The FutureGateway Framework

1 The FutureGateway FrameworkRiccardo Bruno INFN Catania ...

Author: Osborne Fowler

0 downloads 4 Views

1 The FutureGateway FrameworkRiccardo Bruno INFN Catania WACREN e-Research Hackfest

2 Outline Introduction Implementation Authentication and AuthorizationFutureGateway APIs API Usage Installation FutureGateway database 2

3 Introduction

4 FutureGateway (FG) A framework to build Science Gateways:"A Science Gateway is a community-developed set of tools, applications, and data that is integrated via a portal or a suite of applications, usually in a graphical user interface, that is further customized to meet the needs of a specific community." TeraGrid-XSEDE Serves Community Tools applications and data integration Provides GUI (Web, desktop and mobile applications) Customization FutureGateway is the evolution of another product named: Catania Science Gateway Framework (CSGF) FG improves CSGF from well identified weakness especially those things addressing the learning curve 4

5 FG – Design principles Ease the installation and the maintenanceIt provides installation and maintenance scripts All sources available on GitHub Easily customizable for different community needs Flexible and uniform access to distributed computing services The physical access to the distributed infrastructures comes through the JSAGA library (Grid&Cloud Engine) FG executor interfaces (ToscaIDC) Provide a restFull API Back-end portal independency. The FG can operate with any existing portal technology or even linked to mobile and desktop applications Ease SG application development. The FG can be easily piloted by any programming language and/or web framework 5

6 FG - Concepts Tasks Applications InfrastructuresAny activity targeting a DCIs is a task; this ranges from simple batch executions to more sophisticated actions like a PaaS creation and exploitation. Applications It describes the activity to perform against the DCI. Tasks are applications instances. Infrastructures Describe the environment where the application can run. Normally is a resource description necessary to JSAGA adaptors and/or Executor Interfaces to physically access DCIs.

7 API Specifications Available at: http://docs.fgapis.apiary.io/#Endpoints: tasks/applications/infrastructures Infrastructures (not yet available) No specs for users/groups/roles

8 Portlet/WebApp/Web+AjaxFG - Typical usages Liferay Portal APIServer FrontEnd/Daemon Community Portal Portlet Portlet/WebApp/Web+Ajax Final Users Mobile&Desktop Apps Requires (INDIGO) AAI PrivateNet/Firewall/SSL Protection Used for: Molecular Dynamics Climate change INDIGO SG Demonstrator https://sgw.indigo-datacloud.eu New Galaxy portlet Desktop & Mobile apps (Coming soon) The portal manages user access to portlets. CSGF-like setup PTV 8

9 FG Components API Front-end APIServer PortalSetupAccepts API calls in accordance with FG specs Fill-up a queue table of corresponding commands (producer) Manage authN/Z (users/groups/roles) Manage applications Manages database patching mechanism A Python implementation exists: fgAPIServer APIServer Polls over queue table (consumer) Extract tasks to submit Check status and consistency of submitted tasks Retrieve available outputs More daemons may be developed for other languages APIServerDaemon (Actual java implementation) Other implementation (python, …) PortalSetup Manage installation procedures for fgAPIServer, APIServerDaemon and eventually a Liferay Portal installation

10 Queue Daemons and Executor InterfacesAPI Server Queue Daemons and Executor Interfaces Action on DCI Queue Command EI name EI (Action) Action APIServerDaemon/Other daemons Commands (Tasks=Command(Action,EI)) are extracted from the front.end queue Each ‘command’ contains the ‘Target Executor’ field which specifies the Executor Interface name Executor interfaces are dynamically instantiated by the APIServer by its name, applying the specified action on DCI Other queue daemons may extract commands from the queue having their own EIs implemented. Targeting for instance other SAGA implementations or even other systems. New EIs can be easily developed just implementing an abstract class (APIServerDaemon) Current available EIs (APIServerDaemon): GridEngine (A core component of the CSGF using JSAGA and targeting: ssh, rOCCI and wms) SimpleTosca -> ToscaIDC (Indigo orchestrator) 10

11 Architecture (simple view)API Front-end APIServerDaemon GridEngine If. Infrastructures Task queue Executor Interfaces JSAGA Adaptors Mobile and Web apps REST MySQL Applications Tosca If. Other If. GridEngine JSAGA Executor Interfaces Software components responsible to target a specific distributed infrastructure Database Manage the task queue Keep SG application definition Keep application infrastructure parameters Stores information about SG users and their groups and roles xSAGA/Other … Users,Groups and Roles 11

12 Implementation

13 Architecture (current implementation)JSAGA Adaptors Grid and Cloud Engine (GridEngine) TOSCA 1st EI (SimpleTosca) TOSCA 2nd EI (ToscaIDC) Other Interfaces … (…) JSAGA SSH EMI/gLite rOCCI TOSCA Distributed Computing Infrastructures Orchestrator Mid Layer Components DB Interface APIServer DB (fgapiserver) Queue table (as_queue) APIServer (APIServerDaemon) APIServer frontend (fgAPIServer) Executor Interfaces Graphic User Interfaces (Web, Mobile and Desktop applications) REST APIs – APIServer specs. V1.0 rOCCI CLI EMI/gLite APIs ganymed-ssh-2

14 fgAPIServer (front-end)Available on GIT: https://github.com/indigo-dc/fgAPIServer Written in python using Flask microframework This component listens any FutureGateway API REST call in compliancy with specs defined at: This service may run as: Standalone service (Normally under a screen section) (Good for development environments or small requests traffic rate) WSGI application (Suggested for production environments and high requests traffic rate) Different possible configurations: Apache, uWSGI, … The front-end uses a MySQL database to store: Tasks, Applications and Infrastructure with its related data Users/Groups/Roles, Log and Access tokens The task queue 14

15 APIServerDaemon (APIServer)Servlet that runs a daemon on top of Tomcat application server The Java application was necessary since JSAGA is available only via java language. APIServer daemon polls over the task queue table Initially developed to offer a backward compatibility with existing CSGF portals Polling timing and other settings can be configured by a dedicated .properties file APIServerDaemon reads tasks requests from the queue, book them as ‘to process’ and then instruct the correct executor interface for real processing This service timely checks executed task with a simple consistency check algorithm. It re-tries failed requests up to a fixed amount of times. FAILED requests can be reported to the administrator It timely verifies tasks status unitl their termination. The APIServerDaemon manages the task output and updates the DB task tables accordingly 15

16 APIServerDaemon (ControlPanel)This feature exists but is not yet consolidated, it aims to provide a complete overview of daemon activity and eventually configure/manage it https://:/APIServerDaemon/ 16

17 Authentication and Authorization

18 AuthN/Z Baseline CSGF (Shibboleth) Indigo-dc (IAM)Very simple approach Close to the OAuth Thought for environments having no AAI at all Thought to be easily customized limiting changes to just one function CSGF (Shibboleth) The portal manages the AAI A PTV service must be implemented Portal users mapped to FutureGateway API users Indigo-dc (IAM) PTV service has been implemented Mobile and desktop applications contact the portal first

19 FutureGateway Baseline AuthN/Z Log-In APIServer Front-End 1 3 2Username, Password Log Token 1 APIServer Front-End Token check Key 3 Key Log Token Session Token 2 User details User Group(s) Group(s) roles Username Password Timestamp Key … 4 Session Token identifies the user and provides AuthZ informtaiotn Token Expiration Log-In The user log-in providing its own credentials; in the baseline this uses (username and password). This log-in can be handled also with OAUTH authentication (INDIGO AAI, Facebook, Google, …) 1 2 LogToken The baseline Log-In code encrypts username, password and a timestamp in a LogToken string. In OAUTH cases the OAUTH token will be returned after the authentication 3 Token check Incoming LogTokens are processed by the TokenCheck function which in the baseline authentication just maps decrypted credentials with stored users in APIServer DB In OAUTH cases the TokenCheck function will use received user info to map the user with users registered into the APIServer DB. A new session token for mapped user is returned 4 Session Token Session Token will be used to call any further API REST call till token expiration 19

20 AuthN/Z in CSGF IdP eTokenserver Portal LDAP APIServer Front-end AuthN1 LDAP AuthZ 4 R Robot proxy 2 APIServer Front-end 3 5 Resources To be implemented R Validity Check (PTV) AuthN The user is redirected to IdP Login page 1 AuthZ User credentials are retrieved from authenticated users 2 3 Token The portal releases a Token that will be used to contact the APIServer. The API server needs a validy check (PTV) that identifies the Scope as well 4 5 Robot Proxy The APIServer has the responsibility to generate and use Robot-Proxy to access the resources 20

21 FG AuthN/Z in INDIGO IAM Portal APIServer Front-endhttps://www.indigo-datacloud.eu/documents/software-architecture-and-work-plan-wp6-d61 Portal 1 AuthN IAM AuthZ 2 AuthN/Z APIServer Front-end 3 4 Resources Validity Check (PTV) To be implemented IAM The user is redirected to IAM Login page 1 Token Once authenticated the user receives also AuthZ information, stored inside the Token 2 3 Portal and APIServer The portal contacts the ApiServer providing the IAM Token and the API front-end checks the validity of the incoming Token. The Scope and Policies will be applied accordingly to stored AuthZ information Resources IAM Token will be used to access the Resources 4 21

22 Portal Token Validation (PTV)A service normally provided by a Portal which verifies Tokens received by the API server PTV needs HTTP basic authentication to be contacted Base64 encoded username/password to access the service PTV settings are configurable API Server Front-end PTV usage Accept a POST method with the parameter: token="" Answers a JSON saying if given token is valid and the associated portal user (OAUTH2 subject) APIServer front-end uses PTV answer to map portal users with FG users/groups through a configurable mapping file. PTV inside Indigo-dc project, uses LiferayIAM service Liferay7 modula vailable on Git at https://github.com/indigo-dc/LiferayIAM Other PTV services under development (MD use case)

23 FG implementations so farOfficial demonstrator https://sgw.indigo-datacloud.eu Presented at the EGI Community forum in Bari 10-13/11/2015 Helloworld application (Grid&CloudEngine+SSH adapt.) ENES Portlet (Grid&CloudEngine+SSH adapt.) Molecular Dynamics Presented at the CloudScape 02/2016 in Brussels Uses rOCCI JSAGA adaptor with Grid&Cloud Engine on top of a StackServer with Docker based compute node New version will use TOSCA orchestrator and portal integrated with indigo-dc IAM Climate Change (Ophidia+Kepler) Hosted in Poznan, Kepler workflow engine exploits FG APIs to execute jobs on Virtual Appliances Uses SSH JSAGA adaptor with Grid&Cloud Engine Galaxy Hosted in Poznan 23

24 FutureGateway APIs

25 API calls overview Main aim of FG APIs is the task managementA task is any activity a user send to an infrastructure: Run a job in a grid computing site Deploy a VM in a cloud Instantiate a PaaS Execute a job inside a VM, etc. APIs manage: Task collections/Tasks Application collections/Applications Infrastructure collections/Infrastructure currently managed by DB configurations Users/Groups/Roles 25

26 API Server logic Tasks are ‘application’ instances Applications defineExecutable/Infrastructure I/O files Arguments Infrastructures Infrastructure Set of (key,value) pairs, used by Executor Interfaces, defining infrastructure’s access parameters In JSAGA define adaptor requested settings For Grid&Cloud Engine it also contains eTokenServer parameters (X509 robot’s proxy certificates) TOSCA EI foresees the Tosca orchestrator address Users/Groups/Roles Roles are assigned to Groups Users are assigned to Groups Each API call is checked accordingly to the usuer rights 26

27 Conventions Use of ‘curl’ command line toolComplete, powerful and useful tool to send HTTP/HTTPS requests API examples do not use tokens Token management can be switched off from APIServer Daemon front-end configuration file together with the default user It is recommended to use the ‘jq’ command line tool Very useful tool to filter out JSON outputs Produce colorized and human readable outputs 27

28 REST call format Typical API call: Supported HTTP methods:curl –H ”

" [curl options] –X ://:/v1.0/?param_1=value_1&…¶m_n=value_n Supported HTTP methods: GET,POST,PATCH,DELETE Inputs and outputs are in JSON format Any output contain “_links” field helping clients to manage entities. Baseline AAI makes use of Tokens. The Portal Token Valdation service aims to support other AAI systems; such as CSGF AAI and INDIGO IAM. PTV uses Tokens. Tokens are placed in the Header settings -H ”Authentication: Bearer ” 28

29 URL filters In the API URL can be used the ‘user’ filter to execute API call on behalf of another user: User right impersonate_user must be enabled When listing tasks/applications, placing ‘user=*’ will show tasks of all users When listing tasks/applications, placing will show tasks of the user groups User and Group filtering requires special roles user_impersonate group_impersonate If no user is specified. APIServer front-end will use the default user specified in configuration file or the user associated to the provided Token 29

30 Task management – List tasks (GET)List all tasks: curl { "tasks”: [{ id, user, status, description, date, last_change, input_file=[{status,name},], output_files=[{url,name},], application=, arguments=[“arg_1”,…”arg_n”], runtime_data=[{ "creation": ””, "last_change": ” "name": ””, "value": ””, "description": ””},...]],..}, _links=[href,rel] }]} You may use this call to retrieve the tasks execution statuses 30

31 Task management – List user/group tasksList user’s tasks: curl ?user= Where: user = username|* (all user groups List user’s tasks and application: curl ?user=&application= 31

32 Task management – List a given taskView specific task details: curl { id, user, status, description, date, last_change, input_file=[{status,name},], output_files=[{url,name},], application=, arguments=[“arg_1”,…”arg_n”], [runtime_data=[{ "creation": ””, "last_change": ””, "name": ””, "value": ””, "description": ””},...]] _links=[href,rel] } You may use this call to retrieve the task execution status 32

33 Task management – Submit (POST)Submit a task: curl -H "Content-Type: application/json" -X POST -d '{"application":"3","description":"tosca test run" }' Two possible answers: The job goes directly to the execution queue (no input, or pre-configured inputs) Returned JSON informs that task has been triggered for execution The job waits for input files A task detail JSON will be returned 33

34 Upload task files Push job into the queue with input callAccept default files: curl -X POST id>/input?user=brunor Specify input files: curl -X POST -F -F id>/input Input file management is configured during the application installation

35 Task get output When task status is DONE, from task detail JSON output: { ..., "output_files": [ { "url": "file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stdout.txt", "name": "stdout.txt" }, "url": "file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stderr.txt", "name": "stderr.txt" } ], ...} curl "http://localhost:8888/v1.0/file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stdout.txt" fgtest-2del3.cloud.ba.infn.it In this case a plain text content is returned 35

36 Delete task curl -X DELETE This call permanently removes any task reference from the DB as well as the associated task directory in the APIServer front-end file system DELETE action should not be performed by final users directly but rather by an administrator users 36

37 Runtime data and status change (PATCH)Tasks can save runtime information using runtime data feature curl -i -H "Content-Type: application/json" -X PATCH -d '{"runtime_data": [ { "data_name": "test_data", "data_value": "test_value", "data_desc": "test description value"} ]}' In Task details runtime data will be reported as: Users may enforce status change with: curl -i -H "Content-Type: application/json" -X PATCH -d '{"status": "CANCELLED" }' This call must be foreseen by the Executor Interface (GridEngine does not support it) "runtime_data" : [ { "data_name": "name" ,"data_value": "value" ,"data_desc": "description of the value” }, ... ]

38 Applications List (GET)List all applications curl List the app having the given id curl

39 Application installation (POST)Curl -H "Content-Type: application/json” -X POST -d '{ "outcome": "JOB", "description": "hostname test application", "name": "hostname_test", "enabled": true, "parameters": [ { "description": "", "value": "GridEngine", "name": "target_executor” }, "value": "/bin/hostname", "name": "jobdesc_executable” }, "value": "hostname_test_output.txt", "name": "jobdesc_output” }, "value": " hostname_test_error.txt ", "name": "jobdesc_error” } ], Please notice that infrastructures are referenced directly and not using IDs. This because API calls for infrastructure haven’t been implemented yet "infrastructures": [ { "name": (SSH)", "virtual": false, "enabled": true, "parameters": [{ "value": "ssh://localhost:22", "name": "jobservice” }, { "value": "jobtest", "name": "username” }, { "value": "4DuWEMCHy6Nk”, "name": "password” }], "description": "infrastructure for hostname_test"}] }' 39

40 FutureGateway API Usage40

41 Generic considerationsTill now the use of curl has been shown; FG REST APIs may be called by other sources; in particular: AJAX Liferay portlets The use of REST APIs opens to many other programming languages and/or web architectures Many infrastructures require the access to X509 proxy certificates; a VPN service allow to access the eTokenServer service wich provides Robot Proxy certificates (Grid&Cloud Engine) 41

42 AJAX REST calls can be executed from interactive HTML5 pages using AJAX $.ajax({ url: webapp_settings.apiserver_proto+'://' +webapp_settings.apiserver_host +':' +webapp_settings.apiserver_port +webapp_settings.apiserver_path +'/' +webapp_settings.apiserver_ver +'/tasks?user=' +webapp_settings.username, type: "POST", cache: false, dataType: "json", contentType: "application/json; charset=utf-8", data: JSON.stringify(job_desc), success: function(data) { ... }, error: function(jqXHR, textStatus, errorThrown) { ... } See HelloTester example on GIT: https://github.com/FutureGateway/HelloTester 42

43 Liferay Portlet Setup FG contextualization script to install Liferay and ensure LiferaySDK flag is enabled inside setup_config.sh script Liferay Development environment may be installed Liferay 6.2 cd /portlet Execute ./create.sh # then follow the instruction Add the portlet java class in /src/main/webapp/WEB-INF/portlet.xml (tag: portlet-class) In JS page, extract the user name: username : '<%= user.getScreenName() %>’ Use AJAX calls to interact with FG APIs. REST calls may be managed by portlet’ java code but it is not suggested HelloTester’ index.html contains a javascript that could be used as a template file Liferay 7 provides a CLI tool named: blade (see Liferay 7 documentation) Interfaces now in OSGI standard 43

44 Client APIs Although REST APIs keep developer free to implement their own application, several API clients have been developed or in development for the following languages: Java A java class containing methods for all API calls Python fgPyClient; a single class containing methods for all API calls PHP Set of PHP scripts, each for a single API call related to task submission/status/output retrieval

45 eTokenServer and VPN Several adaptors needs to specify X509 proxiesThe GridEngine foresees the use of robot certificates querying the eTokenServer service eTokenServer can be accessible only opening its firewall, enabling a VPN connection or exchanging a certificate To get VPN certificate or eToken certificate please contact: Once obtained the VPN certificate, extract its zip content; chmod to 400 the p12 file received after requesting the VPN access Run openvpn client: sudo openvpn .ovpn While openvpn client works, the host can contact the eTokenServer service Once openVPN works, connect your SSH enabling the same VPN from your local machine 45

46 Installation

47 Installation GITHub: https://github.com/FutureGateway/PortalSetupCore scripts (setup_*.sh) Support: EL5/6/7, deb (Ubuntu), Mac OS X The setup_config.sh file contains global configuration options, then execute setup_FGPortal.sh as ‘futuregateway’ user while remaining setup_*.sh scripts as sudo/root During installation, some binaries are downloaded from please verify its availability first FGRepo.tar.gz is not mandatory, but accelerates installation time APIServerDaemon_lib.tar.gz is mandatory and contains libraries that could not fit in GitHub project (until maven build will be not available) Volunteers are requested to test/develop scripts for different platforms Ubuntu Server LTS Use of core scripts to install on Ubuntu LTS fgSetup.sh – Installs everything just executing it docker-setup.sh – Uses script above to install standard Docker Ubuntu LTS image EGI-FedCloud Using the ‘userdata.txt’ contextualization file, it works on FutureGateway EGI appDB virtual appliance or any other Ubuntu LTS server. It uses fgSetup.sh 47

48 FG Installation on Ubuntu Server 14.04 LTSInstall from a stand-alone installation Extract from GITHub installation file: https://raw.githubusercontent.com/FutureGateway/PortalSetup/master/Ubuntu_14.04/fgSetup.sh Give execution rights and execute the installation file: ./fgSetup.sh futuregateway futuregateway $(cat /root/.ssh/id_rsa.pub) Before to start the execution, other configuration have to be performed and explained in the README.md file in the GITHub page: https://github.com/FutureGateway/PortalSetup/blob/master/Ubuntu_14.04/README.md Install from a docker container Extract from GITHub installation file: https://raw.githubusercontent.com/FutureGateway/PortalSetup/master/Ubuntu_14.04/docker-setup.sh Give execution rights and start the setup script # ./docker-setup.sh 48

49 FG Installation using EGI-FedCloudUsing CLI or the UI the necessary steps are: Get the ‘userdata.txt’ contextualization file, customize its content accodingly to personal needs Get your proxy with fedcloud.egi.eu VO or vo.indigo-dc.eu Select one of the available endpoints supporting the FutureGateway appliance: https://appdb.egi.eu/store/vappliance/futuregateway The contextualization script can work on other existing Ubuntu server LTS based appliances for instance (tested): Image for Ubuntu Server LTS [Ubuntu/14.04 LTS/KVM]_EGI_fedcloud FutureGateway Image (A flat Ubuntu Server LTS) Select a suitable resource template (no particular needs are requested; 1CPU, 1GB of memory is enough) 49

50 EGI FedCloud Contextualization#cloud-config hostname: futuregateway manage_etc_hosts: true package_update: true package_upgrade: true users: - name: futuregateway groups: sudo shell: /bin/bash sudo: ['ALL=(ALL) NOPASSWD:ALL'] lock-passwd: true ssh-import-id: futuregateway ssh-authorized-keys: - ssh-rsa … - path: /root/installFG.sh permissions: "0755" owner: "root" content: | #!/bin/bash # # installFG.sh - Download, configure and execute the fgSetup.sh script # Enable the installation flag, FutureGateway is installing # Configure the appliance options below: FGLIFERAY=0 # Use 0 to skip liferay installation FGREBOOT=0 # Use 1 to reboot after installation, thus FG service will start FGSSH_PORT="2424" # User can specify any ssh port (use 22 for default) FGTOMCAT_USR=”… " # Tomcat administration user name FGTOMCAT_PAS=”… " # Tomcat administration password # Reconfigure SSH port as requested … Place your public SSH keys FGLIFERAY – Skips or not Liferay portal installation. In most cases communities have already a portal FGEREBOOT – Rebooting after context scritp means the FG server will start automatically FGTOMCAT_(USR/PAS) – Tomcat admin user an password 50

51 Customizing fgSetup.shContextualization script creates the file installFG.sh This script contains instructions to customize entries into fgSetup.sh script In particular the fgSetup.sh script contains the instructions to generate the core script: setup_config.sh, which contains any configurable variable used by other setup_* core scripts. More than existing customizations can be be introduced by the user changing the contextualization script, configuring properly the following lines: # Configure the fgSetup.sh script # # Following lines configure the setup_config.sh file contained inside the fgSetup script # You can configure any option just introducing the following commands: # mv fgSetup.sh fgSetup.sh_orig # Make a safe copy of existing fgSetup.sh script # cat fgSetup.sh_orig | sed s/// > fgSetup.sh # set the new value As an example, it is possible to see how SSH port and TOMCAT admin USR and PAS are managed Once configured the fgSetup.sh script the customization script executes the command which starts the installation: runcmd: - /bin/bash /root/installFG.sh 2>/root/install.err > /root/install.out The same way to generate fgSetup and execute it can be exploited to generate other OSes full automatic installations. Volunteers are very welcome to perform such kind of tests. Once the installation completes a flag file will be removed in: /home/futuregateway/.installingFG 51

52 FutureGateway service managementOnce installation scripts complete their job the futuregateway is ready to start Futuregateway has a service control script file: /etc/init.d/futuregateway Usage: /etc/init.d/futuregateway The control script manages two components: The futuregateway REST API front-end This opens a screen section under ‘futuregateway’ user and starts the the python app as stand-alone application The futuregateway APIServerDaemon This starts a Tomcat session and its APIServerDaemon web aplication installed The REST API server front-end should be configured as a wsgi process; in such case futuregateway control script has a flag to ignore the front-end switching of its flag: ENABLEFRONTEND 52

53 Monitoring FutureGatewayREST API Front-end screen –ls There is a screen on: XXXXX.fgAPIServer (…) (Detached) screen XXXXX.fgAPIServer # Attaches to the screen session, use ^C to stop fgapiserver.py execution In the execution dir exists file ‘fgapiserver.log’ (not really used yet) APIServerDaemon Start Tomcat using a defined scrpt: ‘start_tomcat’ Stop Tomcat use defined scripts: ‘stop_tomcat’ and ‘killkava’ that kills any tomcat JAVA process it it persists Watch Tomcat logs with: tail –f $CATALINA_HOME/logs/catalina.out Watch APIServerDaemon logs with: tail –f $CATALINA_HOME/webapps/APIServerDaemon/WEB-INF/logs/APIServerDaemon.log Watch GridEngine logs with: tail –f $CATALINA_HOME/webapps/APIServerDaemon/WEB-INF/logs/GridEngineLog.log SimpleTosca interface logs are included into the APIServerDaemon log APIServer DB mysql -h localhost -P u fgapiserver -pfgapiserver_password fgapiserver Use: asdb|utdb ”” commands to execute a query against APIServer DB or GridEngine’ UsersTracking database; if no argument is given, the interactive mysql client connected to the DB will be available. Setting up variable ASDB/UTDB_OPTS it is possible to configure mySQL client, for instance: ‘ASDB_OPTS=“-N –s”’ 53

54 First test: ‘helloworld’ (SSH)Standard installation contains a working demo example named: ‘helloworld’ and exploits the SSH JSAGA adaptor that points to ‘itself’. cd $FGLOCATION/fgAPIServer/apps/helloworld Open using a text editor file stress_test.sh it contains the REST calls to execute the demo app Configure its values: SUBMIT_COUNT and SUBMIT_DELAY Execute the test: ./stress_test.sh A good practice is to maintain active a tail –f process on APIServerDaemon.log file during the execution Execution can be monitored on APIServer log file or querying the DB ‘asdb "select * from as_queue;"' watching for the ‘status’ field (Your task should be 1 the first time) As soon as the DONE status is reached you can get task info with: curl In the output json paths to output/error files are reported: "output_files": [ { "url": "file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stdout.txt", "name": "stdout.txt” }, To get output file just typing: curl "http://localhost:8888/v1.0/file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stdout.txt” 54

55 SayHello (SSH) A second and more complete ‘sayhello’ test exists and it still exploits JSAGA SSH adaptor showing the use of IOSandboxing cd $FGLOCATION/fgAPIServer/apps/sayhello Open using a text editor file stress_test.sh it contains the REST calls to execute the demo app Configure its values: SUBMIT_COUNT and SUBMIT_DELAY Execute the test: ./stress_test.sh Curl commands in the script shows how to send input files and how to declare output files to retrieve Please feel free to modify this example and make more practice with FG APIs 55

56 Applications Applicaton files are normally under $FGLOCATION/fgAPIServer/apps/ Hostname and SayHello are baseline applications useful for testing new installations Other applications are available on Git inside them following files exist: setup_app.sh; A shell script that installs the application. This file can be used to create new application installations stress_test; A shell script able to submit automatically a set of tasks related to the application Other files; are related to the application input files 56

57 Config APIServer front-end 1/2A configuration file can configure the behavior of the APISercer fron-end [fgapiserver] fgapiver = v1.0 fgapiserver_name = FutureGateway API Server fgapisrv_host = fgapisrv_port = 8888 fgapisrv_debug = True fgapisrv_iosandbox = /tmp fgapisrv_geappid = fgjson_indent = 4 fgapisrv_key = fgapisrv_crt = fgapisrv_logcfg = fgapiserver_log.conf fgapisrv_dbver = fgapisrv_secret = ABCDEF fgapisrv_notoken = True fgapisrv_notokenusr = test fgapisrv_lnkptvflag = False fgapisrv_ptvendpoint= fgapisrv_ptvuser = tokenver_user fgapisrv_ptvpass = tokenver_pass fgapisrv_ptvdefusr = futuregateway fgapisrv_ptvmapfile = fgapiserver_ptvmap.json Host and listening port (unused with WSGI) Task IO file directory GridEngine UsersTraking application Id (fixed for APIServer) Certificate files for https:// connections Token management values APIServerDB settings PTV users/groups mapping file 57

58 Config APIServer front-end 2/2Database section # geapiserver database settings [fgapiserver_db] fgapisrv_db_host = localhost fgapisrv_db_port = 3306 fgapisrv_db_user = fgapiserver fgapisrv_db_pass = fgapiserver_password fgapisrv_db_name = fgapiserver APIServerDB connection settings 58

59 WSGI configuration example (mod_wsgi, apache) ServerName sgw.indigo-datacloud.eu ServerAdmin DocumentRoot /var/www/html ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined SSLEngine on SSLCertificateFile /etc/ssl/certs/sgw_indigo-datacloud_eu.crt SSLCertificateKeyFile /etc/ssl/private/sgw_indigo-datacloud_eu.key SSLCertificateChainFile /etc/ssl/certs/DigiCertCA.crt WSGIDaemonProcess fgapiserver user=futuregateway group=futuregateway processes=5 threads=10 home=/home/futuregateway WSGIProcessGroup futuregateway WSGIScriptAlias /apis /home/futuregateway/FutureGateway/fgapiserver/fgapiserver.wsgi WSGIProcessGroup fgapiserver WSGIApplicationGroup %{GLOBAL} Order deny,allow Allow from all Options All AllowOverride All Require all granted 59

60 APIServerDaemon configurationapisrv_dbhost = localhost apisrv_dbport = 3306 apisrv_dbuser = fgapiserver apisrv_dbpass = fgapiserver_password apisrv_dbname = fgapiserver asdMaxThreads = 100 asdCloseTimeout = 20 gePollingDelay = 4000 gePollingMaxCommands = 5 asControllerDelay = asControllerMaxCommands = 5 # GridEngineDaemon task retry policies asTaskMaxRetries = 5 asTaskMaxWait = # # GridEngine UsersTracking Database settings # Specifying the following JNDI the GridEngine will use the # following connection pool utdb_jndi = jdbc/UserTrackingPool # Not specifying the JNDI as below GridEngine UTDB will be # contacted using following connection settings #utdb_jndi = utdb_host = localhost utdb_port = 3306 utdb_user = tracking_user utdb_pass = usertracking utdb_name = userstracking APIServer DB Polling settings GridEngine JNDI for UsersTrackingDB GridEngine UsersTrackingDB (no longer used, causes problems) 60

61 Database migration A very basic migration helper system exists to keep APIServer DB structure aligned to the latest code changes The migration system relies on a patching mechanism available under: $FGLOCATION/fgAPIServer/db_patches Under the patches directory several scripts will be in charge to apply needed patches. Only one file must be executed to update the DB: ‘patch_apply.sh’ Once aligned the database to the latest version; it will be possible to upgrade easily the code using the updateCode tool Warning message on log files informs about necessary changes to apply as root or su privileges !!! Check log files after execution 61

62 Code changes and maintenanceCode updates are easy to integrate Stop futuregateway service Go to the: $FGLOCATION/fgAPIServer $FGLOCATION/APIServerDaemon Adaptors $FGLOCATION/rOCCI $FGLOCATION/jsaga-adaptor-tosca (SimpleTosca) … next adaptors Execute: git pull (to get latest version) Only JAVA components need the built with: ‘ant all’ command APIServerDaemon: Place generated war file in $CATALINA_HOME/webapps/ then watch the TOMCAT log file (catalina.out) Adaptors: Produce a jar that have to be placed in: $FGLOCATION/APIServerDaemon/web/WEB-INF/lib/ ; Then recompile and reisntall APIServerDaemon (see step above) $FGLICATION/jsaga-1.1.2/lib/ 62

63 FutureGateway databae

64 APIServer DB APIServerDaemon application application_flileapplication_parameter infrastructure Infrastructure_parameter task task_arguments task_input_files Task_output_files runtime_data as_queue APIServerDaemon Application instances are tasks Tasks are executed when referenced in the API Server queue (as_queue) Anything can run on a distributed infrastructure 64

65 Application app_id file_id file path override id name param_idapplication_flile application_parameter infrastructure Infrastructure_parameter id name description creation enabled app_id file_id file path override param_id pname pvalue infra_id Pvalue Instruct JSAGA adaptor using couples (keyname, Keyvalue). Executable Input Ouput Target executor Pre-defined input files file = name of the file path = where the file is located override = if true the file cannot be overwritten by the user using ‘input’ REST call 65

66 Task task_id arg_id argument Id creation file_id last_change filetask_arguments task_input_file Id creation last_change app_id description status iosandbox user task_id arg_id argument file_id file path Executable Input Ouput Target executor Arguments application task_output_file as_queue 66

68 application_parameterjobdesc_executable What will be executed on the remote infrastructure jobdesc_arguments Argument list; (it works in conjunction with task_arguments table) jobdesc_output output file jobdesc_error error_file target_executor Which APIServerDaemon interface will be in charge to execute this task (GridEngine, SimpleTosca, …) 68

70 Infrastructures Infrastructure table should be called: ‘application infrastructure’ It just link applications with infrastructures Infrastructure records are pointing to infrastructure_parameters infrastructure_parameters table keep infrastructure specific settings requested by Executor Interfaces to manage the distributed infrastructure Application

71 Infrastructure parameters Configuration examplesGrid&Cloud Engine supports many JSAGA adaptors; the ones implemented and tested on the FG are: SSH (Remote hosts, clusters, …) rOCCI (Generic clouds, EGI FedCloud, …) wms (EMI-gLite Grid Infrastructures) ToscaIDC Executor Interface 71

76 Users Groups and Roles Users may belong to one or more groupsRoles are linked to Roles Tables: fg_users fg_group fg_role Link tables: fg_group_apps (Important during app installation) fg_group_role (Link roles to groups) Fg_user_group (Link users to groups)

77 Available roles app_install -- Install an application app_change -- Modify an application app_delete -- Delete an application app_view -- Run an application app_run -- Run an application infra_add -- Add an infrastructure infra_change -- Change infrastructure infra_delete -- Delete an infrastructure infra_view -- View an infrastructure infra_attach -- Attach an infrastructure to an application infra_detach -- Detach an infrastructure from an application task_delete -- Delete a task task_view -- View a task task_userdata -- Manage userdata on task user_add -- Can add users user_del -- Can remove users user_change -- Can change users user_impersonate -- Can impersonate any other users group_impersonate -- Can impersonate other users in the same group 77

78 AAI Belongs to the AAI the following tables:as_queue; keep track of task fg_user; keep track of the user specified in task table fg_token; keep track of user tokens together with PTV subject field

79 asdb/utdb Utils ‘asdb’ and ‘utdb’ are two utilities to access respectively: APIServer Database GridEngine’ UsersTrackingDB Providing the command without args a mysql client interactive session will be instantiated Providing as argument a SQL statement it will be executed on the DB Setting up variables (ASDB/UTDB)_OPTS it is possible to specify Mysql client options Example: asdb "select count(*) from as_queue” | count(*) | | | export ASDB_OPTS="-s –N” 5 79

80 Questions ? 80

The FutureGateway Framework

Recommend Documents