1 FutureGateway APIs Riccardo Bruno INFN Catania e-Research Summer Hackfest

2 Outline Introduction Database Tasks Applications Token managementSpecifications APIs considerations REST call format Database Tables overview Tasks API operations on tasks Applications Application configuration API operations on applications Token management Baseline AuthN/Z Portal Token Validation API Usage 2

3 Specifications Defined in 8/2015Specs. available at: 3

4 API calls overview Main aim of FG APIs is the task managementA task is any activity a user send to an infrastructure, such as: run a job in a grid computing site or deploy a VM in a cloud, instantiate a PaaS, execute a job inside a VM, etc. APIs manage: Task collections Task Application collections Application Infrastructure collections Infrastructure (managed by DB configurations yet) Users/Groups/Roles To be implemented (managed by DB configurations now) 4

5 API Server logic Tasks are ‘application’ instances Applications defineExecutable/Infrastructure I/O files Arguments Infrastructures Infrastructure Set of (key,value) pairs, defining infrastructure’s access parameters In JSAGA define adaptor requested settings For Grid&Cloud Engine it also contains eTokenServer parameters (X509 robot’s proxy certificates) Users/Groups/Roles Roles are assigned to Groups Users are assigned to Groups 5

6 REST call format Supported methods:://:/v1.0/?param_1=value_1&…¶m_n=value_n Supported methods: GET,POST,PATCH,DELETE Inputs and outputs are in JSON format Any output contain “_links” field helping clients to manage entities. REST URLs should not be built by client code A baseline Token management provides a customizable basic AAI system. It also provides the Portal Token Valdation written to support INDIGO AAI Users can be also managed by APIs through the parameter user= (useful when operating “on behalf of”) 6

7 Conventions Use of ‘curl’ command line toolComplete, powerful and useful tool to send HTTP/HTTPS requests Use of ‘jq’ command line tool Very useful tool to filter out JSON outputs API examples do not use tokens Token management can be switched off from APIServer Daemon front-end configuration file (default user) 7

8 Database 8

9 APIServer DB APIServerDaemon application application_flileapplication_parameter infrastructure Infrastructure_parameter task task_arguments task_input_files Task_output_files runtime_data as_queue APIServerDaemon Application instances are tasks Tasks are executed when referenced in the API Server queue (as_queue) Anything can run on a distributed infrastructure 9

10 Application app_id file_id file path override id name param_idapplication_flile application_parameter infrastructure Infrastructure_parameter id name description creation enabled app_id file_id file path override param_id pname pvalue infra_id Pvalue Instruct JSAGA adaptor using couples (keyname, Keyvalue). Executable Input Ouput Target executor Pre-defined input files file = name of the file path = where the file is located override = if true the file cannot be overwritten by the user using ‘input’ REST call 10

11 Task task_id arg_id argument Id creation file_id last_change filetask_arguments task_input_file Id creation last_change app_id description status iosandbox user task_id arg_id argument file_id file path Executable Input Ouput Target executor Arguments application task_output_file as_queue 11

12 APIServer DB (task queue)mysql> desc as_queue; | Field | Type | Null | Key | Default | Extra | | task_id | int(10) unsigned | NO | PRI | NULL | | | target_id | int(10) unsigned | YES | | | | | target | varchar(32) | NO | | NULL | | | action | varchar(32) | NO | PRI | NULL | | | status | varchar(32) | NO | | NULL | | | target_status | varchar(32) | YES | | NULL | | | creation | datetime | NO | | NULL | | | last_change | datetime | NO | MUL | NULL | | | action_info | varchar(128) | YES | | NULL | | | retry | int(10) unsigned | NO | | | | | check_ts | datetime | NO | | NULL | | The task queue, links to the task table. The queue is managed both from APIServerDaemon target executor interfaces and the APIServerDaemon. The APIServerDaemon intefaces are in charge to update target_id, target_status, they are the responsible to execute task on the remote ifnrastructure 12

13 Configuration examplesTasks 13

14 Task management – List tasks (GET)List all tasks: curl { “tasks”: [{ id, user, status, description, date, last_change, input_file=[{status,name},], output_files=[{url,name},], application=, arguments=[“arg_1”,…”arg_n”], [runtime_data=[{ "creation": ””, "last_change": ””, "name": ””, "value": ””, "description": ””},...]] _links=[href,rel] }]} Use this call also to retrieve the task execution status List user’s tasks: curl ?user= List user’s tasks and application: curl ?user=&application= If no user is specified. APIServer will use the default user specified in configuration file or the user associated to the provided Token (general rule for all API calls) 14

15 Task management – List a task (GET)View specific task details: curl { id, user, status, description, date, last_change, input_file=[{status,name},], output_files=[{url,name},], application=, arguments=[“arg_1”,…”arg_n”], [runtime_data=[{ "creation": ””, "last_change": ””, "name": ””, "value": ””, "description": ””},...]] _links=[href,rel] } 15

16 Task management – Submit (POST)Submit a task: curl -H "Content-Type: application/json" -X POST -d '{"application":"3","description":"tosca test run" }' Two possible answers: The job goes directly to the execution queue (no input, or pre-configured inputs) Returned JSON informs that task has been enqueued (task id) The job waits for input files A task detail JSON will be returned Push job into the queue with input call curl -X POST id>/input?user=brunor curl -i -X POST -F -F id>/input Input file management is configured in application definition (optional) 16

17 Task get output From task detail JSON output:{ ..., "output_files": [ { "url": "file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stdout.txt", "name": "stdout.txt" }, "url": "file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stderr.txt", "name": "stderr.txt" } ], ...} curl "http://localhost:8888/v1.0/file?path=%2Ftmp%2Fd529ee18-030a-11e6-9fd0-fa163e9e678a%2F1tmpd529ee18030a11e69fd0fa163e9e678a_1&name=stdout.txt" fgtest-2del3.cloud.ba.infn.it In this case a plain text content is returned 17

18 Delete task curl -X DELETE This call permanently removes any task reference from the DB as well as the associated task directory (see as_queue.task_info field) 18

19 Applications 19

20 application_parameterjobdesc_executable What will be executed on the remote infrastructure jobdesc_arguments Argument list; (it works in conjunction with task_arguments table) jobdesc_output output file jobdesc_error error_file target_executor Which APIServerDaemon interface will be in charge to execute this task (GridEngine, SimpleTosca, …) 20

21 application_file | app_id | file_id | file | path | override | | 3 | 1 | tosca_template.yaml | /home/…/fgAPIServer/apps/toscaTest | 0 | | 3 | 2 | tosca_test.sh | /home/…/fgAPIServer/apps/toscaTest | 0 | Several applications foresee a fixed number of input files Input files may or not changed by the user using the APIs (tasks/input REST call) The override flag when true ignores user calls to task/input REST call If no input files are specified in the task creation and all application files have true the override flag, the task submission starts after the POST call to tasks/ The execution starts immediately also when no input files records are defined as well as in input_files list in the POST API call 21

22 Application infrastructures Configuration examplesInfrastructure’ parameters for Grid&Cloud Engine executor. Grid&Cloud Engine supports many JSAGA adaptors; the ones implemented and tested on the FG are: SSH (Remote hosts, clusters, …) rOCCI (Generic clouds, EGI FedCloud, …) wms (EMI-gLite Grid Infrastructures) Tosca (INDIGO-DC, S/P/I/aaS) 22

23 JSAGA ssh:// adaptor JSAGA SSH adaptor requires few parameters:| infra_id | param_id | pname | pvalue | | 1 | 1 | jobservice | ssh://localhost:2424 | | 1 | 2 | username | jobtest | | 1 | 3 | password | NDQwM2Y0ZTRh | JSAGA SSH adaptor requires few parameters: Connection URL: ssh://: Username Password The JSAGA Adaptor foresees also the use of ssh keys; this is not yet supported by the GridEngine target executor 23

24 JSAGA rocci:// adaptor | param_id | pname | pvalue | | | jobservice | rocci://nebula-server-01.ct.infn.it: | | | os_tpl | uuid_chain_reds_generic_vm_centos_6_6_kvm_103 | | | resource_tpl | small | | | attributes_title | sayhello | | | eToken_host | etokenserver.ct.infn.it | | | eToken_port | | | | eToken_id | bc681e2bd4c3ace2a4c54907ea0c379b | | | voms | vo.chain-project.eu | | | voms_role | vo.chain-project.eu | | | rfc_proxy | true | X509 Robot proxy generation parameters rocci:// endpoint foresees different parameters :?prefix=&[protocol=|secured=true/false]&action=&attributes_title=&mixin_os_tpl=&mixin_res_tpl=&link=&wait_ms=&waitsshms= 24

25 JSAGA wms:// adaptor Just provide: WMS endpoint BDII | param_id | pname | pvalue | | | jobservice | wms://wms.ulakbim.gov.tr:7443/glite_wms_wmproxy_server | | | bdii | ldap://bdii.eumedgrid.eu: | | | eToken_host | etokenserver2.ct.infn.it | | | eToken_port | | | | eToken_id | bc681e2bd4c3ace2a4c54907ea0c379b | | | voms | eumed | | | voms_role | eumed | | | rfc_proxy | false | X509 Robot proxy generation parameters Just provide: WMS endpoint BDII Robot proxy parameters 25

26 JSAGA tosca:// adaptor | param_id | pname | pvalue | | | tosca_endpoint | tosca:// :80/orchestrator/deployments | | | tosca_token | AABBCCDDEEFF | | | tosca_template | tosca_template.yaml | | | tosca_parameters | wait_ms=30000&max_waits= | Provides: Tosca endpoint Tosca template yaml Tosca parameters wait_ms – how long wait for remote machine creation max_waits – how many attempts before rising the creation timeout 26

27 Application APIs List all applicatoins: Show specific application:curl { id, name, enabled, description, creation, parameters, input_file=[{name,path,override},], infrastrcture=[{name,description,enabled,virtual,parameters=[{pname,pvalue},]},], _links=[href,rel] } Show specific application: curl Create an application: curl -i -H "Content-Type: application/json" -X POST -d '' Delete an application (not yet refined with consistency checks) curl -i -X DELETE 27

28 Application installation exampleCurl -H "Content-Type: application/json” -X POST -d '{ "outcome": "JOB", "description": "hostname test application", "name": "hostname_test", "enabled": true, "parameters": [ { "description": "", "value": "GridEngine", "name": "target_executor” }, "value": "/bin/hostname", "name": "jobdesc_executable” }, "value": "hostname_test_output.txt", "name": "jobdesc_output” }, "value": " hostname_test_error.txt ", "name": "jobdesc_error” } ], "infrastructures": [ { "name": (SSH)", "virtual": false, "enabled": true, "parameters": [{ "value": "ssh://localhost:22", "name": "jobservice” }, { "value": "jobtest", "name": "username” }, { "value": "4DuWEMCHy6Nk”, "name": "password” }], "description": "infrastructure for hostname_test"}] }' 28

29 Configuration examplesToken management baseline and Portal Token Validator (PTV) 29

30 FutureGateway Baseline AuthN/Z Log-In APIServer Front-End 1 3 2Username, Password Log Token 1 APIServer Front-End Token check Key 3 Key Log Token Session Token 2 User details User Group(s) Group(s) roles Username Password Timestamp Key … 4 Session Token identifies the user and provides AuthZ informtaiotn Token Expiration Log-In The user log-in providing its own credentials; in the baseline this uses (username and password). This log-in can be handled also with OAUTH authentication (INDIGO AAI, Facebook, Google, …) 1 2 LogToken The baseline Log-In code encrypts username, password and a timestamp in a LogToken string. In OAUTH cases the OAUTH token will be returned after the authentication 3 Token check Incoming LogTokens are processed by the TokenCheck function which in the baseline authentication just maps decrypted credentials with stored users in APIServer DB In OAUTH cases the TokenCheck function will use received user info to map the user with users registered into the APIServer DB. A new session token for mapped user is returned 4 Session Token Session Token will be used to call any further API REST call till token expiration 30

31 Baseline Log and Session tokensIn fgapiserver.conf configuration file provide a key value Place the same key into the helper tool ‘mklogtoken.py’, specifying requested username and password Use the generated LOG token to get a session token: ://:/v1.0/auth where in the header there exists "Authorization: Bearer ” It is also possible to get directly a session token with: ://:/v1.0/auth?username=&password= In PTV no need of LOG tokens Once obtained the session token, use it on APIs with: curl -H "Authorization: " "http://localhost:8888/v1.0/ 31

32 FG AuthN/Z in INDIGO IAM Portal APIServerhttps://www.indigo-datacloud.eu/documents/software-architecture-and-work-plan-wp6-d61 Portal 1 AuthN IAM AuthZ 2 AuthN/Z APIServer 3 4 Resources Validity Check To be implemented IAM The user is redirected to IAM Login page 1 Token Once authenticated the user receives also AuthZ information, stored inside the Token 2 3 Portal and APIServer The portal contacts the ApiServer providing the IAM Token and the APIServer checks the validity of the incoming Token. The Scope and Policies will be applied accordingly to stored AuthZ information Resources IAM Token will be used to access the Resources 4 32

33 PortalTokenValidator (soon available)It must be enabled from configuration file (available from db_patch > 0.0.6) Token will come from the portal Any API call verifies the portal token through an endpoint exposed by the portal an accessed via username/password (params in conf file) The portal replies with: User authorization (yes/no) User portal name (optional) User portal group (optional) The APIServer will map the portal(name, group) with a FG defined user/group accordingly to a mapping json file fgapiserver_ptvmap.json 33

34 Available roles app_install -- Install an application app_change -- Modify an application app_delete -- Delete an application app_view -- Run an application app_run -- Run an application infra_add -- Add an infrastructure infra_change -- Change infrastructure infra_delete -- Delete an infrastructure infra_view -- View an infrastructure infra_attach -- Attach an infrastructure to an application infra_detach -- Detach an infrastructure from an application task_delete -- Delete a task task_view -- View a task task_userdata -- Manage userdata on task user_add -- Can add users user_del -- Can remove users user_change -- Can change users user_impersonate -- Can impersonate any other users group_impersonate -- Can impersonate other users in the same group 34

35 API Usage 35

36 Generic considerationsTill now the use of curl has been shown; FG REST APIs may be called by other sources; in particular: AJAX Liferay portlets The use of REST APIs opens to many other programming languages and/or web architectures Many infrastructures require the access to X509 proxy certificates; a VPN service allow to access the eTokenServer service wich provides Robot Proxy certificates (Grid&Cloud Engine) 36

37 AJAX REST calls can be executed from interactive HTML5 pages using AJAX $.ajax({ url: webapp_settings.apiserver_proto+'://' +webapp_settings.apiserver_host +':' +webapp_settings.apiserver_port +webapp_settings.apiserver_path +'/' +webapp_settings.apiserver_ver +'/tasks?user=' +webapp_settings.username, type: "POST", cache: false, dataType: "json", contentType: "application/json; charset=utf-8", data: JSON.stringify(job_desc), success: function(data) { ... }, error: function(jqXHR, textStatus, errorThrown) { ... } See HelloTester example on GIT: https://github.com/FutureGateway/HelloTester 37

38 Liferay Portlet Setup FG contextualization script to install Liferay and ensure LiferaySDK flag is enabled inside setup_config.sh script Liferay Development environment may be installed Liferay 6.2 cd /portlet Execute ./create.sh # then follow the instruction Add the portlet java class in /src/main/webapp/WEB-INF/portlet.xml (tag: portlet-class) In JS page, extract the user name: username : '<%= user.getScreenName() %>’ Use AJAX calls to interact with FG APIs. REST calls may be managed by portlet’ java code but it is not suggested HelloTester’ index.html contains a javascript that could be used as a template file Liferay 7 provides a CLI tool named: blade (see Liferay 7 documentation) Interfaces now in OSGI standard 38

39 eTokenServer and VPN Several adaptors needs to specify X509 proxiesThe GridEngine foresees the use of robot certificates querying the eTokenServer service eTokenServer can be accessible only opening its firewall or enabling a VPN connection To get VPN certificate please contact: or or Once obtained the certificate, extract its zip content; chmod to 400 the p12 file received after requesting the VPN access Run openvpn client: sudo openvpn .ovpn While openvpn client works the host can contact the eTokenServer service Once openVPN works, connect your SSH enabling the same VPN from your local machine 39

40 Questions … ? 40