1 XenoServer Open Platform - Parallel Job ManagementXeno: Greek word ξένος : "foreigner" & "guest” inviting foreign code to run on server as a guest XenoServer Open Platform Parallel Job Management Aytul Arisoy & Nana Liu (Cambridge University & Deutsche Telekom Laboratories) 

2 Ancestor of cloud computing - ‘90sWhat is XenoServer? Public global-scale infrastructure for wide-area distributed computing Global test-bed to enable authenticated external users execute remote code load balancing, limit congestion, self-financing Open source with GNU General Public License (Citrix Xen) Target : reducing communication latency code executes close to the entities it interacts avoid network bottlenecks reduce long-haul network charges provide network presence for transiently-connected mobile devices Initiated in 99, first release in 2003, became open source in 2013 Citrix bought xen but it is under gnu so core part of code is open Aws , rackspace , ibm uses xen hypervisor - 10M users, 300 developers, XenoServers owned by Merchants Public: Any suitably authenticated member of the public, corporation, organization Pay real money for all of the services used Mitigate latency / bandwidth charges Fine-grained: buy resources in small units, vary amount of resource dynamically, buy resources for short periods Support wide-range of existing and future services and apps – don’t require everything to be written in safe language Unlike planetlab, manage services, not machines Ancestor of cloud computing - ‘90s

3 Open Source Distributed Grid ComputingXenoServer project Goals Develop a platform for hosting services & managing resources Find techniques for automatic service placement & adaptive resource management Develop a scalable system for managing access control & auditing Provide programming aids & services for developing code to operate on Xenoservers with mobile devices or widely distributed clients Outcomes - Xen Virtual Machine Monitor = Hypervisor - hosting multiple commodity OS on a single server - resource management, accounting & auditing - Xenoserver Open Platform  - control SW to manage networks of Xenoservers - distributed storage - server discovery - resource management - AAA In 1995 Disney and Pixar used a server farm of 117 uniprocessor and multiprocessor SPARCstation workstations, comprising a total of 294 processors, to render Toy Story [Rob95], the world’s first ever full-length, entirely computergenerated animated movie. Rendering the 114,000 frames of the 77-minute movie required unprecedented amounts of raw computing power; one single-processor computer of that type alone would have needed 43 years of non-stop operation to render the movie. Hypervisor 4.7 Release Aug 2016 limits: x86 Host Limits: 4095 physical cpus, 16 tb physical memory Hvm Guest Limits: 512 virtual cpus, 1 tb virtual memory Open Source Distributed Grid Computing

4 High-Level Architecture ComponentsExecution platform (XenoServer) Securely partition, account and audit resource usage Brokers (e.g. XenoCorp) Intermediary between user and merchant Resource search services (e.g. XenoSearch) Suggest ‘best’ XenoServer(s) for a job Other high-level services and libraries Distributed file systems Control interface Resource management wrappers Analogies can be drawn between the XenoServer platform and aspects of every-day life. XenoCorp’s operation is similar to that of VISA, removing the need for direct trust between merchants and customers. XenoServers are merchants; they provide resources in exchange for money. The XIS is analogous to the yellow pages or on-line shop directories, as it contains a structured list of merchants and their capabilities, and provides basic indexing functionality. XenoSearch is similar to high-level searching services, such as on-line search engines, supporting sophisticated and multidimensional merchant discovery. XenoSearch builds on the XIS to provide advanced, specialised search functionality, such as searching for a set of servers that minimizes the total cost of deploying a particular service or the total round-trip time between the servers and a given set of clients. The use of the XIS is not mandatory, but simplifies the development of XenoSearch services, as it avoids the need for each XenoSearch to communicate with each XenoServer directly. It also allows for easier updates or additions of basic, common search algorithms used by many XenoSearch services. The XenoServer Information Service (XIS) is a service that helps the process of server selection. Servers periodically advertise their resource availability and the XIS provides the basic functionality required for clients to perform simple searches through those advertisements, in order to locate a number of servers that are suitable for hosting a particular service.

5 High-Level Architecture ComponentsExecution platform (XenoServer) Securely partition and account resource usage Single OS image is not mandatory … Commodity (viz. x86) hardware compatible… retain high performance Scalable Motivation for the Xen hypervisor Abstract view of XenoServer’s Design : Tasks are hosted in execution environments; an environment encompasses a set of reserved resources, and hosts one or more tasks belonging to the same service. Resource isolation and protection are enforced between the execution environments, such that several can coexist without being able to adversely affect one another. A privileged execution environment contains the software required for participation in the XenoServer platform, allowing the creation and management of other (client) execution environments on the XenoServer. A client is an entity that deploys services on XenoServers and pays for the resources its services consume. Clients need to be able to locate suitable servers — after describing what “suitable” means in each case, negotiate with XenoServers to reserve the desired resources, and deploy their tasks. Clients expect to receive feedback from the servers regarding the progress of service deployment, as well as information about resource usage by their services and the associated costs. They may also perform management operations on the deployed services, such as stopping and restarting a service, or migrating it to a different XenoServer.

6 XenoServers vs. Grid Grids XenoServers Resource IsolationApplication level Xen VMM QoS isolation Limited Supported code Specific OS or programming language Any OS Topology Not fully exploited Exploited Business model Free, non-profitable Commercial, self-funded Users Co-operative scientists Competing customers

7 Xeno Server Architectureregister_xenoserver validate_purchase_order charge_from_purchase_order XenoCorp register_client create_purchase_order RPC-like advertise_xenoserver update_specifications Session thread created XenoServer Information Service Client lookup_xenoserver find_xenoserver lookup_xenoserver Resource Discovery System It securely divides the resources of a machine among a set of resource-isolated Virtual Machines (VMs) running software on behalf of users. A special Management Virtual Machine (MVM) is used for the administration and control of the XenoServer. The architecture of the prototype XenoServer is shown in right hand side. The Management Virtual Machine (MVM) is booted at the start of the day and allows the owner of a XenoServer to create and manage VMs via the VM control interface (VCI). All configuration and control is performed via this interface, and all policy decisions, such as admission control and yield management, are made by software running within the MVM. Xen itself is responsible only for the mechanisms of facilitating resource multiplexing between VMs. The xend tools are sufficient for local XenoServer administration, but do not allow interaction with the rest of the XenoServer platform. This role is handled by XenoDaemon, a network daemon process that runs within the MVM. XenoDaemon is responsible for interfacing with both clients and XenoCorp, query_xenoserver_status create_session deploy_task (From U. Cambridge Computer Lab. Tech Report)

8 XenoServer Open Platform OverviewXenoServers: host & run clients tasks XenoCorp: authentication, auditing, charging, payment, contracts with clients & XenoServer operators XenoServer Information Service: storing XenoServers’ status updates XenoStore: Used by XIS to provide distributed storage to store the updates We distinguish between a number of entities involved in the XenoServer platform. XenoServers are responsible for running tasks or hosting mobile code on behalf of clients, and are operated by a range of potentially competing organizations. Their role is the same as merchants in a commercial settings. Control-Plane aspects of the system are managed by one or more XenoCorps. These are the analogue of credit card companies such as VISA and MasterCard – existing as trusted third parties between merchants and clients and providing ‘value added’ services to compete for business. Architecture of Open Platform with roles & interfaces Core Infrastructure of the System

9 XenoCorp Server StructureXenoCorp - matchmaking between clients & XenoServers Deamon: receives job from client & return appropriate XenoServers XenoComm: receives XenoServers properties, consult Cashier for pricing schemes of XenoServers to host the job & cache them in Registry module Consult: matchmaking between clients & XenoServers. Performs search & examine parameters on clients behalf Clients: specify priority used by Consult for search. Clients can choose XenoServers to submit job directly XenoCorp is responsible for matchmaking between clients and XenoServers. This task is mainly carried out by the consultant module of XenoCorp servers, many parameters are taken into account, including: execution environmental requirements(operating system), network traffic, server load as it try to perform load balancing. Additionally, XenoCorp should support the functionality of attempting to minimize the cost for the job to be submitted on behalf of the clients if they requested by checking and comparing the pricing schemes of all XenoServers that may be appropriate to host it. Location is also taken into consideration. All of the available XenoServers and their properties is received by XenoComm module and cached in the Registry module in each XenoCorp sever. Clients are given the ability to specify the priorities while searching for suitable XenoServers. The Consult module performs the search and examines all parameters on their behalf. After the search is over, XenoCorp will return a number of appropriate XenoServers, found accoding to those criteria to the client, and supply information about their location, load and pricing schemes. In the end, it is up to the client to choose a XenoServer and submit job directly there for execution. Openness principle in matchmaking client and Xenoserver

10 Execution module for the assigned client taskXenoServer Structure XenoServer Execution environment: set of resources allocated to an OS instance to host clients’ job Environments Manager: store environments currently running or ran in the past in detail Daemon: accepts new job Dispatcher: called to match job requirements with an environment in EnvManager module If succeeds, an environment is dispatched If not, new environment must be initialized Resource Manager: coordinate & allocate resources to execution environments. Execution module for the assigned client task

11 Xen Hypervisor-based XenoServer StructureXen: runs on svr HW & dynamically partitions HW between domains each domain hosts an instance of guest OS Control Plane: control interface to set resource sharing between the running domains. Only accessed by one vm: Domain0 & is privileged Domain0: required by Xen-based XenoServer & runs the app SW to manage control-plane of the platform A single XenoServer may host a range of Guest OS running at the same time, it may also host several independent instances of the same Guest OS running in different domains, perhaps on behalf of different clients. The hypervisor provides just enough abstraction of the machine to allow effective isolation and resource management between these domains. Within the single host, there are now two levels of interface to a given resource: at the bottom level is the raw physical interface between the hypervisor and the device, and above this is the virtualized interface that is presented to the virtual machines. Hypervisor layer virtualizes resources & securely multiplex access

12 XenoServer Selection Algorithm Flow ChartClients and XenoServers register with platform Servers advertise themselves Clients select servers by: - server discovery & selection functionality - directly select known/trusted servers 4. Clients proceed to service deployment 5. Clients submit task specifications to selected servers. - Tasks controlled per local admission control: - resource requirement - availability - resource management policies. - If accepted, tasks launched in execution environments. 6. After execution, further actions may be taken: stop, restart, migrate execution environments to other XenoServers. Resource management for global public computing is a challenging area which encompasses problems of how to describe resources, how to advertise their availability, and how to control the allocation or consumption of resources The general operation of the XenoServer Open Platform consists of four successive stages, which will be analysed in detail in the following sections. First, clients and XenoServers need to register with the platform, in order to be able to participate and trade resources for money. Then, servers advertise themselves and clients select the servers on which their services are to be deployed. To do so, they may use the server discovery and selection functionality provided, or they may directly select servers that are known or trusted by them. Once the servers are selected, clients can proceed with service deployment. Clients submit the deployment specifications of their tasks to each one of the selected servers. Tasks may or may not be accepted for hosting according to the local admission control decisions, based on resource requirement, availability, and resource management policies. If accepted, tasks are launched in execution environments on the servers. After a service is started up in an execution environment, further management actions may be taken to, for example, stop, restart, or migrate execution environments to other XenoServers. Servers account for resources consumed and claim payment from XenoCorp. The process of choosing the XenoServers to be used for the deployment of a distributed service is termed server selection. This can be carried out directly, if clients happen to know which XenoServers are suitable for their requirements. However, in most cases server discovery needs to be carried out. This allows competing XenoServers to publicise their capabilities, available resources and pricing schemes, and clients to find servers that match their requirements. Figure 3.4 shows the entities and interactions involved in the server discovery process. Prior to discovery, resources need to be named and described by individual XenoServers. The next chapter analyses the proposed resource description framework and presents mechanisms for representing resources and pricing units. Then, XenoServers advertise resource availability and clients search through the advertisements The XenoServer Information Service (XIS) indexes advertisements and provides simple lookup functionality. The operation of the XIS is similar to the one of yellow pages; it periodically collects and stores a number of independently produced advertisements in a structured manner, in order to make searching convenient for clients. For more advanced search operations, including multi-dimensional searching, XenoSearch can be used. XenoServer Information Service. Clients can discover the XIS by reading a list of available services advertised on the web — operation 3 in Figure 3.4. The XIS exports interfaces for lookup to the clients, providing basic search functionality that allows searching for advertisements that present values inside a predefined range for a specific token — operation 4 in Figure 3.4. For example, the following query returns all servers connected to the network /16. { Token = IPAddress; MinValue = ; MaxValue = ; } In more detail, the operation of the XIS is as follows. First, at registration time, each XenoServer provides to XenoCorp a URL pointing to the storage location — XenoStore, web, or any other type — where its server advertisements are to be stored. Using this information, XenoCorp periodically produces the Advertisement Locations Catalogue (ALC); this is a list containing URLs to the locations where all registered servers’ advertisements are to be found. The ALC is then itself pushed to one or more globally-readable storage locations, such as web or XenoStore locations — operation 5 in Figure 3.4. The XIS obtains the URLs of these locations from a well-known web server — operation 6 in Figure 3.4. The XIS reads the ALC, periodically pulls fresh — recently written — advertisements from the locations listed in the ALC — operation 7 in Figure 3.4, and 81 stores them in a structured, searchable manner — operation 8 in Figure 3.4. Older advertisements are ignored, as they represent servers likely to be down, thus not pushing fresh advertisements to their storage locations. To prevent repudiation, advertisements not properly signed by the advertisers are also ignored. XenoSearch. Although there is nothing to stop users from using the XIS to select appropriate servers, or from contacting prospective servers directly without introduction, it is anticipated that many will make use of one of a number of available XenoSearch resource discovery systems — operation 9 in Figure 3.4. These support server discovery, receiving sophisticated specifications of user and task requirements and using search algorithms to identify a number of suitable server Client: register client(personal info,address,charging info,temp creds) Server: register_XenoServer(owner_info,admin_info,payment_info, adv_loc,config_loc,temp_creds) XIS: lookup XenoServers(token,min value,max value)

13 Role-based Resource ManagementWhen user requests resources from server: Role declaration: Entities declare roles to group users and define policies Role entry conditions: determine which users are members of which roles Constraint declaration: define how resources are allocated within role memberships Constraint relationship: used to indicate how to resolve overlaps in policies. Resource Availability: Server decides on reservation requests per policies & resource availability Roles, entry conditions, constraints, and constraint relationships are policy elements. Algorithm Steps: 1. Server owners describe policies for resource allocation on servers 2. Policy elements are declared on servers on which they apply 3. For the policies to reach the servers, deployment of policy elements is performed 4. Each server evaluates policies to determine how to handle resource allocation requests role-based resource management (RBRM) scheme, in which the owner of the resources and other stakeholders can express which users or groups of users can be allowed access to which parts of the resources resource management is quantitative; the question then becomes how much access to grant a user to a resource, rather than simply whether to grant access or not. The aim is to allow the server owner and other entities to define policies that dictate how resources of the server are to be apportioned between different users or user groups. The main components of the proposed role-based resource management architecture, shown in Figure 4.5, are the following. There are users, who request resources from a server. Entities declare roles on servers, which identify classes of users for which they wish to define policies, and role entry conditions which determine which users are members of which roles. To define how resources can be allocated to users based on their properties and role memberships, entities specify constraints. Finally, constraint relationships can be used to indicate how to resolve some kinds of overlaps in policies. Each server decides on whether to accept or deny resource reservation requests based on policies and current resource availability. Roles, entry conditions, constraints, and constraint relationships are collectively referred to as policy elements. Server owners and other entities can describe policies for resource allocation on servers. Policy elements are declared on servers on which they apply. For the policies to reach the servers, deployment of policy elements is performed. Finally, each server independently evaluates the policies declared on it to determine how to handle resource allocation requests. Example: resource owner defined a policy suggesting each customer pay by VISA should be granted 5%bw, if cust is local 8%, if not regularly paying 2%. If a cust is local, pays by visa but not regularly. Admin defines conflict resoulution by contrainsts relationship.

14 XenoSearch ArchitectureXenoSearch proceeds in six steps: XenoServers deposit high-level info about their facilities Distilled to low-level formats used by distributed search algorithm Clients issue high-level queries specifying job requirements High-level queries mapped to low-level & issued to the algorithm Result of matching Xenoservers returned to client Client queries matching servers to confirm their status The system operates at two different levels: A high-level which is specific to the XenoServer platform A low-level distributed search algorithm which can be re-targeted for a variety of settings. Use of XenoSearch proceeds in six steps: XenoServers periodically deposit high-level information about the facilities they have and the current load on their resources. These are distilled to the low-level formats used by the distributed search algorithm. Clients issue high-level queries specifying their job requirements. These are also mapped to low-level queries and issued to the distributed search algorithm, ensuring that the resulting query is at least as inclusive as its high-level counterpart. The resulting set of possible matching XenoServer is returned to the client. The client directly queries matching servers to confirm their current status and suitability. Two stage commit: Co-located clusters are analogous to single multi-processor machine in XenoSearch. Queries are k-nearest-neighbor searches around “optimal” points.

15 Location based XenoSearch algorithmResource discovery requirements in distributed computing platform: Index the location Index current resources and total machine resources Design protocol for obtaining updates Specify a query format supporting for location-based constraints. Novel query language developed: Location and resource availability are key factors in the effective use of computing resources for network-centered applications. Location requirements are defined recursively using the primitives of disjuction(V), conjuction(^), proximity(near(A, B, C)), distribution(far(A, B, C)), and terms representing fixed locations(e.g. clients’ position in the network – C) and free servers to locate(S) --- the resource request terms to be matched to machines. These queries are preprocessed to a disjunction normal form in which each disjunction expresses an alternative way to satisfy the query. Each of the nodes in the graph(A…D) is a cluster, which encodes a near(A, B, C…) relationship. Algorithm aims to: minimize distance between selected machines & fixed locations within clusters maximizing distance between all clusters separated by far(…) relationships

16 SUMMARY Address challenges towards global public computing:reusable mechanisms( i.e. representing, advertising, resource discovery) flexible & federated control of resource allocation – orchestration “Openness” principle instead of scheduling Novel role-based resource management Effective service deployment models. Accounting and charging infrastructure A complex service running on the user customized Linux environment can be deployed to multiple XenoServers globally in under 45 seconds Introduces reusable mechanisms( i.e. representing, advertising, and supporting resource discovery) Allow flexible and federate control of resource allocation by all stakeholders Novel role-based resource management framework to expressing, combining distributed management policies. Implements effective service deployment models for launching distributed services on large numbers of machines around the word easily, quickly and efficiently. To keep track of resource consumption and pass charges on to consumers, devices an accounting and charging infrastructure.

17 REFERENCES Global Public Computing, Evangelos Kotsovinos. Computer Laboratory Technical Report UCAM-CL-TR-615, ISSN   The XenoServer Open Platform: Deploying global-scale services for fun and profit, Evangelos Kotsovinos and David Spence. ACM SIGCOMM '03, August 2003  Controlling the XenoServer Open Platform, Steven Hand, Tim Harris, Evangelos Kotsovinos, Ian Pratt. Proceedings of the Sixth IEEE Conference on Open Architectures and Network Programming (OPENARCH 2003), April 2003.  The Xenoserver Computing Infrastructure. Keir A Fraser, Steven M Hand, Timothy L Harris, Ian M Leslie and Ian A Pratt. Technical Report UCAM-CL-TR-552, January 2003.  Xenoservers: Accounted execution of untrusted code, Dickon Reed, Ian Pratt, Paul Menage, Stephen Early, Neil Stratford.  IEEE Hot Topics in Operating Systems (HotOS) VII, March 1999  Xen and the Art of Virtualization, Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt and Andrew Warfield. Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), October 2003  Isolation of Shared Network Resources in Xenoservers, Andrew Warfield, Steven Hand, Timothy Harris and Ian Pratt. PlanetLab Design Note PDN , November 2002  Xen 2002, Paul R Barham, Boris Dragovic, Keir A Fraser, Steven M Hand, Timothy L Harris, Alex C Ho, Evangelos Kotsovinos, Anil V S Madhavapeddy, Rolf Neugebauer, Ian A Pratt, Andrew K Warfield. Technical Report UCAM-CL-TR-553, January 2003.  Global-scale service deployment in the XenoServer platform, Evangelos Kotsovinos, Tim Moreton, Ian Pratt, Russ Ross, Keir Fraser, Steven Hand, Tim Harris. Proceedings of the First Workshop on Real, Large Distributed Systems (WORLDS '04), December 2004, San Francisco  XenoSearch: Distributed Resource Discovery in the XenoServer Open Platform, David Spence and Tim Harris. Proceedings of the Twelfth IEEE International Symposium on High Performance Distributed Computing (HPDC-12), June 2003  Distributed resource discovery and management in the XenoServers Platform, Evangelos Kotsovinos, Timothy L Harris. Proceedings of the 7th CaberNet Radicals Workshop, Bertinoro, Italy , October 2002.  XenoTrust: Event-based distributed trust management, Boris Dragovic, Evangelos Kotsovinos, Steven Hand and Peter Pietzuch.Proceedings of the Second IEEE International Workshop on Trust and Privacy in Digital Business (DEXA-TrustBus'03), September 2003 

18 Thank you

19 BACKUP

20 XenoServer DistributionDeploy XenoCorp Any user, any code, anywhere XenoServer Client XenoCorp, the trusted third party between clients and XenoServers. Its existence is necessary in an inherently untrusted and uncooperative environment, like the one anticipated for the XenoServer platform; as clients and XenoServers do not initially know or trust each other, a trusted broker is required to “guarantee” that XenoServers provide the expected service and clients pay for the charges incurred. XenoCorp issues authentication credentials for clients and XenoServers when they join the platform, stores details about servers’ ownership and administration, provides configuration information and handles charging and payments. Although logically central, XenoCorp may be implemented in a replicated and distributed fashion for fault tolerance and scalability Global services and apps Exploit network topology Open infrastructure Incremental rollout Flexible platform Unified management

21 Format of job description is not standardized, but similarWhat is a Job? A job is a computational task requires processing capabilities (e.g. 64 nodes) subject to constraints (e.g. a job must finish before another starts) Job information provided by the user resource requirements (RSL) CPU architecture, # of nodes, speed memory size per CPU software libraries, licenses I/O capabilities job description Format of job description is not standardized, but similar

22 Job Management in Parallel ComputingAnswers: Whose job you run When to run Where to run How many jobs to run Effectively optimizes the utilization of resources Effectively optimizes the sharing of resources The general operation of the XenoServer Open Platform consists of four successive stages, which will be analysed in detail in the following sections. First, clients and XenoServers need to register with the platform, in order to be able to participate and trade resources for money. Then, servers advertise themselves and clients select the servers on which their services are to be deployed. To do so, they may use the server discovery and selection functionality provided, or they may directly select servers that are known or trusted by them. Once the servers are selected, clients can proceed with service deployment. Clients submit the deployment specifications of their tasks to each one of the selected servers. Tasks may or may not be accepted for hosting according to the local admission control decisions, based on resource requirement, availability, and resource management policies. If accepted, tasks are launched in execution environments on the servers. After a service is started up in an execution environment, further management actions may be taken to, for example, stop, restart, or migrate execution environments to other XenoServers. Servers account for resources consumed and claim payment from XenoCorp. Also referred as Resource Management or Queuing Systems

23 Pros vs. Cons Pros Cons Scalability Open source Community supportOrchestration Ease of deployment Cooperative users Flexible server selection Cons Load balancing Congesting Financing Security DoS Availability Fault Tolerance Untrusted code