1 Climate Change: WRF4G & CAM4GValvanuz Fernández Santander Meteorology Group Dept Applied Mathematics and Comp. Sci. Universidad de Cantabria, Santander, Spain Thanks to: A.S. Cofiño J. Fernández C. Blanco L. Fita M. García-Díez CHAIN Workshop, EGI Technical Forum (21 September 2011), Lyon, France

2 2006. EELA PROJECT. Collaboration among 3 institutions.Motivation 2006. EELA PROJECT. Collaboration among 3 institutions. GOAL: Develop a simulation and analysis tool to predict local impacts of “El Niño” in Latin America A challenging problem with huge socio-economical impact in Latin America (Floodings in the coastal areas, problems with fishing,… ). Explicar donde comenzó el trabajo y porqué Añadir que la colaboración empezó con EELA y partners. ¿Poner algo de cambio climático? El Niño events are characterized by having anomalous Sea Surface Temperature (SST) over the equatorial eastern Pacific.

3 Sensitivity study SST-Precipitation Motivation Sensitivity study SST-Precipitation Hundreds of simulations with different perturbed SST (1997 SST used as pattern) What would happen if the water were warmer/colder? Perturbations are a montercarlo simulation. The pattern we are going to use to do the perturbations is the SST 1997. We are going to run simulations with higher temperatures and lower. We will compare the output precipitation with the precipitation observed for the warmer Temperatures.

4 Climate Models Climate Models are numerical models that simulate the interactions of the atmosphere, oceans, land surface and ice. Global Models such as CAM simulate the whole globe while Regional Models such as WRF provide finer resolution. Modelos climáticos. Dificultades en entornos heterogeneos distribuidos. Ventajas. CAM y WRF. Complex system. Multiple componentes dependientes entre sí. Complex workflow. Representa procesos físicos del sistema climático. CAM y WRF simulan modelos climáticos .Software de un modelo físico concreto. ¿Porqué nadie había ejecutado modelos hasta ahora en GRID?. Características modelos: Requerimientos. Complejidad de ejecución. Post-proceso dependencias, largos, intensivos CPUmemroia (recompilar 1 resolución). Ad-hoc. Tipicamente configurar, compilar. Complejidad de ejecución de los modelos incluso en un ordenador.

5 Climate Models Working with Climate Models is not trivial...Complex compilation. Complex workflow: Models are composed of several executables that have to be run sequentially. Pre & post- processing. Restart capabilities. Large input and output data transfers. Intensive use of CPU and memory (simulations can be running for days even in parallel). Modelos climáticos. Dificultades en entornos heterogeneos distribuidos. Ventajas. CAM y WRF. Complex system. Multiple componentes dependientes entre sí. Complex workflow. Representa procesos físicos del sistema climático. CAM y WRF simulan modelos climáticos .Software de un modelo físico concreto. ¿Porqué nadie había ejecutado modelos hasta ahora en GRID?. Características modelos: Requerimientos. Complejidad de ejecución. Post-proceso dependencias, largos, intensivos CPUmemroia (recompilar 1 resolución). Ad-hoc. Tipicamente configurar, compilar. Complejidad de ejecución de los modelos incluso en un ordenador.

6 Climate Models Configuring climate models to run in new resources (stand-alone severs, clusters, Grid,...) involves: Compiling the model using the compilers and libraries provided by the resource center. Learning to use the new environment (job management commands, data protocols, MPI environments,…). Bringing data to the new infrastructure. Deploying a monitoring service that ensures the correct job execution and provides users with their experiment status. Modelos climáticos. Dificultades en entornos heterogeneos distribuidos. Ventajas. CAM y WRF. Complex system. Multiple componentes dependientes entre sí. Complex workflow. Representa procesos físicos del sistema climático. CAM y WRF simulan modelos climáticos .Software de un modelo físico concreto. ¿Porqué nadie había ejecutado modelos hasta ahora en GRID?. Características modelos: Requerimientos. Complejidad de ejecución. Post-proceso dependencias, largos, intensivos CPUmemroia (recompilar 1 resolución). Ad-hoc. Tipicamente configurar, compilar. Complejidad de ejecución de los modelos incluso en un ordenador.

7 CAM4G Ad-Hoc solution for each scientific problem. Spend human resources. Time-consuming task. Lowering the barrier using the technology. Develop a generic framework to run models regardless the executing environment and data architecture behind them. First approach CAM4G framework. CAM4G provides a CAM compiled version that is not installed in the host resources. Binaries are transferred for each simulation. Monitoring service. It tracks all the events (output files produced, current date being simulated, ...) during the execution. Transparent data access: Support rsync, local copy (NFS), gsiftp. Execution manager in charge of manage the CAM workflow. Technologies used: Gridway, self developed replica catalog and monitoring services. CAM4G: Running CAM on GRID Caso de éxito inserción del Grid en la comunidad de ciencias de la tierra. Se presentá qué es el GRID y las infrastructuras actuales. State of the art of Grid for the climate modeling community EGEE Grid infrastructure to run a sensitivity experiment involving the execution of Experimencias previas: RB, LFC, amga,.... MYSQL. Gridway, self-developped software,

8 El niño experiment Niño experiment. Execution of 750 x 19-month simulations of CAM in the EGEE testbed (200MB of input data each simulation). Environmental Modelling & Software 26 (2011)

9 El niño experiment Niño experiment: 1 simulation (cluster): 2 days750 simulations (GRID): less than 4 days

10 Benefits of Grid… The paper is targeted to the Earth Science community. It describes: Grid technology. Benefits of grid and state of the art of grid for the climate modeling community. Technologies used: Gridway, self developed replica catalog and monitoring services. Niño experiment. Execution of 750 x 19-month simulations of CAM. CAM4G: Running CAM on GRID Caso de éxito inserción del Grid en la comunidad de ciencias de la tierra. Se presentá qué es el GRID y las infrastructuras actuales. State of the art of Grid for the climate modeling community EGEE Grid infrastructure to run a sensitivity experiment involving the execution of Experimencias previas: RB, LFC, amga,.... MYSQL. Gridway, self-developped software,

11 Role of WRF4G in CORDEX New Challenges Why WRF4G?Big community. Currently, registered users. Example of target community: CORDEX initiative CORDEX (COordinated Regional climate Downscaling EXperiment) is a framework to improve coordination of international efforts in regional climate downscaling research. CORDEX was initiated as a result of the Task Force on Regional Climate Downscaling, formed by the World Climate Research Program (WCRP). A set of target regions has been proposed and modeling groups willing to contribute must comply with simulations specifications. There are currently 15 groups planning to contribute to CORDEX with WRF. All of them could benefit from WRF4G.

12 Main improvements in WRF4G framework…WRF4G improvements Main improvements in WRF4G framework… Resource management: CAM4G only gave access to Grid resources. We need a tool that allows the user to access at the same time different computing resources in a transparent way: clusters, stand-alone servers, grid infrastructures. DRM4G (Gridway). Data management: Replica catalog and efficient data transport (not solved yet) . Monitoring: Customized WRF monitor that tracks all the events (output files produced, current date being simulated, ...) during the execution. Modelos climáticos. Dificultades en entornos heterogeneos distribuidos. Ventajas. CAM y WRF. Complex system. Multiple componentes dependientes entre sí. Complex workflow. Representa procesos físicos del sistema climático. CAM y WRF simulan modelos climáticos .Software de un modelo físico concreto. ¿Porqué nadie había ejecutado modelos hasta ahora en GRID?. Características modelos: Requerimientos. Complejidad de ejecución. Post-proceso dependencias, largos, intensivos CPUmemroia (recompilar 1 resolución). Ad-hoc. Tipicamente configurar, compilar. Complejidad de ejecución de los modelos incluso en un ordenador.

13 Computer resource scenariosDesktop/Laptop (UI) Low computational power and storage User interface to other computer resources Workstation Multi-core, shared memory, moderate storage ssh access Local group/institutional cluster Multi-node, distributed memory, large storage ssh access, batch system (PBS, SGE, ...) to submit jobs Mainframe/HPC site Different architectures and memory arrangements ssh or higher security access Grid infrastructure “Cluster of clusters”, geographycally distributed Huge amount of computational power and storage (not trivial to take advantage of it for meteo/climate apps) 13

14 Configuration files [Computing Resources]wrf4gframework.conf [Computing Resources] mycomputer local://localhost? LRMS_TYPE=none; NODECOUNT=1; myworkstation ssh://workstation.unican.es? LRMS_TYPE=none; NODECOUNT=16; PBS_cluster ssh://pbs.cluster.edu? LRMS_TYPE=pbs; QUEUE_NAME=long; NODECOUNT=256; EELAPROD grid://eelaprod_inf? VO=eelaprod; HOST_LIST=eelaprod_hosts.conf

15 Configuration files wrf4gframework.conf resources.wrf4g # WRF4G version to use (packed scripts must be in $WRF4G_APPS) WRF4G_VERSION="1.0" # Name of the packed WRF binaries (the file must be in $WRF4G_APPS) WRF_VERSION="3.1.1_r832INTEL_OMPI" # Common path to save all output and log files WRF4G_BASEPATH=“gsiftp://se01.macc.unican.es/oceano/gmeteo/WORK/ASNA/WRF/experiments" # Path to the preconfigured WRF domains WRF4G_DOMAINPATH="gsiftp://se01.macc.unican.es/oceano/gmeteo/WORK/ASNA/WRF/domains" # Path to the global data for the boundary and initial conditions WRF4G_INPUT=“rsync://server/oceano/gmeteo/DATA" # Path to the packed binaries (WRF4G script, WRF, cdo (preprocessor), ..) WRF4G_APPS="gsiftp://se01.macc.unican.es/oceano/gmeteo/WORK/wrf4g/repository/apps" # Number of parallel processors (cores) per simulation NP=8 # Computer resources to use RESOURCES="myworkstation,PBS_cluster" # Fine tuning ENVIRONMENT='MAXWALLTIME = 36000, MAXMEMORY = 1000'

16 Access to heterogeneous resourcesGridWay metascheduller allows the user transparent access to Grid resources. Provides its own API to add new Resource Managers and Communicators. Grid infrastructures: Globus glite

17 Access to heterogeneous resourcesDRM4G (Distributed Resource Manager) allows the user to merge different computing resources at hand in a transparent way: Local resources (UI) Remote resources (via ssh) Directly in a shell session Interacting with LRMS PBS SGE SLURM ...

18 Monitoring Shell$ wrf4g_status -e exampleRealization GW Stat Chunks Comp.Res WN Run.Sta ext % example__ph P 0/ Prepared example__ph P 0/ Prepared example__ph P 0/ Prepared example__ph P 0/ Prepared example__ph P 0/ Prepared

19 Monitoring Shell$ wrf4g_status -e exampleRealization GW Stat Chunks Comp.Res WN Run.Sta ext % example__ph R 1/4 mycomputer legolas metgrid example__ph R 1/4 mycomputer legolas Down. Bound example__ph W 1/ Submitted example__ph W 1/ Submitted example__ph W 1/ Submitted

20 Monitoring Shell$ wrf4g_status -e exampleRealization GW Stat Chunks Comp.Res WN Run.Sta ext % example__ph R 1/4 mycomputer legolas real example__ph R 1/4 mycomputer legolas metgrid example__ph W 1/ Submitted example__ph W 1/ Submitted example__ph W 1/ Submitted

21 Monitoring Shell$ wrf4g_status -e exampleRealization GW Stat Chunks Comp.Res WN Run.Sta ext % example__ph R 1/4 mycomputer legolas WRF example__ph R 1/4 mycomputer legolas WRF example__ph W 1/ Submitted example__ph W 1/ Submitted example__ph W 1/ Submitted

22 Monitoring Shell$ wrf4g_status -e exampleRealization GW Stat Chunks Comp.Res WN Run.Sta ext % example__ph W 2/ Submitted example__ph W 2/ Submitted example__ph R 1/4 mycomputer legolas WRF example__ph R 1/4 mycomputer legolas WRF example__ph W 1/ Submitted

23 Projects supporting WRF4GEuropean commision (7FP): EELA2: E-science grid facility for Europe and Latin America Partners: 52 institutions in Latin America and Europe Spanish Ministry of Science and Innovation: WRF model port to Grid infrastructures and proof-of-concept for a high-resolution wind hindcast over Europe Universidad de Cantabria Coordinated regional climate downscaling experiment using WRF: a contribution to the CORDEX initiative by the Spanish WRF community Partners: 3 Spanish universities and a supercomputing center

24 WRF4G is freely available for use...Conclusions WRF4G simplifies the design, execution and monitoring of WRF on several computer resources. Current collaboration with other groups that are using WRF4G in clusters and Grid infrastructures to perform different kind of studies. WRF4G is freely available for use...

25 Thank you! Merci! Contact: [email protected]More info: (or just “wrf4g” → Google)