1 NIH/NIEHS Infrastructure ManagementAudrey Brown Social & Scientific Systems Durham, NC
2 Background DRIVER: Branch Director requested REDCap Summer 2016Previous Solutions: NIH Clinical Trails Data Base system Off-site developers, limited functionality, and high cost SSS Internal data collection tools Goals: House NIEHS Clinical Research Unit (CRU) projects Migrate clinical trials data Implement plugins, hooks, APIs SSS provides data management and development Sr. Data Managers, REDCap SMEs, data managers, developer NIEHS/SSS collaborative server management Meet all government security and compliance regulations Internal = DATSTAT (so epr data ) Personnel flexes on the projects dependent upon phase and skills necessary for contract Put together requirements from the principle investigators, worked with OSIO and team to develop what environments would be Branch Director requested use of redcap
3 Infrastructure Development Testing Production Why 3?3 Environments – FISMA Moderate – Network Attached Storage Why 3? Prevents upgrade failure Testing provides live data to test against Development provides playground for developers Allows new server configurations to roll-up Virtual Machine Environment Reuses standard NIEHS VM setup Allows for easy backup and recovery Create new environments as needed Development Testing Production Internal Public Fisma mod = research data Instead of standing up a real server, can create virtual machines instead, housed in one computer, we have 3, if we need 4 its not a big deal, doesn’t take them a long time. For another contract, we would do the same thing for another contract. Less work on server team, not incremental work, not linear. Do it once, and then the next time its faster and easier VMs, comes with the security already built in, because the computer is already set up for that. OS RHEL 7.3 or centos 7.2 (test only) Web Server Apache 2.4.6 Database Server MySQL PHP vsn DEV & Test REDHAT Linux 7.3 Dual socket Dell R-730s 128 gig of RAM 3-1 TB drives on each using Network Attached Storage (NAS) for REDCap PROD VM VMware Given 2 processors 64 gig of ram Network Attached Storage (NAS) for data
4 Beware of Low Hanging FruitWhat not to do Requirements Analysis and Design Build and Test Deploy & Early Life Support Pegasus that everyone wants, and slowly you wind up with a donkey because during requirements you describe something you cant deliver, as analysis and d you start to realize you cant meet them, but you think you can meet some, so you get a quarter horse only, and then during deployment you realize you cant do any, and then you have to shovel yourself out, and then gold plate the shovel instead of adding value, spend time in the trenches. All to say you started with a plan you could not implement. Beware of Low Hanging Fruit
5 Deploy & Early Life SupportThe right way Requirements Analysis and Design Build and Test Deploy & Early Life Support Come up with strawman requirements, basically trying to make a prototype, try to implement them to get a sand man that is a little more solid and you are slowly improving as you go along. Then clayman allows you to realize you can start doing it (come up with a fantastic idea and prototype), then you concretize it and then you can deploy it and get your amazingness of womder woman. Without prototyping it, you don’t really know where you are going and you are in an allusion and in a made up place. Concrete man --- whats better than concrete man. Human 2.0- wonder woman. From strawman to concrete man. So, how do we do that?
6 Preparation before the stormAgreement on Vision Technical Risks Mitigated Operational System Supportable System Assemble your working group early and often: REDCap Administrators (you) Regulatory Officers Project Managers Security Officers Developers Server Team Don’t wait until the last minute, have a plan for growth (3 mo, 6mo, 1-5yr plans) Make deadlines and hold team accountable Watch the pipeline and make sure staff are trained and available for upcoming work Decide how to standardize and what to centralize regardless of organizational structure Plugin/Hooks, new features & versions, compliance documentation, end-user testing & training
7 REDCapCon 2017 Infrastructure Management - Lessons Learned Scalability: Managing Growth Over TimeLuke Stevens Murdoch Children's Research Institute Melbourne, Australia
8 Our Institute Royal Children's Hospital + University of Melbourne Department of Paediatrics + Murdoch Children's Research Institute (MCRI) Largest child health research institute in Australia One of the top five worldwide 1900 Researchers Clinical Sciences Infection and Immunity Population Health Cell Biology Genetics (research and Clinical Genetics Services) Data Science (biostatistics, bioinformatics, data management)
9 Our Institute Royal Children's Hospital + University of Melbourne Department of Paediatrics + Murdoch Children's Research Institute (MCRI) Largest child health research institute in Australia One of the top five worldwide 1900 Researchers Clinical Sciences Infection and Immunity Population Health Cell Biology Genetics (research and Clinical Genetics Services) Data Science (biostatistics, bioinformatics, data management)
10 REDCap@MCRI: Starting SmallLuke Stevens, "Data Management Coordinator" Between researchers and IT, translator! Do stuff to help researchers manage their data better Get the statisticians better quality data to work with Identified need to replace Excel, Access, EpiData There's this REDCap thing that looks interesting. Let's try it... September 2010, Me to IT: "Please can I have a server?"
11 REDCap@MCRI: Starting SmallLuke Stevens, "Data Management Coordinator" Between researchers and IT, translator! Do stuff to help researchers manage their data better Get the statisticians better quality data to work with Identified need to replace Excel, Access, EpiData There's this REDCap thing that looks interesting. Let's try it... September 2010, Me to IT: "Please can I have a server?"
12 REDCap@MCRI: Starting SmallLuke Stevens, "Data Management Coordinator" Between researchers and IT, translator! Do stuff to help researchers manage their data better Get the statisticians better quality data to work with Identified need to replace Excel, Access, EpiData There's this REDCap thing that looks interesting. Let's try it... September 2010, Me to IT: "Please can I have a server?" https://redcap.mcri.edu.au Managed by me Application maintenance, user training and support Server management (backups, ssl certificate etc.)
13 REDCap@MCRI: Growth to NowSep-2010 (v3.3.0) to Aug-2017 (v7.5.2) Consistent strong usage growth Varied usage: at 24-Jul-2017 1613/2166 (74%) research purpose 1237/2166 (57%) with surveys 750/1613 (46%) clinical research 705/1613 (44%) epi/behavioural/psych 184/1613 (11%) basic/bench
14 REDCap@MCRI: Growth to NowAround 0.8 EFT (of me plus one other, employed via funding for project data manager) Application admin (hooks, plugins, version upgrades, Community website!): 0.2 EFT General support, user accounts, consultations: 0.25 EFT (not cost-recovered) Training materials and courses: 0.05 EFT (cost-recovered) Custom project work (project builds, hooks, plugins): 0.3 EFT (cost-recovered) With >2000 projects REDCap now core Institute infrastructure IT handle management of production server Harmonised with other MCRI infrastructure (storage space, server OS, database server, TLS certificate) More paperwork! Still just me on the application management side Local development environment Version upgrades Hook and plugin development
15 REDCap@MCRI: the FutureContinued growth anticipated Increasing expectation of "compliance" (with updated GCP guidelines, if not 21 CFR part 11; potential Epic EMR integration) More paperwork! SOPs for everything Beyond current Institute experience and practice Risk-based approach? Such more stringent requirements do not apply to all projects Clinical Trials? Yes Lab-based research, small pop health survey? What benefit the overhead? How will we balance "compliance" with flexibility for upgrading for new functionality, deploying plugins and hooks?
16
17 REDCap Con 2017: Infrastructure Management – Managing Growth over TimeRobert Morris Senior IT Analyst Duke University Office of Research Informatics
18 REDCap@Duke Usage School of Medicine School of NursingDuke Clinical Research Institute Duke Office of Clinical Research Duke University Health System Other various research departments throughout Duke https://redcap.duke.edu
19 REDCap@Duke Project StatsStarted in Sept ‘09 Steady growth 5K+ projects
20 REDCap@Duke User StatsStarted in Sept ‘09 Steady growth 5900 active users These numbers are as of Aug 7.
21 REDCap@Duke Growth Story – Initial EnvironmentStarted small Prod & Dev using same database Different schema Started with virtualized environment We started with two small servers and 1 small database. These servers did not have high specs and were quite slow. There was no real good policy of having a “dev/test/prod” stack.
22 REDCap@Duke Growth Story - Current EnvironmentEventually, as the use of REDCap grew, we scaled up, creating a more defined and mature environment. Just recently, as part of our infrastructure upgrade, we went to a single server+database pair for each of the “sand/dev/test/prod” environments. Additionally, we implemented the use of SSL certs and an F5 proxy mechanism, to handle URL requests and URL rewrites from redcap.dtmi.duke.edu to redcap.duke.edu With the addition of external users, we’ve added a second ldap server specifically to authenticate these users. Currently have a total of 4 environments – sandbox, dev, test (aka validation) and production; all of which are completely virtualized. Each environment has its own server+database pair and these are dedicated pairs. Each server has specs of: 4 vCPUs 16 GB RAM 250 GB HDD With the databases being similar, just larger drives. They are 500GB instead of 250GB.
23 REDCap@Duke ChallengesMaintenance Windows Establishing regular maintenance windows Monitoring Using/developing monitoring tools Change management Establishing a solid change management plan Documentation Thorough documentation on the infrastructure As we grew, we saw the need for regularly scheduled maintenance windows which allowed us to add in custom hooks and conduct other tasks on a regularly scheduled basis. This lets users know when the system could be unavailable, and gives the sys admin team and the application support a defined schedule to roll out new features. A solid change management plan helped out with this as well. We determined that using monitoring tools would help us stay on top of any issues that might arise, which has been helpful in solving production issues quickly Documentation has been key. We have worked hard to ensure our infrastructure is completely documented so that we have knowledge redundancy and so that everyone who should know can know the system
24 REDCap@Duke Future Automation Containers Other? Install UpgradeDocker, etc Other?
25 Questions?