Revolutionizing Big Data Access and Analysis in the Cloud

THE NEED

Our client, a large research agency, automated its systems for storing, disseminating and analyzing big data about molecular biology, biochemistry, and genomics support for some of the most heavily trafficked governmental sites in the world. These sites experience millions of hits daily from medical workers, researchers, and the general public.

Researchers conducting analyses on the agency’s public datasets typically download them to their research facility computers or to their virtual environments in the cloud. The datasets are often massive, measured in terabytes and sometimes petabytes. Since these public, read-only datasets are copied as many times as there are research teams seeking to work on it, download times are long, and the process is inefficient. The agency was looking for a more efficient way for researchers to access and analyze the datasets faster, as well as free up more of the agency’s networking capabilities.

OUR SOLUTION

eGlobalTech (eGT), a cloud service provider, partnered with this agency to help architect, design, and implement new forms of access to significantly improve the efficiency and speed of research on massive biomedical public datasets. To meet the client’s unique need, eGT employed the emerging “bring compute to the data” paradigm, which enabled researchers to create and fund their own cloud computing resources in one of several clouds where the government agency’s datasets reside. We enabled research teams’ cloud computers to directly connect to, read, and analyze public, read-only datasets without copying or downloading. Researchers see the same version of the data, access is immediate, and efficiencies are significant.

To achieve this result, eGT applied “cloud-native architectures composed in a cloud-agnostic manner” principles to develop the initial architecture portable across Amazon Web Services (AWS) and the Google Compute Platform (GCP), and iteratively refined and operationalized the architecture. eGT applied Infrastructure as Code practices and produced functional, software-defined solutions delivered in time-boxed iterations, while also supporting cloud architecture security engineering from both a compliance and an operational security perspective.

Major highlights of our services include:

  • Designed a multi-cloud architecture to support “Platform as a Service for Big Data Access” model
  • Leveraged tools such as Terraform and Puppet to automate the deployment and operations of cloud environments
  • Developed a portable architecture with initial support for AWS and GCP and eventual support for other providers such as Microsoft Azure
  • Developed required security packages, supported the Accreditation and Authorization (A&A) process, and helped secure Authority to Operate (ATO)
  • Provided on-going SysOps and environment management support to development and DevOps teams

eGT’s extensive experience as a cloud service provider, architecting and migrating applications to the cloud has enabled the development of flexible and creative breakthrough strategies that consider our client’s unique requirements, creative implementations, extremely complex environments, and security needs.

THE RESULTS

Key results of our solution include:

  • Operationalized multi-cloud environments in AWS and GCP
  • Developed required FISMA security packages and secured ATO on time
  • Enabled “Bring Compute to Data” strategy through distributable cloud automation platform

Contact us at info@eGlobalTech.com if you would like to learn more about this project.

Copyright 2018 | eGlobalTech | All rights reserved.