Dealing with Cloud Solutions has been a very interesting and fruitful experience. It is a new and rapidly expanding technology which has me excited for its future. However, despite all the benefits the Google Cloud offers, some Cloud Solutions technology is still in development and not ready for production.
There are very real and beneficial usages for the Google Cloud Platform. It has many benefits and services which can greatly bolster HPCC Systems’ range. The primary focus of this project was to provide a Proof of Concept for future development and to create boilerplates, documentation, and code for future expansion upon the Google Cloud Platform. This project consisted of two phases. The first being the implementation of HPCC Clusters on the Google Kubernetes Engine, while the second regarded the creation and registration of HPCC Clusters on Google Anthos. Since Google Anthos was introduced recently, there are limited resources and inconsistencies within official documentation. As a result, I spent countless hours and days researching for this project. Although I did have some previous experience with Kubernetes, it was fairly limited and I had never used Google Kubernetes Engine. Thankfully, this internship proved very invigorating and illuminating and helped bridge the knowledge gap.
“There are many alternative Cloud providers. Why did you choose Google?” I chose the Google Cloud Platform because, compared to Azure and AWS, it is more cost efficient, has a generous free trial period, and puts a very big focus on security. Google also has virtually no downtime and many internal services such as Anthos, App Engine, and many monitoring services. Also, Google was the original author of Kubernetes which gives it an advantage as its users will always receive the latest versions first.
The installation process is simple: there are a few necessary requirements.
Concluding the install of my HPCC Cluster onto GKE I was greeted with the ECLWatch screen.
After I setup my HPCC Cluster on Google Kubernetes I began running various tests. The first few tests I ran were from the ECL Playground Samples and ECLWatch Data Tutorial. These results were 100% successful. However, there were minor issues with some ECLWatch Data Tutorial tests as a few functions, such as spraying, were not supported. The final test that I ran on ECLWatch were the regression tests, which ended with a few expected errors.
Detailed setup and testing information here.
Inside the Google Kubernetes Engine I experimented around with GKE’s autoscaling functionality. I was able to successfully auto-scale my HPCC Clusters Vertically and Horizontally. There is more detailed information regarding my auto-scaling process here.
Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.
Anthos’s basic building blocks are the Google Kubernetes Engine, GKE On-Prem, and Istio. These three individual services create the fundamental blocks that make up Anthos. As you can see in the diagram above, Anthos can manage multiple clusters and funnel it under a single management platform, despite the clusters operating on different Cloud platforms.
Anthos Service Mesh (ASM) is a suite of tools that aids in monitoring and managing infrastructure layers that manage, observe, and secure communications across all services. The ASM allows HPCC users to view network traffic logs, set service level objectives (SLO), perform health checks, monitor custom services, view service to service relationships to understand relationships between services, and display in-depth telemetry. ASM also comes with a network traffic manager known as Ingress. The Ingress Gateway allows HPCC users to set rules for inbound traffic heading towards ECLWatch. An example usage would be 80% of network traffic directed to stable ECLWatch versions while 20% of network traffic being redirected to new versions.
Anthos Config Management allows HPCC users to manage single cluster, multi-tenant cluster, and multi-cluster Kubernetes deployments by utilizing configuration files stored in Git repositories. i.e. Github or GCSR. Config Management enables configuration-as-code on multi-cloud environments giving HPCC users the ability maintain a consistent set of network and security policies across multiple applications. Config Management can also utilize CI/CD pipelines to automatically test and deploy changes.
As shown in the diagram below, Anthos Config Management syncs the state of the HPCC three cluster(s) using the specified Git repository despite them operating on different cloud platforms. This increases performance as it negates the need for HPCC SRE’s to create thousands of Kubernetes config files for thousands of clusters.
Along with the plethora of tools provided by Anthos Config Management and Anthos Service Mesh, Anthos supports third party programs out of the box: Grafana, Kiali, and Elastic Stack. Although Anthos has Telemetry systems, they only support clusters on Anthos for AWS and Anthos for GKE. Unsupported clusters such as EKS and On-Prem must utilize Grafana, Kiali, and Elastic Stack to receive telemetry data. Grafana and Kiali receive data from another pre-installed third-party program called Prometheus. These third-party telemetry systems will greatly benefit HPCC users by providing a central place to view all logs, metrics, and to debug Kubernetes Clusters despite being on different cloud platforms.
As HPCC Systems is moving towards Native Cloud Development, users will begin experimenting with multiple cloud environments. Utilizing Google Anthos, HPCC users will be able to benefit from a streamlined process in which all cloud environments are packaged and managed under one application. Users will be able to take advantage of the numerous features and components that Anthos provides in its toolkit. However, there are still some issues regarding Anthos. For example, with Istio Sidecar, jobs will never finish. This is due to a bug with K8s. I also experienced several issues setting up ASM which eventually was never resolved. All in all Anthos is still a new technology and has a while to go before it should be used in production. However, it shows a lot of promise.
With the project completed, HPCC Systems developers will now be able to gauge the effectiveness and benefit of utilizing the Google Cloud Platform. HPCC Systems will now be able to make a decision regarding the future of Google’s role in HPCC’s Cloud Native Solution.
I thoroughly enjoyed working for LexisNexis as a summer intern and appreciate in full all the resources and opportunities that the HPCC Community has provided to me. I look forward to continuing this project in Summer 2021. I would like to give special thanks to my mentors Xiaoming Wang and Godson Fortil, as well as Lorraine Chapman.