9 Jul 2020

Deploy HPCC Systems on the Google Cloud Platform

Page Overview

This page will cover the deployment of HPCC Systems on the Google Cloud Platform.

Before we begin…

Before we are able to start, there are some packages and dependencies that need to be installed. For some of these packages you can use homebrew for Mac and Ubuntu. For Windows, you can use chocolatey to make the installation process easier.

Requirements:

Setup:

How do I setup the Visual Studio Code Workplace?

  • Clone the HPCC-Platform repository into your git desktop.
  • Go to Visual Studio Code and open the HPCC-Platform folder.
  • Install git if it hasn’t already been installed. If not, VSC will prompt its installation.
  • Select the latest community version of HPCC-Platform. The current version is “community_7.10.2-1”.

How do I build the Docker image?

Run these commands separately:

cd dockerfiles

.\buildall.sh

Your Workstation and Project is now all set! You are now ready to create your Google Cloud Project!

  • Select the project that you want to enable to Kubernetes API on.
  • After the Kubernetes API is successfully enabled, select the “Create cluster” option.
  • You should now see a page with specifications for the cluster below

  • You can select your own settings for the cluster, but for this blog we will be choosing the “My first cluster” option in the top right corner.
  • After you create the cluster, you will be greeted with this screen below.

  • Select the “connect” button and copy the Command-line access command.
  • Run the command below in the Visual Studio Code Terminal with Powershell or Bash. This must be in the path of the HPCC-Platform Repository.
    gcloud container clusters get-credentials cluster-1 --zone us-central1-c --project [project-id]
    
  • Login and authenticate yourself.

Success! You are now ready to deploy HPCC-Platform on to your Google Kubernetes Cluster!

Deployment

There are two ways to deploy HPCC onto the Google Kubernetes Cluster. The easiest way by far is by using the shell scripts I have provided.

Shell Script Deployment:

Place the two files into the root of your HPCC-Platform project like so:

Run the hpcc-init.sh shell script with the command:

.\hpcc-init.sh

Success! You have deployed HPCC onto the Google Kubernetes Engine!

Manual Deployment:

Enter these commands SEPARATELY into the VSC terminal:

cd helm/examples/nfs

helm install nfsstorage hpcc-nfs/

cd ../../

helm install myhpcc hpcc/ --set global.image.version=latest -f examples/nfs/values-nfs.yaml

Success! You have deployed HPCC onto the Google Kubernetes Engine!

Removal

Like the deployment process, there are two ways to remove your HPCC-Platform Project.

Shell Script Removal:

Run the hpcc-remove.sh shell script with the command:

.\hpcc-remove.sh

Manual Removal:

Enter these commands SEPARATELY into the VSC terminal:

helm uninstall myhpcc

helm uninstall nfsstorage

Verify the Deletion

Run the command(s) below separately:

kubectl get pods

kubectl get pvc

kubectl get pv

You will see nothing being returned if deletion is successful. It may take a while for everything to be deleted, so step away for a few minutes and make yourself a cup of coffee :).

HPCC Tutorial

As with many other blogs, we will be going through a test run of the HPCC-Platform you have installed on your Google Kubernetes Cluster! However, there are a few items that must be addressed beforehand.

Requirements:

Setup:

Run the command featured below in your HPCC-Platform project terminal.

    kubectl cp ./originalperson <eclccserver pod name>:/var/lib/HPCCSystems/hpcc-data/

This command copies over data from your local machine to the Kubernetes Cluster on the file path /var/lib/HPCCSystems/hpcc-data/.

Next, switch over to your datatutorialworkspace project. We will now set up the ECL environment for testing.

Click on the Run button in Visual Studio Code. Or use the keyboard shortcut “Ctrl+Shift+D”. It should look like this:

Select “create a launch.json file” and fill the configuration array with the JSON provided below.

{
    "name": "play-hthor-submit",
    "type": "ecl",
    "request": "launch",
    //Change mode
    "mode": "submit",
    "workspace": "${workspaceRoot}",
    "program": "${file}",
    "protocol": "http",
    "serverAddress": "Your ecl server ip address",
    "port": "your ecl server port",
    "rejectUnauthorized": false,
    //change target cluster
    "targetCluster": "thor",
    "eclccPath": "${config:ecl.eclccPath}",
    "eclccArgs": [],
    "includeFolders": "${config:ecl.includeFolders}",
    "legacyMode": "${config:ecl.legacyMode}",
    "resultLimit": 100,
    "timeoutSecs": 60,
    "user": "",
    "password": ""
}

After the configuration array has been filled, create a folder in root of the project called “TutorialYourName”. We will be uploading ecl scripts here. Your file structure should look like this:

Tutorial:

For this blog we will only be completing one example as all the examples utilize the same process. Thankfully, the HPCC Systems Development team has made the ability to write and execute ECL code straightforward and simple.

In the documentation, it states that we should create a file called “layout_people.ecl”, and to populate it with some ecl code.

Despite what the tutorial says, you MUST have lowercase file names!!!!

Please populate the file with the following ecl code:

EXPORT Layout_People := RECORD
 STRING15 FirstName;
 STRING25 LastName;
 STRING15 MiddleName;
 STRING5 Zip;
 STRING42 Street;
 STRING20 City;
 STRING2 State;
END; 

After doing so, click on the Run button in Visual Studio Code. Or use the keyboard shortcut “Ctrl+Shift+D”. You should see the configuration “play-hthor-submit” at the very top. Click on the green play button to execute your ecl code.

Success! Your ecl code should now be executed! Go to your EclWatch IP Address to look at the process running. You can also see results from EclWatch.

Now repeat the procedure you used for the first example and apply them to the other examples in the documentation.

Disclaimer!

Some features in the tutorial do not work in the current Google Kubernetes Engine. Here is a list of what does not work.

  • Publishing Thor Query
  • Execute using WsECL
  • Compile and Publish the Roxie Query
  • Publish the Roxie query
  • Run the Roxie Query in WsECL

Next Steps:

The Google Cloud Platform leverages many services and technologies which ensures it as a leading cloud competitor in today’s environment. Future work with GCP will utilize it’s hosting services as well as its containerization management services.

Sources:

  1. “GKE Overview Kubernetes Engine Documentation Google Cloud.” Google, Google, cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview.

Thank you for reading my blog and stay tuned for more!

Please feel free to reach out to me for any questions at jeffs.mao@gmail.com


Tags: