Configure CA Certificates (GKE)

The following guide explains how to obtain a Certificate Authority (CA) signed certificate that can be used on a GKE cluster running HPE Machine Learning Inferencing Software for secure communication using HTTPS. Obtaining this certificate is necessary for enabling remote user authentication with identity providers such as Auth0, GitHub, Google, etc. Self-signed certificates are not accepted by identity providers.

Before You Start #

Obtain access to the Google Cloud project where the GKE cluster is running.
Install the gcloud CLI tool.
Review the official Google Cloud documentation on Requesting a certificate using Public CA and an ACME client.

How to Configure CA Certificates (GKE) #

Configure the Google Cloud Project #

Grant yourself the the Public CA External Account Key Creator (publicca.externalAccountKeyCreator) IAM role. Replace the value of --member with your email address:
```
gcloud projects add-iam-policy-binding <your-project-name> --member='user:<your-email-address>' --role=roles/publicca.externalAccountKeyCreator
```

Enable the Public CA APIs:

gcloud services enable publicca.googleapis.com

Install an Automatic Certificate Management Environment (ACME) client:

Tip

An ACME client is any software which can talk to an ACME-enabled Certificate Authority (such as Let’s Encrypt, BuyPass Go, ZeroSSL etc).
```
brew install certbot
```

For alternative steps on installing certbot, refer to the Certbot documentation and select the appropriate operating system.

Register an EAB Key ID and HMAC #

You need to register your ACME account with a Public CA to request certificates from the Public CA. An EAB secret can help you register your ACME account with your chosen Public CA. EAB secret consists of a key ID and a hash-based message authentication code (HMAC).

Request an EAB key ID and HMAC:

gcloud publicca external-account-keys create

Created an external account key
[b64MacKey: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
keyId: YYYYYYYYYYYYYYYYYYYYYYYY]

sudo certbot register --email "<your-email>" --no-eff-email --server "https://dv.acme-v02.api.pki.goog/directory" --eab-kid "<your-keyid-value>" --eab-hmac-key "<your-hmac-value>"

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at https://pki.goog/GTS-SA.pdf. You must agree
in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y
Account registered.

Request & Renew Certificate #

To request and renew a certificate, you must complete an ACME challenge to prove your control of the target domains. The list of challenge types you can choose from can be found in Lets Encrypt’s Documentation.

Since it is not always be possible to make changes to your organization’s DNS servers to create the DNS records required by the dns challenge, this example will focus on the http challenge instead. To use the http challenge, you will need to run a web server. In this example, we will use nginx as our web server.

Set Up NGINX Web Server #

The nginx web server will listen on port 80. To ensure that port 80 is not in use by another application, uninstall HPE Machine Learning Inferencing Software.

Obtain your cluster’s external IP address:

gcloud compute addresses describe ${USER} --region us-central1 --format="value(address)" 2> /dev/null

35.202.39.148

Get the hostname for your IP address, as the CA will only issue a certificate for a hostname, not an IP address:
```
host <your-external-ip>
```
```
148.39.202.35.in-addr.arpa domain name pointer 148.39.202.35.bc.googleusercontent.com.
```
Tip

The . (period) at the end of the hostname shown by the host command is not part of the hostname; do not include it.

Create a file named nginx-webserver.yaml with the following details (this will make your ngnix web server accessible to clients outside the cluster):

apiVersion: v1
kind: Service
metadata:
  name: ngnix-service
spec:
  selector:
    app: nginx
  type: LoadBalancer
  loadBalancerIP: <YOUR-GKE-CLUSTER-EXTERNAL-IP-ADDRESS>
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80

Start up your nginx web application:

kubectl apply -f nginx-webserver.yaml

service/ngnix-service created
deployment.apps/nginx-deployment created

Ensure that your nginx web server is reachable. Replace the hostname your GKE cluster’s hostname, obtained in a previous step:

 wget --no-verbose <your-gke-cluster-hostname>

2024-02-29 07:21:01 URL:http://148.39.202.35.bc.googleusercontent.com/ [615/615] -> "index.html" [1]

View the contents of the index.html file:

cat index.html

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Enter the shell of the nginx pod:

kubectl exec -it svc/ngnix-service -- /bin/bash

Keep this terminal open as you will need it to complete future steps.

Request a Certificate #

Open a separate terminal window.

Create a request for a certificate from the Public CA, but do not press enter yet:

sudo certbot certonly --manual --preferred-challenges "http" --server "https://dv.acme-v02.api.pki.goog/directory" --domains <your-gke-cluster-hostname>

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for 148.39.202.35.bc.googleusercontent.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Create a file containing just this data:

NFT7sAF8nQuBD2-Eic_RLv7ZC7EBEJGLuSYYBu6rcSHLVotGEt5F25MBrkZAIrjF.3WnzK340fSeZhU4VnUsF-6rF-ciwoJQmH4M1ySYlmfI

And make it available on your web server at this URL:

http://148.39.202.35.bc.googleusercontent.com/.well-known/acme-challenge/NFT7sAF8nQuBD2-Eic_RLv7ZC7EBEJGLuSYYBu6rcSHLVotGEt5F25MBrkZAIrjF

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Navigate back to the previous terminal window where you have the nginx pod shell open.

Create the directory structure and file as instructed by the certbot command:

mkdir -p /usr/share/nginx/html/.well-known/acme-challenge
echo "NFT7sAF8nQuBD2-Eic_RLv7ZC7EBEJGLuSYYBu6rcSHLVotGEt5F25MBrkZAIrjF.3WnzK340fSeZhU4VnUsF-6rF-ciwoJQmH4M1ySYlmfI" > /usr/share/nginx/html/.well-known/acme-challenge/NFT7sAF8nQuBD2-Eic_RLv7ZC7EBEJGLuSYYBu6rcSHLVotGEt5F25MBrkZAIrjF

Root Directory

The root directory for the nginx web server is /usr/share/nginx/html.

# grep root /etc/nginx/conf.d/default.conf
   root   /usr/share/nginx/html;

Return to the terminal where you ran the certbot command and press Enter. The paths where the certificate and key were saved will be displayed. You will need this information later on to create the secret needed for HTTPS/TLS:

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/148.39.202.35.bc.googleusercontent.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/148.39.202.35.bc.googleusercontent.com/privkey.pem
This certificate expires on 2024-05-26.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate's expiry date.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Remove the nginx service and deployment:
```
kubectl delete -f nginx-webserver.yaml
```
Use the certificate and key to create a Kubernetes secret that will be passed in as part of your helm install/upgrade command. You can refer to the Configure HTTPS/TLS guide for details.

Configure the CLI #

When using the CLI, you have the option to set the AIOLI_CONTROLLER_CERT_FILE environment variable to the location of the CA certificate, to verify that the server certificate presented by the controller (i.e., server) you are connecting to is valid, and that the server is really the one it claims to be.

Navigate to Google Trust Services.
Scroll to Download CA certificates.
Expand Root CAs and scroll to the GTS Root R1 certificate.
Select the Action dropdown > Select Certificate (PEM) to download.
Copy the downloaded CA root certificate to the location where you’d like to store the certificate:
```
mkdir ~/certs
cp ~/downloads/gtsr1.pem ~/certs/
```

Use the openssl command to validate the server certificate on the controller using the CA root certificate you downloaded:

 openssl s_client -connect <your-cluster-hostname> -CAfile ~/certs/gtsr1.pem

Connecting to 35.202.39.148
CONNECTED(00000005)
depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1
verify return:1
depth=1 C=US, O=Google Trust Services LLC, CN=GTS CA 1P5
verify return:1
depth=0 CN=148.39.202.35.bc.googleusercontent.com
verify return:1
---
Certificate chain
 0 s:CN=148.39.202.35.bc.googleusercontent.com
   i:C=US, O=Google Trust Services LLC, CN=GTS CA 1P5
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb 26 15:09:59 2024 GMT; NotAfter: May 26 15:09:58 2024 GMT
 1 s:C=US, O=Google Trust Services LLC, CN=GTS CA 1P5
   i:C=US, O=Google Trust Services LLC, CN=GTS Root R1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 13 00:00:42 2020 GMT; NotAfter: Sep 30 00:00:42 2027 GMT
 2 s:C=US, O=Google Trust Services LLC, CN=GTS Root R1
   i:C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIExTCCA62gAwIBAgIRAM4L2cbtQy1aEdaAlk653hkwDQYJKoZIhvcNAQELBQAw
RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
TEMxEzARBgNVBAMTCkdUUyBDQSAxUDUwHhcNMjQwMjI2MTUwOTU5WhcNMjQwNTI2
MTUwOTU4WjAxMS8wLQYDVQQDEyYxNDguMzkuMjAyLjM1LmJjLmdvb2dsZXVzZXJj
b250ZW50LmNvbTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABB0ZnxWthe2PrZ2V
WQNuxLX2AjhWJCZS1pPTv4mo/o1FyIxq7WNazm6GFTsjDMpOLFlz+0SQf6ugI0kd
hnk+y8mjggKMMIICiDAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUdwYfiegdC3LYJThJCXg4hFkT5Icw
HwYDVR0jBBgwFoAU1fyeDd8eyt0Il5duK8VfxSv17LgweAYIKwYBBQUHAQEEbDBq
MDUGCCsGAQUFBzABhilodHRwOi8vb2NzcC5wa2kuZ29vZy9zL2d0czFwNS9mTmVq
T3hwMkFiSTAxBggrBgEFBQcwAoYlaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMv
Z3RzMXA1LmRlcjAxBgNVHREEKjAogiYxNDguMzkuMjAyLjM1LmJjLmdvb2dsZXVz
ZXJjb250ZW50LmNvbTAhBgNVHSAEGjAYMAgGBmeBDAECATAMBgorBgEEAdZ5AgUD
MDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmxzLnBraS5nb29nL2d0czFwNS9m
TjdsYXVmSncxWS5jcmwwggEDBgorBgEEAdZ5AgQCBIH0BIHxAO8AdgCi4r/WHt4v
Lweg1k5tN6fcZUOwxrUuotq3iviabfUX2AAAAY3mLiR/AAAEAwBHMEUCIQCHn6dh
tCsAmfDOrfBREyiMk0G+Aqx4EmojZ5+nySFjjwIgcraQi0ktA9zHsp1H1U9ybutk
SVHPT6CXO8srga+vEW0AdQB2/4g/Crb7lVHCYcz1h7o0tKTNuyncaEIKn+ZnTFo6
dAAAAY3mLiSKAAAEAwBGMEQCICAmnDfy/mTrKiTarW7XQdVsFUvMoarO7i9Wig7N
W9JXAiANESgqRMTi4XHKeibBRsoRLH+NuHNtGG2CCUtaQCihFjANBgkqhkiG9w0B
AQsFAAOCAQEAeK37/PgTzo9BhfDps9xHaTMrd6Y7k9tv5TM2hD0WiVa1h/2yW61y
YUII9TNv/urV0CXxZfVLknCDEXTHPSPKeQpya/UeCtL9Th7JRwSz7I1U68211H9y
i2j6LcdWQ+SvfrLn2kjIs02eIEyLLspp7Jx+YviPponLjrDCKpVHPQOvTcLqwp8c
LGlhKnCpAbVKHtUwekzu9+zckF3sVgk6TS56jw+mcvS3vqGQO4pwfHlEBelow3r1
l67WesHl+GNOxjSnZBl8a3LxLBb1CFaq4MZd/JwS25x2wMy0DV8ICTQ0PEOn3YMX
zTXyXX55QfFOWtpt8HEcGQF44h24qrcuhQ==
-----END CERTIFICATE-----
subject=CN=148.39.202.35.bc.googleusercontent.com
issuer=C=US, O=Google Trust Services LLC, CN=GTS CA 1P5
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4347 bytes and written 426 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Tip

If you see an unable to get issuer certificate error confirm you downloaded the correct certificate.

depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1
verify error:num=2:unable to get issuer certificate
...

Set the AIOLI_CONTROLLER_CERT_FILE environment variable to the location of your CA certificate when you run the CLI or use the REST API:

AIOLI_CONTROLLER_CERT_FILE=~/certs/gtsr1.pem  aioli -c <your-gke-cluster-hostname> user whoami
You are logged in as user '<your-username>'

/usr/local/Cellar/curl/8.7.1/bin/curl --cacert ~/certs/gtsr1.pem  -s --show-error -X POST -H "Authorization: " -HContent-Type:application/json <your-gke-cluster-hostname>/api/v1/login -d '{ "username": "admin" }'
{"token":"v2.public.eyJpZCI6NCwidXNlcklkIjoxLCJleHBpcnkiOiIyMDI0LTA0LTE1VDE3OjI3OjI4LjU4NDM5NzQ5M1oiLCJJbmhlcml0ZWRDbGFpbXMiOm51bGx9gDCrYDFNemQ8WFrbuHmbJBzAuSfJfJaCwYuaubX-UvX8T6aarQ8e0JqPYvAgJsXVkc-ZzCXqmKAyLci2g_2XAA.bnVsbA"}