BYO Certs for TKG 1.5.3+ Auth

5 min readMay 21, 2022

Tanzu Kubernetes Grid (TKG) provides the means to deploy consistent and upstream aligned Kubernetes clusters leveraging the Kubernetes subproject Cluster API. TKG 1.4.0 introduced the use of Carvel API Resources format for core and user-managed packages. TKG Auth is delivered by the core package Pinniped.

This process was originally written up for TKG 1.4.0, and then a revised blog was written for TKG 1.4.1 due to a change in how auth was implemented. TKG 1.5.3 introduced a new change that requires a modification to procedure. This version of the post has been updated to accommodate the change.

You configure TKG Auth to meet your specific implementation requirements using IDENTITY_MANAGEMENT_TYPE and LDAP_ or ODIC_ config parameters. However, certain custom configuration patterns can only be applied after management cluster creation because of the dependencies on post-cluster-creation resources. The result of this base configuration, is the use of self-signed certs and auto-generated URIs (exact format is IaaS dependent) for Pinniped and Dex.

^ Initial Pinniped URL ^

^ Once Accepted you Get the Dex URL ^

^ Dex URL after agreeing to proceed ^

We want to present the browser based authentication flow to the user using a friendly FQDN, standard HTTPS port 443 and valid SSL certificates. We will follow these steps.

Setup DNS
Generate Certificates and Add to Cluster
Update Pinniped Core Package
“Post” Post Pinniped Deploy Configuration

Note: For the walk through below, we are assuming you have enabled LDAP and thus Dex is deployed with your TKG Management cluster. If you are using OIDC for auth, you can still follow the steps below, simply skip the references to DEX.

Initial Setup

I’ve deployed the management cluster with IDENTITY_MANAGEMENT_TYPE set to LDAP. My management cluster is named mgmt, and I want my FQDNs to be pinniped.mgmt.tanzu-lab.winterfell.life and dex.mgmt.tanzu-lab.winterfell.life. I have an offline process to generate certificates for my organization.

Setup DNS

First we setup the DNS entries for the custom FQDNs for the Pinniped Supervisor and Dex services.

You can retrieve the External IPs assigned to the services. Use the values from the query below to create DNS entries for pinniped.mgmt.tanzu-lab.winterfell.life and dex.mgmt.tanzu-lab.winterfell.life. Then skip to the next section.

$ kubectl get service pinniped-supervisor -n pinniped-supervisor
$ kubectl get service dexsvc -n tanzu-system-auth

Generate Certificates and Add to Cluster

Next follow your process to generate certificates in your organization for the desired FQDN. Create a secrets in the pinniped-supervisor and tanzu-system-auth namespace called custom-auth-cert-tls with the tls.key, tls.crt, and ca.crt.

It is essential that the ca.crt is part of the certificates. If you are using cert-manager and let’s encrypt, then ca.crt is not added to the secret. You will need to reconstruct a secret that has all three fields.

$ kubectl create secret generic custom-auth-cert-tls \
  --namespace pinniped-supervisor \
  --type=tls \
  --from-file=tls.key=path/to/key/file \
  --from-file=tls.crt=path/to/cert/file \
  --from-file=ca.crt=path/to/ca/file$ kubectl create secret generic custom-auth-cert-tls \
  --namespace tanzu-system-auth \
  --type=tls \
  --from-file=tls.key=path/to/key/file \
  --from-file=tls.crt=path/to/cert/file \
  --from-file=ca.crt=path/to/ca/file

Update Pinniped Core Package

Now that the certificates are in place, we need to update the Pinniped package to tell it about the secrets.

$ MGMT_CLUSTER_NAME=mgmt # udpate accordingly$ kubectl get secret $MGMT_CLUSTER_NAME-pinniped-addon -n tkg-system -ojsonpath="{.data.values\.yaml}" | base64 --decode > /tmp/pinniped-addon-values.yaml

Edit the /tmp/pinniped-addon-values.yaml, setting .custom_tls_secret = “custom-auth-cert-tls”.

...
custom_tls_secret: "custom-auth-cert-tls"
...
pinniped:
...
  supervisor_svc_external_dns: https://pinniped.mgmt.tanzu-lab.winterfell.life # but with your domain
...

Patch the addon secret values.yaml key.

$ NEW_VALUES_YAML=`cat /tmp/pinniped-addon-values.yaml | base64`$ kubectl patch secret $MGMT_CLUSTER_NAME-pinniped-addon -n tkg-system -p '{"data": {"values.yaml": "'$NEW_VALUES_YAML'"}}'

By patching the $MGMT_CLUSTER_NAME-pinniped-addon secret, the pinniped supervisor and dex pods will be re-created and terminate its SSL services with the custom TLS certificates. The pinniped-post-deploy-job will also be re-run.

$ kubectl get jobs -n pinniped-supervisor
NAME                       COMPLETIONS   DURATION   AGE
pinniped-post-deploy-job   1/1           6m44s      6m44s

“Post” Post Pinniped Deploy Configuration (LDAP Only)

The pinniped-post-deploy-job updates resources based upon the configuration. However, for LDAP configuration, it does not know about our dex FQDNs. So we need to manually update the dex config map and OIDCIdentityProvider pinniped resources with references to our dex FQDN used for our custom TLS certificates.

$ DEX_URL=https://dex.mgmt.tanzu-lab.winterfell.life$ kubectl edit cm dex -n tanzu-system-auth# edit and save configmap ......
data:
  config.yaml:
    issuer = <dex_url_from_above>
...# And bounce dex$ kubectl rollout restart deployment dex --namespace tanzu-system-auth# Patch the OIDCIdentityProvider$ kubectl patch oidcidentityprovider upstream-oidc-identity-provider \
 -n pinniped-supervisor \
 --type json \
 -p="[{'op': 'replace', 'path': '/spec/issuer', 'value':'$DEX_URL'}]"

That’s it, you should now be able to retrieve your kubeconfig and access your clusters.

$ MGMT_CLUSTER_NAME=mgmt # udpate accordingly$ tanzu management-cluster kubeconfig get$ kubectl config use-context tanzu-cli-$MGMT_CLUSTER_NAME@$MGMT_CLUSTER_NAME$ kubectl get all

Dex URL with Trusted Cert