In part 3, we will be creating a namespace and deploying a Tanzu Kubernetes Cluster.
In parts 1 and 2, we created our HAProxy VM and implemented workload management which provides our load balancing and supervisor cluster.
Now we need to create a namespace so we can start deploying Tanzu Kubernetes Clusters and workloads.
It seems with default configuration, we are unable to create a namespace with the kubectl tools. So we will need to create it in the vSphere client, then assign VM classes, storage policy, and any additional permissions as needed.
Also to note that, since we are not using NSX-T, we are unable to deploy workload pods directly on to the ESXi hosts. This is why we must deploy Tanzu Kubernetes Clusters as those will run the actual workload.
In the vSphere client, go to the Menu then Workload Management.

Select Create Namespace.

Select the cluster, provide a DNS-compliant name, then a description if you want.

It should load us right in to the new namespace and status should be running.

Next we need to assign our storage policy, so select Add Storage.

Select the storage policy we created previously then click OK.

We should now see the storage policy applied with zero persistent volume claims.

Next we need to assign VM classes that will be available to this namespace. Under VM Service, select Add VM Class.

Select the VM classes we want available. Being test, best effort is sufficient for my needs. Resources are not reserved for Best Effort classes. For production, it’s certainly possible that guaranteed would be warranted, as the resources are reserved for the VMs associated with that cluster.
I just selected 5 of the best effort classes, then OK.

We should now see the associated VM classes we just selected, but we also need to assign the Tanzu content library. So select Add Content Library.

Now add the Tanzu Content Library we created previously and click OK.

Now that we have the namespace created and resources assigned, we can create the Tanzu Kubernetes Cluster. We will do this with the kubectl tools we previously downloaded from our supervisor cluster.
I will be using a YAML file to define the TKC name, version, VM classes, and number of control plane and worker nodes.
Here is my YAML. I’m creating a Tanzu Kubernetes Cluster with name test-cluster in namespace test-ns, version 1.20.9, 1 control plane VM with best effort small using tanzu-storage-policy. There are 2 workers with same VM class and storage policy.
apiVersion: run.tanzu.vmware.com/v1alpha1 kind: TanzuKubernetesCluster metadata: name: test-cluster01 namespace: test-ns spec: distribution: version: v1.20.9 topology: controlPlane: count: 1 class: best-effort-small storageClass: tanzu-storage-policy workers: count: 2 class: best-effort-small storageClass: tanzu-storage-policy
Let’s connect to our Tanzu deployment and validate the assigned storage policy and VM classes.
C:\>kubectl vsphere login --vsphere-username [email protected] --server=10.0.12.129 --insecure-skip-tls-verify KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below Password: Logged in successfully. You have access to the following contexts: 10.0.12.129 test-ns If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. To change context, use `kubectl config use-context <workload name>`
We logged in, via kubectl, to our Tanzu workload manage (which is fronted via HAProxy and the front-end network) with the Administrator SSO account. As we have not replaced the current self-signed certificate, we need the –insecure-skip-tls-verify switch. We can see that we have access to 10.0.12.129 which is primary management, along with the test-ns namespace.
To get the assigned VM classes and storage policies, we need to change to the namespace context then do a get on the API resources.
C:\>kubectl config use-context test-ns Switched to context "test-ns". C:\>kubectl get virtualmachineclassbindings NAME VIRTUALMACHINECLASS AGE best-effort-large best-effort-large 27m best-effort-medium best-effort-medium 27m best-effort-small best-effort-small 27m best-effort-xlarge best-effort-xlarge 27m best-effort-xsmall best-effort-xsmall 27m C:\>kubectl get storageclasses NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE tanzu-storage-policy csi.vsphere.vmware.com Delete Immediate true 15d
We can see that we do have the appropriate VM classes and storage policy assigned. So our cluster YAML should be good.
We will make sure we are in our namespace context, then use kubectl to apply our cluster YAML file. This should create a TKC named test-cluster01 with 1 control plane VM and 2 worker VMs, all using best-effort-small VM class and tanzu-storage-policy for storage.
We will have another command prompt open, also in namespace context, watching events.
C:\>kubectl apply -f C:\kubectl\test-cluster01.yaml tanzukubernetescluster.run.tanzu.vmware.com/test-cluster01 created
If we go back to the vSphere console real quick, we can see OVF templates being deployed and our new control plane VM is being spun up.

C:\>kubectl get events -w LAST SEEN TYPE REASON OBJECT MESSAGE 0s Normal GeneratedKey certificate/test-cluster01-extensions-ca Generated a new private key 0s Normal Requested certificate/test-cluster01-extensions-ca Created new CertificateRequest resource "test-cluster01-extensions-ca-1421064582" 0s Normal CertificateIssued certificaterequest/test-cluster01-extensions-ca-1421064582 Certificate fetched from issuer successfully 0s Warning ErrGetKeyPair issuer/test-cluster01-extensions-ca-issuer Error getting keypair for CA issuer: error decoding certificate PEM block 0s Warning ErrInitIssuer issuer/test-cluster01-extensions-ca-issuer Error initializing issuer: error decoding certificate PEM block 0s Warning ErrGetKeyPair issuer/test-cluster01-extensions-ca-issuer Error getting keypair for CA issuer: error decoding certificate PEM block 0s Warning ErrInitIssuer issuer/test-cluster01-extensions-ca-issuer Error initializing issuer: error decoding certificate PEM block 0s Normal Issued certificate/test-cluster01-extensions-ca Certificate issued successfully 0s Warning ReconcileFailure wcpcluster/test-cluster01 unexpected error while reconciling control plane endpoint for test-cluster01: failed to reconcile loadbalanced endpoint for WCPCluster test-ns/test-cluster01: failed to get control plane endpoint for Cluster test-ns/test-cluster01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses 0s Normal CreateK8sServiceSuccess virtualmachineservice/test-cluster01-control-plane-service CreateK8sService success 0s Warning ReconcileFailure wcpcluster/test-cluster01 unexpected error while reconciling control plane endpoint for test-cluster01: failed to reconcile loadbalanced endpoint for WCPCluster test-ns/test-cluster01: failed to get control plane endpoint for Cluster test-ns/test-cluster01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses 0s Warning ReconcileFailure wcpcluster/test-cluster01 unexpected error while reconciling control plane endpoint for test-cluster01: failed to reconcile loadbalanced endpoint for WCPCluster test-ns/test-cluster01: failed to get control plane endpoint for Cluster test-ns/test-cluster01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Normal SuccessfulCreate machinedeployment/test-cluster01-workers-tcknk Created MachineSet "test-cluster01-workers-tcknk-7449644546" 0s Normal SuccessfulCreate machineset/test-cluster01-workers-tcknk-7449644546 Created machine "test-cluster01-workers-tcknk-7449644546-bvpjx" 0s Normal SuccessfulCreate machineset/test-cluster01-workers-tcknk-7449644546 Created machine "test-cluster01-workers-tcknk-7449644546-kdxnv" 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Normal KeyPairVerified issuer/test-cluster01-extensions-ca-issuer Signing CA verified 0s Normal KeyPairVerified issuer/test-cluster01-extensions-ca-issuer Signing CA verified 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 1s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning Failed to retrieve Node by ProviderID machine/test-cluster01-control-plane-p5fzt the cache is not started, can not read objects 0s Normal GeneratedKey certificate/test-cluster01-antrea Generated a new private key 0s Normal Requested certificate/test-cluster01-antrea Created new CertificateRequest resource "test-cluster01-antrea-1172904443" 0s Normal CertificateIssued certificaterequest/test-cluster01-antrea-1172904443 Certificate fetched from issuer successfully 0s Normal Issued certificate/test-cluster01-antrea Certificate issued successfully 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal GeneratedKey certificate/test-cluster01-auth-svc-cert Generated a new private key 0s Normal Requested certificate/test-cluster01-auth-svc-cert Created new CertificateRequest resource "test-cluster01-auth-svc-cert-828030165" 0s Normal CertificateIssued certificaterequest/test-cluster01-auth-svc-cert-828030165 Certificate fetched from issuer successfully 0s Normal Issued certificate/test-cluster01-auth-svc-cert Certificate issued successfully 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal GeneratedKey certificate/test-cluster01-metrics-server-cert Generated a new private key 0s Normal Requested certificate/test-cluster01-metrics-server-cert Created new CertificateRequest resource "test-cluster01-metrics-server-cert-2884180492" 0s Normal CertificateIssued certificaterequest/test-cluster01-metrics-server-cert-2884180492 Certificate fetched from issuer successfully 0s Normal Issued certificate/test-cluster01-metrics-server-cert Certificate issued successfully 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 1s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal SuccessfulSetNodeRef machine/test-cluster01-control-plane-p5fzt test-cluster01-control-plane-p5fzt 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal SuccessfulSetNodeRef machine/test-cluster01-control-plane-p5fzt test-cluster01-control-plane-p5fzt 0s Normal Ready virtualmachine/test-cluster01-control-plane-p5fzt 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning UpdateFailure virtualmachine/test-cluster01-workers-tcknk-7449644546-bvpjx NetworkInterface.netoperator.vmware.com "tanzu-workload-network-test-cluster01-workers-tcknk-7449644546-bvpjx" not found 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet powered on: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet powered on: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet powered on: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 1s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 1s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal SuccessfulSetNodeRef machine/test-cluster01-workers-tcknk-7449644546-kdxnv test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal SuccessfulSetNodeRef machine/test-cluster01-workers-tcknk-7449644546-bvpjx test-cluster01-workers-tcknk-7449644546-bvpjx 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/test-cluster01-workers-tcknk-7449644546-bvpjx has unhealthy node test-cluster01-workers-tcknk-7449644546-bvpjx 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 1s Normal SuccessfulSetNodeRef machine/test-cluster01-workers-tcknk-7449644546-bvpjx test-cluster01-workers-tcknk-7449644546-bvpjx 0s Normal PhaseChanged tanzukubernetescluster/test-cluster01 cluster changes from creating phase to running phase 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success
It looks like the TKC deployment completed. Let’s now do a describe on our new Tanzu Kubernetes Cluster.
C:\>kubectl describe tkc test-cluster01 Name: test-cluster01 Namespace: test-ns Labels: run.tanzu.vmware.com/tkr=v1.20.9---vmware.1-tkg.1.a4cee5b Annotations: <none> API Version: run.tanzu.vmware.com/v1alpha1 Kind: TanzuKubernetesCluster Metadata: Creation Timestamp: 2021-10-09T20:31:17Z Finalizers: tanzukubernetescluster.run.tanzu.vmware.com Generation: 1 Managed Fields: API Version: run.tanzu.vmware.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:distribution: .: f:version: f:topology: .: f:controlPlane: .: f:class: f:count: f:storageClass: f:workers: .: f:class: f:count: f:storageClass: Manager: kubectl-client-side-apply Operation: Update Time: 2021-10-09T20:31:17Z API Version: run.tanzu.vmware.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: f:labels: .: f:run.tanzu.vmware.com/tkr: f:status: .: f:addons: .: f:authsvc: .: f:conditions: f:name: f:status: f:version: f:cloudprovider: .: f:conditions: f:name: f:status: f:version: f:cni: .: f:conditions: f:name: f:status: f:version: f:csi: .: f:conditions: f:name: f:status: f:version: f:dns: .: f:conditions: f:name: f:status: f:version: f:metrics-server: .: f:conditions: f:name: f:status: f:version: f:proxy: .: f:conditions: f:name: f:status: f:version: f:psp: .: f:conditions: f:name: f:status: f:version: f:clusterApiStatus: .: f:apiEndpoints: f:phase: f:conditions: f:nodeStatus: .: f:test-cluster01-control-plane-p5fzt: f:test-cluster01-workers-tcknk-7449644546-bvpjx: f:test-cluster01-workers-tcknk-7449644546-kdxnv: f:phase: f:vmStatus: .: f:test-cluster01-control-plane-p5fzt: f:test-cluster01-workers-tcknk-7449644546-bvpjx: f:test-cluster01-workers-tcknk-7449644546-kdxnv: Manager: manager Operation: Update Time: 2021-10-09T20:36:54Z Resource Version: 138742703 UID: f41059eb-3f4d-4add-a73b-67234babd06f Spec: Distribution: Full Version: v1.20.9+vmware.1-tkg.1.a4cee5b Version: v1.20.9 Settings: Network: Cni: Name: antrea Pods: Cidr Blocks: 192.168.0.0/16 Service Domain: cluster.local Services: Cidr Blocks: 10.96.0.0/12 Topology: Control Plane: Class: best-effort-small Count: 1 Storage Class: tanzu-storage-policy Workers: Class: best-effort-small Count: 2 Storage Class: tanzu-storage-policy Status: Addons: Authsvc: Conditions: Last Transition Time: 2021-10-09T20:34:32Z Status: True Type: AuthServiceProvisioned Name: authsvc Status: applied Version: 0.1-68-gac93c34 Cloudprovider: Conditions: Last Transition Time: 2021-10-09T20:34:08Z Status: True Type: CPIProvisioned Name: vmware-guest-cluster Status: applied Version: 0.1-89-g3d24a22 Cni: Conditions: Last Transition Time: 2021-10-09T20:34:24Z Status: True Type: CNIProvisioned Name: antrea Status: applied Version: v0.11.4_vmware.2 Csi: Conditions: Last Transition Time: 2021-10-09T20:34:10Z Status: True Type: CSIProvisioned Name: pvcsi Status: applied Version: vsphere70u2-4c32126-73daa24 Dns: Conditions: Last Transition Time: 2021-10-09T20:34:05Z Status: True Type: CoreDNSProvisioned Name: CoreDNS Status: applied Version: v1.7.0_vmware.14 Metrics - Server: Conditions: Last Transition Time: 2021-10-09T20:34:37Z Status: True Type: MetricsServerProvisioned Name: metrics-server Status: applied Version: v0.4.0+vmware.1 Proxy: Conditions: Last Transition Time: 2021-10-09T20:34:05Z Status: True Type: KubeProxyProvisioned Name: kube-proxy Status: applied Version: 1.20.9+vmware.1 Psp: Conditions: Last Transition Time: 2021-10-09T20:34:03Z Status: True Type: PSPProvisioned Name: defaultpsp Status: applied Version: v1.20.9+vmware.1-tkg.1.a4cee5b Cluster API Status: API Endpoints: Host: 10.0.12.131 Port: 6443 Phase: Provisioned Conditions: Last Transition Time: 2021-10-09T20:34:37Z Status: True Type: AddonsReady Last Transition Time: 2021-10-09T20:34:43Z Status: True Type: ControlPlaneReady Last Transition Time: 2021-10-09T20:37:24Z Message: 1/1 Control Plane Node(s) healthy. 2/2 Worker Node(s) healthy Status: True Type: NodesHealthy Last Transition Time: 2021-10-09T20:34:03Z Status: True Type: ProviderServiceAccountsReady Last Transition Time: 2021-10-09T20:34:03Z Status: True Type: RoleBindingSynced Last Transition Time: 2021-10-09T20:34:45Z Status: True Type: ServiceDiscoveryReady Last Transition Time: 2021-10-09T20:34:42Z Status: True Type: StorageClassSynced Last Transition Time: 2021-10-09T20:34:43Z Status: True Type: TanzuKubernetesReleaseCompatible Last Transition Time: 2021-09-30T14:22:59Z Reason: NoUpdates Status: False Type: UpdatesAvailable Node Status: test-cluster01-control-plane-p5fzt: ready test-cluster01-workers-tcknk-7449644546-bvpjx: ready test-cluster01-workers-tcknk-7449644546-kdxnv: ready Phase: running Vm Status: test-cluster01-control-plane-p5fzt: ready test-cluster01-workers-tcknk-7449644546-bvpjx: ready test-cluster01-workers-tcknk-7449644546-kdxnv: ready Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PhaseChanged 2m58s vmware-system-tkg/vmware-system-tkg-controller-manager/tanzukubernetescluster-status-controller cluster changes from creating phase to running phase
Describe also reports the cluster being in the running phase. We can see the API endpoint IP of 10.0.12.131. Again, because we deployed with HAProxy and the frontend network, the exposed IPs will be on our frontend network.
We can also use get tkc and get vm to less details on these resources.
C:\>kubectl get tkc NAME CONTROL PLANE WORKER DISTRIBUTION AGE PHASE TKR COMPATIBLE UPDATES AVAILABLE test-cluster01 1 2 v1.20.9+vmware.1-tkg.1.a4cee5b 26m running True C:\>kubectl get vm NAME POWERSTATE AGE test-cluster01-control-plane-p5fzt poweredOn 26m test-cluster01-workers-tcknk-7449644546-bvpjx poweredOn 23m test-cluster01-workers-tcknk-7449644546-kdxnv poweredOn 23m
Now that we have our first Tanzu Kubernetes Cluster deployed, we will be deploying a workload on to the cluster. Hopefully I will be getting that posted before too long.
Implementing Tanzu in vSphere 7U1 Part 1: Pre-requisites and HAProxy
Implementing Tanzu in vSphere 7U1 Part 2: Enabling Workload Management