In part 3, we will be creating a namespace and deploying a Tanzu Kubernetes Cluster.
In parts 1 and 2, we created our HAProxy VM and implemented workload management which provides our load balancing and supervisor cluster.
Now we need to create a namespace so we can start deploying Tanzu Kubernetes Clusters and workloads.
It seems with default configuration, we are unable to create a namespace with the kubectl tools. So we will need to create it in the vSphere client, then assign VM classes, storage policy, and any additional permissions as needed.
Also to note that, since we are not using NSX-T, we are unable to deploy workload pods directly on to the ESXi hosts. This is why we must deploy Tanzu Kubernetes Clusters as those will run the actual workload.
In the vSphere client, go to the Menu then Workload Management.
Select Create Namespace.
Select the cluster, provide a DNS-compliant name, then a description if you want.
It should load us right in to the new namespace and status should be running.
Next we need to assign our storage policy, so select Add Storage.
Select the storage policy we created previously then click OK.
We should now see the storage policy applied with zero persistent volume claims.
Next we need to assign VM classes that will be available to this namespace. Under VM Service, select Add VM Class.
Select the VM classes we want available. Being test, best effort is sufficient for my needs. Resources are not reserved for Best Effort classes. For production, it’s certainly possible that guaranteed would be warranted, as the resources are reserved for the VMs associated with that cluster.
I just selected 5 of the best effort classes, then OK.
We should now see the associated VM classes we just selected, but we also need to assign the Tanzu content library. So select Add Content Library.
Now add the Tanzu Content Library we created previously and click OK.
Now that we have the namespace created and resources assigned, we can create the Tanzu Kubernetes Cluster. We will do this with the kubectl tools we previously downloaded from our supervisor cluster.
I will be using a YAML file to define the TKC name, version, VM classes, and number of control plane and worker nodes.
Here is my YAML. I’m creating a Tanzu Kubernetes Cluster with name test-cluster in namespace test-ns, version 1.20.9, 1 control plane VM with best effort small using tanzu-storage-policy. There are 2 workers with same VM class and storage policy.
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
name: test-cluster01
namespace: test-ns
spec:
distribution:
version: v1.20.9
topology:
controlPlane:
count: 1
class: best-effort-small
storageClass: tanzu-storage-policy
workers:
count: 2
class: best-effort-small
storageClass: tanzu-storage-policyLet’s connect to our Tanzu deployment and validate the assigned storage policy and VM classes.
C:\>kubectl vsphere login --vsphere-username [email protected] --server=10.0.12.129 --insecure-skip-tls-verify KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below Password: Logged in successfully. You have access to the following contexts: 10.0.12.129 test-ns If the context you wish to use is not in this list, you may need to try logging in again later, or contact your cluster administrator. To change context, use `kubectl config use-context <workload name>`
We logged in, via kubectl, to our Tanzu workload manage (which is fronted via HAProxy and the front-end network) with the Administrator SSO account. As we have not replaced the current self-signed certificate, we need the –insecure-skip-tls-verify switch. We can see that we have access to 10.0.12.129 which is primary management, along with the test-ns namespace.
To get the assigned VM classes and storage policies, we need to change to the namespace context then do a get on the API resources.
C:\>kubectl config use-context test-ns Switched to context "test-ns". C:\>kubectl get virtualmachineclassbindings NAME VIRTUALMACHINECLASS AGE best-effort-large best-effort-large 27m best-effort-medium best-effort-medium 27m best-effort-small best-effort-small 27m best-effort-xlarge best-effort-xlarge 27m best-effort-xsmall best-effort-xsmall 27m C:\>kubectl get storageclasses NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE tanzu-storage-policy csi.vsphere.vmware.com Delete Immediate true 15d
We can see that we do have the appropriate VM classes and storage policy assigned. So our cluster YAML should be good.
We will make sure we are in our namespace context, then use kubectl to apply our cluster YAML file. This should create a TKC named test-cluster01 with 1 control plane VM and 2 worker VMs, all using best-effort-small VM class and tanzu-storage-policy for storage.
We will have another command prompt open, also in namespace context, watching events.
C:\>kubectl apply -f C:\kubectl\test-cluster01.yaml tanzukubernetescluster.run.tanzu.vmware.com/test-cluster01 created
If we go back to the vSphere console real quick, we can see OVF templates being deployed and our new control plane VM is being spun up.
C:\>kubectl get events -w LAST SEEN TYPE REASON OBJECT MESSAGE 0s Normal GeneratedKey certificate/test-cluster01-extensions-ca Generated a new private key 0s Normal Requested certificate/test-cluster01-extensions-ca Created new CertificateRequest resource "test-cluster01-extensions-ca-1421064582" 0s Normal CertificateIssued certificaterequest/test-cluster01-extensions-ca-1421064582 Certificate fetched from issuer successfully 0s Warning ErrGetKeyPair issuer/test-cluster01-extensions-ca-issuer Error getting keypair for CA issuer: error decoding certificate PEM block 0s Warning ErrInitIssuer issuer/test-cluster01-extensions-ca-issuer Error initializing issuer: error decoding certificate PEM block 0s Warning ErrGetKeyPair issuer/test-cluster01-extensions-ca-issuer Error getting keypair for CA issuer: error decoding certificate PEM block 0s Warning ErrInitIssuer issuer/test-cluster01-extensions-ca-issuer Error initializing issuer: error decoding certificate PEM block 0s Normal Issued certificate/test-cluster01-extensions-ca Certificate issued successfully 0s Warning ReconcileFailure wcpcluster/test-cluster01 unexpected error while reconciling control plane endpoint for test-cluster01: failed to reconcile loadbalanced endpoint for WCPCluster test-ns/test-cluster01: failed to get control plane endpoint for Cluster test-ns/test-cluster01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses 0s Normal CreateK8sServiceSuccess virtualmachineservice/test-cluster01-control-plane-service CreateK8sService success 0s Warning ReconcileFailure wcpcluster/test-cluster01 unexpected error while reconciling control plane endpoint for test-cluster01: failed to reconcile loadbalanced endpoint for WCPCluster test-ns/test-cluster01: failed to get control plane endpoint for Cluster test-ns/test-cluster01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses 0s Warning ReconcileFailure wcpcluster/test-cluster01 unexpected error while reconciling control plane endpoint for test-cluster01: failed to reconcile loadbalanced endpoint for WCPCluster test-ns/test-cluster01: failed to get control plane endpoint for Cluster test-ns/test-cluster01: VirtualMachineService LB does not yet have VIP assigned: VirtualMachineService LoadBalancer does not have any Ingresses 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Normal SuccessfulCreate machinedeployment/test-cluster01-workers-tcknk Created MachineSet "test-cluster01-workers-tcknk-7449644546" 0s Normal SuccessfulCreate machineset/test-cluster01-workers-tcknk-7449644546 Created machine "test-cluster01-workers-tcknk-7449644546-bvpjx" 0s Normal SuccessfulCreate machineset/test-cluster01-workers-tcknk-7449644546 Created machine "test-cluster01-workers-tcknk-7449644546-kdxnv" 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Normal KeyPairVerified issuer/test-cluster01-extensions-ca-issuer Signing CA verified 0s Normal KeyPairVerified issuer/test-cluster01-extensions-ca-issuer Signing CA verified 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error fetching REST client config for remote cluster "test-ns/test-cluster01": failed to retrieve kubeconfig secret for Cluster test-ns/test-cluster01: secrets "test-cluster01-kubeconfig" not found 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 1s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: dial tcp 10.0.12.131:6443: connect: connection refused 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileFailure wcpmachine/test-cluster01-control-plane-jx4g2-twvdc vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-control-plane-jx4g2-twvdc 0s Warning ReconcileError machinehealthcheck/test-cluster01-workers-tcknk error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Warning ReconcileError machinehealthcheck/test-cluster01-control-plane error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "test-ns/test-cluster01": Get https://10.0.12.131:6443/api?timeout=10s: EOF 0s Warning Failed to retrieve Node by ProviderID machine/test-cluster01-control-plane-p5fzt the cache is not started, can not read objects 0s Normal GeneratedKey certificate/test-cluster01-antrea Generated a new private key 0s Normal Requested certificate/test-cluster01-antrea Created new CertificateRequest resource "test-cluster01-antrea-1172904443" 0s Normal CertificateIssued certificaterequest/test-cluster01-antrea-1172904443 Certificate fetched from issuer successfully 0s Normal Issued certificate/test-cluster01-antrea Certificate issued successfully 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal GeneratedKey certificate/test-cluster01-auth-svc-cert Generated a new private key 0s Normal Requested certificate/test-cluster01-auth-svc-cert Created new CertificateRequest resource "test-cluster01-auth-svc-cert-828030165" 0s Normal CertificateIssued certificaterequest/test-cluster01-auth-svc-cert-828030165 Certificate fetched from issuer successfully 0s Normal Issued certificate/test-cluster01-auth-svc-cert Certificate issued successfully 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal GeneratedKey certificate/test-cluster01-metrics-server-cert Generated a new private key 0s Normal Requested certificate/test-cluster01-metrics-server-cert Created new CertificateRequest resource "test-cluster01-metrics-server-cert-2884180492" 0s Normal CertificateIssued certificaterequest/test-cluster01-metrics-server-cert-2884180492 Certificate fetched from issuer successfully 0s Normal Issued certificate/test-cluster01-metrics-server-cert Certificate issued successfully 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 1s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm is not yet created: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal SuccessfulSetNodeRef machine/test-cluster01-control-plane-p5fzt test-cluster01-control-plane-p5fzt 0s Normal DetectedUnhealthy machine/test-cluster01-control-plane-p5fzt Machine test-ns/test-cluster01-control-plane/test-cluster01-control-plane-p5fzt/ has unhealthy node 0s Normal SuccessfulSetNodeRef machine/test-cluster01-control-plane-p5fzt test-cluster01-control-plane-p5fzt 0s Normal Ready virtualmachine/test-cluster01-control-plane-p5fzt 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Warning UpdateFailure virtualmachine/test-cluster01-workers-tcknk-7449644546-bvpjx NetworkInterface.netoperator.vmware.com "tanzu-workload-network-test-cluster01-workers-tcknk-7449644546-bvpjx" not found 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet powered on: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet powered on: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm is not yet powered on: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-9644d vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-9644d 0s Warning ReconcileFailure wcpmachine/test-cluster01-workers-fr5mw-7m7td vm does not have an IP address: vmware-system-capw-controller-manager/WCPMachine//test-ns/test-cluster01/test-cluster01-workers-fr5mw-7m7td 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 1s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 1s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal SuccessfulSetNodeRef machine/test-cluster01-workers-tcknk-7449644546-kdxnv test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal SuccessfulSetNodeRef machine/test-cluster01-workers-tcknk-7449644546-bvpjx test-cluster01-workers-tcknk-7449644546-bvpjx 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/ has unhealthy node 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-bvpjx Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-bvpjx/test-cluster01-workers-tcknk-7449644546-bvpjx has unhealthy node test-cluster01-workers-tcknk-7449644546-bvpjx 0s Normal DetectedUnhealthy machine/test-cluster01-workers-tcknk-7449644546-kdxnv Machine test-ns/test-cluster01-workers-tcknk/test-cluster01-workers-tcknk-7449644546-kdxnv/test-cluster01-workers-tcknk-7449644546-kdxnv has unhealthy node test-cluster01-workers-tcknk-7449644546-kdxnv 1s Normal SuccessfulSetNodeRef machine/test-cluster01-workers-tcknk-7449644546-bvpjx test-cluster01-workers-tcknk-7449644546-bvpjx 0s Normal PhaseChanged tanzukubernetescluster/test-cluster01 cluster changes from creating phase to running phase 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success 0s Normal Reconcile gateway/test-cluster01-control-plane-service Success
It looks like the TKC deployment completed. Let’s now do a describe on our new Tanzu Kubernetes Cluster.
C:\>kubectl describe tkc test-cluster01
Name: test-cluster01
Namespace: test-ns
Labels: run.tanzu.vmware.com/tkr=v1.20.9---vmware.1-tkg.1.a4cee5b
Annotations: <none>
API Version: run.tanzu.vmware.com/v1alpha1
Kind: TanzuKubernetesCluster
Metadata:
Creation Timestamp: 2021-10-09T20:31:17Z
Finalizers:
tanzukubernetescluster.run.tanzu.vmware.com
Generation: 1
Managed Fields:
API Version: run.tanzu.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:distribution:
.:
f:version:
f:topology:
.:
f:controlPlane:
.:
f:class:
f:count:
f:storageClass:
f:workers:
.:
f:class:
f:count:
f:storageClass:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-10-09T20:31:17Z
API Version: run.tanzu.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
f:labels:
.:
f:run.tanzu.vmware.com/tkr:
f:status:
.:
f:addons:
.:
f:authsvc:
.:
f:conditions:
f:name:
f:status:
f:version:
f:cloudprovider:
.:
f:conditions:
f:name:
f:status:
f:version:
f:cni:
.:
f:conditions:
f:name:
f:status:
f:version:
f:csi:
.:
f:conditions:
f:name:
f:status:
f:version:
f:dns:
.:
f:conditions:
f:name:
f:status:
f:version:
f:metrics-server:
.:
f:conditions:
f:name:
f:status:
f:version:
f:proxy:
.:
f:conditions:
f:name:
f:status:
f:version:
f:psp:
.:
f:conditions:
f:name:
f:status:
f:version:
f:clusterApiStatus:
.:
f:apiEndpoints:
f:phase:
f:conditions:
f:nodeStatus:
.:
f:test-cluster01-control-plane-p5fzt:
f:test-cluster01-workers-tcknk-7449644546-bvpjx:
f:test-cluster01-workers-tcknk-7449644546-kdxnv:
f:phase:
f:vmStatus:
.:
f:test-cluster01-control-plane-p5fzt:
f:test-cluster01-workers-tcknk-7449644546-bvpjx:
f:test-cluster01-workers-tcknk-7449644546-kdxnv:
Manager: manager
Operation: Update
Time: 2021-10-09T20:36:54Z
Resource Version: 138742703
UID: f41059eb-3f4d-4add-a73b-67234babd06f
Spec:
Distribution:
Full Version: v1.20.9+vmware.1-tkg.1.a4cee5b
Version: v1.20.9
Settings:
Network:
Cni:
Name: antrea
Pods:
Cidr Blocks:
192.168.0.0/16
Service Domain: cluster.local
Services:
Cidr Blocks:
10.96.0.0/12
Topology:
Control Plane:
Class: best-effort-small
Count: 1
Storage Class: tanzu-storage-policy
Workers:
Class: best-effort-small
Count: 2
Storage Class: tanzu-storage-policy
Status:
Addons:
Authsvc:
Conditions:
Last Transition Time: 2021-10-09T20:34:32Z
Status: True
Type: AuthServiceProvisioned
Name: authsvc
Status: applied
Version: 0.1-68-gac93c34
Cloudprovider:
Conditions:
Last Transition Time: 2021-10-09T20:34:08Z
Status: True
Type: CPIProvisioned
Name: vmware-guest-cluster
Status: applied
Version: 0.1-89-g3d24a22
Cni:
Conditions:
Last Transition Time: 2021-10-09T20:34:24Z
Status: True
Type: CNIProvisioned
Name: antrea
Status: applied
Version: v0.11.4_vmware.2
Csi:
Conditions:
Last Transition Time: 2021-10-09T20:34:10Z
Status: True
Type: CSIProvisioned
Name: pvcsi
Status: applied
Version: vsphere70u2-4c32126-73daa24
Dns:
Conditions:
Last Transition Time: 2021-10-09T20:34:05Z
Status: True
Type: CoreDNSProvisioned
Name: CoreDNS
Status: applied
Version: v1.7.0_vmware.14
Metrics - Server:
Conditions:
Last Transition Time: 2021-10-09T20:34:37Z
Status: True
Type: MetricsServerProvisioned
Name: metrics-server
Status: applied
Version: v0.4.0+vmware.1
Proxy:
Conditions:
Last Transition Time: 2021-10-09T20:34:05Z
Status: True
Type: KubeProxyProvisioned
Name: kube-proxy
Status: applied
Version: 1.20.9+vmware.1
Psp:
Conditions:
Last Transition Time: 2021-10-09T20:34:03Z
Status: True
Type: PSPProvisioned
Name: defaultpsp
Status: applied
Version: v1.20.9+vmware.1-tkg.1.a4cee5b
Cluster API Status:
API Endpoints:
Host: 10.0.12.131
Port: 6443
Phase: Provisioned
Conditions:
Last Transition Time: 2021-10-09T20:34:37Z
Status: True
Type: AddonsReady
Last Transition Time: 2021-10-09T20:34:43Z
Status: True
Type: ControlPlaneReady
Last Transition Time: 2021-10-09T20:37:24Z
Message: 1/1 Control Plane Node(s) healthy. 2/2 Worker Node(s) healthy
Status: True
Type: NodesHealthy
Last Transition Time: 2021-10-09T20:34:03Z
Status: True
Type: ProviderServiceAccountsReady
Last Transition Time: 2021-10-09T20:34:03Z
Status: True
Type: RoleBindingSynced
Last Transition Time: 2021-10-09T20:34:45Z
Status: True
Type: ServiceDiscoveryReady
Last Transition Time: 2021-10-09T20:34:42Z
Status: True
Type: StorageClassSynced
Last Transition Time: 2021-10-09T20:34:43Z
Status: True
Type: TanzuKubernetesReleaseCompatible
Last Transition Time: 2021-09-30T14:22:59Z
Reason: NoUpdates
Status: False
Type: UpdatesAvailable
Node Status:
test-cluster01-control-plane-p5fzt: ready
test-cluster01-workers-tcknk-7449644546-bvpjx: ready
test-cluster01-workers-tcknk-7449644546-kdxnv: ready
Phase: running
Vm Status:
test-cluster01-control-plane-p5fzt: ready
test-cluster01-workers-tcknk-7449644546-bvpjx: ready
test-cluster01-workers-tcknk-7449644546-kdxnv: ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal PhaseChanged 2m58s vmware-system-tkg/vmware-system-tkg-controller-manager/tanzukubernetescluster-status-controller cluster changes from creating phase to running phaseDescribe also reports the cluster being in the running phase. We can see the API endpoint IP of 10.0.12.131. Again, because we deployed with HAProxy and the frontend network, the exposed IPs will be on our frontend network.
We can also use get tkc and get vm to less details on these resources.
C:\>kubectl get tkc NAME CONTROL PLANE WORKER DISTRIBUTION AGE PHASE TKR COMPATIBLE UPDATES AVAILABLE test-cluster01 1 2 v1.20.9+vmware.1-tkg.1.a4cee5b 26m running True C:\>kubectl get vm NAME POWERSTATE AGE test-cluster01-control-plane-p5fzt poweredOn 26m test-cluster01-workers-tcknk-7449644546-bvpjx poweredOn 23m test-cluster01-workers-tcknk-7449644546-kdxnv poweredOn 23m
Now that we have our first Tanzu Kubernetes Cluster deployed, we will be deploying a workload on to the cluster. Hopefully I will be getting that posted before too long.
Implementing Tanzu in vSphere 7U1 Part 1: Pre-requisites and HAProxy
Implementing Tanzu in vSphere 7U1 Part 2: Enabling Workload Management