Difference between revisions of "OpenShift v4x health check"

From Bitbull Wiki
Jump to navigation Jump to search
Line 20: Line 20:
 
etcd is a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data
 
etcd is a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data
  
−
==v3.9==
+
==Version 3.9==
 
  [root@master(zabbix/admin) ~]# '''source /etc/etcd/etcd.conf'''
 
  [root@master(zabbix/admin) ~]# '''source /etc/etcd/etcd.conf'''
 
  [root@master(zabbix/admin) ~]# '''etcdctl --cert-file=$ETCD_PEER_CERT_FILE --key-file=$ETCD_PEER_KEY_FILE  --ca-file=/etc/etcd/ca.crt --endpoints=$ETCD_LISTEN_CLIENT_URLS cluster-health'''
 
  [root@master(zabbix/admin) ~]# '''etcdctl --cert-file=$ETCD_PEER_CERT_FILE --key-file=$ETCD_PEER_KEY_FILE  --ca-file=/etc/etcd/ca.crt --endpoints=$ETCD_LISTEN_CLIENT_URLS cluster-health'''
Line 26: Line 26:
 
  '''cluster is healthy'''
 
  '''cluster is healthy'''
  
−
==v4.4==
+
==Version 4.4==
 
  [chris@control(zabbix-dev/system:admin) ~]$ '''oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}''''
 
  [chris@control(zabbix-dev/system:admin) ~]$ '''oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}''''
 
  '''master02,master01,master03 members are available''',  have not started,  are unhealthy,  are unknown
 
  '''master02,master01,master03 members are available''',  have not started,  are unhealthy,  are unknown
Line 33: Line 33:
 
=router=
 
=router=
 
There are many ways to get traffic into the cluster. The most common approach is to use the OpenShift Container Platform router as the ingress point for external traffic destined for services in your OpenShift Container Platform installation.
 
There are many ways to get traffic into the cluster. The most common approach is to use the OpenShift Container Platform router as the ingress point for external traffic destined for services in your OpenShift Container Platform installation.
−
==v3.9==
+
==Version 3.9==
 
  [root@master(zabbix/admin) ~]# '''oc -n default get deploymentconfigs/router'''
 
  [root@master(zabbix/admin) ~]# '''oc -n default get deploymentconfigs/router'''
 
  NAME      REVISION  '''DESIRED  CURRENT'''  TRIGGERED BY
 
  NAME      REVISION  '''DESIRED  CURRENT'''  TRIGGERED BY
Line 41: Line 41:
 
OpenShift Container Platform can build container images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift Container Platform provides an internal, integrated container image registry that can be deployed in your OpenShift Container Platform environment to locally manage images.
 
OpenShift Container Platform can build container images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift Container Platform provides an internal, integrated container image registry that can be deployed in your OpenShift Container Platform environment to locally manage images.
  
−
==v3.9==
+
==Version 3.9==
 
  [root@master(zabbix/admin) ~]# '''oc -n default get deploymentconfigs/docker-registry'''
 
  [root@master(zabbix/admin) ~]# '''oc -n default get deploymentconfigs/docker-registry'''
 
  NAME              REVISION  '''DESIRED  CURRENT'''  TRIGGERED BY
 
  NAME              REVISION  '''DESIRED  CURRENT'''  TRIGGERED BY
 
  docker-registry  1          '''1        1'''        config
 
  docker-registry  1          '''1        1'''        config
  
−
==v4.4==
+
==Version 4.4==
 
  [chris@control(zabbix-dev/system:admin) ~]$ '''oc get all -n openshift-image-registry'''
 
  [chris@control(zabbix-dev/system:admin) ~]$ '''oc get all -n openshift-image-registry'''
 
  NAME                                                  '''READY'''  STATUS    '''RESTARTS'''  AGE
 
  NAME                                                  '''READY'''  STATUS    '''RESTARTS'''  AGE
Line 57: Line 57:
 
  pod/node-ca-ps7v9                                      '''1/1'''    Running  '''0'''          2d23h
 
  pod/node-ca-ps7v9                                      '''1/1'''    Running  '''0'''          2d23h
 
   
 
   
−
  NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S)    AGE
+
  ...
−
service/image-registry            ClusterIP  172.30.229.236  <none>        5000/TCP    40d
 
−
service/image-registry-operator  ClusterIP  None            <none>        60000/TCP  40d
 
 
   
 
   
 
  NAME                    '''DESIRED'''  CURRENT  '''READY'''  UP-TO-DATE  AVAILABLE  NODE SELECTOR            AGE
 
  NAME                    '''DESIRED'''  CURRENT  '''READY'''  UP-TO-DATE  AVAILABLE  NODE SELECTOR            AGE
 
  daemonset.apps/node-ca  5        5        5      5            5          kubernetes.io/os=linux  40d
 
  daemonset.apps/node-ca  5        5        5      5            5          kubernetes.io/os=linux  40d
 
   
 
   
−
  NAME                                              READY  UP-TO-DATE  AVAILABLE  AGE
+
  ...
−
deployment.apps/cluster-image-registry-operator  1/1    1            1          40d
 
−
deployment.apps/image-registry                    1/1    1            1          40d
 
 
   
 
   
 
  NAME                                                        '''DESIRED'''  CURRENT  '''READY'''  AGE
 
  NAME                                                        '''DESIRED'''  CURRENT  '''READY'''  AGE
Line 72: Line 68:
 
  replicaset.apps/cluster-image-registry-operator-7bff4c7595  '''1'''        1        '''1'''      2d23h
 
  replicaset.apps/cluster-image-registry-operator-7bff4c7595  '''1'''        1        '''1'''      2d23h
 
  replicaset.apps/cluster-image-registry-operator-86476f46bc  0        0        0      6d7h
 
  replicaset.apps/cluster-image-registry-operator-86476f46bc  0        0        0      6d7h
−
  replicaset.apps/cluster-image-registry-operator-f9697f69d    0        0        0      40d
+
  ...
−
replicaset.apps/cluster-image-registry-operator-fc9dfb566    0        0        0      3d3h
 
−
replicaset.apps/image-registry-58cc7948d8                    0        0        0      3d3h
 
 
  replicaset.apps/image-registry-688fb696dc                    0        0        0      40d
 
  replicaset.apps/image-registry-688fb696dc                    0        0        0      40d
 
  replicaset.apps/image-registry-6948d8479b                    0        0        0      4d5h
 
  replicaset.apps/image-registry-6948d8479b                    0        0        0      4d5h
 
  replicaset.apps/image-registry-6b6745b4f9                    '''1'''        1        '''1'''      2d23h
 
  replicaset.apps/image-registry-6b6745b4f9                    '''1'''        1        '''1'''      2d23h
−
replicaset.apps/image-registry-7bbdbc5dc7                    0        0        0      6d7h
+
 
−
replicaset.apps/image-registry-9dc4885b                      0        0        0      6d7h
+
 
−
replicaset.apps/image-registry-d4cf5448b                    0        0        0      40d
 
−
replicaset.apps/image-registry-f488f9578                    0        0        0      6d7h
 
−
replicaset.apps/image-registry-f5647c6d8                    0        0        0      40d
 
−
 
−
NAME                        SCHEDULE    SUSPEND  ACTIVE  LAST SCHEDULE  AGE
 
−
cronjob.batch/image-pruner  0 0 * * *  True      0        <none>          2d23h
 
  
 
=DaemonSet=
 
=DaemonSet=
Line 100: Line 88:
 
  openshift-controller-manager            controller-manager            '''3'''        3        '''3'''      3            3          node-role.kubernetes.io/master=  40d
 
  openshift-controller-manager            controller-manager            '''3'''        3        '''3'''      3            3          node-role.kubernetes.io/master=  40d
 
  openshift-dns                            dns-default                  '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
  openshift-dns                            dns-default                  '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
−
  openshift-image-registry                node-ca                      '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
+
  ...
−
openshift-machine-config-operator        machine-config-daemon        '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
−
openshift-machine-config-operator        machine-config-server        '''3'''        3        '''3'''      3            3          node-role.kubernetes.io/master=  40d
 
−
openshift-monitoring                    node-exporter                '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
−
openshift-multus                        multus                        '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
−
openshift-multus                        multus-admission-controller  '''3'''        3        '''3'''      3            3          node-role.kubernetes.io/master=  40d
 
 
  openshift-sdn                            ovs                          '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
  openshift-sdn                            ovs                          '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
  openshift-sdn                            sdn                          '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
 
  openshift-sdn                            sdn                          '''5'''        5        '''5'''      5            5          kubernetes.io/os=linux            40d
Line 113: Line 96:
  
  
−
=ClusterOperators v4x=
+
=ClusterOperators - Version 4x=
 
Conceptually, Operators take human operational knowledge and encode it into software that is more easily shared with consumers.
 
Conceptually, Operators take human operational knowledge and encode it into software that is more easily shared with consumers.
 
<br>
 
<br>
Line 123: Line 106:
 
  cloud-credential                          4.4.4    True        False        '''False'''      40d
 
  cloud-credential                          4.4.4    True        False        '''False'''      40d
 
  cluster-autoscaler                        4.4.4    True        False        '''False'''      40d
 
  cluster-autoscaler                        4.4.4    True        False        '''False'''      40d
−
console                                    4.4.4    True        False        '''False'''      33h
+
   ...
−
csi-snapshot-controller                    4.4.4    True        False        '''False'''      33h
 
−
dns                                        4.4.4    True        False        '''False'''      33h
 
−
etcd                                      4.4.4    True        False        '''False'''      2d20h
 
−
image-registry                            4.4.4    True        False        '''False'''      33h
 
−
ingress                                    4.4.4    True        False        '''False'''      33h
 
−
insights                                  4.4.4    True        False        '''False'''      40d
 
−
kube-apiserver                            4.4.4    True        False        '''False'''      40d
 
−
kube-controller-manager                    4.4.4    True        False        '''False'''      2d23h
 
−
kube-scheduler                            4.4.4    True        False        '''False'''      2d23h
 
−
kube-storage-version-migrator              4.4.4    True        False        '''False'''      33h
 
−
machine-api                                4.4.4    True        False        '''False'''      40d
 
−
machine-config                            4.4.4    True        False        '''False'''      2d19h
 
−
marketplace                                4.4.4    True        False        '''False'''      2d19h
 
−
monitoring                                4.4.4    True        False        '''False'''      2d10h
 
−
network                                    4.4.4    True        False        '''False'''      40d
 
−
node-tuning                                4.4.4    True        False        '''False'''      33h
 
−
openshift-apiserver                        4.4.4    True        False        '''False'''      33h
 
−
openshift-controller-manager              4.4.4    True        False        '''False'''      33h
 
−
openshift-samples                          4.4.4    True        False        '''False'''      7m37s
 
−
operator-lifecycle-manager                4.4.4    True        False        '''False'''      40d
 
−
operator-lifecycle-manager-catalog        4.4.4    True        False        '''False'''      40d
 
−
operator-lifecycle-manager-packageserver   4.4.4    True        False        '''False'''      33h
 
−
service-ca                                4.4.4    True        False        '''False'''      40d
 
 
  service-catalog-apiserver                  4.4.4    True        False        '''False'''      40d
 
  service-catalog-apiserver                  4.4.4    True        False        '''False'''      40d
 
  service-catalog-controller-manager        4.4.4    True        False        '''False'''      40d
 
  service-catalog-controller-manager        4.4.4    True        False        '''False'''      40d
 
  storage                                    4.4.4    True        False        '''False'''      2d23h
 
  storage                                    4.4.4    True        False        '''False'''      2d23h
  
−
=Pods=
+
=Pods (restarts)=
 +
A set of one or more containers that are deployed onto a Node together and share a unique IP and Volumes (persistent storage). Pods also define the security and runtime policy for each container.
 +
 
 +
[chris@control(zabbix-dev/system:admin) ~]$ '''oc get pods --all-namespaces'''
 +
NAMESPACE                                              NAME                                                              '''READY'''  STATUS      '''RESTARTS'''  AGE
 +
openshift-apiserver-operator                            openshift-apiserver-operator-8596449546-kmmt6                    '''1/1'''    Running    '''0'''          2d20h
 +
openshift-apiserver                                    apiserver-95c79c585-b4h7f                                        '''1/1'''    Running    '''0'''          2d20h
 +
openshift-apiserver                                    apiserver-95c79c585-h5pxq                                        '''1/1'''    Running    '''0'''          2d20h
 +
openshift-apiserver                                    apiserver-95c79c585-w2xq2                                        '''1/1'''    Running    '''0'''          2d20h
 +
openshift-authentication-operator                      authentication-operator-66f85cff9-zcjhb                          '''1/1'''    Running    '''0'''          2d20h
 +
openshift-authentication                                oauth-openshift-5d8d554669-9wxng                                  '''1/1'''    Running    '''0'''          34h
 +
openshift-authentication                                oauth-openshift-5d8d554669-vgp8f                                  '''1/1'''    Running    '''0'''          34h
 +
openshift-cloud-credential-operator                    cloud-credential-operator-695f4895db-5nv2b                        '''1/1'''    Running    '''0'''          2d20h
 +
openshift-cluster-machine-approver                      machine-approver-685c8468fb-rpmtq                                '''2/2'''    Running    '''0'''          2d20h
 +
...
 +
 
 +
 
 +
 
  
 
=DeploymentConfig=
 
=DeploymentConfig=

Revision as of 13:12, 26 May 2020

1 Links


2 Nodes

Kubernetes runs your workload by placing containers into Pods to run on Nodes. A node may be a virtual or physical machine, depending on the cluster. Each node contains the services necessary to run Pods 

[chris@control(zabbix-dev/system:admin) ~]$ oc get nodes -o wide
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                   KERNEL-VERSION                CONTAINER-RUNTIME
master01   Ready    master,worker   40d   v1.17.1   192.168.100.221   <none>        RHEL CoreOS 44.81.202005062110-0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.17.4-8.dev.rhaos4.4.git5f5c5e4.el8
master02   Ready    master,worker   40d   v1.17.1   192.168.100.222   <none>        RHEL CoreOS 44.81.202005062110-0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.17.4-8.dev.rhaos4.4.git5f5c5e4.el8
master03   Ready    master,worker   40d   v1.17.1   192.168.100.223   <none>        RHEL CoreOS 44.81.202005062110-0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.17.4-8.dev.rhaos4.4.git5f5c5e4.el8
worker01   Ready    worker          40d   v1.17.1   192.168.100.231   <none>        RHEL CoreOS 44.81.202005062110-0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.17.4-8.dev.rhaos4.4.git5f5c5e4.el8
worker02   Ready    worker          40d   v1.17.1   192.168.100.232   <none>        RHEL CoreOS 44.81.202005062110-0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.17.4-8.dev.rhaos4.4.git5f5c5e4.el8


3 etcd

etcd is a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data

3.1 Version 3.9

[root@master(zabbix/admin) ~]# source /etc/etcd/etcd.conf
[root@master(zabbix/admin) ~]# etcdctl --cert-file=$ETCD_PEER_CERT_FILE --key-file=$ETCD_PEER_KEY_FILE   --ca-file=/etc/etcd/ca.crt --endpoints=$ETCD_LISTEN_CLIENT_URLS cluster-health
member da1c9720d5fee664 is healthy: got healthy result from https://192.168.223.74:2379
cluster is healthy

3.2 Version 4.4

[chris@control(zabbix-dev/system:admin) ~]$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'
master02,master01,master03 members are available,  have not started,  are unhealthy,  are unknown


4 router

There are many ways to get traffic into the cluster. The most common approach is to use the OpenShift Container Platform router as the ingress point for external traffic destined for services in your OpenShift Container Platform installation.

4.1 Version 3.9

[root@master(zabbix/admin) ~]# oc -n default get deploymentconfigs/router
NAME      REVISION   DESIRED   CURRENT   TRIGGERED BY
router    1          1         1         config

5 registry

OpenShift Container Platform can build container images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift Container Platform provides an internal, integrated container image registry that can be deployed in your OpenShift Container Platform environment to locally manage images.

5.1 Version 3.9

[root@master(zabbix/admin) ~]# oc -n default get deploymentconfigs/docker-registry
NAME              REVISION   DESIRED   CURRENT   TRIGGERED BY
docker-registry   1          1         1         config

5.2 Version 4.4

[chris@control(zabbix-dev/system:admin) ~]$ oc get all -n openshift-image-registry
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/cluster-image-registry-operator-7bff4c7595-hkbqx   2/2     Running   0          2d20h
pod/image-registry-6b6745b4f9-wqwdx                    1/1     Running   0          2d22h
pod/node-ca-6wgpw                                      1/1     Running   0          2d23h
pod/node-ca-gjmhw                                      1/1     Running   0          2d23h
pod/node-ca-gnp7n                                      1/1     Running   0          2d23h
pod/node-ca-gtvt9                                      1/1     Running   0          2d23h
pod/node-ca-ps7v9                                      1/1     Running   0          2d23h

...

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/node-ca   5         5         5       5            5           kubernetes.io/os=linux   40d

...

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/cluster-image-registry-operator-6f78cddbbc   0         0         0       4d5h
replicaset.apps/cluster-image-registry-operator-7bff4c7595   1         1         1       2d23h
replicaset.apps/cluster-image-registry-operator-86476f46bc   0         0         0       6d7h
...
replicaset.apps/image-registry-688fb696dc                    0         0         0       40d
replicaset.apps/image-registry-6948d8479b                    0         0         0       4d5h
replicaset.apps/image-registry-6b6745b4f9                    1         1         1       2d23h


6 DaemonSet

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are:

  • running a cluster storage daemon, such as glusterd, ceph, on each node.
  • running a logs collection daemon on every node, such as fluentd or filebeat.
  • running a node monitoring daemon on every node, such as Prometheus Node Exporter, Flowmill, Sysdig Agent, collectd, Dynatrace OneAgent, AppDynamics Agent, Datadog agent, New Relic agent, Ganglia gmond, Instana Agent or Elastic Metricbeat.
[chris@control(zabbix-dev/system:admin) ~]$ oc get daemonset --all-namespaces
NAMESPACE                                NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
openshift-cluster-node-tuning-operator   tuned                         5         5         5       5            5           kubernetes.io/os=linux            2d23h
openshift-controller-manager             controller-manager            3         3         3       3            3           node-role.kubernetes.io/master=   40d
openshift-dns                            dns-default                   5         5         5       5            5           kubernetes.io/os=linux            40d
...
openshift-sdn                            ovs                           5         5         5       5            5           kubernetes.io/os=linux            40d
openshift-sdn                            sdn                           5         5         5       5            5           kubernetes.io/os=linux            40d
openshift-sdn                            sdn-controller                3         3         3       3            3           node-role.kubernetes.io/master=   40d



7 ClusterOperators - Version 4x

Conceptually, Operators take human operational knowledge and encode it into software that is more easily shared with consumers.
Operators are pieces of software that ease the operational complexity of running another piece of software. They act like an extension of the software vendor’s engineering team, watching over a Kubernetes environment (such as OpenShift Container Platform) and using its current state to make decisions in real time. Advanced Operators are designed to handle upgrades seamlessly, react to failures automatically, and not take shortcuts, like skipping a software backup process to save time. 

[chris@control(zabbix-dev/system:admin) ~]$ oc -n default get clusteroperators
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.4     True        False         False      35d
cloud-credential                           4.4.4     True        False         False      40d
cluster-autoscaler                         4.4.4     True        False         False      40d
 ...
service-catalog-apiserver                  4.4.4     True        False         False      40d
service-catalog-controller-manager         4.4.4     True        False         False      40d
storage                                    4.4.4     True        False         False      2d23h

8 Pods (restarts)

A set of one or more containers that are deployed onto a Node together and share a unique IP and Volumes (persistent storage). Pods also define the security and runtime policy for each container. 

[chris@control(zabbix-dev/system:admin) ~]$ oc get pods --all-namespaces
NAMESPACE                                               NAME                                                              READY   STATUS      RESTARTS   AGE
openshift-apiserver-operator                            openshift-apiserver-operator-8596449546-kmmt6                     1/1     Running     0          2d20h
openshift-apiserver                                     apiserver-95c79c585-b4h7f                                         1/1     Running     0          2d20h
openshift-apiserver                                     apiserver-95c79c585-h5pxq                                         1/1     Running     0          2d20h
openshift-apiserver                                     apiserver-95c79c585-w2xq2                                         1/1     Running     0          2d20h
openshift-authentication-operator                       authentication-operator-66f85cff9-zcjhb                           1/1     Running     0          2d20h
openshift-authentication                                oauth-openshift-5d8d554669-9wxng                                  1/1     Running     0          34h
openshift-authentication                                oauth-openshift-5d8d554669-vgp8f                                  1/1     Running     0          34h
openshift-cloud-credential-operator                     cloud-credential-operator-695f4895db-5nv2b                        1/1     Running     0          2d20h
openshift-cluster-machine-approver                      machine-approver-685c8468fb-rpmtq                                 2/2     Running     0          2d20h
...



9 DeploymentConfig

10 ReplicationControlers

  • Result of an Deployment by DeploymentConfig
Retrieved from "http://www.bitbull.ch/wiki/index.php?title=OpenShift_v4x_health_check&oldid=4488"