VerticaAutoscaler

Cannot find CPU metrics with VerticaAutoscaler

You might notice that your VerticaAutoScaler is not scaling correctly according to CPU utilization:

$ kubectl get hpa
NAME                REFERENCE                           TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
autoscaler-name     VerticaAutoscaler/autoscaler-name   <unknown>/50%   3         12        0          19h

$ kubectl describe hpa
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name: autoscaler-name
Namespace: namespace
Labels: <none>
Annotations: <none>
CreationTimestamp: Thu, 12 May 2022 10:25:02 -0400
Reference: VerticaAutoscaler/autoscaler-name
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 50%
Min replicas: 3
Max replicas: 12
VerticaAutoscaler pods: 3 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 7s horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedComputeMetricsReplicas 7s horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

You receive this error because the metrics server is not installed:

$ kubectl top nodes
error: Metrics API not available

To install the metrics server:

1. Download the components.yaml file:

   $ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
   

2. Optionally, disable TLS:

   $ if ! grep kubelet-insecure-tls components.yaml; then
     sed -i 's/- args:/- args:\n - --kubelet-insecure-tls/' components.yaml;
   

3. Apply the YAML file:

   $ kubectl apply -f components.yaml
   

4. Verify that the metrics server is running:

   $ kubectl get svc metrics-server -n namespace
   NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
   metrics-server   ClusterIP   10.105.239.175   <none>        443/TCP   19h
   

CPU request error with VerticaAutoscaler

You might receive an error that states:

failed to get cpu utilization: missing request for cpu

You get this error because you must set resource limits on all containers, including sidecar containers. To correct this error:

1. Verify the error:

   $ kubectl get hpa
   NAME                REFERENCE                           TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   autoscaler-name     VerticaAutoscaler/autoscaler-name   <unknown>/50%   3         12        0          19h
   
   $ kubectl describe hpa
   Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
   Name: autoscaler-name
   Namespace: namespace
   Labels: <none>
   Annotations: <none>
   CreationTimestamp: Thu, 12 May 2022 15:58:31 -0400
   Reference: VerticaAutoscaler/autoscaler-name
   Metrics: ( current / target )
   resource cpu on pods (as a percentage of request): <unknown> / 50%
   Min replicas: 3
   Max replicas: 12
   VerticaAutoscaler pods: 3 current / 0 desired
   Conditions:
   Type Status Reason Message
   ---- ------ ------ -------
   AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
   ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu
   Events:
   Type Reason Age From Message
   ---- ------ ---- ---- -------
   Warning FailedGetResourceMetric 4s (x5 over 64s) horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu
   Warning FailedComputeMetricsReplicas 4s (x5 over 64s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
   

2. Add resource limits to the CR:

   $ cat /tmp/vdb.yaml
   apiVersion: vertica.com/v1beta1
   kind: VerticaDB
   metadata:
     name: vertica-vdb
   spec:
     sidecars:
       - name: vlogger
         image: vertica/vertica-logger:latest
         resources:
           requests:
             memory: "100Mi"
             cpu: "100m"
           limits:
             memory: "100Mi"
             cpu: "100m"
     communal:
       credentialSecret: communal-creds
       endpoint: https://endpoint
           path: s3://bucket-location
     dbName: verticadb
     image: vertica/vertica-k8s:latest
     subclusters:
     - isPrimary: true
       name: sc1
       resources:
         requests:
           memory: "4Gi"
           cpu: 2
         limits:
           memory: "4Gi"
           cpu: 2
       serviceType: ClusterIP
       serviceName: sc1
       size: 3
     upgradePolicy: Auto
   

3. Apply the update:

   $ kubectl apply -f /tmp/vdb.yaml
   verticadb.vertica.com/vertica-vdb created
   

When you set a new CPU resource limit, Kubernetes reschedules each pod in the StatefulSet in a rolling update until all pods have the updated CPU resource limit.