Resource Limits and Usage

Overview

This page documents the preliminary requirements and design of the NDS Labs Workbench resource limit strategy. The NDS Labs Workbench is based on Kubernetes and GlusterFS, so we'll leverage their internal resource limit and enforcement mechanisms. Kubernetes supports limits and quotas on compute resources (CPUs and memory) as well as object counts (services, replication controllers, pods, secrets, etc).  GlusterFS provides mechanisms to enforce storage quotas through explicit quotas or allocated volume size.

Why specify and limit resources?

There are several reasons to enforce resource limits in the NDS Labs Workbench:

1. The Kubernetes system requires the specification of Pod resource requirements for effective scheduling. To enable the specification of resource requirements, the resource quotas must be enabled system-wide.  (Note: it would be possible to grant all projects full access to all system resources if desired, this would lead to competition).
2. We want to prevent individual NDS Labs Workbench projects from exhausting all of the system resources, introducing instability in other projects. This means enforcing quotas on projects.

Enforcement options

We've discussed the following options for enforcing quotas:

Since it isn't possible to know exact usage requirements before launch, we have decided to enforce quotas during runtime.  If launching the stack exceeds quotas, the stack will fail to launch and the user will be notified.  

Kubernetes: Limit ranges, Requests, and ResourceQuotas

Kubernetes defines three concepts related to the implementation and enforcement of resource limits: resource quotas, limit ranges s, and requests. Resource quotas are enabled at the namespace level and define hard limits onresources available to the namespace. Once enabled, all Pods must specify requests for each of the resources quotas that are enabled.  If no request is specified, then Pod creation will fail (403 Forbidden).  

Default values can be specified at the namespace level using LimitRanges.  If a LimitRange is specified on the namespace, then Pods will have resources requests set to the default limit values.

Individual Pods can specify limits (upper bounds of resource requirements) and requests (resource increments). By default, Pods run with unbounded CPU and memory limits. These quotas are independent of actual cluster capacity. 

Resource quotas and limits must be enabled during Kubernetes cluster startup through the --admission-control flag.  

Resource Limits

Note units: https://github.com/kubernetes/kubernetes/blob/master/docs/design/resources.md

• CPUs are in milli-core units (1 core = 1000m).  The default value of cpu=1 is "best effort"

Usage information

• CPU and Memory usage are available via the Grafana API, which are included in the Kubernetes contrib/ansible deployment
• Storage usage information is available via GlusterFS

Requirements

1. Cluster administrator can specify memory, CPU, and storage quotas at the project level
2. Cluster administrator can specify default limits for Pods system-wide
3. Cluster administrator can update quotas
4. Quotas/limits are deleted when the project is deleted
5. Project administrator can view project quotas
6. Project administrator can view memory, CPU, storage usage
7. Service developer can implement resource requests for Pods
8. Service developer can run a tool to capture/estimate resource requirements for a service
9. Cluster ops can monitor cluster-wide resource utilization
10. Test services for exercising limits
    
    1. Exceed memory limits, exceed CPU limits, exceed Object limits
    2. Both internally (single pod/container) and adding a resource that exceeds
    3. What happens if Pod killed due to OOM?
11. Will need to have better error handling for failure conditions – what happens when the stack fails to deploy because a service fails to start?
12. The API server will monitor all Pods to track changes to status and notify site and project administrators.

Implementation

Project Limits

During the POST or PUT of a project, ResourceQuota and LimitRange objects are created for the namespace. The Project object has been modified to include resourceLimits.

• cpuMax:  Maximum CPU for this project
• cpuDefault: Default CPU for Pods without resource requests for this project
• memMax: Maximum total memory for this project
• memDefault: Default memory for Pods without resource requests for this project
  
• storageQuota: Storage allocation
  
  

{
    "id": "demo",
    "name": "demo project",
    "description": "demo project description",
    "namespace": "demo",
    "password": "12345",
	"resourceLimits": {
		"cpuMax": "2",
		"cpuDefault": "1",
		"memMax": "8Gi",
		"memDefault": "100Mi",
		"storageQuota": "10Gb"
	}
}

The above project specification results in the following ResourceQuota and LimitRange objects created for the namespace:

{
    "kind": "ResourceQuota",
    "apiVersion": "v1",
    "metadata": {
        "name": "quota",
    },
    "spec": {
        "hard": {
            "cpu": "2",
            "memory": "8Gi"
        }
    },
}

{
    "kind": "LimitRange",
    "apiVersion": "v1",
    "metadata": {
        "name": "limits",
    },
    "spec": {
        "limits": [
            {
                "type": "Container",
                "default": {
                    "cpu": "1",
                    "memory": "100Mi"
                }
            }
        ]
    }
}

{
    "label": "Clowder"
	"resources": {
    	"limits": [
			{ 
		  		"cpu": "250m",
  		  		"memory": "1G"
    		}
		],
		"requests": [
			{
				"cpu": "100m",
				"memory": "128Mi"
			}
		]
	}
}