Operational Considerations

This page gives recommendations and hints for people managing long lived clusters

Before you begin
Managing Juju
Managing your Kubernetes cluster

Before you begin

This page assumes you understand the basics of Juju and Kubernetes.

Managing Juju

Sizing your controller node

The Juju Controller:

requires about 2 to 2.5GB RAM to operate.
uses a MongoDB database as a storage backend for the configuration and state of the cluster. This database can grow significantly, and can also be the biggest consumer of CPU cycles on the instance
aggregates and stores the log data of all services and units. Therefore, significant storage is needed for long lived models. If your intention is to keep the cluster running, make sure to provision at least 64GB for the logs.

To bootstrap a controller with constraints run the following command:

juju bootstrap --constraints "mem=8GB cpu-cores=4 root-disk=128G"

Juju will select the cheapest instance type matching your constraints on your target cloud. You can also use the instance-type constraint in conjunction with root-disk for strict control. For more information about the constraints available, refer to the official documentation

Additional information about logging can be found in the logging section

SSHing into the Controller Node

By default, Juju will create a pair of SSH keys that it will use to automate the connection to units. They are stored on the client node in ~/.local/share/juju/ssh/

After deployment, Juju Controller is a “silent unit” that acts as a proxy between the client and the deployed applications. Nevertheless it can be useful to SSH into it.

First you need to understand your environment, especially if you run several Juju models and controllers. Run

juju list-models --all
$ juju models --all
Controller: k8s

Model             Cloud/Region   Status     Machines  Cores  Access  Last connection
admin/controller  lxd/localhost  available         1      -  admin   just now
admin/default     lxd/localhost  available         0      -  admin   2017-01-23
admin/whale*      lxd/localhost  available         6      -  admin   3 minutes ago

The first line Controller: k8s refers to how you bootstrapped.

Then you will see 2, 3 or more models listed below.

admin/controller is the default model that hosts all controller units of juju
admin/default is created by default as the primary model to host the user application, such as the Kubernetes cluster
admin/whale is an additional model created if you use conjure-up as an overlay on top of Juju.

Now to ssh into a controller node, you first ask Juju to switch context, then ssh as you would with a normal unit:

juju switch controller

At this stage, you can query the controller model as well:

juju status
Model       Controller  Cloud/Region   Version
controller  k8s           lxd/localhost  2.0.2

App  Version  Status  Scale  Charm  Store  Rev  OS  Notes

Unit  Workload  Agent  Machine  Public address  Ports  Message

Machine  State    DNS           Inst id        Series  AZ
0        started  10.191.22.15  juju-2a5ed8-0  xenial

Note that if you had bootstrapped in HA mode, you would see several machines listed.

Now ssh-ing into the controller follows the same semantic as classic Juju commands:

$ juju ssh 0
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.8.0-34-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.


Last login: Tue Jan 24 16:38:13 2017 from 10.191.22.1
ubuntu@juju-2a5ed8-0:~$

When you are done and want to come back to your initial model, exit the controller and

Then if you need to switch back to your cluster and ssh into the units, run

juju switch default

Managing your Kubernetes cluster

Running privileged containers

By default, juju-deployed clusters only allow running privileged containers on nodes with GPUs. If you need privileged containers on other nodes, you have to enable the allow-privileged config on both kubernetes-master and kubernetes-worker:

juju config kubernetes-master allow-privileged=true
juju config kubernetes-worker allow-privileged=true

Private registry

With the registry action, you can easily create a private docker registry that uses TLS authentication. However, note that a registry deployed with that action is not HA; it uses storage tied to the kubernetes node where the pod is running. Consequently, if the registry pod is migrated from one node to another, you will need to re-publish the images.

Example usage

Create the relevant authentication files. Let’s say you want user userA to authenticate with the password passwordA. Then you’ll do:

echo "userA:passwordA" > htpasswd-plain
htpasswd -c -b -B htpasswd userA passwordA

(the htpasswd program comes with the apache2-utils package)

Assuming that your registry will be reachable at myregistry.company.com, you already have your TLS key in the registry.key file, and your TLS certificate (with myregistry.company.com as Common Name) in the registry.crt file, you would then run:

juju run-action kubernetes-worker/0 registry domain=myregistry.company.com htpasswd="$(base64 -w0 htpasswd)" htpasswd-plain="$(base64 -w0 htpasswd-plain)" tlscert="$(base64 -w0 registry.crt)" tlskey="$(base64 -w0 registry.key)" ingress=true

If you then decide that you want to delete the registry, just run:

juju run-action kubernetes-worker/0 registry delete=true ingress=true