Kubernetes (k8s) is a container orchestration platform that’s transforming the way software services are deployed and managed. For less than the cost of production-grade infrastructure, one can purchase multiple Raspberry Pi boards, network them together, and create a fully functioning k8s cluster for personal development and experimentation. This post details how to put together 3x Raspberry Pi 3 B+ hardware boards, link them on a private network, and install a fully functioning k8s cluster for personal use and experimentation.
Parts List
The following parts were used in this tutorial:
- 3x Raspberry Pi 3 B+ boards
- 3x SanDisk Extreme 32GB MicroSD Cards
- 1x Netgear 8-port Gigabit Switch
- 1x GeauxRobot Raspberry Pi 3 Model B 5-Layer Clear Case
- 3x Cat6 Ethernet Cable
- 1x Anker PowerPort 6
- MicroSD reader (or SD card reader with SD/MicroSD card adapter)
Note that the case is to make connections easier and cleaner but is not strictly required. Additionally, you might consider HDMI cords and keyboard peripheral, but these instructions will not require UI-based changes (they will assume you are connecting to the Raspberry Pi instances remotely via your laptop/personal device).
Finally, you can certainly expand your cluster to include additional Raspberry Pi nodes - this tutorial utilizes 3 total (1x master and 2x worker nodes), but you can expand the tutorial to use as many as you feel fit.
Flashing the MicroSD Cards
You’ll need to flash each of the MicroSD cards with an operating system for the Raspberry Pi instances. The OS used for this
tutorial is the Hypriot OS. The easiest way to flash the cards is to utilize the flash
utility that
is available from the same provider.
First, we’ll prepare a boot script that will configure the OS according to how you expect the system to operate/be configured. You’ll
likely want to include the default wireless SSID and passphrase to enable your Raspberry Pi instances to get online and be reachable as
soon as they boot, so you’ll insert your wireless network SSID and wireless passphrase in the ssid
and psk
parameters in the file
configuration.yaml
below as indicated:
#cloud-config
# Set your hostname here, the manage_etc_hosts will update the hosts file entries as well
hostname: black-pearl
manage_etc_hosts: true
# You could modify this for your own user information
users:
- name: pirate
gecos: "Hypriot Pirate"
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
groups: users,docker,video
plain_text_passwd: hypriot
lock_passwd: false
ssh_pwauth: true
chpasswd: { expire: false }
package_upgrade: false
# # WiFi connect to HotSpot
# # - use `wpa_passphrase SSID PASSWORD` to encrypt the psk
write_files:
- content: |
allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp
path: /etc/network/interfaces.d/wlan0
- content: |
country=us
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="<WIRELESS_SSID>"
psk="<WIRELESS_PASSPHRASE>"
proto=RSN
key_mgmt=WPA-PSK
pairwise=CCMP
auth_alg=OPEN
}
path: /etc/wpa_supplicant/wpa_supplicant.conf
# These commands will be ran once on first boot only
runcmd:
# Pickup the hostname changes
- 'systemctl restart avahi-daemon'
# Activate WiFi interface
- 'ifup wlan0'
Note that it is best not to use clear-text passphrases in the above, and if you’d like to instead create an encrypted psk,
you can use the wpa_passphrase
command-line to do so. If you do, ensure that when you insert your encrypted passphrase, you
do not use double-quotes in the value (double-quotes are only required for when a clear-text passphrase is specified).
Next we’ll download the Hypriot OS image file and flash utility:
- Download the Hypriot image - the version used in this tutorial is 1.11.4.
- Download the flash tool:
curl -LO https://github.com/hypriot/flash/releases/download/2.3.0/flash chmod +x flash
Next, you’ll want to flash each card in sequence. Insert each card into the USB reader for your device and run the following
command specifying the hostname as node1
, node2
, and node3
where applicable for the SD card inserted. This parameter overrides
the hostname
property in the configuration file, making it easier so you don’t need to open/edit the file for each run:
./flash --hostname node1 --userdata wifi.yaml hypriotos-rpi-v1.11.4.img.zip
Once each of the cards is successfully flashed, you can continue assembling your cluster!
Assembling
It’s not the prettiest/cleanest layout, but this is an example of the assembly in case it helps:
You’ll now want to mount the Raspberry Pi boards in the case. Next, the assembly should be pretty straightforward, but in case it helps, follow these steps:
- Connect a Cat6 cable between each of the Raspberry Pi board ethernet adapters and the Netgear switch - the port used is not important.
- Connect a micro USB cable between each of the Raspberry Pi board micro USB power plugs and the Anker 6-way USB power ports.
- Plug the MicroSD card for each of the nodes into a Raspberry Pi instance.
- Connect a Cat6 cable between your wireless router and the Netgear switch - the port used is not important.
- Plug the power cable for the Anker 6-way USB power adapter and Netgear Switch into a wall power outlet.
At this point, your Raspberry Pi instances should begin to boot and configure. Note that it can take a few minutes for them to stabilize as the first thing that occurs is an expansion of the SD card (to enable all space to be usable) and initial configuration according to the YAML file you created.
Communicating with Raspberry Pi Instances
After a few minutes, you should be able to see your Raspberry Pi nodes on your local wifi network. From a local device connected
to the same wireless network as the Raspberry Pi instances, run the command sudo arp -a
- this should display a list of IP and
Mac addresses detected on your network. Within this list should be tags of node1
, node2
, and node3
. If you don’t see the names
of the nodes, you might need to go about this a different way (such as accessing your wireless router to view connected devices which
should contain the node names).
Once you identify the nodes, take note of the IP addresses for each. This should be an IP address assigned by your wireless router.
You might consider inserting these into your local /etc/hosts
file to make it easier to reach the Raspberry Pi instances.
Using your favorite split terminal (tmux, etc.), SSH into each of the Raspberry Pi nodes using the default username/password of
pirate
/hypriot
. Immediately upon login, you should likely consider changing the default password for the pirate
user. In addition
it’s most efficient to insert your public SSH key into a file under the pirate
user’s home directory ~/.ssh/authorized_keys
so you
can SSH to each host using the pirate
user without a passphrase (only using your SSH keys).
Note: Under certain circumstances, the default route to the internet on the Pi instances via the wireless ethernet device may be lost, causing
you to lose internet connectivity. This most often occurrs when performing a major upgrade of the operating system, etc. Under these circumstances,
you can run the following command, which will restore internet connectivity via the wireless adapter (replacing the via
IP gateway with the
actual gateway of your wireless network): sudo ip route add default via 192.168.86.1 dev wlan0
Master Network Configuration
Setting up the k8s master node is the first step. Create a file /etc/network/interfaces.d/eth0
with the following contents (we’ll be
following the subnet recommendation from the “Terraform: Up and Running” book as referenced in the credits section at the end of this
blog):
allow-hotplug eth0
iface eth0 inet static
address 10.0.0.2
netmask 255.255.255.0
broadcast 10.0.0.255
Once the above file is created, reboot the k8s master node so it can claim the IP address. Once booted, SSH back into the device and run
the ifconfig
command to ensure the eth0
device has the IP address 10.0.0.2
.
DHCP Server/Worker Node Network Configuration
Next we’ll need to install a DHCP server to ensure the remaining node2
and node3
nodes (and any additional you may wish to add) can
automatically obtain IP addresses in the same subnet:
sudo apt-get -y install isc-dhcp-server
Edit the file /etc/dhcp/dhcpd.conf
to include the following directive. Replace the hardware
values for each of the nodes with the
respective ether
(MAC address) that shows up for the eth0
network device on each of the nodes when running the ifconfig
command:
option domain-name "cluster.home";
option domain-name-servers 8.8.8.8, 8.8.4.4;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
subnet 10.0.0.0 netmask 255.255.255.0 {
option broadcast-address 10.0.0.255;
group {
host node1 {
hardware ethernet b8:27:eb:d9:e4:38;
fixed-address 10.0.0.2;
}
host node2 {
hardware ethernet b8:27:eb:7f:7b:bb;
fixed-address 10.0.0.3;
}
host node3 {
hardware ethernet b8:27:eb:29:4e:03;
fixed-address 10.0.0.4;
}
}
}
You’ll also need to update the file /etc/default/isc-dhcp-server
to ensure the following property is present and specified:
INTERFACESv4="eth0"
# you can leave the following property blank as we'll not be using IPV6
INTERFACESv6=""
Then, restart the DHCP server for the configuration to take effect: sudo systemctl restart isc-dhcp-server
.
Once the above is complete (and the DHCP server restarts gracefully), issue a reboot command on each of your node2
and node3
instances.
When they have finished rebooting, SSH back into the devices and ensure each has their eth0
network adapter specified with a 10.0.0.3
and
10.0.0.4
IP address, respectively, while the wlan0
(wireless) adapter remains specified as an IP on your wireless network, ensuring they
can continue to access the public internet.
Kubernetes Software Installation
On each of the nodes, add the encryption key for the k8s packages, add the k8s repository, and install the packages required to run k8s:
sudo su -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" \
>> /etc/apt/sources.list.d/kubernetes.list
apt-get -y update
apt-get -y upgrade
apt-get -y install kubelet kubeadm kubectl kubernetes-cni
The required software is now installed on each of the nodes.
Configuring the k8s Master
Let’s now configure the master instance - run the following commands on the node1
k8s master instance::
sudo kubeadm init --pod-network-cidr 10.244.0.0/16 \
--apiserver-advertise-address 10.0.0.2 \
--apiserver-cert-extra-sans kubernetes.cluster.home
The above command will take several minutes to download container images, configure network components, etc. Once complete, there are 2 important items the logs will print that need to be taken care of in the following sections.
Setting Up kubectl
Command
In order to interact with the k8s cluster from any node, you’ll need to copy the admin.conf
over to the host you’re using to access the cluster.
Assuming you’re going to interact with the cluster from the k8s master instance (node1
), run the following commands to set up configuration for
your user to successfully interact with the cluster (obviously this can be performed wherever you wish to interact with the cluster from):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Joining Worker Nodes
Now that the k8s master is configured you need to configure the worker nodes to connect to the master and be usable in the cluster. Pay attention to the command output when running the master install and capture the command to join a worker node to the master instance/cluster. It should look something like the following:
sudo kubeadm join 10.0.0.2:6443 --token <TOKEN> \
--discovery-token-ca-cert-hash <TOKEN_HASH>
Run the above on each of the node2
and node3
worker nodes.
Showing Node Status
Run the command kubectl get nodes
to list the nodes in the cluster. You should see something similar to the following, with the NotReady
showing
for all nodes due to the fact that we have not yet installed cluster (Pod-to-Pod) networking:
NAME STATUS ROLES AGE VERSION
node1 NotReady master 5m v1.17.0
node2 NotReady <none> 4m v1.17.0
node3 NotReady <none> 3m v1.17.0
Cluster (Pod-to-Pod) Networking
We will be using Flannel for our Cluster networking, which enables Pod-to-Pod communication. The following steps will deploy several containers that will initialize and configure the network fabric to enable pods to communicate. First, download a default flannel deployment configuration:
curl https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml \
> kube-flannel.yaml
Next, we’ll need to update the configuration to use the host-gw
configuration instead of vxlan
, and specify using arm
instead of arm64
. Edit
the kube-flannel.yaml
file and replace all instances of vxlan
with host-gw
, as well as replace any instance of arm64
with arm
. Once complete,
apply the flannel configuration:
kubectl apply -f kube-flannel.yaml
At this point, several containers should be deployed and start to configure the network fabric. You can re-run kubectl get nodes
over the next several
minutes and should start to see each node change state to Ready
, indicating the network configuration is complete on that node.
Firewall and Traffic Forwarding
On each node, data exchange between the wlan0
and eth0
devices is needed and is configured through IPTables routing rules and
sysctl
settings. Configure IP forwarding by editing /etc/sysctl.conf
:
# edit this property to be un-commented and set to a value of "1"
net.ipv4.ip_forward=1
Next, add the following statements to the file /etc/rc.local
to ensure the IPTables rules are correctly specified. Note that this step must be
done after the k8s software has been installed and configured otherwise the IPTables rules specified below will be overwritten.
iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT
When done, reboot your Raspberry Pi instance for the settings to take effect.
Dashboard
Some users prefer a UI to investigate issues and see the state and configuration of their cluster. We’ll deploy the default k8s dashboard following the default instructions here. First, we’ll deploy the dashboard:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta6/aio/deploy/recommended.yaml
Dashboard User and Binding
In order to access the dashboard, we’ll need to create a user and bind the user to enable access. The steps are to create a service account and then
bind the service account to enable access. Create a file dashboard-adminuser.yaml
with the following contents:
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
Apply the configuration to create the service account:
kubectl apply -f dashboard-adminuser.yaml
Next, create a binding configuration file named adminuser-binding.yaml
with the following contents:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
Apply the adminuser binding:
kubectl apply -f adminuser-binding.yaml
Dashboard User Token
In order to access the dashboard, a token is required. Run the following command, and copy the token
value to be used in order to access
the dashboard:
kubectl -n kubernetes-dashboard describe secret \
$(kubectl -n kubernetes-dashboard get secret \
| grep admin-user \
| awk '{print $1}')
Accessing the Dashboard
Now that we have a user and associated token, we can access the dashboard. From your local device, port forward to the k8s master node node1
:
ssh -L8001:localhost:8001 pirate@node1
Once on the k8s master node, run the following command to start the dashboard:
kubectl proxy
You should see a message indicating the dashboard is being served from 127.0.0.1:8001
. From your local device (not the k8s master, but the
device you port forwarded from), access the dashboard via the following URL in a browser:
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/overview?namespace=default
Use the Token
option to log in, and paste the token you captured from the “Dashboard User Token” section above. You should now see the
dashboard and all configurations/resources in your cluster!
Connecting from Local Device
There is a caveat if you wish to connect to the cluster from a local device (laptop or otherwise). Due to the fact that the eth0
interface
is hosting the cluster IP for the node subnet, if your local device is not on the same subnet/network (10.x
), you will likely need to SSH tunnel
and port forward the interface for kubectl
to operate. First, copy the /home/pirate/.kube/config
file to your local machine ~/.kube/config
file.
Then, port forward SSH-tunnel into the k8s master node1
instance for the kubectl
interface:
ssh -L6443:localhost:6443 pirate@node1
Finally, open a new window and run the following command to see the node status for your cluster:
kubectl get nodes --insecure-skip-tls-verify=true
If all goes well, you should see your cluster node status displayed!
Note: If you don’t want to have to include the --insecure-skip-tls-verify=true
switch for each command and you aren’t worried about man-in-the-middle
attacks (or don’t want to mess with certs), you can update the cluster config to remove cert checking/TLS verification entirely by using the following
command: kubectl config set-cluster kubernetes --insecure-skip-tls-verify=true
. Note that it is much preferred that you resolve the certificate validation
issues than use this method, but for quick starts, this may suffice for your needs.
Credit
The above tutorial was pieced together with some information from the following sites/resources: