NVidia Nano Node Platform

Contains the specific instructions and ansible scripts for the NVidia Nano Node Platform.

Hardware needed
Bootstrap Steps
Enabling root SSH access
Setup the Extra Drive
Configure Docker to use External Media
Configure the Network Interface, Udev Rules
Install the Internet sharing service
Install k3s
Configure k3s to Use External Media
Configure Network Manager to Not Manage the k3s Networks
Set up k3s Shutdown Service
k3s Service Startup Customizations
k3s GPU access Config
Test k3s and Docker's GPU Access
Configure The Local Dev Docker Registry
Configuring the Sensors

Hardware needed

NVIDIA Jetson Nano, either Micro-USB or DC power supply, barrel to set the jumper next to barrel connector to use barrel power, uSD memory cards, SD card reader, etc.
5 pin USB cable
Sensors: camera, microphone, & BME680.

Note: The jetson nano can NOT have an IP on the ethernet in the 10.42.0.0 IP space as k3s internal network uses that subnet. it breaks stuff.

Connecting the hardware

TODO: list the instructions on what other hardware needs to be connected at this time (to ensure the future ansible script will work)

512GB USB drive
usb2.0 extension to usb <-> ethernet dongle

Bootstrap Steps

Install NVidia Nano OS version 4.4.1 for Jetson Nano Developer Kit
Follow the instructions on the Getting Started with Jetson Nano Developer Kit website to write the image to the microSD card
1. Make sure to follow the correct set of instructions corresponding to your operating system
Insert the microSD card into the Nano
1. The microSD card slot is located on the underside of the Nano
Dev Note (TODO): insert in image where the sd card is located
Jumper the J48 Power Selector Header Pins
1. Pins not jumpered:
2. Pins Jumpered:
Connect an ethernet cable that is connected to the internet into the port. This is where the nano will get access to WAN (Wide Area Network ie internet).
Connect your computer to the Nano via it's micro USB port
Follow the instructions on Nvidia's website to set up the Nano according to your operating system
Once you are connected to the nano go through the initial set up
1. Set username to waggle and passwd to waggle
Dev note: come up with a better password later
1. Set partition size to 0 or leave blank
2. When you get to the Network Configuration Screen, select the eth0 option
3. Set hostname to localhost for now
  
  Note: The hostname will change later to node id
4. Use default nvpmodel (MAXN) (10W)
Dev note: we may be able to set this with ansible later. There are only 2 modes. the maxn is mode 001

You should now be in the nano's command line!

Now let's setup root SSH access

Enabling `root` SSH access (Jetson Nano Host)

Get IP for eth0

run command ifconfig eth0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 000.000.0.000  netmask 000.000.000.0  broadcast 000.000.0.000
inet6 fe80::4ab0:2dff:fe5b:fe63  prefixlen 64  scopeid 0x20<link>
inet6 0000:000:0000:0000:0000:0000:0000:0000  prefixlen 00  scopeid 0x0<global>
inet6 0000:000:0000:0000:0000:0000:0000:0000  prefixlen 00  scopeid 0x0<global>
inet6 0000:000:0000:0000:0000:0000:0000:0000  prefixlen 00  scopeid 0x0<global>
inet6 0000:000:0000:0000:0000:0000:0000:0000  prefixlen 00  scopeid 0x0<global>
inet6 0000:000:0000:0000:0000:0000:0000:0000  prefixlen 00  scopeid 0x0<global>
ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
RX packets 1371647  bytes 1819095948 (1.8 GB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 916252  bytes 87050219 (87.0 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
device interrupt 151  base 0x5000

The set of numbers next to inet is your ip address
```
inet 000.000.0.000
```

Run command sudo su
1. enter in your password ie waggle
Set root user password
1. run comand passwd root
2. set password to waggle
Dev note: come up with a better password later
Enable root user ssh login
1. run command vim /etc/ssh/sshd_config
  1. Go to the bottom of the file
  2. Enter in PermitRootLogin yes and PasswordAuthentication yes (Make sure they are in seperate lines)
  3. Save and exit the file
Note: In vim to enter in insert mode press 'i'. To exit insert mode press 'esc'. To save and quit type in ':wq' and press enter when not in insert mode
Open another terminal and SSH as root into the nano ssh root@<ip>

Set up SSH public key authentication to connect to a remote system (client side)

Getting a List of SSH Commands and Syntax

ssh

Set up public key authentication using SSH on a Linux or macOS computer

To generate RSA keys, on the command line, enter: ssh-keygen -t rsa

Use SFTP or SCP to copy the public key file (for example, ~/.ssh/id_rsa.pub) to your account on the remote system

scp ~/.ssh/id_rsa.pub waggle@IP-Address

SSH Access

ssh waggle@<ip> e.g., 10.0.0.151

SSH Access

ssh-keygen -R <ip> e.g., 10.0.0.151

Install the same docker file from https://hub.docker.com/r/waggle/gpu-stress-test/tags

docker pull waggle/gpu-stress-test:1.0.1

You won't see the any output on your current terminal. You can open another one.

sudo docker run -it --rm --runtime nvidia --network host waggle/gpu-stress-test:1.0.1 -m 2

Ansible Provisioning

Host machine (your laptop) Ansible setup

You will need to install ansible to the machine that you are using to ssh connect to the nano.

TOOD: fill in steps for configuring your laptop to run Nano. With a link to some options (brew and/or pip for mac).

Note: ansible is not needed on the Nano in order to perform ansible provisioning.

Setup the Extra Drive

Insert 512GB samsung usb stick into the nano
The drive should enumerate as sda1
1. run command lsblk to check
Clear the current partition table and create GPT table
```
fdisk --wipe always --wipe-partitions always /dev/sda
g
w
```
1. run command fdisk --wipe always --wipe-partitions always /dev/sda
2. press 'g' and then enter
3. press 'w' and then enter
Make the SWAP partition (16GB SWAP)
```
fdisk --wipe always --wipe-partitions always /dev/sda
n
""
""
+16G
t
19
w
```
1. run command fdisk --wipe always --wipe-partitions always /dev/sda
2. press 'n' and then enter
3. leave blank and press enter
4. leave blank and press enter
5. type in '+16G' and press enter
6. press 't' and press enter
7. type in '19' and press enter
8. press 'w' and press enter
Make the overlayfs 32GB
```
fdisk --wipe always --wipe-partitions always /dev/sda
n
""
""
+32G
w
```
1. run command fdisk --wipe always --wipe-partitions always /dev/sda
2. press 'n' and then enter
3. leave blank and press enter
4. leave blank and press enter
5. type in '+32G' and press enter
6. press 'w' and press enter
Make the plugin-data (the rest of the space)
```
  fdisk --wipe always --wipe-partitions always /dev/sda
  n
  ""
  ""
  ""
  w
```
1. run command fdisk --wipe always --wipe-partitions always /dev/sda
2. press 'n' and then enter
3. leave blank and press enter
4. leave blank and press enter
5. leave blank and press enter
6. press 'w' and press enter

Turn on the swap

Run command mkswap /dev/sda1 -L ext-swap

  root@localhost:~ mkswap /dev/sda1 -L ext-swap
Setting up swapspace version 1, size = 16 GiB (17179865088 bytes)
LABEL=ext-swap, UUID=6863907e-fe44-4d50-956b-cdc98490a059

Put the swap in the startup partition file
1. Run command echo "/dev/sda1 swap swap defaults,nofail 0 0" >> /etc/fstab
Setup the overlayfs partition
1. run command mkfs.ext4 /dev/sda2
2. run command e2label /dev/sda2 system-data

Note: We will actually enable the overlayfs at the very end (TODO)

Set the default mount of /media/plugin-data in the /etc/fstab

Run command mkfs.ext4 /dev/sda3
Run command e2label /dev/sda3 plugin-data

Run command:

echo "/dev/sda3 /media/plugin-data ext4 defaults,nofail,x-systemd.after=local-fs-pre.target,x-systemd.before=local-fs.target 0 2" >> /etc/fstab

Reboot the nano by running command reboot
SSH into the nano as root again
run command lsblk to see the drive configured correctly

Dev Note: insert in the correct output here

Install Helpful Tools

Install helpful tools

apt-get update && apt-get install -y \
dnsutils \
htop \
iftop \
iotop \
jq \
nmap

Remove Uneeded Items

Remove installs we don't need or conflict

apt-get purge -y \
isc-dhcp-server -y \
whoopsie
apt-get autoremove -y

Disable apt upgrade services

systemctl disable apt-daily.service && systemctl disable apt-daily.timer
systemctl disable apt-daily-upgrade.service && systemctl disable apt-daily-upgrade.timer

Disable the motd news updates

chmod -x /etc/update-motd.d/*
echo 'ENABLED=0' > /etc/default/motd-news
systemctl disable motd-news.service && systemctl disable motd-news.timer

Add the WaggleOS MOTD

Copy file /ROOTFS/etc/update-motd.d/05-waggle to system and give execute permissions

scp ROOTFS/etc/update-motd.d/05-waggle <ip>:/etc/update-motd.d/05-waggle

chmod +x /etc/update-motd.d/05-waggle

Configure the Network Interface, Udev Rules

Go into rules.d directory by running this command cd /etc/udev/rules.d/

Create a file using vim by running this command vim 10-waggle.rules

Go into insert mode, and paste the following content

## WAN configuration
# all: find the Nvidia native ethernet interface, assign to WAN
KERNEL=="eth*", ATTR{address}=="48:b0:2d:*", NAME="wan0"

## LAN configuration
KERNEL=="eth*", ATTR{address}=="f8:e4:3b:*", NAME="lan0"

Escape insert mode, save and quit file

Reboot device to see eth0 change to wan0

Install the Internet sharing service

```bash
wget https://github.com/waggle-sensor/waggle-internet-share/releases/download/v1.4.1/waggle-internet-share_1.4.1_all.deb

dpkg -i waggle-internet-share_1.4.1_all.deb
```
> Note: This creates a service that ensures that hardware on the 10.31.81.1 has Internet access (by sharing the Nano's Internet uplink)

Configure `journalctl` to save historical logs

create the journal log folder

mkdir -p /var/log/journal

add the journalctl config

mkdir -p /etc/systemd/journald.conf.d/

scp ROOTFS/etc/systemd/journald.conf.d/10-waggle-journald.conf <ip>:/etc/systemd/journald.conf.d/10-waggle-journald.conf

Configure Docker to use External Media

Stop docker by running command service docker stop

Note: Docker is already installed by the native L4T / Jetpack so we will use that

Move docker by running command `mv /var/lib/docker /media/plugin-data/
Create a symbolic link by running command ln -s /media/plugin-data/docker/ /var/lib/docker
Start up docker by running command service docker start

Install k3s

Install curl by running this command apt-get install curl

Install K3s via curl by running this command:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.20.2+k3s1 INSTALL_K3S_SKIP_ENABLE=true K3S_CLUSTER_SECRET=4tX0DUZ0uQknRtVUAKjt sh -

Configure k3s to Use External Media

Check the status of k3s by runnig this command service k3s status

Output should look like this:

● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; disabled; vendor preset: enabled)
Active: inactive (dead)
 Docs: https://k3s.io

Run this command to save external media dir as a variable NVMEMOUNT=/media/plugin-data
1. Check if the variable was saved by running this command ls ${NVMEMOUNT}
```
ls ${NVMEMOUNT}
docker  lost+found
```

Run the following commands to create the neccessary directories in the external media

mkdir -p ${NVMEMOUNT}/k3s/etc/rancher
mkdir -p ${NVMEMOUNT}/k3s/kubelet
mkdir -p ${NVMEMOUNT}/k3s/rancher

Run the following commands to create symbolic links in the newly created directories

ln -s ${NVMEMOUNT}/k3s/etc/rancher /etc/rancher
ln -s ${NVMEMOUNT}/k3s/kubelet /var/lib/kubelet
ln -s ${NVMEMOUNT}/k3s/rancher /var/lib/rancher

Enable k3s by running this command systemctl enable k3s.service

Output should look like this:

Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

Configure Network Manager to Not Manage the k3s Networks

Go to directory /etc/NetworkManager/conf.d/
Create a file using vim by running this command vim cni.conf
1. Go into insert mode, and paste the following content
```
[keyfile]
unmanaged-devices=interface-name:flannel.1;interface-name:veth*;interface-name:cni0
```
2. Escape insert mode, save and quit file

Set up k3s Shutdown Service

Go to directory /etc/systemd/system/

Create a file using vim by running this command vim waggle-k3s-shutdown.service

Go into insert mode, and paste the following content

[Unit]
Description=Gracefully k3s Shutdown
DefaultDependencies=no
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/k3s-killall.sh
TimeoutStartSec=0

[Install]
WantedBy=shutdown.target

Escape insert mode, save and quit file

Enable service by running the following commands

systemctl daemon-reload
systemctl enable waggle-k3s-shutdown

k3s Service Startup Customizations

Go to directory /etc/
Create a directory running this command mkdir waggle
Go into the waggle directory you just created

Create a file using vim by running this command vim config.ini

Go into insert mode, and paste the following content

[system]
name = wd-nanocore

[hardware]
lan-interface = lan0
wlan-interface = wan0
wifi-interface = wifi0
modem-interface = modem0

[network]
static-ip-nx = 10.31.81.1

[registration]
host = beekeeper.sagecontinuum.org
port = 49190
user = sage_registration
key = /etc/waggle/sage_registration
keycert = /etc/waggle/sage_registration-cert.pub

[reverse-tunnel]
host = beekeeper.sagecontinuum.org
port = 49190
key = /etc/waggle/bk_key.pem
pubkey = /etc/waggle/bk_pubkey.pem

Escape insert mode, save and quit file

Install python3-click by running this command apt-get install python3-click
Travel to home dir by running this command cd ~
Install the waggle-nodeid tool by running the follwing commands
```
wget https://github.com/waggle-sensor/waggle-nodeid/releases/download/v1.0.7/waggle-nodeid_1.0.7_all.deb

dpkg -i waggle-nodeid_1.0.7_all.deb
```
Note: This creates a waggle-nodeid service that should run on start-up and create the /etc/waggle/node-id file
Reboot the device so the service creates the /etc/waggle/node-id file
Execute this command to test if the service is working service waggle-nodeid status
```
Dev Note: insert correct output here
```
- If hostname does not resemble 'wd-nanocore-<id>' after the test was done then do the following:
  1. Go to home directory cd ~
  2. Install waggle-node-hostname by running the following commands:
```
wget https://github.com/waggle-sensor/waggle-node-hostname/releases/download/v1.2.1/waggle-node-hostname_1.2.1_all.deb

dpkg -i waggle-node-hostname_1.2.1_all.deb
```
  3. Run the following commands:
```
touch /etc/waggle/bk_key.pem

/usr/bin/waggle_node_hostname.py -n /etc/waggle/node-id

rm /etc/waggle/bk_key.pem
```
    Note: this is a work-around that will fixed better later (TODO)
  4. Reboot the device, Hostname should change after that

Configure the lan0 network interface to bind to 10.31.81.1

Go to directory /etc/NetworkManager/system-connections/

Create a file using vim by running this command vim lan

Go into insert mode, and paste the following content

[connection]
id=lan
type=ethernet
autoconnect=true
interface-name=lan0
permissions=

[ethernet]
auto-negotiate=true
mac-address-blacklist=

[ipv4]
address1=10.31.81.1/24
dns-search=
method=manual
never-default=true

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=ignore

Escape insert mode, save and quit file

Apply 0600 permission to lan file by running this command chmod 0600 lan

Create a file using vim by running this command vim wan

Go into insert mode, and paste the following content

[connection]
id=wan
type=ethernet
autoconnect=true
interface-name=wan0
permissions=

[ethernet]
auto-negotiate=true
mac-address-blacklist=

[ipv4]
dns-search=
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=ignore

Escape insert mode, save and quit file

Apply 0600 permission to wan file by running this command chmod 0600 wan

Create a file using vim by running this command vim wifi

Go into insert mode, and paste the following content

[connection]
id=wifi
type=wifi
autoconnect=true
interface-name=wifi0
permissions=

[wifi]
mac-address-blacklist=
mode=infrastructure
#ssid=<access_point>

[wifi-security]
auth-alg=open
#key-mgmt=wpa-psk
#psk=<password>

[ipv4]
dns-search=
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=ignore

Escape insert mode, save and quit file

Apply 0600 permission to wan file by running this command chmod 0600 wifi

Install and configure dnsmasq to create the device's internal 10.31.81.1/24 network (for use by kubernetes).

Install dnsmasq by running this command apt-get install dnsmasq
Go to directory /etc/dnsmasq.d

Create a file using vim by running this command vim 10-waggle-base.conf

Go into insert mode, and paste the following content

## Basic global config
log-dhcp
#log-queries

## LAN DNS resolution / forwarding
listen-address=10.31.81.1
interface=lan0
bind-interfaces

## Global DHCP Options
dhcp-authoritative
bogus-priv
# gateway
dhcp-option=3,10.31.81.1
# DNS server
dhcp-option=6,10.31.81.1

## DHCP address assignment
dhcp-mac=set:rpi,DC:A6:32:*:*:*
dhcp-mac=set:rpi,3A:35:41:*:*:*
dhcp-mac=set:rpi,E4:5F:01:*:*:*
dhcp-mac=set:rpi,28:CD:C1:*:*:*
dhcp-range=tag:rpi,10.31.81.4,10.31.81.4,infinite
dhcp-mac=set:camera,9C:8E:CD:*:*:* # AMCREST camera
dhcp-mac=set:camera,E4:30:22:*:*:* # Hanwha camera
dhcp-mac=set:camera,00:03:C5:*:*:* # Mobotix camera
dhcp-range=tag:camera,10.31.81.10,10.31.81.19,infinite
# general DHCP pool
dhcp-range=10.31.81.50,10.31.81.254,10m

Escape insert mode, save and quit file

Go to directory /etc/systemd/system
Create a directory by running this command mkdir dnsmasq.service.d
Go into this newly created directory cd dnsmasq.service.d
Create a file using vim by running this command vim 10-waggle-base.conf
1. Go into insert mode, and paste the following content
```
[Unit]
# start after the lan0 bound interface is up
Requires=network-online.target
After=network-online.target

[Service]
Restart=always
RestartSec=30
```
2. Escape insert mode, save and quit file
Reload daemon by running this command systemctl daemon-reload

Set up k3s service override

Go to directory /etc/systemd/system
Create a directory by running this command mkdir k3s.service.d
Go into the directory you just created cd k3s.service.d

Create a file using vim by running this command vim override.conf

Go into insert mode, and paste the following content

[Unit]
After=waggle-nodeid.service
Wants=waggle-nodeid.service
# desire to start after dnsmasq creates the lan IP
After=dnsmasq.service
Wants=dnsmasq.service

[Service]
# Fail service if Node ID file does not exist
ExecStartPre=/usr/bin/test -e /etc/waggle/node-id
ExecStartPre=/usr/bin/nmcli conn up lan
ExecStartPre=/etc/waggle/k3s_config/pre-run.sh
ExecStart=
ExecStart=/bin/bash -ce "/usr/local/bin/k3s server \
  --node-name $(cat /etc/waggle/node-id).$(waggle-get-config -s system -k name) \
  --disable=traefik \
  --bind-address $(waggle-get-config -s network -k static-ip-nx) \
  --node-ip $(waggle-get-config -s network -k static-ip-nx) \
  --advertise-address $(waggle-get-config -s network -k static-ip-nx) \
  --flannel-iface $(waggle-get-config -s hardware -k lan-interface) \
  --resolv-conf /etc/waggle/k3s_config/resolv.conf \
"

Escape insert mode, save and quit file

Reload daemon by running this command systemctl daemon-reload

Install waggle-commontools to read the config
1. Travel to your home directory by running this command cd ~
2. Install waggle-commontools by running the following commands:
```
wget https://github.com/waggle-sensor/waggle-common-tools/releases/download/v1.0.0/waggle-common-tools_1.0.0_all.deb

dpkg -i waggle-common-tools_1.0.0_all.deb
```

k3s GPU access Config

Go to directory /etc/waggle/
Create a new directory by running this command mkdir k3s_config
Go into the directory you just created cd k3s_config

Create a file using vim by running this command vim config.toml.tmpl

Go into insert mode, and paste the following content

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"


[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "rancher/pause:3.1"


[plugins.cri.containerd]
  disable_snapshot_annotations = true
  snapshotter = "overlayfs"


[plugins.cri.cni]
  bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
  conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"


[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"


[plugins.cri.containerd.runtimes.runc.options]
  BinaryName = "nvidia-container-runtime"

Escape insert mode, save and quit file

Create a file using vim by running this command vim pre-run.sh

Go into insert mode, and paste the following content

#!/bin/bash -e

# Script to be run before every execution of the K3s server

CONFIG_DEST=/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
CONFIG_SRC=/etc/waggle/k3s_config/config.toml.tmpl

# ensure the destination path exists
mkdir -p $(dirname $CONFIG_DEST)

cp $CONFIG_SRC $CONFIG_DEST

Escape insert mode, save and quit file

Give execution permission to pre-run.sh by running this command chmod +x pre-run.sh
Create a file using vim by running this command vim resolv.conf
1. Go into insert mode, and paste the following content
```
# Use local dnsmasq DNS (backed by systemd-resolve)
nameserver 10.31.81.1
```
2. Escape insert mode, save and quit file

Test k3s and Docker's GPU Access

Test k3s service override by running the following commands:

service k3s status
systemctl cat k3s

Make sure lan and wan are connected before testing k3s service

wan0
u should have your uplink (to network switch) plugged into the nano ethernet port

lan0
u should have the ethernet dongle connected to a usb port
(this is what you connect the camera to)

if this is not the case then you will get this error when running service k3s status:

Process: 7538 ExecStartPre=/usr/bin/nmcli conn up lan (code=exited, status=4)

k3s service should start and the service override should work

Dev Note: Insert correct Ouput here

If service k3s status returns active then test k3s and run basic pod by running the following commands:

kubectl get pod -A
kubectl get node

The output should look like this:

NAME                           STATUS   ROLES                  AGE   VERSION
000048b02d5bfe58.wd-nanocore   Ready    control-plane,master   19h   v1.20.2+k3s1

Test dnsmasq by running the following commands:
```
service dnsmasq status
systemctl cat dnsmasq.service
```
Dev Note: insert the correct output here
Test docker's GPU access by running this command docker run -ti --rm --gpus all waggle/gpu-stress-test:latest
- To see if the gpu is being used check the gpu frequency (GR3D_FREQ) by running this command tegrastats in a seperate terminal
If service k3s status returns active and kubectl get node returns an active node then test k3s' GPU access by running this command kubectl run gpu-test --image=waggle/gpu-stress-test:latest --attach=true

Note: You may see the error error: timed out waiting for the condition. that is okay. it just means it is taking a long time to create the container in k3s
- It takes a while for the pod to create, to watch the pod creation status run this command watch kubectl get pod
- To see if the gpu is being used check the gpu frequency (GR3D_FREQ) by running this command tegrastats in a seperate terminal
Once the pod stops running (~5 min) delete the pod by running this command kubectl delete pod gpu-test &

Configure The Local Dev Docker Registry

Dev Note: Skipping the sage and docker.io mirror registries as they just add a complexity we don't need

Setup the local docker registry mirrors (as the k3s config uses them)

Go to directory /etc/systemd/system/

Create a file using vim by running this command vim waggle-registry-local.service

Go into insert mode, and paste the following content

[Unit]
Description=Waggle Docker Registry (Local)
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=30s
ExecStart=/usr/bin/docker run --rm -p 5000:5000 --name local_registry \
    -v /media/plugin-data/docker_registry/local:/var/lib/registry \
    -v /etc/waggle/docker/certs:/certs \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
    -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
    registry:2
ExecStop=/usr/bin/docker stop local_registry
ExecStopPost=-/usr/bin/docker container rm local_registry

[Install]
WantedBy=default.target

Escape insert mode, save and quit file

Go to directory /etc/waggle/
Create a directory by running this command mkdir docker
Travel into the newly created directory cd docker
Create a directory by running this command mkdir certs
Travel into the newly created directory cd certs

Create a file using vim by running this command vim domain.crt

Go into insert mode, and paste the following content

Escape insert mode, save and quit file

Create a file using vim by running this command vim domain.key

Go into insert mode, and paste the following content

Escape insert mode, save and quit file

Configure the local dev docker keys

Go to home directory cd ~ and run the following commands:

chmod 600 /etc/waggle/docker/certs/domain.*

mkdir -p /etc/docker/certs.d/10.31.81.1\:5000/

cp /etc/waggle/docker/certs/domain.crt /etc/docker/certs.d/10.31.81.1\:5000/

mkdir -p /usr/local/share/ca-certificates

cp /etc/waggle/docker/certs/domain.crt /usr/local/share/ca-certificates/docker.crt

update-ca-certificates

After running update-ca-certificates the ouput should look like the following:

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

Make the directories for the docker registries mkdir -p /media/plugin-data/docker_registry/local

Enable the docker registry services by running the following commands:

systemctl daemon-reload
systemctl enable waggle-registry-local.service

To test the local dev docker registry run the following commands:
```
curl https://10.31.81.1:5000/v2/_catalog
```
- Output:
```
{"repositories":[]}
```

Ensure you can pull from local registry by running the following commands:

docker pull ubuntu:latest

docker tag ubuntu:latest 10.31.81.1:5000/joe:latest

curl https://10.31.81.1:5000/v2/_catalog

Output

{"repositories":[]}

Ensure you can push to local registry by running this command:

docker push 10.31.81.1:5000/joe:latest

Output

13e8c0db60e7: Pushed
latest: digest: sha256:0f744430d9643a0ec647a4addcac14b1fbb11424be434165c15e2cc7269f70f8 size: 529

Note: while doing the above push you should see logs in docker logs -f local_registry

You should now see a repository by running this command curl https://10.31.81.1:5000/v2/_catalog
- Output
```
{"repositories":["joe"]}
```

Add fallback DNS

ROOTFS file: ROOTFS/etc/systemd/resolved.conf.d/10-waggle-resolved.conf

Add wifi dongle support

ROOTFS file: ROOTFS/etc/udev/rules.d/10-waggle.rules ROOTFS file: ROOTFS/etc/modprobe.d/rtl8821cu.conf ROOTFS file: ROOTFS/etc/modprobe.d/rtl8822bu.conf

Note: the modprobe conf files above disable the auto-power management (which is unstable)

Add 'waggle' hotspot support

ROOTFS file: ROOTFS/etc/NetworkManager/system-connections/wifi-waggle

chmod 600 /etc/NetworkManager/system-connections/*

Note: the device will auto-connect to a ssid: waggle / passwd: Why1Not@Waggle network now

Enable NetworkManager Connectivity checks

ROOTFS file: ROOTFS/etc/NetworkManager/conf.d/99-connectivity.conf

echo 'net.ipv4.conf.default.rp_filter = 2' >> /etc/sysctl.conf
echo 'net.ipv4.conf.all.rp_filter = 2' >> /etc/sysctl.conf

Note: this change ensures that default ip route that is used actually has an Internet connection

Set the node friendly ID (VSN)

# echo N001 > /etc/waggle/vsn

Note: the above instructions hard-code a static VSN. In the future we will want to generate a random VSN (NXXX where X is a [0-9A-Z]) (TODO)

Register node and establish connection to Beehive

Install the registration and reverse tunnel services

wget https://github.com/waggle-sensor/waggle-bk-registration/releases/download/v2.2.2/waggle-bk-registration_2.2.2_all.deb
wget https://github.com/waggle-sensor/waggle-bk-reverse-tunnel/releases/download/v2.3.2/waggle-bk-reverse-tunnel_2.3.2_all.deb
dpkg -i waggle-bk-registration_2.2.2_all.deb
dpkg -i waggle-bk-reverse-tunnel_2.3.2_all.deb

note: we either need to reboot after the above steps or force start the services

Copy the Beekeeper known hosts public CA

ROOTFS: ROOTFS/etc/ssh/ssh_known_hosts

chmod 644 /etc/ssh/ssh_known_hosts

Copy registration private key

ROOTFS file: ROOTFS/etc/waggle/sage_registration

chmod 600 /etc/waggle/sage_registration

note: this file is not checked into the code at this time, need to triple-check its safe to do so. (TODO)

Copy the registration certificate

Obtain the regristration certificate from the "node registration portal" and copy to: /etc/waggle/sage_registration-cert.pub

chmod 600 /etc/waggle/sage_registration-cert.pub

Note: for the initial test the "production" registration certificate was used, which not be what is used long term Note: this will be the registration cert that is obtained from the "node registration portal"

Enable admin reverse tunnel access

ROOTFS: ROOTFS/root/.ssh/authorized_keys.prod

cat /root/.ssh/authorized_keys.prod >> /root/.ssh/authorized_keys

note: the above file is appended to the currently existing authorized_keys file, as the file was already created and has a key in it to enable the ansible ssh session

note: at this time an admin should be able to login to the nano via the reverse tunnel through the beekeeper administrative services

Get WES running

After registration, the Beehive should automatically push WES to the node and start running its pods

Note: this section is intended to show what it looks like when WES is running, as a verification of things working

# kubectl get pod
NAME                                        READY   STATUS              RESTARTS   AGE
wes-audio-server-75847fd59-5ql4n            0/1     Pending             0          11m
wes-gps-server-6dd8f84cb9-pkrvf             0/1     Pending             0          11m
node-exporter-mdc5x                         1/1     Running             0          11m
wes-upload-agent-pcjdn                      1/1     Running             0          11m
wes-scoreboard-679ccdddb7-zfxms             1/1     Running             0          11m
wes-metrics-agent-5frxw                     1/1     Running             0          11m
wes-device-labeler-8vqdh                    1/1     Running             0          11m
wes-node-influxdb-0                         1/1     Running             0          11m
wes-camera-provisioner-1657315800-56t8f     0/1     ContainerCreating   0          4m56s
wes-sciencerule-checker-646b4c6c4-h7clz     1/1     Running             0          11m
wes-data-sharing-service-84b7958dd9-l4642   1/1     Running             4          11m
wes-plugin-scheduler-c665b68b5-72r5z        1/1     Running             0          11m
wes-rabbitmq-0                              1/1     Running             1          11m
wes-node-influxdb-loader-6b58f7474-25k8n    1/1     Running             3          11m

You should start seeing metrics being published to the Beehive data store

$ curl -s 1-H 'Content-Type: application/json' https://data.sagecontinuum.org/api/v1/query -d '{"start": "-125s","filter": {"vsn": "N001"}}'  | grep uptime
{"timestamp":"2022-07-08T21:37:50.542686843Z","name":"sys.uptime","value":12617.5,"meta":{"host":"000048b02d5bfe58.wd-nanocore","node":"000048b02d5bfe58","vsn":"N001"}}

Configuring the Sensors

BME680

Connect in the 5 pin USB cable into the environmental sensor with the black wire connected to the ground pin
With the Nano powered off, connect the other end of the cable into the nano on the bottom leftmost set of 5 pins on the 40-pin expansion header. The black wire should connect to the ground pin and the red wire should connect to the pin labeled 3v3

Run this command to see if the sensor is being detected by i2c. i2cdetect -y -r 1 0x76 0x77

root@wd-nanocore-000048B02D5BFE58:~# i2cdetect -y -r 1 0x76 0x77
    0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30:                                                 
40:                                                 
50:                                                 
60:                                                 
70:                   76 --

Note: The sensor can either be on 0x76 or 0x77. Based on what address it's on the number will change in the documentation

The kernel supplied by nvidia doesn't support the BME680 sensor, so we just have to get the kernel source of a kernel version supporting the sensor (4.19) wget http://www.kernel.org/pub/linux/kernel/v4.x/linux-4.19.116.tar.gz
Extract the drivers/iio/chemical subtree of the kernel source for building the bme680 sensors kernel module tar --strip-components=3 -xzf linux-4.19.116.tar.gz linux-4.19.116/drivers/iio/chemical
After extracting, we have to configure the build. To do so, prepend the following lines to the Makefile in this subtree vim chemical/Makefile
```
CONFIG_BME680=m
CONFIG_BME680_I2C=m
```
Afterwards build the kernel module and install the binaries to the modules folder by running the following commands:
1. make -C /lib/modules/uname -r/build M=$PWD
2. make -C /lib/modules/uname -r/build M=$PWD modules_install
Build the module's dependency list by running the command depmod -a
Run command modprobe bme680_i2c to see if the module was successfully installed

Note: If there is no output then the module was installed correctly

To use the sensor we have to register it with the kernels i2c subsystem so run this command echo bme680 0x76 > /sys/bus/i2c/devices/i2c-1/new_device

Note: change number to 77 if that's the address the sensor is on

Check if the BME680 sensor is registered by running the command cat /sys/bus/iio/devices/*/name
```
root@<id>:~# cat /sys/bus/iio/devices/*/name

ina3221x
bme680
```

Once the sensor is registered, the sensors readings can be acquired using the sysfs interface:

root@<id>:~#  grep ^ /dev/null /sys/bus/i2c/devices/i2c-1/1-0076/iio:device1/*input*

/sys/bus/i2c/devices/i2c-1/1-0076/iio:device1/in_humidityrelative_input:63.827000000
/sys/bus/i2c/devices/i2c-1/1-0076/iio:device1/in_pressure_input:984.630000000
/sys/bus/i2c/devices/i2c-1/1-0076/iio:device1/in_resistance_input:7234
/sys/bus/i2c/devices/i2c-1/1-0076/iio:device1/in_temp_input:24510

Note: If your sensor is on 0x77 the relative path will be /sys/bus/i2c/devices/i2c-1/1-0077/iio:device1/*input*

If your outputs in steps 11 & 12 are similiar then configure the nano to set bme680 on boot
1. To load the bme680_i2c kernel module on startup add it to the modules.conf file by running this command echo bme680_i2c >> /etc/modules-load.d/modules.conf
2. Next, we have to register the sensor with the kernels i2c subsystem on startup. To do so we have to create a rc.local file since ubuntu 18.04 doesn't have it by default. vim /etc/rc.local
  1. Go into insert mode, and paste the following content
```
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Add bme680 to i2c
echo bme680 0x76 > /sys/bus/i2c/devices/i2c-1/new_device

exit 0
```
  2. Escape insert mode, save and quit file
  3. Give execution permission to the file chmod +x /etc/rc.local
Reboot and on startup run steps 11 & 12 again to see if the BME680 sensor is set up on boot

TODO ITEMS

currently working on

enable the overlayfs (this may need to be done before registration completes)
- its okay if the registration cert exists on the read-only partition, as its timed anyway and not a big secret

later

we need the ip_set kernel modules maybe
check service startup order (svg) to confirm its all good
get microphone running and wes configued to set the nano core as the node running the audio-server
the camera will have an IP in the range of 10.31.81.10 - 19. do we want to set the Amcrest camera to a static IP (i.e. 10.31.8.20 ) ?
- we also need to figure out how to connect the camera to pyWaggle / WES. we don't want to have to run the "camera provisioner"
add version to /etc/waggle_version_os
install and have the Nano's use the wan-tunnel to route all Internet traffice through beekeeper
figure out how to generate a random VSN
- NXXX where X is a random value between [0-9A-Z]

Optional / research / unknown

update the instructions for creating a "dummy" user on install and then adding the creation of the waggle user in the Ansible script so that it can be updated and easily versioned (instead of relying on the initial creation). can tie this into sudoers access for waggle etc.
- this may not be necessary as the user can operate as root on the node, and we should encourage that.
(optional) create /var/lib/nvpmodel/status file to set the default operating mode and fmode (fan mode) to cool
- it seems that cool is not allowed. so need to figure out what fan modes ARE supported, if any
figure out if we want to allow root user to be able to login with ssh with password or not??
the first time the external drive enumerated as sdb instead of sda, why? we need to make sure this is consistent
remove dhcpd service from running (its not running on a WSN)
we may not need the sage and docker.io mirror, as they just take up additional space and maybe take longer. there are no k3s agents to take advantage of the mirrors
- plus this assumes a sagecontinuum.org ECR which we dont need to assume
remove the X server and other gui elements that take up a alot of space/resources
- this also includes extras that NVidia put in there (examples maybe?)
some basic unit test / sanity test / health script to ensure the device is healthy (see the waggle-sanity-check - for the students)
- check lan IP
- check k3s is running
- check docker is running
- check docker mirrors working
- check internet connection
update the MOTD to point out the url to go for registration
add support for RPI PxE booting, to allow for an easy downstream "agent"
- this also requires the fix to allow the rpi to use the local docker registry on the core (TBD)
add a Ansible script for configuring the Nano to operate as an "agent" instead of a "core"
- this includes the required ssh key access to be able to ssh to the agents from the core (for admin stuff)
- this includes the local docker registry access

the lsblk is showing up as, why does it have the UUID in it?

├─sda2         8:2    1    16G  0 part /media/1aef5c37-a088-4343-9a55-530d68442109
└─sda3         8:3    1 428.3G  0 part /media/df5990dc-414c-4aa4-975d-774bc8811a8a

auto-mount problem (see: systemctl --failed)

Files

README.md

Latest commit

History

README.md

File metadata and controls

NVidia Nano Node Platform

Table of Contents

Hardware needed

Connecting the hardware

Bootstrap Steps

You should now be in the nano's command line!

Enabling root SSH access (Jetson Nano Host)

Set up SSH public key authentication to connect to a remote system (client side)

Getting a List of SSH Commands and Syntax

Set up public key authentication using SSH on a Linux or macOS computer

Use SFTP or SCP to copy the public key file (for example, ~/.ssh/id_rsa.pub) to your account on the remote system

SSH Access

SSH Access

Install the same docker file from https://hub.docker.com/r/waggle/gpu-stress-test/tags

You won't see the any output on your current terminal. You can open another one.

Ansible Provisioning

Host machine (your laptop) Ansible setup

Setup the Extra Drive

Install Helpful Tools

Remove Uneeded Items

Add the WaggleOS MOTD

Configure the Network Interface, Udev Rules

Install the Internet sharing service

Configure journalctl to save historical logs

Configure Docker to use External Media

Install k3s

Configure k3s to Use External Media

Configure Network Manager to Not Manage the k3s Networks

Set up k3s Shutdown Service

k3s Service Startup Customizations

k3s GPU access Config

Test k3s and Docker's GPU Access

Configure The Local Dev Docker Registry

Add fallback DNS

Add wifi dongle support

Add 'waggle' hotspot support

Enable NetworkManager Connectivity checks

Set the node friendly ID (VSN)

Register node and establish connection to Beehive

Get WES running

Configuring the Sensors

BME680

TODO ITEMS

currently working on

later

Optional / research / unknown

Enabling `root` SSH access (Jetson Nano Host)

Configure `journalctl` to save historical logs