diff --git a/README.md b/README.md index 9cdea42..e825008 100644 --- a/README.md +++ b/README.md @@ -1,160 +1,256 @@ -# OTA Community Edition (mono)-lith +# OTA-Community-Edition -Easy to run, secure, open source [tuf](https://theupdateframework.io/)/[uptane](https://uptane.github.io/) over the air (OTA) updates. +- The OTA Community Edition is an open-source server software using uptane to deliver over-the-air (OTA) updates to compatible clients. -You may not want or need to run [ota-community-edition](https://github.com/advancedtelematic/ota-community-edition) using a microservice architecure. A monolith might fit your use case better if you just want to try `ota-community-edition` or if your organization doesn't need to serve millions of devices. A monolith architecture is easier to deploy and manage, and uses less resources. +- Uptane is an open and secure software update framework design which protects software delivered over-the-air to automobile electronic control units (ECUs). -This project bundles all the scala apps included in [ota-community-edition](https://github.com/advancedtelematic/ota-community-edition) into a application that can be executed in a single container. The app can be easily configured using a single configuration file and avoids the usage of environment variables to simplifly configuration. Additionally, a `docker-compose` file is provided to run the application. This means you no longer need a kubernetes cluster if you just want to try or test `ota-community-edition`. +- OTA-CE comprises of a number of services which together make up the OTA system. + `reposerver, keyserver, director, deviceregistry, campaigner and treehub` -For small deployments, you don't need kubernetes. This solution can fit your organization better. With this app you could run ota in a single machine/vm + mariadb and kafka. +- The source code for the servers is available on [Advanced Telematic's Github](https://github.com/advancedtelematic) and is licensed under the MPL2.0 -## Active branches +- Docker container images of the latest build are available on Docker Hub : [Advanced Telematic](https://hub.docker.com/u/advancedtelematic) & [ota-lith](https://hub.docker.com/r/uptane/ota-lith) -Currently there are three active branches in this repository: +## Prerequisite +### Linux/Debian setup with the following installed: +``` +sudo apt install asn1c build-essential cmake curl libarchive-dev libboost-dev libboost-filesystem-dev libboost-log-dev libboost-program-options-dev libcurl4-openssl-dev libpthread-stubs0-dev libsodium-dev libsqlite3-dev pkg-config libssl-dev python3 uuid-runtime +``` +- [docker](https://docs.docker.com/engine/install/) (with [docker compose](https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command) version 2.6.x & above) -- `master`. `ota-community-edition` running without webapp and with kafka, using a single scala app. +- [aktualizr](https://github.com/advancedtelematic/aktualizr) -- `webapp`. The webapp is broken in `ota-community-edition` and therefore is not included in `ota-lith/master`. The `webapp` branch includes patched version of `webapp` that does not rely on user profile and is therefore working with both `ota-community-edition` and `ota-lith`. -## Dependency management - -This project follows the upstream [UPTANE OTA](https://uptane.github.io/) (see [sources](https://github.com/uptane/)) projects. +## Install OTA-CE +#### All the following commands are run from the `ota-community-edition` directory -The following forks/branches are included: +### 1. Add the following host names to `/etc/hosts` : +> paste the following on the last line of `/etc/hosts` +``` +0.0.0.0 reposerver.ota.ce +0.0.0.0 keyserver.ota.ce +0.0.0.0 director.ota.ce +0.0.0.0 treehub.ota.ce +0.0.0.0 deviceregistry.ota.ce +0.0.0.0 campaigner.ota.ce +0.0.0.0 app.ota.ce +0.0.0.0 ota.ce +``` +### 2. Run the script `gen-server-certs.sh` : +> Once you clone the repository and are inside it, generate the required certificates. +> This is required to provision the device using aktualizr. +```bash +#make sure the script is executable +chmod +x scripts/gen-server-certs.sh + +./scripts/gen-server-certs.sh +``` -- tuf https://github.com/uptane/ota-tuf -- director https://github.com/uptane/director -- device-registry https://github.com/uptane/ota-device-registry -- campaigner https://github.com/simao/campaigner -- treehub https://github.com/uptane/treehub -- libats https://github.com/uptane/libats -The dependencies are managed using -[git-subtree](https://man.archlinux.org/man/git-subtree.1) under the -`repos` directory on this repository. +### 3. Pull latest ota-lith image from docker +> Read more about [ota-lith](https://github.com/simao/ota-lith) +```bash +export img=uptane/ota-lith:$(git rev-parse master) +docker pull $img +docker tag $img uptane/ota-lith:latest +``` +### 4. Run docker compose +> Run in daemon mode +```bash +docker compose -f ota-ce.yaml up -d -To update to the latest changes from upstream ota-tuf, you could use for example: +# stop using followinng +docker compose -f ota-ce.yaml down +``` +> Run without daemon mode +```bash +docker compose -f ota-ce.yaml up - git subtree pull --prefix repos/ota-tuf git@github.com:uptane/ota-tuf.git master --squash +#stop by hitting ctrl+c or by running the following command in another terminal +docker compose -f ota-ce.yaml down +``` -If you wish to make changes to ota-tuf, you could edit directly -`repos/ota-tuf` and then commit the changes, then use `git-subtree` to -split the changes and open a pull request upstream. However, a simpler -way would be to just use `git diff` to generate a patch and apply that -patch to the repository separately: +### 5. You can now create a virtual device - cd repos/ota-tuf - git diff . > tuf.patch - cd /home/user/my-ota-tuf - patch -p3 < tuf.patch - - # normal git flow to create a PR for ota-tuf - -And then once those changes are merged upstream you could use `git -subtree pull` to incorporate those changes. - -## Building - -To build a container running the services, run `sbt docker:publishLocal` +> This will create new directories `certs` and `ota-ce-gen` the first time you run it and a device in `ota-ce-gen/devices/:uuid` where `uuid` is the id of the new device. +```bash +chmod +x scripts/gen-device.sh -## Configuration +./scripts/gen-device.sh +``` -Configuration is done through a single config file, rather than using enviroment variables. This is meant to simplify the deployment scripts and just pass a `configFile=file.conf` argument to the container, or when needed, using system properties (`-Dkey=value`). An example config file is provied in `ota-lith-ce.conf`. -## Running +### 6. Provision that device with aktualizr -If you already have kafka and mariadb instances you can just run the ota-lith binary using sbt or docker. +> Before you try to update your virtual device, make sure it connects with aktualizr +```bash +cd ota-ce-gen/devices/:uuid -### Using sbt +aktualizr --run-mode=once --config=config.toml +``` -You'll need a valid ota-lith.conf, then run: +### 7. Create credentials +> Credentials are needed to send updates to device. This will create a `credentials.zip` file in the `ota-ce-gen` directory +```bash +chmod +x scripts/get-credentials.sh - sbt -Dconfig.file=$(pwd)/ota-lith.conf run +./scripts/get-credentials.sh +``` -### Using docker +## Deploy updates -The scala apps run in a single container, but you'll need kafka and mariadb. Write a valid ota-lith.conf. +You can use [ota-cli](https://github.com/simao/ota-cli/) to deploy updates. After provisioning devices (see above). - sbt docker:publishLocal - docker run --name=ota-lith -v $(pwd)/ota-lith.conf:/tmp/ota-lith.conf uptane/ota-lith:latest -Dconfig.file=/tmp/ota-lith.conf - -If you don't have `sbt` or prefer to use a pre built image, you can use: +### 1. Install ota-cli +```bash +git clone https://github.com/simao/ota-cli.git - export img=uptane/ota-lith:$(git rev-parse master) - docker run --name=ota-lith -v $(pwd)/ota-lith.conf:/tmp/ota-lith.conf $img -Dconfig.file=/tmp/ota-lith.conf +cd ota-cli -## Running With Docker Compose +#run the following and follow the onscreen instructions +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -If you don't have kafka or mariadb running and just want to try ota-ce, run using docker-compose: +#Build the tool +make ota +``` -1. Generate the required certificates using `scripts/gen-server-certs.sh` +### 2. Initialize ota-cli +```bash +ota init --credentials /path/to/ota-ce-gen/credentials.zip --campaigner http://campaigner.ota.ce --director http://director.ota.ce --registry http://deviceregistry.ota.ce +``` -2. Update /etc/hosts with the following host names: +### 3. Add package to update +> Name a file with content in it that you would like to update to device as `update.bin` . It is not necessary to call it that, you can name it anything but for the sake of this tutorial name it `update.bin`. +```bash +# The output of this command is important to create the target file in the next step +ota package add -n mypkg -v 0.0.1 --path /path/to/update.bin --binary --hardware ota-ce-device +# You can regenerate the same output with this step +ota package list ``` -0.0.0.0 reposerver.ota.ce -0.0.0.0 keyserver.ota.ce -0.0.0.0 director.ota.ce -0.0.0.0 treehub.ota.ce -0.0.0.0 deviceregistry.ota.ce -0.0.0.0 campaigner.ota.ce -0.0.0.0 app.ota.ce -0.0.0.0 ota.ce + example output: +```json +{ + "signatures": [ + { + "keyid": "f62cfe66e8e986f4aac9161dcb231a9ccbff6db7b7907a4cd797f4db975d727c", + "method": "ed25519", + "sig": "dD2y2kX08uPsLLC9Uab8b7g0xdUqd3W/GUWc8dri6CE2OaA7rBKFO+M/GgZSIKXq8xzsO1tWP4POcei60NymAA==" + } + ], + "signed": { + "_type": "Targets", + "expires": "2022-10-12T14:06:56Z", + "targets": { + "mypkg-0.0.1": { + "custom": { + "createdAt": "2022-09-11T14:06:55Z", + "hardwareIds": [ + "ota-ce-device" + ], + "name": "mypkg", + "targetFormat": "BINARY", + "updatedAt": "2022-09-11T14:06:55Z", + "uri": null, + "version": "0.0.1" + }, + "hashes": { + "sha256": "70b22c2b8c857d1416a16e04cbfdf2b4d00d8b84000cc809af2061301ccfecb1" + }, + "length": 830 + } + }, + "version": 1 + } +} ``` -3. build docker image or pull from docker +### 4. Create target.toml file +> You have to use the name, version, length and hash using the json output recieved above and write a `target.toml` file. -`sbt docker:publishLocal` +For example the content of `target.toml` file wrt the above example output will be: +``` +[ota-ce-device] +target_format = "binary" +generate_diff = true + +# required metadata specifying update target +[ota-ce-device.to] +name = "mypkg" +version = "0.0.1" +length = 830 +hash = "70b22c2b8c857d1416a16e04cbfdf2b4d00d8b84000cc809af2061301ccfecb1" +method = "sha256" +``` +### 5. Create update +> The output of this command will be an `uuid` which you will use next to --update parameter in next step +```bash +ota update create -t /path/to/target.toml +``` +### 6. Serve the Update +> Using the uuid you get in the above command and the uuid of the device you want to update. +```bash +ota update launch --update --device +``` +> If it works , the o/p of the command will be `OK` or else it will give an error -Or: +### 7. Update the device +```bash +cd ota-ce-gen/devices/:uuid - export img=uptane/ota-lith:$(git rev-parse master) - docker pull $img - docker tag $img uptane/ota-lith:latest +aktualizr --run-mode=once --config=config.toml -4. Run docker-compose - -`docker-compose -f ota-ce.yaml up` +# you should find the update in the following path -5. Test +cd ota-ce-gen/devices/[uuid]/storage/images/ -For example `curl director.ota.ce/health/version` +``` -6. You can now create device credentials and provision devices -Run `scripts/gen-device.sh`. This will create a new dir in `ota-ce-gen/devices/:uuid` where `uuid` is the id of the new device. You can run `aktualizr` in that directory using: - aktualizr --run-mode=once --config=config.toml - -7. You can now deploy updates to the devices +To deploy an update using the API and a custom campaign, see [api-updates.md](docs/api-updates.md). -## Deploy updates +To deploy an update using [ota-cli](https://github.com/simao/ota-cli/) with a custom campaign see [updates-ota-cli.md](docs/updates-ota-cli.md). -You can either use the API directly or use [ota-cli](https://github.com/simao/ota-cli/) to deploy updates. After provisioning devices (see above). +## Dependency management + +This project follows the upstream [UPTANE OTA](https://uptane.github.io/) (see [sources](https://github.com/uptane/)) projects. -Before using the api or `ota-cli` you will need to generate a valid `credentials.zip`. Run `scripts/get-credentials.zip`. +The following forks/branches are included: -To deploy an update using the API and a custom campaign, see [api-updates.md](docs/api-updates.md). +- tuf https://github.com/uptane/ota-tuf +- director https://github.com/uptane/director +- device-registry https://github.com/uptane/ota-device-registry +- campaigner https://github.com/simao/campaigner +- treehub https://github.com/uptane/treehub +- libats https://github.com/uptane/libats -To deploy an update using [ota-cli](https://github.com/simao/ota-cli/) with or without a custom campaign see [updates-ota-cli.md](docs/updates-ota-cli.md). +The dependencies are managed using +[git-subtree](https://man.archlinux.org/man/git-subtree.1) under the +`repos` directory on this repository. -## FAQ +To update to the latest changes from upstream ota-tuf, you could use for example: -### Who maintains ota-lith + git subtree pull --prefix repos/ota-tuf git@github.com:uptane/ota-tuf.git master --squash -ota-lith is a collection of scripts and configurations that aggregates -upstream projects and makes it easier to run a complete OTA -solution. These scripts and configurations are maintained by -[simao](https://github.com/simao). +If you wish to make changes to ota-tuf, you could edit directly +`repos/ota-tuf` and then commit the changes, then use `git-subtree` to +split the changes and open a pull request upstream. However, a simpler +way would be to just use `git diff` to generate a patch and apply that +patch to the repository separately: -The actual OTA implementation is implemented and maintained by -multiple uptane contributors under the [uptane -organization](https://github.com/uptane/). + cd repos/ota-tuf + git diff . > tuf.patch + cd /home/user/my-ota-tuf + patch -p3 < tuf.patch + + # normal git flow to create a PR for ota-tuf + +And then once those changes are merged upstream you could use `git +subtree pull` to incorporate those changes. + -There are currently multiple contributors to the upstream UPTANE -repositories, most notably [toradex](https://toradex.com) which runs -the same uptane implementation as part of the [torizon -platform](https://app.torizon.io) +## FAQ ### How does it keep up with changes on the Uptane Standard? @@ -166,8 +262,7 @@ and creating the containers again. However, `webapp` and `campaigner` projects are not part of the UPTANE repositories and they are independently maintained by HERE Technologies -GmbH. These changes might or might not be merged back into `ota-lith`, -depending on the complexity of merging the changes. +GmbH. ### What is the relationship with Advanced Telematic OTA Community Edition? diff --git a/ota-ce.yaml b/ota-ce.yaml index 82caa5d..e98da46 100644 --- a/ota-ce.yaml +++ b/ota-ce.yaml @@ -4,25 +4,39 @@ # TODO: Kafka +#Using comments as markers to interact with ansible builtin block file modules to configure uptanedemo.org +#To setup https using letsencrypt in reverse proxy container we use the makers: REVERSE PROXY COMMAND, REVERSE PROXY CERTIFICATES, REVERSE PROXY PORT, REVERSE PROXY VOLUME +#To setup the landing page for uptanedemo.org using an nginx container we use the makerr: LANDING PAGE CONTAINER +#To setup the ota-lith services i.e. director, deviceregistry.. etc with reverse proxy we use the markers: OTA LITH LABELS & NEW OTA LITH LABELS + services: reverse-proxy: # The official v2 Traefik docker image image: traefik:v2.3 # Enables the web UI and tells Traefik to listen to docker +#BEGIN REVERSE PROXY COMMAND command: --api.insecure=true --providers.docker --providers.docker.exposedbydefault=false +#END REVERSE PROXY COMMAND +#BEGIN REVERSE PROXY CERTIFICATES +#END REVERSE PROXY CERTIFICATES ports: # The HTTP port - "80:80" +#BEGIN REVERSE PROXY PORT +#END REVERSE PROXY PORT # The Web UI (enabled by --api.insecure=true) - "8080:8080" volumes: +#BEGIN REVERSE PROXY VOLUME +#END REVERSE PROXY VOLUME # So that Traefik can listen to the Docker events - /var/run/docker.sock:/var/run/docker.sock - +#BEGIN LANDING PAGE CONTAINER +#END LANDING PAGE CONTAINER gateway: image: nginx:1.13.7 restart: always - command: ["nginx-debug", "-g", "daemon off;"] + command: ["nginx-debug", "-g", "daemon off;"] expose: - '80' - '443' @@ -67,6 +81,9 @@ services: command: - "/opt/ota-lith/bin/ota-lith" - "-Dconfig.file=/tmp/ota-lith.conf" +#BEGIN NEW OTA LITH LABELS +#END NEW OTA LITH LABELS +#BEGIN OTA LITH LABELS labels: - traefik.enable=true - traefik.http.routers.reposerver.service=reposerver @@ -87,10 +104,11 @@ services: - traefik.http.routers.campaigner.service=campaigner - traefik.http.routers.campaigner.rule=Host(`campaigner.ota.ce`) - traefik.http.services.campaigner.loadbalancer.server.port=7600 +#END OTA LITH LABELS volumes: - ./ota-lith-ce.conf:/tmp/ota-lith.conf - objects:/var/lib/ota-lith - + ota-lith-daemons: image: uptane/ota-lith:latest restart: always diff --git a/ota-ce/gateway.conf b/ota-ce/gateway.conf index 48f9f41..ba32d91 100644 --- a/ota-ce/gateway.conf +++ b/ota-ce/gateway.conf @@ -1,7 +1,11 @@ server { error_log /var/log/nginx/error.log info; listen 8443 ssl; +#BEGIN DGW NGINX CONF +#END DGW NGINX CONF +#BEGIN OTA CE NGINX CONF server_name ota.ce; +#END OTA CE NGINX CONF ssl_certificate /etc/ssl/gateway/server.chain.pem; ssl_certificate_key /etc/ssl/gateway/server.key; ssl_verify_client on; @@ -32,14 +36,17 @@ server { proxy_set_header x-ats-namespace $deviceNamespace; proxy_pass http://ota-lith:7600; } - +#BEGIN DIRECTOR NGINX CONF +#END DIRECTOR NGINX CONF +#BEGIN OTA CE DIRECTOR location /director/ { rewrite ^/director/(.*)$ /api/v1/device/${deviceUuid}/$1 break; proxy_set_header x-ats-namespace $deviceNamespace; proxy_set_header Host director.ota.ce; proxy_pass http://reverse-proxy; } - +#END OTA CE DIRECTOR + location /repo/ { rewrite ^/repo/(.*)$ /api/v1/user_repo/$1 break; proxy_set_header x-ats-namespace $deviceNamespace; diff --git a/scripts/gen-device.sh b/scripts/gen-device.sh index b1189dc..f4413c3 100755 --- a/scripts/gen-device.sh +++ b/scripts/gen-device.sh @@ -1,11 +1,20 @@ #!/bin/bash +#Using comments as markers to interact with ansible builtin block file modules to configure uptanedemo.org +#To change how ota-ce-gen and cert directories are configured when interacting with uptanedemo.org, we use the markers: CE DIR CONF and UPTANE DEMO DIR CONF +#To change how the script will interact with uptanedemo.org apis, we use the markers: UPTANE DEMO API CONF and CE API CONF + set -euo pipefail DEVICE_UUID=${DEVICE_UUID:-$(uuidgen | tr "[:upper:]" "[:lower:]")} CWD=$(dirname "$0") + +#BEGIN UPTANE DEMO DIR CONF +#END UPTANE DEMO DIR CONF +#BEGIN CE DIR CONF SERVER_DIR=$CWD/../ota-ce-gen DEVICES_DIR=${SERVER_DIR}/devices +#END CE DIR CONF device_id=$DEVICE_UUID device_dir="${DEVICES_DIR}/${DEVICE_UUID}" @@ -18,7 +27,7 @@ openssl pkcs8 -topk8 -nocrypt -in "${device_dir}/pkey.ec.pem" -out "${device_dir openssl req -new -key "${device_dir}/pkey.pem" \ -config <(sed "s/\$ENV::DEVICE_UUID/${DEVICE_UUID}/g" "${CWD}/certs/client.cnf") \ -out "${device_dir}/${device_id}.csr" - + openssl x509 -req -days 365 -extfile "${CWD}/certs/client.ext" -in "${device_dir}/${device_id}.csr" \ -CAkey "${DEVICES_DIR}/ca.key" -CA "${DEVICES_DIR}/ca.crt" -CAcreateserial -out "${device_dir}/client.pem" @@ -60,6 +69,11 @@ tls_clientcert_path = "client.pem" tls_pkey_path = "pkey.pem" EOF +#BEGIN UPTANE DEMO API CONF +#END UPTANE DEMO API CONF +#BEGIN CE API CONF curl -X PUT -d "${body}" http://deviceregistry.ota.ce/api/v1/devices -s -S -v -H "Content-Type: application/json" -H "Accept: application/json, */*" echo "https://ota.ce:30443" > ${device_dir}/gateway.url +#END CE API CONF + diff --git a/scripts/gen-server-certs.sh b/scripts/gen-server-certs.sh index a8200e3..d3ca2f5 100755 --- a/scripts/gen-server-certs.sh +++ b/scripts/gen-server-certs.sh @@ -1,11 +1,13 @@ #!/bin/bash +# To generate server certs for uptanedemo.org , run the script like follows: +#./gen-server-certs.sh uptanedemo.org set -euo pipefail SERVER_DIR=ota-ce-gen DEVICES_DIR=ota-ce-gen/devices CWD=$(dirname $0) -SERVER_NAME=ota.ce +SERVER_NAME=${1:-ota.ce} if [ -d "$SERVER_DIR" ] || [ -d "$DEVICES_DIR" ] ; then echo "${SERVER_DIR} or ${DEVICES_DIR} exists, aborting" diff --git a/scripts/get-credentials.sh b/scripts/get-credentials.sh index e825f44..38e72ff 100755 --- a/scripts/get-credentials.sh +++ b/scripts/get-credentials.sh @@ -1,17 +1,23 @@ #!/bin/bash +#Using comments as markers to interact with ansible builtin block file modules to configure uptanedemo.org +# To configure apis when interacting with uptanedemo.org we use the markers: HTTPS and HTTP +# ./get-credentials.sh https uptanedemo.org + set -euox pipefail SERVER_DIR=ota-ce-gen +SERVER_BASE_URI=${2:-ota.ce} + namespace="x-ats-namespace:default" -keyserver="keyserver.ota.ce" -reposerver="reposerver.ota.ce" -director="director.ota.ce" +keyserver="${1:-http}://keyserver.${SERVER_BASE_URI}" +reposerver="${1:-http}://reposerver.${SERVER_BASE_URI}" +director="${1:-http}://director.${SERVER_BASE_URI}" -curl --silent --fail ${director}/health || echo "$director not running" -curl --silent --fail ${keyserver}/health || echo "$keyserver not running" -curl --silent --fail ${reposerver}/health || echo "$reposerver not running" +curl --silent --fail ${director}/health/version || echo "$director not running" +curl --silent --fail ${keyserver}/health/version || echo "$keyserver not running" +curl --silent --fail ${reposerver}/health/version || echo "$reposerver not running" curl -X POST "${reposerver}/api/v1/user_repo" -H "${namespace}" @@ -29,16 +35,21 @@ keys=$(curl -s -f "${keyserver}/api/v1/root/${id}/keys/targets/pairs") echo ${keys} | jq '.[0] | {keytype, keyval: {public: .keyval.public}}' > "${SERVER_DIR}/targets.pub" echo ${keys} | jq '.[0] | {keytype, keyval: {private: .keyval.private}}' > "${SERVER_DIR}/targets.sec" -echo "http://reposerver.ota.ce" > "${SERVER_DIR}/tufrepo.url" -echo "http://ota.ce:30443" > "${SERVER_DIR}/autoprov.url" +#BEGIN HTTPS +#END HTTPS + +#BEGIN HTTP +echo "http://reposerver.${SERVER_BASE_URI}" > "${SERVER_DIR}/tufrepo.url" +echo "http://${SERVER_BASE_URI}:30443" > "${SERVER_DIR}/autoprov.url" cat > "${SERVER_DIR}/treehub.json" <