The Things Stack components are primarily built in Go, we use React for web front-ends. It is assumed that you have decent knowledge and experience with these technologies. If you want to get more familiar with Go, we strongly recommend to take A Tour of Go.
The Things Network's development tooling uses Mage. Under the hood, mage
calls other tools such as git
, go
, yarn
, docker
etc. Recent versions are supported; Node v12.x and Go v1.13.x.
- Follow Go's installation guide to install Go.
- Download Node.js from their website and install it.
- Follow Yarn's installation guide to install Yarn.
- Follow the guides to install Docker and to install Docker Compose.
If you are unfamiliar with forking projects on GitHub or cloning them locally, please see the GitHub documentation.
As most of the tasks will be managed by make
and mage
we will first initialize the tooling:
$ make init
You may want to run this commands from time to time.
Now you can initialize the development databases with some defaults.
$ ./mage dev:dbStart # This requires Docker to be running.
$ ./mage dev:initStack
This starts a CockroachDB and Redis database in Docker containers, creates a database, migrates tables and creates a user admin
with password admin
.
You can use the following commands to start, stop and erase databases.
$ ./mage dev:dbStart # Starts all databases in a Docker container.
$ ./mage dev:dbStop # Stops all databases.
# The contents of the databases will be saved in .env/data
$ ./mage dev:dbErase # Stops all databases and erase storage.
CockroachDB is a distributed SQL database that we use in the Identity Server.
You can use ./mage dev:dbSQL
to enter an SQL shell.
Redis is an in-memory data store that we use as a database for "hot" data.
You can use ./mage dev:dbRedisCli
to enter a Redis-CLI shell.
The folder structure of the project looks as follows:
.
├── .editorconfig configuration for your editor, see editorconfig.org
├── CODEOWNERS maintainers of folders who are required to approve pull requests
├── CONTRIBUTING.md guidelines for contributing: branching, commits, code style, etc.
├── DEVELOPMENT.md guide for setting up your development environment
├── docker-compose.yml deployment file (including databases) for Docker Compose
├── Dockerfile formula for building Docker images
├── LICENSE the license that explains what you're allowed to do with this code
├── Makefile dev/test/build tooling
├── .mage dev/test/build tooling
├── README.md general information about this project
│ ...
├── api contains the protocol buffer definitions for our API
├── cmd contains the different binaries that form The Things Stack for LoRaWAN
│ ├── internal contains internal files shared between the different binaries
│ │ ...
│ ├── ttn-lw-cli the command-line-interface for The Things Stack for LoRaWAN
│ └── ttn-lw-stack bundles the server binaries that form The Things Stack for LoRaWAN
├── config configuration for our JavaScript SDK and frontend
├── doc detailed documentation on the workings of The Things Stack for LoRaWAN
├── pkg contains all libraries used in The Things Stack for LoRaWAN
│ ├── component contains the base component; all other components extend this component
│ ├── config package for configuration using config files, environment and CLI flags
│ ├── console package that provides the web server for the console
│ ├── errors package for rich errors that include metadata and cross API boundaries
│ ├── log package for logging
│ ├── messages contains non-proto messages (such as the messages that are sent over MQTT)
│ ├── metrics package for metrics collection
│ ├── ttnpb contains generated code from our protocol buffer definitions and some helper functions
│ ├── types contains primitive types
│ ├── webui contains js code for the console and oauth provider
│ └── ...
├── public frontend code will be compiled to this folder - not added to git
├── release binaries will be compiled to this folder - not added to git
└── sdk source code for our SDKs
└── js source code for our JavaScript SDK
Our APIs are defined in .proto
files in the api
folder. These files describe the messages and interfaces of the different components of The Things Stack. If this is the first time you hear the term "protocol buffers" you should probably read the protocol buffers documentation before you continue.
From the .proto
files, we generate code using the protoc
compiler. As we plan to compile to a number of different languages, we decided to put the compiler and its dependencies in a Docker image. The actual commands for compilation are handled by our tooling, so the only thing you have to execute when updating the API, is:
$ ./mage proto:clean proto:all jsSDK:definitions
The documentation site for The Things Stack is built from the doc
folder.
All content is stored as Markdown files in doc/content
.
In order to build the documentation site with the right theme, you need to run
./mage docs:deps
from time to time.
You can start a development server with live reloading by running
./mage docs:server
. This command will print the address of the server.
The documentation site can be built by running ./mage docs:build
. This will
output the site to docs/public
.
For more details on how our documentation site is written, see the Hugo docs.
The Things Stack for LoRaWAN includes two frontend applications: the Console and OAuth Provider. Both applications use React as frontend framework. The console
and oauth
packages of the backend expose their respective web servers and handle all logic that cannot be done in the browser. Otherwise both applications are single page applications (SPA) that run entirely in the browser.
The folder structure of the frontend looks as follows:
./pkg/webui
├── assets assets (eg. vectors, images) used by the frontend
├── components react components shared throughout the frontend
├── console root of the console application
│ ├── api api definitions to communicate with The Things Stack
│ ├── containers container components
│ ├── lib utility classes and functions
│ ├── store redux actions, reducers and logic middlewares
│ ├── views whole view components of the console (~pages)
├── containers global react container components
├── lib global utility classes and functions
├── locales frontend and backend locale jsons used for i18n
├── oauth root of the oauth application
│ ├── api api definitions to communicate with The Things Stack
│ ├── store redux actions, reducers and logic middlewares
│ ├── views whole view components of the oauth provider (~pages)
├── styles global stylus (~css) styles and mixins
├── console.js entry point of the console app
├── oauth.js entry point of the oauth app
├── manifest.go generated manifest of the frontend, containing file hashes
├── template.go go template module used to render the frontend HTML
For development purposes, the frontend can be run using webpack-dev-server
. After following the Getting Started section to initialize The Things Stack and doing an initial build of the frontend via ./mage js:build
, it can be served using:
$ export NODE_ENV=development
$ ./mage js:serve
The development server runs on http://localhost:8080
and will proxy all api calls to port 1885
. The serve command watches any changes inside pkg/webui
and refreshes automatically.
In order to set up The Things Stack to support running the frontend via webpack-dev-server
, the following environment setup is needed:
NODE_ENV="development"
TTN_LW_LOG_LEVEL="debug"
TTN_LW_IS_OAUTH_UI_JS_FILE="libs.bundle.js oauth.js"
TTN_LW_CONSOLE_UI_JS_FILE="libs.bundle.js console.js"
TTN_LW_CONSOLE_UI_CANONICAL_URL="http://localhost:8080/console"
TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL="http://localhost:8080/oauth/authorize"
TTN_LW_CONSOLE_OAUTH_TOKEN_URL="http://localhost:8080/oauth/token"
TTN_LW_IS_OAUTH_UI_CANONICAL_URL="http://localhost:8080/oauth"
TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL="http://localhost:8080/oauth.js"
TTN_LW_CONSOLE_UI_ASSETS_BASE_URL="http://localhost:8080/assets"
We want our code to be consistent across our projects, so we'll have to agree on a number of formatting rules. These rules should usually usually be applied by your editor. Make sure to install the editorconfig plugin for your editor.
Our editorconfig contains the following rules:
- We use the utf-8 character set.
- We use LF line endings.
- We have a final newline in each file.
- We we trim whitespace at the end of each line (except in Markdown).
- All
.go
files are indented using tabs - The
Makefile
and all.make
files are indented using tabs - All other files are indented using two spaces
We don't have strict rules for line length, but in our experience the following guidelines result in nice and readable code:
- If a line is longer than 80 columns, try to find a "natural" break
- If a line is longer than 120 columns, insert a line break
- In very special cases, longer lines are tolerated
Go code can be automatically formatted using tools such as gofmt
and goimports
. The Go language server can also help with formatting code. There are many editor plugins that automatically format your code when you save your files. We highly recommend using those.
We use revive
to lint Go code and eslint
to lint JavaScript code. These tools should automatically be installed when initializing your development environment.
Please respect the following guidelines for content in our documentation site:
- The title of a doc page is already rendered by the build system as a h1, don't add an extra one.
- Use title case for headings.
- A documentation page starts with an introduction, and then the first heading. The first paragraph of the introduction is typically a summary of the page. Use a
<!--more-->
to indicate where the summary ends. - Since the title is a
h1
, everything in the content is at leasth2
(##
). - Paragraphs typically consist of at least two sentences.
- Use an empty line between all blocks (headings, paragraphs, lists, ...).
- Prefer text over bullet lists or enumerations. For bullets, use
-
, for enumerations1.
etc. - Explicitly call this product "The Things Stack". Not "the stack" etc. You can use the shortcode
{{% tts %}}
which will expand to "The Things Stack". - Avoid shortening, i.e. write "it is" instead of "it's".
- Write guides as a goal-oriented journey.
- Unless already clear from context, use a clearer term than "user", especially if there are multiple kinds (network operator, gateway owner, application developer, ...).
- The user does not have a gender (so use they/them/their).
- Taking screenshots is done as follows:
- In Chrome: activate the Developer Tools and toggle the Device Toolbar. In the Device Toolbar, select Laptop with HiDPI screen (add it if not already there), and click Capture Screenshot in the menu on the right.
- In Firefox: enter Responsive Design Mode. In the Device Toolbar, select "Laptop with HiDPI screen" (add it if not already there) and Take a screenshot of the viewport.
- Use
**Strong**
when referring to buttons in the Console - Use fenced code blocks with a language:
bash
for lists of environment variables:SOME_ENV="value"
.bash
for CLI examples. Prefix commands with$
. Wrap strings with double quotes""
(except when working with JSON, which already uses double quotes).- Wrap large CLI output with
<details><summary>Show CLI output</summary> ... output here ... </details>
. yaml
(notyml
) for YAML. Wrap strings with single quotes''
(because of frequent Go templates that use double quotes).
All API method names should follow the naming convention of VerbNoun
in upper camel case, where the verb uses the imperative mood and the noun is the resource type.
The following snippet defines the basic CRUD definitions for a resource named Type
.
Note also that the order of the methods is defined by CRUD.
CreateType
GetType
ListTypes (returns slice)
UpdateType
DeleteType
AddTypeAttribute
SetTypeAttribute
GetTypeAttribute
ListTypeAttributes (returns slice)
RemoveTypeAttribute
Variable names should be short and concise.
We follow the official go guidelines and try to be consistent with Go standard library as much as possible, everything not defined in the tables below should follow Go standard library naming scheme. In general, variable names are English and descriptive, omitting abbreviations as much as possible (except for the tables below), as well as putting adjectives and adverbs before the noun and verb respectively.
entity | name | example type |
---|---|---|
context | ctx | context.Context |
mutex | mu | sync.Mutex |
configuration | conf | go.thethings.network/lorawan-stack/pkg/config.Config |
logger | logger | go.thethings.network/lorawan-stack/pkg/log.Logger |
message | msg | go.thethings.network/lorawan-stack/api/gateway.UplinkMessage |
status | st | go.thethings.network/lorawan-stack/api/gateway.Status |
server | srv | go.thethings.network/lorawan-stack/pkg/network-server.Server |
ID | id | string |
unique ID | uid | string |
counter | cnt | int |
gateway | gtw | |
application | app | |
end device | dev | |
user | usr / user | |
transmit | tx / Tx | |
receive | rx / Rx |
The EUI naming scheme can be found in the well-known variable names section bellow.
In case both of the words have an implementation-specific meaning, the variable name is the combination of first letter of each word.
entity | name |
---|---|
wait group | wg |
Application Server | as |
Gateway Server | gs |
Identity Server | is |
Join Server | js |
Network Server | ns |
In case one of the words specifies the meaning of the variable in a specific language construct context, the variable name is the combination of abbrevations of the words.
These are the names of variables that occur often in the code. Be consistent in naming them, even when their meaning is obvious from the context.
entity | name |
---|---|
gateway ID | gtwID |
gateway EUI | gtwEUI |
application ID | appID |
application EUI | appEUI |
join EUI | joinEUI |
device ID | devID |
device EUI | devEUI |
user ID | usrID / userID |
Events are defined with
events.Define("event_name", "event description")
The event name is usually of the form component.entity.action
. Examples are ns.up.receive_duplicate
and is.user.update
. We have some exceptions, such as ns.up.join.forward
, which is specifically used for join messages.
The event description describes the event in simple English. The description is capitalized by the frontend, so the message should be lowercase, and typically doesn't end with a period. Event descriptions will be translated to different languages.
Errors are defined with
errors.Define<Type>("error_name", "description with `{attribute}`", "other", "public", "attributes")
Error definitions must be defined as close to the return statements as possible; in the same package, and preferably above the concerning function(s). Avoid exporting error definitions unless they are meaningful to other packages, i.e. for testing the exact error definition. Keep in mind that error definitions become part of the API.
Prefer using a specific error type, i.e. errors.DefineInvalidArgument()
. If you are using a cause (using WithCause()
), you may use Define()
to fallback to the cause's type.
The error name in snake case is a short and unique identifier of the error within the package. There is no need to append _failed
or _error
or prepend failed_to_
as an error already indicates something went wrong. Be consistent in wording (i.e. prefer the more descriptive missing_field
over no_field
), order (i.e. prefer the more clear missing_field
over field_missing
) and avoid entity abbreviations.
The error description in lower case, with only names in title case, is a concise plain English text that is human readable and understandable. Do not end the description with a period. You may use attributes, in snake case, in the description defined between backticks (`
) by putting the key in curly braces ({ }
). See below for naming conventions. Only provide primitive types as attribute values using WithAttributes()
. Error descriptions will be translated to different languages.
Any name
defined in the following statements:
- Logging field key:
logger.WithField("name", "value")
- Event name:
events.Define("name", "description")
- Error name:
errors.Define("name", "description")
- Error attribute:
errors.Define("example", "description `{name}`")
- Task identifier:
c.RegisterTask("name", ...)
Shall be snake case, optionally having an event name prepended with a dotted namespace, see above. The spacer _
shall be used in LoRaWAN terms: DevAddr
is dev_addr
, AppSKey
is app_s_key
, etc.
All comments should be English sentences, starting with a capital letter and ending with a period.
Every Go package should have a package comment. Every top-level Go type, const, var and func should have a comment. These comments are recognized by Go's tooling, and added to the Godoc for our code. See Effective Go for more details.
Although Go code should typically explain itself, it's sometimes important to add additional comments inside funcs to communicate what a block of code does, how a block of code does that, or why it's implemented that way.
Comments can also be used to indicate steps to take in the future (TODOs). Such comments look as follows:
// TODO: Open the pod bay doors (https://github.com/TheThingsNetwork/lorawan-stack/issues/<number>).
In our API definitions (.proto
files) we'd like to see short comments on every service, method, message and field. Code that is generated from these files does not have to comply with guidelines (such as Go's guideline for starting the comment with the name of the thing that is commented on).
We do our best to make all text that could be visible to users available for translation. This means that all text of the console's user interface, as well as all text that it may forward from the backend, needs to be defined in such a way that it can be translated into other languages than English.
In the API, the enum descriptions, error messages and event descriptions available for translation. Enum descriptions are defined in pkg/ttnpb/i18n.go
. Error messages and event descriptions are defined with errors.Define(...)
and events.Define(...)
respectively.
These messages are then collected in the config/messages.json
file, which will be processed in the frontend build process, but may also be used by other (native) user interfaces. When you define new enums, errors or events or when you change them, the messages need to be updated into the config/messages.json
file.
$ ./mage go:messages
If you forget to do so, this will cause a CI failure.
Adding translations of messages to other languages than English is a matter of adding key/value pairs to translations
in config/messages.json
.
The frontend uses react-intl
, which helps us greatly to define text messages used in the frontend.
The workflow for defining messages is as follows:
- Add a
react-intl
message usingintl.defineMessages({…})
- This can be done either inline, or by adding it to
pkg/webui/lib/shared-messages.js
- Use the message in components (e.g.
sharedMessage.myMessage
)
After adding messages this way, it needs to be added the locales file pkg/webui/locales/*.js
by using:
$ ./mage js:translations
Note: When using
./mage js:serve
, this command will be run automatically after any change.
The message definitions in pkg/webui/locales
can be used to provide translations in other languages (e.g. fr.js
). Keep in mind that locale files are checked in and committed, any discrepancy in the locales file with the defined messages will lead to a CI failure.
$ ./mage go:test js:test jsSDK:test
There is a single binary for the server, ttn-lw-stack
, as well as a binary for the command-line interface ttn-lw-cli
. The single binary contains all components start one or multiple components. This allows you to run The Things Stack with one command in simple deployment scenarios, as well as distributing micro-services for more advanced scenarios.
We provide binary releases for all supported platforms, including packages for various package managers at https://github.com/TheThingsNetwork/lorawan-stack/releases. We suggest you use the compiled packages we provide in production scenarios.
Before the binaries are built, the frontend needs to be built. You can control whether to build the frontend for production or development by setting the NODE_ENV
environment variable to either development
or production
.
The difference of a development build includes:
- Including source maps
- Using DLL bundle for modules to reduce build time
- A couple of special build options to improve usage with
webpack-dev-server
The frontend can then be built using:
$ ./mage js:build
For development/testing purposes we suggest to run the binaries directly via go run
:
$ go run ./cmd/ttn-lw-stack start
It is also possible to use go build
, or release snapshots, as described below.
You can build a release snapshot with go run github.com/goreleaser/goreleaser --snapshot
.
Note: You will at least need to have
rpm
andsnapcraft
in yourPATH
.
This will compile binaries for all supported platforms, deb
, rpm
and Snapcraft packages, release archives in dist
, as well as Docker images.
Note: The operating system and architecture represent the name of the directory in
dist
in which the binaries are placed. For example, the binaries for Darwin x64 (macOS) will be located atdist/darwin_amd64
.
Releasing a new version consists of the following steps:
- Creating a
release/<version>
branch(further, called "release branch") (e.g.release/3.2.1
). - Updating the
CHANGELOG.md
file:
- Change the Unreleased section to the new version and add date obtained via
date +%Y-%m-%d
(e.g.## [3.2.1] - 2019-10-11
) - Check if we didn't forget anything important
- Remove empty subsections
- Update the list of links in the bottom of the file
- Add new Unreleased section:
## [Unreleased] ### Added ### Changed ### Deprecated ### Removed ### Fixed ### Security
- Updating the
SECURITY.md
file with the supported versions - Bumping the version
- Writing the version files
- Creating the version bump commit
- Creating a pull request from release branch containing all changes made so far to
master
- Merging all commits from release branch to
master
locally viagit merge --ff-only release/<version>
- Creating the version tag
- Pushing the version tag
- Pushing
master
- Building the release and pushing to package managers (this is done by CI)
Our development tooling helps with this process. The mage
command has the following commands for version bumps:
$ ./mage version:bumpMajor # bumps a major version (from 3.4.5 -> 4.0.0).
$ ./mage version:bumpMinor # bumps a minor version (from 3.4.5 -> 3.5.0).
$ ./mage version:bumpPatch # bumps a patch version (from 3.4.5 -> 3.4.6).
$ ./mage version:bumpRC # bumps a release candidate version (from 3.4.5-rc1 -> 3.4.5-rc2).
$ ./mage version:bumpRelease # bumps a pre-release to a release version (from 3.4.5-rc1 -> 3.4.5).
These bumps can be combined (i.e. version:bumpMinor version:bumpRC
bumps 3.4.5 -> 3.5.0-rc1). Apart from these bump commands, we have commands for writing version files (version:files
), creating the bump commit (version:commitBump
) and the version tag (version:tag
).
A typical release process is executed directly on the master
branch and looks like this:
$ version=$(./mage version:bumpPatch version:current)
$ git checkout -b "release/${version}"
$ ${EDITOR:-vim} CHANGELOG.md SECURITY.md # edit CHANGELOG.md and SECURITY.md
$ git add CHANGELOG.md SECURITY.md
$ ./mage version:bumpPatch version:files version:commitBump
$ git push origin "release/${version}"
After this, open a pull request from release/${version}
. After it is approved:
$ git checkout master
$ git merge --ff-only "release/${version}"
$ ./mage version:bumpPatch version:tag
$ git push origin ${version}
$ git push origin master
After pushing the tag, our CI system will start building the release. When this is done, you'll find a new release on the releases page. After this is done, you'll need to edit the release notes. We typically copy-paste these from CHANGELOG.md
.
The Console will render a blank page and you can see backend logs like e.g.:
INFO Request handled duration=40.596µs error=error:pkg/errors/web:unknown (Not Found) message=Not Found method=GET namespace=web remote_addr=[::1]:50450 request_id=01DZ2CJDWKAFS10QD1NKZ1D56H status=404 url=/assets/console.36fcac90fa2408a19e4b.js
You might also see error messages in the Console such as:
Uncaught ReferenceError: libs_472226f4872c9448fc26 is not defined
at eval (eval at dll-reference libs_472226f4872c9448fc26 (console.js:26130), <anonymous>:1:18)
at Object.dll-reference libs_472226f4872c9448fc26 (console.js:26130)
at …
The stack has not been restarted after the Console bundle has changed. In production mode, The Things Stack will access the bundle via a filename that contains a content-hash, which is set during the build process of the Console. The hash cannot be updated during runtime and will take effect only after a restart.
- Restart the The Things Stack
The bundle files have been deleted. This might happen e.g. when a mage target encountered an error and quit before running through.
- Rebuild the Console
./mage js:clean js:build
- Restart the The Things Stack
If you switch between production and development builds of the Console, you might forget to re-run the build process and to restart The Things Stack. Likewise, you might have arbitrary config options set that are specific to a respective build type.
- Double check whether you have set the correct environment:
echo $NODE_ENV
, it should be eitherproduction
ordevelopment
- Double check whether your The Things Stack config is set correctly (especially
TTN_LW_CONSOLE_UI_JS_FILE
,TTN_LW_CONSOLE_UI_CANONICAL_URL
and similar settings) - Make sure to rebuild the Console
./mage js:clean js:build
- Restart The Things Stack
console.4e67a17c1ce5a74f3f50.js:104 Uncaught TypeError: m.subscribe is not a function
at Object../pkg/webui/console/api/index.js (console.4e67a17c1ce5a74f3f50.js:104)
at o (console.4e67a17c1ce5a74f3f50.js:1)
at Object../pkg/webui/console/store/middleware/logics/index.js (console.4e67a17c1ce5a74f3f50.js:104)
at o (console.4e67a17c1ce5a74f3f50.js:1)
at Object.<anonymous> (console.4e67a17c1ce5a74f3f50.js:104)
at Object../pkg/webui/console/store/index.js (console.4e67a17c1ce5a74f3f50.js:104)
at o (console.4e67a17c1ce5a74f3f50.js:1)
at Module../pkg/webui/console.js (console.4e67a17c1ce5a74f3f50.js:104)
at o (console.4e67a17c1ce5a74f3f50.js:1)
at Object.0 (console.4e67a17c1ce5a74f3f50.js:104)
The bundle integrates an old version of the JS SDK. This is likely a caching/linking issue of the JS SDK dependency.
- Re-establish a proper module link between the Console and the JS SDK
- Run
./mage js:cleanDeps js:deps
- Check whether the
ttn-lw
symlink exists insidenode_modules
and whether it points to the right destination:lorawan-stack/sdk/js/dist
- If you have cloned multiple
lorawan-stack
forks in different locations,yarn link
might associate the JS SDK module with the SDK on another ttn repository
- If you have cloned multiple
- Rebuild the Console and (only after the build has finished) restart The Things Stack
- Run
./mage
runs in silent mode by default. In verbose mode, you might get more helpful error messages
Run mage in verbose mode: ./mage -v {target}
A lot of problems during build stem from fragmented, incomplete runs of mage targets (due to arbitrary errors happening during a run). Oftentimes, it then helps to build the entire Web UI from scratch: ./mage jsSDK:cleanDeps jsSDK:clean js:cleanDeps js:clean js:build
, and (re-)start The Things Stack after running this.