From cdd5944f0346918d977f4e7c595f160ac2979024 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 14 Sep 2015 10:13:34 +0200 Subject: [PATCH 01/11] Update release documentation --- docs/git-repository.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/git-repository.md b/docs/git-repository.md index 2dfb2b656..d718cc1e9 100644 --- a/docs/git-repository.md +++ b/docs/git-repository.md @@ -81,15 +81,14 @@ git flow release start '1.4.0' ### Finish release ```shell -git flow release finish '1.4.0' - +git checkout master +git pull origin master git checkout develop -git push github develop --tags -git checkout master -git push github develop --tags +git flow release finish '1.4.0' git checkout develop +git push origin develop:develop master:master '1.4.0' ``` Go to Github repository /releases and add description of the release. From 30659c251b46b09f35c9b94f5a3ac080c8d8b215 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 14 Sep 2015 15:35:39 +0200 Subject: [PATCH 02/11] Update deployment documentation --- docs/deployment.md | 73 +++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 27 deletions(-) diff --git a/docs/deployment.md b/docs/deployment.md index 8b77351ed..f90ce0a6f 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -2,49 +2,66 @@ To deploy to production server: -1. Clone the repository master branch without history: +1. Stop the running containers + ```shell + docker exec osmaxx-starter /usr/local/bin/docker-compose stop + ``` + +2. Rename the old source and clone the repository master branch without history: ```shell git clone --depth 1 -b master https://github.com/geometalab/osmaxx.git osmaxx && cd osmaxx git submodule init && git submodule update ``` Repository details see [Project repository](git-repository.md). -2. Link production configuration for docker-compose, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). -3. Add target specific environment variables to compose-production.yml +3. Link production configuration for docker-compose, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). +4. Add target specific environment variables to compose-production.yml * If you are using an nginx proxy (jwilder/nginx-proxy), you need to set the environment variable `VIRTUAL_HOST=osmaxx.yourdomain.tld` -4. Build the containers. Use docker-compose container instead of native installation: +5. Build the containers. Use docker-compose container instead of native installation: - a. Create the docker-compose container: - ```shell - docker create \ + a. Create the docker-compose container if not already exists: + ```shell + # "up" && "--no-recreate": default arguments used by the systemd service starter + docker create \ --name osmaxx-starter -v "/path/to/source/repo:/app" \ -v "/var/run/docker.sock:/var/run/docker.sock" \ -e "COMPOSE_PROJECT_NAME=osmaxx" \ - "dduportal/docker-compose:1.2.0" - ``` - - b. Build osmaxx containers: - - ```shell - docker start osmaxx-starter build --no-cache - ``` - -5. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). + "dduportal/docker-compose:1.3.1" up --no-recreate + ``` + + b. Pull images & build osmaxx containers: + ```shell + # pull newest images + # "--rm" removes the temporary used compose-container + docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.3.1" pull + + # build own containers + docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.3.1" build + ``` + +6. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). For docker-compose container run: ```shell - docker exec -ti osmaxx-starter /usr/local/bin/docker-compose \ - run webapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" + # database needs to be running before the webapp container + docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.3.1" up -d database + + # apply migrations + # "-it" is needed to get an interactive terminal + # "run --rm" removes the temporary used webapp container + docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" -it --rm "dduportal/docker-compose:1.3.1" run --rm webapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" + + # stop running containers + docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.3.1" stop ``` -6. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). -7. Load database container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). -8. Add a system startup script running `docker-compose up` +7. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). +8. Load database container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). +9. Add a system startup script running `docker-compose up` E.g. /etc/systemd/system/osmaxx.service: ```shell - [Unit] Description=Start osmaxx application @@ -59,13 +76,15 @@ To deploy to production server: WantedBy=multi-user.target ``` **important**: osmax-starter container needs to be created before! - -9. Enable startup service + + "/usr/bin/docker start -a osmaxx-starter" will use the default arguments configured on create. + +1. Enable startup service ```shell sudo systemctl enable docker-osmaxx.service ``` - -10. Start the containers + +11. Start the containers ```shell sudo systemctl start docker-osmaxx.service ``` From 76577b9c995fc33a44fe93f8a19afa910e77a446 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 14 Sep 2015 17:51:17 +0200 Subject: [PATCH 03/11] Fix production configuration (missing socket) --- compose-production.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/compose-production.yml b/compose-production.yml index 9bbd8e3bb..9a519eac2 100755 --- a/compose-production.yml +++ b/compose-production.yml @@ -41,9 +41,11 @@ celery: extends: file: compose-common.yml service: base + command: honcho start -f Procfile.celery + volumes: + - /var/run/docker.sock:/var/run/docker.sock volumes_from: - shareddata - command: honcho start -f Procfile.celery links: - rabbitmq:rabbit - database From 6999694426ad19b1969ce54c2a0ac764c754d9de Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 08:31:58 +0200 Subject: [PATCH 04/11] Comment email shipping because it's not configured production ready yet --- osmaxx-py/excerptconverter/converter_helper.py | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 7ab3e2634..2a6255f32 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -1,5 +1,5 @@ from django.contrib import messages -from django.core.mail import send_mail +#from django.core.mail import send_mail from django.utils.translation import ugettext_lazy as _ import stored_messages @@ -35,12 +35,14 @@ def inform_user(self, message_type, message_text, email=True): if email: if hasattr(self.user, 'email'): - send_mail( - '[OSMAXX] '+message_text, - message_text, - 'no-reply@osmaxx.hsr.ch', - [self.user.email] - ) + pass + # TODO: fix it + # send_mail( + # '[OSMAXX] '+message_text, + # message_text, + # 'no-reply@osmaxx.hsr.ch', + # [self.user.email] + # ) else: self.inform_user( messages.WARNING, From 48aad94c9ce47792966de3ee2cb067a6e3a24417 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 08:34:10 +0200 Subject: [PATCH 05/11] Add production setup script --- setup.production.sh | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 setup.production.sh diff --git a/setup.production.sh b/setup.production.sh new file mode 100644 index 000000000..4054213a2 --- /dev/null +++ b/setup.production.sh @@ -0,0 +1,24 @@ +#!/bin/bash + +# Setup a docker compose container used to start the production application +# Setup application & database + +DOCKER_COMPOSE_TAG="1.3.1" + +# create container +docker create --name osmaxx-starter -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" up --no-recreate + +# stop containers & cleanup +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" stop +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" rm -f + +# pull & build +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" pull +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" build + +# migrate +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" up -d database +sleep 3 +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" -it --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" \ + run --rm webapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.3.1" stop From b795672a1b2849cf9376364aa9493f8c603ea109 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 10:13:06 +0200 Subject: [PATCH 06/11] Update production setup --- setup.production.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.production.sh b/setup.production.sh index 4054213a2..a51189c88 100644 --- a/setup.production.sh +++ b/setup.production.sh @@ -21,4 +21,4 @@ docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "C sleep 3 docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" -it --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" \ run --rm webapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" -docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.3.1" stop +docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" stop From 62c2c99c6c588069c552d6b490a6344ef37f6c9e Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 10:13:17 +0200 Subject: [PATCH 07/11] Add development setup --- setup.development.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 setup.development.sh diff --git a/setup.development.sh b/setup.development.sh new file mode 100644 index 000000000..1f63d740e --- /dev/null +++ b/setup.development.sh @@ -0,0 +1,18 @@ +#!/bin/bash + +# Setup a docker compose container used to start the development application +# Setup application & database + +# stop containers & cleanup +docker-compose stop +docker-compose rm -f + +# pull & build +docker-compose pull +docker-compose build + +# migrate +docker-compose up -d databasedev +sleep 3 +docker-compose run webappdev /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' + From 2c57f611389fda182206fe6e74089fcff3db3959 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 10:17:38 +0200 Subject: [PATCH 08/11] Update README --- README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b46c32680..7bc5bc8fb 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ To run this project locally, you need docker and docker-compose installed (https://docs.docker.com/installation/ubuntulinux/ and https://docs.docker.com/compose/install/). -### Initialization/Docker container bootstrapping +### Initialization ```shell # For development: @@ -38,7 +38,12 @@ ln -s compose-development.yml docker-compose.yml ln -s compose-production.yml docker-compose.yml ``` -To setup all the containers and their dependencies, run +### Docker container bootstrapping + +Take a look at the scripts ```setup.development.sh``` and ```setup.production.sh``. +These scripts will setup the container forest, run migrations and create a superuser (interactive). + +To setup all the containers and their dependencies by hand, run ```shell docker-compose build @@ -52,7 +57,7 @@ docker-compose up -d databasedev; docker-compose run webappdev /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' # For production: -docker-compose up -d databaseprod; +docker-compose up -d database; docker-compose run webapp /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' ``` From dff0603660c5692ec8ba835a6e74e2b0297abe65 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 10:21:49 +0200 Subject: [PATCH 09/11] Add hint for removal of data container --- setup.production.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.production.sh b/setup.production.sh index a51189c88..2de5ccdf5 100644 --- a/setup.production.sh +++ b/setup.production.sh @@ -10,6 +10,7 @@ docker create --name osmaxx-starter -v "$(pwd):/app" -v "/var/run/docker.sock:/ # stop containers & cleanup docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" stop +# TODO: data containers should not be removed! docker run -v "$(pwd):/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:$DOCKER_COMPOSE_TAG" rm -f # pull & build From 569ba1abff6878d234f6aa3ffd55503bb354a175 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 10:53:51 +0200 Subject: [PATCH 10/11] Remove temporary conctainers --- setup.development.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.development.sh b/setup.development.sh index 1f63d740e..8fefefbe8 100644 --- a/setup.development.sh +++ b/setup.development.sh @@ -14,5 +14,5 @@ docker-compose build # migrate docker-compose up -d databasedev sleep 3 -docker-compose run webappdev /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' +docker-compose run --rm webappdev /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' From 146c14779b2342c8d6188a5baa652caa2df29e3b Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Tue, 15 Sep 2015 10:54:59 +0200 Subject: [PATCH 11/11] Make setup scripts executable --- setup.development.sh | 0 setup.production.sh | 0 2 files changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 setup.development.sh mode change 100644 => 100755 setup.production.sh diff --git a/setup.development.sh b/setup.development.sh old mode 100644 new mode 100755 diff --git a/setup.production.sh b/setup.production.sh old mode 100644 new mode 100755