From 7d38b372c19fa00da646d97f446ac06bed3b376b Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Thu, 6 Aug 2015 13:16:14 +0200 Subject: [PATCH 01/94] Update deployment process --- docs/deployment.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/deployment.md b/docs/deployment.md index e4f15abbf..2b3deffc5 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -2,11 +2,18 @@ To deploy to production server: -1. Clone the repository, Details see [Project repository](git-repository.md). +1. Clone the repository master branch without history: + + ```shell + git clone --depth 1 -b master git@github.com:geometalab/osmaxx.git osmaxx && cd osmaxx + git submodule init && git submodule update + ``` + Repository details see [Project repository](git-repository.md). 2. Link production configuration for docker-compose, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). -3. Build the containers -4. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). -5. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). -6. Load database container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). -7. Add a system startup script running `docker-compose up` -8. Start the containers +3. Add target specific environment variables to compose-production.yml +4. Build the containers +5. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). +6. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). +7. Load database container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). +8. Add a system startup script running `docker-compose up` +9. Start the containers From 132ced7dfc9e05b50351aedab5007772b1556978 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Thu, 6 Aug 2015 13:59:17 +0200 Subject: [PATCH 02/94] Change git link to https url --- docs/deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment.md b/docs/deployment.md index 2b3deffc5..dc9c9c1a3 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -5,7 +5,7 @@ To deploy to production server: 1. Clone the repository master branch without history: ```shell - git clone --depth 1 -b master git@github.com:geometalab/osmaxx.git osmaxx && cd osmaxx + git clone --depth 1 -b master https://github.com/geometalab/osmaxx.git osmaxx && cd osmaxx git submodule init && git submodule update ``` Repository details see [Project repository](git-repository.md). From 70f05bdb064a643f74ebff91c306311e9326dbe7 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Thu, 6 Aug 2015 14:08:05 +0200 Subject: [PATCH 03/94] Add docker-compose container commands --- docs/deployment.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/deployment.md b/docs/deployment.md index dc9c9c1a3..c8eac260a 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -12,8 +12,42 @@ To deploy to production server: 2. Link production configuration for docker-compose, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). 3. Add target specific environment variables to compose-production.yml 4. Build the containers + If there is no docker-compose on the target system, use a docker-compose container: + + ```shell + docker run \ + -v "/path/to/source/repo:/app" \ + -v "/var/run/docker.sock:/var/run/docker.sock" \ + -e "COMPOSE_PROJECT_NAME=osmaxx" \ + --rm \ + "dduportal/docker-compose:1.2.0" build + ``` 5. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). + For docker-compose container run: + + ```shell + docker run \ + -v "/path/to/source/repo:/app" \ + -v "/var/run/docker.sock:/var/run/docker.sock" \ + -e "COMPOSE_PROJECT_NAME=osmaxx" \ + --rm -ti \ + "dduportal/docker-compose:1.2.0" run osmaxxwebapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" + ``` 6. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). 7. Load database container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). 8. Add a system startup script running `docker-compose up` + E.g. /etc/systemd/system/osmaxx.service: + + ```shell + [Unit] + Description=Start osmaxx application + + [Service] + ExecStart=docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" up -d + ExecStop=docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" stop + Restart=always + + [Install] + WantedBy=multi-user.target + ``` 9. Start the containers From 20e422d1eb0c46126fa7ada251f3d3057087e0ea Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 10 Aug 2015 09:39:49 +0200 Subject: [PATCH 04/94] Add --no-cache to build command --- docs/deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment.md b/docs/deployment.md index c8eac260a..4a644b868 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -20,7 +20,7 @@ To deploy to production server: -v "/var/run/docker.sock:/var/run/docker.sock" \ -e "COMPOSE_PROJECT_NAME=osmaxx" \ --rm \ - "dduportal/docker-compose:1.2.0" build + "dduportal/docker-compose:1.2.0" build --no-cache ``` 5. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). For docker-compose container run: From edc280e576a20ad8651eee0f3a93b2dbfa049252 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Mon, 10 Aug 2015 09:41:55 +0200 Subject: [PATCH 05/94] update whitenoise and social auth --- osmaxx-py/requirements/base.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/requirements/base.txt b/osmaxx-py/requirements/base.txt index 0952f990c..e4fce88b2 100644 --- a/osmaxx-py/requirements/base.txt +++ b/osmaxx-py/requirements/base.txt @@ -6,8 +6,8 @@ django-secure==1.0.1 django-stored-messages==1.3.0 psycopg2==2.6.1 -python-social-auth==0.2.11 -whitenoise==2.0.1 +python-social-auth==0.2.12 +whitenoise==2.0.2 # WSGI Handler # needed for production and docker images From 6e72da7dd2705194568605cf2baf1113f1feeda7 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 10 Aug 2015 09:42:04 +0200 Subject: [PATCH 06/94] Update build command for deployment --- docs/deployment.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/deployment.md b/docs/deployment.md index 4a644b868..3a43cd611 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -11,8 +11,13 @@ To deploy to production server: Repository details see [Project repository](git-repository.md). 2. Link production configuration for docker-compose, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). 3. Add target specific environment variables to compose-production.yml -4. Build the containers - If there is no docker-compose on the target system, use a docker-compose container: +4. Build the containers. + + ```shell + docker-compose build --no-cache + ``` + + If there is **no docker-compose installed** on the target system, use a docker-compose container: ```shell docker run \ From 84b3f68beb5eec79ed8c59f45f525097afa0ae28 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Mon, 10 Aug 2015 16:09:54 +0200 Subject: [PATCH 07/94] refactoring of tests --- test.sh | 128 +++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 94 insertions(+), 34 deletions(-) diff --git a/test.sh b/test.sh index 52692acb4..ac9f3cafe 100755 --- a/test.sh +++ b/test.sh @@ -7,52 +7,112 @@ GREEN="\e[32m" MAGENTA="\e[95m" RESET="\e[0m" +# This function will be called when running this script +function main(){ + # This function will be called when running this script + run_tests_in_development; + run_tests_in_production; +} -# development or production? -if [[ $(ls -l docker-compose.yml) == *"production.yml"* ]]; then +function run_tests_in_development() { + echo -e "${MAGENTA}" + echo -e "=== Development mode ===" + echo -e "${RESET}" + WEBAPP_CONTAINER="osmaxxwebappdev" + CELERY_CONTAINER="osmaxxcelerydev" + DB_CONTAINER="osmaxxdatabasedev" + COMPOSE_FILE="compose-development.yml" + run_tests; +} + +function run_tests_in_production() { echo -e "${MAGENTA}" echo -e "=== Production mode ===" echo -e "${RESET}" WEBAPP_CONTAINER="osmaxxwebapp" CELERY_CONTAINER="osmaxxcelery" -else - echo -e "${MAGENTA}=== Develpment mode ===" - echo -e "${RESET}" - WEBAPP_CONTAINER="osmaxxwebappdev" - CELERY_CONTAINER="osmaxxcelerydev" -fi + DB_CONTAINER="osmaxxdatabase" + COMPOSE_FILE="compose-production.yml" + run_tests; +} +function reset_containers() { + echo -e "resetting all containers"; + dcompose stop -t 0 &> test.log; + dcompose rm -f &> test.log; + dcompose build &> test.log; +} -# application tests -echo -e "${MAGENTA}-------------------" -echo -e "Application checks:" -echo -e "-------------------${RESET}" -docker-compose run $WEBAPP_CONTAINER /bin/bash -c "python3 manage.py check" +function reset_container() { + echo -e "resetting container ${CONTAINER_TO_BE_RESETTED};" + dcompose stop -t 0 ${CONTAINER_TO_BE_RESETTED} &> test.log; + dcompose rm -f ${CONTAINER_TO_BE_RESETTED} &> test.log; + dcompose build ${CONTAINER_TO_BE_RESETTED} &> test.log; +} +function dcompose() { + docker-compose -f ${COMPOSE_FILE} "${@}"; +} -echo -e "${MAGENTA}" -echo -e "------------------" -echo -e "Application tests:" -echo -e "------------------${RESET}" -docker-compose run $WEBAPP_CONTAINER /bin/bash -c "python3 manage.py test" +function run_tests() { + # start clean + reset_containers; + application_tests; + docker_volume_configuration_tests; + echo -e "cleaning up all containers" + reset_containers; +} + +function application_tests() { + # application tests + echo -e "${MAGENTA}-------------------" + echo -e "Application checks:" + echo -e "-------------------${RESET}" + dcompose run $WEBAPP_CONTAINER /bin/bash -c 'python3 manage.py check' &> test.log + if [ $? -eq 0 ]; then + echo -e "${GREEN}Checks passed successfully.${RESET}" + else + echo -e "${RED}Checks failed. Please have a look at the test.log!${RESET}" + fi + + echo -e "${MAGENTA}" + echo -e "------------------" + echo -e "Application tests:" + echo -e "------------------${RESET}" + dcompose run $WEBAPP_CONTAINER /bin/bash -c "python3 manage.py test" &> test.log + if [ $? -eq 0 ]; then + echo -e "${GREEN}Tests passed successfully.${RESET}" + else + echo -e "${RED}Tests failed. Please have a look at the test.log!${RESET}" + fi + +} + +function docker_volume_configuration_tests() { + # docker volume configuration tests + + echo -e "${MAGENTA}" + echo -e "-------------------------" + echo -e "Volume integration tests:" + echo -e "-------------------------${RESET}" -# docker volume configuration tests + dcompose run $CELERY_CONTAINER /bin/bash -c "touch $TEST_FILE" &> test.log + if [ $? -ne 0 ]; then + echo -e "${RED}Test file creation failed ${RESET}" + fi -echo -e "${MAGENTA}" -echo -e "-------------------------" -echo -e "Volume integration tests:" -echo -e "-------------------------${RESET}" -if [[ ! ${docker-compose run $CELERY_CONTAINER /bin/bash -c "touch $TEST_FILE"} ]]; then - echo -e "${RED}Test file creation failed ${RESET}" -fi + dcompose run $WEBAPP_CONTAINER /bin/bash -c "if [ ! -f $TEST_FILE ]; then exit 1; else exit 0; fi;" &> test.log + if [ $? -eq 0 ]; then + echo -e "${GREEN}Shared test file found: volume mount correct ${RESET}" + else + echo -e "${RED}Test file does not exist: volume mount incorrect ${RESET}" + fi -if [[ ${docker-compose run $WEBAPP_CONTAINER /bin/bash -c "if [ ! -f $TEST_FILE ]; then exit 1; else exit 0; fi;"} ]]; then - echo -e "${GREEN}Shared test file found: volume mount correct ${RESET}" -else - echo -e "${RED}Test file does not exist: volume mount incorrect ${RESET}" -fi + dcompose run $CELERY_CONTAINER /bin/bash -c "rm $TEST_FILE" &> test.log + if [ $? -ne 0 ]; then + echo -e "${RED}Test file clean up failed ${RESET}" + fi +} -if [[ ! ${docker-compose run $CELERY_CONTAINER /bin/bash -c "rm $TEST_FILE"} ]]; then - echo -e "${RED}Test file clean up failed ${RESET}" -fi +main; \ No newline at end of file From 5d09a4571e31c798e4d0a4af5974bdbeb282115e Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Mon, 10 Aug 2015 16:11:23 +0200 Subject: [PATCH 08/94] adding data persistence to database and removing networking from data container --- compose-development.yml | 7 +++++++ compose-production.yml | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/compose-development.yml b/compose-development.yml index 76011488a..96466251f 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -6,6 +6,9 @@ osmaxxshareddatadev: # the parent directory should be mounted, not the directories itself # otherwise we get OSError: Device or resource busy - /data + # database + - /database/data + net: "none" osmaxxsourcedev: build: docker/volume-mount @@ -61,6 +64,10 @@ osmaxxdatabasedev: extends: file: compose-common.yml service: osmaxxdatabasebase + volumes_from: + - osmaxxshareddatadev + environment: + - PGDATA=/database/data osmaxxemaildev: image: python:3 diff --git a/compose-production.yml b/compose-production.yml index 7361483e2..d5f093e25 100755 --- a/compose-production.yml +++ b/compose-production.yml @@ -3,6 +3,8 @@ osmaxxshareddata: command: /bin/true volumes: - /data + - /database/data + net: "none" # TODO: this file is only a placeholder for now! osmaxxwebapp: @@ -54,6 +56,10 @@ osmaxxdatabase: extends: file: compose-common.yml service: osmaxxdatabasebase + volumes_from: + - osmaxxshareddata + environment: + - PGDATA=/database/data osmaxxemail: image: catatnight/postfix From 2c49d69265b16b660dd3dfaa6ba0367af0a2cd4d Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Mon, 10 Aug 2015 16:11:45 +0200 Subject: [PATCH 09/94] test new setup --- test.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/test.sh b/test.sh index ac9f3cafe..c18da9ed5 100755 --- a/test.sh +++ b/test.sh @@ -60,6 +60,7 @@ function run_tests() { reset_containers; application_tests; docker_volume_configuration_tests; + persisting_database_data_tests; echo -e "cleaning up all containers" reset_containers; } @@ -115,4 +116,22 @@ function docker_volume_configuration_tests() { fi } +function persisting_database_data_tests() { + # clean containers needed for testing this... + reset_containers; + if dcompose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then + echo -e "${RED}Migrations could not be applied!${RESET}" + else + echo -e "${GREEN}Migrations applied successfully.${RESET}" + fi + + CONTAINER_TO_BE_RESETTED=${DB_CONTAINER} reset_container; + + if dcompose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then + echo -e "${GREEN}Database migrations retained correctly.${RESET}" + else + echo -e "${RED}Database migrations not retained, data only container not working correctly!${RESET}" + fi +} + main; \ No newline at end of file From 093cddb95fe13adb617a4c6b30bcf07a6486105e Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Mon, 10 Aug 2015 16:12:08 +0200 Subject: [PATCH 10/94] ignore log output from tests --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 46dececec..0cd7d5d3f 100644 --- a/.gitignore +++ b/.gitignore @@ -72,3 +72,4 @@ docker-compose.yml osmaxx-py/src/ # ignore environment file for local production environment .env +*.log From a76fd85bdb390c29c9a9f01c5c9ef4e0ba0b759f Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Mon, 10 Aug 2015 16:14:18 +0200 Subject: [PATCH 11/94] fix no newline --- test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test.sh b/test.sh index c18da9ed5..38652e8a1 100755 --- a/test.sh +++ b/test.sh @@ -134,4 +134,4 @@ function persisting_database_data_tests() { fi } -main; \ No newline at end of file +main; From ed68ddeffef6899358120780845ea527f730674d Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Tue, 11 Aug 2015 10:02:22 +0200 Subject: [PATCH 12/94] Add full path for docker call in service file --- docs/deployment.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deployment.md b/docs/deployment.md index 3a43cd611..79a900eb0 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -48,8 +48,8 @@ To deploy to production server: Description=Start osmaxx application [Service] - ExecStart=docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" up -d - ExecStop=docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" stop + ExecStart=/usr/bin/docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" up -d + ExecStop=/usr/bin/docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" stop Restart=always [Install] From c5c69bb82c80d08d01e981a48155181dd585d4de Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 11 Aug 2015 09:59:10 +0200 Subject: [PATCH 13/94] refactoring for tests, since they are run on the production machine as well --- test.sh | 103 ++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 66 insertions(+), 37 deletions(-) diff --git a/test.sh b/test.sh index 38652e8a1..d5f6555d4 100755 --- a/test.sh +++ b/test.sh @@ -7,81 +7,111 @@ GREEN="\e[32m" MAGENTA="\e[95m" RESET="\e[0m" -# This function will be called when running this script function main(){ # This function will be called when running this script - run_tests_in_development; - run_tests_in_production; + if [[ $(ls -l docker-compose.yml) == *"development.yml"* ]]; then + run_development_tests; + else + run_production_tests; + fi } -function run_tests_in_development() { +function run_development_tests() { echo -e "${MAGENTA}" echo -e "=== Development mode ===" echo -e "${RESET}" + WEBAPP_CONTAINER="osmaxxwebappdev" CELERY_CONTAINER="osmaxxcelerydev" DB_CONTAINER="osmaxxdatabasedev" COMPOSE_FILE="compose-development.yml" - run_tests; + + setup; + + application_checks; + application_tests; + + reset; + + docker_volume_configuration_tests; + + reset; + + persisting_database_data_tests; + + tear_down; } -function run_tests_in_production() { +function run_production_tests() { echo -e "${MAGENTA}" echo -e "=== Production mode ===" echo -e "${RESET}" + WEBAPP_CONTAINER="osmaxxwebapp" CELERY_CONTAINER="osmaxxcelery" DB_CONTAINER="osmaxxdatabase" COMPOSE_FILE="compose-production.yml" - run_tests; + + # this is run on the actual production machine as well, so we don't mess with the containers (setup/teardown) + docker_volume_configuration_tests; + application_checks; +} + +function setup() { + # does the same as reset and tear_down, but it makes the execution of the tests more readable. + reset_containers; +} + +function reset() { + reset_containers; +} + +function tear_down() { + reset_containers; } function reset_containers() { - echo -e "resetting all containers"; - dcompose stop -t 0 &> test.log; - dcompose rm -f &> test.log; - dcompose build &> test.log; + docker_compose stop -t 0 &> test.log; + docker_compose rm -f &> test.log; + docker_compose build &> test.log; } function reset_container() { - echo -e "resetting container ${CONTAINER_TO_BE_RESETTED};" - dcompose stop -t 0 ${CONTAINER_TO_BE_RESETTED} &> test.log; - dcompose rm -f ${CONTAINER_TO_BE_RESETTED} &> test.log; - dcompose build ${CONTAINER_TO_BE_RESETTED} &> test.log; + CONTAINER_TO_BE_RESETTED=$1 + docker_compose stop -t 0 ${CONTAINER_TO_BE_RESETTED} &> test.log; + docker_compose rm -f ${CONTAINER_TO_BE_RESETTED} &> test.log; + docker_compose build ${CONTAINER_TO_BE_RESETTED} &> test.log; } -function dcompose() { +function docker_compose() { docker-compose -f ${COMPOSE_FILE} "${@}"; } +#################### CONCRETE TEST IMPLEMENTATIONS #################### -function run_tests() { - # start clean - reset_containers; - application_tests; - docker_volume_configuration_tests; - persisting_database_data_tests; - echo -e "cleaning up all containers" - reset_containers; -} - -function application_tests() { +function application_checks() { # application tests echo -e "${MAGENTA}-------------------" echo -e "Application checks:" echo -e "-------------------${RESET}" - dcompose run $WEBAPP_CONTAINER /bin/bash -c 'python3 manage.py check' &> test.log + + docker_compose run $WEBAPP_CONTAINER /bin/bash -c 'python3 manage.py check' &> test.log + if [ $? -eq 0 ]; then echo -e "${GREEN}Checks passed successfully.${RESET}" else echo -e "${RED}Checks failed. Please have a look at the test.log!${RESET}" fi +} +function application_tests() { echo -e "${MAGENTA}" echo -e "------------------" echo -e "Application tests:" echo -e "------------------${RESET}" - dcompose run $WEBAPP_CONTAINER /bin/bash -c "python3 manage.py test" &> test.log + + docker_compose run $WEBAPP_CONTAINER /bin/bash -c "python3 manage.py test" &> test.log + if [ $? -eq 0 ]; then echo -e "${GREEN}Tests passed successfully.${RESET}" else @@ -98,36 +128,35 @@ function docker_volume_configuration_tests() { echo -e "Volume integration tests:" echo -e "-------------------------${RESET}" - dcompose run $CELERY_CONTAINER /bin/bash -c "touch $TEST_FILE" &> test.log + docker_compose run $CELERY_CONTAINER /bin/bash -c "touch $TEST_FILE" &> test.log if [ $? -ne 0 ]; then echo -e "${RED}Test file creation failed ${RESET}" fi - dcompose run $WEBAPP_CONTAINER /bin/bash -c "if [ ! -f $TEST_FILE ]; then exit 1; else exit 0; fi;" &> test.log + docker_compose run $WEBAPP_CONTAINER /bin/bash -c "if [ ! -f $TEST_FILE ]; then exit 1; else exit 0; fi;" &> test.log if [ $? -eq 0 ]; then echo -e "${GREEN}Shared test file found: volume mount correct ${RESET}" else echo -e "${RED}Test file does not exist: volume mount incorrect ${RESET}" fi - dcompose run $CELERY_CONTAINER /bin/bash -c "rm $TEST_FILE" &> test.log + docker_compose run $CELERY_CONTAINER /bin/bash -c "rm $TEST_FILE" &> test.log if [ $? -ne 0 ]; then echo -e "${RED}Test file clean up failed ${RESET}" fi } function persisting_database_data_tests() { - # clean containers needed for testing this... - reset_containers; - if dcompose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then + + if docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then echo -e "${RED}Migrations could not be applied!${RESET}" else echo -e "${GREEN}Migrations applied successfully.${RESET}" fi - CONTAINER_TO_BE_RESETTED=${DB_CONTAINER} reset_container; + reset_container ${DB_CONTAINER}; - if dcompose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then + if docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then echo -e "${GREEN}Database migrations retained correctly.${RESET}" else echo -e "${RED}Database migrations not retained, data only container not working correctly!${RESET}" From b0b5ad3312cd0dbd1a7da368664294f1ad6cd81c Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 11 Aug 2015 10:29:55 +0200 Subject: [PATCH 14/94] reomved redundant osmaxx from docker-compose.yml --- compose-common.yml | 10 ++++----- compose-development.yml | 48 ++++++++++++++++++++--------------------- compose-production.yml | 46 +++++++++++++++++++-------------------- test.sh | 8 +++---- 4 files changed, 56 insertions(+), 56 deletions(-) diff --git a/compose-common.yml b/compose-common.yml index e1c295c7f..4d9b5c596 100755 --- a/compose-common.yml +++ b/compose-common.yml @@ -1,7 +1,7 @@ -osmaxxdatabasebase: +databasebase: image: geometalab/postgis:9.4 -osmaxxbase: +base: build: . environment: # shared data @@ -9,8 +9,8 @@ osmaxxbase: - DJANGO_MEDIA_ROOT=/data/media - DJANGO_PRIVATE_MEDIA_ROOT=/data/private_media # --link some-rabbit:rabbit "just works" - - DJANGO_CELERY_BROKER_URL=amqp://guest@osmaxxrabbit - - DJANGO_DATABASE_URL=postgis://postgres@osmaxxdatabase/postgres + - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit + - DJANGO_DATABASE_URL=postgis://postgres@database/postgres - DJANGO_EMAIL_HOST=localhost # Settings above likely need to be changed, depending on the environment ########################################################################## @@ -41,5 +41,5 @@ osmaxxbase: - DJANGO_CSRF_COOKIE_HTTPONLY=true - DJANGO_X_FRAME_OPTIONS='DENY' -osmaxxrabbitmqbase: +rabbitmqbase: image: rabbitmq:3 diff --git a/compose-development.yml b/compose-development.yml index 76011488a..96e1504f0 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -1,4 +1,4 @@ -osmaxxshareddatadev: +shareddatadev: image: buildpack-deps:jessie command: /bin/true volumes: @@ -7,62 +7,62 @@ osmaxxshareddatadev: # otherwise we get OSError: Device or resource busy - /data -osmaxxsourcedev: +sourcedev: build: docker/volume-mount volumes: - osmaxx-py:/home/osmaxx/source - ~/.cache/pip:/home/osmaxx/.cache/pip -osmaxxwebappdev: +webappdev: extends: file: compose-common.yml - service: osmaxxbase + service: base command: honcho start -f Procfile.django.dev ports: - "8000:8000" expose: - "8000" volumes_from: - - osmaxxsourcedev - - osmaxxshareddatadev + - sourcedev + - shareddatadev links: - - osmaxxrabbitmqdev:rabbit - - osmaxxdatabasedev - - osmaxxemaildev - - osmaxxcelerydev + - rabbitmqdev:rabbit + - databasedev + - emaildev + - celerydev environment: - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - - DJANGO_DATABASE_URL=postgis://postgres@osmaxxdatabasedev/postgres + - DJANGO_DATABASE_URL=postgis://postgres@databasedev/postgres - DJANGO_DEBUG=true -osmaxxrabbitmqdev: +rabbitmqdev: extends: file: compose-common.yml - service: osmaxxrabbitmqbase + service: rabbitmqbase -osmaxxcelerydev: +celerydev: extends: file: compose-common.yml - service: osmaxxbase + service: base command: honcho start -f Procfile.celery volumes_from: - - osmaxxsourcedev - - osmaxxshareddatadev + - sourcedev + - shareddatadev links: - - osmaxxrabbitmqdev:rabbit - - osmaxxdatabasedev - - osmaxxemaildev + - rabbitmqdev:rabbit + - databasedev + - emaildev environment: - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - - DJANGO_DATABASE_URL=postgis://postgres@osmaxxdatabasedev/postgres + - DJANGO_DATABASE_URL=postgis://postgres@databasedev/postgres - DJANGO_DEBUG=true -osmaxxdatabasedev: +databasedev: extends: file: compose-common.yml - service: osmaxxdatabasebase + service: databasebase -osmaxxemaildev: +emaildev: image: python:3 ports: - 1025:1025 diff --git a/compose-production.yml b/compose-production.yml index 7361483e2..402368702 100755 --- a/compose-production.yml +++ b/compose-production.yml @@ -1,64 +1,64 @@ -osmaxxshareddata: +shareddata: image: buildpack-deps:jessie command: /bin/true volumes: - /data # TODO: this file is only a placeholder for now! -osmaxxwebapp: +webapp: extends: file: compose-common.yml - service: osmaxxbase + service: base ports: - "8000:8000" volumes_from: - - osmaxxshareddata + - shareddata command: honcho start -f Procfile.django links: - - osmaxxrabbitmq:rabbit - - osmaxxdatabase - - osmaxxemail - - osmaxxcelery + - rabbitmq:rabbit + - database + - email + - celery environment: - APP_PORT=8000 - APP_HOST=0.0.0.0 - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - - DJANGO_DATABASE_URL=postgis://postgres@osmaxxdatabase/postgres + - DJANGO_DATABASE_URL=postgis://postgres@database/postgres - DJANGO_EMAIL_USE_TLS=false - - DJANGO_EMAIL_HOST=osmaxxemail + - DJANGO_EMAIL_HOST=email - DJANGO_EMAIL_HOST_PASSWORD=osmaxx - DJANGO_EMAIL_HOST_USERNAME=osmaxx - DJANGO_EMAIL_PORT=25 -osmaxxrabbitmq: +rabbitmq: extends: file: compose-common.yml - service: osmaxxrabbitmqbase + service: rabbitmqbase -osmaxxcelery: +celery: extends: file: compose-common.yml - service: osmaxxbase + service: base volumes_from: - - osmaxxshareddata + - shareddata command: honcho start -f Procfile.celery links: - - osmaxxrabbitmq:rabbit - - osmaxxdatabase - - osmaxxemail + - rabbitmq:rabbit + - database + - email environment: - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - - DJANGO_DATABASE_URL=postgis://postgres@osmaxxdatabase/postgres + - DJANGO_DATABASE_URL=postgis://postgres@database/postgres -osmaxxdatabase: +database: extends: file: compose-common.yml - service: osmaxxdatabasebase + service: databasebase -osmaxxemail: +email: image: catatnight/postfix environment: - - maildomain=osmaxx.hsr.ch + - maildomain=.hsr.ch - smtp_user=osmaxx:osmaxx expose: - 25 diff --git a/test.sh b/test.sh index 52692acb4..78ee2420b 100755 --- a/test.sh +++ b/test.sh @@ -13,13 +13,13 @@ if [[ $(ls -l docker-compose.yml) == *"production.yml"* ]]; then echo -e "${MAGENTA}" echo -e "=== Production mode ===" echo -e "${RESET}" - WEBAPP_CONTAINER="osmaxxwebapp" - CELERY_CONTAINER="osmaxxcelery" + WEBAPP_CONTAINER="webapp" + CELERY_CONTAINER="celery" else echo -e "${MAGENTA}=== Develpment mode ===" echo -e "${RESET}" - WEBAPP_CONTAINER="osmaxxwebappdev" - CELERY_CONTAINER="osmaxxcelerydev" + WEBAPP_CONTAINER="webappdev" + CELERY_CONTAINER="celerydev" fi From da0c2ebf64062aa97b0cefabbd86d60b5e90e35c Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 11 Aug 2015 10:45:26 +0200 Subject: [PATCH 15/94] deleted oblivious comment --- compose-production.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/compose-production.yml b/compose-production.yml index 402368702..66ab5d4ee 100755 --- a/compose-production.yml +++ b/compose-production.yml @@ -4,7 +4,6 @@ shareddata: volumes: - /data -# TODO: this file is only a placeholder for now! webapp: extends: file: compose-common.yml From e6b49c49ab551d60eaa029e00466cec7aac3ac76 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Tue, 11 Aug 2015 12:18:57 +0200 Subject: [PATCH 16/94] Update deployment documentation --- docs/deployment.md | 48 +++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/docs/deployment.md b/docs/deployment.md index 79a900eb0..61fe11c33 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -11,32 +11,32 @@ To deploy to production server: Repository details see [Project repository](git-repository.md). 2. Link production configuration for docker-compose, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). 3. Add target specific environment variables to compose-production.yml -4. Build the containers. + * If you are using an nginx proxy (jwilder/nginx-proxy), you need to set the environment variable + `VIRTUAL_HOST=osmaxx.yourdomain.tld` +4. Build the containers. Use docker-compose container instead of native installation: + a. Create the docker-compose container: ```shell - docker-compose build --no-cache - ``` - - If there is **no docker-compose installed** on the target system, use a docker-compose container: - - ```shell - docker run \ + docker create \ + --name osmaxx-starter -v "/path/to/source/repo:/app" \ -v "/var/run/docker.sock:/var/run/docker.sock" \ -e "COMPOSE_PROJECT_NAME=osmaxx" \ - --rm \ - "dduportal/docker-compose:1.2.0" build --no-cache + "dduportal/docker-compose:1.2.0" + ``` + + b. Build osmaxx containers: + + ```shell + docker start osmaxx-starter build --no-cache ``` + 5. Run migrations and add create super user, Details see [Docker container bootstrapping](../README.md#initializationdocker-container-bootstrapping). For docker-compose container run: ```shell - docker run \ - -v "/path/to/source/repo:/app" \ - -v "/var/run/docker.sock:/var/run/docker.sock" \ - -e "COMPOSE_PROJECT_NAME=osmaxx" \ - --rm -ti \ - "dduportal/docker-compose:1.2.0" run osmaxxwebapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" + docker exec osmaxx-starter /usr/local/bin/docker-compose \ + run webapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" ``` 6. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). 7. Load database container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). @@ -44,15 +44,23 @@ To deploy to production server: E.g. /etc/systemd/system/osmaxx.service: ```shell + [Unit] Description=Start osmaxx application - + [Service] - ExecStart=/usr/bin/docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" up -d - ExecStop=/usr/bin/docker run -v "/path/to/source/repo:/app" -v "/var/run/docker.sock:/var/run/docker.sock" -e "COMPOSE_PROJECT_NAME=osmaxx" --rm "dduportal/docker-compose:1.2.0" stop + ExecStart=/usr/bin/docker start -a osmaxx-starter + ExecStop=/usr/bin/docker exec osmaxx-starter /usr/local/bin/docker-compose stop --timeout 60 + ExecStop=/usr/bin/docker kill --signal=INT osmaxx-starter + ExecStop=/usr/bin/docker stop -t 10 osmaxx-starter Restart=always - + [Install] WantedBy=multi-user.target ``` + **important**: osmax-starter container needs to be created before! + 9. Start the containers + ```shell + sudo systemctl start docker-osmaxx.service + ``` From 4fdbc295750198ee8a94e2ec54418dc21a816daf Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Wed, 12 Aug 2015 10:40:51 +0200 Subject: [PATCH 17/94] Fix deployment documentation migrations command --- docs/deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment.md b/docs/deployment.md index 61fe11c33..8d737bd1a 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -35,7 +35,7 @@ To deploy to production server: For docker-compose container run: ```shell - docker exec osmaxx-starter /usr/local/bin/docker-compose \ + docker exec -ti osmaxx-starter /usr/local/bin/docker-compose \ run webapp /bin/bash -c "python3 manage.py migrate && python3 manage.py createsuperuser" ``` 6. Load data container content from old container if there is one, Details see [Useful Docker commands](project-development-environment.md#useful-docker-commands). From 436af2cea05a8c7a8e7430d03de63442fdde3ef3 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 17 Aug 2015 10:50:43 +0200 Subject: [PATCH 18/94] Add system service enable command --- docs/deployment.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/deployment.md b/docs/deployment.md index 8d737bd1a..8b77351ed 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -60,7 +60,12 @@ To deploy to production server: ``` **important**: osmax-starter container needs to be created before! -9. Start the containers +9. Enable startup service + ```shell + sudo systemctl enable docker-osmaxx.service + ``` + +10. Start the containers ```shell sudo systemctl start docker-osmaxx.service ``` From 4d9f0ef61886a72907a630660952acac757c2acc Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 17 Aug 2015 13:55:58 +0200 Subject: [PATCH 19/94] Update container names in documentation --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 94d47f997..9507d9bcd 100644 --- a/README.md +++ b/README.md @@ -48,20 +48,20 @@ Then initiate the project defaults by running the following command: ```shell # For development: -docker-compose run osmaxxwebappdev /bin/bash -c 'python3 manage.py migrate && python3 manage.py createsuperuser' +docker-compose run webappdev /bin/bash -c 'python3 manage.py migrate && python3 manage.py createsuperuser' # For production: -docker-compose run osmaxxwebapp /bin/bash -c 'python3 manage.py migrate && python3 manage.py createsuperuser' +docker-compose run webapp /bin/bash -c 'python3 manage.py migrate && python3 manage.py createsuperuser' ``` Alternative to this command, bootstrap the container and execute the commands inside the container by hand: ```shell # For development: -docker-compose run osmaxxwebappdev /bin/bash +docker-compose run webappdev /bin/bash # For production: -docker-compose run osmaxxwebapp /bin/bash +docker-compose run webapp /bin/bash ``` Inside the container: From b05a292b59b28d43b34b14c34fcca289c94a61bd Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 17 Aug 2015 15:29:50 +0200 Subject: [PATCH 20/94] Ignore IE exception --- .../static/excerptexport/scripts/formModeSwitcher.js | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js index c1973db00..838ff1efb 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js @@ -26,8 +26,13 @@ formPartsSwitcher.value = formModeParam; } + // IE may throw an error here "Object doesn't support this action". + // It seems IE won't trigger a change event on select boxes + // I didn't found a solution to get the supported actions, + // so the error is still there: Fuck you IE + // Note: The switcher is working anyway formPartsSwitcher.dispatchEvent(new Event('change')); - } + }; window.addEventListener('load', function() { var formPartNodes = document.querySelectorAll('[data-form-part]'); @@ -47,6 +52,11 @@ if(formPartsSwitcher.value != formPartId) { formPartsSwitcher.value = formPartId; + // IE may throw an error here "Object doesn't support this action". + // It seems IE won't trigger a change event on select boxes + // I didn't found a solution to get the supported actions, + // so the error is still there: Fuck you IE + // Note: The switcher is working anyway formPartsSwitcher.dispatchEvent(new Event('change')); } }; From b91ca34a2ef7d9302ef1c1f99519529efb3d2bdc Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 17 Aug 2015 16:27:01 +0200 Subject: [PATCH 21/94] Improve visual appereance in IE --- .../excerptexport/stylesheets/ie-fixes.css | 18 ++++++++++++++++++ .../static/excerptexport/stylesheets/main.css | 2 -- .../templates/excerptexport/layouts/base.html | 1 + 3 files changed, 19 insertions(+), 2 deletions(-) create mode 100644 osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css new file mode 100644 index 000000000..91c0c4510 --- /dev/null +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css @@ -0,0 +1,18 @@ +@media all and (-ms-high-contrast: none), (-ms-high-contrast: active) { + /* readd IE default style to reatyle it isable for IE uesers + * IE does not support styling for select and option as buttons :-( + */ + select.btn-group, .select-button-group > select.btn-group { + font-size: 1.5rem !important; + display: block; + width: auto!important; + border: 1px solid rgb(33,33,33) !important; + } + select.btn-group > option:checked { + border-color: none; + background-color: rgb(51, 152, 254); + } + select.btn-group > option { + font-size: 1.5rem !important; + } +} diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/main.css b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/main.css index 7d478c194..49f7b952c 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/main.css +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/main.css @@ -140,9 +140,7 @@ optgroup:before { display:inline-block; border: none; background: none; - background: none; -moz-appearance: none; - -ms-appearance: none; -webkit-appearance: none; appearance: none; overflow: hidden; diff --git a/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/layouts/base.html b/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/layouts/base.html index 55942fa8d..8508d23fa 100644 --- a/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/layouts/base.html +++ b/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/layouts/base.html @@ -15,6 +15,7 @@ +
From 1982ab82f09d5d26cf7c562ca5e89084c3d49951 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 18 Aug 2015 13:41:49 +0200 Subject: [PATCH 22/94] enable docker-compose in Dockerfile --- Dockerfile | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 5ce087892..056e0e86a 100755 --- a/Dockerfile +++ b/Dockerfile @@ -1,9 +1,14 @@ FROM geometalab/python3-gis ENV USER osmaxx -# ENV USERID 1000 -# ENV GROUPID 1000 -# RUN groupadd -g $GROUPID $USER && useradd -g $USERID --create-home --home-dir /home/$USER -g $USER $USER + +ENV COMPOSE_VERSION 1.3.1 + +RUN DEBIAN_FRONTEND=noninteractive apt-get update -q \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y -q --no-install-recommends curl ca-certificates \ + && curl -o /usr/local/bin/docker-compose -L \ + "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-Linux-x86_64" \ + && chmod +x /usr/local/bin/docker-compose ENV HOME /home/$USER From 3401e40328de56b77c62c457c92c87334689dc73 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 18 Aug 2015 13:42:48 +0200 Subject: [PATCH 23/94] link docker into container which needs docker-compose --- compose-development.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/compose-development.yml b/compose-development.yml index bcac3cd90..3fe496ce3 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -48,6 +48,8 @@ celerydev: file: compose-common.yml service: base command: honcho start -f Procfile.celery + volumes: + - /var/run/docker.sock:/var/run/docker.sock volumes_from: - sourcedev - shareddatadev From 4ab92708800642237572efdb41819deccf977030 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 18 Aug 2015 13:44:22 +0200 Subject: [PATCH 24/94] start with the excerpt converter --- .../gis_excerpt_converter/__init__.py | 0 .../docker-compose-conversion-blackbox.yml | 18 ++++++++++++ .../execute_converting_blackbox.py | 28 +++++++++++++++++++ 3 files changed, 46 insertions(+) create mode 100644 osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py create mode 100644 osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml create mode 100644 osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml b/osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml new file mode 100644 index 000000000..306782ffa --- /dev/null +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml @@ -0,0 +1,18 @@ +bootstrap: + image: geometalab/osmaxx-postgis-conversion-bootstrap:latest + links: + - db + environment: + - PGHOST=db + volumes: + - tmp/osmosis/:/tmp/osmosis/ +excerpt: + image: geometalab/osmaxx-postgis-conversion-export:latest + links: + - db + environment: + - PGHOST=db + volumes: + - results:/home/excerpt/data/ +db: + image: geometalab/postgis-with-translit \ No newline at end of file diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py new file mode 100644 index 000000000..1a44e7449 --- /dev/null +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python +import os +import shutil +import subprocess +import tempfile + + +with tempfile.TemporaryDirectory() as tmp_dir: + original_curdir = os.path.curdir + + try: + print(tmp_dir) + shutil.copyfile( + os.path.join('blackbox', 'docker-compose-conversion-blackbox.yml'), + os.path.join(tmp_dir, 'docker-compose.yml') + ) + os.chdir(tmp_dir) + subprocess.check_call("docker-compose build", shell=True) + subprocess.check_call("docker-compose run bootstrap sleep 10", shell=True) + subprocess.check_call("docker-compose run bootstrap sh main-bootstrap.sh 8.775449276 47.1892350573 8.8901920319 47.2413633153", shell=True) + subprocess.check_call("docker-compose run excerpt python excerpt.py 8.775449276 47.1892350573 8.8901920319 47.2413633153 -f spatialite", shell=True) + subprocess.check_call("docker-compose stop --timeout 0", shell=True) + subprocess.check_call("docker-compose rm -f", shell=True) + except: + # inform_user() + raise + finally: + os.chdir(original_curdir) From 3832ff7ae9c413b4374480ba6dabafd528d2ba19 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 18 Aug 2015 14:59:18 +0200 Subject: [PATCH 25/94] document dcoker issues with docker-compose --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index 9507d9bcd..1eb901718 100644 --- a/README.md +++ b/README.md @@ -99,3 +99,17 @@ LINE 1: ..."django_site"."domain", "django_site"."name" FROM "django_si... ``` You forgot to **run the migrations** + + +#### Tests failed, but worked before with no apparent change + +Do not run `docker-compose build --no-cache`. Use `docker-compose rm -f && docker-compose build`, or +if you really want to start clean, remove all docker containers + +`docker rm -f $(docker ps -q)` + +and remove all images + +`docker rmi -f $(docker images -q)` + +*WARNING*: This removes all containers/images on the machine. \ No newline at end of file From a025d326abe300381cf3b60d8479b0c3e77d7c9d Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Tue, 18 Aug 2015 14:59:18 +0200 Subject: [PATCH 26/94] document dcoker issues with docker-compose --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index 9507d9bcd..5a6eec9d4 100644 --- a/README.md +++ b/README.md @@ -99,3 +99,17 @@ LINE 1: ..."django_site"."domain", "django_site"."name" FROM "django_si... ``` You forgot to **run the migrations** + + +#### Tests failed, but worked before with no apparent change + +Do not run `docker-compose build --no-cache`. Use `docker-compose rm -f && docker-compose build`, or +if you really want to start clean, remove all docker containers + +`docker rm -f $(docker ps -q)` + +and remove all images + +`docker rmi -f $(docker images -q)` + +*WARNING*: This removes all containers/images on the machine. From b60c7a9837be7e65952c8d92a0ac89598fff0157 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Tue, 18 Aug 2015 15:16:18 +0200 Subject: [PATCH 27/94] avoid shell invocation for command execution --- .../execute_converting_blackbox.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py index 1a44e7449..c46165954 100644 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py @@ -15,12 +15,12 @@ os.path.join(tmp_dir, 'docker-compose.yml') ) os.chdir(tmp_dir) - subprocess.check_call("docker-compose build", shell=True) - subprocess.check_call("docker-compose run bootstrap sleep 10", shell=True) - subprocess.check_call("docker-compose run bootstrap sh main-bootstrap.sh 8.775449276 47.1892350573 8.8901920319 47.2413633153", shell=True) - subprocess.check_call("docker-compose run excerpt python excerpt.py 8.775449276 47.1892350573 8.8901920319 47.2413633153 -f spatialite", shell=True) - subprocess.check_call("docker-compose stop --timeout 0", shell=True) - subprocess.check_call("docker-compose rm -f", shell=True) + subprocess.check_call("docker-compose build".split(' ')) + subprocess.check_call("docker-compose run bootstrap sleep 10".split(' ')) + subprocess.check_call("docker-compose run bootstrap sh main-bootstrap.sh 8.775449276 47.1892350573 8.8901920319 47.2413633153".split(' ')) + subprocess.check_call("docker-compose run excerpt python excerpt.py 8.775449276 47.1892350573 8.8901920319 47.2413633153 -f spatialite".split(' ')) + subprocess.check_call("docker-compose stop --timeout 0".split(' ')) + subprocess.check_call("docker-compose rm -f".split(' ')) except: # inform_user() raise From b60191c1bc80ddce8f02a968e046bb3d4926c519 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Tue, 18 Aug 2015 15:36:05 +0200 Subject: [PATCH 28/94] remember actual current dir --- .../gis_excerpt_converter/execute_converting_blackbox.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py index c46165954..2d3f16092 100644 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py @@ -6,7 +6,7 @@ with tempfile.TemporaryDirectory() as tmp_dir: - original_curdir = os.path.curdir + original_cwd = os.getcwd() try: print(tmp_dir) @@ -25,4 +25,4 @@ # inform_user() raise finally: - os.chdir(original_curdir) + os.chdir(original_cwd) From 6a5633f991b3ab7f10dc7c0dd67dd3835bbb7ff8 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Tue, 18 Aug 2015 16:46:56 +0200 Subject: [PATCH 29/94] extract bounding box arguments substring --- .../gis_excerpt_converter/execute_converting_blackbox.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py index 2d3f16092..7eb1964ff 100644 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py @@ -17,8 +17,9 @@ os.chdir(tmp_dir) subprocess.check_call("docker-compose build".split(' ')) subprocess.check_call("docker-compose run bootstrap sleep 10".split(' ')) - subprocess.check_call("docker-compose run bootstrap sh main-bootstrap.sh 8.775449276 47.1892350573 8.8901920319 47.2413633153".split(' ')) - subprocess.check_call("docker-compose run excerpt python excerpt.py 8.775449276 47.1892350573 8.8901920319 47.2413633153 -f spatialite".split(' ')) + bbox_args = '8.775449276 47.1892350573 8.8901920319 47.2413633153' + subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % bbox_args).split(' ')) + subprocess.check_call(("docker-compose run excerpt python excerpt.py %s" % bbox_args).split(' ')) subprocess.check_call("docker-compose stop --timeout 0".split(' ')) subprocess.check_call("docker-compose rm -f".split(' ')) except: From 95cd93897a1848078cc13365666b4d2f94a60ad7 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Wed, 19 Aug 2015 14:41:46 +0200 Subject: [PATCH 30/94] Bugfix: Resolve problem of inheritance and relation of BoundingGeometry --- .../migrations/0004_auto_20150819_1417.py | 19 +++++++++++++++++++ .../excerptexport/models/bounding_geometry.py | 16 +++++++++++++++- .../osmaxx/excerptexport/models/excerpt.py | 10 +++++++++- osmaxx-py/osmaxx/excerptexport/views.py | 4 ++-- 4 files changed, 45 insertions(+), 4 deletions(-) create mode 100644 osmaxx-py/osmaxx/excerptexport/migrations/0004_auto_20150819_1417.py diff --git a/osmaxx-py/osmaxx/excerptexport/migrations/0004_auto_20150819_1417.py b/osmaxx-py/osmaxx/excerptexport/migrations/0004_auto_20150819_1417.py new file mode 100644 index 000000000..3fa342c4d --- /dev/null +++ b/osmaxx-py/osmaxx/excerptexport/migrations/0004_auto_20150819_1417.py @@ -0,0 +1,19 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.db import models, migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('excerptexport', '0003_auto_20150729_1059'), + ] + + operations = [ + migrations.RenameField( + model_name='excerpt', + old_name='bounding_geometry', + new_name='_bounding_geometry', + ), + ] diff --git a/osmaxx-py/osmaxx/excerptexport/models/bounding_geometry.py b/osmaxx-py/osmaxx/excerptexport/models/bounding_geometry.py index d8b271066..7943188eb 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/bounding_geometry.py +++ b/osmaxx-py/osmaxx/excerptexport/models/bounding_geometry.py @@ -4,7 +4,18 @@ class BoundingGeometry(models.Model): - pass + @property + def type(self): + return type(self).__name__ + + @property + def geometry_instance(self): + if hasattr(self, 'bboxboundinggeometry'): + return self.bboxboundinggeometry + elif hasattr(self, 'osmosispolygonfilterboundinggeometry'): + return self.osmosispolygonfilterboundinggeometry + else: + return self class OsmosisPolygonFilterBoundingGeometry(BoundingGeometry): @@ -13,6 +24,9 @@ class OsmosisPolygonFilterBoundingGeometry(BoundingGeometry): """ polygon_file = models.FileField(storage=get_private_upload_storage()) + def __str__(self): + return 'Polygon file: ' + self.polygon_file + class BBoxBoundingGeometry(BoundingGeometry): south_west = models.PointField() diff --git a/osmaxx-py/osmaxx/excerptexport/models/excerpt.py b/osmaxx-py/osmaxx/excerptexport/models/excerpt.py index 79babfbe1..3e3573d5b 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/excerpt.py +++ b/osmaxx-py/osmaxx/excerptexport/models/excerpt.py @@ -9,7 +9,15 @@ class Excerpt(models.Model): is_active = models.BooleanField(default=True, verbose_name=_('is active')) owner = models.ForeignKey(User, related_name='excerpts', verbose_name=_('owner')) - bounding_geometry = models.OneToOneField('BoundingGeometry', verbose_name=_('bounding geometry')) + _bounding_geometry = models.OneToOneField('BoundingGeometry', verbose_name=_('bounding geometry')) + + @property + def bounding_geometry(self): + return self._bounding_geometry.geometry_instance + + @bounding_geometry.setter + def bounding_geometry(self, bounding_geometry): + self._bounding_geometry = bounding_geometry def __str__(self): return self.name diff --git a/osmaxx-py/osmaxx/excerptexport/views.py b/osmaxx-py/osmaxx/excerptexport/views.py index c2c561813..0c01f4fdc 100644 --- a/osmaxx-py/osmaxx/excerptexport/views.py +++ b/osmaxx-py/osmaxx/excerptexport/views.py @@ -29,9 +29,9 @@ class NewExtractionOrderView(LoginRequiredMixin, FrontendAccessRequiredMixin, View): def get(self, request, excerpt_form_initial_data=None): active_excerpts = Excerpt.objects.filter(is_active=True) - active_bbox_excerpts = active_excerpts.filter(bounding_geometry__bboxboundinggeometry__isnull=False) + active_bbox_excerpts = active_excerpts.filter(_bounding_geometry__bboxboundinggeometry__isnull=False) active_file_excerpts = active_excerpts.filter( - bounding_geometry__osmosispolygonfilterboundinggeometry__isnull=False) + _bounding_geometry__osmosispolygonfilterboundinggeometry__isnull=False) view_model = { 'user': request.user, 'export_options_form': ExportOptionsForm(ConverterManager.converter_configuration(), auto_id='%s'), From 9146bebbf124f3f7809d2874cb21e83faddb3bfd Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Wed, 19 Aug 2015 14:42:25 +0200 Subject: [PATCH 31/94] Add bounding box coordinated to list items --- .../excerptexport/partials/excerpt_option.html | 14 ++++++++++++++ .../templates/new_excerpt_export.html | 8 ++++---- 2 files changed, 18 insertions(+), 4 deletions(-) create mode 100644 osmaxx-py/osmaxx/excerptexport/templates/excerptexport/partials/excerpt_option.html diff --git a/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/partials/excerpt_option.html b/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/partials/excerpt_option.html new file mode 100644 index 000000000..eb82c264f --- /dev/null +++ b/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/partials/excerpt_option.html @@ -0,0 +1,14 @@ +{% if excerpt.bounding_geometry.type == 'BBoxBoundingGeometry' %} + +{% else %} + +{% endif %} diff --git a/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/templates/new_excerpt_export.html b/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/templates/new_excerpt_export.html index 73f02803f..dfd272785 100644 --- a/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/templates/new_excerpt_export.html +++ b/osmaxx-py/osmaxx/excerptexport/templates/excerptexport/templates/new_excerpt_export.html @@ -33,22 +33,22 @@

{% trans 'Existing excerpt' %}

From b2f3e835cef0742370fc8973fb41b2f33061deb4 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Wed, 19 Aug 2015 16:48:29 +0200 Subject: [PATCH 32/94] Update excerpt on map on selection of existing excerpt in list --- .../static/excerptexport/scripts/map.js | 40 ++++++++++++++++++- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 0090b80f3..bda9df4bc 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -13,6 +13,9 @@ var inputElementWest = document.getElementById('new_excerpt_bounding_box_west'); var inputElementEast = document.getElementById('new_excerpt_bounding_box_east'); var inputElementSouth = document.getElementById('new_excerpt_bounding_box_south'); +var selectExistingExcerpts = document.getElementById('existing_excerpt.id'); +var formPartsSwitcher = document.getElementById('form-mode'); + var locationFilter = new L.LocationFilter({ enable: true, @@ -26,15 +29,28 @@ function updateBboxTextInputsWith(bounds) { inputElementSouth.value = bounds._southWest.lat; } +/** + * if the shown excerpt on the map changes and the change was not made by selecting an existing excerpt in the list -> a user changed the excerpt, so change the form part the + */ +function showNewExcerptPart(locationFilterBounds) { + if(!(selectExistingExcerpts.value != "" && + selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-north') == locationFilterBounds._northEast.lat && + selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-west') == locationFilterBounds._southWest.lng && + selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-east') == locationFilterBounds._northEast.lng && + selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-south') == locationFilterBounds._southWest.lat)) { + formPartsSwitcher.value = 'new-excerpt'; + formPartsSwitcher.dispatchEvent(new Event('change')); + } +} + // push initial values to text inputs updateBboxTextInputsWith(locationFilter.getBounds()); // update text input values upon change on map locationFilter.on("change", function (e) { - console.log(locationFilter); - console.log(locationFilter.getBounds()); updateBboxTextInputsWith(e.bounds); + showNewExcerptPart(e.bounds); }); @@ -54,3 +70,23 @@ inputElementNorth.addEventListener('change', updateMapBoundingBox); inputElementWest.addEventListener('change', updateMapBoundingBox); inputElementEast.addEventListener('change', updateMapBoundingBox); inputElementSouth.addEventListener('change', updateMapBoundingBox); + +selectExistingExcerpts.addEventListener('change', function(event) { + var excerptOption = event.explicitOriginalTarget; + if(excerptOption.getAttribute('data-geometry') == 'boundingbox') { + var bounds = locationFilter.getBounds(); + bounds._northEast.lat = excerptOption.getAttribute('data-north'); + bounds._southWest.lng = excerptOption.getAttribute('data-west'); + bounds._northEast.lng = excerptOption.getAttribute('data-east'); + bounds._southWest.lat = excerptOption.getAttribute('data-south'); + locationFilter.setBounds(bounds); + locationFilter.enable(); + } else { + locationFilter.disable(); + } +}); + +formPartsSwitcher.addEventListener('change', function(event) { + locationFilter.enable(); +}); + From a8b1f2f7aec242be7121783e09f5dd1d87087f19 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Wed, 19 Aug 2015 18:10:34 +0200 Subject: [PATCH 33/94] Refactoring: Clean up the map script --- .../static/excerptexport/scripts/map.js | 203 ++++++++++-------- 1 file changed, 111 insertions(+), 92 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index bda9df4bc..552d00aa4 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -1,92 +1,111 @@ -/** - * @author Raphael Das Gupta - */ -var map = L.map('map').setView([0, 0], 2); -// add an OpenStreetMap tile layer -L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', { - attribution: '© OpenStreetMap contributors' -}).addTo(map); - - -var inputElementNorth = document.getElementById('new_excerpt_bounding_box_north'); -var inputElementWest = document.getElementById('new_excerpt_bounding_box_west'); -var inputElementEast = document.getElementById('new_excerpt_bounding_box_east'); -var inputElementSouth = document.getElementById('new_excerpt_bounding_box_south'); - -var selectExistingExcerpts = document.getElementById('existing_excerpt.id'); -var formPartsSwitcher = document.getElementById('form-mode'); - - -var locationFilter = new L.LocationFilter({ - enable: true, - enableButton: false -}).addTo(map); - -function updateBboxTextInputsWith(bounds) { - inputElementNorth.value = bounds._northEast.lat; - inputElementWest.value = bounds._southWest.lng; - inputElementEast.value = bounds._northEast.lng; - inputElementSouth.value = bounds._southWest.lat; -} - -/** - * if the shown excerpt on the map changes and the change was not made by selecting an existing excerpt in the list -> a user changed the excerpt, so change the form part the - */ -function showNewExcerptPart(locationFilterBounds) { - if(!(selectExistingExcerpts.value != "" && - selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-north') == locationFilterBounds._northEast.lat && - selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-west') == locationFilterBounds._southWest.lng && - selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-east') == locationFilterBounds._northEast.lng && - selectExistingExcerpts.querySelector('option[value="'+selectExistingExcerpts.value+'"]').getAttribute('data-south') == locationFilterBounds._southWest.lat)) { - formPartsSwitcher.value = 'new-excerpt'; - formPartsSwitcher.dispatchEvent(new Event('change')); - } -} - - -// push initial values to text inputs -updateBboxTextInputsWith(locationFilter.getBounds()); - -// update text input values upon change on map -locationFilter.on("change", function (e) { - updateBboxTextInputsWith(e.bounds); - showNewExcerptPart(e.bounds); -}); - - -/** - * update map on coordinate input field change - */ -function updateMapBoundingBox() { - var bounds = locationFilter.getBounds(); - bounds._northEast.lat = inputElementNorth.value; - bounds._southWest.lng = inputElementWest.value; - bounds._northEast.lng = inputElementEast.value; - bounds._southWest.lat = inputElementSouth.value; - locationFilter.setBounds(bounds); -} - -inputElementNorth.addEventListener('change', updateMapBoundingBox); -inputElementWest.addEventListener('change', updateMapBoundingBox); -inputElementEast.addEventListener('change', updateMapBoundingBox); -inputElementSouth.addEventListener('change', updateMapBoundingBox); - -selectExistingExcerpts.addEventListener('change', function(event) { - var excerptOption = event.explicitOriginalTarget; - if(excerptOption.getAttribute('data-geometry') == 'boundingbox') { - var bounds = locationFilter.getBounds(); - bounds._northEast.lat = excerptOption.getAttribute('data-north'); - bounds._southWest.lng = excerptOption.getAttribute('data-west'); - bounds._northEast.lng = excerptOption.getAttribute('data-east'); - bounds._southWest.lat = excerptOption.getAttribute('data-south'); - locationFilter.setBounds(bounds); - locationFilter.enable(); - } else { - locationFilter.disable(); - } -}); - -formPartsSwitcher.addEventListener('change', function(event) { - locationFilter.enable(); -}); - +(function(){ + var ExcerptManager = function(locationFilter, inputElementsNewBoundingBox, selectElementExistingExcerpts, formElementPartsSwitcher) { + this.locationFilter = locationFilter; + this.inputElementsNewBoundingBox = inputElementsNewBoundingBox; + this.selectElementExistingExcerpts = selectElementExistingExcerpts; + this.formElementPartsSwitcher = formElementPartsSwitcher; + + + /** + * Synchronize coordinates in input fields to excerpt on map + */ + this.updateInputElementsBoundingBox = function() { + var locationFilterBounds = this.locationFilter.getBounds(); + this.inputElementsNewBoundingBox.inputElementNorth.value = locationFilterBounds._northEast.lat; + this.inputElementsNewBoundingBox.inputElementWest.value = locationFilterBounds._southWest.lng; + this.inputElementsNewBoundingBox.inputElementEast.value = locationFilterBounds._northEast.lng; + this.inputElementsNewBoundingBox.inputElementSouth.value = locationFilterBounds._southWest.lat; + }; + + /** + * Synchronize excerpt on map to coordinates in input fields + */ + this.updateMapExcerpt = function() { + var locationFilterBounds = this.locationFilter.getBounds(); + locationFilterBounds._northEast.lat = this.inputElementsNewBoundingBox.inputElementNorth.value; + locationFilterBounds._southWest.lng = this.inputElementsNewBoundingBox.inputElementWest.value; + locationFilterBounds._northEast.lng = this.inputElementsNewBoundingBox.inputElementEast.value; + locationFilterBounds._southWest.lat = this.inputElementsNewBoundingBox.inputElementSouth.value; + this.locationFilter.setBounds(locationFilterBounds); + }; + + /** + * if the shown excerpt on the map changes and the change was not made by selecting an existing excerpt in the list + * -> a user changed the excerpt, so change the form part to 'new-excerpt' + */ + this.userChangeExcerptOnMapShowNewExcerptPart = function() { + var locationFilterBounds = this.locationFilter.getBounds(); + var select = this.selectElementExistingExcerpts; + if(!(selectElementExistingExcerpts.value != "" && + select.querySelector('option[value="'+select.value+'"]').getAttribute('data-north') == locationFilterBounds._northEast.lat && + select.querySelector('option[value="'+select.value+'"]').getAttribute('data-west') == locationFilterBounds._southWest.lng && + select.querySelector('option[value="'+select.value+'"]').getAttribute('data-east') == locationFilterBounds._northEast.lng && + select.querySelector('option[value="'+select.value+'"]').getAttribute('data-south') == locationFilterBounds._southWest.lat)) { + this.formElementPartsSwitcher.value = 'new-excerpt'; + this.formElementPartsSwitcher.dispatchEvent(new Event('change')); + } + }; + + + // update coordinates input elements on change of excerpt on map + this.locationFilter.on("change", function (event) { + this.updateInputElementsBoundingBox(); + this.userChangeExcerptOnMapShowNewExcerptPart(); + }.bind(this)); + + // update excerpt on map on change of coordinates input elements + Object.keys(this.inputElementsNewBoundingBox).forEach(function(inputElementKey) { + this.inputElementsNewBoundingBox[inputElementKey].addEventListener('change', this.updateMapExcerpt.bind(this)); + }.bind(this)); + + // update excerpt on map on selection of existing excerpt in list + this.selectElementExistingExcerpts.addEventListener('change', function(event) { + var excerptOption = event.explicitOriginalTarget; + if(excerptOption.getAttribute('data-geometry') == 'boundingbox') { + var bounds = this.locationFilter.getBounds(); + bounds._northEast.lat = excerptOption.getAttribute('data-north'); + bounds._southWest.lng = excerptOption.getAttribute('data-west'); + bounds._northEast.lng = excerptOption.getAttribute('data-east'); + bounds._southWest.lat = excerptOption.getAttribute('data-south'); + this.locationFilter.setBounds(bounds); + // enable excerpt on map for case it was disable before (e.g. by clicking on a not-boundingbox excerpt in the list) + this.locationFilter.enable(); + } else { + // no existing excerpt of type boundingbox -> disable excerpt because we are not able to show not-boundingbox excerpts on map + this.locationFilter.disable(); + } + }.bind(this)); + + // enable excerpt on map on change of form mode (existing excerpt or new excerpt) + this.formElementPartsSwitcher.addEventListener('change', function(event) { + this.locationFilter.enable(); + }.bind(this)); + }; + + + var map = map = L.map('map').setView([0, 0], 2); + // add an OpenStreetMap tile layer + L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', { + attribution: '© OpenStreetMap contributors' + }).addTo(map); + + var locationFilter = new L.LocationFilter({ + enable: true, + enableButton: false + }).addTo(map); + + var excerptManager = new ExcerptManager( + locationFilter, + { + inputElementNorth: document.getElementById('new_excerpt_bounding_box_north'), + inputElementWest: document.getElementById('new_excerpt_bounding_box_west'), + inputElementEast: document.getElementById('new_excerpt_bounding_box_east'), + inputElementSouth: document.getElementById('new_excerpt_bounding_box_south') + }, + document.getElementById('existing_excerpt.id'), + document.getElementById('form-mode') + ); + + // push initial values to text inputs + excerptManager.updateInputElementsBoundingBox(); +})(); From 59f53c14ad98ae11c8f9e552a6d127bd0163ce29 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Thu, 20 Aug 2015 09:04:19 +0200 Subject: [PATCH 34/94] --wip-- --- Dockerfile | 2 +- .../excerptconverter/gis_excerpt_converter/__init__.py | 1 + .../gis_excerpt_converter/execute_converting_blackbox.py | 7 +++++-- .../tests/test_postgis_excerpt_converter.py | 7 +++++++ 4 files changed, 14 insertions(+), 3 deletions(-) create mode 100644 osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py diff --git a/Dockerfile b/Dockerfile index 056e0e86a..27cd425e0 100755 --- a/Dockerfile +++ b/Dockerfile @@ -2,7 +2,7 @@ FROM geometalab/python3-gis ENV USER osmaxx -ENV COMPOSE_VERSION 1.3.1 +ENV COMPOSE_VERSION 1.3.3 RUN DEBIAN_FRONTEND=noninteractive apt-get update -q \ && DEBIAN_FRONTEND=noninteractive apt-get install -y -q --no-install-recommends curl ca-certificates \ diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py index e69de29bb..158915d23 100644 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py @@ -0,0 +1 @@ +from .execute_converting_blackbox import PostGisExcerptConverter diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py index 7eb1964ff..d6a446a2b 100644 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py +++ b/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py @@ -3,7 +3,10 @@ import shutil import subprocess import tempfile +#from excerptconverter.baseexcerptconverter import BaseExcerptConverter +#class PostGisExcerptConverter(BaseExcerptConverter): +# pass with tempfile.TemporaryDirectory() as tmp_dir: original_cwd = os.getcwd() @@ -11,12 +14,12 @@ try: print(tmp_dir) shutil.copyfile( - os.path.join('blackbox', 'docker-compose-conversion-blackbox.yml'), + os.path.join(os.path.dirname(__file__), 'blackbox', 'docker-compose-conversion-blackbox.yml'), os.path.join(tmp_dir, 'docker-compose.yml') ) os.chdir(tmp_dir) subprocess.check_call("docker-compose build".split(' ')) - subprocess.check_call("docker-compose run bootstrap sleep 10".split(' ')) + subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) bbox_args = '8.775449276 47.1892350573 8.8901920319 47.2413633153' subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % bbox_args).split(' ')) subprocess.check_call(("docker-compose run excerpt python excerpt.py %s" % bbox_args).split(' ')) diff --git a/osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py b/osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py new file mode 100644 index 000000000..1a8ae091b --- /dev/null +++ b/osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py @@ -0,0 +1,7 @@ +from excerptconverter.gis_excerpt_converter import PostGisExcerptConverter +from excerptconverter.baseexcerptconverter import BaseExcerptConverter + +class PostGisExcerptConverterTestCase(TestCase): + def test_foo(self): + BaseExcerptConverter.available_converters.append(PostGisExcerptConverter) + BaseExcerptConverter.converter_configuration() From 8a47675d2ed75ba8f766525e5a49800f6fedc556 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Thu, 20 Aug 2015 16:39:49 +0200 Subject: [PATCH 35/94] Add gis excerpt converter --- osmaxx-py/config/settings/common.py | 6 +- .../gis_excerpt_converter/__init__.py | 1 - .../gisexcerptconverter/__init__.py | 5 + .../docker-compose-conversion-blackbox.yml | 4 +- .../execute_converting_blackbox.py | 0 .../gis_excerpt_converter.py | 192 ++++++++++++++++++ 6 files changed, 203 insertions(+), 5 deletions(-) delete mode 100644 osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py create mode 100644 osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py rename osmaxx-py/excerptconverter/{gis_excerpt_converter => gisexcerptconverter}/blackbox/docker-compose-conversion-blackbox.yml (78%) rename osmaxx-py/excerptconverter/{gis_excerpt_converter => gisexcerptconverter}/execute_converting_blackbox.py (100%) create mode 100644 osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py diff --git a/osmaxx-py/config/settings/common.py b/osmaxx-py/config/settings/common.py index dc44a8032..f50d0a5f0 100644 --- a/osmaxx-py/config/settings/common.py +++ b/osmaxx-py/config/settings/common.py @@ -50,10 +50,12 @@ LOCAL_APPS = ( 'osmaxx.excerptexport', 'osmaxx.social_auth', - 'excerptconverter.dummyexcerptconverter' + 'excerptconverter.dummyexcerptconverter', + 'excerptconverter.gisexcerptconverter' ) CELERY_IMPORTS = [ - 'excerptconverter.dummyexcerptconverter.dummy_excerpt_converter' + 'excerptconverter.dummyexcerptconverter.dummy_excerpt_converter', + 'excerptconverter.gisexcerptconverter.gis_excerpt_converter' ] # See: https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py b/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py deleted file mode 100644 index 158915d23..000000000 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from .execute_converting_blackbox import PostGisExcerptConverter diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py b/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py new file mode 100644 index 000000000..87fcaad02 --- /dev/null +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py @@ -0,0 +1,5 @@ +#from .execute_converting_blackbox import PostGisExcerptConverter +from excerptconverter.baseexcerptconverter import BaseExcerptConverter +from .gis_excerpt_converter import GisExcerptConverter + +BaseExcerptConverter.available_converters.append(GisExcerptConverter) diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml b/osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml similarity index 78% rename from osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml rename to osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml index 306782ffa..0a900dfe8 100644 --- a/osmaxx-py/excerptconverter/gis_excerpt_converter/blackbox/docker-compose-conversion-blackbox.yml +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml @@ -13,6 +13,6 @@ excerpt: environment: - PGHOST=db volumes: - - results:/home/excerpt/data/ + - /data/results:/home/excerpt/data/ db: - image: geometalab/postgis-with-translit \ No newline at end of file + image: geometalab/postgis-with-translit diff --git a/osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gisexcerptconverter/execute_converting_blackbox.py similarity index 100% rename from osmaxx-py/excerptconverter/gis_excerpt_converter/execute_converting_blackbox.py rename to osmaxx-py/excerptconverter/gisexcerptconverter/execute_converting_blackbox.py diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py new file mode 100644 index 000000000..69a2f7385 --- /dev/null +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -0,0 +1,192 @@ +import os +import shutil +import subprocess +import tempfile +import time + +from celery import shared_task + +from django.utils.translation import ugettext_lazy as _ +from django.contrib import messages + +from excerptconverter.baseexcerptconverter import BaseExcerptConverter + +from osmaxx.excerptexport import models + + +class GisExcerptConverter(BaseExcerptConverter): + @staticmethod + def name(): + return 'GIS' + + @staticmethod + def export_formats(): + return { + 'spatialite': { + 'name': 'SpatiaLite (SQLite)', + 'file_extension': 'sqlite', + 'mime_type': 'application/x-sqlite3' + }, + 'gpkg': { + 'name': 'Geo package', + 'file_extension': 'gpkg', + 'mime_type': 'application/octet-stream' # Not verified! + }, + 'shp': { + 'name': 'Shape file', + 'file_extension': 'shp', + 'mime_type': 'application/octet-stream' # Not verified! + } + } + + @staticmethod + def export_options(): + return { + # 'coordinate_reference_system': { + # 'label': 'Coordinate reference system', + # 'type': 'choice', + # 'default': 'pseudomerkator', + # 'groups': [ + # { + # 'name': 'Global coordinate reference systems', + # 'values': [ + # {'name': 'pseudomerkator', 'label': 'Pseudo merkator'}, + # {'name': 'wgs72', 'label': 'WGS 72'}, + # {'name': 'wgs84', 'label': 'WGS 84'} + # ] + # }, + # { + # 'name': 'UTM zones for your export', + # 'values': [ + # {'name': 'utm32', 'label': 'UTM zone 32'}, + # {'name': 'utm33', 'label': 'UTM zone 33'} + # ] + # } + # ] + # }, + # 'detail_level': { + # 'label': 'Detail level', + # 'type': 'choice', + # 'default': 'verbatim', + # 'values': [ + # {'name': 'verbatim', 'label': 'Verbatim'}, + # {'name': 'simplified', 'label': 'Simplified'} + # ] + # } + } + + @staticmethod + def extract_excerpts(execution_configuration, extraction_order, bbox_args): + """ + Extract excerpt for chosen formats (execution_configuration) using docker-compose + to trigger the conversion process (defined in blackbox/docker-compose-conversion-blackbox.yml) + + :param execution_configuration: + :param extraction_order: + :param bbox_args example: '8.775449276 47.1892350573 8.8901920319 47.2413633153' + :return: + """ + for export_format_key, export_format_config in GisExcerptConverter.export_formats(): + if export_format_key in execution_configuration['formats']: + extraction_command = "docker-compose run excerpt python excerpt.py %(bbox_args)s -f %(format)s" % { + 'bbox_args': bbox_args, + 'format': export_format_key + } + subprocess.check_call(extraction_command.split(' ')) + + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.SUCCESS, + _('The extraction of "%(file_type)s" of order "%(extraction_order)s") was successful.') % { + 'file_type': export_format_config['name'], + 'extraction_order': extraction_order + }, + False + ) + + @staticmethod + @shared_task + def execute_task(extraction_order_id, supported_export_formats, execution_configuration): + wait_time = 0 + # wait for the db to be updated! + extraction_order = None + while extraction_order is None: + try: + extraction_order = models.ExtractionOrder.objects.get(pk=extraction_order_id) + except models.ExtractionOrder.DoesNotExist: + time.sleep(5) + wait_time += 5 + if wait_time > 30: + raise + + with tempfile.TemporaryDirectory() as tmp_dir: + original_cwd = os.getcwd() + + try: + print(tmp_dir) + shutil.copyfile( + os.path.join(os.path.dirname(__file__), 'blackbox', 'docker-compose-conversion-blackbox.yml'), + os.path.join(tmp_dir, 'docker-compose.yml') + ) + os.chdir(tmp_dir) + subprocess.check_call("docker-compose build".split(' ')) + # database needs time to be ready + subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) + + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.INFO, + _('The extraction of the order "%s" is ready to start.') % extraction_order, + False + ) + + bounding_geometry = extraction_order.excerpt.bounding_geometry + if type(bounding_geometry) == models.BBoxBoundingGeometry: + bbox_args = '%(excerpt_south_border)s %(excerpt_west_border)s %(excerpt_north_border)s ' \ + '%(excerpt_east_border)s' % { + 'excerpt_north_border': bounding_geometry.north, + 'excerpt_west_border': bounding_geometry.west, + 'excerpt_south_border': bounding_geometry.south, + 'excerpt_east_border': bounding_geometry.east + } + + subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % + bbox_args).split(' ')) + GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args) + + elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.ERROR, + _('GIS excerpt converter is not yet able to extract polygon excerpts.'), + False + ) + + else: + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.ERROR, + _('GIS excerpt converter is not yet able to extract excerpts of type %s.') % + type(bounding_geometry).__name__, + False + ) + + subprocess.check_call("docker-compose stop --timeout 0".split(' ')) + subprocess.check_call("docker-compose rm -f".split(' ')) + + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.INFO, + _('The extraction of the order "%s" has been finished.') % extraction_order, + False + ) + except: + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.ERROR, + _('The extraction of order %s failed. Please contact an administrator.') % extraction_order, + False + ) + raise + finally: + os.chdir(original_cwd) From 2966f3f6d2c06c2a1b8dde341fc8b342f06e603b Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 21 Aug 2015 08:49:54 +0200 Subject: [PATCH 36/94] Bugfix: prevent DummyExcerptConverter setting status and creating notifications without files to convert --- .../dummy_excerpt_converter.py | 75 ++++++++++--------- 1 file changed, 40 insertions(+), 35 deletions(-) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index e068b29fd..1719ba1fc 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -81,41 +81,46 @@ def create_output_files(execution_configuration, extraction_order, supported_exp @staticmethod @shared_task def execute_task(extraction_order_id, supported_export_formats, execution_configuration): - wait_time = 0 - # wait for the db to be updated! - extraction_order = None - while extraction_order is None: - try: - extraction_order = models.ExtractionOrder.objects.get(pk=extraction_order_id) - except models.ExtractionOrder.DoesNotExist: - time.sleep(5) - wait_time += 5 - if wait_time > 30: - raise - - fake_work_waiting_time_in_seconds = 5 - - # now set the new state - extraction_order.state = models.ExtractionOrderState.WAITING - extraction_order.save() - - time.sleep(fake_work_waiting_time_in_seconds) - - # now set the new state - extraction_order.state = models.ExtractionOrderState.PROCESSING - extraction_order.save() - - message_text = _('Your extraction order "%s" has been started' % extraction_order) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.INFO, message_text, False) - - DummyExcerptConverter.create_output_files(execution_configuration, extraction_order, supported_export_formats) + if len(execution_configuration['formats']) > 0: + wait_time = 0 + # wait for the db to be updated! + extraction_order = None + while extraction_order is None: + try: + extraction_order = models.ExtractionOrder.objects.get(pk=extraction_order_id) + except models.ExtractionOrder.DoesNotExist: + time.sleep(5) + wait_time += 5 + if wait_time > 30: + raise + + fake_work_waiting_time_in_seconds = 5 + + # now set the new state + extraction_order.state = models.ExtractionOrderState.WAITING + extraction_order.save() + + time.sleep(fake_work_waiting_time_in_seconds) + + # now set the new state + extraction_order.state = models.ExtractionOrderState.PROCESSING + extraction_order.save() + + message_text = _('Your extraction order "%s" has been started' % extraction_order) + BaseExcerptConverter.inform_user(extraction_order.orderer, messages.INFO, message_text, False) + + DummyExcerptConverter.create_output_files( + execution_configuration, + extraction_order, + supported_export_formats + ) - time.sleep(fake_work_waiting_time_in_seconds) + time.sleep(fake_work_waiting_time_in_seconds) - # now set the new state - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() + # now set the new state + extraction_order.state = models.ExtractionOrderState.FINISHED + extraction_order.save() - # inform the user of the status change. - message_text = _('Your extraction order %s has been processed' % extraction_order) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.SUCCESS, message_text) + # inform the user of the status change. + message_text = _('Your extraction order %s has been processed' % extraction_order) + BaseExcerptConverter.inform_user(extraction_order.orderer, messages.SUCCESS, message_text) From f4022692a0bb8ee15bfb14751be4e18bc63e114b Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 21 Aug 2015 08:50:37 +0200 Subject: [PATCH 37/94] Add temporary directory as file copy pipe --- compose-development.yml | 2 ++ compose-production.yml | 1 + 2 files changed, 3 insertions(+) diff --git a/compose-development.yml b/compose-development.yml index 3fe496ce3..9902486df 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -8,6 +8,8 @@ shareddatadev: - /data # database - /database/data + # temporary results directory used as pipe between multiple docker forests + - /tmp/osmaxx-results:/results net: "none" sourcedev: diff --git a/compose-production.yml b/compose-production.yml index 82f9e8ccb..9bf380699 100755 --- a/compose-production.yml +++ b/compose-production.yml @@ -4,6 +4,7 @@ shareddata: volumes: - /data - /database/data + - /tmp/osmaxx-results:/results net: "none" webapp: From d9f80c4dae943932e6b7c15f3a3081e9b11d3292 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 21 Aug 2015 08:51:30 +0200 Subject: [PATCH 38/94] Get gis excerpt converter working --- .../docker-compose-conversion-blackbox.yml | 2 +- .../gis_excerpt_converter.py | 30 ++++++++++++++++--- 2 files changed, 27 insertions(+), 5 deletions(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml b/osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml index 0a900dfe8..b1aa57b79 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/blackbox/docker-compose-conversion-blackbox.yml @@ -13,6 +13,6 @@ excerpt: environment: - PGHOST=db volumes: - - /data/results:/home/excerpt/data/ + - /tmp/osmaxx-results:/home/excerpt/data/ db: image: geometalab/postgis-with-translit diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 69a2f7385..9b44dd047 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -3,6 +3,8 @@ import subprocess import tempfile import time +import sys +from os import listdir from celery import shared_task @@ -86,7 +88,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): :param bbox_args example: '8.775449276 47.1892350573 8.8901920319 47.2413633153' :return: """ - for export_format_key, export_format_config in GisExcerptConverter.export_formats(): + for export_format_key, export_format_config in GisExcerptConverter.export_formats().items(): if export_format_key in execution_configuration['formats']: extraction_command = "docker-compose run excerpt python excerpt.py %(bbox_args)s -f %(format)s" % { 'bbox_args': bbox_args, @@ -119,6 +121,9 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise + extraction_order.state = models.ExtractionOrderState.WAITING + extraction_order.save() + with tempfile.TemporaryDirectory() as tmp_dir: original_cwd = os.getcwd() @@ -152,7 +157,10 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % bbox_args).split(' ')) - GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args) + if len(execution_configuration['formats']) > 0: + extraction_order.state = models.ExtractionOrderState.PROCESSING + extraction_order.save() + GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args) elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: BaseExcerptConverter.inform_user( @@ -177,16 +185,30 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config BaseExcerptConverter.inform_user( extraction_order.orderer, messages.INFO, - _('The extraction of the order "%s" has been finished.') % extraction_order, + _('The extraction of the order "%(extraction_order)s" has been finished. ' + 'Created files: %(created_files)s.') % { + 'extraction_order': extraction_order, + 'created_files': str([file_or_dir for file_or_dir in listdir('/results')]) + }, False ) + extraction_order.state = models.ExtractionOrderState.FINISHED + extraction_order.save() except: + extraction_order.state = models.ExtractionOrderState.CANCELED + extraction_order.save() + BaseExcerptConverter.inform_user( extraction_order.orderer, messages.ERROR, - _('The extraction of order %s failed. Please contact an administrator.') % extraction_order, + _('The extraction of order %(extraction_order)s failed: %(error)s. ' + 'Please contact an administrator.') % { + 'extraction_order': extraction_order, + 'error': sys.exc_info()[0] + }, False ) raise finally: os.chdir(original_cwd) + print('Created files: ' + str([file for file in listdir('/results')])) From abecf34a87a8534bb9682bffd860ea622bc84b74 Mon Sep 17 00:00:00 2001 From: Tobias Blaser Date: Fri, 21 Aug 2015 15:11:28 +0200 Subject: [PATCH 39/94] Fix file handling after extraction --- compose-development.yml | 2 +- osmaxx-py/config/settings/common.py | 1 + .../excerptconverter/converter_manager.py | 4 +- .../dummy_excerpt_converter.py | 85 +++++++++-------- .../gis_excerpt_converter.py | 93 +++++++++++++++---- 5 files changed, 122 insertions(+), 63 deletions(-) diff --git a/compose-development.yml b/compose-development.yml index 9902486df..5dbb80c5b 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -15,7 +15,7 @@ shareddatadev: sourcedev: build: docker/volume-mount volumes: - - osmaxx-py:/home/osmaxx/source + - ./osmaxx-py:/home/osmaxx/source - ~/.cache/pip:/home/osmaxx/.cache/pip webappdev: diff --git a/osmaxx-py/config/settings/common.py b/osmaxx-py/config/settings/common.py index f50d0a5f0..6fd0ceabc 100644 --- a/osmaxx-py/config/settings/common.py +++ b/osmaxx-py/config/settings/common.py @@ -200,6 +200,7 @@ MEDIA_ROOT = env.str('DJANGO_MEDIA_ROOT', default=str(ROOT_DIR('..', 'media'))) PRIVATE_MEDIA_ROOT = env.str('DJANGO_PRIVATE_MEDIA_ROOT', default=str(ROOT_DIR.path('..', 'private_media'))) +RESULT_MEDIA_ROOT = '/results' # See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url MEDIA_URL = '/media/' diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index dd0aae6d9..12437dc95 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -29,7 +29,9 @@ def __init__(self, extraction_order, def execute_converters(self): for Converter in self.available_converters: - if Converter.__name__ in self.extraction_order.extraction_configuration: + if (Converter.__name__ in self.extraction_order.extraction_configuration and + len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0): + Converter.execute( self.extraction_order, self.extraction_order.extraction_configuration[Converter.__name__], diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 1719ba1fc..0deb530ed 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -81,46 +81,45 @@ def create_output_files(execution_configuration, extraction_order, supported_exp @staticmethod @shared_task def execute_task(extraction_order_id, supported_export_formats, execution_configuration): - if len(execution_configuration['formats']) > 0: - wait_time = 0 - # wait for the db to be updated! - extraction_order = None - while extraction_order is None: - try: - extraction_order = models.ExtractionOrder.objects.get(pk=extraction_order_id) - except models.ExtractionOrder.DoesNotExist: - time.sleep(5) - wait_time += 5 - if wait_time > 30: - raise - - fake_work_waiting_time_in_seconds = 5 - - # now set the new state - extraction_order.state = models.ExtractionOrderState.WAITING - extraction_order.save() - - time.sleep(fake_work_waiting_time_in_seconds) - - # now set the new state - extraction_order.state = models.ExtractionOrderState.PROCESSING - extraction_order.save() - - message_text = _('Your extraction order "%s" has been started' % extraction_order) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.INFO, message_text, False) - - DummyExcerptConverter.create_output_files( - execution_configuration, - extraction_order, - supported_export_formats - ) - - time.sleep(fake_work_waiting_time_in_seconds) - - # now set the new state - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() - - # inform the user of the status change. - message_text = _('Your extraction order %s has been processed' % extraction_order) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.SUCCESS, message_text) + wait_time = 0 + # wait for the db to be updated! + extraction_order = None + while extraction_order is None: + try: + extraction_order = models.ExtractionOrder.objects.get(pk=extraction_order_id) + except models.ExtractionOrder.DoesNotExist: + time.sleep(5) + wait_time += 5 + if wait_time > 30: + raise + + fake_work_waiting_time_in_seconds = 5 + + # now set the new state + extraction_order.state = models.ExtractionOrderState.WAITING + extraction_order.save() + + time.sleep(fake_work_waiting_time_in_seconds) + + # now set the new state + extraction_order.state = models.ExtractionOrderState.PROCESSING + extraction_order.save() + + message_text = _('Your extraction order "%s" has been started' % extraction_order) + BaseExcerptConverter.inform_user(extraction_order.orderer, messages.INFO, message_text, False) + + DummyExcerptConverter.create_output_files( + execution_configuration, + extraction_order, + supported_export_formats + ) + + time.sleep(fake_work_waiting_time_in_seconds) + + # now set the new state + extraction_order.state = models.ExtractionOrderState.FINISHED + extraction_order.save() + + # inform the user of the status change. + message_text = _('Your extraction order %s has been processed' % extraction_order) + BaseExcerptConverter.inform_user(extraction_order.orderer, messages.SUCCESS, message_text) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 9b44dd047..7254e05af 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -4,18 +4,23 @@ import tempfile import time import sys -from os import listdir from celery import shared_task +from django.conf import settings from django.utils.translation import ugettext_lazy as _ from django.contrib import messages +from django.core.files import File +from django.core.files.storage import FileSystemStorage from excerptconverter.baseexcerptconverter import BaseExcerptConverter from osmaxx.excerptexport import models +private_storage = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT) + + class GisExcerptConverter(BaseExcerptConverter): @staticmethod def name(): @@ -89,22 +94,77 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): :return: """ for export_format_key, export_format_config in GisExcerptConverter.export_formats().items(): + index = 0 if export_format_key in execution_configuration['formats']: + index += 1 extraction_command = "docker-compose run excerpt python excerpt.py %(bbox_args)s -f %(format)s" % { 'bbox_args': bbox_args, 'format': export_format_key } subprocess.check_call(extraction_command.split(' ')) - BaseExcerptConverter.inform_user( - extraction_order.orderer, - messages.SUCCESS, - _('The extraction of "%(file_type)s" of order "%(extraction_order)s") was successful.') % { - 'file_type': export_format_config['name'], - 'extraction_order': extraction_order - }, - False - ) + if len(os.listdir(settings.RESULT_MEDIA_ROOT)) > 0: + for result_file_name in os.listdir(settings.RESULT_MEDIA_ROOT): + # gis files are packaged in a zip file + if GisExcerptConverter.create_output_file(extraction_order, result_file_name): + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.SUCCESS, + _('Extraction of "%(file_type)s" of extraction order "%(order_id)s" was successful. ' + '(File %(file_index)s of %(number_of_files)s of %(converter_name)s converter)') % { + 'file_type': export_format_config['name'], + 'file_index': index, + 'number_of_files': len(execution_configuration['formats']), + 'converter_name': GisExcerptConverter.name(), + 'order_id': extraction_order.id + }, + False + ) + else: + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.ERROR, + _('The extraction of "%(file)s" of extraction order "%(order_id)s" failed.') % { + 'file': result_file_name, + 'order_id': extraction_order.id + }, + False + ) + else: + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.ERROR, + _('The extraction of "%(file_type)s" of extraction order "%(order_id)s" failed.') % { + 'file_type': export_format_config['name'], + 'order_id': extraction_order.id + }, + False + ) + + @staticmethod + def create_output_file(extraction_order, result_file_name): + """ + Move file to private media storage and add OutputFile to Extractionorder + + :return: True if file created successful + """ + output_file = models.OutputFile.objects.create( + mime_type='application/zip', + extraction_order=extraction_order + ) + + if not os.path.exists(private_storage.location): + os.makedirs(private_storage.location) + + file_name = str(output_file.public_identifier) + '.zip' + result_file_path = os.path.abspath(os.path.join(settings.RESULT_MEDIA_ROOT, result_file_name)) + target_file_path = os.path.abspath(os.path.join(private_storage.location, file_name)) + + shutil.move(result_file_path,target_file_path) + output_file.file = private_storage.open(target_file_path) + output_file.save() + + return os.path.isfile(target_file_path) @staticmethod @shared_task @@ -155,9 +215,9 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config 'excerpt_east_border': bounding_geometry.east } - subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % - bbox_args).split(' ')) if len(execution_configuration['formats']) > 0: + subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % + bbox_args).split(' ')) extraction_order.state = models.ExtractionOrderState.PROCESSING extraction_order.save() GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args) @@ -185,10 +245,8 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config BaseExcerptConverter.inform_user( extraction_order.orderer, messages.INFO, - _('The extraction of the order "%(extraction_order)s" has been finished. ' - 'Created files: %(created_files)s.') % { - 'extraction_order': extraction_order, - 'created_files': str([file_or_dir for file_or_dir in listdir('/results')]) + _('The extraction of the order "%(extraction_order)s" has been finished.') % { + 'extraction_order': extraction_order }, False ) @@ -204,11 +262,10 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config _('The extraction of order %(extraction_order)s failed: %(error)s. ' 'Please contact an administrator.') % { 'extraction_order': extraction_order, - 'error': sys.exc_info()[0] + 'error': str(sys.exc_info()) }, False ) raise finally: os.chdir(original_cwd) - print('Created files: ' + str([file for file in listdir('/results')])) From e661b07917c5df60630a70d854d1bd85c1ca1523 Mon Sep 17 00:00:00 2001 From: Tobias Blaser Date: Fri, 21 Aug 2015 17:30:42 +0200 Subject: [PATCH 40/94] Improve result file handling and user reporting --- .../base_excerpt_converter.py | 16 ++++++++++++++ .../dummy_excerpt_converter.py | 11 +++------- .../gis_excerpt_converter.py | 21 ++++--------------- .../excerptexport/models/extraction_order.py | 20 ++++++++++++++++++ 4 files changed, 43 insertions(+), 25 deletions(-) diff --git a/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py b/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py index d757201a8..4536ed714 100644 --- a/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py @@ -7,6 +7,8 @@ from django.core.mail import send_mail from django.utils.translation import ugettext_lazy as _ +from osmaxx.excerptexport import models + class BaseExcerptConverter(metaclass=abstract_base_class.ABCMeta): available_converters = [] @@ -32,6 +34,20 @@ def export_options(): def execute_task(extraction_order_id, supported_export_formats, execution_configuration): raise NotImplemented + @staticmethod + def file_conversion_finished_of_extraction_order(extraction_order): + if extraction_order.output_files.count() >= len(extraction_order.extraction_formats): + BaseExcerptConverter.inform_user( + extraction_order.orderer, + messages.SUCCESS, + _('The extraction of the order "%(order_id)s" has been finished.') % { + 'order_id': extraction_order.id + }, + True + ) + extraction_order.state = models.ExtractionOrderState.FINISHED + extraction_order.save() + @staticmethod def inform_user(user, message_type, message_text, email=True): stored_messages.api.add_message_for( diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 0deb530ed..e3c066775 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -105,7 +105,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config extraction_order.state = models.ExtractionOrderState.PROCESSING extraction_order.save() - message_text = _('Your extraction order "%s" has been started' % extraction_order) + message_text = _('The Dummy conversion of extraction order "%s" has been started.' % extraction_order.id) BaseExcerptConverter.inform_user(extraction_order.orderer, messages.INFO, message_text, False) DummyExcerptConverter.create_output_files( @@ -116,10 +116,5 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config time.sleep(fake_work_waiting_time_in_seconds) - # now set the new state - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() - - # inform the user of the status change. - message_text = _('Your extraction order %s has been processed' % extraction_order) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.SUCCESS, message_text) + # now set the new state (if all files have been processed) and inform the user about the state + BaseExcerptConverer.file_conversion_finished_of_extraction_order(extraction_order) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 7254e05af..5f24a0b56 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -194,14 +194,13 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config os.path.join(tmp_dir, 'docker-compose.yml') ) os.chdir(tmp_dir) - subprocess.check_call("docker-compose build".split(' ')) # database needs time to be ready subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) BaseExcerptConverter.inform_user( extraction_order.orderer, messages.INFO, - _('The extraction of the order "%s" is ready to start.') % extraction_order, + _('The GIS extraction of the order "%s" is has been started.') % extraction_order.id, False ) @@ -242,26 +241,14 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config subprocess.check_call("docker-compose stop --timeout 0".split(' ')) subprocess.check_call("docker-compose rm -f".split(' ')) - BaseExcerptConverter.inform_user( - extraction_order.orderer, - messages.INFO, - _('The extraction of the order "%(extraction_order)s" has been finished.') % { - 'extraction_order': extraction_order - }, - False - ) - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() + BaseExcerptConverter.file_conversion_finished_of_extraction_order(extraction_order) except: - extraction_order.state = models.ExtractionOrderState.CANCELED - extraction_order.save() - BaseExcerptConverter.inform_user( extraction_order.orderer, messages.ERROR, - _('The extraction of order %(extraction_order)s failed: %(error)s. ' + _('The extraction of order %(order_id)s failed: %(error)s. ' 'Please contact an administrator.') % { - 'extraction_order': extraction_order, + 'order_id': extraction_order.id, 'error': str(sys.exc_info()) }, False diff --git a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py index e17763ffc..5e8f0ab00 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py +++ b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py @@ -45,6 +45,26 @@ def extraction_configuration(self): @extraction_configuration.setter def extraction_configuration(self, value): + """ + :return example: + { + 'gis': { + 'formats': ['txt', 'file_gdb'], + 'options': { + 'coordinate_reference_system': 'wgs72', + 'detail_level': 'verbatim' + } + }, + 'routing': { ... } + } + """ if not value: value = {} self._extraction_configuration = json.dumps(value) + + @property + def extraction_formats(self): + extraction_formats = [] + for export_format_key, export_format in self.extraction_configuration.items(): + extraction_formats = extraction_formats + export_format['formats'] + return extraction_formats \ No newline at end of file From 231dbbef830c1a6633d1968680287751779b2ed6 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Mon, 24 Aug 2015 10:45:08 +0200 Subject: [PATCH 41/94] fix grammar in comments --- .../excerptexport/scripts/formModeSwitcher.js | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js index 838ff1efb..e3bd5a8ed 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js @@ -27,10 +27,10 @@ } // IE may throw an error here "Object doesn't support this action". - // It seems IE won't trigger a change event on select boxes - // I didn't found a solution to get the supported actions, - // so the error is still there: Fuck you IE - // Note: The switcher is working anyway + // It seems IE won't trigger a change event on select boxes. + // I didn't find a solution to get the supported actions, + // so the error is still there: Fuck you IE. + // Note: The switcher is working regardless. formPartsSwitcher.dispatchEvent(new Event('change')); }; @@ -53,10 +53,10 @@ if(formPartsSwitcher.value != formPartId) { formPartsSwitcher.value = formPartId; // IE may throw an error here "Object doesn't support this action". - // It seems IE won't trigger a change event on select boxes - // I didn't found a solution to get the supported actions, - // so the error is still there: Fuck you IE - // Note: The switcher is working anyway + // It seems IE won't trigger a change event on select boxes. + // I didn't find a solution to get the supported actions, + // so the error is still there: Fuck you IE. + // Note: The switcher is working regardless. formPartsSwitcher.dispatchEvent(new Event('change')); } }; From b5661617b43c9025b3376d380ddb9895c3dfc1e9 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 10:51:50 +0200 Subject: [PATCH 42/94] Fix bad comment --- .../static/excerptexport/scripts/formModeSwitcher.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js index e3bd5a8ed..f9a7a2634 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js @@ -29,7 +29,7 @@ // IE may throw an error here "Object doesn't support this action". // It seems IE won't trigger a change event on select boxes. // I didn't find a solution to get the supported actions, - // so the error is still there: Fuck you IE. + // so the error is still there: Use a better browser. // Note: The switcher is working regardless. formPartsSwitcher.dispatchEvent(new Event('change')); }; @@ -55,7 +55,7 @@ // IE may throw an error here "Object doesn't support this action". // It seems IE won't trigger a change event on select boxes. // I didn't find a solution to get the supported actions, - // so the error is still there: Fuck you IE. + // so the error is still there: Use a better browser. // Note: The switcher is working regardless. formPartsSwitcher.dispatchEvent(new Event('change')); } From 3214313522d3aab10b99ae2e68ba6963c1a2ce5d Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 10:54:04 +0200 Subject: [PATCH 43/94] Improve comment --- .../static/excerptexport/scripts/formModeSwitcher.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js index f9a7a2634..e10147ab1 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/formModeSwitcher.js @@ -30,7 +30,7 @@ // It seems IE won't trigger a change event on select boxes. // I didn't find a solution to get the supported actions, // so the error is still there: Use a better browser. - // Note: The switcher is working regardless. + // Note: The switcher is working on IE but IE users don't have the same features as other users formPartsSwitcher.dispatchEvent(new Event('change')); }; @@ -56,7 +56,7 @@ // It seems IE won't trigger a change event on select boxes. // I didn't find a solution to get the supported actions, // so the error is still there: Use a better browser. - // Note: The switcher is working regardless. + // Note: The switcher is working on IE but IE users don't have the same features as other users formPartsSwitcher.dispatchEvent(new Event('change')); } }; From db3df529311a3648e9bb4e2e65a81b5456e7093a Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Mon, 24 Aug 2015 10:54:19 +0200 Subject: [PATCH 44/94] fix grammar & spelling in comments --- .../static/excerptexport/stylesheets/ie-fixes.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css index 91c0c4510..45f11da82 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css @@ -1,6 +1,6 @@ @media all and (-ms-high-contrast: none), (-ms-high-contrast: active) { - /* readd IE default style to reatyle it isable for IE uesers - * IE does not support styling for select and option as buttons :-( + /* Re-add IE default style to restyle it isable for IE uesers, as + * IE does not support styling for select and option as buttons. :-( */ select.btn-group, .select-button-group > select.btn-group { font-size: 1.5rem !important; From a301165ca38f07eb6c5f555240076cc211cd8b21 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 10:55:56 +0200 Subject: [PATCH 45/94] Fix comment typo --- .../excerptexport/static/excerptexport/stylesheets/ie-fixes.css | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css index 45f11da82..2daf7805e 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/stylesheets/ie-fixes.css @@ -1,5 +1,5 @@ @media all and (-ms-high-contrast: none), (-ms-high-contrast: active) { - /* Re-add IE default style to restyle it isable for IE uesers, as + /* Re-add IE default style to restyle it usable for IE users, as * IE does not support styling for select and option as buttons. :-( */ select.btn-group, .select-button-group > select.btn-group { From f62bce36d3974af0a62a10da68ccb5aab6d7bf25 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 11:29:00 +0200 Subject: [PATCH 46/94] Fix susicions assignement --- .../osmaxx/excerptexport/static/excerptexport/scripts/map.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 552d00aa4..5604b20e9 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -83,7 +83,7 @@ }; - var map = map = L.map('map').setView([0, 0], 2); + var map = L.map('map').setView([0, 0], 2); // add an OpenStreetMap tile layer L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', { attribution: '© OpenStreetMap contributors' From 6005b0a2cba2b51d3503a8fb2b9f8046a5cb5e7b Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 11:45:06 +0200 Subject: [PATCH 47/94] Fix indentation --- .../static/excerptexport/scripts/map.js | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 5604b20e9..994f7e90c 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -84,15 +84,15 @@ var map = L.map('map').setView([0, 0], 2); - // add an OpenStreetMap tile layer - L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', { - attribution: '© OpenStreetMap contributors' - }).addTo(map); + // add an OpenStreetMap tile layer + L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', { + attribution: '© OpenStreetMap contributors' + }).addTo(map); - var locationFilter = new L.LocationFilter({ - enable: true, - enableButton: false - }).addTo(map); + var locationFilter = new L.LocationFilter({ + enable: true, + enableButton: false + }).addTo(map); var excerptManager = new ExcerptManager( locationFilter, From 524e88e6d48659693b412db7bec6a35ccb8895c3 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 12:10:02 +0200 Subject: [PATCH 48/94] extract expression --- .../static/excerptexport/scripts/map.js | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 994f7e90c..3aac9710c 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -28,6 +28,17 @@ locationFilterBounds._southWest.lat = this.inputElementsNewBoundingBox.inputElementSouth.value; this.locationFilter.setBounds(locationFilterBounds); }; + + this.isSelectOptionSelectedAndExcerptOnMapInSyncWithInputFields = function(select, locationFilterBounds) { + if(select.value != "") { + var optionElement = select.querySelector('option[value="'+select.value+'"]'); + } + return select.value != "" && + optionElement.getAttribute('data-north') == locationFilterBounds._northEast.lat && + optionElement.getAttribute('data-west') == locationFilterBounds._southWest.lng && + optionElement.getAttribute('data-east') == locationFilterBounds._northEast.lng && + optionElement.getAttribute('data-south') == locationFilterBounds._southWest.lat + } /** * if the shown excerpt on the map changes and the change was not made by selecting an existing excerpt in the list @@ -36,11 +47,7 @@ this.userChangeExcerptOnMapShowNewExcerptPart = function() { var locationFilterBounds = this.locationFilter.getBounds(); var select = this.selectElementExistingExcerpts; - if(!(selectElementExistingExcerpts.value != "" && - select.querySelector('option[value="'+select.value+'"]').getAttribute('data-north') == locationFilterBounds._northEast.lat && - select.querySelector('option[value="'+select.value+'"]').getAttribute('data-west') == locationFilterBounds._southWest.lng && - select.querySelector('option[value="'+select.value+'"]').getAttribute('data-east') == locationFilterBounds._northEast.lng && - select.querySelector('option[value="'+select.value+'"]').getAttribute('data-south') == locationFilterBounds._southWest.lat)) { + if(!this.isSelectOptionSelectedAndExcerptOnMapInSyncWithInputFields(select, locationFilterBounds)) { this.formElementPartsSwitcher.value = 'new-excerpt'; this.formElementPartsSwitcher.dispatchEvent(new Event('change')); } From e55699d9566bfa004557c6b68cb251014084c973 Mon Sep 17 00:00:00 2001 From: Tobias Blaser Date: Mon, 24 Aug 2015 12:25:00 +0200 Subject: [PATCH 49/94] Fix conversion status handling --- compose-development.yml | 3 ++ compose-production.yml | 1 + osmaxx-py/excerptconverter/__init__.py | 1 + .../base_excerpt_converter.py | 41 ---------------- .../excerptconverter/converter_helper.py | 49 +++++++++++++++++++ .../dummy_excerpt_converter.py | 48 ++++++++++-------- .../gis_excerpt_converter.py | 35 ++++++------- .../tests/test_converter_manager.py | 4 +- .../tests/test_gis_excerpt_converter.py | 7 +++ .../tests/test_postgis_excerpt_converter.py | 7 --- .../osmaxx/excerptexport/tests/test_views.py | 4 ++ 11 files changed, 111 insertions(+), 89 deletions(-) create mode 100644 osmaxx-py/excerptconverter/converter_helper.py create mode 100644 osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py delete mode 100644 osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py diff --git a/compose-development.yml b/compose-development.yml index 5dbb80c5b..d619f3e36 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -39,6 +39,8 @@ webappdev: - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - DJANGO_DATABASE_URL=postgis://postgres@databasedev/postgres - DJANGO_DEBUG=true + - DJANGO_EMAIL_HOST=emaildev + rabbitmqdev: extends: @@ -63,6 +65,7 @@ celerydev: - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - DJANGO_DATABASE_URL=postgis://postgres@databasedev/postgres - DJANGO_DEBUG=true + - DJANGO_EMAIL_HOST=emaildev databasedev: extends: diff --git a/compose-production.yml b/compose-production.yml index 9bf380699..9bbd8e3bb 100755 --- a/compose-production.yml +++ b/compose-production.yml @@ -51,6 +51,7 @@ celery: environment: - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - DJANGO_DATABASE_URL=postgis://postgres@database/postgres + - DJANGO_EMAIL_HOST=email database: extends: diff --git a/osmaxx-py/excerptconverter/__init__.py b/osmaxx-py/excerptconverter/__init__.py index 5ebcd00af..897e42f45 100644 --- a/osmaxx-py/excerptconverter/__init__.py +++ b/osmaxx-py/excerptconverter/__init__.py @@ -1,2 +1,3 @@ # flake8: noqa from .converter_manager import ConverterManager +from .converter_helper import ConverterHelper diff --git a/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py b/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py index 4536ed714..0b7a3b10f 100644 --- a/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py @@ -1,10 +1,7 @@ import abc as abstract_base_class -import stored_messages from celery import shared_task -from django.contrib import messages -from django.core.mail import send_mail from django.utils.translation import ugettext_lazy as _ from osmaxx.excerptexport import models @@ -34,44 +31,6 @@ def export_options(): def execute_task(extraction_order_id, supported_export_formats, execution_configuration): raise NotImplemented - @staticmethod - def file_conversion_finished_of_extraction_order(extraction_order): - if extraction_order.output_files.count() >= len(extraction_order.extraction_formats): - BaseExcerptConverter.inform_user( - extraction_order.orderer, - messages.SUCCESS, - _('The extraction of the order "%(order_id)s" has been finished.') % { - 'order_id': extraction_order.id - }, - True - ) - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() - - @staticmethod - def inform_user(user, message_type, message_text, email=True): - stored_messages.api.add_message_for( - users=[user], - level=message_type, - message_text=message_text - ) - - if email: - if hasattr(user, 'email'): - send_mail( - '[OSMAXX] '+message_text, - message_text, - 'no-reply@osmaxx.hsr.ch', - [user.email] - ) - else: - BaseExcerptConverter.inform_user( - user, - messages.WARNING, - _("There is no email address assigned to your account. " - "You won't be notified by email on process finish!"), - email=False) - @classmethod def converter_configuration(cls): return { diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py new file mode 100644 index 000000000..6586962df --- /dev/null +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -0,0 +1,49 @@ +from django.contrib import messages +from django.core.mail import send_mail +from django.utils.translation import ugettext_lazy as _ + +import stored_messages + +from osmaxx.excerptexport import models + +from excerptconverter.baseexcerptconverter import BaseExcerptConverter + +class ConverterHelper: + def __init__(self, extraction_order): + self.extraction_order = extraction_order + self.user = extraction_order.orderer + + def file_conversion_finished(self): + if self.extraction_order.output_files.count() >= len(self.extraction_order.extraction_formats): + self.inform_user( + messages.SUCCESS, + _('The extraction of the order "%(order_id)s" has been finished.') % { + 'order_id': self.extraction_order.id + }, + True + ) + self.extraction_order.state = models.ExtractionOrderState.FINISHED + self.extraction_order.save() + + def inform_user(self, message_type, message_text, email=True): + stored_messages.api.add_message_for( + users=[self.user], + level=message_type, + message_text=message_text + ) + + if email: + if hasattr(self.user, 'email'): + send_mail( + '[OSMAXX] '+message_text, + message_text, + 'no-reply@osmaxx.hsr.ch', + [self.user.email] + ) + else: + self.inform_user( + messages.WARNING, + _("There is no email address assigned to your account. " + "You won't be notified by email on process finish!"), + email=False + ) \ No newline at end of file diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index e3c066775..9bdbf23a4 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -1,5 +1,6 @@ import time import os +import traceback from celery import shared_task @@ -10,6 +11,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter +from excerptconverter import ConverterHelper from osmaxx.excerptexport import models @@ -52,7 +54,7 @@ def export_options(): } @staticmethod - def create_output_files(execution_configuration, extraction_order, supported_export_formats): + def create_output_files(execution_configuration, extraction_order, supported_export_formats, converter_helper): for format_key in execution_configuration['formats']: output_file = models.OutputFile.objects.create( mime_type=supported_export_formats[format_key]['mime_type'], @@ -73,10 +75,10 @@ def create_output_files(execution_configuration, extraction_order, supported_exp if private_storage.exists(file_name): message_text = _('"%s" created successful' % file_name) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.SUCCESS, message_text, False) + converter_helper.inform_user(messages.SUCCESS, message_text, False) else: message_text = _('Creation of "%s" failed!' % file_name) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.ERROR, message_text, False) + converter_helper.inform_user(messages.ERROR, message_text, False) @staticmethod @shared_task @@ -93,28 +95,32 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise - fake_work_waiting_time_in_seconds = 5 + try: + converter_helper = ConverterHelper(extraction_order) + fake_work_waiting_time_in_seconds = 5 - # now set the new state - extraction_order.state = models.ExtractionOrderState.WAITING - extraction_order.save() + # now set the new state + extraction_order.state = models.ExtractionOrderState.WAITING + extraction_order.save() - time.sleep(fake_work_waiting_time_in_seconds) + time.sleep(fake_work_waiting_time_in_seconds) - # now set the new state - extraction_order.state = models.ExtractionOrderState.PROCESSING - extraction_order.save() + # now set the new state + extraction_order.state = models.ExtractionOrderState.PROCESSING + extraction_order.save() - message_text = _('The Dummy conversion of extraction order "%s" has been started.' % extraction_order.id) - BaseExcerptConverter.inform_user(extraction_order.orderer, messages.INFO, message_text, False) + message_text = _('The Dummy conversion of extraction order "%s" has been started.') % extraction_order.id + converter_helper.inform_user(messages.INFO, message_text, False) - DummyExcerptConverter.create_output_files( - execution_configuration, - extraction_order, - supported_export_formats - ) + DummyExcerptConverter.create_output_files( + execution_configuration, + extraction_order, + supported_export_formats, converter_helper + ) - time.sleep(fake_work_waiting_time_in_seconds) + time.sleep(fake_work_waiting_time_in_seconds) - # now set the new state (if all files have been processed) and inform the user about the state - BaseExcerptConverer.file_conversion_finished_of_extraction_order(extraction_order) + # now set the new state (if all files have been processed) and inform the user about the state + converter_helper.file_conversion_finished() + except: + converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), False) \ No newline at end of file diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 5f24a0b56..3bd606290 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -4,16 +4,18 @@ import tempfile import time import sys +import traceback from celery import shared_task from django.conf import settings from django.utils.translation import ugettext_lazy as _ -from django.contrib import messages from django.core.files import File from django.core.files.storage import FileSystemStorage +from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter +from excerptconverter import ConverterHelper from osmaxx.excerptexport import models @@ -83,7 +85,7 @@ def export_options(): } @staticmethod - def extract_excerpts(execution_configuration, extraction_order, bbox_args): + def extract_excerpts(execution_configuration, extraction_order, bbox_args, converter_helper): """ Extract excerpt for chosen formats (execution_configuration) using docker-compose to trigger the conversion process (defined in blackbox/docker-compose-conversion-blackbox.yml) @@ -107,8 +109,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): for result_file_name in os.listdir(settings.RESULT_MEDIA_ROOT): # gis files are packaged in a zip file if GisExcerptConverter.create_output_file(extraction_order, result_file_name): - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.SUCCESS, _('Extraction of "%(file_type)s" of extraction order "%(order_id)s" was successful. ' '(File %(file_index)s of %(number_of_files)s of %(converter_name)s converter)') % { @@ -121,8 +122,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): False ) else: - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of "%(file)s" of extraction order "%(order_id)s" failed.') % { 'file': result_file_name, @@ -131,8 +131,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): False ) else: - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of "%(file_type)s" of extraction order "%(order_id)s" failed.') % { 'file_type': export_format_config['name'], @@ -181,6 +180,8 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise + + converter_helper = ConverterHelper(extraction_order) extraction_order.state = models.ExtractionOrderState.WAITING extraction_order.save() @@ -197,8 +198,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # database needs time to be ready subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.INFO, _('The GIS extraction of the order "%s" is has been started.') % extraction_order.id, False @@ -219,19 +219,17 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config bbox_args).split(' ')) extraction_order.state = models.ExtractionOrderState.PROCESSING extraction_order.save() - GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args) + GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args, converter_helper) elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract polygon excerpts.'), False ) else: - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract excerpts of type %s.') % type(bounding_geometry).__name__, @@ -241,15 +239,14 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config subprocess.check_call("docker-compose stop --timeout 0".split(' ')) subprocess.check_call("docker-compose rm -f".split(' ')) - BaseExcerptConverter.file_conversion_finished_of_extraction_order(extraction_order) + converter_helper.file_conversion_finished() except: - BaseExcerptConverter.inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of order %(order_id)s failed: %(error)s. ' 'Please contact an administrator.') % { 'order_id': extraction_order.id, - 'error': str(sys.exc_info()) + 'error': traceback.format_tb(sys.exc_info()) }, False ) diff --git a/osmaxx-py/excerptconverter/tests/test_converter_manager.py b/osmaxx-py/excerptconverter/tests/test_converter_manager.py index 0fc07e9c3..ebcbb0edd 100644 --- a/osmaxx-py/excerptconverter/tests/test_converter_manager.py +++ b/osmaxx-py/excerptconverter/tests/test_converter_manager.py @@ -6,6 +6,7 @@ from excerptconverter import ConverterManager from excerptconverter.baseexcerptconverter import BaseExcerptConverter from excerptconverter.dummyexcerptconverter import DummyExcerptConverter +from excerptconverter.gisexcerptconverter import GisExcerptConverter from osmaxx.excerptexport import models @@ -141,7 +142,8 @@ def test_converter_configuration(self): collections.OrderedDict(sorted({ SomeExcerptConverter.__name__: SomeExcerptConverter.converter_configuration(), DummyExcerptConverter.__name__: DummyExcerptConverter.converter_configuration(), - TestExcerptConverter.__name__: TestExcerptConverter.converter_configuration() + TestExcerptConverter.__name__: TestExcerptConverter.converter_configuration(), + GisExcerptConverter.__name__: GisExcerptConverter.converter_configuration() }.items())) ) diff --git a/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py b/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py new file mode 100644 index 000000000..58c03312d --- /dev/null +++ b/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py @@ -0,0 +1,7 @@ +from django.test.testcases import TestCase + +from excerptconverter.gisexcerptconverter import GisExcerptConverter +from excerptconverter.baseexcerptconverter import BaseExcerptConverter + +class GisExcerptConverterTestCase(TestCase): + pass diff --git a/osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py b/osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py deleted file mode 100644 index 1a8ae091b..000000000 --- a/osmaxx-py/excerptconverter/tests/test_postgis_excerpt_converter.py +++ /dev/null @@ -1,7 +0,0 @@ -from excerptconverter.gis_excerpt_converter import PostGisExcerptConverter -from excerptconverter.baseexcerptconverter import BaseExcerptConverter - -class PostGisExcerptConverterTestCase(TestCase): - def test_foo(self): - BaseExcerptConverter.available_converters.append(PostGisExcerptConverter) - BaseExcerptConverter.converter_configuration() diff --git a/osmaxx-py/osmaxx/excerptexport/tests/test_views.py b/osmaxx-py/osmaxx/excerptexport/tests/test_views.py index ef96c4150..c36128a32 100644 --- a/osmaxx-py/osmaxx/excerptexport/tests/test_views.py +++ b/osmaxx-py/osmaxx/excerptexport/tests/test_views.py @@ -68,6 +68,10 @@ def setUp(self): 'options': { 'detail_level': 'verbatim' } + }, + 'GisExcerptConverter': { + 'formats': [], + 'options': {} } } From 74fda8e531a46b03eab84567a5db432aebaac1f8 Mon Sep 17 00:00:00 2001 From: Tobias Blaser Date: Mon, 24 Aug 2015 12:31:45 +0200 Subject: [PATCH 50/94] Clean up --- .../execute_converting_blackbox.py | 32 ------------------- .../gis_excerpt_converter.py | 1 + 2 files changed, 1 insertion(+), 32 deletions(-) delete mode 100644 osmaxx-py/excerptconverter/gisexcerptconverter/execute_converting_blackbox.py diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/execute_converting_blackbox.py b/osmaxx-py/excerptconverter/gisexcerptconverter/execute_converting_blackbox.py deleted file mode 100644 index d6a446a2b..000000000 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/execute_converting_blackbox.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python -import os -import shutil -import subprocess -import tempfile -#from excerptconverter.baseexcerptconverter import BaseExcerptConverter - -#class PostGisExcerptConverter(BaseExcerptConverter): -# pass - -with tempfile.TemporaryDirectory() as tmp_dir: - original_cwd = os.getcwd() - - try: - print(tmp_dir) - shutil.copyfile( - os.path.join(os.path.dirname(__file__), 'blackbox', 'docker-compose-conversion-blackbox.yml'), - os.path.join(tmp_dir, 'docker-compose.yml') - ) - os.chdir(tmp_dir) - subprocess.check_call("docker-compose build".split(' ')) - subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) - bbox_args = '8.775449276 47.1892350573 8.8901920319 47.2413633153' - subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % bbox_args).split(' ')) - subprocess.check_call(("docker-compose run excerpt python excerpt.py %s" % bbox_args).split(' ')) - subprocess.check_call("docker-compose stop --timeout 0".split(' ')) - subprocess.check_call("docker-compose rm -f".split(' ')) - except: - # inform_user() - raise - finally: - os.chdir(original_cwd) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 3bd606290..7c641faf5 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -51,6 +51,7 @@ def export_formats(): @staticmethod def export_options(): return { + # has to be implemented next: # 'coordinate_reference_system': { # 'label': 'Coordinate reference system', # 'type': 'choice', From 00222b18c2f0260d2cc394d0d4632263573da689 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 14:27:29 +0200 Subject: [PATCH 51/94] Refactoring: rename pseudo private field --- .../migrations/0005_auto_20150824_1408.py | 19 +++++++++++++++++++ .../osmaxx/excerptexport/models/excerpt.py | 6 +++--- osmaxx-py/osmaxx/excerptexport/views.py | 6 ++++-- 3 files changed, 26 insertions(+), 5 deletions(-) create mode 100644 osmaxx-py/osmaxx/excerptexport/migrations/0005_auto_20150824_1408.py diff --git a/osmaxx-py/osmaxx/excerptexport/migrations/0005_auto_20150824_1408.py b/osmaxx-py/osmaxx/excerptexport/migrations/0005_auto_20150824_1408.py new file mode 100644 index 000000000..392a8d8da --- /dev/null +++ b/osmaxx-py/osmaxx/excerptexport/migrations/0005_auto_20150824_1408.py @@ -0,0 +1,19 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.db import models, migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('excerptexport', '0004_auto_20150819_1417'), + ] + + operations = [ + migrations.RenameField( + model_name='excerpt', + old_name='_bounding_geometry', + new_name='bounding_geometry_raw_reference', + ), + ] diff --git a/osmaxx-py/osmaxx/excerptexport/models/excerpt.py b/osmaxx-py/osmaxx/excerptexport/models/excerpt.py index 3e3573d5b..fa2d9b2b9 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/excerpt.py +++ b/osmaxx-py/osmaxx/excerptexport/models/excerpt.py @@ -9,15 +9,15 @@ class Excerpt(models.Model): is_active = models.BooleanField(default=True, verbose_name=_('is active')) owner = models.ForeignKey(User, related_name='excerpts', verbose_name=_('owner')) - _bounding_geometry = models.OneToOneField('BoundingGeometry', verbose_name=_('bounding geometry')) + bounding_geometry_raw_reference = models.OneToOneField('BoundingGeometry', verbose_name=_('bounding geometry')) @property def bounding_geometry(self): - return self._bounding_geometry.geometry_instance + return self.bounding_geometry_raw_reference.geometry_instance @bounding_geometry.setter def bounding_geometry(self, bounding_geometry): - self._bounding_geometry = bounding_geometry + self.bounding_geometry_raw_reference = bounding_geometry def __str__(self): return self.name diff --git a/osmaxx-py/osmaxx/excerptexport/views.py b/osmaxx-py/osmaxx/excerptexport/views.py index 0c01f4fdc..6ced1ac3d 100644 --- a/osmaxx-py/osmaxx/excerptexport/views.py +++ b/osmaxx-py/osmaxx/excerptexport/views.py @@ -29,9 +29,11 @@ class NewExtractionOrderView(LoginRequiredMixin, FrontendAccessRequiredMixin, View): def get(self, request, excerpt_form_initial_data=None): active_excerpts = Excerpt.objects.filter(is_active=True) - active_bbox_excerpts = active_excerpts.filter(_bounding_geometry__bboxboundinggeometry__isnull=False) + active_bbox_excerpts = active_excerpts.filter( + bounding_geometry_raw_reference__bboxboundinggeometry__isnull=False + ) active_file_excerpts = active_excerpts.filter( - _bounding_geometry__osmosispolygonfilterboundinggeometry__isnull=False) + bounding_geometry_raw_reference__osmosispolygonfilterboundinggeometry__isnull=False) view_model = { 'user': request.user, 'export_options_form': ExportOptionsForm(ConverterManager.converter_configuration(), auto_id='%s'), From b6b743c91bdb87345a9a7aa765717930b0e47319 Mon Sep 17 00:00:00 2001 From: Wasabi Developer Date: Mon, 24 Aug 2015 14:30:00 +0200 Subject: [PATCH 52/94] Refactoring: Simplify huge expression --- .../excerptexport/static/excerptexport/scripts/map.js | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 3aac9710c..37e4a6c39 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -30,10 +30,11 @@ }; this.isSelectOptionSelectedAndExcerptOnMapInSyncWithInputFields = function(select, locationFilterBounds) { - if(select.value != "") { - var optionElement = select.querySelector('option[value="'+select.value+'"]'); + if(select.value == "") { + return false; } - return select.value != "" && + var optionElement = select.querySelector('option[value="'+select.value+'"]'); + return optionElement.getAttribute('data-north') == locationFilterBounds._northEast.lat && optionElement.getAttribute('data-west') == locationFilterBounds._southWest.lng && optionElement.getAttribute('data-east') == locationFilterBounds._northEast.lng && From 31ece18398e84430a8a0ecc4fc65f5fb678169b1 Mon Sep 17 00:00:00 2001 From: Nicola Date: Mon, 31 Aug 2015 09:45:21 +0200 Subject: [PATCH 53/94] missing newline at end of file --- osmaxx-py/osmaxx/excerptexport/models/extraction_order.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py index 5e8f0ab00..0046a1f41 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py +++ b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py @@ -67,4 +67,4 @@ def extraction_formats(self): extraction_formats = [] for export_format_key, export_format in self.extraction_configuration.items(): extraction_formats = extraction_formats + export_format['formats'] - return extraction_formats \ No newline at end of file + return extraction_formats From 38025ed4462483f882f799e503daf8863bc0ea29 Mon Sep 17 00:00:00 2001 From: Nicola Date: Mon, 31 Aug 2015 09:45:46 +0200 Subject: [PATCH 54/94] missing newline at end of file --- .../dummyexcerptconverter/dummy_excerpt_converter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 9bdbf23a4..de122cdff 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -123,4 +123,4 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # now set the new state (if all files have been processed) and inform the user about the state converter_helper.file_conversion_finished() except: - converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), False) \ No newline at end of file + converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), False) From 45695de46a3c19917735f9c6cde4a1f3344eb8f3 Mon Sep 17 00:00:00 2001 From: Nicola Date: Mon, 31 Aug 2015 09:46:00 +0200 Subject: [PATCH 55/94] missing newline at end of file --- osmaxx-py/excerptconverter/converter_helper.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 6586962df..cba696bc0 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -46,4 +46,4 @@ def inform_user(self, message_type, message_text, email=True): _("There is no email address assigned to your account. " "You won't be notified by email on process finish!"), email=False - ) \ No newline at end of file + ) From bce503e0ae556471469fef989ee2ce61409244f7 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Mon, 31 Aug 2015 10:36:18 +0200 Subject: [PATCH 56/94] Add TODO comments about not having dummy in common settings --- osmaxx-py/config/settings/common.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/osmaxx-py/config/settings/common.py b/osmaxx-py/config/settings/common.py index 6fd0ceabc..f37cae221 100644 --- a/osmaxx-py/config/settings/common.py +++ b/osmaxx-py/config/settings/common.py @@ -50,10 +50,12 @@ LOCAL_APPS = ( 'osmaxx.excerptexport', 'osmaxx.social_auth', + # TODO: Remove dummy converter or move out of common settings (e.g. to test-specific settings) 'excerptconverter.dummyexcerptconverter', 'excerptconverter.gisexcerptconverter' ) CELERY_IMPORTS = [ + # TODO: Remove dummy converter or move out of common settings (e.g. to test-specific settings) 'excerptconverter.dummyexcerptconverter.dummy_excerpt_converter', 'excerptconverter.gisexcerptconverter.gis_excerpt_converter' ] From 61cccf3a65eb0704ffa8cd1d41a7a37778b5b61a Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Mon, 31 Aug 2015 11:09:09 +0200 Subject: [PATCH 57/94] remove commented-out import --- osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py | 1 - 1 file changed, 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py b/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py index 87fcaad02..2da02b729 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/__init__.py @@ -1,4 +1,3 @@ -#from .execute_converting_blackbox import PostGisExcerptConverter from excerptconverter.baseexcerptconverter import BaseExcerptConverter from .gis_excerpt_converter import GisExcerptConverter From 93c99caf8cff87f4c70219e38394623c5ef419e5 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Thu, 3 Sep 2015 13:05:28 +0200 Subject: [PATCH 58/94] relative path is implicit --- compose-development.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/compose-development.yml b/compose-development.yml index d619f3e36..e7dbdd529 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -15,7 +15,7 @@ shareddatadev: sourcedev: build: docker/volume-mount volumes: - - ./osmaxx-py:/home/osmaxx/source + - osmaxx-py:/home/osmaxx/source - ~/.cache/pip:/home/osmaxx/.cache/pip webappdev: From c9361adb0c55b7a44876c5732ede5f209b1755ac Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Thu, 3 Sep 2015 13:07:57 +0200 Subject: [PATCH 59/94] use DJANGO_RESULT_MEDIA_ROOT from environment --- compose-common.yml | 1 + osmaxx-py/config/settings/common.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/compose-common.yml b/compose-common.yml index 4d9b5c596..07d98b494 100755 --- a/compose-common.yml +++ b/compose-common.yml @@ -8,6 +8,7 @@ base: - DJANGO_STATIC_ROOT=/data/static - DJANGO_MEDIA_ROOT=/data/media - DJANGO_PRIVATE_MEDIA_ROOT=/data/private_media + - DJANGO_RESULT_MEDIA_ROOT=/results # --link some-rabbit:rabbit "just works" - DJANGO_CELERY_BROKER_URL=amqp://guest@rabbit - DJANGO_DATABASE_URL=postgis://postgres@database/postgres diff --git a/osmaxx-py/config/settings/common.py b/osmaxx-py/config/settings/common.py index f37cae221..69778ae4e 100644 --- a/osmaxx-py/config/settings/common.py +++ b/osmaxx-py/config/settings/common.py @@ -202,7 +202,7 @@ MEDIA_ROOT = env.str('DJANGO_MEDIA_ROOT', default=str(ROOT_DIR('..', 'media'))) PRIVATE_MEDIA_ROOT = env.str('DJANGO_PRIVATE_MEDIA_ROOT', default=str(ROOT_DIR.path('..', 'private_media'))) -RESULT_MEDIA_ROOT = '/results' +RESULT_MEDIA_ROOT = env.str('DJANGO_RESULT_MEDIA_ROOT', default=str(ROOT_DIR.path('..', 'results_media'))) # See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url MEDIA_URL = '/media/' From 42c9fff118d09d44b889fad619fe907c570fac8c Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Thu, 3 Sep 2015 13:11:56 +0200 Subject: [PATCH 60/94] fix flake8 warnings --- .../baseexcerptconverter/base_excerpt_converter.py | 4 ---- osmaxx-py/excerptconverter/converter_helper.py | 1 - osmaxx-py/excerptconverter/converter_manager.py | 2 +- .../dummyexcerptconverter/dummy_excerpt_converter.py | 3 ++- .../gisexcerptconverter/gis_excerpt_converter.py | 11 +++++++---- .../tests/test_gis_excerpt_converter.py | 2 -- 6 files changed, 10 insertions(+), 13 deletions(-) diff --git a/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py b/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py index 0b7a3b10f..acc8e359f 100644 --- a/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/baseexcerptconverter/base_excerpt_converter.py @@ -2,10 +2,6 @@ from celery import shared_task -from django.utils.translation import ugettext_lazy as _ - -from osmaxx.excerptexport import models - class BaseExcerptConverter(metaclass=abstract_base_class.ABCMeta): available_converters = [] diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index cba696bc0..2d182c0e5 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -6,7 +6,6 @@ from osmaxx.excerptexport import models -from excerptconverter.baseexcerptconverter import BaseExcerptConverter class ConverterHelper: def __init__(self, extraction_order): diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index 12437dc95..01663dc5f 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -30,7 +30,7 @@ def __init__(self, extraction_order, def execute_converters(self): for Converter in self.available_converters: if (Converter.__name__ in self.extraction_order.extraction_configuration and - len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0): + len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0): Converter.execute( self.extraction_order, diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index de122cdff..fd759e644 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -1,5 +1,6 @@ -import time import os +import sys +import time import traceback from celery import shared_task diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 7c641faf5..285ce9bef 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -10,7 +10,6 @@ from django.conf import settings from django.utils.translation import ugettext_lazy as _ -from django.core.files import File from django.core.files.storage import FileSystemStorage from django.contrib import messages @@ -160,7 +159,7 @@ def create_output_file(extraction_order, result_file_name): result_file_path = os.path.abspath(os.path.join(settings.RESULT_MEDIA_ROOT, result_file_name)) target_file_path = os.path.abspath(os.path.join(private_storage.location, file_name)) - shutil.move(result_file_path,target_file_path) + shutil.move(result_file_path, target_file_path) output_file.file = private_storage.open(target_file_path) output_file.save() @@ -181,7 +180,6 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise - converter_helper = ConverterHelper(extraction_order) extraction_order.state = models.ExtractionOrderState.WAITING extraction_order.save() @@ -220,7 +218,12 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config bbox_args).split(' ')) extraction_order.state = models.ExtractionOrderState.PROCESSING extraction_order.save() - GisExcerptConverter.extract_excerpts(execution_configuration, extraction_order, bbox_args, converter_helper) + GisExcerptConverter.extract_excerpts( + execution_configuration, + extraction_order, + bbox_args, + converter_helper + ) elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: converter_helper.inform_user( diff --git a/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py b/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py index 58c03312d..bd4eeafd7 100644 --- a/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/tests/test_gis_excerpt_converter.py @@ -1,7 +1,5 @@ from django.test.testcases import TestCase -from excerptconverter.gisexcerptconverter import GisExcerptConverter -from excerptconverter.baseexcerptconverter import BaseExcerptConverter class GisExcerptConverterTestCase(TestCase): pass From 9cc6cbe2a2bc5fc787008c964527be8d92746972 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Thu, 3 Sep 2015 13:16:40 +0200 Subject: [PATCH 61/94] use boolean flags as named arguments --- osmaxx-py/excerptconverter/converter_helper.py | 2 +- .../dummy_excerpt_converter.py | 8 ++++---- .../gisexcerptconverter/gis_excerpt_converter.py | 14 +++++++------- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 2d182c0e5..5e50fc3e8 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -19,7 +19,7 @@ def file_conversion_finished(self): _('The extraction of the order "%(order_id)s" has been finished.') % { 'order_id': self.extraction_order.id }, - True + email=True ) self.extraction_order.state = models.ExtractionOrderState.FINISHED self.extraction_order.save() diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index fd759e644..f79547228 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -76,10 +76,10 @@ def create_output_files(execution_configuration, extraction_order, supported_exp if private_storage.exists(file_name): message_text = _('"%s" created successful' % file_name) - converter_helper.inform_user(messages.SUCCESS, message_text, False) + converter_helper.inform_user(messages.SUCCESS, message_text, email=False) else: message_text = _('Creation of "%s" failed!' % file_name) - converter_helper.inform_user(messages.ERROR, message_text, False) + converter_helper.inform_user(messages.ERROR, message_text, email=False) @staticmethod @shared_task @@ -111,7 +111,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config extraction_order.save() message_text = _('The Dummy conversion of extraction order "%s" has been started.') % extraction_order.id - converter_helper.inform_user(messages.INFO, message_text, False) + converter_helper.inform_user(messages.INFO, message_text, email=False) DummyExcerptConverter.create_output_files( execution_configuration, @@ -124,4 +124,4 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # now set the new state (if all files have been processed) and inform the user about the state converter_helper.file_conversion_finished() except: - converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), False) + converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 285ce9bef..55bb854f2 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -119,7 +119,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve 'converter_name': GisExcerptConverter.name(), 'order_id': extraction_order.id }, - False + email=False ) else: converter_helper.inform_user( @@ -128,7 +128,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve 'file': result_file_name, 'order_id': extraction_order.id }, - False + email=False ) else: converter_helper.inform_user( @@ -137,7 +137,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve 'file_type': export_format_config['name'], 'order_id': extraction_order.id }, - False + email=False ) @staticmethod @@ -200,7 +200,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config converter_helper.inform_user( messages.INFO, _('The GIS extraction of the order "%s" is has been started.') % extraction_order.id, - False + email=False ) bounding_geometry = extraction_order.excerpt.bounding_geometry @@ -229,7 +229,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract polygon excerpts.'), - False + email=False ) else: @@ -237,7 +237,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config messages.ERROR, _('GIS excerpt converter is not yet able to extract excerpts of type %s.') % type(bounding_geometry).__name__, - False + email=False ) subprocess.check_call("docker-compose stop --timeout 0".split(' ')) @@ -252,7 +252,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config 'order_id': extraction_order.id, 'error': traceback.format_tb(sys.exc_info()) }, - False + email=False ) raise finally: From d9f7d113a0a0a4ffcc27533b741729d60a913f67 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Thu, 3 Sep 2015 14:45:50 +0200 Subject: [PATCH 62/94] use __all__ for __init__.py --- osmaxx-py/excerptconverter/__init__.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/__init__.py b/osmaxx-py/excerptconverter/__init__.py index 897e42f45..9f7d3d1c3 100644 --- a/osmaxx-py/excerptconverter/__init__.py +++ b/osmaxx-py/excerptconverter/__init__.py @@ -1,3 +1,5 @@ -# flake8: noqa from .converter_manager import ConverterManager -from .converter_helper import ConverterHelper + +__all__ = [ + ConverterManager +] From a630554c12ed1c4286c800a78e744d22b1e27595 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Thu, 3 Sep 2015 17:27:15 +0200 Subject: [PATCH 63/94] make tests run again --- README.md | 12 ++++++++---- compose-development.yml | 2 +- test.sh | 22 +++++++++++++++------- 3 files changed, 24 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 9507d9bcd..7ad0028a9 100644 --- a/README.md +++ b/README.md @@ -48,26 +48,30 @@ Then initiate the project defaults by running the following command: ```shell # For development: -docker-compose run webappdev /bin/bash -c 'python3 manage.py migrate && python3 manage.py createsuperuser' +docker-compose up -d databasedev; +docker-compose run webappdev /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' # For production: -docker-compose run webapp /bin/bash -c 'python3 manage.py migrate && python3 manage.py createsuperuser' +docker-compose up -d databaseprod; +docker-compose run webapp /bin/bash -c './manage.py migrate && ./manage.py createsuperuser' ``` Alternative to this command, bootstrap the container and execute the commands inside the container by hand: ```shell # For development: +docker-compose up -d databasedev docker-compose run webappdev /bin/bash # For production: +docker-compose up -d database docker-compose run webapp /bin/bash ``` Inside the container: -1. Execute migrations: `$ python3 manage.py migrate` -2. (optional, recommended) setup a superuser: `$ python3 manage.py createsuperuser` +1. Execute migrations: `$ ./manage.py migrate` +2. (optional, recommended) setup a superuser: `$ ./manage.py createsuperuser` ### Running the project diff --git a/compose-development.yml b/compose-development.yml index bcac3cd90..23a5c1e58 100755 --- a/compose-development.yml +++ b/compose-development.yml @@ -13,7 +13,7 @@ shareddatadev: sourcedev: build: docker/volume-mount volumes: - - osmaxx-py:/home/osmaxx/source + - ./osmaxx-py:/home/osmaxx/source - ~/.cache/pip:/home/osmaxx/.cache/pip webappdev: diff --git a/test.sh b/test.sh index b9dc49b77..003899f48 100755 --- a/test.sh +++ b/test.sh @@ -63,10 +63,14 @@ function setup() { # does the same as reset, but it makes the execution of the tests more readable. echo '' > ${LOGFILE} reset_containers; + docker_compose up -d ${DB_CONTAINER}; + sleep 3; } function reset() { reset_containers; + docker_compose up -d ${DB_CONTAINER}; + sleep 3; } function tear_down() { @@ -75,14 +79,14 @@ function tear_down() { function reset_containers() { docker_compose stop -t 0 &>> ${LOGFILE}; - docker_compose rm -f &>> ${LOGFILE}; + docker_compose rm -vf &>> ${LOGFILE}; docker_compose build &>> ${LOGFILE}; } function reset_container() { local CONTAINER_TO_BE_RESETTED=$1 docker_compose stop -t 0 ${CONTAINER_TO_BE_RESETTED} &>> ${LOGFILE}; - docker_compose rm -f ${CONTAINER_TO_BE_RESETTED} &>> ${LOGFILE}; + docker_compose rm -vf ${CONTAINER_TO_BE_RESETTED} &>> ${LOGFILE}; docker_compose build ${CONTAINER_TO_BE_RESETTED} &>> ${LOGFILE}; } @@ -154,20 +158,24 @@ function docker_volume_configuration_tests() { } function persisting_database_data_tests() { - - if docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then - log "${RED}Migrations could not be applied!${RESET}" - else + docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' > /tmp/irgendesfile + if grep -q 'Applying excerptexport' /tmp/irgendesfile; then log "${GREEN}Migrations applied successfully.${RESET}" + else + log "${RED}Migrations could not be applied!${RESET}" fi reset_container ${DB_CONTAINER}; + docker_compose up -d ${DB_CONTAINER}; + + docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' > /tmp/irgendesfile - if docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' | grep -q 'No migrations to apply'; then + if grep -q 'No migrations to apply' /tmp/irgendesfile; then log "${GREEN}Database migrations retained correctly.${RESET}" else log "${RED}Database migrations not retained, data only container not working correctly!${RESET}" fi + rm -f /tmp/irgendesfile; } main; From 323878f734c4f4336f25768efd0ebe9a04c560be Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Fri, 4 Sep 2015 08:52:37 +0200 Subject: [PATCH 64/94] use a better named file for migration test results --- test.sh | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/test.sh b/test.sh index 003899f48..fa0ef9947 100755 --- a/test.sh +++ b/test.sh @@ -158,8 +158,10 @@ function docker_volume_configuration_tests() { } function persisting_database_data_tests() { - docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' > /tmp/irgendesfile - if grep -q 'Applying excerptexport' /tmp/irgendesfile; then + MIGRATION_RESULT_FILE='/tmp/temporary_osmaxx_migration_result' + docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' > ${MIGRATION_RESULT_FILE} + + if grep -q 'Applying excerptexport' ${MIGRATION_RESULT_FILE}; then log "${GREEN}Migrations applied successfully.${RESET}" else log "${RED}Migrations could not be applied!${RESET}" @@ -168,14 +170,14 @@ function persisting_database_data_tests() { reset_container ${DB_CONTAINER}; docker_compose up -d ${DB_CONTAINER}; - docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' > /tmp/irgendesfile + docker_compose run $WEBAPP_CONTAINER bash -c './manage.py migrate' > ${MIGRATION_RESULT_FILE} - if grep -q 'No migrations to apply' /tmp/irgendesfile; then + if grep -q 'No migrations to apply' ${MIGRATION_RESULT_FILE}; then log "${GREEN}Database migrations retained correctly.${RESET}" else log "${RED}Database migrations not retained, data only container not working correctly!${RESET}" fi - rm -f /tmp/irgendesfile; + rm -f ${MIGRATION_RESULT_FILE}; } main; From b92907fe5b616c45feade288c1c6ba94cdb6e33a Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Fri, 4 Sep 2015 09:04:47 +0200 Subject: [PATCH 65/94] small test optimization to ensure the newest images are being used --- test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test.sh b/test.sh index fa0ef9947..692f4333a 100755 --- a/test.sh +++ b/test.sh @@ -60,8 +60,8 @@ function run_production_tests() { } function setup() { - # does the same as reset, but it makes the execution of the tests more readable. echo '' > ${LOGFILE} + docker_compose pull; reset_containers; docker_compose up -d ${DB_CONTAINER}; sleep 3; From 1db75f3d3d78a4c969d69e751f015a06c9a9578b Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Fri, 4 Sep 2015 09:26:01 +0200 Subject: [PATCH 66/94] use free function instead of classes --- .../excerptconverter/converter_helper.py | 69 +++++++++---------- .../dummy_excerpt_converter.py | 19 ++--- .../gis_excerpt_converter.py | 31 +++++---- 3 files changed, 61 insertions(+), 58 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 5e50fc3e8..635d522ae 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -7,42 +7,39 @@ from osmaxx.excerptexport import models -class ConverterHelper: - def __init__(self, extraction_order): - self.extraction_order = extraction_order - self.user = extraction_order.orderer +def file_conversion_finished(extraction_order): + if extraction_order.output_files.count() >= len(extraction_order.extraction_formats): + inform_user( + extraction_order.orderer, + messages.SUCCESS, + _('The extraction of the order "%(order_id)s" has been finished.') % { + 'order_id': extraction_order.id + }, + email=True + ) + extraction_order.state = models.ExtractionOrderState.FINISHED + extraction_order.save() - def file_conversion_finished(self): - if self.extraction_order.output_files.count() >= len(self.extraction_order.extraction_formats): - self.inform_user( - messages.SUCCESS, - _('The extraction of the order "%(order_id)s" has been finished.') % { - 'order_id': self.extraction_order.id - }, - email=True - ) - self.extraction_order.state = models.ExtractionOrderState.FINISHED - self.extraction_order.save() - def inform_user(self, message_type, message_text, email=True): - stored_messages.api.add_message_for( - users=[self.user], - level=message_type, - message_text=message_text - ) +def inform_user(user, message_type, message_text, email=True): + stored_messages.api.add_message_for( + users=[user], + level=message_type, + message_text=message_text + ) - if email: - if hasattr(self.user, 'email'): - send_mail( - '[OSMAXX] '+message_text, - message_text, - 'no-reply@osmaxx.hsr.ch', - [self.user.email] - ) - else: - self.inform_user( - messages.WARNING, - _("There is no email address assigned to your account. " - "You won't be notified by email on process finish!"), - email=False - ) + if email: + if hasattr(user, 'email'): + send_mail( + '[OSMAXX] '+message_text, + message_text, + 'no-reply@osmaxx.hsr.ch', + [user.email] + ) + else: + inform_user( + messages.WARNING, + _("There is no email address assigned to your account. " + "You won't be notified by email on process finish!"), + email=False + ) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index f79547228..448666b48 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -12,7 +12,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter -from excerptconverter import ConverterHelper +from excerptconverter.converter_helper import inform_user, file_conversion_finished from osmaxx.excerptexport import models @@ -55,7 +55,7 @@ def export_options(): } @staticmethod - def create_output_files(execution_configuration, extraction_order, supported_export_formats, converter_helper): + def create_output_files(execution_configuration, extraction_order, supported_export_formats): for format_key in execution_configuration['formats']: output_file = models.OutputFile.objects.create( mime_type=supported_export_formats[format_key]['mime_type'], @@ -75,11 +75,13 @@ def create_output_files(execution_configuration, extraction_order, supported_exp output_file.save() if private_storage.exists(file_name): + message_level = messages.SUCCESS message_text = _('"%s" created successful' % file_name) - converter_helper.inform_user(messages.SUCCESS, message_text, email=False) else: + message_level = messages.ERROR message_text = _('Creation of "%s" failed!' % file_name) - converter_helper.inform_user(messages.ERROR, message_text, email=False) + user = extraction_order.orderer + inform_user(user, message_level, message_text, email=False) @staticmethod @shared_task @@ -97,7 +99,6 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config raise try: - converter_helper = ConverterHelper(extraction_order) fake_work_waiting_time_in_seconds = 5 # now set the new state @@ -111,17 +112,17 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config extraction_order.save() message_text = _('The Dummy conversion of extraction order "%s" has been started.') % extraction_order.id - converter_helper.inform_user(messages.INFO, message_text, email=False) + inform_user(extraction_order.orderer, messages.INFO, message_text, email=False) DummyExcerptConverter.create_output_files( execution_configuration, extraction_order, - supported_export_formats, converter_helper + supported_export_formats ) time.sleep(fake_work_waiting_time_in_seconds) # now set the new state (if all files have been processed) and inform the user about the state - converter_helper.file_conversion_finished() + file_conversion_finished(extraction_order) except: - converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) + inform_user(extraction_order.orderer, messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 55bb854f2..4994d1f47 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -14,7 +14,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter -from excerptconverter import ConverterHelper +from excerptconverter.converter_helper import inform_user, file_conversion_finished from osmaxx.excerptexport import models @@ -85,7 +85,7 @@ def export_options(): } @staticmethod - def extract_excerpts(execution_configuration, extraction_order, bbox_args, converter_helper): + def extract_excerpts(execution_configuration, extraction_order, bbox_args): """ Extract excerpt for chosen formats (execution_configuration) using docker-compose to trigger the conversion process (defined in blackbox/docker-compose-conversion-blackbox.yml) @@ -109,7 +109,8 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve for result_file_name in os.listdir(settings.RESULT_MEDIA_ROOT): # gis files are packaged in a zip file if GisExcerptConverter.create_output_file(extraction_order, result_file_name): - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.SUCCESS, _('Extraction of "%(file_type)s" of extraction order "%(order_id)s" was successful. ' '(File %(file_index)s of %(number_of_files)s of %(converter_name)s converter)') % { @@ -122,7 +123,8 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve email=False ) else: - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.ERROR, _('The extraction of "%(file)s" of extraction order "%(order_id)s" failed.') % { 'file': result_file_name, @@ -131,7 +133,8 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve email=False ) else: - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.ERROR, _('The extraction of "%(file_type)s" of extraction order "%(order_id)s" failed.') % { 'file_type': export_format_config['name'], @@ -180,7 +183,6 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise - converter_helper = ConverterHelper(extraction_order) extraction_order.state = models.ExtractionOrderState.WAITING extraction_order.save() @@ -197,7 +199,8 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # database needs time to be ready subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.INFO, _('The GIS extraction of the order "%s" is has been started.') % extraction_order.id, email=False @@ -221,19 +224,20 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config GisExcerptConverter.extract_excerpts( execution_configuration, extraction_order, - bbox_args, - converter_helper + bbox_args ) elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.ERROR, _('GIS excerpt converter is not yet able to extract polygon excerpts.'), email=False ) else: - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.ERROR, _('GIS excerpt converter is not yet able to extract excerpts of type %s.') % type(bounding_geometry).__name__, @@ -243,9 +247,10 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config subprocess.check_call("docker-compose stop --timeout 0".split(' ')) subprocess.check_call("docker-compose rm -f".split(' ')) - converter_helper.file_conversion_finished() + file_conversion_finished(extraction_order.orderer) except: - converter_helper.inform_user( + inform_user( + extraction_order.orderer, messages.ERROR, _('The extraction of order %(order_id)s failed: %(error)s. ' 'Please contact an administrator.') % { From c69312947d74a51531c74ba2cc5c75c137c51655 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Fri, 4 Sep 2015 09:34:57 +0200 Subject: [PATCH 67/94] small refactoring for test script --- test.sh | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/test.sh b/test.sh index 692f4333a..4cd89b617 100755 --- a/test.sh +++ b/test.sh @@ -63,14 +63,10 @@ function setup() { echo '' > ${LOGFILE} docker_compose pull; reset_containers; - docker_compose up -d ${DB_CONTAINER}; - sleep 3; } function reset() { reset_containers; - docker_compose up -d ${DB_CONTAINER}; - sleep 3; } function tear_down() { @@ -81,6 +77,8 @@ function reset_containers() { docker_compose stop -t 0 &>> ${LOGFILE}; docker_compose rm -vf &>> ${LOGFILE}; docker_compose build &>> ${LOGFILE}; + docker_compose up -d ${DB_CONTAINER}; + sleep 3; } function reset_container() { From f0b3800f16ad4ec150bf7281b8988fcd839f3b56 Mon Sep 17 00:00:00 2001 From: Nicola Jordan Date: Fri, 4 Sep 2015 11:17:48 +0200 Subject: [PATCH 68/94] fix new docker-compose startup sequence for gisconverter --- .../gisexcerptconverter/gis_excerpt_converter.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 4994d1f47..61258fd2d 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -196,8 +196,12 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config os.path.join(tmp_dir, 'docker-compose.yml') ) os.chdir(tmp_dir) - # database needs time to be ready - subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) + # use newest images + subprocess.check_output("docker-compose pull".split(' ')) + # database needs to be ready + subprocess.check_output("docker-compose up -d db".split(' ')) + # wait for the db to be up + subprocess.check_output("sleep 10".split(' ')) inform_user( extraction_order.orderer, @@ -245,7 +249,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config ) subprocess.check_call("docker-compose stop --timeout 0".split(' ')) - subprocess.check_call("docker-compose rm -f".split(' ')) + subprocess.check_call("docker-compose rm -vf".split(' ')) file_conversion_finished(extraction_order.orderer) except: From 308692d1e945597bfed775ef060320e9b8e2caa8 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 12:06:01 +0200 Subject: [PATCH 69/94] refactoring: extract method for condition --- osmaxx-py/excerptconverter/converter_manager.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index 01663dc5f..312c6d472 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -29,11 +29,14 @@ def __init__(self, extraction_order, def execute_converters(self): for Converter in self.available_converters: - if (Converter.__name__ in self.extraction_order.extraction_configuration and - len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0): + if self.is_needed_for_configuration(Converter): Converter.execute( self.extraction_order, self.extraction_order.extraction_configuration[Converter.__name__], self.run_as_celery_tasks ) + + def is_needed_for_configuration(self, Converter): + return (Converter.__name__ in self.extraction_order.extraction_configuration and + len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0) From 20b497831ca5b7fc2c2a3d0e9b715c249616d7fe Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 13:39:59 +0200 Subject: [PATCH 70/94] refactoring: extract converter filtering to list comprehension --- osmaxx-py/excerptconverter/converter_manager.py | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index 312c6d472..9dbf23555 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -28,14 +28,13 @@ def __init__(self, extraction_order, self.run_as_celery_tasks = run_as_celery_tasks def execute_converters(self): - for Converter in self.available_converters: - if self.is_needed_for_configuration(Converter): - - Converter.execute( - self.extraction_order, - self.extraction_order.extraction_configuration[Converter.__name__], - self.run_as_celery_tasks - ) + needed_converters = [C for C in self.available_converters if C.__name__ in self.is_needed_for_configuration(C)] + for Converter in needed_converters: + Converter.execute( + self.extraction_order, + self.extraction_order.extraction_configuration[Converter.__name__], + self.run_as_celery_tasks + ) def is_needed_for_configuration(self, Converter): return (Converter.__name__ in self.extraction_order.extraction_configuration and From 4804f054acb985e29520fe3a45c86385b0d0ea65 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 13:41:38 +0200 Subject: [PATCH 71/94] refactoring: extract list comprehension into method --- osmaxx-py/excerptconverter/converter_manager.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index 9dbf23555..a329efe0a 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -28,7 +28,7 @@ def __init__(self, extraction_order, self.run_as_celery_tasks = run_as_celery_tasks def execute_converters(self): - needed_converters = [C for C in self.available_converters if C.__name__ in self.is_needed_for_configuration(C)] + needed_converters = self._get_converters_from_configuration() for Converter in needed_converters: Converter.execute( self.extraction_order, @@ -36,6 +36,9 @@ def execute_converters(self): self.run_as_celery_tasks ) + def _get_converters_from_configuration(self): + return [C for C in self.available_converters if C.__name__ in self.is_needed_for_configuration(C)] + def is_needed_for_configuration(self, Converter): return (Converter.__name__ in self.extraction_order.extraction_configuration and len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0) From 4b68c53b4294810e7c44e2dc2d4d8f0f340b0093 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 13:43:39 +0200 Subject: [PATCH 72/94] refactoring: rename local variable --- osmaxx-py/excerptconverter/converter_manager.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index a329efe0a..d61856819 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -28,8 +28,8 @@ def __init__(self, extraction_order, self.run_as_celery_tasks = run_as_celery_tasks def execute_converters(self): - needed_converters = self._get_converters_from_configuration() - for Converter in needed_converters: + converters_in_configuration = self._get_converters_from_configuration() + for Converter in converters_in_configuration: Converter.execute( self.extraction_order, self.extraction_order.extraction_configuration[Converter.__name__], From a65a51e7994e60df83cb78242d8eff890530540e Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 13:48:07 +0200 Subject: [PATCH 73/94] refactoring: rename method --- osmaxx-py/excerptconverter/converter_manager.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index d61856819..de34dd8d9 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -37,8 +37,9 @@ def execute_converters(self): ) def _get_converters_from_configuration(self): - return [C for C in self.available_converters if C.__name__ in self.is_needed_for_configuration(C)] + return [C for C in self.available_converters + if C.__name__ in self._config_has_formats_provided_by_converter(C)] - def is_needed_for_configuration(self, Converter): + def _config_has_formats_provided_by_converter(self, Converter): return (Converter.__name__ in self.extraction_order.extraction_configuration and len(self.extraction_order.extraction_configuration[Converter.__name__]['formats']) > 0) From 7b098a1391719401b9a3605ac4d152c4e1d96569 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 14:01:56 +0200 Subject: [PATCH 74/94] re-raise caught exceptions in dummy converter after reporting them to the user --- .../dummyexcerptconverter/dummy_excerpt_converter.py | 1 + 1 file changed, 1 insertion(+) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 448666b48..2b7c0bc5b 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -126,3 +126,4 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config file_conversion_finished(extraction_order) except: inform_user(extraction_order.orderer, messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) + raise From cbe1866d3376eb563c692887677f325984a24332 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 14:09:36 +0200 Subject: [PATCH 75/94] don't leak stacktraces to end users --- .../dummyexcerptconverter/dummy_excerpt_converter.py | 5 ++--- .../gisexcerptconverter/gis_excerpt_converter.py | 7 ++----- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 2b7c0bc5b..2cc06b587 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -1,7 +1,5 @@ import os -import sys import time -import traceback from celery import shared_task @@ -125,5 +123,6 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # now set the new state (if all files have been processed) and inform the user about the state file_conversion_finished(extraction_order) except: - inform_user(extraction_order.orderer, messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) + message_text = _('The Dummy conversion of extraction order "%s" failed.') % extraction_order.id + inform_user(extraction_order.orderer, messages.ERROR, message_text, email=False) raise diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 61258fd2d..a8774e2bf 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -3,8 +3,6 @@ import subprocess import tempfile import time -import sys -import traceback from celery import shared_task @@ -256,10 +254,9 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config inform_user( extraction_order.orderer, messages.ERROR, - _('The extraction of order %(order_id)s failed: %(error)s. ' + _('The extraction of order %(order_id)s failed. ' 'Please contact an administrator.') % { - 'order_id': extraction_order.id, - 'error': traceback.format_tb(sys.exc_info()) + 'order_id': extraction_order.id }, email=False ) From 5cc37233813fd76dc197ef957f4bda8e6b81d7c1 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 14:29:52 +0200 Subject: [PATCH 76/94] fix list comprehension condition --- osmaxx-py/excerptconverter/converter_manager.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/converter_manager.py b/osmaxx-py/excerptconverter/converter_manager.py index 9dbf23555..5c1c4714a 100644 --- a/osmaxx-py/excerptconverter/converter_manager.py +++ b/osmaxx-py/excerptconverter/converter_manager.py @@ -28,7 +28,7 @@ def __init__(self, extraction_order, self.run_as_celery_tasks = run_as_celery_tasks def execute_converters(self): - needed_converters = [C for C in self.available_converters if C.__name__ in self.is_needed_for_configuration(C)] + needed_converters = [C for C in self.available_converters if self.is_needed_for_configuration(C)] for Converter in needed_converters: Converter.execute( self.extraction_order, From 846a81c73b21b627d2eff0502e7b109e69be545d Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Fri, 4 Sep 2015 14:38:16 +0200 Subject: [PATCH 77/94] centralize creation of private_storage --- .../dummyexcerptconverter/dummy_excerpt_converter.py | 6 +----- .../gisexcerptconverter/gis_excerpt_converter.py | 5 +---- osmaxx-py/osmaxx/excerptexport/views.py | 5 +---- osmaxx-py/osmaxx/utils.py | 4 ++++ 4 files changed, 7 insertions(+), 13 deletions(-) create mode 100644 osmaxx-py/osmaxx/utils.py diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 2cc06b587..445a0f3b6 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -3,8 +3,6 @@ from celery import shared_task -from django.conf import settings -from django.core.files.storage import FileSystemStorage from django.core.files.base import ContentFile from django.utils.translation import ugettext_lazy as _ from django.contrib import messages @@ -13,9 +11,7 @@ from excerptconverter.converter_helper import inform_user, file_conversion_finished from osmaxx.excerptexport import models - - -private_storage = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT) +from osmaxx.utils import private_storage class DummyExcerptConverter(BaseExcerptConverter): diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index a8774e2bf..207b9027e 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -8,16 +8,13 @@ from django.conf import settings from django.utils.translation import ugettext_lazy as _ -from django.core.files.storage import FileSystemStorage from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter from excerptconverter.converter_helper import inform_user, file_conversion_finished from osmaxx.excerptexport import models - - -private_storage = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT) +from osmaxx.utils import private_storage class GisExcerptConverter(BaseExcerptConverter): diff --git a/osmaxx-py/osmaxx/excerptexport/views.py b/osmaxx-py/osmaxx/excerptexport/views.py index 6ced1ac3d..3e3706901 100644 --- a/osmaxx-py/osmaxx/excerptexport/views.py +++ b/osmaxx-py/osmaxx/excerptexport/views.py @@ -9,8 +9,6 @@ from django.contrib import messages from django.utils.translation import ugettext_lazy as _ from django.views.generic import View -from django.core.files.storage import FileSystemStorage -from django.conf import settings from .models import ExtractionOrder, Excerpt, OutputFile, BBoxBoundingGeometry from .models.extraction_order import ExtractionOrderState @@ -22,8 +20,7 @@ from . import settings as excerptexport_settings from .forms import ExportOptionsForm, NewExcerptForm from excerptconverter import ConverterManager - -private_storage = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT) +from osmaxx.utils import private_storage class NewExtractionOrderView(LoginRequiredMixin, FrontendAccessRequiredMixin, View): diff --git a/osmaxx-py/osmaxx/utils.py b/osmaxx-py/osmaxx/utils.py new file mode 100644 index 000000000..cb617e290 --- /dev/null +++ b/osmaxx-py/osmaxx/utils.py @@ -0,0 +1,4 @@ +from django.core.files.storage import FileSystemStorage +from django.conf import settings + +private_storage = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT) From ce34e9c859f82af5a2fd3ca8e23bebf248ffb0a6 Mon Sep 17 00:00:00 2001 From: Raphael Das Gupta Date: Thu, 10 Sep 2015 15:31:58 +0200 Subject: [PATCH 78/94] set order state to failed when an exception occurred --- .../gisexcerptconverter/gis_excerpt_converter.py | 3 +++ osmaxx-py/osmaxx/excerptexport/models/extraction_order.py | 1 + 2 files changed, 4 insertions(+) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 207b9027e..7f55e0691 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -248,6 +248,9 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config file_conversion_finished(extraction_order.orderer) except: + extraction_order.state = models.ExtractionOrderState.FAILED + extraction_order.save() + inform_user( extraction_order.orderer, messages.ERROR, diff --git a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py index 0046a1f41..415bc0097 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py +++ b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py @@ -16,6 +16,7 @@ class ExtractionOrderState(enum.Enum): PROCESSING = 3 FINISHED = 4 CANCELED = 5 + FAILED = 6 class ExtractionOrder(models.Model): From 24bfd76ba2c7d4256245ed3bb0be02507864b3cf Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Thu, 10 Sep 2015 17:40:36 +0200 Subject: [PATCH 79/94] Fix broken __all__ export --- osmaxx-py/excerptconverter/__init__.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/osmaxx-py/excerptconverter/__init__.py b/osmaxx-py/excerptconverter/__init__.py index 9f7d3d1c3..7ac164e1e 100644 --- a/osmaxx-py/excerptconverter/__init__.py +++ b/osmaxx-py/excerptconverter/__init__.py @@ -1,5 +1,4 @@ from .converter_manager import ConverterManager +from .converter_helper import ConverterHelper -__all__ = [ - ConverterManager -] +__all__ = ['ConverterManager', 'ConverterHelper'] From 8ecc35bbb79f2b72c17680680e20c9058cb3fa97 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 10:27:20 +0200 Subject: [PATCH 80/94] Fix form mode switching caused by semicolon insertion --- .../osmaxx/excerptexport/static/excerptexport/scripts/map.js | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 37e4a6c39..100243ed1 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -1,3 +1,4 @@ +'use strict'; (function(){ var ExcerptManager = function(locationFilter, inputElementsNewBoundingBox, selectElementExistingExcerpts, formElementPartsSwitcher) { this.locationFilter = locationFilter; @@ -34,11 +35,12 @@ return false; } var optionElement = select.querySelector('option[value="'+select.value+'"]'); - return + return ( optionElement.getAttribute('data-north') == locationFilterBounds._northEast.lat && optionElement.getAttribute('data-west') == locationFilterBounds._southWest.lng && optionElement.getAttribute('data-east') == locationFilterBounds._northEast.lng && optionElement.getAttribute('data-south') == locationFilterBounds._southWest.lat + ); } /** From 399f2b429edb650a8455dcd742c805621cc9db2f Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 10:53:37 +0200 Subject: [PATCH 81/94] Revert "use free function instead of classes" This reverts commit 1db75f3d3d78a4c969d69e751f015a06c9a9578b. --- .../excerptconverter/converter_helper.py | 69 ++++++++++--------- .../dummy_excerpt_converter.py | 19 +++-- .../gis_excerpt_converter.py | 31 ++++----- 3 files changed, 58 insertions(+), 61 deletions(-) diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 635d522ae..5e50fc3e8 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -7,39 +7,42 @@ from osmaxx.excerptexport import models -def file_conversion_finished(extraction_order): - if extraction_order.output_files.count() >= len(extraction_order.extraction_formats): - inform_user( - extraction_order.orderer, - messages.SUCCESS, - _('The extraction of the order "%(order_id)s" has been finished.') % { - 'order_id': extraction_order.id - }, - email=True - ) - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() +class ConverterHelper: + def __init__(self, extraction_order): + self.extraction_order = extraction_order + self.user = extraction_order.orderer + def file_conversion_finished(self): + if self.extraction_order.output_files.count() >= len(self.extraction_order.extraction_formats): + self.inform_user( + messages.SUCCESS, + _('The extraction of the order "%(order_id)s" has been finished.') % { + 'order_id': self.extraction_order.id + }, + email=True + ) + self.extraction_order.state = models.ExtractionOrderState.FINISHED + self.extraction_order.save() -def inform_user(user, message_type, message_text, email=True): - stored_messages.api.add_message_for( - users=[user], - level=message_type, - message_text=message_text - ) + def inform_user(self, message_type, message_text, email=True): + stored_messages.api.add_message_for( + users=[self.user], + level=message_type, + message_text=message_text + ) - if email: - if hasattr(user, 'email'): - send_mail( - '[OSMAXX] '+message_text, - message_text, - 'no-reply@osmaxx.hsr.ch', - [user.email] - ) - else: - inform_user( - messages.WARNING, - _("There is no email address assigned to your account. " - "You won't be notified by email on process finish!"), - email=False - ) + if email: + if hasattr(self.user, 'email'): + send_mail( + '[OSMAXX] '+message_text, + message_text, + 'no-reply@osmaxx.hsr.ch', + [self.user.email] + ) + else: + self.inform_user( + messages.WARNING, + _("There is no email address assigned to your account. " + "You won't be notified by email on process finish!"), + email=False + ) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 448666b48..f79547228 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -12,7 +12,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter -from excerptconverter.converter_helper import inform_user, file_conversion_finished +from excerptconverter import ConverterHelper from osmaxx.excerptexport import models @@ -55,7 +55,7 @@ def export_options(): } @staticmethod - def create_output_files(execution_configuration, extraction_order, supported_export_formats): + def create_output_files(execution_configuration, extraction_order, supported_export_formats, converter_helper): for format_key in execution_configuration['formats']: output_file = models.OutputFile.objects.create( mime_type=supported_export_formats[format_key]['mime_type'], @@ -75,13 +75,11 @@ def create_output_files(execution_configuration, extraction_order, supported_exp output_file.save() if private_storage.exists(file_name): - message_level = messages.SUCCESS message_text = _('"%s" created successful' % file_name) + converter_helper.inform_user(messages.SUCCESS, message_text, email=False) else: - message_level = messages.ERROR message_text = _('Creation of "%s" failed!' % file_name) - user = extraction_order.orderer - inform_user(user, message_level, message_text, email=False) + converter_helper.inform_user(messages.ERROR, message_text, email=False) @staticmethod @shared_task @@ -99,6 +97,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config raise try: + converter_helper = ConverterHelper(extraction_order) fake_work_waiting_time_in_seconds = 5 # now set the new state @@ -112,17 +111,17 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config extraction_order.save() message_text = _('The Dummy conversion of extraction order "%s" has been started.') % extraction_order.id - inform_user(extraction_order.orderer, messages.INFO, message_text, email=False) + converter_helper.inform_user(messages.INFO, message_text, email=False) DummyExcerptConverter.create_output_files( execution_configuration, extraction_order, - supported_export_formats + supported_export_formats, converter_helper ) time.sleep(fake_work_waiting_time_in_seconds) # now set the new state (if all files have been processed) and inform the user about the state - file_conversion_finished(extraction_order) + converter_helper.file_conversion_finished() except: - inform_user(extraction_order.orderer, messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) + converter_helper.inform_user(messages.ERROR, traceback.format_tb(sys.exc_info()), email=False) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 4994d1f47..55bb854f2 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -14,7 +14,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter -from excerptconverter.converter_helper import inform_user, file_conversion_finished +from excerptconverter import ConverterHelper from osmaxx.excerptexport import models @@ -85,7 +85,7 @@ def export_options(): } @staticmethod - def extract_excerpts(execution_configuration, extraction_order, bbox_args): + def extract_excerpts(execution_configuration, extraction_order, bbox_args, converter_helper): """ Extract excerpt for chosen formats (execution_configuration) using docker-compose to trigger the conversion process (defined in blackbox/docker-compose-conversion-blackbox.yml) @@ -109,8 +109,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): for result_file_name in os.listdir(settings.RESULT_MEDIA_ROOT): # gis files are packaged in a zip file if GisExcerptConverter.create_output_file(extraction_order, result_file_name): - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.SUCCESS, _('Extraction of "%(file_type)s" of extraction order "%(order_id)s" was successful. ' '(File %(file_index)s of %(number_of_files)s of %(converter_name)s converter)') % { @@ -123,8 +122,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): email=False ) else: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of "%(file)s" of extraction order "%(order_id)s" failed.') % { 'file': result_file_name, @@ -133,8 +131,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): email=False ) else: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of "%(file_type)s" of extraction order "%(order_id)s" failed.') % { 'file_type': export_format_config['name'], @@ -183,6 +180,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise + converter_helper = ConverterHelper(extraction_order) extraction_order.state = models.ExtractionOrderState.WAITING extraction_order.save() @@ -199,8 +197,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # database needs time to be ready subprocess.check_output("docker-compose run bootstrap sleep 10".split(' ')) - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.INFO, _('The GIS extraction of the order "%s" is has been started.') % extraction_order.id, email=False @@ -224,20 +221,19 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config GisExcerptConverter.extract_excerpts( execution_configuration, extraction_order, - bbox_args + bbox_args, + converter_helper ) elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract polygon excerpts.'), email=False ) else: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract excerpts of type %s.') % type(bounding_geometry).__name__, @@ -247,10 +243,9 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config subprocess.check_call("docker-compose stop --timeout 0".split(' ')) subprocess.check_call("docker-compose rm -f".split(' ')) - file_conversion_finished(extraction_order.orderer) + converter_helper.file_conversion_finished() except: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of order %(order_id)s failed: %(error)s. ' 'Please contact an administrator.') % { From 1c9f3b95cca8439ec8c4a5a5cb1959e8b2edaff2 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 11:04:23 +0200 Subject: [PATCH 82/94] Document reason for class instead of free function --- osmaxx-py/excerptconverter/converter_helper.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 5e50fc3e8..7ab3e2634 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -7,6 +7,8 @@ from osmaxx.excerptexport import models +# functions using database (extraction_order) must be instance methods of a class +# -> free functions will not work: database connection error class ConverterHelper: def __init__(self, extraction_order): self.extraction_order = extraction_order From 7acf60ca4335f1e93ec66a4cd617a6af8687d6d8 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 12:14:43 +0200 Subject: [PATCH 83/94] Merge fixes and review changes --- osmaxx-py/excerptconverter/__init__.py | 5 +- .../excerptconverter/converter_helper.py | 71 ++++++++++--------- .../dummy_excerpt_converter.py | 18 ++--- .../gis_excerpt_converter.py | 31 ++++---- .../static/excerptexport/scripts/map.js | 4 +- 5 files changed, 66 insertions(+), 63 deletions(-) diff --git a/osmaxx-py/excerptconverter/__init__.py b/osmaxx-py/excerptconverter/__init__.py index 9f7d3d1c3..7ac164e1e 100644 --- a/osmaxx-py/excerptconverter/__init__.py +++ b/osmaxx-py/excerptconverter/__init__.py @@ -1,5 +1,4 @@ from .converter_manager import ConverterManager +from .converter_helper import ConverterHelper -__all__ = [ - ConverterManager -] +__all__ = ['ConverterManager', 'ConverterHelper'] diff --git a/osmaxx-py/excerptconverter/converter_helper.py b/osmaxx-py/excerptconverter/converter_helper.py index 635d522ae..7ab3e2634 100644 --- a/osmaxx-py/excerptconverter/converter_helper.py +++ b/osmaxx-py/excerptconverter/converter_helper.py @@ -7,39 +7,44 @@ from osmaxx.excerptexport import models -def file_conversion_finished(extraction_order): - if extraction_order.output_files.count() >= len(extraction_order.extraction_formats): - inform_user( - extraction_order.orderer, - messages.SUCCESS, - _('The extraction of the order "%(order_id)s" has been finished.') % { - 'order_id': extraction_order.id - }, - email=True - ) - extraction_order.state = models.ExtractionOrderState.FINISHED - extraction_order.save() +# functions using database (extraction_order) must be instance methods of a class +# -> free functions will not work: database connection error +class ConverterHelper: + def __init__(self, extraction_order): + self.extraction_order = extraction_order + self.user = extraction_order.orderer + def file_conversion_finished(self): + if self.extraction_order.output_files.count() >= len(self.extraction_order.extraction_formats): + self.inform_user( + messages.SUCCESS, + _('The extraction of the order "%(order_id)s" has been finished.') % { + 'order_id': self.extraction_order.id + }, + email=True + ) + self.extraction_order.state = models.ExtractionOrderState.FINISHED + self.extraction_order.save() -def inform_user(user, message_type, message_text, email=True): - stored_messages.api.add_message_for( - users=[user], - level=message_type, - message_text=message_text - ) + def inform_user(self, message_type, message_text, email=True): + stored_messages.api.add_message_for( + users=[self.user], + level=message_type, + message_text=message_text + ) - if email: - if hasattr(user, 'email'): - send_mail( - '[OSMAXX] '+message_text, - message_text, - 'no-reply@osmaxx.hsr.ch', - [user.email] - ) - else: - inform_user( - messages.WARNING, - _("There is no email address assigned to your account. " - "You won't be notified by email on process finish!"), - email=False - ) + if email: + if hasattr(self.user, 'email'): + send_mail( + '[OSMAXX] '+message_text, + message_text, + 'no-reply@osmaxx.hsr.ch', + [self.user.email] + ) + else: + self.inform_user( + messages.WARNING, + _("There is no email address assigned to your account. " + "You won't be notified by email on process finish!"), + email=False + ) diff --git a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py index 445a0f3b6..385b944f0 100644 --- a/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py @@ -8,7 +8,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter -from excerptconverter.converter_helper import inform_user, file_conversion_finished +from excerptconverter import ConverterHelper from osmaxx.excerptexport import models from osmaxx.utils import private_storage @@ -49,7 +49,7 @@ def export_options(): } @staticmethod - def create_output_files(execution_configuration, extraction_order, supported_export_formats): + def create_output_files(execution_configuration, extraction_order, supported_export_formats, converter_helper): for format_key in execution_configuration['formats']: output_file = models.OutputFile.objects.create( mime_type=supported_export_formats[format_key]['mime_type'], @@ -74,8 +74,7 @@ def create_output_files(execution_configuration, extraction_order, supported_exp else: message_level = messages.ERROR message_text = _('Creation of "%s" failed!' % file_name) - user = extraction_order.orderer - inform_user(user, message_level, message_text, email=False) + converter_helper.inform_user(message_level, message_text, email=False) @staticmethod @shared_task @@ -93,6 +92,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config raise try: + converter_helper = ConverterHelper(extraction_order) fake_work_waiting_time_in_seconds = 5 # now set the new state @@ -106,19 +106,21 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config extraction_order.save() message_text = _('The Dummy conversion of extraction order "%s" has been started.') % extraction_order.id - inform_user(extraction_order.orderer, messages.INFO, message_text, email=False) + converter_helper.inform_user(messages.INFO, message_text, email=False) DummyExcerptConverter.create_output_files( execution_configuration, extraction_order, - supported_export_formats + supported_export_formats, + converter_helper ) time.sleep(fake_work_waiting_time_in_seconds) # now set the new state (if all files have been processed) and inform the user about the state - file_conversion_finished(extraction_order) + converter_helper.file_conversion_finished() except: + # TODO: log stack trace message_text = _('The Dummy conversion of extraction order "%s" failed.') % extraction_order.id - inform_user(extraction_order.orderer, messages.ERROR, message_text, email=False) + converter_helper.inform_user(messages.ERROR, message_text, email=False) raise diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 7f55e0691..cafe0160e 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -11,7 +11,7 @@ from django.contrib import messages from excerptconverter.baseexcerptconverter import BaseExcerptConverter -from excerptconverter.converter_helper import inform_user, file_conversion_finished +from excerptconverter import ConverterHelper from osmaxx.excerptexport import models from osmaxx.utils import private_storage @@ -80,7 +80,7 @@ def export_options(): } @staticmethod - def extract_excerpts(execution_configuration, extraction_order, bbox_args): + def extract_excerpts(execution_configuration, extraction_order, bbox_args, converter_helper): """ Extract excerpt for chosen formats (execution_configuration) using docker-compose to trigger the conversion process (defined in blackbox/docker-compose-conversion-blackbox.yml) @@ -104,8 +104,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): for result_file_name in os.listdir(settings.RESULT_MEDIA_ROOT): # gis files are packaged in a zip file if GisExcerptConverter.create_output_file(extraction_order, result_file_name): - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.SUCCESS, _('Extraction of "%(file_type)s" of extraction order "%(order_id)s" was successful. ' '(File %(file_index)s of %(number_of_files)s of %(converter_name)s converter)') % { @@ -118,8 +117,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): email=False ) else: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of "%(file)s" of extraction order "%(order_id)s" failed.') % { 'file': result_file_name, @@ -128,8 +126,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args): email=False ) else: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of "%(file_type)s" of extraction order "%(order_id)s" failed.') % { 'file_type': export_format_config['name'], @@ -178,6 +175,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config if wait_time > 30: raise + converter_helper = ConverterHelper(extraction_order) extraction_order.state = models.ExtractionOrderState.WAITING extraction_order.save() @@ -198,8 +196,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config # wait for the db to be up subprocess.check_output("sleep 10".split(' ')) - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.INFO, _('The GIS extraction of the order "%s" is has been started.') % extraction_order.id, email=False @@ -223,20 +220,19 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config GisExcerptConverter.extract_excerpts( execution_configuration, extraction_order, - bbox_args + bbox_args, + converter_helper ) elif type(bounding_geometry) == models.OsmosisPolygonFilterBoundingGeometry: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract polygon excerpts.'), email=False ) else: - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('GIS excerpt converter is not yet able to extract excerpts of type %s.') % type(bounding_geometry).__name__, @@ -246,13 +242,12 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config subprocess.check_call("docker-compose stop --timeout 0".split(' ')) subprocess.check_call("docker-compose rm -vf".split(' ')) - file_conversion_finished(extraction_order.orderer) + converter_helper.file_conversion_finished() except: extraction_order.state = models.ExtractionOrderState.FAILED extraction_order.save() - inform_user( - extraction_order.orderer, + converter_helper.inform_user( messages.ERROR, _('The extraction of order %(order_id)s failed. ' 'Please contact an administrator.') % { diff --git a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js index 37e4a6c39..100243ed1 100644 --- a/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js +++ b/osmaxx-py/osmaxx/excerptexport/static/excerptexport/scripts/map.js @@ -1,3 +1,4 @@ +'use strict'; (function(){ var ExcerptManager = function(locationFilter, inputElementsNewBoundingBox, selectElementExistingExcerpts, formElementPartsSwitcher) { this.locationFilter = locationFilter; @@ -34,11 +35,12 @@ return false; } var optionElement = select.querySelector('option[value="'+select.value+'"]'); - return + return ( optionElement.getAttribute('data-north') == locationFilterBounds._northEast.lat && optionElement.getAttribute('data-west') == locationFilterBounds._southWest.lng && optionElement.getAttribute('data-east') == locationFilterBounds._northEast.lng && optionElement.getAttribute('data-south') == locationFilterBounds._southWest.lat + ); } /** From 387bd444751a8214c69de33ae9ca89d4a4dc58ef Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 13:43:50 +0200 Subject: [PATCH 84/94] Improve user messages --- .../gisexcerptconverter/gis_excerpt_converter.py | 2 +- osmaxx-py/osmaxx/excerptexport/views.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index cafe0160e..b49734eab 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -107,7 +107,7 @@ def extract_excerpts(execution_configuration, extraction_order, bbox_args, conve converter_helper.inform_user( messages.SUCCESS, _('Extraction of "%(file_type)s" of extraction order "%(order_id)s" was successful. ' - '(File %(file_index)s of %(number_of_files)s of %(converter_name)s converter)') % { + '(of %(number_of_files)s files of %(converter_name)s converter)') % { 'file_type': export_format_config['name'], 'file_index': index, 'number_of_files': len(execution_configuration['formats']), diff --git a/osmaxx-py/osmaxx/excerptexport/views.py b/osmaxx-py/osmaxx/excerptexport/views.py index 3e3706901..ea6a4999b 100644 --- a/osmaxx-py/osmaxx/excerptexport/views.py +++ b/osmaxx-py/osmaxx/excerptexport/views.py @@ -97,7 +97,7 @@ def post(self, request): converter_manager.execute_converters() messages.success(request, _( - 'Successful creation of extraction order extraction order %(id)s. ' + 'Queued extraction order %(id)s. ' 'The conversion process will start soon.' ) % {'id': extraction_order.id}) return HttpResponseRedirect( From bd1157b5a4cf04faff9a4be29262f9975886197a Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 14:11:00 +0200 Subject: [PATCH 85/94] Change type of info message to info --- osmaxx-py/osmaxx/excerptexport/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/osmaxx/excerptexport/views.py b/osmaxx-py/osmaxx/excerptexport/views.py index ea6a4999b..6ab761bbd 100644 --- a/osmaxx-py/osmaxx/excerptexport/views.py +++ b/osmaxx-py/osmaxx/excerptexport/views.py @@ -96,7 +96,7 @@ def post(self, request): converter_manager = ConverterManager(extraction_order) converter_manager.execute_converters() - messages.success(request, _( + messages.info(request, _( 'Queued extraction order %(id)s. ' 'The conversion process will start soon.' ) % {'id': extraction_order.id}) From 0bee4ae8177484273781f27b340d8ab21d58cf1e Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 14:21:59 +0200 Subject: [PATCH 86/94] Refactoring: remove unused mime type declarations --- .../gisexcerptconverter/gis_excerpt_converter.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index b49734eab..d24e17f1a 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -27,18 +27,15 @@ def export_formats(): return { 'spatialite': { 'name': 'SpatiaLite (SQLite)', - 'file_extension': 'sqlite', - 'mime_type': 'application/x-sqlite3' + 'file_extension': 'sqlite' }, 'gpkg': { 'name': 'Geo package', - 'file_extension': 'gpkg', - 'mime_type': 'application/octet-stream' # Not verified! + 'file_extension': 'gpkg' }, 'shp': { 'name': 'Shape file', - 'file_extension': 'shp', - 'mime_type': 'application/octet-stream' # Not verified! + 'file_extension': 'shp' } } From 04f598f8f6758d26a7f06ef65950f8867daa8875 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 14:37:11 +0200 Subject: [PATCH 87/94] Remove print --- .../gisexcerptconverter/gis_excerpt_converter.py | 1 - 1 file changed, 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index d24e17f1a..339c409f9 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -180,7 +180,6 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config original_cwd = os.getcwd() try: - print(tmp_dir) shutil.copyfile( os.path.join(os.path.dirname(__file__), 'blackbox', 'docker-compose-conversion-blackbox.yml'), os.path.join(tmp_dir, 'docker-compose.yml') From c57316f275a5e8a2c60b3bff6558618d8f0d5e08 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 15:08:19 +0200 Subject: [PATCH 88/94] Document reason for missing list comprehension --- osmaxx-py/osmaxx/excerptexport/models/extraction_order.py | 1 + 1 file changed, 1 insertion(+) diff --git a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py index 415bc0097..ef544815e 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py +++ b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py @@ -67,5 +67,6 @@ def extraction_configuration(self, value): def extraction_formats(self): extraction_formats = [] for export_format_key, export_format in self.extraction_configuration.items(): + # merge lists to flat nested structure extraction_formats = extraction_formats + export_format['formats'] return extraction_formats From 3a500c1fc3070cbf2049f7c5812da47f56f2e580 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 15:18:50 +0200 Subject: [PATCH 89/94] Refactor code --- .../gisexcerptconverter/gis_excerpt_converter.py | 13 ++++++------- .../osmaxx/excerptexport/models/extraction_order.py | 4 ++-- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index 339c409f9..a393bf997 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -200,13 +200,12 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config bounding_geometry = extraction_order.excerpt.bounding_geometry if type(bounding_geometry) == models.BBoxBoundingGeometry: - bbox_args = '%(excerpt_south_border)s %(excerpt_west_border)s %(excerpt_north_border)s ' \ - '%(excerpt_east_border)s' % { - 'excerpt_north_border': bounding_geometry.north, - 'excerpt_west_border': bounding_geometry.west, - 'excerpt_south_border': bounding_geometry.south, - 'excerpt_east_border': bounding_geometry.east - } + bbox_args = ' '.join([ + bounding_geometry.south, + bounding_geometry.west, + bounding_geometry.north, + bounding_geometry.east + ]) if len(execution_configuration['formats']) > 0: subprocess.check_call(("docker-compose run bootstrap sh main-bootstrap.sh %s" % diff --git a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py index ef544815e..7ab946eb2 100644 --- a/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py +++ b/osmaxx-py/osmaxx/excerptexport/models/extraction_order.py @@ -66,7 +66,7 @@ def extraction_configuration(self, value): @property def extraction_formats(self): extraction_formats = [] - for export_format_key, export_format in self.extraction_configuration.items(): + for export_format_config in self.extraction_configuration.values(): # merge lists to flat nested structure - extraction_formats = extraction_formats + export_format['formats'] + extraction_formats = extraction_formats + export_format_config['formats'] return extraction_formats From d40d2d5d271cc7b01cf1ab55ec0462a3226dd2cc Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 16:57:02 +0200 Subject: [PATCH 90/94] Fix bbox_args (cast to string) --- .../gisexcerptconverter/gis_excerpt_converter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py index a393bf997..67446a34a 100644 --- a/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py +++ b/osmaxx-py/excerptconverter/gisexcerptconverter/gis_excerpt_converter.py @@ -200,7 +200,7 @@ def execute_task(extraction_order_id, supported_export_formats, execution_config bounding_geometry = extraction_order.excerpt.bounding_geometry if type(bounding_geometry) == models.BBoxBoundingGeometry: - bbox_args = ' '.join([ + bbox_args = ' '.join(str(coordinate) for coordinate in [ bounding_geometry.south, bounding_geometry.west, bounding_geometry.north, From edfdc4dbcb7dc66454042be111d9009c34f0d079 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Fri, 11 Sep 2015 17:02:43 +0200 Subject: [PATCH 91/94] Remove dummyexcerptconverter from common config --- osmaxx-py/config/settings/common.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/osmaxx-py/config/settings/common.py b/osmaxx-py/config/settings/common.py index 69778ae4e..1a4406e7c 100644 --- a/osmaxx-py/config/settings/common.py +++ b/osmaxx-py/config/settings/common.py @@ -50,13 +50,9 @@ LOCAL_APPS = ( 'osmaxx.excerptexport', 'osmaxx.social_auth', - # TODO: Remove dummy converter or move out of common settings (e.g. to test-specific settings) - 'excerptconverter.dummyexcerptconverter', 'excerptconverter.gisexcerptconverter' ) CELERY_IMPORTS = [ - # TODO: Remove dummy converter or move out of common settings (e.g. to test-specific settings) - 'excerptconverter.dummyexcerptconverter.dummy_excerpt_converter', 'excerptconverter.gisexcerptconverter.gis_excerpt_converter' ] From 314158c5bbc2d386764f9a951227c9f483a56bb6 Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 14 Sep 2015 09:47:20 +0200 Subject: [PATCH 92/94] Update locales --- osmaxx-py/locale/de_CH/LC_MESSAGES/django.po | 81 +++++++++++++++----- osmaxx-py/locale/en_UK/LC_MESSAGES/django.po | 81 +++++++++++++++----- osmaxx-py/locale/en_US/LC_MESSAGES/django.po | 81 +++++++++++++++----- 3 files changed, 186 insertions(+), 57 deletions(-) diff --git a/osmaxx-py/locale/de_CH/LC_MESSAGES/django.po b/osmaxx-py/locale/de_CH/LC_MESSAGES/django.po index 4812f22c3..72e5a8b8f 100644 --- a/osmaxx-py/locale/de_CH/LC_MESSAGES/django.po +++ b/osmaxx-py/locale/de_CH/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2015-08-04 14:34+0200\n" +"POT-Creation-Date: 2015-09-14 09:46+0200\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -17,30 +17,75 @@ msgstr "" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" -#: excerptconverter/baseexcerptconverter/base_excerpt_converter.py:55 +#: excerptconverter/converter_helper.py:21 +#, python-format +msgid "The extraction of the order \"%(order_id)s\" has been finished." +msgstr "" + +#: excerptconverter/converter_helper.py:47 msgid "" "There is no email address assigned to your account. You won't be notified by " "email on process finish!" msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:75 +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:73 #, python-format msgid "\"%s\" created successful" msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:78 +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:76 #, python-format msgid "Creation of \"%s\" failed!" msgstr "" #: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:108 #, python-format -msgid "Your extraction order \"%s\" has been started" +msgid "The Dummy conversion of extraction order \"%s\" has been started." +msgstr "" + +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:124 +#, python-format +msgid "The Dummy conversion of extraction order \"%s\" failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:106 +#, python-format +msgid "" +"Extraction of \"%(file_type)s\" of extraction order \"%(order_id)s\" was " +"successful. (of %(number_of_files)s files of %(converter_name)s converter)" +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:119 +#, python-format +msgid "" +"The extraction of \"%(file)s\" of extraction order \"%(order_id)s\" failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:128 +#, python-format +msgid "" +"The extraction of \"%(file_type)s\" of extraction order \"%(order_id)s\" " +"failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:197 +#, python-format +msgid "The GIS extraction of the order \"%s\" is has been started." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:225 +msgid "GIS excerpt converter is not yet able to extract polygon excerpts." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:232 +#, python-format +msgid "GIS excerpt converter is not yet able to extract excerpts of type %s." msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:120 +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:247 #, python-format -msgid "Your extraction order %s has been processed" +msgid "" +"The extraction of order %(order_id)s failed. Please contact an administrator." msgstr "" #: osmaxx/excerptexport/forms/export_options_form.py:15 @@ -93,23 +138,23 @@ msgstr "" msgid "bounding geometry" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:22 +#: osmaxx/excerptexport/models/extraction_order.py:23 msgid "state" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:23 +#: osmaxx/excerptexport/models/extraction_order.py:24 msgid "process start date" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:25 +#: osmaxx/excerptexport/models/extraction_order.py:26 msgid "extraction options" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:28 +#: osmaxx/excerptexport/models/extraction_order.py:29 msgid "orderer" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:29 +#: osmaxx/excerptexport/models/extraction_order.py:30 msgid "excerpt" msgstr "" @@ -326,23 +371,21 @@ msgstr "" msgid "Export options" msgstr "" -#: osmaxx/excerptexport/views.py:93 +#: osmaxx/excerptexport/views.py:92 msgid "Invalid excerpt." msgstr "" -#: osmaxx/excerptexport/views.py:101 +#: osmaxx/excerptexport/views.py:100 #, python-format -msgid "" -"Successful creation of extraction order extraction order %(id)s. The " -"conversion process will start soon." +msgid "Queued extraction order %(id)s. The conversion process will start soon." msgstr "" -#: osmaxx/excerptexport/views.py:109 +#: osmaxx/excerptexport/views.py:108 #, python-format msgid "Creation of extraction order \"%s\" failed." msgstr "" -#: osmaxx/excerptexport/views.py:113 +#: osmaxx/excerptexport/views.py:112 msgid "Invalid export options." msgstr "" diff --git a/osmaxx-py/locale/en_UK/LC_MESSAGES/django.po b/osmaxx-py/locale/en_UK/LC_MESSAGES/django.po index 4812f22c3..72e5a8b8f 100644 --- a/osmaxx-py/locale/en_UK/LC_MESSAGES/django.po +++ b/osmaxx-py/locale/en_UK/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2015-08-04 14:34+0200\n" +"POT-Creation-Date: 2015-09-14 09:46+0200\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -17,30 +17,75 @@ msgstr "" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" -#: excerptconverter/baseexcerptconverter/base_excerpt_converter.py:55 +#: excerptconverter/converter_helper.py:21 +#, python-format +msgid "The extraction of the order \"%(order_id)s\" has been finished." +msgstr "" + +#: excerptconverter/converter_helper.py:47 msgid "" "There is no email address assigned to your account. You won't be notified by " "email on process finish!" msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:75 +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:73 #, python-format msgid "\"%s\" created successful" msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:78 +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:76 #, python-format msgid "Creation of \"%s\" failed!" msgstr "" #: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:108 #, python-format -msgid "Your extraction order \"%s\" has been started" +msgid "The Dummy conversion of extraction order \"%s\" has been started." +msgstr "" + +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:124 +#, python-format +msgid "The Dummy conversion of extraction order \"%s\" failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:106 +#, python-format +msgid "" +"Extraction of \"%(file_type)s\" of extraction order \"%(order_id)s\" was " +"successful. (of %(number_of_files)s files of %(converter_name)s converter)" +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:119 +#, python-format +msgid "" +"The extraction of \"%(file)s\" of extraction order \"%(order_id)s\" failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:128 +#, python-format +msgid "" +"The extraction of \"%(file_type)s\" of extraction order \"%(order_id)s\" " +"failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:197 +#, python-format +msgid "The GIS extraction of the order \"%s\" is has been started." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:225 +msgid "GIS excerpt converter is not yet able to extract polygon excerpts." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:232 +#, python-format +msgid "GIS excerpt converter is not yet able to extract excerpts of type %s." msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:120 +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:247 #, python-format -msgid "Your extraction order %s has been processed" +msgid "" +"The extraction of order %(order_id)s failed. Please contact an administrator." msgstr "" #: osmaxx/excerptexport/forms/export_options_form.py:15 @@ -93,23 +138,23 @@ msgstr "" msgid "bounding geometry" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:22 +#: osmaxx/excerptexport/models/extraction_order.py:23 msgid "state" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:23 +#: osmaxx/excerptexport/models/extraction_order.py:24 msgid "process start date" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:25 +#: osmaxx/excerptexport/models/extraction_order.py:26 msgid "extraction options" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:28 +#: osmaxx/excerptexport/models/extraction_order.py:29 msgid "orderer" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:29 +#: osmaxx/excerptexport/models/extraction_order.py:30 msgid "excerpt" msgstr "" @@ -326,23 +371,21 @@ msgstr "" msgid "Export options" msgstr "" -#: osmaxx/excerptexport/views.py:93 +#: osmaxx/excerptexport/views.py:92 msgid "Invalid excerpt." msgstr "" -#: osmaxx/excerptexport/views.py:101 +#: osmaxx/excerptexport/views.py:100 #, python-format -msgid "" -"Successful creation of extraction order extraction order %(id)s. The " -"conversion process will start soon." +msgid "Queued extraction order %(id)s. The conversion process will start soon." msgstr "" -#: osmaxx/excerptexport/views.py:109 +#: osmaxx/excerptexport/views.py:108 #, python-format msgid "Creation of extraction order \"%s\" failed." msgstr "" -#: osmaxx/excerptexport/views.py:113 +#: osmaxx/excerptexport/views.py:112 msgid "Invalid export options." msgstr "" diff --git a/osmaxx-py/locale/en_US/LC_MESSAGES/django.po b/osmaxx-py/locale/en_US/LC_MESSAGES/django.po index 4812f22c3..72e5a8b8f 100644 --- a/osmaxx-py/locale/en_US/LC_MESSAGES/django.po +++ b/osmaxx-py/locale/en_US/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2015-08-04 14:34+0200\n" +"POT-Creation-Date: 2015-09-14 09:46+0200\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -17,30 +17,75 @@ msgstr "" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" -#: excerptconverter/baseexcerptconverter/base_excerpt_converter.py:55 +#: excerptconverter/converter_helper.py:21 +#, python-format +msgid "The extraction of the order \"%(order_id)s\" has been finished." +msgstr "" + +#: excerptconverter/converter_helper.py:47 msgid "" "There is no email address assigned to your account. You won't be notified by " "email on process finish!" msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:75 +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:73 #, python-format msgid "\"%s\" created successful" msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:78 +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:76 #, python-format msgid "Creation of \"%s\" failed!" msgstr "" #: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:108 #, python-format -msgid "Your extraction order \"%s\" has been started" +msgid "The Dummy conversion of extraction order \"%s\" has been started." +msgstr "" + +#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:124 +#, python-format +msgid "The Dummy conversion of extraction order \"%s\" failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:106 +#, python-format +msgid "" +"Extraction of \"%(file_type)s\" of extraction order \"%(order_id)s\" was " +"successful. (of %(number_of_files)s files of %(converter_name)s converter)" +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:119 +#, python-format +msgid "" +"The extraction of \"%(file)s\" of extraction order \"%(order_id)s\" failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:128 +#, python-format +msgid "" +"The extraction of \"%(file_type)s\" of extraction order \"%(order_id)s\" " +"failed." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:197 +#, python-format +msgid "The GIS extraction of the order \"%s\" is has been started." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:225 +msgid "GIS excerpt converter is not yet able to extract polygon excerpts." +msgstr "" + +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:232 +#, python-format +msgid "GIS excerpt converter is not yet able to extract excerpts of type %s." msgstr "" -#: excerptconverter/dummyexcerptconverter/dummy_excerpt_converter.py:120 +#: excerptconverter/gisexcerptconverter/gis_excerpt_converter.py:247 #, python-format -msgid "Your extraction order %s has been processed" +msgid "" +"The extraction of order %(order_id)s failed. Please contact an administrator." msgstr "" #: osmaxx/excerptexport/forms/export_options_form.py:15 @@ -93,23 +138,23 @@ msgstr "" msgid "bounding geometry" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:22 +#: osmaxx/excerptexport/models/extraction_order.py:23 msgid "state" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:23 +#: osmaxx/excerptexport/models/extraction_order.py:24 msgid "process start date" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:25 +#: osmaxx/excerptexport/models/extraction_order.py:26 msgid "extraction options" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:28 +#: osmaxx/excerptexport/models/extraction_order.py:29 msgid "orderer" msgstr "" -#: osmaxx/excerptexport/models/extraction_order.py:29 +#: osmaxx/excerptexport/models/extraction_order.py:30 msgid "excerpt" msgstr "" @@ -326,23 +371,21 @@ msgstr "" msgid "Export options" msgstr "" -#: osmaxx/excerptexport/views.py:93 +#: osmaxx/excerptexport/views.py:92 msgid "Invalid excerpt." msgstr "" -#: osmaxx/excerptexport/views.py:101 +#: osmaxx/excerptexport/views.py:100 #, python-format -msgid "" -"Successful creation of extraction order extraction order %(id)s. The " -"conversion process will start soon." +msgid "Queued extraction order %(id)s. The conversion process will start soon." msgstr "" -#: osmaxx/excerptexport/views.py:109 +#: osmaxx/excerptexport/views.py:108 #, python-format msgid "Creation of extraction order \"%s\" failed." msgstr "" -#: osmaxx/excerptexport/views.py:113 +#: osmaxx/excerptexport/views.py:112 msgid "Invalid export options." msgstr "" From 1b3843e6731e9da6bdf70b400508123253d7489a Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 14 Sep 2015 09:52:18 +0200 Subject: [PATCH 93/94] Update documentation --- docs/useful-commands.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/useful-commands.md b/docs/useful-commands.md index d5ff57fa4..53716e365 100644 --- a/docs/useful-commands.md +++ b/docs/useful-commands.md @@ -5,18 +5,18 @@ ### Update migration information ```shell -docker-compose run osmaxxwebappdev /bin/bash -c 'python3 manage.py makemigrations' +docker-compose run webappdev /bin/bash -c 'python3 manage.py makemigrations' ``` ### Run migrations on database ```shell -docker-compose run osmaxxwebapp /bin/bash -c 'python3 manage.py migrate' +docker-compose run webapp /bin/bash -c 'python3 manage.py migrate' ``` ## Run tests ```shell -docker-compose run osmaxxwebappdev /bin/bash -c 'python3 manage.py test' +docker-compose run webappdev /bin/bash -c 'python3 manage.py test' ``` @@ -25,12 +25,12 @@ docker-compose run osmaxxwebappdev /bin/bash -c 'python3 manage.py test' ### Create superuser ```shell -docker-compose run webapp /bin/bash -c 'python3 manage.py createsuperuser' +docker-compose run webappdev /bin/bash -c 'python3 manage.py createsuperuser' ``` ### Update locales Please update locales only on release. Otherwise you will get huge diffs in feature pull requests. ```shell -docker-compose run webapp /bin/bash -c 'python3 manage.py makemessages -a' +docker-compose run webappdev /bin/bash -c 'python3 manage.py makemessages -a' ``` From 9bc114ca7470e7e588c6dcfd9ab4651061f49b7c Mon Sep 17 00:00:00 2001 From: Wasabideveloper Date: Mon, 14 Sep 2015 09:58:28 +0200 Subject: [PATCH 94/94] Update documentation --- docs/git-repository.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/git-repository.md b/docs/git-repository.md index 2d13d1fd6..2dfb2b656 100644 --- a/docs/git-repository.md +++ b/docs/git-repository.md @@ -49,9 +49,7 @@ For developers have write access to this repository: Implementation of feature #123 (Merge will close #123 ). * locales **NOT** compiled -> do this on release (prevent huge locale diffs in changes) - * ran flake8 - * ran check - * ran test + * ran tests * tested views by hand: * /orders/new [get/post] * /orders/{id}