Add documentation content

README.md

@@ -4,18 +4,18 @@ Froide
 [![Build Status](https://travis-ci.org/stefanw/froide.png?branch=master)](https://travis-ci.org/stefanw/froide)
 
 
-Froide is a Freedom Of Information Portal written in Python using Django 1.4.3.
-The name comes from Freedom of Information (de). Also Froide
-sounds like Freude which is German for joy.
+Froide is a Freedom Of Information Portal written in Python using Django 1.5.
 
-It is used by the German FoI site fragdenstaat.de, but it is fully
+It is used by the German and the Austrian FOI site, but it is fully
 internationalized and written in English.
 
 Docs
 ----
 
 [Read the documentation](http://readthedocs.org/docs/froide/en/latest/) including a [Getting Started Guide](http://readthedocs.org/docs/froide/en/latest/gettingstarted/).
 
+Froide is supported by the [Open Knowledge Foundation](http://okfn.org/).
+
 
 License
 -------

docs/about.rst

@@ -2,9 +2,8 @@
 About
 =====
 
-Froide was designed to mimic the functionality of `What do they know <http://whatdotheyknow.com>`_ – a Freedom of Information portal in the UK written in Ruby on Rails 2.3. At the time when a German FoI portal was needed, the general FoI solution forked from WDTK called `Alaveteli <http://alaveteli.org>`_ was hard to install and not ready for reuse.
-That's why Froide was developed as a fresh start, fully
-internationalized and configurable written in Django 1.3 to power `Frag den Staat <https://fragdenstaat.de>`_.
+Froide is a Freedom of Information portal software written in Python with Django 1.5.
+
 
 Development Goals
 -----------------
@@ -36,20 +35,26 @@ Features
   request-unique email address (using SMTP) and the platform will receive answers on
   that mail address (by accessing an IMAP account).
 - Search functionality for Requests and Public Bodies.
-- Error Reporting interface via Sentry.
+- A read/write REST-API
+- Redaction of PDFs
 
 Dependencies
 ------------
 
 A detailed list of Python package dependencies can be found in `requirements.txt`, but here is a general overview:
 
-- Django 1.3 - the Web framework
-- South 0.7.3 - the database migration framework (development dependency)
-- Sphinx 1.0.7 - the documentation tool (development dependency)
-- Celery 2.2.5 - task queue for background processing
-- Haystack 1.2.0 - abstraction layer for search
+- Django 1.5 - the Web framework
+- Celery 3.X - task queue for background processing
+- Haystack 2.X-beta - abstraction layer for search
+
+A development goal is that, even though a task queue (like Celery) and a search server (like Solr) are highly recommended, they are not necessary for either development or production setup and can be replaced with Cronjobs and database queries respectively (results/performance will probably degrade, but it will work nonetheless).
 
-A development goal is that, even though a task queue (like Celery) and a search server (like Solr) are highly recommended, they are not necessary for either development or production setup and can be replaced with Cronjobs and database queries respectively (results/performance will probably degrade, but it should work).
+History
+-------
+
+Froide was designed to mimic the functionality of `What do they know <http://whatdotheyknow.com>`_ – a Freedom of Information portal in the UK written in Ruby on Rails 2.3. At the time when a German FoI portal was needed, the general FoI solution forked from WDTK called `Alaveteli <http://alaveteli.org>`_ was hard to install and not ready for reuse.
+That's why Froide was developed in spring 2011 as a fresh start, fully
+internationalized and configurable written in Django 1.3 to power `Frag den Staat <https://fragdenstaat.de>`_ which launched in August 2011.
 
 Name
 ----

docs/admin.rst

@@ -0,0 +1,6 @@
+============
+Froide Admin
+============
+
+The Django admin exposes database objects to staff users and superusers.
+Superusers have access to all objects while staff users can be given specific read/write/delete access to kinds of objects (e.g. only public bodies). Users can also given a group that has certain rights associated with it.

docs/conf.py

@@ -41,7 +41,7 @@
 
 # General information about the project.
 project = u'froide'
-copyright = u'2011, Stefan Wehrmeyer'
+copyright = u'2013, Stefan Wehrmeyer, Open Knowledge Foundation'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/configuration.rst

@@ -4,7 +4,7 @@ Configuration
 
 Froide can be configured in many ways to reflect the needs of your local FoI portal.
 
-All configuration is kept in the Django `settings.py` file. Individual settings can be overwritten by placing a `local_settings.py` file on the Python path (e.g. in the same directory) and redefining the configuration key in there.
+The `custom_settings.py.example` file that comes with froide has all the settings from the `settings.py` file but they are commented out. You can copy this file to `custom_settings.py`
 
 Froide Configuration
 --------------------
@@ -37,6 +37,9 @@ The following keys in that dictionary must be present:
   *integer* The id of the Freedom of Information law in the database
   that is used by default (e.g. 1)
 
+**search_engine_query**
+  *string* You can give a URL with string formatting placeholders `query` and `domain` in them that will be presented to the user as the URL for web searches. The default is a Google search.
+
 
 Greeting Regexes
 ----------------
@@ -77,7 +80,7 @@ You must adapt the standard Django parameters for sending email.
 Configure the backend depending on your environment (development vs.
 production)::
 
-    # development environment:
+    # development/testing environment:
     EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
     # production environment:
     EMAIL_BACKEND = 'djcelery_email.backends.CeleryEmailBackend'
@@ -136,52 +139,17 @@ right before the mail is
 sent out (the changed address is not stored). This allows for some
 testing of sending and receiving mails to and from public bodies wihtout spamming them.
 
-Setting Up Search with Solr
----------------------------
-
-Froide uses `django-haystack` to interface with a search. Solr is
-recommended, but thanks to `django-haystack` you can use something
-else as well.
-
-Haystack configuration for solr works like so::
-
-    HAYSTACK_SITECONF = 'froide.search_sites'
-    HAYSTACK_SEARCH_ENGINE = 'solr'
-    HAYSTACK_SOLR_URL = 'http://127.0.0.1:8983/solr'
-
-If you have a solr multicore setup, your solr URL would probably look more like this::
-
-    HAYSTACK_SOLR_URL = 'http://127.0.0.1:8983/solr/froide'
-
-For details, please refer to the `Haystack Documentation <http://haystacksearch.org>`_.
-
-Setting Up Background Processing with Celery
---------------------------------------------
-
-The following part in `settings.py` does the configuration of Celery.
-Overwrite the `CELERY*` values with your own in `local_settings.py`::
-
-    import djcelery
-    djcelery.setup_loader()
-
-    CELERY_IMPORTS = ("foirequest.tasks", )
-
-    CELERY_RESULT_BACKEND = "database"
-    CELERY_RESULT_DBURI = "sqlite:///dev.db"
-
-    CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
-
-For details please refer to the `django-celery documentation <http://django-celery.readthedocs.org/en/latest/index.html>`_.
 
 Some more settings
 ------------------
 
-Configure the name and default domain URL (without trailing slash) of your site with the following settings::
+Configure the name, default domain URL and default email (without trailing slash) of your site with the following settings::
 
     SITE_NAME = 'FroIde'
     SITE_URL = 'http://localhost:8000'
+    SITE_EMAIL = '[email protected]'
 
-You can give a URL with string formatting placeholders `query` and `domain` in them that will be presented to the user as the URL for web searches via the setting `SEARCH_ENGINE_QUERY`. The default is a Google search.
+More suggestions of settings you can change can be found in the `custom_settings.py.example` file that comes with froide.
 
 
 Securing your site
@@ -193,11 +161,10 @@ parts configurable by `local_settings` you can use the following
 setting::
 
     SECRET_URLS = {
-        "admin": "my-secret-admin",
-        "sentry": "my-secret-sentry"
+        "admin": "my-secret-admin"
     }
 
-It's also recommended to protect the admin and sentry further via HTTP
+It's also recommended to protect the admin further via HTTP
 auth in your production reverse proxy (e.g. nginx).
 
 The app `djangosecure <https://github.com/carljm/django-secure/>`_ is part of Froide
@@ -212,3 +179,5 @@ Some Django settings related to security and SSL::
     SESSION_COOKIE_AGE = 3628800 # six weeks for usability
     SESSION_COOKIE_HTTPONLY = True
     SESSION_COOKIE_SECURE = True
+
+Make sure that your frontend server transports the information that HTTPS is used to the web server.

docs/development.rst

@@ -0,0 +1,36 @@
+Development
+===========
+
+Run tests
+---------
+
+Froide has a test suite. You have to install the test dependencies first::
+
+    pip install -r requirements-test.txt
+
+The default test configuration requires a running elasticsearch instance.
+You either have to install elasticsearch or change the configuration to
+another search engine.
+
+Then you can run the tests::
+
+    make test
+
+This also does test coverage analysis that you can then
+find at `htmlcov/index.html`.
+
+
+Build docs
+----------
+
+The docs can be build with Sphinx but first you must install the theme.
+Excecute these commands from the top level of the froide directory::
+
+  git submodule init
+  git submodule update
+
+Then change into the `docs` directory and type::
+
+  make html
+
+The documentation will be build and you can find the resulting html in `docs/_build/html`.

docs/gettingstarted.rst

@@ -9,71 +9,129 @@ more advanced features are discussed at the end.
 Set up the development environment
 ----------------------------------
 
-You should be using `virtualenv` and it is suggested you
-also use `virtualenvwrapper`. Setup a virtual environment for development like so::
+You should be using a Python virtual environment.
+Setup a virtual environment for development with `virtualenv`like so::
 
-    mkvirtualenv --no-site-packages froide
+    virtualenv --no-site-packages ~/pyenv
+    . ~/pyenv/bin/activate
 
-Get the source code with Git from the official GitHub repository or from
-your fork::
+Get the source code with Git from the GitHub repository::
 
     git clone git://github.com/stefanw/froide.git
     cd froide
 
 Install the requirements inside the virtual env with `pip`::
 
-    which pip
-    <should display your virtual env pip>
     pip install -r requirements.txt
 
-If only your global `pip` is available, run `easy_install pip`. The dependency installation takes a couple of seconds, but after that everything is in place.
+The dependency installation may take a couple of minutes, but after that everything is in place.
 
-Copy `custom_settings.py.example` to `custom_settings.py`::
+Sync and migrate and *do NOT* create a superuser just yet::
+
+    python manage.py syncdb --noinput --migrate
+
+Now you can already start the web server::
+
+    python manage.py runserver
+
+Visit `http://localhost:8000 <http://localhost:8000>`_ and there is your running Froide instance!
+
+You can quit the server (Ctrl+C) and create a superuser account::
+
+    python manage.py createsuperuser
+
+
+Add basic database objects
+--------------------------
+
+Run the web server again and login to the admin interface at `http://localhost:8000/admin/ <http://localhost:8000/admin/>`_ with the credentials of your superuser.
+
+The first thing you should do is create a jurisdiction. Click on "Jurisdiction" in the "Publicbody" section and then on "Add Jurisdiction".
+Name your jurisdiction however you want (e.g. Federal). If you only ever intend to have one, the name will not even show up. Click "Save" to continue.
+
+Go back into the public body section and add an FOI law. Give it a name (e.g. Freedom of Information Act) and choose a jurisdiction. There are many more things that can be configured, but you can leave them for now.
+
+Now you can add your first public body by going back to the public body section and clicking on "Add" next to "Public Bodies". Give it a name (e.g. Ministry of the Interior).
+Click on the little plus-sign next to topic to add a topic for this public body. The classification is to distinguish different areas of government (e.g. "Ministry", "Council").
+If you want to make a request to this public body, it needs to have an email address.
+Select your previously created jurisdiction and FOI law and save.
+
+You should also fill out your user details like your name in the user section of the admin.
+
+Now you are ready to make your first request. Go back to the front page and give it a go. You can also find out more about the :doc:`admin`.
+
+
+Custom configuration
+--------------------
+
+By default the Django Web server uses the `settings.py` file in the froide directory (the `froide.settings` Python module). This will be fine for your first experiments but if you want to customize your froide instance you will want your own settings file.
+
+Go into the froide directory and copy the `custom_settings.py.example` to `custom_settings.py`::
 
     cd froide
     cp custom_settings.py.example custom_settings.py
 
-The development environment uses SQLite. You can override the default Django settings values for froide in your `custom_settings.py`, if you want, but you don't have to.
+Now you can customize that settings file to your liking. Note that it imports the original `settings.py` at the top, so you only have to change the values you need to change.
 
-Your `custom_settings.py` file (you can name it differently) is your Django settings file that you must give on each Django manage.py command. You can also set the `DJANGO_SETTINGS_MODULE` environment variable.
+In order for Django to start your web server with these new settings you either have to set the environment variable `DJANGO_SETTINGS_MODULE` or give the settings module as a command line argument.
 
-Sync and migrate and *do NOT* create a superuser just yet::
+Either type::
+
+    export DJANGO_SETTINGS_MODULE=froide.custom_settings
+    python manage.py runserver
 
-    python manage.py syncdb --noinput --migrate --settings=froide.custom_settings
+The environment variable will be set for this shell session. If you open a new shell, you have to export the settings again.
+Or::
 
-Now you can create a superuser account::
+    python manage.py runserver --settings=froide.custom_settings
 
-    python manage.py createsuperuser --settings=froide.custom_settings
+Since it can be annoying to type this (or to remember to export the environment variable), you can create a shortcut shell script, create an alias, put an export in your virtualenv's `activate` file or use `virtualenvwrapper <http://virtualenvwrapper.readthedocs.org/en/latest/>`_ and put the export in your `postactivate` hook.
 
-That's it for a setup that basically works. Run this::
 
-    python manage.py runserver --settings=froide.custom_settings
 
-and go to `http://localhost:8000 <http://localhost:8000>`_. You should
-be greeted by a working Froide installation. It doesn't have any data
-inside, but you can change that by going to `http://localhost:8000/admin/ <http://localhost:8000/admin/>`_ and logging in with your superuser account.
+Search with Haystack
+--------------------
+
+In order to get a real search engine running you need to override the `HAYSTACK_CONNECTIONS` setting with the details of your search engine. Find out `how to configure your search engine at the Haystack Docs <http://django-haystack.readthedocs.org/en/latest/tutorial.html#modify-your-settings-py>`_.
+
+An example configuration for solr would look like this::
+
+    HAYSTACK_CONNECTIONS = {
+        'default': {
+            'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
+            'URL': 'http://127.0.0.1:8983/solr/froide'
+        }
+    }
+
+
+Background Tasks with Celery
+----------------------------
 
-For more information on the different models you find in the admin visit :doc:`models`.
+From the standard settings file everything is already setup for background tasks except that they are not running in the background.
 
-Run tests
----------
+You need to change the `CELERY_ALWAYS_EAGER` setting to `False` in your custom settings::
 
-Froide has a test suite. Copy `test_settings.py.example` to `test_settings.py`. `test_settings.py` does not use your `custom_settings.py` changes.
+    CELERY_ALWAYS_EAGER = False
 
-You can then run for tests::
+You need a broker for Celery. Find out more at the `Celery Docs <http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#choosing-a-broker>`_.
 
-    make test
+We recommend `RabbitMQ <http://www.rabbitmq.com/>`_ as broker. Install it and then start it in a different terminal like this::
 
-This also does test coverage analysis that you can then
-find at `htmlcov/index.html`.
-Note some tests will not work without a search engine like solr running.
+    rabbitmq-server
 
+After you started the broker open yet another terminal, activate your virtual environment and run the celery worker like this::
 
-Search with Haystack and Solr
------------------------------
+    python manage.py celeryd -l INFO -B
 
+Now your server will send background tasks to Celery. Lots of common tasks are designed as background tasks so that an ongoing HTTP request can send a response more quickly. The following things are designed as background tasks:
 
-Background Processing with Celery
----------------------------------
+- Search Indexing: Updates to database objects are indexed in the background
+- Email Sending: When an action triggers an email, it's sent in the background
+- Denormalized Counts on database objects
 
+Celery also takes the role of `cron` and handles periodic tasks. You should set up periodic tasks in the admin under "Djcelery - Periodic tasks". Here is a recommended configuration:
 
+- Fetch Mail: every minute
+- Detect Overdue at Midnight: 0 0 * * * (m/h/d/dM/MY)
+- Batch Update Followers every 24 hours: 0 0 * * * (m/h/d/dM/MY)
+- Remind users to classify there requests: 0 7 6 * * (m/h/d/dM/MY)