From 0a5a0e109cf090fcd3a1cf9a63d614d7c81a0e5f Mon Sep 17 00:00:00 2001 From: Daniel Mizyrycki Date: Sat, 14 May 2016 22:44:36 -0700 Subject: [PATCH] Add python3 support. Improve webserver logic using bottle instead of flask and ajax long polling instead of websockets. Improve debugging. Improve tox build and tests. Reduce and update dependencies. --- .travis.yml | 6 +-- README.rst | 74 ++++++++++++++++--------- pex/setup.cfg | 3 +- requirements.txt | 6 +-- scripts/sphinxserve | 10 +++- setup.cfg | 2 +- sphinxserve/__init__.py | 22 ++++---- sphinxserve/lib.py | 104 ++++++++++++++++++++---------------- tests/test_requirements.txt | 6 +-- tox.ini | 51 +++++++++++------- 10 files changed, 169 insertions(+), 115 deletions(-) mode change 120000 => 100755 scripts/sphinxserve diff --git a/.travis.yml b/.travis.yml index 501a90d..1bd5608 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,11 +7,11 @@ matrix: osx_image: beta-xcode6.3 python: - - 2.7 + - 3.4 install: - - sudo bash -c 'python2.7 <(curl https://bootstrap.pypa.io/get-pip.py)' - - sudo pip install tox + - sudo bash -c 'python3.4 <(curl https://bootstrap.pypa.io/get-pip.py)' + - sudo pip3 install tox script: - tox diff --git a/README.rst b/README.rst index a2a28af..ba42b45 100644 --- a/README.rst +++ b/README.rst @@ -41,17 +41,29 @@ Design considerations sphinxserve was originally conceived as a Python and Linux project that can visualize sphinx document modifications in real time while working on them. At its core, sphinxserve uses the awesome projects `gevent`_ to provide -concurrency and event coordination, `flask`_ for web communication, Sphinx -for reStructucturedText rendering and of course `Python`_. sphinxserve used to -control browser reloading with xdotool, introducing a complex system tool -dependency. On release 0.7, sphinxserve decouples from this system dependency -using instead flask-sockets python package. The tradeoff here was to -temporarily drop python3 support until the gevent ecosystem officially -supports python3 which should be soon. Also, the filesystem notification tool -was upgraded to watchdog, removing another system dependency and making -the code more generic and cleaner. With these changes, as of release 0.7.4, -sphinxserve is able to run in other platforms as OSX and Windows for example. +concurrency and event coordination, `bottle`_ for web communication, +`watchdog`_ for filesystem events, `Sphinx`_ for reStructucturedText rendering +and of course `Python`_. +History +======= + +release 0.8: sphinxserve fully supports python3. bottle replaces flask and +ajax long polling replaces websockets to simplify even more the web server +logic. + +release 0.7.4: sphinxserve is able to run in other platforms as OSX and Windows +for example. + +release 0.7: sphinxserve decoupled from xdotool using flask-sockets python +package. The tradeoff was to temporarily drop python3 support until the gevent +ecosystem officially supported python3. Also, the filesystem notification tool +was upgraded to watchdog, removing another system dependency and making the +code more generic and cleaner. + +release <0.7: sphinxserve used to control browser reloading with xdotool, +a complex system tool dependency only available on Unix systems and tested +on Linux. Installation ============ @@ -77,10 +89,10 @@ Ubuntu>=14, Centos>=7 and Arch distros on Linux and in Yosemite on OSX. Linux ----- -System dependencies: glibc linux>=3, python>=2.7,<3 and a web browser +System dependencies: glibc linux>=3, python>=2.7 and a web browser supporting websockets (Firefox, Chrome, etc) on Linux:: - $ wget -O ~/bin/sphinxserve https://github.com/mzdaniel/sphinxserve/releases/download/0.7.4/sphinxserve-linux + $ wget -O ~/bin/sphinxserve https://github.com/mzdaniel/sphinxserve/releases/download/0.7.5/sphinxserve-linux $ chmod 755 ~/bin/sphinxserve OSX @@ -88,23 +100,24 @@ OSX Yosemite already has all needed dependencies:: - $ wget -O ~/bin/sphinxserve https://github.com/mzdaniel/sphinxserve/releases/download/0.7.4/sphinxserve-osx + $ wget -O ~/bin/sphinxserve https://github.com/mzdaniel/sphinxserve/releases/download/0.7.5/sphinxserve-osx $ chmod 755 ~/bin/sphinxserve Python package ~~~~~~~~~~~~~~ -Linux system dependencies: glibc linux>=3, python>=2.7,<3, the C toolchain +Linux system dependencies: glibc linux>=3, python>=2.7, the C toolchain (package names dependent on linux distro) to compile gevent and a web browser -supporting websockets. pip automatically downloads sphinxserve and its python +supporting javascript. pip automatically downloads sphinxserve and its python dependencies, compiles and builds wheel binary packages as needed and finally install sphinxserve. -OSX system dependencies: Xcode. Verified to work on Yosemite. +OSX system dependencies: Verified to work on Yosemite, python >=2.7 and +a web browser supporting javascript ajax with just pip installing. -Windows system dependencies: Verified to work on Windows 7, python >=2.7,<3 and -a web browser supporting websockets with just pip installing. +Windows system dependencies: Verified to work on Windows 7, python >=2.7 and +a web browser supporting javascript ajax with just pip installing. In all systems:: @@ -157,14 +170,29 @@ your browser to localhost:8888. Any saved changes on rst or txt files will trigger docs rebuild. +Local test/build +================ + +Assumptions for this section: A unix system, python2.7, 3.4 or 3.5, and +pip >= 8.1. Although git is recommended, it is not required. + +We use tox to test sphinxserve in virtualenvs for python2.7, 3.4 and 3.5 +Tox is a generic virtualenv manager and test command line tool. It handles the +creation of virtualenvs with proper python dependencies for testing, pep8 +checking and building: + + $ git clone https://github.com/mzdaniel/sphinxserve; cd sphinxserve + $ pip install tox + $ tox + + Thanks! ======= * `Guido van Rossum`_ and `Linus Torvalds`_ * Georg Brandl & David Goodger for `Sphinx`_ and `reStructuredText`_ * Denis Bilenko, Armin Rigo & Christian Tismer for `Gevent`_ and `Greenlet`_ -* Armin Ronacher for `Flask`_ -* Jeffrey Gelens & Kenneth Reitz for `gevent websocket`_ and `flask sockets`_ +* Marcel Hellkamp for `bottle`_ * Yesudeep Mangalapilly for `watchdog`_ * Holger Krekel for `pytest`_ and `tox`_ * Eric Holscher for `Read The Docs`_ @@ -177,13 +205,11 @@ Thanks! .. _Guido van Rossum: http://en.wikipedia.org/wiki/Guido_van_Rossum .. _Linus Torvalds: http://en.wikipedia.org/wiki/Linus_Torvalds .. _python: https://www.python.org -.. _sphinx: http://sphinx-doc.org/tutorial.html +.. _Sphinx: http://sphinx-doc.org/tutorial.html .. _restructuredtext: http://docutils.sourceforge.net/rst.html .. _gevent: http://gevent.org .. _greenlet: https://github.com/python-greenlet/greenlet -.. _flask: http://flask.pocoo.org -.. _gevent websocket: https://bitbucket.org/Jeffrey/gevent-websocket -.. _flask sockets: https://github.com/kennethreitz/flask-sockets +.. _bottle: http://bottlepy.org/docs/dev/index.html .. _watchdog: https://github.com/gorakhargosh/watchdog .. _pytest: http://pytest.org .. _pex: https://github.com/pantsbuild/pex diff --git a/pex/setup.cfg b/pex/setup.cfg index f1e17c1..45e26e4 100644 --- a/pex/setup.cfg +++ b/pex/setup.cfg @@ -32,4 +32,5 @@ classifier = [files] packages = sphinxserve_pex -scripts = scripts/sphinxserve +data_files = + bin = scripts/sphinxserve diff --git a/requirements.txt b/requirements.txt index e7cfa9f..7111b3e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ -flask-sockets>=0.1 -gevent>=1.1b2 -loadconfig>=0.1 +bottle==0.12.9 +gevent>=1.1.1 +loadconfig>=0.1.1 sphinx>=1.2.3 watchdog>=0.8.3 diff --git a/scripts/sphinxserve b/scripts/sphinxserve deleted file mode 120000 index a8ccf3f..0000000 --- a/scripts/sphinxserve +++ /dev/null @@ -1 +0,0 @@ -../sphinxserve/__main__.py \ No newline at end of file diff --git a/scripts/sphinxserve b/scripts/sphinxserve new file mode 100755 index 0000000..4366ccd --- /dev/null +++ b/scripts/sphinxserve @@ -0,0 +1,9 @@ +#!/usr/bin/env python + +import gevent.monkey +gevent.monkey.patch_all() + +from sphinxserve import main +import sys + +main(sys.argv) diff --git a/setup.cfg b/setup.cfg index 2847e44..9e866f8 100644 --- a/setup.cfg +++ b/setup.cfg @@ -20,8 +20,8 @@ classifier = Intended Audience :: System Administrators License :: OSI Approved :: MIT License Programming Language :: Python - Programming Language :: Python :: 2 Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3 Topic :: Documentation Topic :: Documentation :: Sphinx Topic :: Text Processing :: Markup diff --git a/sphinxserve/__init__.py b/sphinxserve/__init__.py index 02c8750..5493383 100755 --- a/sphinxserve/__init__.py +++ b/sphinxserve/__init__.py @@ -13,12 +13,12 @@ from gevent.event import Event from gevent import spawn, joinall from loadconfig import Config -from loadconfig.lib import capture_stream, write_file +from loadconfig.lib import write_file import logging as log import os from os.path import exists from sphinx import build_main -from sphinxserve.lib import fs_event_ctx, Webserver +from sphinxserve.lib import capture_streams, fs_event_ctx, Webserver import sys from textwrap import dedent @@ -128,26 +128,24 @@ def render(self): while True: self.watch_ev.wait() # Wait for docs changes self.watch_ev.clear() - with capture_stream() as stdout: + with capture_streams() as streams: self.build() - log.debug(stdout.getvalue()) + log.debug(streams.getvalue()) self.render_ev.set() def build(self): - return build_main([ - 'sphinx-build', - self.c.sphinx_path, - os.path.join(self.c.sphinx_path, self.c.output) - ]) + '''Render reStructuredText files with sphinx''' + return build_main(['sphinx-build', self.c.sphinx_path, + os.path.join(self.c.sphinx_path, self.c.output)]) def manage(self): '''Manage web server, watcher and sphinx docs renderer ''' - with capture_stream() as stdout, capture_stream('stderr') as stderr: + with capture_streams() as streams: ret = self.build() if ret != 0: - sys.exit(stderr.getvalue()) - log.debug(stdout.getvalue()) + sys.exit(streams.getvalue()) + log.debug(streams.getvalue()) workers = [spawn(self.serve), spawn(self.watch), spawn(self.render)] joinall(workers) diff --git a/sphinxserve/lib.py b/sphinxserve/lib.py index 730b05a..2056625 100644 --- a/sphinxserve/lib.py +++ b/sphinxserve/lib.py @@ -5,78 +5,69 @@ import os os.read = tp_read +from bottle import get, run, static_file from contextlib import contextmanager -from flask import Flask -from flask_sockets import Sockets import gevent -from gevent.pywsgi import WSGIServer from gevent.queue import Queue from gevent import sleep -from geventwebsocket.handler import WebSocketHandler from loadconfig.lib import Ret +from loadconfig.py6 import cStringIO import re import socket -from sys import platform +import sys from textwrap import dedent from watchdog.events import PatternMatchingEventHandler from watchdog.observers import Observer -if platform.startswith('win') or platform.startswith('darwin'): +if sys.platform.startswith('win') or sys.platform.startswith('darwin'): from watchdog.observers.polling import PollingObserver as Observer # noqa class Webserver(object): '''Serve static content from path, featuring asynchronous reload. - reload uses flask-sockets for async gevent communication on the server - and websockets on the browser. + Page reload is triggered by sphinx rst updates using gevent on the server + and executed after ajax long polling on the browser. ''' - def __init__(self, path, host, port, sig_reload): - self.path = path - self.host = host - self.port = port - self.signal = sig_reload + def __init__(self, path, host, port, reload_ev): + self.path, self.host, self.port = path, host, port + self.reload_ev = reload_ev def run(self): reload_js = dedent('''\ ''') - app = Flask( - __name__, - static_url_path='', - static_folder=self.path - ) - sockets = Sockets(app) - - @app.route('/') - def root(): - return app.send_static_file('index.html') - - @app.after_request + def after_request(response): '''Add reload javascript and remove googleapis fonts''' - response.direct_passthrough = False - if response.content_type.startswith('text/html'): - response.data = re.sub('()', r'{}\1'.format(reload_js), - response.data, flags=re.IGNORECASE) - if response.content_type.startswith('text/css'): - response.data = re.sub( - '@import url\(.+fonts.googleapis.com.+\);', '', - response.data, flags=re.IGNORECASE) - return response - - @sockets.route('/ws') - def ws_socket(ws): - '''Reload browser''' - self.signal.wait() - self.signal.clear() - ws.close() - - WSGIServer((self.host, int(self.port)), app, - handler_class=WebSocketHandler).serve_forever() + r = response + r.body = r.body.read().decode('utf-8') if getattr( + r.body, 'read', False) else r.body + if r.content_type.startswith('text/html'): + r.body = re.sub('()', r'{}\1'.format(reload_js), + r.body, flags=re.IGNORECASE) + if r.content_type.startswith('text/css'): + r.body = re.sub('@import url\(.+fonts.googleapis.com.+\);', '', + r.body, flags=re.IGNORECASE) + return r + + @get('') + def serve_static(path): + path = path + '/index.html' if path.endswith('/') else path + response = static_file(path, root=self.path) + return after_request(response) + + @get('/_svwait') + def wait_server_event(): + '''Block long polling javascript until reload event''' + self.reload_ev.wait() + self.reload_ev.clear() + + run(host=self.host, port=int(self.port), server='gevent') @contextmanager @@ -110,6 +101,25 @@ def fs_event(self): del evh +@contextmanager +def capture_streams(): + r'''Capture streams (stdout & stderr) in a string + >>> with capture_streams() as streams: + ... print('Hi there') + >>> streams.getvalue() + 'Hi there\n' + ''' + stdout = sys.stdout + stderr = sys.stderr + data = cStringIO() + sys.stdout = data + sys.stderr = data + yield data + sys.stdout = stdout + sys.stderr = stderr + data.flush() + + class Timeout(gevent.Timeout): '''Add expired attribute to Timeout context manager''' def __init__(self, *args, **kwargs): diff --git a/tests/test_requirements.txt b/tests/test_requirements.txt index ff7916b..70d3fd3 100644 --- a/tests/test_requirements.txt +++ b/tests/test_requirements.txt @@ -1,3 +1,3 @@ -hacking==0.10.1 -pytest==2.7.0 -requests==2.7.0 +flake8==2.5.4 +pytest==2.9.0 +requests==2.9.1 diff --git a/tox.ini b/tox.ini index e436ddc..957fb86 100644 --- a/tox.ini +++ b/tox.ini @@ -1,42 +1,58 @@ [tox] -envlist = clean, flake8, py27, build -minversion = 1.9 +envlist = clean, flake8, py27, py34, py35, build +minversion = 2.3.1 skipsdist = True skip_missing_interpreters = true toxworkdir = /tmp/tox/sphinxserve -[flake8] -exclude = .git -ignore = H102,E113,E121,E127,E128,H202,H301,H304,H405,H803 - [testenv] -basepython = python2.7 -deps = -rrequirements.txt - -rtests/test_requirements.txt whitelist_externals = /bin/sh /bin/rm -commands = - py27: py.test -c tests/pytest.ini {posargs} [testenv:clean] deps = commands = rm -rf dist sphinxserve.egg-info .eggs pex/scripts +[flake8] +exclude = .git +ignore = H102,E113,E121,E127,E128,E402,H202,H301,H304,H405,H803 + [testenv:flake8] deps = -rtests/test_requirements.txt commands = flake8 {toxinidir} +[testenv:py27] +deps = -rrequirements.txt + -rtests/test_requirements.txt +commands = py.test -c tests/pytest.ini {posargs} + +[testenv:py34] +deps = -rrequirements.txt + -rtests/test_requirements.txt +commands = py.test -c tests/pytest.ini {posargs} + +[testenv:py35] +deps = -rrequirements.txt + -rtests/test_requirements.txt +commands = py.test -c tests/pytest.ini {posargs} + [testenv:build-whl] recreate= True -deps = wheel==0.24.0 +deps = wheel==0.29.0 commands = {[testenv:clean]commands} - pip wheel --wheel-dir=dist sphinx<1.3 gevent>=1.1b1 {toxinidir} + pip wheel --wheel-dir=dist sphinx<1.3 {toxinidir} + +[testenv:dev] +envdir = /tmp/tox/dev +usedevelop = True +commands = [testenv:build] +basepython = python3.4 recreate= True -deps = wheel==0.24.0 - pex==1.0.1 +deps = wheel==0.29.0 + pex==1.1.6 commands = {[testenv:build-whl]commands} pip wheel --wheel-dir=dist -rpex/pex_requirements.txt sh -c '{envdir}/bin/pex -v --disable-cache --no-index -f dist \ @@ -44,8 +60,3 @@ commands = {[testenv:build-whl]commands} sh -x -c 'openssl sha1 pex/scripts/sphinxserve' pip wheel --wheel-dir=dist pex/ sh -x -c 'openssl sha1 dist/sphinxserve_pex*' - -[testenv:dev] -envdir = /tmp/tox/dev -usedevelop = True -commands =