Skip to content

Commit

Permalink
Streamline docker usage (pandas-dev#49981)
Browse files Browse the repository at this point in the history
* Streamline docker usage

* Updated CI

* newline

* update python.pythonpath in devcontainer.json

Co-authored-by: MarcoGorelli <>
  • Loading branch information
WillAyd authored Dec 2, 2022
1 parent 39e0964 commit 9e6bbde
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 63 deletions.
3 changes: 1 addition & 2 deletions .devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,7 @@
// You can edit these settings after create using File > Preferences > Settings > Remote.
"settings": {
"terminal.integrated.shell.linux": "/bin/bash",
"python.condaPath": "/opt/conda/bin/conda",
"python.pythonPath": "/opt/conda/bin/python",
"python.pythonPath": "/usr/local/bin/python",
"python.formatting.provider": "black",
"python.linting.enabled": true,
"python.linting.flake8Enabled": true,
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/code-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ jobs:
run: docker build --pull --no-cache --tag pandas-dev-env .

- name: Show environment
run: docker run -w /home/pandas pandas-dev-env mamba run -n pandas-dev python -c "import pandas as pd; print(pd.show_versions())"
run: docker run --rm pandas-dev-env python -c "import pandas as pd; print(pd.show_versions())"

requirements-dev-text-installable:
name: Test install requirements-dev.txt
Expand Down
49 changes: 10 additions & 39 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,42 +1,13 @@
FROM quay.io/condaforge/mambaforge
FROM python:3.10.8
WORKDIR /home/pandas

# if you forked pandas, you can pass in your own GitHub username to use your fork
# i.e. gh_username=myname
ARG gh_username=pandas-dev
ARG pandas_home="/home/pandas"
RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y build-essential

# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive
# hdf5 needed for pytables installation
RUN apt-get install -y libhdf5-dev

# Configure apt and install packages
RUN apt-get update \
&& apt-get -y install --no-install-recommends apt-utils git tzdata dialog 2>&1 \
#
# Configure timezone (fix for tests which try to read from "/etc/localtime")
&& ln -fs /usr/share/zoneinfo/Etc/UTC /etc/localtime \
&& dpkg-reconfigure -f noninteractive tzdata \
#
# cleanup
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=dialog

# Clone pandas repo
RUN mkdir "$pandas_home" \
&& git clone "https://github.com/$gh_username/pandas.git" "$pandas_home" \
&& cd "$pandas_home" \
&& git remote add upstream "https://github.com/pandas-dev/pandas.git" \
&& git pull upstream main

# Set up environment
RUN mamba env create -f "$pandas_home/environment.yml"

# Build C extensions and pandas
SHELL ["mamba", "run", "--no-capture-output", "-n", "pandas-dev", "/bin/bash", "-c"]
RUN cd "$pandas_home" \
&& export \
&& python setup.py build_ext -j 4 \
&& python -m pip install --no-build-isolation -e .
RUN python -m pip install --upgrade pip
RUN python -m pip install --use-deprecated=legacy-resolver \
-r https://raw.githubusercontent.com/pandas-dev/pandas/main/requirements-dev.txt
CMD ["/bin/bash"]
30 changes: 9 additions & 21 deletions doc/source/development/contributing_environment.rst
Original file line number Diff line number Diff line change
Expand Up @@ -228,34 +228,22 @@ with a full pandas development environment.

Build the Docker image::

# Build the image pandas-yourname-env
docker build --tag pandas-yourname-env .
# Or build the image by passing your GitHub username to use your own fork
docker build --build-arg gh_username=yourname --tag pandas-yourname-env .
# Build the image
docker build -t pandas-dev .

Run Container::

# Run a container and bind your local repo to the container
docker run -it -w /home/pandas --rm -v path-to-local-pandas-repo:/home/pandas pandas-yourname-env
# This command assumes you are running from your local repo
# but if not alter ${PWD} to match your local repo path
docker run -it --rm -v ${PWD}:/home/pandas pandas-dev

Then a ``pandas-dev`` virtual environment will be available with all the development dependencies.
When inside the running container you can build and install pandas the same way as the other methods

.. code-block:: shell
root@... :/home/pandas# conda env list
# conda environments:
#
base * /opt/conda
pandas-dev /opt/conda/envs/pandas-dev
.. note::
If you bind your local repo for the first time, you have to build the C extensions afterwards.
Run the following command inside the container::

python setup.py build_ext -j 4
.. code-block:: bash
You need to rebuild the C extensions anytime the Cython code in ``pandas/_libs`` changes.
This most frequently occurs when changing or merging branches.
python setup.py build_ext -j 4
python -m pip install -e . --no-build-isolation --no-use-pep517
*Even easier, you can integrate Docker with the following IDEs:*

Expand Down

0 comments on commit 9e6bbde

Please sign in to comment.