diff --git a/ci-scripts/docker/README.md b/ci-scripts/docker/README.md index 9cf7370a2..753d2fee3 100644 --- a/ci-scripts/docker/README.md +++ b/ci-scripts/docker/README.md @@ -1,216 +1,22 @@ -# Chapter 1. Testing on your own computer -## Step 1. +Author: Thomas Ferreira de Lima +email: thomas@tlima.me -Make sure you have the quay.io/pypa/manylinux1_x86_64 image. +This folder contains scripts to be run inside docker images. See instructions on how to test this yourself in ci-scripts/docker/development_notes. + +## docker_build.sh + +We need two environment variables to get going: ```bash -$ docker images -REPOSITORY TAG IMAGE ID CREATED SIZE -quay.io/pypa/manylinux1_x86_64 latest 1c8429c548f2 2 months ago 879MB -hello-world latest 4ab4c602aa5e 3 months ago 1.84kB -# My image was old: -$ docker pull quay.io/pypa/manylinux1_x86_64 -Using default tag: latest -latest: Pulling from pypa/manylinux1_x86_64 -7d0d9526f38a: Already exists -3324bfadf9cb: Pull complete -20f27c7e3062: Pull complete -5bc21fc5fe97: Pull complete -Digest: sha256:a13b2719fb21daebfe25c0173d80f8a85a2326dd994510d7879676e7a2193500 -Status: Downloaded newer image for quay.io/pypa/manylinux1_x86_64:latest +DOCKER_IMAGE="quay.io/pypa/manylinux1_x86_64" +PY_VERSION="cp37-cp37m" ``` -## Step 2. - -This step was inspired by https://dev.to/jibinliu/how-to-persist-data-in-docker-container-2m72 -Create a volume for klayout. This is necessary because docker containers don't persist data. +The script must be run inside an image pulled from $DOCKER_IMAGE and with klayout's git repo cloned in /io. Inside the git clone folder, run: ```bash -$ docker volume create klayout-persist -$ docker volume inspect klayout-persist -[ - { - "CreatedAt": "2018-12-18T15:01:48Z", - "Driver": "local", - "Labels": {}, - "Mountpoint": "/var/lib/docker/volumes/klayout-persist/_data", - "Name": "klayout-persist", - "Options": {}, - "Scope": "local" - } -] +docker run --rm -e DOCKER_IMAGE -e PY_VERSION -v `pwd`:/io $DOCKER_IMAGE $PRE_CMD "/io/ci-scripts/docker/docker_build.sh"; +# $PRE_CMD is empty for now (useless currently). ``` -## Step 3. - -Build image `myimage` with: - -```bash -$ docker build -t myimage:latest -f Dockerfile.x86_64 . -``` - -This creates an image called `myimage` (temporary). This image will not overwrite old ones. Tip: prune old, unused images with `docker image prune`. - -Then I run the docker with a terminal shell and load the volume klayout-persist in /persist: - -```bash -$ docker run --name klayout --mount source=klayout-persist,target=/persist -it myimage -``` - -## Step 4. - -In the shell, pull master from klayout. - -```bash -cd /persist -git clone https://github.com/lightwave-lab/klayout.git -mkdir -p wheelhouse -cd klayout -# make wheel with python 3.6 (for example) -/opt/python/cp36-cp36m/bin/python setup.py bdist_wheel -d /persist/wheelhouse/ -cd /persist -auditwheel repair "wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-linux_x86_64.whl" -w wheelhouse/ -# Need to manually fix the wheel -#/opt/python/cp36-cp36m/bin/pip install klayout --no-index -f /wheelhouse -``` - -The produced wheel from auditwheel, klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl, is defective in the following way: dbcore.so etc. have RPATHs reset to `$ORIGIN/.libs`, so we need to move all .so's `lib_*` into `.libs`, as well as `db_plugins`. We also need to change the dist-info/RECORD file paths. This is a bug from auditwheel, it should either have added a new RPATH, $ORIGIN/.libs, where it places libz, libcurl, libexpat, instead of renaming the existing ones, or moved the files to the right place. - - -Procedure to fix the wheel: - -```bash -unzip wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl -d tempwheel -cd tempwheel/klayout -mv lib_* db_plugins .libs/ -cd ../klayout-0.26.0.dev8.dist-info/ -sed -i 's/^klayout\/lib_/klayout\/.libs\/lib_/g' RECORD -sed -i 's/^klayout\/db_plugins/klayout\/.libs\/db_plugins/g' RECORD -cd ../ -rm -f ../wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl -zip -r ../wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl ./* -cd .. -rm -rf tempwheel -``` - -Now we can install and test: -```bash -/opt/python/cp36-cp36m/bin/pip install klayout --no-index -f /persist/wheelhouse -cd /persist/klayout -/opt/python/cp36-cp36m/bin/python -m unittest testdata/pymod/import_db.py testdata/pymod/import_rdb.py testdata/pymod/import_tl.py -# Tests passed! -``` - -Encoded this behavior in a script called fix_wheel.sh. now you only need to run `./fix_wheel.sh wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl`, and it will overwrite the wheel. - -## Step 5. Iterate over all python versions. - -For that, we need something like: - -```bash -# Compile wheels -for PYBIN in /opt/python/*/bin; do - "${PYBIN}/python" setup.py bdist_wheel -d /persist/wheelhouse/ -done - -# Bundle external shared libraries into the wheels via auditwheel -for whl in /persist/wheelhouse/*linux_*.whl; do - auditwheel repair "$whl" -w /persist/wheelhouse/ -done - -# Fix each wheel generated by auditwheel -for whl in /persist/wheelhouse/*manylinux1_*.whl; do - ./ci-scripts/docker/fix_wheel.sh "$whl" -done - -# Install packages and test -TEST_HOME=/persist/klayout/testdata -for PYBIN in /opt/python/*/bin/; do - "${PYBIN}/pip" install klayout --no-index -f /persist/wheelhouse - "${PYBIN}/python" $TEST_HOME/pymod/import_db.py - "${PYBIN}/python" $TEST_HOME/pymod/import_rdb.py - "${PYBIN}/python" $TEST_HOME/pymod/import_tl.py - -``` - -I tested step 1-5 with both quay.io/pypa/manylinux1_x86_64 and quay.io/pypa/manylinux1_i686. So far the only failure was with `cp27-cp27mu` which gave this import error: - -`ImportError: /opt/python/cp27-cp27mu/lib/python2.7/site-packages/klayout/.libs/lib_pya.so: undefined symbol: PyUnicodeUCS2_AsUTF8String` - -I noticed that the ccache folder ended up with 800MB. I was hoping that the gcc compilation could reuse a lot of previously built objects but that didn't happen. I think that's because each python comes with its own header. So going forward it doesn't make sense to create a docker image for every python version. I will just cache a ccache folder via travis. -The ccache folder after a single build has 657MB. Go figure. - -I discovered that fix_wheel script was actually not properly working. So instead I looked into fixing `auditwheel` directly. Here's the commit that fixes it: https://github.com/thomaslima/auditwheel/tree/87f5306ec02cc68020afaa9933543c898b1d47c1 - -So now the plan is to change the `docker_build.sh` script so it uses the proper auditwheel, instead of their default. - -# Chapter 2. Testing CI flow with docker - -# Step 1. Testing commands in own computer - -First cloned with: -```bash -git clone git@github.com:lightwave-lab/klayout.git -b tmp/manylinux -cd klayout -``` - -Let's work with a few environment variables, like in https://github.com/pypa/python-manylinux-demo/blob/master/.travis.yml - -DOCKER_IMAGE options: quay.io/pypa/manylinux1_x86_64 quay.io/pypa/manylinux1_i686 - -PY_VERSION options: cp27-cp27m cp27-cp27mu cp34-cp34m cp35-cp35m cp36-cp36m cp37-cp37m - -Total of 2x6 = 12 possibilities - -```bash -export DOCKER_IMAGE=quay.io/pypa/manylinux1_x86_64 -export PY_VERSION="cp36-cp36m" -docker pull $DOCKER_IMAGE -mkdir -p ccache -mkdir -p wheelhouse -docker run --name klayout -v `pwd` -it $DOCKER_IMAGE -``` - -Inside docker shell: -```bash -yum install -y zlib-devel -yum install -y zip - -yum install -y ccache -ln -s /usr/bin/ccache /usr/lib64/ccache/c++ -ln -s /usr/bin/ccache /usr/lib64/ccache/cc -ln -s /usr/bin/ccache /usr/lib64/ccache/gcc -ln -s /usr/bin/ccache /usr/lib64/ccache/g++ -echo $PATH -# /usr/lib64/ccache:/opt/rh/devtoolset-2/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -export CCACHE_DIR="/io/ccache" - -# Compile wheel -/opt/python/$PY_VERSION/bin/python setup.py bdist_wheel -d /io/wheelhouse/ - -# Bundle external shared libraries into the wheels via auditwheel -for whl in /io/wheelhouse/*linux_*.whl; do - auditwheel repair "$whl" -w /io/wheelhouse/ -done - -# Fix each wheel generated by auditwheel -for whl in /io/wheelhouse/*manylinux1_*.whl; do - ./ci-scripts/docker/fix_wheel.sh "$whl" -done - -``` - -# Step 2. Automating step 1 in travis (CI). -DOCKER_IMAGE options: quay.io/pypa/manylinux1_x86_64 quay.io/pypa/manylinux1_i686 - -PY_VERSION options: cp27-cp27m cp27-cp27mu cp34-cp34m cp35-cp35m cp36-cp36m cp37-cp37m - -Build: spawn 12 travis jobs, one for each combination of word-size and python version. -Output: populated ./ccache with compiled objects and wheels inside ./wheelhouse/, one useless, `*linux_*.whl` and one useful `*manylinux1_*.whl`. -Post-build: -- cache `./ccache` -- deploy `./wheelhouse/*manylinux1_*.whl` to dropbox (./deploy folder) - -# Step 3. Automating deployment to PyPI: - -TBD +This command will generate a wheel and place it in `wheelhouse/klayout-*manylinux1*.whl`. This is the wheel that needs to be uploaded to PyPI via twine. See ci-scripts/twine/README.md. diff --git a/ci-scripts/docker/Dockerfile.i686 b/ci-scripts/docker/development_notes/Dockerfile.i686 similarity index 100% rename from ci-scripts/docker/Dockerfile.i686 rename to ci-scripts/docker/development_notes/Dockerfile.i686 diff --git a/ci-scripts/docker/Dockerfile.x86_64 b/ci-scripts/docker/development_notes/Dockerfile.x86_64 similarity index 100% rename from ci-scripts/docker/Dockerfile.x86_64 rename to ci-scripts/docker/development_notes/Dockerfile.x86_64 diff --git a/ci-scripts/docker/development_notes/README.md b/ci-scripts/docker/development_notes/README.md new file mode 100644 index 000000000..5a05684a5 --- /dev/null +++ b/ci-scripts/docker/development_notes/README.md @@ -0,0 +1,221 @@ +Author: Thomas Ferreira de Lima +email: thomas@tlima.me + +I wrote these notes as I was learning how to use docker and how to build python packages inside a docker image prepared by the pypa team. They require us to build there to allow wheels to have the `manylinux1` tag, meaning that these wheels would be compatible with most linux distributions around. Chapter 1 is about testing in my own computer (MacOS Mojave) and Chapter 2 is about how to automate this build using travis-ci.org. + +# Chapter 1. Testing on your own computer +## Step 1. + +Make sure you have the quay.io/pypa/manylinux1_x86_64 image. + +```bash +$ docker images +REPOSITORY TAG IMAGE ID CREATED SIZE +quay.io/pypa/manylinux1_x86_64 latest 1c8429c548f2 2 months ago 879MB +hello-world latest 4ab4c602aa5e 3 months ago 1.84kB +# My image was old: +$ docker pull quay.io/pypa/manylinux1_x86_64 +Using default tag: latest +latest: Pulling from pypa/manylinux1_x86_64 +7d0d9526f38a: Already exists +3324bfadf9cb: Pull complete +20f27c7e3062: Pull complete +5bc21fc5fe97: Pull complete +Digest: sha256:a13b2719fb21daebfe25c0173d80f8a85a2326dd994510d7879676e7a2193500 +Status: Downloaded newer image for quay.io/pypa/manylinux1_x86_64:latest +``` + +## Step 2. + +This step was inspired by https://dev.to/jibinliu/how-to-persist-data-in-docker-container-2m72 +Create a volume for klayout. This is necessary because docker containers don't persist data. + +```bash +$ docker volume create klayout-persist +$ docker volume inspect klayout-persist +[ + { + "CreatedAt": "2018-12-18T15:01:48Z", + "Driver": "local", + "Labels": {}, + "Mountpoint": "/var/lib/docker/volumes/klayout-persist/_data", + "Name": "klayout-persist", + "Options": {}, + "Scope": "local" + } +] +``` + +## Step 3. + +Build image `myimage` with: + +```bash +$ docker build -t myimage:latest -f Dockerfile.x86_64 . +``` + +This creates an image called `myimage` (temporary). This image will not overwrite old ones. Tip: prune old, unused images with `docker image prune`. + +Then I run the docker with a terminal shell and load the volume klayout-persist in /persist: + +```bash +$ docker run --name klayout --mount source=klayout-persist,target=/persist -it myimage +``` + +## Step 4. + +In the shell, pull master from klayout. + +```bash +cd /persist +git clone https://github.com/lightwave-lab/klayout.git +mkdir -p wheelhouse +cd klayout +# make wheel with python 3.6 (for example) +/opt/python/cp36-cp36m/bin/python setup.py bdist_wheel -d /persist/wheelhouse/ +cd /persist +auditwheel repair "wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-linux_x86_64.whl" -w wheelhouse/ +# Need to manually fix the wheel +#/opt/python/cp36-cp36m/bin/pip install klayout --no-index -f /wheelhouse +``` + +The produced wheel from auditwheel, klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl, is defective in the following way: dbcore.so etc. have RPATHs reset to `$ORIGIN/.libs`, so we need to move all .so's `lib_*` into `.libs`, as well as `db_plugins`. We also need to change the dist-info/RECORD file paths. This is a bug from auditwheel, it should either have added a new RPATH, $ORIGIN/.libs, where it places libz, libcurl, libexpat, instead of renaming the existing ones, or moved the files to the right place. + + +Procedure to fix the wheel: + +```bash +unzip wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl -d tempwheel +cd tempwheel/klayout +mv lib_* db_plugins .libs/ +cd ../klayout-0.26.0.dev8.dist-info/ +sed -i 's/^klayout\/lib_/klayout\/.libs\/lib_/g' RECORD +sed -i 's/^klayout\/db_plugins/klayout\/.libs\/db_plugins/g' RECORD +cd ../ +rm -f ../wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl +zip -r ../wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl ./* +cd .. +rm -rf tempwheel +``` + +Now we can install and test: +```bash +/opt/python/cp36-cp36m/bin/pip install klayout --no-index -f /persist/wheelhouse +cd /persist/klayout +/opt/python/cp36-cp36m/bin/python -m unittest testdata/pymod/import_db.py testdata/pymod/import_rdb.py testdata/pymod/import_tl.py +# Tests passed! +``` + +Encoded this behavior in a script called fix_wheel.sh. now you only need to run `./fix_wheel.sh wheelhouse/klayout-0.26.0.dev8-cp36-cp36m-manylinux1_x86_64.whl`, and it will overwrite the wheel. + +## Step 5. Iterate over all python versions. + +For that, we need something like: + +```bash +# Compile wheels +for PYBIN in /opt/python/*/bin; do + "${PYBIN}/python" setup.py bdist_wheel -d /persist/wheelhouse/ +done + +# Bundle external shared libraries into the wheels via auditwheel +for whl in /persist/wheelhouse/*linux_*.whl; do + auditwheel repair "$whl" -w /persist/wheelhouse/ +done + +# Fix each wheel generated by auditwheel +for whl in /persist/wheelhouse/*manylinux1_*.whl; do + ./ci-scripts/docker/fix_wheel.sh "$whl" +done + +# Install packages and test +TEST_HOME=/persist/klayout/testdata +for PYBIN in /opt/python/*/bin/; do + "${PYBIN}/pip" install klayout --no-index -f /persist/wheelhouse + "${PYBIN}/python" $TEST_HOME/pymod/import_db.py + "${PYBIN}/python" $TEST_HOME/pymod/import_rdb.py + "${PYBIN}/python" $TEST_HOME/pymod/import_tl.py + +``` + +I tested step 1-5 with both quay.io/pypa/manylinux1_x86_64 and quay.io/pypa/manylinux1_i686. So far the only failure was with `cp27-cp27mu` which gave this import error: + +`ImportError: /opt/python/cp27-cp27mu/lib/python2.7/site-packages/klayout/.libs/lib_pya.so: undefined symbol: PyUnicodeUCS2_AsUTF8String` + +I noticed that the ccache folder ended up with 800MB. I was hoping that the gcc compilation could reuse a lot of previously built objects but that didn't happen. I think that's because each python comes with its own header. So going forward it doesn't make sense to create a docker image for every python version. I will just cache a ccache folder via travis. +The ccache folder after a single build has 657MB. Go figure. + +I discovered that fix_wheel script was actually not properly working. So instead I looked into fixing `auditwheel` directly. Here's the commit that fixes it: https://github.com/thomaslima/auditwheel/tree/87f5306ec02cc68020afaa9933543c898b1d47c1 + +So now the plan is to change the `docker_build.sh` script so it uses the proper auditwheel, instead of their default. + +# Chapter 2. Testing CI flow with docker + +# Step 1. Testing commands in own computer + +First cloned with: +```bash +git clone git@github.com:lightwave-lab/klayout.git -b tmp/manylinux +cd klayout +``` + +Let's work with a few environment variables, like in https://github.com/pypa/python-manylinux-demo/blob/master/.travis.yml + +DOCKER_IMAGE options: quay.io/pypa/manylinux1_x86_64 quay.io/pypa/manylinux1_i686 + +PY_VERSION options: cp27-cp27m cp27-cp27mu cp34-cp34m cp35-cp35m cp36-cp36m cp37-cp37m + +Total of 2x6 = 12 possibilities + +```bash +export DOCKER_IMAGE=quay.io/pypa/manylinux1_x86_64 +export PY_VERSION="cp36-cp36m" +docker pull $DOCKER_IMAGE +mkdir -p ccache +mkdir -p wheelhouse +docker run --name klayout -v `pwd` -it $DOCKER_IMAGE +``` + +Inside docker shell: +```bash +yum install -y zlib-devel +yum install -y zip + +yum install -y ccache +ln -s /usr/bin/ccache /usr/lib64/ccache/c++ +ln -s /usr/bin/ccache /usr/lib64/ccache/cc +ln -s /usr/bin/ccache /usr/lib64/ccache/gcc +ln -s /usr/bin/ccache /usr/lib64/ccache/g++ +echo $PATH +# /usr/lib64/ccache:/opt/rh/devtoolset-2/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +export CCACHE_DIR="/io/ccache" + +# Compile wheel +/opt/python/$PY_VERSION/bin/python setup.py bdist_wheel -d /io/wheelhouse/ + +# Bundle external shared libraries into the wheels via auditwheel +for whl in /io/wheelhouse/*linux_*.whl; do + auditwheel repair "$whl" -w /io/wheelhouse/ +done + +# Fix each wheel generated by auditwheel +for whl in /io/wheelhouse/*manylinux1_*.whl; do + ./ci-scripts/docker/fix_wheel.sh "$whl" +done + +``` + +# Step 2. Automating step 1 in travis (CI). +DOCKER_IMAGE options: quay.io/pypa/manylinux1_x86_64 quay.io/pypa/manylinux1_i686 + +PY_VERSION options: cp27-cp27m cp27-cp27mu cp34-cp34m cp35-cp35m cp36-cp36m cp37-cp37m + +Build: spawn 12 travis jobs, one for each combination of word-size and python version. +Output: populated ./ccache with compiled objects and wheels inside ./wheelhouse/, one useless, `*linux_*.whl` and one useful `*manylinux1_*.whl`. +Post-build: +- cache `./ccache` +- deploy `./wheelhouse/*manylinux1_*.whl` to dropbox (./deploy folder) + +# Step 3. Automating deployment to PyPI: + +TBD diff --git a/ci-scripts/docker/fix_wheel.sh b/ci-scripts/docker/development_notes/fix_wheel.sh similarity index 100% rename from ci-scripts/docker/fix_wheel.sh rename to ci-scripts/docker/development_notes/fix_wheel.sh diff --git a/ci-scripts/docker/manylinux-docker.sh b/ci-scripts/docker/development_notes/manylinux-docker.sh similarity index 100% rename from ci-scripts/docker/manylinux-docker.sh rename to ci-scripts/docker/development_notes/manylinux-docker.sh diff --git a/ci-scripts/twine/README.md b/ci-scripts/twine/README.md new file mode 100644 index 000000000..3916de5d4 --- /dev/null +++ b/ci-scripts/twine/README.md @@ -0,0 +1,15 @@ +After building all the travis wheels, go to the folder where the wheels were deployed. In my case, for example, `/Users/tlima/Dropbox/Apps/travis-deploy/Builds/klayout/dist-pymod/0.26.0.dev10`. + +Then, run the command + +```bash +travis upload *.whl +``` + +, which will ask for an username and password related to your account. + +After this upload was successful, you can upload the source tarball. Go to klayout's git folder and run `python setup.py sdist`. Inside `dist/`, you'll find a tarball named, e.g., `klayout-0.26.0.dev10.tar.gz`. So just run + +```bash +travis upload klayout-0.26.0.dev10.tar.gz +```