Releasing a new version of khmer

This document is for khmer release managers, and details the process for making a new release of the khmer project.

How to make a khmer release candidate

Michael R. Crusoe, Luiz Irber, and C. Titus Brown have all been release makers, following this checklist by MRC.

  1. The below should be done in a clean checkout:

    cd `mktemp -d`
    git clone git@github.com:dib-lab/khmer.git
    cd khmer
    
  2. (Optional) Check for updates to versioneer:

    pip install --upgrade versioneer
    versioneer-installer
    
    git diff
    
    ./setup.py versioneer
    git diff
    git commit -m -a "new version of versioneer.py"
    # or
    git checkout -- versioneer.py khmer/_version.py khmer/__init__.py MANIFEST.in
    
  3. Review the git logs since the last release and diffs (if needed) and ensure that the ChangeLog is up to date:

    git log --minimal --patch `git describe --tags --always --abbrev=0`..HEAD
    
  4. Review the issue list for any new bugs that will not be fixed in this release. Add them to doc/known-issues.txt

  5. Verify that the build is clean: http://ci.oxli.org/job/khmer-master/

  6. Submit a build to Coverity Scan if it hasn't been done recently. You can get the token from https://gitlab.msu.edu/ged-lab/ged-internal-docs/wikis/coverity-scan or https://scan.coverity.com/projects/621?tab=project_settings

    virtualenv coverityenv
    source coverityenv/bin/activate
    make install-dependencies
    make clean
    cov_analysis_dir=~/src/coverity/cov-analysis-linux64-7.5.0/ make coverity-build
    COVERITY_TOKEN=${COVERITY_TOKEN} make coverity-upload
    
  7. Set your new version number and release candidate:

    new_version=1.3
    rc=rc1
    

    and then tag the release candidate with the new version number prefixed by the letter 'v':

    git tag v${new_version}-${rc}
    git push --tags git@github.com:dib-lab/khmer.git
    
  8. Test the release candidate. Bonus: repeat on Mac OS X:

    cd ..
    virtualenv testenv1
    virtualenv testenv2
    virtualenv testenv3
    virtualenv testenv4
    # First we test the tag
    
    cd testenv1
    source bin/activate
    git clone --depth 1 --branch v${new_version}-${rc} https://github.com/dib-lab/khmer.git
    cd khmer
    make install-dependencies
    make test
    normalize-by-median.py --version 2>&1 | grep khmer\ ${new_version}-${rc} && \
            echo 1st manual version check passed
    pip uninstall -y khmer; pip uninstall -y khmer; make install
    mkdir ../not-khmer # if there is a subdir named 'khmer' nosetest will execute tests
    # there instead of the installed khmer module's tests
    pushd ../not-khmer; nosetests khmer --attr '!known_failing'; popd
    
    
    # Secondly we test via pip
    
    cd ../../testenv2
    source bin/activate
    pip install -U setuptools==3.4.1
    pip install -e git+https://github.com/dib-lab/khmer.git@v${new_version}-${rc}#egg=khmer
    cd src/khmer
    make install-dependencies
    make dist
    make test
    cp dist/khmer*tar.gz ../../../testenv3/
    pip uninstall -y khmer; pip uninstall -y khmer; make install
    cd ../.. # no subdir named khmer here, safe for nosetesting installed khmer module
    normalize-by-median.py --version 2>&1 | grep khmer\ ${new_version}-${rc} && \
            echo 2nd manual version check passed
    nosetests khmer --attr '!known_failing'
    
    # Is the distribution in testenv2 complete enough to build another
    # functional distribution?
    
    cd ../testenv3/
    source bin/activate
    pip install -U setuptools==3.4.1
    pip install khmer*tar.gz
    pip install nose
    tar xzf khmer*tar.gz
    cd khmer*
    make dist
    make test
    pip uninstall -y khmer; pip uninstall -y khmer; make install
    mkdir ../not-khmer
    pushd ../not-khmer ; nosetests khmer --attr '!known_failing' ; popd
    
  9. Publish the new release on the testing PyPI server. You will need to change your PyPI credentials as documented here: https://wiki.python.org/moin/TestPyPI. You may need to re-register:

    python setup.py register --repository test
    

    Now, upload the new release:

    python setup.py sdist upload -r test
    

    Test the PyPI release in a new virtualenv:

    cd ../../testenv4
    source bin/activate
    pip install -U setuptools==3.4.1
    pip install screed nose
    pip install -i https://testpypi.python.org/pypi --pre --no-clean khmer
    nosetests khmer --attr '!known_failing'
    normalize-by-median.py --version 2>&1 | grep khmer\ ${new_version}-${rc} && \
            echo 3rd manual version check passed
    cd build/khmer
    make test
    
  10. Do any final testing (BaTLab and/or acceptance tests).

  11. Make sure any release notes are merged into doc/release-notes/.

How to make a final release

When you've got a thoroughly tested release candidate, cut a release like so:

  1. Create the final tag and publish the new release on PyPI (requires an authorized account).:

    cd ../../../khmer
    git tag v${new_version}
    python setup.py register sdist upload
    
  2. Delete the release candidate tag and push the tag updates to GitHub.:

    git tag -d v${new_version}-${rc}
    git push git@github.com:dib-lab/khmer.git
    git push --tags git@github.com:dib-lab/khmer.git
    
  3. Add the release on GitHub, using the tag you just pushed. Name it 'version X.Y.Z', and copy and paste in the release notes.

  4. Make a binary wheel on OS X.:

    virtualenv build
    cd build
    source bin/activate
    pip install -U setuptools==3.4.1 wheel
    pip install --no-clean khmer==${new_version}
    cd build/khmer
    ./setup.py bdist_wheel upload
    
  5. Update Read the Docs to point to the new version. Visit https://readthedocs.org/builds/khmer/ and 'Build Version: master' to pick up the new tag. Once that build has finished check the "Activate" box next to the new version at https://readthedocs.org/dashboard/khmer/versions/ under "Choose Active Versions". Finally change the default version at https://readthedocs.org/dashboard/khmer/advanced/ to the new version.

  6. Delete any RC tags created:

    git tag -d ${new_version}-${rc}
    git push origin :refs/tags/${new_version}-${rc}
    
  7. Tweet about the new release.

  8. Send email including the release notes to khmer@lists.idyll.org and khmer-announce@lists.idyll.org

BaTLab testing

The UW-Madison Build and Test Lab provides the khmer project with a free cross-platform testing environment.

  1. Connect to their head node:

    ssh mcrusoe@submit-1.batlab.org
    
  2. Move into the khmer directory and download a release from PyPI's main server or the test PyPI server:

    cd khmer/
    wget https://testpypi.python.org/packages/source/k/khmer/khmer-1.0.1-rc3.tar.gz
    vim khmer-v1.0.inputs # change the 'scp_file' to point to the release
    vim khmer-v1.0.run-spec # change 'project_version' at bottom
    nmi_submit khmer-v1.0.run-spec
    

Setuptools Bootstrap

ez_setup.py is from https://bitbucket.org/pypa/setuptools/raw/bootstrap/

Before major releases it should be examined to see if there are new versions available and if the change would be useful

Versioning Explanation

Versioneer, from https://github.com/warner/python-versioneer, is used to determine the version number and is called by Setuptools and Sphinx. See the files versioneer.py, the top of khmer/__init__.py, khmer/_version.py, setup.py, and doc/conf.py for the implementation.

The version number is determined through several methods: see https://github.com/warner/python-versioneer#version-identifiers

If the source tree is from a git checkout then the version number is derived by git describe --tags --dirty --always. This will be in the format ${tagVersion}-${commits_ahead}-${revision_id}-${isDirty}. Example: v0.6.1-18-g8a9e430-dirty

If from an unpacked tarball then the name of the directory is queried.

Lacking either of the two git-archive will record the version number at the top of khmer/_version.py via the $Format:%d$ and $Format:%H$ placeholders enabled by the "export-subst" entry in .gitattributes.

Non source distributions will have a customized khmer/_version.py that contains hard-coded version strings. (see build/*/khmer/_version.py after a python setup.py build for an example)

ez_setup.py bootstraps Setuptools (if needed) by downloading and installing an appropriate version

comments powered by Disqus