Skip to content

Release Process

Jump to bottom Edit New page
seanh edited this page Oct 2, 2012 · 6 revisions

Table of Contents

Where We Publish CKAN Releases

We publish CKAN releases:

  • As tags on GitHub, e.g. ckan-1.7.1 etc., you can download the source code for a tag from Github as a .zip or .tar.gz file, or you can clone the git repository and checkout the tag.
  • As Python packages on PyPI
  • As Ubuntu packages on apt.ckan.org

Changelog

You can see a list of changes in each CKAN release in the CKAN CHANGELOG.

Doing a CKAN Beta Release

Release a beta when you have new features you want to try out on select users. We normally try out a beta release on thedatahub.org for a week before it is released properly.

1. Run the most thorough tests on the code.

2. Create a new release branch: 

 $ git checkout -b release-v1.2

and push this new branch to the CKAN GitHub project.

3. In ckan/__init__.py change the version number to the new version with a b after it, e.g. 1.2b.

4. Change pip-requirements.txt to point to the new release branch.

5. If you have not got the latest Babel and Transifex clients installed you need to do: 

 $ pip install --upgrade Babel
 $ pip install transifex-client

You will also need to create a ~/.transifexrc file like this: 

[http://www.transifex.net]
hostname = http://www.transifex.net
username = <username>
password = <password>
token =

(token should be left blank).

6. Extract new strings from the CKAN source code into the ckan.pot file: 

 $ python setup.py extract_messages

The pot file is a text file that contains the original, untranslated strings extracted from the CKAN source code.

7. Pull new and updated translations from Transifex into the ckan.po files: 

 $ tx pull --all --force

The po files are text files, one for each language that CKAN is translated into, that contains the translated strings next to the originals. Translators edit the po files (on Transifex) to update the translations. We never edit the po files locally, always edit them on Transifex and then pull the changes down from Transifex.

--force tells Transifex to update all ckan.po files, regardless of mtime. Normally Transifex would only update ckan.po files whose mtimes are older than those of the versions on Transifex, but since we're using source control the mtimes of your files will be when you last checked them out from source control rather than when someone last edited them, so --force is needed.

Now run our script that checks for mistakes in the ckan.po files: 

 $ pip install polib
 $ paster check-po-files ckan/i18n/*/LC_MESSAGES/ckan.po

If the script finds any mistakes correct them on Transifex and then run the tx pull command again, don't edit the ckan.pot files directly. Repeat until the script finds no mistakes.

8. Edit .tx/config, on line 4 to set the Transifex 'resource' to the new major release name (if different), using dashes instead of dots. e.g. v1.2, v1.2.1 and v1.2.2 all share: [ckan.1-2]

9. Update the ckan.po files with the new strings from the ckan.pot file: 

 $ python setup.py update_catalog --no-fuzzy-matching

Any new or updated strings from the CKAN source code will get into the po files, and any strings in the po files that no longer exist in the source code will be deleted (along with their translations).

We use the --no-fuzzy-matching option because fuzzy matching often causes problems with Babel and Transifex.

10. Create a new resource in the CKAN project on Transifex by pushing the new pot and po files: 

 $ tx push --source --translations --force

Because it reads the new version number in the .tx/config file, tx will create a new resource on Transifex rather than updating the existing resource (updating an existing resource, especially with the --force option, can result in translations being deleted from Transifex).

11. Update the ckan.mo files by compiling the po files: 

 $ python setup.py compile_catalog

The mo files are the files that CKAN actually reads when displaying strings to the user.

12. Commit all the above changes to git and push them to the CKAN project on GitHub.

13. Send an email to the ckan-discuss list asking people to translate the new strings on Transifex, e.g.:

We're preparing to release CKAN 1.2 in about a week's time and there are new strings that need translating from English to other languages before then. If you can help with translating the words and sentences and keep CKAN open to people across the world, please sign up here to contribute: https://www.transifex.net/projects/p/ckan/resource/1-2

Once you've registered on Transifex, email me your username and I'll provide permissions on the site.

14. On the release branch, update the minified versions of the javascript and css files using the paster minify command: 

 $ paster --plugin=ckan minify ckan/public/base
 $ git commit ckan/public/base -m "Update minified js and css files"
 $ git push

15. Merge the release branch back into the master branch. On the master branch, set the version number in ckan/__init__.py to the number for the next CKAN release with an a after it, e.g. 1.3a, and update pip-requirements.txt to point to the master branch not the release branch. Commit the merge and push it to the CKAN project on GitHub.

Doing a proper CKAN release

Once the beta has been tested manually on a live instance for a week you can do a release.

1. Run the most thorough tests. 

 $ nosetests ckan/tests --ckan --ckan-migration --with-pylons=test-core.ini

2. Summarise the tickets going in this release in the ckan/doc/CHANGELOG.rst and ensure that the date is filled in.

  • To see merged in branches compared to the previous release, do:
   $ git branch -a --merged >merged-current.txt
   $ git branch -a --merged ckan-1.5 >merged-previous.txt
   $ diff merged-previous.txt merged-current.txt
  • Also look at commits to master for ticket numbers - use gitk
  • Also look at tickets closed during the period
3. Remove the beta letter in the version number in ckan/.py e.g. 1.1b -> 1.1

4. Commit this change: 

 $ git commit -a -m '[release]: Version number updated for release.'

5. Pull in the latest translations from Transifex (you need --force, because mercurial doesn't restore file timestamps): 

 $ tx pull --all --force

6. If you've not got msgfmt, then: 

 $ sudo apt-get install gettext

Run checks on the translations. Iron out any errors manually. 

 $ find . -name "*.po" | xargs -L 1 --verbose msgfmt --check

7. Compile translations 

 $ python setup.py compile_catalog 
 $ find . -name "ckan.?o" | xargs git add
 $ git commit -m '[i18n]: Updated translations from Transifex.'

8. Check the docs compile ok: 

 $ rm build/sphinx -rf
 $ python setup.py build_sphinx

9. Tag the repo with the version number (commits change). (add -f to the tag if you want to overwrite a previous tag with this release) e.g. 

 $ git tag -a -m '[release]: Release tag' ckan-1.1 
 $ git push --tags

10. Upload the release to PyPI. You will need a PyPI account with admin permissions on the ckan package, and your credentials should be defined on a ~/.pypirc file, as described here. (If you make a mistake, you can always go into http://pypi.python.org/pypi/ckan to remove the release file and then reupload it.) 

 $ python setup.py sdist upload

11. Create the install package using: http://jenkins.ckan.org 

 * On apt.ckan.org call it, for example: ckan-1.6

12. Close branch and merge to default: 

 $ git checkout master
 $ git merge --no-ff release-v1.3.2

During this merge you will no doubt get conflicts of the i18n files. Ensure that you keep the version on default and ignore any changes on the branch (Ctrl-Shift-2 in kdiff3). The newer translations will be pulled from Transifex in a later step.

13. Manually revert two files: pip-requirements.txt ckan/.py

14. test and commit the merge. 

 $ nosetests --ckan ckan/tests
 $ hg ci -m '[merge] from release-v1.3.2.'

15. Enable the new version of the docs: http://readthedocs.org/dashboard/ckan/versions/ Check it is displayed at: http://docs.ckan.org/

16. If there have been any unreleased changes to OKF dependencies like vdm, ckanclient, datautil etc. make sure you release them too.

17. Write a CKAN Blog post and email to ckan-discuss about the release, including the bit of changelog.

ckanclient releases

  1. Increase version number in ckanclient/.py
  2. Add a changelog for this new version in ckanclient/.py
  3. Check-in the changes
  4. Tag the version: 
 $ hg tag "ckanclient-v0.5"
  1. Create a PYPI config file (~/.pypirc) if necessary:
 [distutils]
 index-servers =   pypi
 [pypi]
 #repository: http://www.python.org/pypi
 username: joebloggs
 password: mypassword
  1. Produce the distribution and upload to PYPI
 $ python setup.py sdist upload
  1. See the upload here: http://www.python.org/pypi/ckanclient
  2. Check that the setup.py for various branches of CKAN have the new version in their range (as appropriate).
  3. Check that the various pip-requirements.txt files also match the new version (as appropriate).

vdm releases

  1. Increase version number in vdm/.py
  2. Add a changelog for this new version in vdm/CHANGELOG.txt
  3. Check-in the changes
  4. Tag the version: 
 $ hg tag "vdm-0.8"
  1. Create a PYPI config file (~/.pypirc) if necessary:
 [distutils]
 index-servers =   pypi
 [pypi]
 #repository: http://www.python.org/pypi
 username: joebloggs
 password: mypassword
  1. Produce the distribution and upload to PYPI
 $ python setup.py sdist upload
  1. See the upload here: http://www.python.org/pypi/vdm
  2. Check that the setup.py for various branches of CKAN have the new version in their range (as appropriate).
  3. Check that the various pip-requirements.txt files also match the new version (as appropriate).
Add a custom footer
Add a custom sidebar

Clone this wiki locally