pypi:, keeping a DRY long_description

 Python  Comments Off on pypi:, keeping a DRY long_description
May 172014

I like the idea of listing changes to my distribution in the long_desciption in So a release a go, I started appending docs/changes.rst to my README.rst file that I am using for pypi. It was a simple doc with bulleted lists. The world was good.

    with open('README.rst') as h_rst:
    with open('docs/changes.rst') as h_rst:

However, I grew unsatisified with my changes.rst file and wanted it to link, ala cross-references to the documentation, so when reading the docs a user could quickly go see the docs for that item. For example:

* added “preserve_mtime“ parameter to :meth:`.put`, optionally updates the remote file’s st_mtime to match the local file.

Sphinx liked it, I liked it.
However, python check --restructuredtext --strict gagged when it saw it, so would PyPi.

Harumph! I mutter to myself. I want it all, I want Don’t Repeat Yourself(DRY), I want my change log to display on PyPi, I want cross-references in my docs. However, cross-references don’t make sense in the long_description, what they link to isn’t there. I do not want to update changes in two different places, I am already vexed with making sure that just the one document is updated. After all, who likes writing docs more than writing code?

Well, how do I get the cross-references scrubbed out for the long_description? Here is what I coded up:

with open('README.rst') as h_rst:

with open('docs/changes.rst') as h_rst:
    BUF =
    BUF = BUF.replace('``', '$')        # protect existing code markers
    for xref in [':meth:', ':attr:', ':class:', ':func:']:
        BUF = BUF.replace(xref, '')     # remove xrefs
    BUF = BUF.replace('`', '``')        # replace refs with code markers
    BUF = BUF.replace('$', '``')        # restore existing code markers

It is a Decorate, Scrub, Transform, Undecorate kind of pattern. Stripping out the :roles: tags left the single ` markers. So, if I wanted to change those to ``, I have to hide all of the existing ``, scrub, transform, and then unhide the original ``. And so that is what I did.

I imagine a determined individual could create a regex to cover the entire breadth of sphinx directives and make a sphinx_to_pypi converter. But for me, my itch is scratched. Maybe it will help someone else too.

Here is the resulting long_description on pypi and the changes doc on

Who else is wrangling long_description from RestructuredText documentation? What are you doing?

 Posted by at 1:04 pm
May 042014

0.1.3, 0.1.4, 0.1.5 – The Lost Versions

2014-05-05, I am updating this post to give the short answer: use python register to update your meta-data on pypi. It can be run repeatedly and will modify the meta-data on pypi for the distribution.

and now the original blog post…

During the first launch of YamJam, I went through a series of releases because of rendering issues of README.rst on pypi. WAT!, I say to myself and a few other choice words. I had just created 6 pages of rst, that compiled just fine for my sphinx generated documentation. I had cut and pasted the top portion of my index.rst with some text edits — what can be going on? I uploaded my package with twine – no errors. pypi was seeming to say, everything is peachy, then turning its back and mumbling FU. (Things should not fail silently.) I pulled up my shiny new package on pypi and was met with unseemly, unformatted text instead of a spiffy display.

I google for answers, I review other dists in pypi, pulling up their repos and reading through their readmes — Aha! I say to myself, I don’t see anyone else using an :alt: on their build status badge. I remove mine, go through the release procedures again, I upload and was slapped down yet again.

I grow angry and frustrated, I recheck my readme, I google for issues related. I find mention of “run the docutils command on it”, but no mention of what command. I review even more dists on pypi, seeing other broken readme renderings. I am unsure of what to do. I think — “It must be my .. code:: python, I see lots of :: , I rip out mine and replace with ::. I try again and FAIL.

I reread the rst, I put it through sphinx just by itself and see a warning about duplicate references. I had `view <url1>`_ for docs and `view <url2>`_ . One of those little edits, I mentioned earlier. You think, looks like an href, smells like an href — but no, it is different. That did it, 3 revisions later, I’m happy with the display of my readme on pypi.

It was a day later, while enhancing my release and CI scripts that I found the python check and python check --restructuredtext commands. I tested it locally, and sure enough, it would “warn” but not set an return code, so my release script couldn’t detect the failure. I figure, that messing up your pypi page, should equal FAIL not warn. Ok, so I’ll submit a patch to make it ERROR and set a non-Zero return code. I find the code repos and start reading through the code and discover an option I hadn’t seen. -s, strict. That will cause it to FAIL and set a non-Zero return code. So off to the CI script I go, I add the strict option and the test passes when it should have failed. If you don’t have docutils installed, and I didn’t on my CI, it just returns the same response as if it had passed. FACE PALM. pip install docutils

As it stands now, I can detect rst that will cause pypi to fail silently so I am good in that regard, and you should be to, now that you have read this.

Sidebar: why dosen’t upload and twine upload automatically run the checks supplied by in their strictest mode? Fail early, the cost is less

Which brings me to unnecessary binding. Why is the description on pypi so tightly and unnecessarily bound to a distribution release. Forcing a new release to fix render problems and typos? Pypi will let us upload packages but not let us edit the description in a web interface WITHOUT having to do an entire re-release? Use the readme as a starting point, let us edit without re-releasing.

Alex G, if you happen to read this — please make this possible on warehouse. Mr Gaynor, Tear Down this binding! Also, Give us download stats, with as much info as possible so we can weed out mirror requests.

Anyone who is thinking, “I’m going to rewrite pip because of X, DON’T” We have had too many installers, to many distutils and setuptools. is a cacophony of knobs, buttons, dials, many fighting each other. “There should be one– and preferably only one –obvious way to do it.” Unless it has to do with, then it should be as confusing as possible. There needs to be a pypi/ BDFaW (Benevolent Dictator For a While). If the new solution doesn’t solve the current problem, or creates new problems, it is not a solution, it is just change for the sake of change. Eggs and wheels – harumph, I say.

I enjoy writing code in Python, I endure creating a release.

The world can and should be a better place.

Post PyCon momentum and

 Python  Comments Off on Post PyCon momentum and
May 012014

Since I didn’t get to attend this year, again, I’ve been watching the PyCon videos (thanks and PyCon!) Out of all the video’s, Carl Meyer’s talk on “Set your code free…”, struck a nerve.

I have a project, YamJam, that I have been using since 2009. The main idea is a framework that allows you to factor out sensitive data from your code before you upload to a 3rd party repos. I’ve got internal and external projects that have been using it for quite a while and it makes these refactorings a breeze. I have other open source projects that get a lot more attention but yet have a niche audience, so I am experimenting to see if it is the lack of documentation and being a conforming pypi dist that is limiting it’s appeal. Yamjam should be popular, because it can scratch an itch caused by Djano. That itch being, “What do you do with” It’s got lots of sensitive data that shouldn’t be checked into to a repos but it also has a lot of code that should. It also makes deployment between dev, staging and production easy to do with a checkout.

To that end, I’ve been creating a proper and complete test suite (was doctests) using py.test and tox, Continuous Integration with, documentation via and sphinx, spiffing up the setup and dist with the latest distutils and uploading with twine instead of upload.

Through this whole process, I’ve had a lot of new experiences that I am going to be blogging about in the upcoming weeks. Things I like, really like, things that are annoying and some things that are counter to my way of thinking. During this process I’ve also been filing bug reports when I’ve encountered them and sending out feedback for improvements along the way.

After moving my code from subversion on google code to mercurial on bitbucket (hg convert), I started looking for a CI service to use. Off the bat, I looked at Travis-ci, but unfortunately, travis is a github snob. If you are not hosting your code on github or mirroring off of github then travis-ci is not an option. Some google searching showed that pylint (a tool I like very much and use) moved from internal tools to bitbucket and . So off to drone I go. took about 10 minutes from signing in via my bitbucket account to running my first integration. When I learned that allows you to view the settings for other open source projects build environments, I had python 27, 32, 33 and 34 tests running via tox less than an hour later. What higher praise could I give a service than saying from 0 to testing in 10 minutes? I really, really like and recommend that you check them out. Unlike Travis-ci, supports bitbucket, github and google code. Options, I like. It is the same reason I prefer bitbucket to github. Bitbucket supports mercurial and git. I use both dvcs systems. I prefer mercurial. I like that bitbucket allows me to make the choice. Choice makes me happy. Check out my setup on drone.

More to follow.