May 172014
 

I like the idea of listing changes to my distribution in the long_desciption in setup.py. So a release a go, I started appending docs/changes.rst to my README.rst file that I am using for pypi. It was a simple doc with bulleted lists. The world was good.

    with open('README.rst') as h_rst:
        LONG_DESCRIPTION = h_rst.read()
    
    with open('docs/changes.rst') as h_rst:
        LONG_DESCRIPTION += h_rst.read()

However, I grew unsatisified with my changes.rst file and wanted it to link, ala cross-references to the documentation, so when reading the docs a user could quickly go see the docs for that item. For example:


* added “preserve_mtime“ parameter to :meth:`.put`, optionally updates the remote file’s st_mtime to match the local file.

Sphinx liked it, I liked it.
However, python setup.py check --restructuredtext --strict gagged when it saw it, so would PyPi.

Harumph! I mutter to myself. I want it all, I want Don’t Repeat Yourself(DRY), I want my change log to display on PyPi, I want cross-references in my docs. However, cross-references don’t make sense in the long_description, what they link to isn’t there. I do not want to update changes in two different places, I am already vexed with making sure that just the one document is updated. After all, who likes writing docs more than writing code?

Well, how do I get the cross-references scrubbed out for the long_description? Here is what I coded up:

with open('README.rst') as h_rst:
    LONG_DESCRIPTION = h_rst.read()

with open('docs/changes.rst') as h_rst:
    BUF = h_rst.read()
    BUF = BUF.replace('``', '$')        # protect existing code markers
    for xref in [':meth:', ':attr:', ':class:', ':func:']:
        BUF = BUF.replace(xref, '')     # remove xrefs
    BUF = BUF.replace('`', '``')        # replace refs with code markers
    BUF = BUF.replace('$', '``')        # restore existing code markers
LONG_DESCRIPTION += BUF

It is a Decorate, Scrub, Transform, Undecorate kind of pattern. Stripping out the :roles: tags left the single ` markers. So, if I wanted to change those to ``, I have to hide all of the existing ``, scrub, transform, and then unhide the original ``. And so that is what I did.

I imagine a determined individual could create a regex to cover the entire breadth of sphinx directives and make a sphinx_to_pypi converter. But for me, my itch is scratched. Maybe it will help someone else too.

Huzzah!
Here is the resulting long_description on pypi and the changes doc on pysftp.rtfd.org

Who else is wrangling long_description from RestructuredText documentation? What are you doing?

 Posted by at 1:04 pm
May 042014
 

0.1.3, 0.1.4, 0.1.5 – The Lost Versions

2014-05-05, I am updating this post to give the short answer: use python setup.py register to update your meta-data on pypi. It can be run repeatedly and will modify the meta-data on pypi for the distribution.

and now the original blog post…

During the first launch of YamJam, I went through a series of releases because of rendering issues of README.rst on pypi. WAT!, I say to myself and a few other choice words. I had just created 6 pages of rst, that compiled just fine for my sphinx generated documentation. I had cut and pasted the top portion of my index.rst with some text edits — what can be going on? I uploaded my package with twine – no errors. pypi was seeming to say, everything is peachy, then turning its back and mumbling FU. (Things should not fail silently.) I pulled up my shiny new package on pypi and was met with unseemly, unformatted text instead of a spiffy display.

I google for answers, I review other dists in pypi, pulling up their repos and reading through their readmes — Aha! I say to myself, I don’t see anyone else using an :alt: on their build status badge. I remove mine, go through the release procedures again, I upload and was slapped down yet again.

I grow angry and frustrated, I recheck my readme, I google for issues related. I find mention of “run the docutils command on it”, but no mention of what command. I review even more dists on pypi, seeing other broken readme renderings. I am unsure of what to do. I think — “It must be my .. code:: python, I see lots of :: , I rip out mine and replace with ::. I try again and FAIL.

I reread the rst, I put it through sphinx just by itself and see a warning about duplicate references. I had `view <url1>`_ for docs and `view <url2>`_ . One of those little edits, I mentioned earlier. You think, looks like an href, smells like an href — but no, it is different. That did it, 3 revisions later, I’m happy with the display of my readme on pypi.

It was a day later, while enhancing my release and CI scripts that I found the python setup.py check and python setup.py check --restructuredtext commands. I tested it locally, and sure enough, it would “warn” but not set an return code, so my release script couldn’t detect the failure. I figure, that messing up your pypi page, should equal FAIL not warn. Ok, so I’ll submit a patch to make it ERROR and set a non-Zero return code. I find the code repos and start reading through the code and discover an option I hadn’t seen. -s, strict. That will cause it to FAIL and set a non-Zero return code. So off to the CI script I go, I add the strict option and the test passes when it should have failed. If you don’t have docutils installed, and I didn’t on my CI, it just returns the same response as if it had passed. FACE PALM. pip install docutils

As it stands now, I can detect rst that will cause pypi to fail silently so I am good in that regard, and you should be to, now that you have read this.

Sidebar: why dosen’t setup.py upload and twine upload automatically run the checks supplied by setup.py in their strictest mode? Fail early, the cost is less

Which brings me to unnecessary binding. Why is the description on pypi so tightly and unnecessarily bound to a distribution release. Forcing a new release to fix render problems and typos? Pypi will let us upload packages but not let us edit the description in a web interface WITHOUT having to do an entire re-release? Use the readme as a starting point, let us edit without re-releasing.

Alex G, if you happen to read this — please make this possible on warehouse. Mr Gaynor, Tear Down this binding! Also, Give us download stats, with as much info as possible so we can weed out mirror requests.

Anyone who is thinking, “I’m going to rewrite pip because of X, DON’T” We have had too many installers, to many distutils and setuptools. setup.py is a cacophony of knobs, buttons, dials, many fighting each other. “There should be one– and preferably only one –obvious way to do it.” Unless it has to do with setup.py, then it should be as confusing as possible. There needs to be a pypi/setup.py BDFaW (Benevolent Dictator For a While). If the new solution doesn’t solve the current problem, or creates new problems, it is not a solution, it is just change for the sake of change. Eggs and wheels – harumph, I say.

I enjoy writing code in Python, I endure creating a release.

The world can and should be a better place.

May 012014
 

Since I didn’t get to attend this year, again, I’ve been watching the PyCon videos (thanks pyvideo.org and PyCon!) Out of all the video’s, Carl Meyer’s talk on “Set your code free…”, struck a nerve.

I have a project, YamJam, that I have been using since 2009. The main idea is a framework that allows you to factor out sensitive data from your code before you upload to a 3rd party repos. I’ve got internal and external projects that have been using it for quite a while and it makes these refactorings a breeze. I have other open source projects that get a lot more attention but yet have a niche audience, so I am experimenting to see if it is the lack of documentation and being a conforming pypi dist that is limiting it’s appeal. Yamjam should be popular, because it can scratch an itch caused by Djano. That itch being, “What do you do with settings.py?” It’s got lots of sensitive data that shouldn’t be checked into to a repos but it also has a lot of code that should. It also makes deployment between dev, staging and production easy to do with a checkout.

To that end, I’ve been creating a proper and complete test suite (was doctests) using py.test and tox, Continuous Integration with drone.io, documentation via readthedocs.org and sphinx, spiffing up the setup and dist with the latest distutils and uploading with twine instead of setup.py upload.

Through this whole process, I’ve had a lot of new experiences that I am going to be blogging about in the upcoming weeks. Things I like, really like, things that are annoying and some things that are counter to my way of thinking. During this process I’ve also been filing bug reports when I’ve encountered them and sending out feedback for improvements along the way.

After moving my code from subversion on google code to mercurial on bitbucket (hg convert), I started looking for a CI service to use. Off the bat, I looked at Travis-ci, but unfortunately, travis is a github snob. If you are not hosting your code on github or mirroring off of github then travis-ci is not an option. Some google searching showed that pylint (a tool I like very much and use) moved from internal tools to bitbucket and http://drone.io/ . So off to drone I go.

http://drone.io/ took about 10 minutes from signing in via my bitbucket account to running my first integration. When I learned that drone.io allows you to view the settings for other open source projects build environments, I had python 27, 32, 33 and 34 tests running via tox less than an hour later. What higher praise could I give a service than saying from 0 to testing in 10 minutes? I really, really like drone.io and recommend that you check them out. Unlike Travis-ci, drone.io supports bitbucket, github and google code. Options, I like. It is the same reason I prefer bitbucket to github. Bitbucket supports mercurial and git. I use both dvcs systems. I prefer mercurial. I like that bitbucket allows me to make the choice. Choice makes me happy. Check out my setup on drone.

More to follow.

A date with JSON?

 Python  Comments Off
Dec 012011
 

I’ve run in to this situation a few times and end up having to query my gBrain for the answer. When using json as a transport from python to html/javascript, I frequently end up needing to move date and time data. However, the builtin json module is not happy when you ask it to serialize a date/time object.

>>> json.dumps(datetime.datetime.now())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.6/json/encoder.py", line 367, in encode
    chunks = list(self.iterencode(o))
  File "/usr/lib/python2.6/json/encoder.py", line 317, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/usr/lib/python2.6/json/encoder.py", line 323, in _iterencode_default
    newobj = self.default(o)
  File "/usr/lib/python2.6/json/encoder.py", line 344, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2011, 12, 1, 0, 50, 53, 152215) is not JSON serializable

So this means we need to figure out a work around. The trick is to let the json module know what it should do with a date/time object while leaving the rest of it in place. So, no replacing the default handler.

What we need to do is subclass the JSONEncoder and override the default method

class JSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if hasattr(obj, 'isoformat'): #handles both date and datetime objects
            return obj.isoformat()
        else:
            return json.JSONEncoder.default(self, obj)

Using the hasattr and looking for ‘isoformat’ method will allow this to handle both date objects and datetime objects. So all that is left to demonstrate is how to put it together with the json.dumps method.

>>> json.dumps(datetime.datetime.now(), cls=JSONEncoder)
'"2011-12-01T00:58:34.479929"'

Ok, so now you have this ISO formatted string containing date information, how do you get that converted to a javascript date object once it has been transmitted across the abyss and now resides in a javscript callback function?

var d = new Date("2011-12-01T00:58:34.479929");

Happy Data Slinging!

 Posted by at 1:22 am
Nov 292011
 

So, I’m hacking on a Python for Android project, which is built over top of the SL4A project. I’m currently using the remote method of development where you fire up an interpreter and share it in public mode. Then you import android.py and instantiate an instance of Android with IP and Port information of your public server. You can then hack in your favorite editor on a laptop instead of using a thumb-board or other. It looks like this:

import android  # The SL4A android.py module should be on your sys.path.

ip = '192.168.x.xxxx'
port = 35766
droid = android.Android((ip, port))

What happens is the __getattr__ method of the Android object uses magic to change droid.getenvironment() into an RPC call to the public server and then return the result back as a named tuple. Nice. Being the nosey bugger that I tend to be, I modified the code to add a debug param to the __init__ method, that when set, would print out what was being sent out over RPC and then the raw tuple results. A snippet of the modification goes like this:

  def __getattr__(self, name):
    def rpc_call(*args):
      if self._debug:
          print "droid.%s%s" % (name, str(args))
      res = self._rpc(name, *args)
      if self._debug:
          print "\t%s" % str(res)
      return res
    return rpc_call

You can easily see where I put my “if self._debug” logic in place. Now if I use my modified android.py I can turn on the debug flag and get some 411 on the magic that is going on. It ends up looking like this:

droid.eventWait(3000,)
	Result(id=28, result=None, error=None)
droid.eventWait(3000,)
	Result(id=29, result={u'data': u'@end', u'name': u'dmt:fromClient.speak', u'time': 1322542655069000L}, error=None)
droid.wakeLockRelease()
	Result(id=30, result=None, error=None)
 Posted by at 12:49 am
Nov 262011
 

While I was playing around with Python for Andoid, I was using the webViewShow method to load an interactive html page and set up message passing both from

html/js -> python

and from

python -> html/js

Part of this requires that I knock out and hack some html/js code. However, I am using the Remote method with a public server on my android device, since I am too lazy to set up eclipse and a full blown android dev env. The example code they show, uses an html file located on the sdcard of the device. Of course this brings its own problems, since now I have to mount, edit, unmount between each hack cycle. Ick. Well I could just tell it to load the html from an off device server, but being lazy ( I think I mentioned that already. ) I didn’t want to rsync back and forth to my remote server, set up directories, etc. Also, I didn’t want to set up a Django install just to serve a hacky html script.

So I think to myself, man there has to be some light-weight way to serve this up locally while I’m hacking. So I think, hey CherryPy, but then I remember Edna and it hits me, I can serve static pages out of a directory with just python. A little google-fu and this page appears, giving just the needed incantation.

 python -m SimpleHTTPServer 8080

Which happily serves everything out of the directories below it on the specified port (8080 in this case). I make a little adjustment in to my webViewShow and change it from file:/// to http://my-dev-ip:8080/thefile.html and all is good with the world. As I hack, changes to the html are pulled and served immediately.

Did I mention, Python Rocks!

 Posted by at 1:13 am
Apr 272008
 

App Engine is a pretty dramatic thought departure for lots of programmers who are used to writing an app that runs on a single server and access a single database.  Case in point, there has been a recurring topic of auto-increment fields on the  App Engine list — people trying to implement their own version of it since it is not a native datastore type.

Using an auto-increment field is not the way to go.  It is viable when you only have 1 database but the datastore for your app is going to/can be replicated out to other machines.  This would mean that their exists times, when datastore’ != datastore” — over time datastore’ would be sync’d with datastore” so that datastore’ == datastore”   — this would lead one to believe that there will be times when the idea of an auto-increment field will not be synchronizable or that the result of the synchronization would be less than satisfactory.  My belief that auto-increment fields are the wrong idea in this environment is strengthened by the fact that they are not offered as an intrinsic datatype.

The way to go, in my opinion, is to use UUIDs. (see links below)
  http://docs.python.org/lib/module-uuid.html
  http://www.faqs.org/rfcs/rfc4122.html

Other Thoughts on the topic:

  • data access is very expensive, using a UUID should be faster
  • UUID1 or UUID4 would be the types to consider
  • UUID1 is preferable as it would introduce some machine significance which should make the chances for a collision to be even more remote than for a UUID4 (random)
 Posted by at 10:45 am

Greedy Coin Changer

 Python  Comments Off
Apr 262008
 

Noah Gift over on O’Reilly OnLamp Blog has an article on building a greedy coin changer. That is, given a value, say 71 cents, calculate the fewest coins needed to make the amount. He had listed a number of solutions, but I felt I could do it a bit more pythonic. ;)

#!/usr/bin/env python
"""implement a greedy coin changer, returning the
fewest coins to make the change requested."""
#coin_list can be expanded to include silver dollars 
# and 50 cent pieces by just expanding the coin list
# to [100,50,25,10,5,1] the reulting answer 
#structure will modify itself to reflect 

coin_list = [25,10,5,1]
change_requested = .71
remaining = change_requested * 100 
change_returned = []    #result structure

for coin in coin_list:
    num_coins,remaining =  divmod(remaining,coin)
    change_returned.append(int(num_coins))
    
print change_returned
print remaining

The benefits of this version, are no conditional logic is needed, the coin structure can be modified and the answer will modify itself accordingly.

Apr 082008
 

Now this is truly an interesting development. Google’s just announced App Engine is sure to super-charge the Python community and convert a number of disillusioned developers of other languages in to Pythonistas. There have been lots of interesting comments floating in the blogosphere about what this could mean.

I think it is a great opportunity on a smaller scale than anyone might imagine. Sure, this could serve as the platform for the next YouTube type social-2.x site, but what I think this really means, is that Google is rounding out the Google Apps for Domains by giving the ability to create something more than a brochure-ware style site offered by their current Sites for Google Apps.

Many are looking for Google to use this as an opportunity to expand advertising revenue, and that is certainly possible for widely popular webX.x sites but what they really needed is another tool/knife to hold to the competition’s throats. Looking at the tea leaves in the bottom of my glass, I see something more akin to a SharePoint attack; Going after the S in SMB market.

App Engine allows for authenticating users via Google system, how much longer until we can interact with other Google services in a similar fashion?? Calendaring, GTalk, etc — I’m not talking mashups, something much more refined.

 Posted by at 10:23 pm
Jan 222008
 

There is a great article over on SnapLogic,  SnapLogic Blog Squishy design with Python: Designing in code

The gist of the article is that when developing APIs are never as complete as we want them when we are developing a new system and if you are using a static language you’ve got lots of ramifications to consider and code to rework when you have to expand an API.  However, dynamic languages have a real advantage here and they go on to give a very real example and how they dealt with it.

 Posted by at 12:41 am
Jan 222008
 

There is an interesting write up on plugin architecture – g :: A Simple Plugin Framework

A project that I’m working on is going to require a plugin framework for a number of things: Logic, Data Storage, Reporting and I’ve been keeping my eyes open for papers/articles on plugin frameworks.  Do you know of any resources/articles?

 Posted by at 12:08 am
May 012007
 

Who would have guessed that MS would be the first to really put python in the browser? I know you can jump through hoops to make a browser support python but it was not something you could do for a public website.

There have been a bunch of announcements in the past couple of days about IronPython and SilverLight — will they beat firefox?
Personally, I’m giddy about the prospect of using Python instead of javascript.  John Udell’s blog entry tonight specifically mentions
that exact combination.
http://blog.jonudell.net/2007/05/01/watching-anders-hejlsberg-reinvent-the-relationship-between-programs-and-data/

Finally, things are starting to get exciting again.

 Posted by at 9:51 pm

PyCon: Talk to people.

 pycon, Python  Comments Off
Feb 272007
 

Ned Batchelder blogged some good PyCon advice tonight, “Talk to people.” I completely agree with him on that point.

This was my second PyCon and I spoke to a vastly larger group of people than I did when I attend PyCon 2002. While I did say Hello to a number, the one real conversation I had was with Tim Peters. This year I wasn’t the least bit shy about introducing myself and talking to others. I met a number of great people who just happen to be excellent pythonistas. There was Chris and Ted from ChiPy, Alvin from San Francisco, Christian Tismer, Mark Ramm, Michael Foord, Alex, Ben, Doug (nice job on the pycon schedule) and many, many more. I’ll be blogging about a number of these conversations as time goes on.

 Posted by at 12:20 am
Feb 242007
 

ExoTagging
After listening to the wonderful things being said about SQLAlchemy at PyCon, I wondered if the following idiom could be implemented as a Domain Object with SQLAlchemy.

Often we have legacy data that we have read only access, to which we need to associate additional meta information. Before this has always meant that the application had to be aware of the relationship of the meta data record in the ExoTagging DB and how to map it to the legacy read-only data. If we could map the two into a single Domain Object it would make are apps a lot cleaner. I spoke to Mark Ramm and he believed that it would be possible and in fact it might even be easy although he hadn’t attempted it before.

I call the idiom ExoTagging because it is logically quite like the idea of Tags, except that tags are kept in a table that resides in a database that is separate from the data that is being tagged. Hence Exo-Tag.

Has anyone else already implemented something like this with SQLAlchemy, if so would you mind sharing your experiences?

 Posted by at 4:15 pm
Feb 242007
 

I believe that it is time to resurrect the Omaha Python Users Group. I attended that Python Advocacy meeting on Day 1 of PyCon and listened to Jeff Rush and then talked to the leader of ChiPy, Chris McAvoy. There is an advocacy open session tonight @9pm after the OLPC show and tell session. I am hoping to work with others there to set up a system to share web resources, mailing list, website, etc between users groups. People should be able to go to the python site and “Find a local users Group”. The users groups could video tape speakers, and then share those videos for use at other user group meetings and for advocacy in general. A central point for available speakers and a centralized place for companies like O’Reilly, to offer products, books, and other give-aways that could be requested and used by the different users groups.

 Posted by at 10:50 am