Monday, May 4, 2015

concurrent.futures backport updated

It's been a while since I last updated the backport of the concurrent.futures package for Python versions older than 3.2. I was looking at my github repositories and noticed that I wasn't marked as watching the pythonfutures repository. As I viewed the issue tracker, I noticed a huge slew of issues in it, some of them dating back to 2013! "Oops", I said to myself. I immediately started working on the bugs, mostly applying patches from upstream CPython code. I also decided to drop support for Python 2.5 and 3.1, since nobody seems to be using them anymore. Likewise, the "futures" top level package (which was an alias to concurrent.futures) is now gone.

Several issues were fixed by the 3.0.1 release:

  • CPython issue 16284 (bug #32; memory leak in thread and process pools)
  • CPython issue 11777 (bug #28; map() doesn't start tasks until generator is used)
  • CPython issue 15015 (accessing a nonexistent attribute)
  • CPython issue 20367 (behavior of concurrent.futures.as_completed() for duplicate arguments)
  • backport specific bug #20 (map() is greedy; fixed in 3.0.1)
Many apologies for the unreasonable delays in getting these fixed!

There are still some major issues with ProcessPoolExecutor, but the upstream code in Python 3.3+ now uses Python 3 specific features and I'm not brave enough to try and backport that. I'm not sure it's even possible. So if you need reliable process pools, you're going to have to switch to Python 3 :)