Dive Into Mark: Universal Encoding Detector

Character encoding auto-detection in Python 2 and 3. As smart as your browser. Open source. More great open-source code from Mark Pilgrim.

>>> import urllib
>>> urlread = lambda url: urllib.urlopen(url).read()
>>> import chardet
>>> chardet.detect(urlread("http://google.cn/"))
{'encoding': 'GB2312', 'confidence': 0.99}

>>> chardet.detect(urlread("http://yahoo.co.jp/"))
{'encoding': 'EUC-JP', 'confidence': 0.99}

>>> chardet.detect(urlread("http://amazon.co.jp/"))
{'encoding': 'SHIFT_JIS', 'confidence': 1}

>>> chardet.detect(urlread("http://pravda.ru/"))
{'encoding': 'windows-1251', 'confidence': 0.9355}

>>> chardet.detect(urlread("http://auction.co.kr/"))
{'encoding': 'EUC-KR', 'confidence': 0.99}

>>> chardet.detect(urlread("http://haaretz.co.il/"))
{'encoding': 'windows-1255', 'confidence': 0.99}

>>> chardet.detect(urlread("http://www.nectec.or.th/tindex.html"))
{'encoding': 'TIS-620', 'confidence': 0.7675}

>>> chardet.detect(urlread("http://feedparser.org/docs/"))
{'encoding': 'utf-8', 'confidence': 0.99}

Great to know!

Dive Into Python 3 - httplib2

Fantastic writing on a fine HTTP library "httplib2" by Mark Pilgrim. Dive Into Python 3 - HTTP Web Services Shows the value of the feature set of HTTP - compression, caching, getting around local and third-party caches, temporary and permanent redirects. "httplib2" supports them all, by default, so you can be a good HTTP-Citizen from the word GO. I read the Reddit comments on this article, and it was pretty depressing. There is a weird reaction to a well designed piece of library code - the better designed and more comprehensive a library is, the more vocal and numerous the irrational criticism for it is.

Image via Wikipedia

People are so terrified to learn a new library, because of a lack of unallocated healthy plastic brain neurons, and because they are terrified of their meagre mental gifts being so publicly obvious, that they have to publicly use all the the odious tools of the Art of Controversy to denigrate it. "EasyInstall" and "Eggs" in Python seem to suffer the worst from this - EasyInstall is hardly perfect, but it helps smart people with tricky Python packaging problems, and it will certainly play a part, in some future incarnation, to the issue of Python packaging for all who wish to do it well. Because of that, and because of Python programmers with a lack of unallocated healthy plastic brain neurons who are terrified of their

Image via Wikipedia

meagre mental gifts being so publicly obvious, you get a lot of trash talk and irrational criticism of the "EasyInstall" library. (Phillip Eby's personality is no help here, and I am hardly his best apologist for his personality defects, but the man does some good work.) The Reddit comments of Mark Pilgrim's "httplib2" write-up is the nittiest of nit-picking, and conflating of pertinent issues. Some idiot was trying to argue that local disc caching is sometimes higher latency than caching on a local network call. I guess some weird local network with a massive RAM cache and only a single workstation user. Dumb stuff. Then some idiot complained that httplib2 does too many "correct" things by default, and it was interfering with him writing a mobile device application, badly.

Image via Wikipedia

All you can do is call it like you see it, and hope the silent crowd of competent developers recognize the garbage comments for what it is. And maintain healthy brain tissues.