Take the 2-minute tour ×

I'm programming in Python and I'm obtaining information from a web page through the urllib2 library. The problem is that that page can provide me with non-ASCII characters, like 'ñ', 'á', etc. In the very moment urllib2 gets this character, it provokes an exception, like this:

File "c:\Python25\lib\httplib.py", line 711, in send
File "<string>", line 1, in sendall:
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 74: ordinal not in range(128)

I need to handle those characters. I mean, I don't want to handle the exception but to continue the program. Is there any way to, for example (I don't know if this is something stupid), use another codec rather than the ASCII? Because I have to work with those characters, insert them in a database, etc.

share|improve this question
It would be useful if you could say, also, whether you're using Python 3+, or something earlier. –  Sixten Otto Oct 29 '09 at 17:04
Couldn't be Py3k since the urllib2 module has been removed (wrapped into urllib)... –  Tim Pietzcker Oct 29 '09 at 17:09
Duplicate: stackoverflow.com/questions/1020892/… –  S.Lott Oct 29 '09 at 18:19
add comment

3 Answers

up vote 3 down vote accepted

You just read a set of bytes from the socket. If you want a string you have to decode it:

yourstring = receivedbytes.decode("utf-8") 

(substituting whatever encoding you're using for utf-8)

Then you have to do the reverse to send it back out:

outbytes = yourstring.encode("utf-8")
share|improve this answer
add comment

You want to use unicode for all your work if you can.

You probably will find this question/answer useful:


share|improve this answer
add comment

You might want to look into using an actual parsing library to find this information. lxml, for instance, already addresses Unicode encode/decode using the declared character set.

share|improve this answer
Unfortunately a lot of website produce improperly encoded documents, generally the encoding will be mostly correct, but there will be sporadic invalid byte sequences. Some applications won't have to worry about this, but if you are crawling random public web sites, it will be a problem. –  mikerobi Apr 25 '12 at 21:09
add comment

Your Answer

