Python 3.0 Support about mutagen HOT 39 CLOSED

I managed to add Python 3 support, but only for mp3 files and id3 headers, so it can be done.  Both reading and writing works in the minimal testing I've done.

 It's more of a hack than a port, but it works for my current needs.

Well, the good news is that Python 3 reverts to that behavior.  ;)

Let me know if you'd like me to revise my patch in any way.

Indeed, the problem appears to be that I'm too old and remembering pre-new-style behavior.

I believe they were hashable before.  All I did was make their existing hashing behavior explicit, by inheriting their parent class's __hash__ method (or lack thereof).  (For example, id3.Frame subclasses explicitly inherit Frame's __hash__ method, which raises a TypeError.)

This was a subtle change that happened with the advent of new-style classes, so you may not have been aware of this behavior.  See < http://mail.python.org/pipermail/python-bugs-list/2003-February/016089.html > and < http://www.mail-archive.com/[email protected]/msg116409.html >.

To your example, DictMixIn and ID3TimeStamp were hashable because they inherited object's __hash__ method.  (Python 3 changes that behavior, blocking the inheritance, which is why the -3 flag produces a warning.)  If your intent was for those objects not to be hashable, you could easily make that explicit by defining __hash__ to raise a TypeError.

Likewise, flac.MetadataBlock inherited object's __hash__ method, so that it and all its subclasses were hashable, regardless of whether they defined __eq__ or __cmp__.  Again, if you want the subclasses that define __eq__ or __cmp__ not to be hashable, you can explicitly define their __hash__ method to raise a TypeError.

A test in 2.4:

>>> class InheritedHash(object):
...     def __eq__(self, other): return self.name == other.name
... 
>>> class Unhashable(object):
...     def __hash__(self): raise TypeError("unhashable object")
... 
>>> class SimpleUnhashable(object):
...     __hash__ = None           
... 
>>> a = InheritedHash()
>>> b = Unhashable()
>>> c = SimpleUnhashable()
>>> d = { a: "whee" }
>>> d = { b: "whee" }
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in __hash__
TypeError: unhashable object
>>> d = { c: "whee" }
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: 'NoneType' object is not callable

Python 2.6 cleans up this last case by producing the following output without requiring you to raise a TypeError in all of your classes:

>>> d = { c: "whee" }
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'SimpleUnhashable'

To conclude, I believe my patch introduced no change in behavior, but it seems that it uncovered some undesirable implicit behavior, given that these objects are mutable.  I'd be happy to submit a revised patch, but you may be best suited to determine which objects are intended to be mutable and which are not.

Your patch seems to add __hash__ to some things that did not have it before, thus making them hashable. Why?

For example, DictMixin and ID3TimeStamp both have comparison methods, but not __hash__, which is correct for them, but you gave them object.__hash__. Likewise, flac.MetadataBlock said nothing about hashes so subclasses were only hashable if they did not define __eq__ or __cmp__; your patch gives several of them explicit hashes, making them both mutable and hashable.

If you run ./setup.py coverage it will generate a coverage report of the test suite. I am not sure why the warning didn't trigger for you, because the reduce call in mutagen.flac is reached 29,990 times during the test suite according to my report.

Here's an initial patch, based on 1.20.  This patch was tested by running "python -3 setup.py test".  Before patch, it produced 55 warnings.  After, it produces none.

Notes:

1) I assumed that you wanted no change in behavior, so all the explicit __hash__ functions simply inherit from the parent class (as they do by default in Python 2.x).  There may be some of these classes for which you want to harmonize __hash__ with __cmp__/__eq__, but that would change the functionality.

2) The import of the "sets" module in test_id3.py appeared to have no purpose, so I removed it.  If you'd rather preserve it, I can easily wrap it inside a test for Python version.

3) Some deprecation warnings may not appear until code is exercised, so I may not have caught all of the warnings.  For example, the functools.reduce() warning mentioned in my initial report doesn't get triggered by the the test suite.  I caught it in my usage:

    import mutagen.flac
    f = mutagen.File("foo.flac")  # triggers functools.reduce warning

I suspect that means this code isn't getting tested by the test suite.  I'm not sure what tests would be appropriate.

Patches for such concerns will be applied if they're not crap; issue 74 already contained one. Mutagen still tries to target Python 2.3, so keep that in mind when writing patches.

I have a slightly revised request:  can you at least update it so that running it with the "-3" flag (under 2.6 or later) doesn't issue deprecation warnings?  Some day Python 3 will probably take over, so I use the -3 flag to keep myself from getting into habits that I'll have to break later.

The particular warnings I get when I import mutagen are:
mutagen/id3.py:763: DeprecationWarning: Overriding __cmp__ blocks inheritance of __hash__ in 3.x
  class ID3TimeStamp(object):
mutagen/id3.py:1125: DeprecationWarning: Overriding __eq__ blocks inheritance of __hash__ in 3.x
  class TextFrame(Frame):
(lots of the above two, for various classes)
and
mutagen/flac.py:38: DeprecationWarning: reduce() not supported in 3.x; use functools.reduce()

Both are very easy to tweak in a 2.x-compatible way that will leave you (and others) in a better position to port to 3.x some day.

Do you really find serious value in Python 3? I mean something that really helps you program better, not that philosophically speaking, variables first bound within expressions should not leak to the enclosing local scope.

I'm kinda working on it on and off.
There are a freaking million bytes vs string vs the old unicde errors.
And then there are more subtle errors.
And then I get annoyed with it and go do something else for a while. :P

But do you really plan to stay with Python 2 forever?

If no one is working on this, I'm going to close it as WontFix. Python 3 remains dead to me.

The following works fine in Python 2.x. It might be considered not as clean. On the other hand, it's a pretty dumb feature to rewrite your entire language for.

def foo(*args, **kwargs):
    my_bool = kwargs.pop("stupid_special_flag", False)
    assert(not kwargs)

99% of my code would work fine in 2.x. The problem is the 1% that doesn't, because those are often pivotal points of the program. One of the biggest examples is keyword-only arguments to functions. I don't use them very much, but in the few places I do, its absolutely critical that I do.

A good example is a class of mine that takes an arbitrary number of arbitrary-typed arguments, as well as an optional boolean flag. You can't trim the last entry from args if its a boolean, because args could be intended to contain booleans and the flag intended to be unset. A keyword-only arg is the only solution, as far as I'm aware. Feel free to prove me wrong though. :P

If they were to back-port support for keyword-only args (since regular args work the same as always, so it wouldn't break old code), I'd keep using 2.x as well, since lots of the new features are indeed being backported.

Another difference I noticed is that in 2.x,
[b for c in d]
will result in the variable c floating around in the local scope, while this doesn't happen in 3.x.
I discovered this when mutagen/_util.py threw a NameError at me after some clean-up code tried to del two variables that it didn't have to.

Woah. I see what you mean, davydm: There are TONS of places where there are errors due to mix-ups between str and bytes. Like assertions that fail because 'a' != b'a', and the like. I did notice that str.encode/decode work in 2.x, so perhaps putting in a lot of those in the 2.x source and running it through 2to3 again will help solve some problems.

Ideally, a quick shot through 2to3 should yield runable code, so that will be my goal.

nerdywhiteguy, nearly everything you describe (new exception syntax, decorator use) is what 2to3 does. That's not the hard part. What's hard is making sure everything is still correct. I haven't downloaded the tarball to check (because, well, it's a tarball rather than a patch) but I find it very unlikely that our test suite comes anywhere close to passing.

Mutagen uses "old" Python constructs because it is an "old" Python program written by even "older" Python programmers. There's no point in churning code for that kind of stuff.

Also, I still don't see 3.x being relevant. Frankly I still have hope it will get over itself and die already. If I were you, I'd rewrite your projects to support 2.x. There's zero downside to supporting it.

Absolutely do not just post a tarball; those make it nearly impossible to see what changed. Instead follow the instructions on the Source tab to check out mutagen into some directory. Then make all your changes. Then in the root directory, run `svn diff > my.patch` to generate a patch.

While 2.x won't cease to exist for a long time, more and more projects are going to be written in 3.x as time goes on. I know from personal experience that writing code that works _simultaneously_ in both can be quite a headache, but making little syntax changes to the existing 2.x code now can make it _tons_ easier to port to 3.x later. When looking at the current source, the most obvious to me is davydm's #4: There are already places in the source that use the constructor method, so why not use it everywhere? It it just a matter of when stuff was written? If its just a matter of taking the time to go through the entire source and making little syntax adjustments like this, I'd be perfectly willing to do so.

Alternate solutions to #1 and 2 (assuming 2to3 doesn't do it satisfactorily) are to use explicit constructors like long(100) for 100L and unicode('foo') for u'foo'. Then saying
if 'long' not in dir(__builtins__):
  long = int
if 'unicode' not in dir(__builtins__):
  unicode = str
should make everyone happy.

Also, I noticed a few other syntax oddities in the source, like one spot where it goes
def score....
score = staticmethod(score)
instead of just saying
@staticmethod
def score....
Is this like this deliberately? or has it just been that way since the dawn of time and never changed? Or am I missing something?

I've got a couple of personal projects that could _really_ benefit from seeing mutagen work in 3.x, so if there is some way I could contribute these minor little syntax tweaks I'd love to. (I'm a total n00b at the whole svn thing, so how would I go about possibly joining this project and submitting patches? Or should I just make the changes and post it in a tarball here?)

My machine still gets 14% more pystones on 2.6 than 3.1. Most of the performance improvements are getting backported. Most of the slow things are not.

Just so no-one is left in the dark: I haven't done much work on mutagen with py3k for a while -- I have to admit I was a little defeated when I was about 90% done and a new version of mutagen came out with quite significant changes -- not that I'm complaining (: I just don't want anyone to think I'm holding some amazing conversion here where no-one can see it (: I will get back on to it at some point -- the urgency has dropped since I've found (imperfect) ways to use a locally-installed py26 in preference to py31 on both windows and *nix. But I still maintain this is an important step forward -- Mutagen ROCKS and py3k has proven itself to, if nothing else, provide significant performance boosts in areas where I've used it.

Hi Davydm, were you able to get Mutagen to work on 3.1?

It's basically impossible to support both reasonably old versions Python 2.x and Python 
3 (Google stills run Python 2.4 internally, I believe; "old" but common versions of OS 
X still come with 2.4 and 2.5). I think Mutagen still supports 2.3, although I have to 
admit I haven't tested it in a while, so 2.4 might be our baseline now.

I have my doubts that Python 2 will cease to exist before, let's say, 2015.

Yeah, reading the comment about 2to3 versus looking at the actual source (and working
through issues as I come across them) makes me think that any conversion script is
going to cause more headaches than it solves. Sure, a script could sort out the
changes in syntax for, eg, exception handling or unicode strings being the "norm".
But that was a minor part of the deal -- I'm working through problems with when
something should be a bytes() type and when it shouldn't -- as well as how to achieve
things where were quite easy in py26 (such as finding the first occurrence of "\xff"
in a string of bytes). Where possible, I'm trying to keep the code working for both
python 2.x and py3k. Like I said above, I'm not a zealot for either platform -- I
just realise that, at some point, there will cease to be a python[N-1] which means
that I need to get stuff working on python[N] and mutagen has saved me a lot of
headache and hassle wrt fappy -- if I can overcome the issues that I have on a py3k
interpreter (and it just seems that I need more time than I thought: I'm slowly
getting there) then I would gladly contribute them back -- if anyone is interested
and my code meets the standards requirements for mutagen code.

Because Mutagen actually handles Unicode in complicated but correct ways, 2to3 crapped 
all over it last time I tried it.

davydm: I haven't looked at your code, but from your description it sounds like
you're trying to port mutagen directly, rather than using the 2to3 tool. The best
approach to achieve compatibility with py3k is to use 2to3 to generate the py3k
version automatically. In case 2to3 doesn't produce working code, any ambiguities
should be fixed in the 2.x source, not the converted source. http://docs.python.org/library/2to3.html http://diveintopython3.org/porting-code-to-python-3-with-2to3.html

gah, ignore me -- there are apparently kinks to work out first. I will beat this
though (:

Also, I forgot to add that I use io.StringIO when cStringIO.StringIO can't be loaded.

Hi

I have done preliminary porting to get this to work under python 3.1. Basically, the
install works and I'm assuming that the following hold true:

1) since all strings in python 3.1 are supposed to be utf, the preceding 'u' for a
string declaration isn't required -- I've removed them
2) since the short and long int have been unified, the trailing 'L' for a long int
should be unnecessary -- so I've dropped them.
3) Python 3.1 doesn't seem to like tuple variables in function declarations -- or,
rather, doesn't provide the syntactic sugar of automatically expanding them into the
named variables. For example, the following definition is no longer valid:
def __key_sort((key1, v1), (key2, v2)):
I've replaced the only occurrences of this with boilerplate code to check if the
variables sent in are tuples with minimum length 2 and then expanded the tuples out
into variables of the same names (key1, v1, key2, v2). Since this happens in code
that I don't really traverse, I would appreciate feedback.
4) I've replaced all exception handling which used the comma-notation with the more
functional-style notation (which works just fine under py2.6), so, for example:
except Exception, e:
is replaced with
except Exception as e
and
raise Exception, "foo to the bar"
is replaced with
raise Exception("foo to the bar")

The problem with (1) and (2) is that I can't seem to put them in try blocks or
equivalent -- basically, they have to be "as-is" in the code, which I'm assuming will
break 2.6 usage, since I'm assuming the original author put them in there for a reason.

I would appreciate it if anyone interested (in particular, the mutagen author, Joe
Wreschnig, without whose excellent code, fappy (also to be found on google code)
would not exist) would get back to me. Whilst the transition to py31 is a little
painful from py26 (I've been putting it off for *ages*), it's gonna happen some time,
and I have to assume that the py devs have good reasons for breaking backwards
compat. Personally, I want to deploy to as many people as possible, so, where
possible, I just try to write code that works in both. Points (1) and (2) above are
sticky wrt cross-interpreter code, but I'm open for suggestions. I don't believe that
3.1-specific or 2.6-specific code is "the way to go" -- rather, let's try make this
rad library work *everywhere*  (:

Python 3 is meant to be faster but since it's new... The code for it will change over
time. I tried doing a 2 to 3 for Musicbrainz Picard. Broke the code fully. :P Wud be
good to have Python 3 support at some point tho, when everyone else decides to switch
over.

Sounds fine.  

I'm not into python enough to know why 3 is worse than 2 (I just downloaded it
because it was newer of course.)  But I trust your experience, so I'll switch back. 
The only difference I can notice is that 'print' is a function now and not a
statement (Hope those are the right terms.)  Plus I want to use mutagen, so I have to
go back.

There's no Python 3 support.

We'd basically need another maintainer for a Python 3 version who would have to do
most of the porting by hand, since I'm not interested in doing it (Python 3 is shit),
and 2to3 also falls flat on its face trying to convert Mutagen.