hateful semi-circular dependencies

Recently, I’ve been trying to help do my part to prevent needless memory consumption. Today, at work, Dieter and I were talking about ways to reduce Email::Simple’s significant memory consumption. Here’s an example: I created an eight megabyte email (50 lines of headers, 10,000 of body) and wrote a program that slurps it and makes a new Email::Simple, giving a crude (ps-based) memory check as it goes.

knight!rjbs:~/code/pep/Email-Simple/trunk$ perl readmail big.msg
just started                :   1364    28328
after require File::Slurp   :   2232    28704
after slurping              :  17876    44336
after require Email::Simple :  17936    44336
after construction          :  57060    83416

Those numbers are resident set size and vsize. This means that when creating a new Email::Simple for this message, we’re using eight times its size in memory while constructing, and end up still using four times its size when we’ve got the object built.

Some smart-but-not-clever (I hope) optimization cuts this down drastically:

knight!rjbs:~/code/pep/Email-Simple/trunk$ perl -I lib readmail big.msg   
just started                :   1364    28328
after require File::Slurp   :   2232    28704
after slurping              :  17880    44336
after require Email::Simple :  17940    44336
after construction          :  25800    52152

We end up using about the message’s size in resident memory, having peaked at using about four times its size during construction. This optimization, though, is based on referring to the body as a reference. If we’re willing to give up the scalar that held the slurped content, and pass it to the constructor as a reference, we can save even more memory:

knight!rjbs:~/code/pep/Email-Simple/trunk$ perl -I lib readmailref big.msg
just started                :   1364    28328
after require File::Slurp   :   2232    28704
after slurping              :  17876    44336
after require Email::Simple :  17936    44336
after construction          :  17988    44336

We end up only using about 50kB more for the object, and during construction don’t exceed the vsize that we’d reached slurping the file! Fantastic!

Well, except for one thing. Email:: code has a habit of exposing its guts, in lieu of a sufficient API. Maybe the original idea was that this would only be done internal to Email:: development, but that hardly helps once the code is popular! See, we’re changing what’s stored in $self->{body}, which users should only access with the body and body_set methods. Sadly, though, Email::MIME feels itself to be above these restrictions, and fiddles with the guts directly.

This is a pain because if I were to release this optimized version as a normal upgrade to Email::Simple, it would break Email::MIME on any system on which it was installed. I can’t just up the version of Email::Simple that Email::MIME requires, because, well, that won’t help. I don’t want to make Email::MIME a prerequisite of Email::Simple because (a) that’s stupid and (b) it would create a circular dependency. I don’t really want to distribute them together either.

If I weren’t also maintaining Email::MIME, I could just take some kind of sadistic glee in punishin the wicked: if you stick you hand into my objects’ guts, you should expect bad things to happen. I think my only option is this ugly hack: when Makefile.PL is run, if an older Email::MIME is installed, require a newer one. Otherwise, do nothing. I just have horrible visions of accidentally creating a circular dependency in the future.

I think I’m going to need to go through all the Email:: code and deal with this kind of gut-touching before it causes me, or anyone else, future grief.

Written on November 28, 2006
📧 email
🐫 perl
🧑🏽‍💻 programming