Language Detection in Haraka

English emails only please!

A while ago someone (somewhere – maybe the Node.js mailing list, I forget) mentioned that they had ported PEAR’s language detection module to Node. This struck me as a great thing to have in Haraka, as I’m continually bombarded with foreign spam which slips through the filters (mostly mainsleaze, and the SBL clearly misses it). There’s only me on my domain, so blocking anything not in English is a reasonable thing to do.

The first step to this though had to be decoding mail body text properly to UTF-8. There’s been a TODO line in Haraka forever to do this. I just never got around to it.

So after work today I made that work. It took a fair bit of hackery as previously I was decoding base64 line by line, but that can easily break encoding routines because you get partial characters. So I modified this to decode the entire thing at once into a single buffer, then it was just a matter of using node-iconv to convert (and cope with errors).

Then language detection was just a matter of finding the right body part – in emails you can have both a text/plain part and a HTML part saying the same thing (or sometimes different things), so in that case I prefer the HTML part as it’s more likely to contain relevant text. Then I strip HTML and pass it to node-languagedetect (available on npm). This returns a list of possibilities and confidences. Of which I just choose the top one. Bingo.

66 lines of code. It’s pretty horrible so I won’t put it into the distro. But come the next release (when the iconv stuff is there) I’ll be putting it to the test live on my server.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s