Announcing Haraka v2.6.0

With thanks to a ton of hard work from the Haraka development team I’m very proud to announce the release of Haraka v2.6.0.

This release is the culmination of a huge number of patches and changes, so please check your configuration carefully when upgrading.
The following is the full list of changes in this release:
* other bug fixes
* updated a few tests so test suite passes on Windows
* log.syslog: handle failure to load node-syslog
* plugin directory is $ENV definable (@martin1yness)
* logging timestamps were static, fixed by @cloudbuy
* queue/rabbitmq_amqplib, new plugin for RabbitMQ using amqplib (@esevece)
* outbound:
    * plugins can set the outbound IP (during get_mx)
    * only replace line endings if not \r\n
    * bannering fixes
    * added support for per recipient routes
* tls: don’t register hooks upless certs exist
* removed contrib/geolite-mirror-simple.pl (replaced by
  docs update pointing to maxmind-geolite-mirror)
* rcpt.routes: new plugin by @msimerson
* make haproxy IPv6 compatible
* record_envelope_addresses: new plugin by @deburau
* prevent_credential_leaks: new plugin by @smfreegard
* config:
    * configfile: added .yaml support
    * improved config file ‘watch’ logic
    * Allow hyphens in params in config files (@abhas)
    * cached requests include options in cache key name
* asn: updates for node 0.11 compat
* dnsbl: use aysync.each vs forEach (avoid race condition)
* spamassassin: improved config loading and test coverage
* geoip: deprecate geoip-lite in favor of maxmind, IPv6 compatible
* disable SSLv3 (due to POODLE)
* dkim & spf, updates for node 0.11 compatibiilty
* karma: move neighbor scoring from code to karma.ini
    * move excludes list to karma.ini
    * apply awards before adding message header & permit rejection at queue
    * karma.ini: score updates for access & uribl plugins
    * score denials issued by skipped plugins
    * add scores for specific DNSBLs
* add transaction body filters (@chazomaticus)
    * change bannering to use them
* helo.checks: fix timeout bug
    * match_re now validates and pre-compiles all REs
    * Add new proto_mismatch check
* p0f: add register(), load config once, early
* server: improved config handling
* data.headers: add Delivered-To check
* rcpt_to.ldap: new plugin by @abhas
* smtp_client: only load tls_* when cfg.enable_tls
* added plugins/host_list_base
* Platform independent temp dir (thanks @martinvd)
* move deprecated docs into docs/deprecated
* Switch to Phusion baseimage instead of stock Ubuntu (thanks @Synchro)
* dkim_verify: new plugin by @smfreegard
* many new tests
* improved URI parser (for URIBL plugin)
* Allow mixed case STARTTLS command
* Install Node via package manager (Mohd Rozi)
* Fix a couple crit errors (@Illirgway)
* Add noisy/bulk out-of-band rule support to MessaageSniffer plugin
* initial support for rabbitmq plugin (@samuelharden)
* bounce, added non_local_msgid checks and much faster lookups
* vpopmail: fail faster during a CRAM-MD5 auth attempt with an invalid user
* fcrdns: handle a null hostname
* Improve HAProxy support code and documentation
* tls: reworked for efficiency and linear style
* access: test hostname validity before PSL lookup
    * load lists into objects (vs arrays), for much faster runtime access
* host_list: huge performance increase, esp for many hosts
Documentation is up to date on http://haraka.github.com/ where you can read more about the many changes in there.
This release has been tested on Node v0.10 and v0.12 and IO.js. It should work on Node v0.8 but we do not recommend it.
Thanks for your continued support of Haraka.

Solving the express ‘array of routes’ problem

Today on hacker news people have been discussing an issue Netflix had with a bug in their route reload code using Express. Now in express, routes aren’t meant to be reloaded (and there are plenty of options for restarting without stopping accepting connections – I’ve even written a simple one myself), but they hacked it in anyway, and every reload resulted in the processing of routes slowing down.

The discussion on hacker news didn’t focus on the actual problem Netflix had, more on the issue that express uses a simple (recursive) loop through an array of routes to find the appropriate route for the request, continuing on to the next matching route if next() is called.

While this isn’t a problem for our systems at work, some people in the discussion and elsewhere said there’s no way to solve this without breaking the semantics of how express currently works (routes have a defined order, and multiple routes can match a single request). Suggestions of a path-based DFA were quickly rebutted.

However that’s a naive way to solve the problem.

V8 has a very fast regular expression engine that compiles regular expressions to x86 machine code. This could be exploited to implement the current routing logic using a large single regular expression (actually N regular expressions, where N is the number of routes, but I’ll come to that).

The implementation is very simple:

  • Take each route (which can be a string, a simplified regexp, or a full regexp) and convert it to a regexp (presumably the router already does this internally because /foo/:id has to match any value for :id).
  • Combine the regexps with bracket matches and alternation: ((re1)|(re2)|(re3)|(re4)|(re5))
  • Generate N regexps, depending on number of routes: ((re1)|(re2)|(re3)|(re4)|(re5)) and ((re2)|(re3)|(re4)|(re5)) and ((re3)| (re4)|(re5)) and ((re4)|(re5)) and ((re5))
  • When a request comes in, check the first regexp, if it matches, you know which alternation matched based on the return from RegExp.prototype.exec(), from there you pick the route function based on an index in an array of route functions.
  • Alternating regexps match in order so this preserves the Express semantics.
  • If next() is called, you go to the next regexp in the list (e.g. ((re2)|(re3)|(re4)|(re5))), and run that. Use the same algorithm (with index + 1) to find the appropriate route function.

This solves the performance problem because the V8 regexp engine finds the appropriate route extremely quickly, and keeps the Express ordering and multiple routes semantics in-tact.

I haven’t proved this is any faster yet, I’m just assuming. I suspect not much for small numbers of routes, and I don’t know if the JS engine has size limits that would prevent it working on large numbers of routes.

Also, this is just an idea I had – I welcome people telling me why it couldn’t work as perhaps my understanding of the Express internals isn’t as strong as I hoped.


Egnyte Support for Email It In

Email It In has just added support for the Egnyte hybrid cloud storage service. Unlike the other services Email It In supports (Google Drive, OneDrive and Dropbox), Egnyte allows their customers to maintain storage on-premises, as well as providing a cloud storage service.

To add Egnyte support we worked very closely with both Egnyte and their users to ensure we offered the best service possible. Users of Egnyte can sign up for free trials at https://emailitin.com/


Haraka and shellshock

It has recently been shown that Qmail is vulnerable to shellshock if you use a pipe filter in a .qmail file (as I do on one of my own machines).

I want Haraka users to know that if you have Haraka in front of Qmail, you are NOT vulnerable to this.
The reason being that Haraka validates MAIL FROM commands according to RFC 5321 rules, whereas Qmail does not – it simply passes any string through untested into the environment. I believe the same safety to be true of Qpsmtpd, though I have not tested it there.
All that said, upgrade your servers anyway. This is a nasty bug with multiple attack vectors.

Irritate your developers

Nothing motivates a developer more than being irritated by something.

It’s why we’re hackers in the first place. I remember my first real programming experience being using the “Freeze Frame” cartridge on the Commodore 64 so I could change the number of lives I had in a game. The irritation of not being able to finish the game drew me into hacking. Most programmers have similar stories – of how something irritated them enough to start hacking on it.

We can use this to our advantage in our day to day work. One excellent example I remember hearing when I was in the Perl community (I think it was from @chromatic, but I don’t recall exactly) was at the end of each day, write a test that fails based on what you’re working on. You’ll come in next morning and want/need to fix that test.

My most recent examples of irritating myself have been when trying out new external systems that implement webhooks. When I implemented Stripe payments for EmailItIn I had no idea which webhooks I might need to pay attention to. So rather than pour over the documentation for hours until I figured it out, I just set it up to irritate me – every call from Stripe webhooks emails me the entire JSON structure they send. Over time this becomes irritating fairly quickly, and so like any good hacker I start to notice patterns and things I’m interested in and not interested in, and I can setup code to filter out irrelevant webhooks, and write code that deals with things like “subscription_cancelled”.

Another way you can use this to your advantage is to email every single error (or in Node, every call to console.error()) on your site. This might scare a lot of people, but if you’re seeing errors often enough to irritate you, that is something that needs fixed, fast. And if you don’t do this those errors often get lost in your logs (you do keep logs, right?). At Ideal Candidate we email all developers when any of the following conditions occur:

  • A console.error() call – we shim this to provide a stack trace in the email
  • A server side exception which brings the server down
  • A client side exception occurs – this does a POST back to our server which triggers the email

Does this amount of emails scare you? If it does you probably have far too many errors occurring in your application. Give it a try – you might find a lot of issues that you didn’t even know you had.
Continue reading


SOLVED: Drobo doesn’t remount at boot on OS X Mavericks

For a long time I loved my drobo. Then I upgraded to Mavericks and all went to shit. On reboot I would have to carefully unmount the now read only drobo, and remount, a process that took around 30 minutes. Only then would it be properly mounted.

I recently discovered HFS journaling wasn’t enabled on the drive. Turning that on in disk utility (a big button in the toolbar) fixed the reboot woes.

Putting this blog post here in the hopes I will help others with the same problem.


Haraka v2.5.0

After a massive development effort a new stable release of Haraka is out.

v2.5.0 contains a huge number of changes, of which these are the highlights:

  • A new feature called ResultStore (or Results) which allows plugins to communicate information they found to each other. This is primarily used by the karma plugin to use the results of other plugins to penalise senders
  • A -o/–order option to bin/haraka to show the order all currently configured plugins will run.
  • IPv6 support on outbound via `echo 1 > config/outbound.ipv6_enabled`
  • Attachment streams now have access to the MIME header for that attachment via stream.header
  • A new “deferred” hook for outbound mail, called when mail is temporarily denied
  • Outbound “bounce” hooks now receive an Error object, containing the MX and recipient information. Existing uses of the error being treated as a string should still function as normal
  • Outbound gets more parameters on the “delivered” hook to allow detailed analysis of sent mail
  • Outbound UUID now distinguishes between different domains using .1, .2 etc just as inbound gives a new index for each transaction
  • Outbound won’t try and send to domains publishing NULL MX (draft-delany-nullmx-02)
  • Outbound template can use `extended_reason` to show a more detailed error
  • Log lines are coloured if sent to the terminal
  • Log lines can have timestamps prepended via `echo 1 > config/log_timestamps`
  • Net_utils gets a large number of new support functions
  • Listening on port 465 automatically enables SSL support on that socket
  • New plugins: connect.asn, access, connect.fcrdns, and relay
  • A new utility: `haraka_grep` – a grep-like tool for Haraka log lines which displays all log lines for a given connection when it finds matching lines in the logs
  • Haraka is now continuously tested by Travis-CI
  • Dependencies are now more strictly managed with “~Version” in package.json

This release also features many updates to existing plugins, and many bug fixes particularly in the Outbound sending engine.

Installation or upgrading is as simple as “npm install -g Haraka”, and restarting your server, with the caveat that we urge anyone upgrading to test upgrades thoroughly before putting them into production.

As usual the development of Haraka could not be done without the help of many in the community. Please see the git logs for a full list of all changes and contributors.