With thanks to a ton of hard work from the Haraka development team I’m very proud to announce the release of Haraka v2.6.0.
Today on hacker news people have been discussing an issue Netflix had with a bug in their route reload code using Express. Now in express, routes aren’t meant to be reloaded (and there are plenty of options for restarting without stopping accepting connections – I’ve even written a simple one myself), but they hacked it in anyway, and every reload resulted in the processing of routes slowing down.
The discussion on hacker news didn’t focus on the actual problem Netflix had, more on the issue that express uses a simple (recursive) loop through an array of routes to find the appropriate route for the request, continuing on to the next matching route if
next() is called.
While this isn’t a problem for our systems at work, some people in the discussion and elsewhere said there’s no way to solve this without breaking the semantics of how express currently works (routes have a defined order, and multiple routes can match a single request). Suggestions of a path-based DFA were quickly rebutted.
However that’s a naive way to solve the problem.
V8 has a very fast regular expression engine that compiles regular expressions to x86 machine code. This could be exploited to implement the current routing logic using a large single regular expression (actually N regular expressions, where N is the number of routes, but I’ll come to that).
The implementation is very simple:
- Take each route (which can be a string, a simplified regexp, or a full regexp) and convert it to a regexp (presumably the router already does this internally because
/foo/:idhas to match any value for
- Combine the regexps with bracket matches and alternation:
- Generate N regexps, depending on number of routes:
- When a request comes in, check the first regexp, if it matches, you know which alternation matched based on the return from
RegExp.prototype.exec(), from there you pick the route function based on an index in an array of route functions.
- Alternating regexps match in order so this preserves the Express semantics.
next()is called, you go to the next regexp in the list (e.g.
((re2)|(re3)|(re4)|(re5))), and run that. Use the same algorithm (with index + 1) to find the appropriate route function.
This solves the performance problem because the V8 regexp engine finds the appropriate route extremely quickly, and keeps the Express ordering and multiple routes semantics in-tact.
I haven’t proved this is any faster yet, I’m just assuming. I suspect not much for small numbers of routes, and I don’t know if the JS engine has size limits that would prevent it working on large numbers of routes.
Also, this is just an idea I had – I welcome people telling me why it couldn’t work as perhaps my understanding of the Express internals isn’t as strong as I hoped.
Email It In has just added support for the Egnyte hybrid cloud storage service. Unlike the other services Email It In supports (Google Drive, OneDrive and Dropbox), Egnyte allows their customers to maintain storage on-premises, as well as providing a cloud storage service.
To add Egnyte support we worked very closely with both Egnyte and their users to ensure we offered the best service possible. Users of Egnyte can sign up for free trials at https://emailitin.com/
It has recently been shown that Qmail is vulnerable to shellshock if you use a pipe filter in a .qmail file (as I do on one of my own machines).
Nothing motivates a developer more than being irritated by something.
It’s why we’re hackers in the first place. I remember my first real programming experience being using the “Freeze Frame” cartridge on the Commodore 64 so I could change the number of lives I had in a game. The irritation of not being able to finish the game drew me into hacking. Most programmers have similar stories – of how something irritated them enough to start hacking on it.
We can use this to our advantage in our day to day work. One excellent example I remember hearing when I was in the Perl community (I think it was from @chromatic, but I don’t recall exactly) was at the end of each day, write a test that fails based on what you’re working on. You’ll come in next morning and want/need to fix that test.
My most recent examples of irritating myself have been when trying out new external systems that implement webhooks. When I implemented Stripe payments for EmailItIn I had no idea which webhooks I might need to pay attention to. So rather than pour over the documentation for hours until I figured it out, I just set it up to irritate me – every call from Stripe webhooks emails me the entire JSON structure they send. Over time this becomes irritating fairly quickly, and so like any good hacker I start to notice patterns and things I’m interested in and not interested in, and I can setup code to filter out irrelevant webhooks, and write code that deals with things like “subscription_cancelled”.
Another way you can use this to your advantage is to email every single error (or in Node, every call to console.error()) on your site. This might scare a lot of people, but if you’re seeing errors often enough to irritate you, that is something that needs fixed, fast. And if you don’t do this those errors often get lost in your logs (you do keep logs, right?). At Ideal Candidate we email all developers when any of the following conditions occur:
- A console.error() call – we shim this to provide a stack trace in the email
- A server side exception which brings the server down
- A client side exception occurs – this does a POST back to our server which triggers the email
Does this amount of emails scare you? If it does you probably have far too many errors occurring in your application. Give it a try – you might find a lot of issues that you didn’t even know you had.
For a long time I loved my drobo. Then I upgraded to Mavericks and all went to shit. On reboot I would have to carefully unmount the now read only drobo, and remount, a process that took around 30 minutes. Only then would it be properly mounted.
I recently discovered HFS journaling wasn’t enabled on the drive. Turning that on in disk utility (a big button in the toolbar) fixed the reboot woes.
Putting this blog post here in the hopes I will help others with the same problem.
After a massive development effort a new stable release of Haraka is out.
v2.5.0 contains a huge number of changes, of which these are the highlights:
- A new feature called ResultStore (or Results) which allows plugins to communicate information they found to each other. This is primarily used by the karma plugin to use the results of other plugins to penalise senders
- A -o/–order option to
bin/harakato show the order all currently configured plugins will run.
- IPv6 support on outbound via `echo 1 > config/outbound.ipv6_enabled`
- Attachment streams now have access to the MIME header for that attachment via stream.header
- A new “deferred” hook for outbound mail, called when mail is temporarily denied
- Outbound “bounce” hooks now receive an Error object, containing the MX and recipient information. Existing uses of the error being treated as a string should still function as normal
- Outbound gets more parameters on the “delivered” hook to allow detailed analysis of sent mail
- Outbound UUID now distinguishes between different domains using .1, .2 etc just as inbound gives a new index for each transaction
- Outbound won’t try and send to domains publishing NULL MX (draft-delany-nullmx-02)
- Outbound template can use `extended_reason` to show a more detailed error
- Log lines are coloured if sent to the terminal
- Log lines can have timestamps prepended via `echo 1 > config/log_timestamps`
- Net_utils gets a large number of new support functions
- Listening on port 465 automatically enables SSL support on that socket
- New plugins: connect.asn, access, connect.fcrdns, and relay
- A new utility: `haraka_grep` – a grep-like tool for Haraka log lines which displays all log lines for a given connection when it finds matching lines in the logs
- Haraka is now continuously tested by Travis-CI
- Dependencies are now more strictly managed with “~Version” in package.json
This release also features many updates to existing plugins, and many bug fixes particularly in the Outbound sending engine.
Installation or upgrading is as simple as “npm install -g Haraka”, and restarting your server, with the caveat that we urge anyone upgrading to test upgrades thoroughly before putting them into production.
As usual the development of Haraka could not be done without the help of many in the community. Please see the git logs for a full list of all changes and contributors.