Uncategorized

Announcing Haraka 2.6.1

The Haraka development team is proud to announce the release of v2.6.1 of the popular mail server, Haraka.

This release adds some small new features, but is mostly a bug fix release for the 2.6 series. In particular a bug in mail_from.is_resolvable was fixed which can cause Haraka to crash.

Other changes in this release:

  • added sedation timers for config file re-reading
  • Add AUTH support to outbound
  • tests/spf: quiet excessive DEBUG noise
  • allow domains with underscore
  • correct name of domains config file in access
  • Fix SMTP AUTH in smtp_forward/proxy and add docs
  • Fix opts not being passed to HMailItem _bounce function
  • log.syslog will try strong-fork-syslog (for node 0.12 compat)
  • improvements to Plugin docs
  • rename net_utils.is_rfc1918 -> is_private_ip
    • IPv6 compat
    • test coverage
    • add IPv6 unique local fc00::/7
  • pre-populated config/plugins
  • added utils.extend, copies props onto objects

Upgrading for most users should be as simple as running “npm install -g Haraka”, and copying over the files “extra-tlds”, “top-level-tlds”, “three-level-tlds”, “two-level-tlds” and “public-suffix-list”.

Enjoy, and for any issues please use github issues or join us on #haraka on irc.freenode.net

Standard
Uncategorized

Announcing Haraka v2.6.0

With thanks to a ton of hard work from the Haraka development team I’m very proud to announce the release of Haraka v2.6.0.

This release is the culmination of a huge number of patches and changes, so please check your configuration carefully when upgrading.
The following is the full list of changes in this release:
* other bug fixes
* updated a few tests so test suite passes on Windows
* log.syslog: handle failure to load node-syslog
* plugin directory is $ENV definable (@martin1yness)
* logging timestamps were static, fixed by @cloudbuy
* queue/rabbitmq_amqplib, new plugin for RabbitMQ using amqplib (@esevece)
* outbound:
    * plugins can set the outbound IP (during get_mx)
    * only replace line endings if not \r\n
    * bannering fixes
    * added support for per recipient routes
* tls: don’t register hooks upless certs exist
* removed contrib/geolite-mirror-simple.pl (replaced by
  docs update pointing to maxmind-geolite-mirror)
* rcpt.routes: new plugin by @msimerson
* make haproxy IPv6 compatible
* record_envelope_addresses: new plugin by @deburau
* prevent_credential_leaks: new plugin by @smfreegard
* config:
    * configfile: added .yaml support
    * improved config file ‘watch’ logic
    * Allow hyphens in params in config files (@abhas)
    * cached requests include options in cache key name
* asn: updates for node 0.11 compat
* dnsbl: use aysync.each vs forEach (avoid race condition)
* spamassassin: improved config loading and test coverage
* geoip: deprecate geoip-lite in favor of maxmind, IPv6 compatible
* disable SSLv3 (due to POODLE)
* dkim & spf, updates for node 0.11 compatibiilty
* karma: move neighbor scoring from code to karma.ini
    * move excludes list to karma.ini
    * apply awards before adding message header & permit rejection at queue
    * karma.ini: score updates for access & uribl plugins
    * score denials issued by skipped plugins
    * add scores for specific DNSBLs
* add transaction body filters (@chazomaticus)
    * change bannering to use them
* helo.checks: fix timeout bug
    * match_re now validates and pre-compiles all REs
    * Add new proto_mismatch check
* p0f: add register(), load config once, early
* server: improved config handling
* data.headers: add Delivered-To check
* rcpt_to.ldap: new plugin by @abhas
* smtp_client: only load tls_* when cfg.enable_tls
* added plugins/host_list_base
* Platform independent temp dir (thanks @martinvd)
* move deprecated docs into docs/deprecated
* Switch to Phusion baseimage instead of stock Ubuntu (thanks @Synchro)
* dkim_verify: new plugin by @smfreegard
* many new tests
* improved URI parser (for URIBL plugin)
* Allow mixed case STARTTLS command
* Install Node via package manager (Mohd Rozi)
* Fix a couple crit errors (@Illirgway)
* Add noisy/bulk out-of-band rule support to MessaageSniffer plugin
* initial support for rabbitmq plugin (@samuelharden)
* bounce, added non_local_msgid checks and much faster lookups
* vpopmail: fail faster during a CRAM-MD5 auth attempt with an invalid user
* fcrdns: handle a null hostname
* Improve HAProxy support code and documentation
* tls: reworked for efficiency and linear style
* access: test hostname validity before PSL lookup
    * load lists into objects (vs arrays), for much faster runtime access
* host_list: huge performance increase, esp for many hosts
Documentation is up to date on http://haraka.github.com/ where you can read more about the many changes in there.
This release has been tested on Node v0.10 and v0.12 and IO.js. It should work on Node v0.8 but we do not recommend it.
Thanks for your continued support of Haraka.
Standard
Uncategorized

Solving the express ‘array of routes’ problem

Today on hacker news people have been discussing an issue Netflix had with a bug in their route reload code using Express. Now in express, routes aren’t meant to be reloaded (and there are plenty of options for restarting without stopping accepting connections – I’ve even written a simple one myself), but they hacked it in anyway, and every reload resulted in the processing of routes slowing down.

The discussion on hacker news didn’t focus on the actual problem Netflix had, more on the issue that express uses a simple (recursive) loop through an array of routes to find the appropriate route for the request, continuing on to the next matching route if next() is called.

While this isn’t a problem for our systems at work, some people in the discussion and elsewhere said there’s no way to solve this without breaking the semantics of how express currently works (routes have a defined order, and multiple routes can match a single request). Suggestions of a path-based DFA were quickly rebutted.

However that’s a naive way to solve the problem.

V8 has a very fast regular expression engine that compiles regular expressions to x86 machine code. This could be exploited to implement the current routing logic using a large single regular expression (actually N regular expressions, where N is the number of routes, but I’ll come to that).

The implementation is very simple:

  • Take each route (which can be a string, a simplified regexp, or a full regexp) and convert it to a regexp (presumably the router already does this internally because /foo/:id has to match any value for :id).
  • Combine the regexps with bracket matches and alternation: ((re1)|(re2)|(re3)|(re4)|(re5))
  • Generate N regexps, depending on number of routes: ((re1)|(re2)|(re3)|(re4)|(re5)) and ((re2)|(re3)|(re4)|(re5)) and ((re3)| (re4)|(re5)) and ((re4)|(re5)) and ((re5))
  • When a request comes in, check the first regexp, if it matches, you know which alternation matched based on the return from RegExp.prototype.exec(), from there you pick the route function based on an index in an array of route functions.
  • Alternating regexps match in order so this preserves the Express semantics.
  • If next() is called, you go to the next regexp in the list (e.g. ((re2)|(re3)|(re4)|(re5))), and run that. Use the same algorithm (with index + 1) to find the appropriate route function.

This solves the performance problem because the V8 regexp engine finds the appropriate route extremely quickly, and keeps the Express ordering and multiple routes semantics in-tact.

I haven’t proved this is any faster yet, I’m just assuming. I suspect not much for small numbers of routes, and I don’t know if the JS engine has size limits that would prevent it working on large numbers of routes.

Also, this is just an idea I had – I welcome people telling me why it couldn’t work as perhaps my understanding of the Express internals isn’t as strong as I hoped.

Standard
Uncategorized

Egnyte Support for Email It In

Email It In has just added support for the Egnyte hybrid cloud storage service. Unlike the other services Email It In supports (Google Drive, OneDrive and Dropbox), Egnyte allows their customers to maintain storage on-premises, as well as providing a cloud storage service.

To add Egnyte support we worked very closely with both Egnyte and their users to ensure we offered the best service possible. Users of Egnyte can sign up for free trials at https://emailitin.com/

Standard
Uncategorized

Haraka and shellshock

It has recently been shown that Qmail is vulnerable to shellshock if you use a pipe filter in a .qmail file (as I do on one of my own machines).

I want Haraka users to know that if you have Haraka in front of Qmail, you are NOT vulnerable to this.
The reason being that Haraka validates MAIL FROM commands according to RFC 5321 rules, whereas Qmail does not – it simply passes any string through untested into the environment. I believe the same safety to be true of Qpsmtpd, though I have not tested it there.
All that said, upgrade your servers anyway. This is a nasty bug with multiple attack vectors.
Standard
Uncategorized

Irritate your developers

Nothing motivates a developer more than being irritated by something.

It’s why we’re hackers in the first place. I remember my first real programming experience being using the “Freeze Frame” cartridge on the Commodore 64 so I could change the number of lives I had in a game. The irritation of not being able to finish the game drew me into hacking. Most programmers have similar stories – of how something irritated them enough to start hacking on it.

We can use this to our advantage in our day to day work. One excellent example I remember hearing when I was in the Perl community (I think it was from @chromatic, but I don’t recall exactly) was at the end of each day, write a test that fails based on what you’re working on. You’ll come in next morning and want/need to fix that test.

My most recent examples of irritating myself have been when trying out new external systems that implement webhooks. When I implemented Stripe payments for EmailItIn I had no idea which webhooks I might need to pay attention to. So rather than pour over the documentation for hours until I figured it out, I just set it up to irritate me – every call from Stripe webhooks emails me the entire JSON structure they send. Over time this becomes irritating fairly quickly, and so like any good hacker I start to notice patterns and things I’m interested in and not interested in, and I can setup code to filter out irrelevant webhooks, and write code that deals with things like “subscription_cancelled”.

Another way you can use this to your advantage is to email every single error (or in Node, every call to console.error()) on your site. This might scare a lot of people, but if you’re seeing errors often enough to irritate you, that is something that needs fixed, fast. And if you don’t do this those errors often get lost in your logs (you do keep logs, right?). At Ideal Candidate we email all developers when any of the following conditions occur:

  • A console.error() call – we shim this to provide a stack trace in the email
  • A server side exception which brings the server down
  • A client side exception occurs – this does a POST back to our server which triggers the email

Does this amount of emails scare you? If it does you probably have far too many errors occurring in your application. Give it a try – you might find a lot of issues that you didn’t even know you had.
Continue reading

Standard
Uncategorized

SOLVED: Drobo doesn’t remount at boot on OS X Mavericks

For a long time I loved my drobo. Then I upgraded to Mavericks and all went to shit. On reboot I would have to carefully unmount the now read only drobo, and remount, a process that took around 30 minutes. Only then would it be properly mounted.

I recently discovered HFS journaling wasn’t enabled on the drive. Turning that on in disk utility (a big button in the toolbar) fixed the reboot woes.

Putting this blog post here in the hopes I will help others with the same problem.

Standard