Uncategorized

Setting Up Haraka to send to mailchimp/sendgrid/SES etc

One really useful way to use Haraka at a business is to be a local fast cache to forward on to mailchimp/sendgrid/SES or one of the many available transactional mail senders out there. These services offer bounce management, tracking, and also manage deliverability for you, and yet often their sending systems are slow, which can affect your app. By installing Haraka locally you can keep all those benefits, but have your system return from sending mail much faster.

First, install haraka and create an instance of it:

$ sudo npm install -g Haraka
... # lots of output
$ haraka -i /var/apps/haraka

Now edit `/var/apps/haraka/config/plugins` to define what Haraka plugins will get loaded. Here’s my simple config for this setup:

# this lets you view how much stuff your haraka instance is doing
process_title
# this lets you receive inbound mail, but is mostly not required
rcpt_to.in_host_list
# this is a custom plugin I'll detail below
relay_via_external
# this lets you configure who can relay (i.e. your app)
relay
# Check mail headers are valid
data.headers
# this lets you configure where inbound mail goes (also mostly not required)
queue/smtp_forward
# this is just a safety net
max_unrecognized_commands

You’ll need an entry in the `config/host_list` file:

$ hostname > /var/apps/haraka/config/host_list

You’ll want to allow localhost to relay:

$ echo 127.0.0.1 > /var/apps/haraka/config/relay_acl_allow
$ echo ::1 >> /var/apps/haraka/config/relay_acl_allow

Copy this into your `/var/apps/haraka/config/relay.ini` file:

[relay]
acl=true
relay=all

Prevent `smtp_forward` from processing outbound mail:

$ echo 'enable_outbound=false' > /var/apps/haraka/config/smtp_forward.ini

Edit your `smtp.ini` file to make it listen on a higher port so you don’t have to listen on port 25 by setting the `listen=` line to: listen=[::0]:2525, and set the `nodes=` line to use as many CPUs as you think is reasonable for your setup.

Finally, copy this plugin into `/var/apps/haraka/plugins/relay_via_external`:

var external_smtp = {
    auth_user: '<your username here>',
    auth_pass: '<enter password here>',
    priority: 0,
    exchange: 'smtp.mailgun.org', # or whoever else
};

exports.hook_get_mx = function (next, hmail, domain) {
    // All relaying goes via external service
    return next(OK, external_smtp);
}

Remember to edit the username and password – get them from your mail service provider.

And that’s it. There are various ways of launching Haraka (see the contrib directory on github) which keep it starting at startup. But to test it you can simply launch: `haraka -c /var/apps/haraka`.

Now just make your app use localhost on port 2525 as your mail server, with no authentication necessary, and your mail will go out MUCH faster.

Standard
Uncategorized

ANNOUNCE: Haraka v2.8.8

  • Changes
    • removed UPGRADE.doc to wiki
  • Improvements
    • support + wildcard in aliases plugin #1531
    • Support dkim_sign with outbound.send_email() #1512
    • spf: always check remote IP, then public IP if != pass #1528
    • spf: diplay IP used for SPF eval #1528
  • Bug Fixes
    • handle missing wss section in http.ini #1542
    • fix leak on socket write error #1541
    • add results property to outbound transaction #1535
    • don’t unref unref’d wss server #1521
Standard
Uncategorized

Announce: Haraka v2.8.5

I haven’t bothered posting about the earlier 2.8.x releases (beyond 2.8.0) because they have mostly fixed small bugs that we introduced in the 2.8.0 release. This is the first release to add significantly new features.

  • Changes
    • The connection object is now passed to get_plain_passwd. Older modules should continue to work as-is.
    • The reseed_rng plugin now just uses the Crypto module from core. Though it seems this plugin should be irrelevant with newer versions of node.js
  • New Features
    • Outbound mail now uses pooled connections, only sending a QUIT message if the connection has been idle for a while.
  • Improvements
    • Shut down and reload (via haraka -c <path> --graceful) is now graceful – allowing current connections to finish and plugins to clean up before ending.
  • Bug Fixes
    • Bind maxmind version to ignore API change (#1492)
    • Fix encodings when banners are used (#1477)
    • Various DKIM fixes (#1495)

Upgrading should be fairly transparent, especially for current 2.8.x users.

Happy Haraking!

Standard
Uncategorized

ANNOUNCE: Haraka 2.8.2

Note: We shortly after this released 2.8.3 which fixes config merging, a feature we added in 2.8.0. It didn’t seem to really deserve a blog post.

This is a minor bugfix release, mostly to fix the broken config/plugins file we accidentally shipped with v2.8.0.

Changes

  • Added Node v6 to travis tests

New Features

  • Added bin/haraka –qunstick <domain> to flush all mails for that domain (#1460)

Improvements

  • Make bin/haraka –qlist show much more information (#1452)
  • Allow CIDR ranges in no_tls_hosts (#1450)

Bug Fixes

  • 2.8.0 was shipped with a broken config/plugins. (#1453)
  • Stop haraka dying when ldap connections fail (#1456)
  • Pick up domain specific config correctly in ldap (#1456)
Standard
Uncategorized

Announce: Haraka v2.8.0

This release represents a huge leap forwards for Haraka users everywhere. We came very close to calling this v3, but since we didn’t break any APIs we stuck with v2.8.

This release contains work from 15 contributors around the world, and we thank them all for their time and effort.

Upgrading is simply a matter of running: npm install -g Haraka. Following an upgrade to 2.8 we recommend deleting any config files in your personal Haraka config folder that you have not modified. The reason for this is that Haraka will now load non-modified config data from the core Haraka folder. This change makes future upgrades much easier.

The major new features in this release are:

  • The ability to write plugins as npm packages
  • The merging of config data to allow minimal config in your local config folder
  • Many TLS fixes including the ability to use outbound TLS without a certificate, meaning we now enable outbound TLS by default

The full list of changes are:

  • Changes
    • updated dependency versions (#1426, #1425)
    • use utf8 encoding for body filters (#1429)
    • remove spameatingmonkey from tests (#1421)
    • replace ./constants.js with haraka-constants (#1353)
    • Document HMail and TODO items (#1343)
    • Copy only a minimal config/* by default (#1341).
    • cfreader/* removed to haraka/haraka-config (#1350)
    • outbound and smtp_client honor tls.ini settings (#1350)
    • outbound TLS defaults to enabled
    • lint: remove all unused variables (#1358)
    • replace ./address.js with address-rfc2181 (#1359)
  • New Features
    • smtp_forward: accepts a list of backend hosts, thanks @kgeoss (#1333)
    • config: add array[] syntax to INI files (#1345)
    • plugins.js: support require(‘./config’) in plugins
    • Load plugin config from own folder and merge (#1335)
    • Allow original email’s Subject to be included in bounce message (#1337)
    • new queue/smtp_bridge plugin, thanks @jesucarr (#1351)
  • Improvements
    • early_talker: supports IP whitelisting (#1423)
    • loading plugins as packages (#1278)
    • removed TLD stuff to haraka/haraka-tld (#1301)
    • removed unused ‘require(‘redis’) in plugins/karma (#1348)
    • improved MIME header support per rfc2231 (#1344)
    • tls options can be defined for outbound and smtp_* (#1357)
    • explicitly disable SSLv2 (#1395)
    • cache STUN results
    • xclient plugin improvements (#1405)
    • tls: Set verify=NO correctly when no certificate presented (#1400)
    • improved message header decoding (#1403, #1406)
    • bounce: skip single_recipient check for relays/private_ips (#1385)
    • rspamd docs: Clarify usage of check.private_ip (#1383)
    • if rcpt_to returns DSN in msg, log it properly (#1375)
  • Bug Fixes
    • fix out-of-range errors from banner insertion (#1334)
    • dkim_verify: Call next only after message_stream ended (#1330)
    • outbound: remove type check from pid match (#1322)
    • lint: enable no-shadown and remove all shadow variables (#1349)
    • spf: fix log_debug syntax (#1416)
    • auto_proxy: fix a starttls loop (#1392)
    • fcrdns: corrected err variable name (#1391)
    • rspamd: Fix undefined variable (#1396)
    • dkim_verify: Fix header handling (#1371)
    • smtp_client: fix remote_ip (#1362)

Happy Haraking!

Standard
Uncategorized

Express, Lost Sessions, and Windows

One of the harder bugs I’ve ever had to track down.

Most of our office uses Macs. One developer alone uses Windows. He would regularly see the weirdest issue where he couldn’t log into our app – he’d try time and time again, and maybe on the 4th or 5th try it would work.

He even hacked in some delays via Javascript’s setTimeout() function to try and fix it, knowing it was a weird bug with lost sessions.

After helping him debug a problem on his machine, and seeing him go through this over and over, it was finally time to get down to the meat of this issue. Why were sessions not getting saved for Windows users, but were just fine for Mac users?

Every problem related to a call to res.redirect(). First problem located.

So why would this work fine for Mac and not Windows? It wasn’t even browser specific.

Thankfully I have a large back-story in various networking related activities. Windows and Macs have different networking stacks, and different ways they cut apart packets and so on.

I also knew quite well how Express (the Node.js HTTP server stack we use) saved the session, due to various unrelated deep dives into the code.

Express saves the session by hijacking res.end(). It turns out that when you do a res.redirect(), on Windows it will likely get the headers in a single packet, and the body in another, but perform the redirect before even seeing the body (because it’s empty and irrelevant). But res.end() isn’t called until the HTTP request is completed. This means the Windows boxes can get the redirect, request the redirected URL, and get access to an unsaved session before res.end() has time to completely save the session.

The fix? Now in our code we hijack res.redirect() to perform req.session.save() before performing the actual redirect.

This fix should probably go into express-session. I will work on a pull request.

Code for the curious (put this in some middleware):

 var redirect = res.redirect;
 res.redirect = function (path) {
 res.redirect = redirect;

 if (req.session) {
 req.session.save(function (err) {
 if (err) console.error(err);
 _finish();
 });
 }
 else {
 _finish();
 }

 function _finish () {
 if (/\&utm_/.test(path)) {
 return res.redirect(path);
 }
 if (req.query.utm_campaign && req.query.utm_medium && req.query.utm_source) {
 var extras = qs.stringify({
 utm_campaign: req.query.utm_campaign,
 utm_medium: req.query.utm_medium,
 utm_source: req.query.utm_source,
 });
 if (/\?./.test(path)) {
 path = path + '&' + extras;
 }
 else {
 path = path + '?' + extras;
 }
 }
 return res.redirect(path); 
 }
 }
Standard
Uncategorized

Announcing Haraka v2.7.3

The Haraka core team are happy to announce Haraka v2.7.3.

  • Changes
    • smtp_proxy & qmail-queue: default to enabled for outbound deliveries (previously used Outbound), to better matches user expectations.
  • New Features
    • outbound: allow passing notes to send_email (#1295)
  • Improvements
    • logging: emit log message queue before shutting down (#1296)
    • result_store: permit redis pub/sub to work when host != localhost (#1277)
    • tests: quiet the extremely verbose messages (#1282)
    • rspamd: add timeout error handling (#1276)
    • watch: fix display of early_talker results (#1281)
    • spamassassin: publish results to result_store (#1280)
    • karma: can now connect to redis on hosts other than localhost (#1275)
    • geoip & p0f: don’t log empty/null values from RFC 1918 connects (#1267)
    • redis: make plugin params match docs (#1273)
    • mailbody: small refactoring (#1315)
    • smtp_proxy & qmail-queue: default to enabled for outbound (#1308)
  • Bug Fixes
    • redis: use correct path for db.select (#1273)
    • count errors correctly (#1274)
    • logger: ignore null arguments (#1299)
    • connection: pause for hook_reset_transaction (#1303)
    • rcpt_to.routes: update redis usage for compat with redis plugin (#1302)
    • smtp_forward: use correct config path to auth settings (#1327)
    • messagestream: correctly pass options parameter to get_data (#1316)
    • spf: honour configuration for mfrom scope (#1322)
    • outbound: Add missing dash to ‘Final-Recipient’ header name (#1320)
Standard