Uncategorized

Angular.js: ng-click still fires when div is (ng)disabled

I’m not sure if this is considered a bug in Angular.js or not, but after much hunting I found a work-around.

The idea is to disable a button after it has been clicked to prevent button mashing. But a lot of the buttons in Ideal Candidate have been designed using plain <div> tags. Not sure if that’s a good or bad idea – I’m not a UI guy – it is what it is.

The basic problem is ng-disabled works fine for buttons:

<button ng-click="do_something()" ng-disabled="button_clicked">Click Me</button>
angular.module('ngToggle', [])
    .controller('AppCtrl',['$scope', function($scope){
    $scope.button_clicked = false;
    $scope.do_something = function() {
        alert("Clicked!");
        $scope.button_clicked = true;
        return false;
    }
}]);

But if you change your button to a <div> the ng-disabled flag no longer works. It will set the element to disabled, but it won’t prevent the ng-click from firing if you mash the button.

The simple solution is to change your ng-click attribute to check the parameter:

<div ng-click="button_clicked || do_something()" ng-disabled="button_clicked">Click Me</div>

This works perfectly.

Standard
Uncategorized

I ain’t afraid of no Ghost

A while ago I bought the domain name baudehlo.com for this blog, and wanted to try out some blog software.

So I had a brief play with Ghost today. It was fairly straightforward for me to install (being someone who knows Node.js and understands the ecosystem), but in the end I decided it just wasn’t ready for prime time. WordPress.com offers me so much beyond just hosting – they provide comments, stats, subscriptions, etc. So I decided to pay the $99 a year to get my domain name on here, which also allowed me to get rid of the ugly font used in the headlines. So I’m not moving my blog – it’s staying right here, but has redirected now to baudehlo.com. Enjoy!

Standard
Uncategorized

Announcing Haraka v2.4.0 – LMTP support

We decided to quickly push out 2.4.0 due to finding a bug in outbound which was discovered while testing the new LMTP support for inbound mail.

LMTP is the last piece in the puzzle of totally replacing your inbound SMTP server with Haraka. Now Postfix/Exim/Qmail are not required for a Haraka installation – just use LMTP to deliver directly to your IMAP server or to `procmail -z`.

Upgrading is as simple as “sudo npm install -g Haraka” and restarting your Haraka server.

Full changes:

  • Trim whitespace when reading “list” type config files (such as config/plugins)
  • Added LMTP via queue/lmtp plugin
  • Fixed bug in outbound when temp failing some of the recipients that would prevent delivery working to those recipients for future delivery attempts
  • Add additional details/parameters to delivered hook for outbound mail
  • Removed the hmail.bounce_extra object as that information now stored with the rcpt_to list
  • Store the RCPT TO rejection reason on the address object

Thanks to Peter Janotta for starting the work on LMTP support. See http://haraka.github.io/manual/plugins/queue/lmtp.html for documentation.

Standard
Uncategorized

Fixing async.waterfall parameter handling

Users of node.js should be familiar with the excellent async library for wrangling some of Node’s async craziness. One thing that frustrates me is when using async.waterfall you can’t easily vary the number of parameters passed. Two uses cases occur for this: one: skipping to the end, and two: just straight up ignoring results.

A typical example looks like this:

    async.waterfall([
        ... // some functions skipped that may or may not set "skip" flag
        function (cb) {
            if (skip) return cb();
            model.companies.get(user.company.id, cb);
        },
        function (c, cb) {
            if (skip) return cb();
            // do something with c...

However that fails under normal async because it cares about the number of parameters passed to cb() matching those of the next function.

I managed to fix that by monkeypatching async.iterator as follows:

// Async monkeypatch
async.iterator = function (tasks) {
    var makeCallback = function (index) {
        var fn = function () {
            if (tasks.length) {
                var args = Array.prototype.slice.call(arguments);
                while (args.length < tasks[index].length) {
                    args.unshift(null);
                }
                tasks[index].apply(null, args.slice(Math.max(0, arguments.length - tasks[index].length)));
            }
            return fn.next();
        };
        fn.next = function () {
            return (index < tasks.length - 1) ? makeCallback(index + 1): null;
        };
        return fn;
    };
    return makeCallback(0);
};

All it does is make sure the number of parameters matches, filling in nulls when required, or truncating as required.

Hope you find this useful.

Standard
Uncategorized

ANNOUNCE: Haraka v2.3.0

A new year and a new Haraka. This series of changes is rather large, and so anyone deploying over a current installation is urged to test carefully before deploying to production. Having said that we have performed extensive testing on production systems, including a bedding-in session at Craigslist.

The big changes in this release include some new plugins including vpopmaild auth, Qmail::Deliverable recipient checking, DKIM signing support, a bounce plugin for dealing with unwanted bounces, a geoip plugin, a karma plugin like the Qpsmtpd one, and a delay_deny plugin to control when rejections occur. Updated documentation for all these can be found on our web site.

Full list of changes below in order they appeared:

  • Fixes to memory leak when watching config files for changes
  • Support for badly formatted MAIL FROM/RCPT TO lines
  • Fix a memory corruption when fixing line endings
  • Fix breakpoints in plugins when using node inspector
  • Reload config in relay_force_routing without restart
  • Don’t re-attempt TLS upgrade if upgraded already and STARTTLS is re-advertised
  • Improved outbound logging
  • Pass failed recipients to bounce hook in outbound processing
  • Added startup checks to ensure Haraka has been installed correctly
  • Handle case of Haraka server running out of disk space better
  • In mail_from.is_resolvable: move re_bogus_ip into config
  • Added auth/auth_vpopmaild plugin – SMTP AUTH against a vpopmaild server
  • Fixed graph plugin to work with sqlite3
  • Added rcpt_to.qmail_deliverable plugin – Authenticate inbound RCPT TOs against Qmail::Deliverable daemon
  • Added data.headers plugin which merges all the functionality of other default header checks into one manageable place. This deprecates data.noreceived, data.rfc5322_header_checks, and data.nomsgid
  • Added documentation for logging system
  • Added DKIM per-domain signing support
  • Added p0f plugin
  • In relay_acl, if host is allowed by acl, don’t deny the recipient because the domain isn’t in the allow list
  • Add Authentication-Results header (RFC 5451) to all emails
  • Fixed writing the todo file in outbound for newer Node versions
  • Added Karma plugin to support penalizing consistently evil senders
  • Added GeoIP plugin including distance calculation from your mail server
  • Added bounce plugin for handling incoming bounce messages in various ways
  • Fix underscores in documentation so web version doesn’t look so weird
  • By default prevent SMTP AUTH unless on a private IP or using TLS WARNING: May break some uses of Haraka, but is worth it for security
  • In lookup_rdns.strict, check whitelist before looking up IP
  • Big rewrite of the SpamAssassin plugin for simplicity and mainly to pass through X-Spam-* headers provided
  • Added delay_deny plugin allowing more flexibility on when to reject mail
  • Improvements to ini file parsing allowing floats and negative integers, and specifying boolean keys
  • Fix issue causing a CRIT/crash with lost transaction/connection while sending inbound to ongoing SMTP server
  • Allow setting of spamd_user for spamassassin plugin

Big thanks to our contributors for all these changes, especially Matt Simerson who we have stolen from the Qpsmtpd project and Steve Freegard who continues his tireless work on the project.

Standard
Uncategorized

Converting a Rails app to Node.js

This is going to be a fairly long post detailing the process I recently went through converting a Rails app to Node.js.

I recently joined a new startup as CTO/Lead. They had paid for an MVP of their product which was written using Rails.

The rails app used many pieces of well known Rails and Ruby infrastructure, including:

  • Capistrano for deployment
  • Sidekiq for background tasks
  • Thin + Rails + Unicorn for runtime
  • CoffeeScript with Angular.JS for client side
  • SASS for stylesheets
  • Compass for SASS/CSS add-ons
  • HAML for server side templates
  • MySQL and MongoDB for storage
  • Many other Gems, many of which I didn’t (and some I still don’t) know their purpose

Why?

Now many people are going to ask “why convert it?” at this stage. This was a fairly complete app, and it mostly worked, but I ripped it apart and replaced it almost wholesale. I thought long and hard about whether I should do this or not, so here were my reasons:

  • I don’t know Ruby well, and know the Rails ecosystem even less well (whereas I know Node.js well).
  • Hiring – I’ve had good experiences in previous roles finding frontend JavaScript programmers who can be tasked with backend work too. It’s getting harder to find good Rails people these days, and they have to skip over the language “barrier” to do UI work (and often choose to do UI work in CoffeeScript for this reason).
  • Magic – Rails (and partly Ruby) is full of “magic” – lots of things just magically happen, and if you don’t understand the Rails system really well (such as the “asset pipeline”) then you are quickly lost. Node.js is explicit in everything it does – you may have to write a bit more code – but you know where and why something happened.
  • Performance – just starting the Rails app on a fast machine took a minute (which can make development painful), and the Ruby runtime is orders of magnitude slower than Node’s V8 runtime, so much so that the app now feels noticeably faster.
  • Ability to add and change things – if I kept the app in Rails it would hinder my ability to add and modify code.
  • Queryability – the use of ActiveRecord for everything meant that they only way to ask the system questions was via ActiveRecord. Using the DB directly (especially with data split across MySQL and MongoDB) was impossible due to much functionality implemented in the Ruby side of the model.
  • PostgreSQL – I’m a postgres guy – I don’t trust MySQL.
  • Referential integrity – Rails punts (by default) on referential integrity in the database. This was a serious WTF for me.
  • Porting time – we were pre-alpha still, with no customers, so porting now rather than later was a smart choice allowing us to move forwards faster.

That’s most of the reasons. I’m sure there were more, but you get the idea.

The rails app was ~6000 lines of Ruby, 4000 lines of SASS, 1500 lines of HAML, and 2500 lines of coffeescript (which remained unchanged).

I estimated 6 weeks for figuring everything out and porting it over. In the end I finished in around 3 to 4 weeks. Most of that time was figuring out how Rails did things. I learned a lot about Rails in the process, and like some of what I see – it definitely does a lot for you. But doing the same level of work in Node didn’t take much, and I’m glad I went through the process.

Here’s the setup I ended up with today:

  • Express server mostly stock aside from connect-memcached for sessions
  • PostgreSQL as the only database – database schema constructed by hand written SQL
  • Jade for server side templates
  • SCSS via node-sass for stylesheets with Compass working
  • Front end code mostly untouched (CoffeeScript and Angular.js) – will probably convert soon to pure JS
  • Deploy_to_runit for deployment: https://github.com/baudehlo/deploy_to_runit

Here’s the directory structure I use for Node+Express projects:

lib       - general libraries
  model   - database accessors
public    - files that get served to the front end
  assets  - files that are considered part of the app
routes    - files that provide express routes
  api/v1  - files providing REST API endpoints
views     - Jade templates

Delivering Stylesheets and JavaScript

Both SASS/SCSS and CoffeeScript need to be converted server side to be interpreted correctly by browsers (OK not strictly true, but it’s most compatible that way). Unfortunately the modules for Node.js don’t support the original “SASS” syntax, which is indent based, but they do support SCSS which is a one-step conversion.

The original SASS files were converted to SCSS using Ruby’s SASS module which comes with sass-convert to do the conversion for you – a simple matter of a shell script to convert every file to SCSS format. Despite the name node-sass cannot process SASS files. Thankfully I prefer the look of SCSS (the syntax is closer to CSS).

In order to emulate the Rails asset pipeline I convert the SCSS and take the SHA256 sum, and provide a link that is unique as until the CSS changes, this allows you to provide cache headers and yet always deliver up to date CSS.

Here’s most of the code for compiling the SCSS and serving it:

var css = sass.renderSync({
    file: __dirname + '/public/assets/stylesheets/application.scss',
    includePaths: [__dirname + '/public/assets/stylesheets'],
});

var shasum = crypto.createHash('sha256');
shasum.update(css);
var css_sha = shasum.digest('hex');

app.get('*', function (req, res, next) {
    res.locals.stylesheet_sha = css_sha;
});
app.get('/assets/stylesheets/application-' + css_sha + '.css', function (req, res) {
    res.type('css');
    res.setHeader('Cache-Control', 'public, max-age=31557600');
    res.end(css);
});

Then in your template you can use the stylesheet_sha variable to include you stylesheet. The final version of this code actually uses file watching in dev mode to recompile the SCSS when the stylesheets change, allowing me to dynamically develop with the server running. I use the one-file CSS delivery even in development mode because unlike JavaScript you rarely need to see the source of the CSS with modern debug tools.

Adding in Compass was a rather more complicated affair. If you google for “compass + node.js” you come across things like “node-compass” which uses Ruby on each request to serve up the compass files. Not what I wanted – I wanted to remove the dependency on Ruby altogether. After much hunting I discovered that Compass was mostly a bunch of SCSS files. Some places said that there was some Ruby code that ran too, but I wanted to see how far I could get just using them as straight SCSS files. So I imported the files (from gems/compass-0.12.2/frameworks/compass/stylesheets/compass) into my project, and magically it all seemed to Just Work ™. There are probably edge cases of Compass I’m not using which are broken, but for now this is good enough.

Next coffeescript:

app.use(require('coffee-middleware')({
    src: __dirname + '/public',
    compress: false,
}));

That was easy. In production I do something similar to the CSS code – compiling it statically all into one file, minimizing it using uglify-js, and provide it using a SHA sum link.

Routes

Now I had the basics served up I needed to work on routes (and soon, views).

I have a standard way of serving up all routes. In my main app.js file I have:

module.exports = app;
find.fileSync(/\.js$/, __dirname + '/routes').forEach(function (route_file) {
    require(route_file);
});

That allows nested folders under the “routes” folder, and loads every file in there. Then each file can get at the app via:

var app = module.parent.exports;
app.get(...);

My routes are logically separated into separate files relevant to each part of the application.

Model

I don’t use ORMs. I think they shield you too much from the guts of what is happening at the SQL level, and they tend to provide a layer of abstraction too deep and hard to debug when they go wrong.

But writing model APIs is simple enough, and much can still be abstracted away into a separate library, such that a typical model file looks like this:

"use strict";

var db = require('./db');

exports.get = function get (id, cb) {
    db.get_one_row("SELECT * FROM Answers WHERE id=$1", [id], cb);
}

exports.get_by_submission_id = function get_by_submission_id (submission_id, cb) {
    db.query("SELECT * FROM Answers WHERE submission_id=$1 ORDER BY question_id", [submission_id], cb);
}

exports.count_by_submission_id = function count_by_submission_id (submission_id, cb) {
    db.get_one_row("SELECT count(*) as num_answers FROM Answers WHERE submission_id=$1", [submission_id], cb);
}

exports.upsert = function upsert (submission_id, question_id, answer, cb) {
    db.query("INSERT INTO Answers (submission_id, question_id, value)\
        SELECT $1,$2,$3 WHERE NOT EXISTS \
        (SELECT 1 FROM Answers WHERE submission_id=$1 AND question_id=$2)\
        RETURNING id", [submission_id, question_id, answer],
    function (err, rows) {
        if (err) return cb(err);
        if (rows.length && rows[0].id) {
            return cb();
        }
        db.query("UPDATE Answers SET value=$1 WHERE submission_id=$2 AND question_id=$3", [answer, submission_id, question_id], cb);
    })
}

Now there’s a lot I’m not saying here – creating all the routes and models appropriately was a large part of the work in converting the app to Node. Mostly I relied on Chrome’s web inspector to see what the Ruby version sent over the wire for each endpoint (JSON in most cases), and creating the same data feed in Node. This was a time consuming process, but valuable because it taught me about the entire structure of the application, and also helped me fix some of the weaknesses in the data model.

Converting HAML to Jade

This was the next biggest part of this project. Jade is very similar to HAML in style, but slightly cleaner in syntax. It is slightly less flexible in some things (e.g. no dynamic includes), but overall I came out of it knowing a lot about Jade and even created the #jadejs IRC channel to help other people.

After a number of hand conversions I came up with a set of regexp substitutions I applied to each file (I did this in SublimeText, but perl or sed or just about anything would work):

Fix tags:
Search: ^(\s*)\%
Replace: $1

Fix attribute wrappers: tag{foo: bar} becomes tag(foo: bar)
Search: (\w)\{(.*)\}
Replace: $1($2)

Fix attribute “key: ‘value’” to “key=’value’”
Search: (\w):\s*(["'])
Replace: $1=$2

The remainder of the contents I converted carefully by hand.

One thing which was a constant irritation was the use of nested values in attributes due to using AngularJS, I’d see a lot of: {ng:{show: 'some_condition'}} which translates to: (ng-show='some_condition'). I much prefer the Jade representation of this because it looks like HTML attributes, rather than JS or Ruby code.

The next thing I discovered which caused problems was helpers. Rails provides a bunch of default helpers for forms, input fields, etc. I decided to use Jade’s powerful mixin feature for this. So Rails’ = f.text_field :email became +form_text_field('email', data.user.email) – slightly more complex, but clearer where the default value comes from. The code for this mixin itself (which I keep in a separate file) is:

mixin form_text_field(name, default)
    input(id=name, name=name, size=30, type="text", value=default)

A more complex mixin can be created for select fields:

mixin form_select_field(name, object, include_blank)
    select(id=name, name=name)
        if include_blank
            option(value="")
        if (Array.isArray(object))
            each val in object
                option(value=val)= val
        else
            each val, key in object
                option(value=key)= val

Mixins can also have sub-blocks so the following works:

mixin form(action)
    form(accept-charset="UTF-8", action=action, method="POST")
        input(name="utf8" type="hidden" value="✓")
        block

+form('/post')
    form_text_field('name', data.name)

One complexity I found was that HAML allows for dynamic includes, which Jade doesn’t. This allows HAML to implement something like:

= popup 'roles/new'

Which behind the scenes will load the template in views/roles/new and include it in the block of the popup.

Whereas in Jade you have to use two lines as follows:

+popup('roles/new')
    include ../roles/new

Not too much of a hassle, and once again makes it explicit about what is happening.

Next Steps

The main thing I left as-is was the front end code, written in CoffeeScript. At some point I may convert this to plain Javascript, which I just find easier to understand as it’s now the same language as the backend, though CS obviously brings with it some nice shortened versions of the verbose JavaScript way of doing things.

This blog was also turned into a presentation for the Toronto Node.js user group. You can view the slides below:

Standard
Uncategorized

ANNOUNCE: Haraka v2.2.6

This is a minor bug fix release of Haraka fixing the following bugs:

  • A large base64 javascript field could cause infinite backtracking in the regular expressions in URL matching in the uribl plugin
  • Allow for multiple EHLO/EHLO commands as per the RFC
  • Fix STARTTLS capabilities if multiple EHLOs are sent

Thanks to Steve Freegard (Fort Anti Spam) for these fixes.

Upgrading is as simple as “sudo npm -g install Haraka” and restarting your Haraka servers.

Standard