Laravel and NodeJS messaging using Redis Pub/Sub

I was recently working on this project that was composed of two different parts: a web application built in PHP with Laravel, and an AWS Lambda function written in NodeJS. In the past, both applications exchanged data using a common MySQL database. With time, this setup showed up very inefficient. As the number of “messages” sent and received increased, the database started to not handling well the volume of reads and writes required to support both “applications” — the Lambda function is not an application per se but you know what I mean, right?

The first thing we tried was changing the database schema to focus on performance, rather than on data integrity. We dropped some constraints and changed how the data was stored to achieve that. The updates soon showed themselves not enough.

In a second iteration, we started playing around with Redis. Due to its nature, a key/value store and not a relational database, it’s a lot faster than MySQL. The first attempt using Redis involved simply moving the data we’re storing into the database to a set. It seemed to work well but just after a few tests on a staging server we realized that approach wouldn’t work for the system needs. When retrieving the data using the SCAN command, the order of returned elements is not guaranteed. And that was an important downside for us, the business logic required us to read the data in the same order it was written.

Finally, we got to the setup we have now: both sides — the web app and the Lambda function — were updated to use Redis Pub/Sub implementation. Laravel supports Redis out of the box, which was a nice thing to have. For the NodeJS part, we used NodeRedis.

Subscribing to a channel

As I mentioned, Laravel already has an interface to deal with Redis. It still needs an underlying client, but most of the operations are pretty straightforward. You may refer to the Laravel docs for more info. Subscribing to a channel requires a single method call:

Redis::subscribe([ 'channel_name' ], function ($message) {
    /* Do whatever you need with the message */
}

I’m using an Artisan command to start this listener, this way:

class Subscriber extends Command
{
    protected $signature = 'redis:subscriber';

    protected $description = '...';

    public function handle()
    {
        Redis::subscribe([ 'channel_name' ], function ($message) {
            $this->processMessage($message);
        });
    }

    public function processMessage(string $message)
    {
        /* Handles the received message */
        $this->info(sprintf('Message received: %s', $message));
    }
}

Now we simply have to trigger the command to start listening to the channel.

You’ll notice that after a minute without receiving any data, the next time the subscriber gets a message an error will be thrown. That’s because the connection timed out. To fix that, we added the following settings to the config/database.php file, inside the "redis" block:

'read_write_timeout' => 0,
'persistent' => 1,

Publishing to the channel

On the NodeJS side, we need the aforementioned library. To install it:

$ npm install redis

After that, we’ll need to write our Lambda function that publishes to the channel. Since the focus is the Pub/Sub flow, I’m not using any particular logic to create the message here, just returning the attribute received with the event.

const redis = require('redis');
const client = redis.createClient();

const handler = (event, context) => {
    const message = processEvent(event);
    client.publish('channel_name', message);
    return context.done(null, {
        message,
    });
};

const processEvent = (event) => {
    /* Handles the event and return the message to publish */
    return event.message;
};

exports.handler = handler;

Notice I’m not passing any properties to the createClient function. You’ll probably want to set the host or any other custom configuration you have to properly connect to the Redis instance. Check the NodeRedis docs for more info about the available properties.

Testing all together

First, start the Artisan command. If you used the same name from my example above, you should be able to run the following:

$ php artisan redis:subscriber

Then, you have to run your Lambda function to publish messages. You can do that after deploying the code to AWS. Or, you can run it locally with a mockup of the Lambda env. Something like this:

const http = require('http');

// This is where the Lambda function is
const lambda = require('./lambda');

const context = {
    done: (error, success) => {
        if (error) {
            console.error('FAIL:', error);
            return;
        }
        console.log('OK:', success);
    },
};

const server = http.createServer((request, response) => {
    let data = '';
    request.on('data', (chunk) => {
        data += chunk;
    });
    request.on('end', () => {
        if (data) {
            const event = JSON.parse(data);
            lambda.handler(event, context);
        }
        response.end();
    });
});

server.on('clientError', (error, socket) => {
    socket.end('HTTP/1.1 400 Bad Request\r\n\r\n');
});

server.listen(3000);

This stub is a very basic mockup of Lambda env. It lacks some better error handling and validation. But for the purpose of this test, it does what we need. I strongly don’t recommend using this code in production, though.

If you named the script above, for instance, as web.js, you should be able to run it:

$ node web.js

And then invoke the function with cURL:

$ curl -d '{"message":"Hello world!"}' http://localhost:3000

The request body (with the -d param in the command) will be parsed as JSON and sent to the Lambda function as the event. If you check the function again, you’ll notice we’re using the message attribute there.

After executing that command, you should see two different outputs in your console. One from the Lambda mockup, which may look like this:

OK: { message: 'Hello world!' }

And another from the Artisan command:

Message received: Hello world!

The output will change in according to the message with the request body.

Conclusion

In this sample code, I showed the basics of Redis Pub/Sub. You don’t necessarily need AWS Lambda to use it. I just wanted to show up a “nearly” real-life use case. Sure, this is still not a real application, but I hope you got the idea.

You may have noticed, but this is a way to build what the cool kids out there call Microservices. If this is all new to you, maybe this is an opportunity to give it a chance and try to build your first distributed application.

Got comments or questions? Feel free to share them below.

Advertisements

Using MutationObserver API and undo.js to track DOM changes

I was recently working on this project that had a kind of drag and drop builder. Users should be able to drag items into a specific building area to assembly stuff — I’m sorry for not giving more details, by the time I’m writing this the product is not yet released. One of the features I had to implement was a pair of buttons to undo and redo actions.

Every time an item is added to the builder, a DOM change happens. So, I started to look for ways to track changes in the DOM. That’s when I found about MutationObserver API. That web API is implemented by most of the modern browsers, Internet Explorer 11+ included, and allows you to observe DOM changes using a simple approach. The second tool that helped me build this was undo.js. It’s a simple Javascript class that allows developers to implement actions tracking in an exceptionally easy way.

After coding this feature, it seemed to me that combination of tools might be useful for a lot of other use cases. So I wrote this fiddle to demonstrate how they can be used together:

https://jsfiddle.net/straube/wh35f5yp

If you have any question, the comments area below is all yours.

Using make to build a NodeJS app

Do you know you can use the good ol’ make to build your NodeJS app? I wrote a little about it in my latest post on Kettle’s Blog:

https://blog.kettle.io/building-aws-lambda-apps-using-make/

Passing Node args to Mocha tests

This is a really quick tip. I was looking around on the internet for a way to pass Node arguments when calling Mocha binary. And I couldn’t find anything useful. Then I tried the following and it worked:

$ test node --expose-gc ./node_modules/.bin/mocha [...]

The --expose-gc argument is just an example. You can pass any argument accepted by Node program.

In my specific case, I was trying to load dotenv config. In the end, the project’s MakeFile looked like:

test:
    @NODE_ENV=test node -r dotenv/config ./node_modules/.bin/mocha \
        --require should \
        --reporter spec \
        --harmony \
        --bail \
        tests

.PHONY: test

Photo credit: Matt Benson

Mixing HTTP and WebSocket routes in a Koa-based application

I’ve started to use the koa-websocket package. And it took me some time to figure out how to mix HTTP and WebSocket routes in a single Koa-based application. I’m not sure if this solution is obvious, but I’m sharing it anyway.

First of all, we’ll need two separate routers for regular HTTP and WebSocket routes:

// Creating a Koa app instance.
const app = require('koa')();
// "Websockifying" the application.
const socket = (require('koa-websocket'))(app);
// Loading router package
const router = require('koa-router');

// Here they are, our 2 routers
const http = router();
const ws = router();

Then, we can write our routes, plugging them to the specific router:

http.get('/', function *(next) {
    this.status = 200;
    this.body = 'Hello!';
});

ws.get('/socket', function *(next) {
    this.websocket.send('Hey!');
    this.websocket.on('message', function (message) {
        console.log(message);
    });
});

Finally, let’s make the app use the routers we created:

app.use(http.routes()).use(http.allowedMethods());
app.ws.use(ws.routes()).use(ws.allowedMethods());

Notice that the second router was added to app.ws instead of app directly.

And… That’s it.

Photo credit: Aron Van de Pol

Simple Gallery now as a jQuery plugin

One year ago I wrote a very simple image gallery script, based on jQuery. Now I updated it and completely rewrote it as a jQuery plugin. The code is available on GitHub: https://github.com/straube/simple-gallery

Photo credit: Jessica Ruscello

Simple Gallery — Uma simples galeria baseada em jQuery

Na maioria dos projetos que desenvolvemos na CodeKings, acabamos não conseguindo usar as opções de galerias disponíveis na web. Isso acontece porque a estrutura do plugin não é compatível com o layout ou faz muito mais coisas do que realmente precisamos. Nesses casos, acabamos escrevendo um script simples que resolve mais rapidamente do que se tivermos que adaptar alguma solução pronta. Com o tempo, percebi que esse script não mudava muito de um projeto para outro, então resolvi escrever uma versão genérica que é bem simples de customizar.

Publiquei o projeto no GitHub: https://github.com/straube/simple-gallery.