Laravel and NodeJS messaging using Redis Pub/Sub

I was recently working on this project that was composed of two different parts: a web application built in PHP with Laravel, and an AWS Lambda function written in NodeJS. In the past, both applications exchanged data using a common MySQL database. With time, this setup showed up very inefficient. As the number of “messages” sent and received increased, the database started to not handling well the volume of reads and writes required to support both “applications” — the Lambda function is not an application per se but you know what I mean, right?

The first thing we tried was changing the database schema to focus on performance, rather than on data integrity. We dropped some constraints and changed how the data was stored to achieve that. The updates soon showed themselves not enough.

In a second iteration, we started playing around with Redis. Due to its nature, a key/value store and not a relational database, it’s a lot faster than MySQL. The first attempt using Redis involved simply moving the data we’re storing into the database to a set. It seemed to work well but just after a few tests on a staging server we realized that approach wouldn’t work for the system needs. When retrieving the data using the SCAN command, the order of returned elements is not guaranteed. And that was an important downside for us, the business logic required us to read the data in the same order it was written.

Finally, we got to the setup we have now: both sides — the web app and the Lambda function — were updated to use Redis Pub/Sub implementation. Laravel supports Redis out of the box, which was a nice thing to have. For the NodeJS part, we used NodeRedis.

Subscribing to a channel

As I mentioned, Laravel already has an interface to deal with Redis. It still needs an underlying client, but most of the operations are pretty straightforward. You may refer to the Laravel docs for more info. Subscribing to a channel requires a single method call:

Redis::subscribe([ 'channel_name' ], function ($message) {
    /* Do whatever you need with the message */
}

I’m using an Artisan command to start this listener, this way:

class Subscriber extends Command
{
    protected $signature = 'redis:subscriber';

    protected $description = '...';

    public function handle()
    {
        Redis::subscribe([ 'channel_name' ], function ($message) {
            $this->processMessage($message);
        });
    }

    public function processMessage(string $message)
    {
        /* Handles the received message */
        $this->info(sprintf('Message received: %s', $message));
    }
}

Now we simply have to trigger the command to start listening to the channel.

You’ll notice that after a minute without receiving any data, the next time the subscriber gets a message an error will be thrown. That’s because the connection timed out. To fix that, we added the following settings to the config/database.php file, inside the "redis" block:

'read_write_timeout' => 0,
'persistent' => 1,

Publishing to the channel

On the NodeJS side, we need the aforementioned library. To install it:

$ npm install redis

After that, we’ll need to write our Lambda function that publishes to the channel. Since the focus is the Pub/Sub flow, I’m not using any particular logic to create the message here, just returning the attribute received with the event.

const redis = require('redis');
const client = redis.createClient();

const handler = (event, context) => {
    const message = processEvent(event);
    client.publish('channel_name', message);
    return context.done(null, {
        message,
    });
};

const processEvent = (event) => {
    /* Handles the event and return the message to publish */
    return event.message;
};

exports.handler = handler;

Notice I’m not passing any properties to the createClient function. You’ll probably want to set the host or any other custom configuration you have to properly connect to the Redis instance. Check the NodeRedis docs for more info about the available properties.

Testing all together

First, start the Artisan command. If you used the same name from my example above, you should be able to run the following:

$ php artisan redis:subscriber

Then, you have to run your Lambda function to publish messages. You can do that after deploying the code to AWS. Or, you can run it locally with a mockup of the Lambda env. Something like this:

const http = require('http');

// This is where the Lambda function is
const lambda = require('./lambda');

const context = {
    done: (error, success) => {
        if (error) {
            console.error('FAIL:', error);
            return;
        }
        console.log('OK:', success);
    },
};

const server = http.createServer((request, response) => {
    let data = '';
    request.on('data', (chunk) => {
        data += chunk;
    });
    request.on('end', () => {
        if (data) {
            const event = JSON.parse(data);
            lambda.handler(event, context);
        }
        response.end();
    });
});

server.on('clientError', (error, socket) => {
    socket.end('HTTP/1.1 400 Bad Request\r\n\r\n');
});

server.listen(3000);

This stub is a very basic mockup of Lambda env. It lacks some better error handling and validation. But for the purpose of this test, it does what we need. I strongly don’t recommend using this code in production, though.

If you named the script above, for instance, as web.js, you should be able to run it:

$ node web.js

And then invoke the function with cURL:

$ curl -d '{"message":"Hello world!"}' http://localhost:3000

The request body (with the -d param in the command) will be parsed as JSON and sent to the Lambda function as the event. If you check the function again, you’ll notice we’re using the message attribute there.

After executing that command, you should see two different outputs in your console. One from the Lambda mockup, which may look like this:

OK: { message: 'Hello world!' }

And another from the Artisan command:

Message received: Hello world!

The output will change in according to the message with the request body.

Conclusion

In this sample code, I showed the basics of Redis Pub/Sub. You don’t necessarily need AWS Lambda to use it. I just wanted to show up a “nearly” real-life use case. Sure, this is still not a real application, but I hope you got the idea.

You may have noticed, but this is a way to build what the cool kids out there call Microservices. If this is all new to you, maybe this is an opportunity to give it a chance and try to build your first distributed application.

Got comments or questions? Feel free to share them below.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.