Operator overloading in Python and ORMs

If you worked with ORMs in Python such as SQLAlchemy or Django ORM, you probably saw or wrote code like this when dealing with queries:

result = query.filter(User.email == "me@domain.com")

This is something that bothered me when I first saw it. Coming from languages like PHP and JavaScript, that equality operation as a function argument doesn’t make a lot of sense. In those languages, the operation inside the function call would be evaluated to a boolean. Always passing true or false to the filter function would never have the desired side effect.

There it comes operator overloading. The email model attribute is a specific type (e.g., Column) that overrides the __eq__ function. When the == operator is applied, instead of calling the base equality operation, it uses the custom implementation.

Internally, this would be something like this:

class Column:
    # [...]
    def __eq__(self, other):
        return Comparison(self, "=", other)

This Comparison instance can then be transformed into email = 'me@domain.com' when compiling the query to actual SQL.

Here I’m talking specifically about equality, but this is also valid for other operators, such as comparisons (>, >=, <, <=), etc.

Going beyond ORMs and queries, this can be applied to any class in Python. Even built-in types such as lists use this internally, for instance, when you write this:

result = ["a"] + ["b"]

Python internally does this:

result = ["a"].__add__(["b"])

And both give the same result:

print(result) # ['a', 'b']

Photo by Jay Heike

It’s been a while…

… since the last time I wrote here, 3 years ago. And a lot has changed. Back then, I was still doing some PHP, but slowly shifting toward a fully JavaScript/TypeScript stack, working with Node.js and Edge Functions on the backend, and React and Next.js on the frontend. Then, at the end of October 2023, I joined MileIQ. There, I’ve been mainly doing backend on a Python-based stack. I like changes, and, even though this was quite a challenge because I had zero previous work experience with Python, it was definitely a challenge, but one I’ve really enjoyed. I’ve learned a ton over the past year. And not just the language or stack: I started using Terraform, leveled up my Docker and Kubernetes skills, worked on CI/CD pipelines in new ways, got introduced to a new project management methodology (Shape Up), and also improved my game in communication and feedback, just to mention a few.

My main takeaway from this recent experience is that I found myself again in software engineering. I’d been drifting for a while, letting external factors steer my career. But now I’m thinking ahead again about what I want, where I want to go, and how to get there. I believe a great deal of it comes from the difference that the working culture one is involved in can make.

Photo by Rory McKeever.

A PHP engineer working with Node.js

Most of the software projects I engaged in during my career had PHP as their main programming language. Yet, I also had the opportunity to work with other stacks. One of these times I developed the backend of a waterway monitoring app using Node.js. The main purpose of the app was to provide relevant data for better fishing. It was a big challenge, not only because I didn’t have much experience with the platform but also because the scope implied dealing with a huge volume of data. The backend included two parts: the data fetching and processing service, and an API.

The data service part took care of fetching data from public sources, parsing, filtering, and storing it. Web scrapping techniques were used to fetch the data. The sources used different formats varying from HTML to CSV. This way, there was a parser for each one. All data were then filtered and shaped into Javascript objects, following a single convention. To, finally, be stored in the database.

There was the API seating on the other side of the database, fetching the data requested by the mobile app. Both historical and forecast data were provided by the API. These included water info — such as CFS (cubic feet per second, which is the flow rate), water height, turbidity, and a couple of other values — and weather details. Users could filter the data by specific points in a given river (there were thousands of these points, called stations) and create alerts for when a sensor reached a specific value. These alerts triggered push notifications.

This is a brief explanation of the main features and the challenges involved in this project. Unfortunately, the app didn’t pass the beta phase due to business-related factors. In the first version, MySQL was picked as the datastore. As soon as the database grew in size, it showed to be not the best choice. I’d probably have to migrate to a storage alternative that fits better the context. Cassandra could be an option. Also, coming from PHP and its mostly synchronous flow, I had to get used to all the callbacks and promises required to write the logic. In any case, dealing with large amounts of data might not be easy but it’s really fun.

Photo by Manuel Sardo on Unsplash

Installing Composer v1 in a CircleCI build

Composer v2 has been released for a while now, however some older projects still might rely on its previous major version. When using the official images from CircleCI to build a project, you’ll probably get the latest version of Composer installed. If that’s causing a build to break, it’s possible to downgrade Composer by adding the following line to .circleci/config.yml file, before the composer install command:

# Overriding composer installation to use v1
- run: curl -sS https://getcomposer.org/installer -o composer-setup.php
- run: sudo php composer-setup.php --filename=composer --install-dir=/usr/local/bin --version=1.10.19
- run: sudo rm composer-setup.php

Notice the setup command now specifies a version. Unfortunately, it’s not possible to pass only the major version to version argument and hope for it to install the latest release for it. If it’s important for your scenario, you may have to update your CI config in time to time to have Composer always up-to-date.


Photo by Wan San Yip on Unsplash

Add Bootstrap styles to a project without Webpack

The directions below are not only applicable to Bootstrap but to any SASS-based framework or library.

Introduction

I started working on a small project that compiles to a small set of static HTML+CSS. Since I don’t have any Javascript in it, using Webpack didn’t look the right way to include the Bootstrap files I wanted. Especially since Webpack requires a JS file as the entry.

Steps

First of all, you need the SASS client to compile the styles. On MacOS you can use Dart SASS. Installing it from Homebrew is a matter of one command line:

npm instal sass/sass/sass

If you’re not using any NPM dependencies, i.e. don’t have a package.json file, you may go to your project’s root dir and create it with:

npm init -y

Then, add the following scripts to the package.json — notice that if you already have a package.json in your project, the script names below (watch and build) may already exists. In that case, add some suffix to avoid conflicts. watch-styles and build-styles, for instance.

"scripts": {
    "watch": "sass --watch --load-path=node_modules --style=compressed src/styles/app.scss dist/styles/app.css",
    "build": "sass --load-path=node_modules --style=compressed src/styles/app.scss dist/styles/app.css"
}

In these scripts, the respective paths to the main SASS file and target CSS files are src/styles/app.scss and dist/styles/app.css. You may have to tweak those paths to match your current project structure.

In case you noted, there is a --load-path option in the script. If needed, multiple options like that can be added. With that option pointing to node_modules dir, for example, it’s possible to import Bootstrap into your SASS files this way:

@import "bootstrap/scss/bootstrap";

Now, simply run the scripts from the command line, through npm binary. To watch for changes and automatically compile the styles (useful during development):

npm run watch

To simply build the CSS file from the SASS source:

npm run build

Conclusion

Despite NPM is part of the recipe, it’s not required. With SASS installed, it’s possible to call it directly from the command line. For example:

sass --watch --load-path=node_modules --style=compressed src/styles/app.scss dist/styles/app.css

However, since you probably are using or will use NPM to load Bootstrap or any other framework/library, the scripts are a nice shorthand.

One way or another, running SASS client directly is an approach to get rid of using Webpack or another bundler for just compile SASS.


Photo by JOSHUA COLEMAN on Unsplash

Avoiding state update on unmounted React component within useEffect hook

The hooks introduced with React 16.8 are a nice way to control state and other features on functional React components. They discard the necessity of third-party libraries such as Recompose – which is the inspiration for React hooks, by the way.

One way one can apply the useEffect hook, for example, is to mimic the behavior of componentDidMount event to bind a value returned by an asynchronous function to a given state. Like:

const React, { useEffect, useState } from 'react';

export default Component = () => {
    const [ data, setData ] = useState(null);
    useEffect(() => {
        (async () {
            setData(await someAsyncApiCall());
        })();
    }, []);
    return (<span>{ data }</span>);

Calling the useEffect hook with an empty array will make it be called only when the component was mounted, i.e. is the same as componentDidMount event.

However, if the component has unmounted some point before the async call finishes, it’s likely to get the following error in the console:

Can’t perform a React state update on an unmounted component

That happens because the async function called setData, which is the state setter, after the component has unmounted. To fix that error, it’s required to cancel the async function before changing the state in case the component did unmount.

In a “regular” React component, there is the componentWillUnmount event. One could set a flag to indicate the component is not mounted anymore to avoid setting the state late. Simulating this same approach with hooks is not hard. A function returned by a useEffect hook with an empty array in the second argument (the case above) run when the component will unmount. So, one could have the following:

const React, { useEffect, useState } from 'react';

export default Component = () => {
    const [ data, setData ] = useState(null);
    useEffect(() => {
        let mounted = true; // Indicate the mount state
        (async () {
            const data = await someAsyncApiCall();
            if (!mounted) return; // Exits if comp. is not mounted
            setData(data);
        })();
        return () => { // Runs when component will unmount
            mounted = false;
        };
    }, []);
    return (<span>{ data }</span>);

Now the error is gone since the state is not updated if the component is not mounted anymore.

I particularly am fond of hooks. I used to add Recompose to projects in the past and like the way the code looks with functional components. There is also, the performance benefits.

If you have questions or suggestions, feel free to express yourself in the comments below.

Custom setters with React useState hook

One of the recent additions to React was its hooks. With that, developers don’t need to rely on libraries such as Recompose to add state (and other features) to functional components.

If you’re one of those devs and were used to create custom setters for states, it’s possible you find the new useState hook somewhat limited. But you still can write your own setters, wrapping the setter returned by useState into another function.

Consider the following Recompose code:

withStateHandlers({
    counter: 0,
}, {
    increment: ({ counter }) => {
        return () => ({
            counter: counter + 1,
        });
    },
    decrement: ({ counter }) => {
        return () => ({
            counter: counter - 1,
        });
    },
});

You may write the same logic with React hooks this way:

const [ counter, setCounter ] = useState(0);
const increment = () => {
    setCounter(counter + 1);
};
const decrement = () => {
    setCounter(counter - 1);
};

This probably seems pretty obvious but there are some cases where you may need to apply some side-effect to your setter and this approach would be useful. Here it goes an example, given the same state above:

const [ counter, _setCounter ] = useState(0);
const setCounter = (counter) => {
    // Some side-effect here ...
    _setCounter(counter);
    // ... or there
};

For questions and suggestions, please leave a comment below.

Checking DNS records of a specific nameserver

I’m migrating a site from GoDaddy to HostGator (I personally would prefer to move over AWS but that’s the client’s choice in this specific case) and I wanted to get the IP of the new server on HostGator. I couldn’t find the IP anywhere on their control panel. Pinging the temporary URL they provide for testing was returning a different address — I suspect they have some kind of proxy in place for this temp URLs.

I know the host command could give me the DNS settings, which would include the A record with the IP address. However, the domain was still set to use GoDaddy’s nameservers (NS). Running host "name.tld" would give me the DNS records from GoDaddy. However, I was wondering if I could run the command against a specific nameserver. It makes sense having a way to look for records that way, right?

Looking at the man page for host command, there is a server argument. The signature looks like host [options] host [server]. So, I ended up running:

host -a "name.tld" "new.nameserver.tld"

And, voilà! I got the IP address for the new web server.

Got questions? Feel free to comment below.

Laravel and NodeJS messaging using Redis Pub/Sub

I was recently working on this project that was composed of two different parts: a web application built in PHP with Laravel, and an AWS Lambda function written in NodeJS. In the past, both applications exchanged data using a common MySQL database. With time, this setup showed up very inefficient. As the number of “messages” sent and received increased, the database started to not handling well the volume of reads and writes required to support both “applications” — the Lambda function is not an application per se but you know what I mean, right?

The first thing we tried was changing the database schema to focus on performance, rather than on data integrity. We dropped some constraints and changed how the data was stored to achieve that. The updates soon showed themselves not enough.

In a second iteration, we started playing around with Redis. Due to its nature, a key/value store and not a relational database, it’s a lot faster than MySQL. The first attempt using Redis involved simply moving the data we’re storing into the database to a set. It seemed to work well but just after a few tests on a staging server we realized that approach wouldn’t work for the system needs. When retrieving the data using the SCAN command, the order of returned elements is not guaranteed. And that was an important downside for us, the business logic required us to read the data in the same order it was written.

Finally, we got to the setup we have now: both sides — the web app and the Lambda function — were updated to use Redis Pub/Sub implementation. Laravel supports Redis out of the box, which was a nice thing to have. For the NodeJS part, we used NodeRedis.

Subscribing to a channel

As I mentioned, Laravel already has an interface to deal with Redis. It still needs an underlying client, but most of the operations are pretty straightforward. You may refer to the Laravel docs for more info. Subscribing to a channel requires a single method call:

Redis::subscribe([ 'channel_name' ], function ($message) {
    /* Do whatever you need with the message */
}

I’m using an Artisan command to start this listener, this way:

class Subscriber extends Command
{
    protected $signature = 'redis:subscriber';

    protected $description = '...';

    public function handle()
    {
        Redis::subscribe([ 'channel_name' ], function ($message) {
            $this->processMessage($message);
        });
    }

    public function processMessage(string $message)
    {
        /* Handles the received message */
        $this->info(sprintf('Message received: %s', $message));
    }
}

Now we simply have to trigger the command to start listening to the channel.

You’ll notice that after a minute without receiving any data, the next time the subscriber gets a message an error will be thrown. That’s because the connection timed out. To fix that, we added the following settings to the config/database.php file, inside the "redis" block:

'read_write_timeout' => 0,
'persistent' => 1,

Publishing to the channel

On the NodeJS side, we need the aforementioned library. To install it:

$ npm install redis

After that, we’ll need to write our Lambda function that publishes to the channel. Since the focus is the Pub/Sub flow, I’m not using any particular logic to create the message here, just returning the attribute received with the event.

const redis = require('redis');
const client = redis.createClient();

const handler = (event, context) => {
    const message = processEvent(event);
    client.publish('channel_name', message);
    return context.done(null, {
        message,
    });
};

const processEvent = (event) => {
    /* Handles the event and return the message to publish */
    return event.message;
};

exports.handler = handler;

Notice I’m not passing any properties to the createClient function. You’ll probably want to set the host or any other custom configuration you have to properly connect to the Redis instance. Check the NodeRedis docs for more info about the available properties.

Testing all together

First, start the Artisan command. If you used the same name from my example above, you should be able to run the following:

$ php artisan redis:subscriber

Then, you have to run your Lambda function to publish messages. You can do that after deploying the code to AWS. Or, you can run it locally with a mockup of the Lambda env. Something like this:

const http = require('http');

// This is where the Lambda function is
const lambda = require('./lambda');

const context = {
    done: (error, success) => {
        if (error) {
            console.error('FAIL:', error);
            return;
        }
        console.log('OK:', success);
    },
};

const server = http.createServer((request, response) => {
    let data = '';
    request.on('data', (chunk) => {
        data += chunk;
    });
    request.on('end', () => {
        if (data) {
            const event = JSON.parse(data);
            lambda.handler(event, context);
        }
        response.end();
    });
});

server.on('clientError', (error, socket) => {
    socket.end('HTTP/1.1 400 Bad Request\r\n\r\n');
});

server.listen(3000);

This stub is a very basic mockup of Lambda env. It lacks some better error handling and validation. But for the purpose of this test, it does what we need. I strongly don’t recommend using this code in production, though.

If you named the script above, for instance, as web.js, you should be able to run it:

$ node web.js

And then invoke the function with cURL:

$ curl -d '{"message":"Hello world!"}' http://localhost:3000

The request body (with the -d param in the command) will be parsed as JSON and sent to the Lambda function as the event. If you check the function again, you’ll notice we’re using the message attribute there.

After executing that command, you should see two different outputs in your console. One from the Lambda mockup, which may look like this:

OK: { message: 'Hello world!' }

And another from the Artisan command:

Message received: Hello world!

The output will change in according to the message with the request body.

Conclusion

In this sample code, I showed the basics of Redis Pub/Sub. You don’t necessarily need AWS Lambda to use it. I just wanted to show up a “nearly” real-life use case. Sure, this is still not a real application, but I hope you got the idea.

You may have noticed, but this is a way to build what the cool kids out there call Microservices. If this is all new to you, maybe this is an opportunity to give it a chance and try to build your first distributed application.

Got comments or questions? Feel free to share them below.

Using MutationObserver API and undo.js to track DOM changes

I was recently working on this project that had a kind of drag and drop builder. Users should be able to drag items into a specific building area to assembly stuff — I’m sorry for not giving more details, by the time I’m writing this the product is not yet released. One of the features I had to implement was a pair of buttons to undo and redo actions.

Every time an item is added to the builder, a DOM change happens. So, I started to look for ways to track changes in the DOM. That’s when I found about MutationObserver API. That web API is implemented by most of the modern browsers, Internet Explorer 11+ included, and allows you to observe DOM changes using a simple approach. The second tool that helped me build this was undo.js. It’s a simple Javascript class that allows developers to implement actions tracking in an exceptionally easy way.

After coding this feature, it seemed to me that combination of tools might be useful for a lot of other use cases. So I wrote this fiddle to demonstrate how they can be used together:

https://jsfiddle.net/straube/wh35f5yp

If you have any question, the comments area below is all yours.

Create your own website, free on WordPress.com.Create your own website.
Start free