Today we are going to become a bit more technical than usual. I wanted to talk about an open source ETL-tool called Node Red. Let’s get to it!
To start this thing off I probably should explain what an ETL-tool is. ETL stands for Extract Transfer and Load. These tools are used to automate integrations and automations between different systems. A common way to do integrations i either to buy an off-the-shelf product that is integration your two products, write the integration in a programming language or use an ETL-tool. The goal for the ETL tool is to create the integration cheaper then writing it with code – both in the initial build out but also in maintenance.
Node Red was create as a platform for making it easy to automations on the RaspBerry Pie one-card computers. But it is definitely not limited to that platform. The software is written in Node-js and can there fore be ran on anything that runs Node-js. Today that would be any platform out there.
You create your initial setup just as any other node-project. You create your package.json file – do a npm install. And you are ready to go. The power here is that you can use the node ecosystem for libraries and extensions. There are a ton of node-red specific libraries and modules that you can install for easy work within Node-Red but you can essentially use any library out there – you just need to know how to link it into the system.
When Node is up and running you can log into its web user interface. Here you create the automation or the integration by dragging nodes from a command pallet to your canvas. On the canvas you configure the nodes and connect them using lines. There is quite a few nodes that comes with the base system and there are also alot third-party modules that you can easily install with npm – and after that you can simply drag and drop those on to the canvas as well.
The automation or integration you build gets represented in a json-text file and you can put that file, together with the other node-files, into a repository and tie that to your normal code development workflow. The json is readable, but probably not your preferred way of editing the integration is through the ui.
I have used node red both for private project and professional projects. And to give you and idea.
I have a webpage where people can sign up for a newsletter. The newsletter tool I’m using for this project didn’t have a well working integration with wordpress and elementor. In wordpress I could create a simple form that submitted the content of that form into a web-hook in node-red. A posting to that web-hook would fire of a flow in node red that takes the form data – transforms it – than it uses the APIs of the newsletter provider to enroll the new subscriber. A very simple flow right! This could easily be extend to have node sending an email auto-responder to the customer and also put a message into slack say that there is a new newsletter subscriber.
You can do quite advanced automations in node-red because – if the nodes you have doesn’t fit your need – you can use a function node to write javascript that can use external libraries that you linked. It is also quite easy to create your own node modules if you want to do this.
That was a quick intro and explanation to Node-red. If you think this is interesting, send me a note, you can find the details for doing that over at martinhaagen.se/about – links in the show notes, and I can talk more about this in the future.