Lob offers direct mail automation via a suite of APIs designed for developers. Even though we now offer a low-code solution to automate direct mail (via our Campaigns product), we will always be a developer-first company. We are excited to announce an open-source project that gives back to our developer community.
TLDR: We’ve released @lob/sqs-consumer, an open-source library that makes it easier to work with AWS SQS; this includes support for AWS SDK V3, updated dependencies, a new Jest test suite, and fixes to major issues (ex: enables support for FIFO queues).
AWS SQS, or Simple Queue Service, is a piece of infrastructure offered by Amazon. Like the name suggests, it’s a queue, or a way to make our data stand in line and wait to get processed. When there are multiple actors feeding data in and just as many reading data out, you need a faucet of sorts to control that flow. It’s a very powerful tool for processing data in a controlled manner.
Owned by the British Broadcasting Corporation, “sqs-consumer” is an existing open-source library in GitHub for working with SQS. This library provides an easy way to configure the multitude of SQS settings, all with minimal code. It’s like a remote control that allows you to define only the pieces that you really care about when working with SQS—popping messages from the queue, deleting them, setting time-out intervals, and changing that depending on errors, and so on. It has been a pretty popular library—we’ve used it in Lob API and other services—but Lob and the open-source community have major concerns.
The bbc/sqs-consumer library is no longer being maintained. Pull requests involving major issues and features are being ignored, and most importantly, it’s not been upgraded to the new version of AWS SDK (V3).
To fully understand the differences between V2 and V3, see this article, or the AWS docs, but from our perspective, the benefits of AWS SDK V3 are:
In short, upgrading to the latest version of the AWS SDK is necessary to improve the stability and maintainability of our platform.
We have services at Lob that need to be upgraded to V3 but they use sqs-consumer, which is stuck on V2. It would appear the current existing solution has been abandoned and is not being actively maintained, thus won’t ever be upgraded to V3 of the SDK. Finally, if we want to use V3 of the SDK, we’d have to write our own custom boilerplate code for SQS (reinvent the wheel). Arg!
It took a few dedicated days of work during a hackathon, but on October 28, 2022, we released @lob/sqs-consumer! It's an open fork of the previous package, but it makes a few key changes:
sqs-consumer got a glow up!
Migrating over and swapping to the lob/sqs-consumer library is very easy. All you need to do is create a new V3 client; everything else, your code, your configuration, it all stays the same. You can download this package here.
This new-and-improved library removed major roadblocks and has allowed us—and developers—to move forward with development in AWS SDK V3. This is a huge win for our engineers, and we look forward to feedback and contribution from the developer community.
UPDATE: In December, after a year and a half of radio silence, the BBC started making some updates to the original package! Not saying we had anything to do with that, but... 😉
Special thanks to Software Engineer, Sishaar Rao, with help from Staff Software Engineer, Landon Barnickle, and Senior Software Engineer, Hanqing Chen.
