Engineering at Lob: What We Do In Our 20% Time

lobsters working together

All Lobster engineers are permitted to spend roughly 20% of their time to work on things outside the scope of traditional sprint work. They are encouraged to work on things that they are passionate about and can impact Lob’s business, sense of community, or employee productivity.

The 20% time projects vary widely in topic and scope. For example, some Lobsters build tools to help other business units run more efficiently, while others contribute to open source. The only hard guidelines are that projects must bring value to Lob, and any code written must adhere to our standard software development processes including full test coverage and thorough code reviews.

In this post, we’ve asked our crustaceans what they have been working on in their 20% time. We’ve also included a fun fact about each Lobster.

 

Anthony once rang the opening bell for the New York Stock Exchange.

I’m working on a Lob Library in Elixir. Elixir is one of the newer technologies that we’ve been using on the engineering team at Lob and it made sense to me to offer a library in Elixir if we’re utilizing it internally at Lob.

 

Taylor recently went on an adventure to Eastern Europe and was fascinated by seeing the remains of the USSR.

I am working on adding a key grant module to the AWS Terraform provider. This module will allow specific grants to be added to a KMS key that dictate who can access a specific key and, possibly more importantly, in what context the access is taking place. Knowledge of Golang and the AWS KMS API were both necessary to effectively contribute.

 

Stewart was held as a baby by Mr. Rogers.

While working on an internal API service built in Phoenix, an Elixir web framework, I noticed Lob’s API standards and best practices weren’t exactly followed by the automatically generated code. I’ve used 20% time to create custom templates which follow our best practices more closely. Phoenix uses these templates to generate code that automatically includes metrics, exception tracking, and logging out of the box. The goal is to remove the repetition of building fundamental features that all new services should have.

 

Tony recently took his most favorite trip to the Galapagos Islands, the place that sparked Darwin’s theory of evolution.

I was tackling the problem of fraudulent accounts mailing checks which was causing our operations team a huge headache trying to manage. They had a manual review process for detecting spam accounts and banning them from our system. I integrated our system with a machine learning fraud detection platform that eliminated the need for the manual review process, and reduced the occurrence of fraudulent account activity by 99%.

 

Matt says “Saturn is the only planet in the solar system with a density less than that of water—so if you had a bathtub big enough to fit all the planets, Saturn is the only one that would float.”

We use the Elastic stack quite extensively throughout our codebase, and historically, deployments of our Logstash services have been very manual and very error-prone. On top of that, only two of our engineers (including myself) knew how to make these deployments, so working with Logstash was a big pain point for engineers. During my 20% time, I focused on automating these deployments by moving these services over to Docker and Convox, a powerful tool for managing AWS infrastructure. Now, instead of having to SSH into an EC2 instance, checkout the latest configs, ensure that all the plugins are up to date, etc., an engineer’s changes can be deployed with a single command or the click of a button.

 

Damian lived on a boat for 5 years as a kid and spent summers in high school sailing around the South Pacific.

I’ve been working on a Go library for our core API. I’m a big fan of Go as a language, so it’s been a fun challenge to design an idiomatic, developer-friendly API wrapper. It’s been a great learning experience, and as a side benefit, having this library will let us more easily use Go in future projects.

 

Pallavi got caught in the worst flooding in New Orleans since Katrina and ended up on a local news station trying to get her car out of the mud.

I’ve spent a significant amount of my 20% time on tracking more granular database and query metrics. I worked with Docker and the Datadog agent to log custom Postgres metrics. I also added query timing metrics at the application layer to our SQL query builder of choice, Knex. We have rolled out these new metrics to several services to get more insight into how our database queries are performing.

 

Lob is hiring software engineers and more! Visit our careers page to learn more about our open positions.