It was a dark and stormy night in May when our internal metric readings for memory usage spiked in our Address Verification service for no obvious reason. Even weirder than that, our largest API and main service had suddenly started crashing and restarting every several hours—a dark night indeed. The following is a summary of our painful investigation over several weeks, involving multiple team members and endless :confused_dog: and :computer_rage_bang_head: emojis. But (spoiler alert!), after the storm comes a rainbow.
Read on to understand how Docker handles file writes to memory and the role hapi plays. TL;DR: We learned how Docker memory caching works and found a 7-year old file management bug that led to the discovery of a secret hidden hapi vulnerability; all of which helped stabilize our containers and product offering.
Traditionally for those who have worked with Docker and Datadog, the main way to measure memory usage in a container is to look for Docker.mem.in_use. Now for reasons unbeknownst to me, Docker.mem.in_use used to only measure Rss memory usage rather than the combination of both Docker.mem.rss and Docker.mem.cache:
RSS memory usage is essentially what most would think about when they see a memory leak; a piece of the heap not being garbage collected, or an object that continues to grow infinitely. And that’s why a large chunk of our investigation looked towards traditional sources of memory leaks. But after several weeks with little progress, I started to piece together why our container memory readings had spiked and why the memory leak in our main service had started at roughly the same time.
During my investigation I discovered that on May 9th, Datadog made a change in how memory data is read: mem_in_use is now a combination of both rss and cache usage.
This was when a lightbulb went off: None of our Docker containers had any mounts of volumes attached, so whenever we wrote to disk, we would be writing to memory instead. Logically it makes quite a bit of sense, if you don’t have disk space the only other place you can write to is memory(ram).
This actually even goes a step further; there might be instances where you may have a mount or volume but you still notice docker.mem.cache going up. Well certain linux distributions will freely cache disk read/writes to memory if there is ample space, and will automatically start freeing it up once you start running low. For a deeper breakdown, here’s a great article: Linux Ate my RAM!
So:
import { promises as fs } from 'fs'
await fs.writeFile(...)
…wasn’t writing to disk but straight to memory. Specifically for our Address Verification Service, we load nearly 4 gigabytes of files into memory at runtime in order to operate.
Now this helped us identify and resolve the problem in our Address Verification service: we were improperly using cached Docker builds. Once we pushed a fix for that, our memory readings went back to normal. For those that are confused why that might be the case, the details need to be kept in house, but in short: if our assets had been properly cached as a docker layer, the 4 GB of data in that case would be counted as part of the image size and would be part of Docker’s storage driver. That way it would no longer be using memory to store those files and instead would properly be a part of the disk space.
With one service fixed, I just had to fix the memory problem in our core API. To understand how I fixed it, I think it’s relevant to provide some background into hapi.js. We use hapi as our main framework for our api and are currently running v20. As a company that deals heavily with turning assets into printable mail pieces we needed a framework that makes it easy to work with parsing payloads.
The main options of dealing with payloads that Hapi provides are:
The first option, Data, is fairly straightforward: if you pass a JSON body into a Post request it’ll parse it into JSON data. Streams will do the same thing, but if you get a multipart upload (i.e., a file is uploaded), it’ll read that file into a stream without any additional tooling needed. The third is similar, but it’ll write a file to a specified directory rather than give you a stream:
'File’: the incoming payload is written to a temporary file in the directory specified by the uploads settings. If the payload is 'multipart/form-data' and parse is true, field values are presented as text while files are saved to disk. Note that it is the sole responsibility of the application to clean up the files generated by the framework. This can be done by keeping track of which files are used (e.g. using the request.app object), and listening to the server 'response' event to perform cleanup.
Now there’s two things that developers should be aware of:
While the hapi docs do let us know that we should be deleting assets, they fail to raise a pretty important concern in that it’s super important to listen to a server response event rather than manually handling file deletion within your normal code paths. The huge issue is that payload parsing occurs first in the hapi lifecycle before almost anything else—stuff like authentication and request validation happen after a payload is parsed—creating somewhat of a vulnerability if you’re not careful.
Say a malicious actor, let’s call him Bob, had a personal vendetta against Lob. They could continually try to hit our endpoints with ill-formatted requests, along with a massive 200 mb asset attached. That request payload would be parsed and the file would be written to memory. Now we’re smart enough to create some validators that will check any files attached to make sure they’re less than a certain size, let’s say 20 mb in this case. That same request would certainly fail payload validation and then we would send a 422 response. But that 200 mb of data is now sitting in memory, and with a couple more requests it would start causing our containers to run out of memory. So rather than making sure these files were deleted our normal code path and controllers, we had to make sure they were being deleted regardless of whether they made it that far in the pipeline.
All it took was a little chunk of code to clean up response payloads right before a response.
Say we’re handling a request like this:
/v1/upload
{
“Upload_name”: “Testing”,
“file_we_want_to_upload”: “/tmp/file.html”
}
We need a response handler that’ll delete that file right before sending a response regardless of whether it was successful or not.
module.exports = {
name: 'cleanUpAssets',
register: async function (server) {
server.events.on('response', async function (request) {
rimraf(request.payload.file_we_want_to_upload) }
};
And with these changes our memory was no longer spiking! The best part? Even after we shrunk our container memory limit by 1/4th, we were still seeing stable memory readings. Celebration ensued! Now the emojis were of the :partyporg: :dancingpickle: variety.
This reduction in cpu and memory is likely to result in AWS savings from right-sizing containers; there may even be future savings when we complete our migration to Nomad.
Engineers may occasionally get called stubborn, but in the Case of the Mysterious Memory Leak, tenacity paid off. In leveling up our Docker and hapi knowledge, we uncovered a legacy bug and a potential vulnerability, and the changes we made resulted in a more stable, secure, and cost-effective operation.
