Below are some tips that we have aggregated to help startups as they grow on AWS. Most of these lessons came as a direct result of some mishap or mistake that happened in the first year of Lob. These are things that we wish we knew from day 1 that would have eliminated some early hiccups.
When trying to scale engineering one of the best things we learned was to completely separate the development infrastructure from the production infrastructure. This means development and production should be on completely separate accounts. This way only those who absolutely need access to the production infrastructure have the credentials. Meanwhile, all other developers can run tests using the development account without accidently deleting an important AMI, changing a security group, or taking down a set of core servers.
Not everyone in the organization is an AWS expert. From day 1, create a set of simple IAM groups that restrict access and give permissions to users. Ensure that both production and development accounts have these groups. This will eliminate a new hire or intern accidentally destroying something important. Be disciplined with your IAM access, never sign into the master account unless you have to update billing.
Create at least 3 different users to access your DB - an admin, a read-only user, and a read-write user. Your application should be the only one with write access. Give everyone else read-only access. This will eliminate the probability of someone dropping or overwriting your entire users table. It happens. Ensure that your database is multi-az, has automated backs, and has DB snapshots turned on. This way when something goes wrong you can revert. Also save a full copy of your DB to S3 every night at midnight.
Make sure to set up your development and production infrastructure to be identical on AWS. That includes VPC, security groups, IAM groups, AMI’s, S3 buckets. Never cross share information. This will allow your development workflow to be almost identical.
Since you will most likely want to share domains across both production and development accounts there are complicated ways to share NS records so both accounts can have access to the same domain. That way you can have www.yourwebsite.com on production and internal.yourwebsite.com on your dev. If you want even cleaner separation than you can have an entire different domain for your development account.
Don’t rely on AMI’s. They get stale quickly and can have bad data stored on them. It is best to always start with a fresh server build. This will also force you to learn how to use Chef or Puppet early on and how best automate server configuration.
Add alerts to all your AWS and DB instances. This is an easy way to know when your servers have high CPU, are down, or are running out of memory. It will also give you warnings when they are close to certain thresholds.
Let us know if you found these helpful!
