Jeff Bezos has a lot of money. Most think he acquired his wealth from people buying everything from books and toilet paper on the Amazon storefront. The truth is that nowadays, Bezos is making a lot of money via Amazon Web Services (AWS).
We have met AWS users spending hundreds of thousands per month on database services, some of which are spending up to 40% of their monthly AWS bill on databases. And a large portion of this database revenue comes from people overpaying for their unoptimized RDS instances. In other words, they are paying Amazon more money to run their MySQL and PostgreSQL databases than what they actually need to support their applications. But it does not need to be this way.
In this article, I want to discuss three ways that we have seen AWS customers unnecessarily padding Bezos’ bank account with expensive database costs. I will also mention what Bezos does with that extra money that you give him each month. Some of his purchases are admittedly pretty good; for example, he got a $22,000 soft-serve ice cream machine for his $165M house in Los Angeles. But as we will see, other purchases are more exorbitant. I hope this will motivate the AWS DBMS user community to do something about it (spoiler: OtterTune automatic database tuning can help!) and bring Bezos’ exotic spending habits back down to earth.
#1 - Using Larger Instance Sizes Than Needed = Backup Superyacht
The great thing about running your database in the cloud is Amazon makes it easy to scale vertically and use a bigger instance size. Instead of procuring new hardware and then migrating your database to the new machine (which can take weeks or months), AWS lets you upgrade your database with more CPU and RAM with only a few clicks. Such easy scaling is helpful if your database grows and you find that query performance is getting slower over time. But there is a limit to how much faster your DBMS can get by scaling vertically, especially if you do not adjust its configuration as you change the system’s hardware. But we see customers blindly increasing their RDS instance sizes without first checking whether they can optimize their DBMS for its current instance size. And this is, of course, giving Bezos more money than he needs.
To show again the limits of scaling vertically, we ran a quick experiment using the industry-standard TPC-C benchmark on PostgreSQL RDS (v13). We start with the smallest instance size (db.m5.large) and then scale up the instance size to measure the database’s performance. For each instance, use the same database size and workload generator settings. We also let OtterTune optimize the DBMS’s configuration for each instance. The graph below shows that for this workload, using an instance size larger than db.m5.8xlarge does not make a difference. The application cannot push more queries through the system, and thus using a larger instance is a waste of money. Capacity planning for databases is a challenging problem. Using an automated tool like OtterTune to manage the DBMS’s configuration is the first step towards solving it.
What Bezos Bought with This Money:
It is hard to quantify how many people are paying for a larger RDS instance than they need, but I am sure there are many. Bezos is getting rich off people’s indolence in managing their DBMSs. He then uses that money to buy a backup yacht for his new $500m superyacht. It is well known that database billionaires like to build superyachts (Oracle’s Larry Ellison had four boats at one point). But buying a smaller yacht so your main yacht does not get lonely is next-level garishness.
#2 - Running with the Default Configuration = $23M "Museum" Mansion
Part of what makes DBMSs so enigmatic is that their performance and scalability are highly dependent on their configurations. Further exacerbating this problem is that the default configurations of DBMS knobs are notoriously bad. Know that the defaults always produce suboptimal results, even if they are “pre-tuned” by a cloud database provider like Amazon Web Services.
To give you a sense of how much using the default configuration can cost you, we ran another experiment on two PostgreSQL RDS databases. For the first database, we used a db.m5.4xlarge instance (16 vCPUs, 64GB RAM) with 4000 provisioned IOPs. For the second database, we used a smaller db.m5.2xlarge instance (8 vCPUs, 32GB RAM) with 2000 provisioned IOPS. We then ran the TPC-C benchmark on each database instance twice: once with Amazon's default RDS configuration and once using OtterTune's optimized configuration.
The graph below shows that the OtterTune-optimized PostgreSQL instance on the smaller instance achieves the same throughput performance as the larger instance still using the default configuration. But the OtterTune-optimized instance only costs $8,544/year while the other database costs $17,088/year. That means if your database is using the default configuration, you could be paying Amazon 50% more than you need to for it.
What Bezos Bought with This Money:
And so what does Jeff Bezos do with all that extra money that he earns from people using the default database configuration? He bought a $23M museum in Washington, DC and turned it into a new house for himself. The property was already the largest mansion in the city at 22,000 square feet before Bezos bought it. But after paying for extensive renovations with his database monies, the house is even larger at 34,000 square feet.
#3 - Paying For Unnecessary Provisioned IOPs = $42M Underground Clock
Using Provisioned IOPs (PIOPs) is an essential optimization in AWS to ensure that your PostgreSQL and MySQL RDS databases achieve consistent read/write performance. But PIOPs are expensive: the maximum amount of 80,000 PIOPs for a single node DBMS costs $16,000 per month as of September 2021. We have met customers that are paying AWS more money for PIOPs than for the RDS instances themselves.
Like using an instance size that is too large, people are often overpaying Amazon for more PIOPs than their RDS databases need. The challenge, of course, is that it is hard to determine the right PIOPs setting for your database as it depends on many factors. For example, suppose your application executes mostly read queries (i.e., it does not perform many writes). Then the amount of data that the DBMS reads from the disk will partly depend on whether the application’s working set (table and index data) fits in memory. PIOPs levels for write-heavy workloads are more challenging to predict because DBMSs have many internal components that write data (e.g., double-writeback buffer, write-ahead log, background writer, autovacuum).
Given this complexity, an all too common approach to improving RDS performance problems that seem to be related to disk I/O is to increase PIOPs blindly. Adjusting this setting may boost your database performance at first, but this will have diminishing returns. Unless you take the time to analyze your application’s behavior and how it uses the database, you may set the PIOPs level too high and thus overpay for unused resources.
To illustrate this point, we ran the TPC-C benchmark again on a PostgreSQL instance (db.m5.2xlarge) and scaled up the PIOPs setting for the database from the smallest amount (1000 IOPs) to the maximum amount (80,000 IOPs). We use the same instance size for every trial in this deployment but allow OtterTune to generate a new optimized configuration for each PIOPs level.
As shown in the graph above, the additional PIOPs increase PostgreSQL’s throughput by almost 6x for a 10x cost increase: from 1k ($200/month) to 10k ($2000/month). But beyond 10k PIOPs, PostgreSQL’s performance does not improve. These results do not mean PostgreSQL cannot take advantage of additional PIOPs beyond 10k; it does not make a difference for our write-heavy workload (TPC-C). So at 80k PIOPs, we would pay an extra $14,000/month for faster storage but not getting any benefit.
What Bezos Bought with This Money:
We have found that many others have also misconfigured PIOPs for their RDS instances in similar ways. And Bezos is laughing all the way to the bank. He then takes all his PIOPs winnings and writes a check to buy a $42M clock inside a mountain that only ticks once per year. From a scientific point of view, I agree that this clock is kind of cool, but it is clearly something that you buy when you already have everything. It is like getting something from The Sharper Image but for billionaires.
We love AWS and the good people who develop, manage, and operate it. We do a lot of work with AWS RDS users. These DBMSs are a great value if you use them responsibly. OtterTune can help you do that.
Jeffrey Bezos image source: Daniel Oberhaus, CC BY 4.0 https://creativecommons.org/licenses/by/4.0, via Wikimedia Commons