How to Architect a SaaS Application That Scales From 100 Users to 100,000

Here is the moment this article is written for. You’re at 800 customers. Response times that were instant in January take a beat in September. One engineer says you need a bigger database instance. Another says the architecture “won’t scale” and floats a rewrite. A third says it’s fine. They are all guessing, and every option quoted has a different number of zeros on it.

The good news: SaaS systems are boringly consistent about how they fail under growth. The bottlenecks arrive in a predictable order, each has a cheapest intervention, and almost none of them require the dramatic version. This is the playbook we use as a SaaS development company, organized by the order of magnitude where each wall appears.

The Scaling Ladder: What Actually Breaks at Each Order of Magnitude

Two things about this table before the details. The user counts are markers, not laws: a data-heavy analytics product hits the 10,000-user wall at 2,000 users, a simple CRUD tool coasts to 30,000. What’s stable is the order. And notice what never appears as a first move: microservices, Kubernetes, or a rewrite. Those are answers to organizational scale more than user scale, and reaching for them early is how teams acquire the complexity of 100,000 users at the traffic of 1,000, the self-inflicted version of load-bearing code.

Multi-Tenancy: The Decision That’s Hard to Reverse

Most scaling decisions are reversible; this one mostly isn’t, which is why it comes first. Multi-tenancy has three broad shapes: pooled (all tenants share tables, separated by a tenant ID on every row), siloed (a database per tenant), and bridged (pooled for the many, siloed for the few big ones). Pooled is the right default for almost every B2B SaaS: cheapest to operate, simplest to deploy, and it scales further than founders expect. Siloed buys isolation and per-tenant compliance stories at the cost of operational sprawl.

The part that matters at 800 customers: whichever model you’re in, enforce it like a law now. Every query scoped by tenant, enforced in one shared layer rather than by developer memory, because a single unscoped query is both a security incident and the thing that makes later migration miserable. And keep the bridge option honest: the day a whale customer demands isolation, you want “we move your tenant to its own database” to be a project, not a rewrite. Retrofitting tenancy discipline into a codebase that grew without it is precisely the renovation problem from the load-bearing code post: possible, slow, and best avoided by an early decision.

The Database: A Long, Boring Stretch Before Anything Exotic

Database scaling advice on the internet skips to the exciting end: sharding, distributed SQL, event sourcing. Here is the actual sequence, in order of cost. First, indexes and query tuning: at the 1,000-user wall, the overwhelming majority of “we need a bigger server” situations are five slow queries and two missing indexes. Read the slow query log before reading a single vendor page. Second, one vertical upsize: paying for a larger instance is not a failure of engineering; it is often the cheapest engineering decision available, buying months of runway for a known monthly price. Third, cache the hot reads (next section). Fourth, read replicas: when dashboards and reports genuinely saturate the primary, route reads to replicas, and accept the new complexity of replication lag. Last, and rarely, partitioning and sharding: real solutions with real costs in application complexity, justified when write volume has a visible ceiling on a single primary, and almost never before.

Donald Knuth’s fifty-year-old warning remains the best database scaling advice ever written:

“Premature optimization is the root of all evil.”

Donald Knuth, Structured Programming with go to Statements (1974)

The corollary founders need: premature distribution is its most expensive modern form. Every hop up this ladder adds permanent operational complexity, so the discipline is to take each step when a gauge demands it, not when a conference talk does.

Async Everything: The Request Path You Must Keep Short

The 10,000-user wall is rarely the database; it is the request path doing work the user never asked to wait for. A “create invoice” request that also renders a PDF, sends two emails, fires three webhooks, and updates analytics is a request that takes four seconds on a good day and times out on a bad one, and it holds a worker hostage the whole time.

The rule is one sentence: if a step can fail without the user needing to know immediately, it does not belong in the request. The request validates, writes the record, enqueues everything else, and responds. Workers handle the rest with retries, which also buys you a property money can’t otherwise purchase: when the email provider has an outage, your app doesn’t. This is the single highest-leverage architectural change on the whole ladder, it works with any queue technology your stack already offers, and it is also what keeps the team fast, because background jobs can be changed and redeployed without touching the request path, the drag mechanics we covered in terminal velocity.

Caching Layers, in Order of Return on Effort

Caching advice fails when it starts with “add Redis” instead of with the order of returns. First, the CDN for static assets: an afternoon of configuration that removes the largest bytes from your servers entirely. Second, HTTP caching headers on anything publicly cacheable. Third, application-level caching of expensive reads: the rendered dashboard, the computed report, the settings object read on every request; this is where Redis or Memcached earns its place, keyed carefully and invalidated on write. Fourth, database-adjacent caching (materialized views, precomputed aggregates) for the analytics queries that will never be fast live. Each layer is roughly a week of work; take them in order and stop when the gauges relax. The revenue logic is Deloitte’s: tenths of a second move conversion rates by whole percentage points, which prices a caching sprint rather well.

One warning: every cache is a new way to serve stale data. Cache things whose staleness has a known, acceptable window, and invalidate on write for the rest. “Why does the dashboard show yesterday’s number” is a support ticket; “why did the invoice show the old amount” is an incident.

Observability: Seeing the Wall 60 Days Out

Everything above assumes you know which wall is next, and that knowledge comes from exactly four gauges. Scaling emergencies are almost never surprises; they are gauges nobody was watching.

p95 latency, as a trend: averages hide suffering; the 95th percentile is what your unluckiest users feel every day. Alert on three weeks of drift, not one bad afternoon. Database saturation: CPU, connection count, and IOPS plotted against their ceilings, reviewed weekly; a resource at 60% and climbing 5% a month is a dated appointment with a wall, roughly 60 days out. Queue depth and oldest-job age: a backlog that grows across days announces worker capacity failure about a week before customers feel it. The ten slowest queries, week over week: fix the new entrants while they’re cheap. Half a day to set up on any monitoring stack, and it converts every decision in this article from argument into arithmetic.

The Scaling Readiness Checklist

• Every query is tenant-scoped through one enforced layer, not by convention.
• The slow query log is reviewed on a schedule, and the top ten have owners.
• Anything the user isn’t waiting for runs through a queue with retries.
• Caching exists at the CDN and hot-read layers, with deliberate invalidation.
• p95 latency, database saturation, and queue depth are graphed with alerts on trends.
• The next bottleneck has a name, a gauge, and a planned intervention, written down.

Six yeses means growth is an engineering schedule. Fewer means the next wall picks its own date, and walls prefer launch weeks.

FAQs

Our SaaS is slowing down. How do we tell if we need a bigger server or a different architecture?

Read the gauges before buying anything: the slow query log, p95 latency trend, database saturation, and queue depth. In most sub-10,000-user systems the answer is five slow queries and missing indexes, fixable in days. A bigger instance is the right call when the whole database is uniformly busy; architecture changes are the right call only when a gauge shows a structural ceiling.

When should a SaaS application move to microservices?

Later than the internet suggests. Microservices solve team-coordination problems (many teams shipping independently) more than user-scale problems. A well-factored monolith with queues, caching, and read replicas comfortably serves 100,000 users. Split services when team boundaries demand it, not when a user milestone arrives.

Which multi-tenant architecture should we choose: shared database or database per tenant?

Pooled (shared tables with a tenant ID, enforced in one shared layer) is the right default for most B2B SaaS: cheapest to run and simplest to ship. Move specific large tenants to their own database when isolation or compliance demands it, which is a manageable project if tenancy discipline was enforced from the start.

How do we know a scaling wall is coming before customers feel it?

Watch four gauges weekly: p95 latency trend, database CPU, connections and IOPS against their ceilings, queue depth with oldest-job age, and the ten slowest queries week over week. A resource climbing steadily toward a ceiling gives you roughly 60 days of warning, which is enough time to fix things calmly and in order.

Do we need sharding to reach 100,000 users?

Usually not. Sharding is the last resort after indexes, caching, async processing, one or two vertical upsizes, read replicas, and partitioning, because it permanently complicates every query and migration. Most applications reach 100,000 users without it; the ones that need it earlier have unusually write-heavy workloads and will see it clearly in the write-capacity gauge. An experienced web development services partner should show you that evidence before proposing it.

Sources

• Deloitte and Google, Milliseconds Make Millions (37 brands, 30 million user sessions)
• web.dev, Milliseconds Make Millions case study summary
• Donald Knuth, Structured Programming with go to Statements, ACM Computing Surveys (1974)

Looking for timeline and cost estimates for your app?

Contact us

Ankush Mathur

Ankush Mathur leads technology at Techuz as CTO & Technical Project Manager, where he helps startups and enterprises architect and scale their software. He's spent his career moving from hands-on development to technology leadership, and enjoys writing about engineering practices, AI, and the decisions behind building solid products.