Micro Servers: Unix Philosophy With a Hosting Bill

Because Apparently One Computer Was Too Easy

There is a particular point in every self-hoster’s life where they look at a perfectly functional server and think:

What if I bought another server?

This is usually how trouble starts.

For years the fashionable answer to almost every infrastructure problem has been increasingly complicated orchestration. Containers inside clusters, service meshes talking to service meshes, dashboards monitoring dashboards and approximately seventeen YAML files standing between you and displaying a webpage.

All because somebody needed to serve 300KB of HTML.

I have nothing against any of these technologies. They solve legitimate problems.

I just don’t particularly enjoy using a forklift to move a chair.

My own infrastructure has slowly been moving in another direction which I have started thinking of as micro-server architecture.

Not microservices in the fashionable Silicon Valley sense.

No Kubernetes.

No cluster.

No service discovery daemon having an existential crisis at 3am.

Just several deliberately small Linux servers, each responsible for a clearly defined job.

Basically:

One boring little computer. One boring little responsibility.

And the more I look at it, the less this feels like a new architectural idea.

It feels suspiciously like an old Unix idea wearing a VPS bill.

The Original Accident

ATHENSNOISE.ROCKS already works somewhat like this.

The website runs on one small VPS.

The radio runs on another.

The web server is basically Apache serving static files generated by Hugo. It doesn’t need much CPU, doesn’t need a database and generally spends most of its life wondering why I rented an entire computer for it.

The radio server has a completely different workload.

It runs AzuraCast, Docker, Liquidsoap, Icecast and the supporting services required to keep an internet radio station alive.

These two machines have very little reason to live together.

If the radio server catches fire, the website should still work.

If I screw up Apache configuration at midnight because apparently I enjoy making poor decisions, the radio should continue broadcasting.

That separation wasn’t originally some grand architectural strategy.

It was simply practical.

But over time I’ve come to appreciate the model.

The Micro-Server Idea

The basic idea is embarrassingly simple.

Instead of buying one increasingly powerful server and putting everything on it, use several inexpensive VPS instances with clearly defined roles.

Something like:

                    Internet
                        |
          +-------------+-------------+
          |                           |
     Web Server                   Radio Server
   Hugo + Apache              AzuraCast + Icecast
      1 GB RAM                    2 GB RAM
          |                           |
          |                      Block Storage
          |
     Future Services
          |
     +----+------+
     |           |
   Forum       Other
   Server      Services

A forum, for example, doesn’t necessarily need to be installed onto the existing web server simply because there happens to be some disk space available.

It could get its own little VPS.

A Linode 2 GB instance is currently roughly another $12/month.

That machine becomes:

forum.athensnoise.rocks

and its entire reason for existing is:

run the forum.

There is something deeply Unix-like about this arrangement.

And that part deserves more than one passing sentence.

Unix Philosophy, Just One Level Higher

The more I think about this architecture, the less it feels like some new infrastructure idea and the more it feels like applying an old Unix principle at a slightly different level.

The classic Unix philosophy is usually summarized as:

Do one thing and do it well.

The original idea applies to programs.

A tool shouldn’t try to edit text, send email, manage databases, render video and make coffee.

It should have a clear job.

Then other tools can be combined around it.

grep finds things.

sort sorts things.

wc counts things.

None of them needs a product manager.

The important part isn’t that every executable must perform exactly one microscopic operation.

It’s the architectural instinct behind it:

keep responsibilities understandable and keep boundaries clear.

That principle scales surprisingly well.

A process can have a clear responsibility.

A service can have a clear responsibility.

And, if the economics make sense, a machine can have a clear responsibility too.

So instead of thinking:

server01

I increasingly prefer thinking:

web
radio
forum
backup

The hostname itself should almost tell me why the machine exists.

If I SSH into the radio server, I shouldn’t discover a forgotten WordPress installation, a Git repository, an IRC bot and somebody’s experimental PostgreSQL database hiding behind the sofa.

The radio server exists to run the radio.

The web server exists to serve the website.

If a forum gets its own machine, that machine exists to run the forum.

This isn’t about architectural purity.

Sometimes several closely related services obviously belong together.

AzuraCast, Liquidsoap and Icecast are separate pieces of software, but together they form one logical system: the radio.

Splitting each of those onto individual VPS instances would not make the design more Unix-like.

It would make me an idiot with a larger hosting bill.

Unix philosophy was never:

make everything as small as physically possible.

It was closer to:

make things small enough that you can still understand what they do.

That’s an important distinction.

The goal is not maximum decomposition.

The goal is minimum unnecessary coupling.

And that is where the micro-server idea becomes interesting.

A machine becomes another composable component.

It has inputs.

It has outputs.

It exposes a small number of network services.

It can be replaced.

It can be upgraded independently.

It can fail without necessarily dragging unrelated systems down with it.

In that sense, DNS and the network become the pipes connecting the larger pieces together.

Not literally Unix pipes, obviously.

Trying to grep TCP packets directly tends to upset people.

But conceptually the architecture is similar.

The website does not need to understand how Liquidsoap works.

The radio server does not care how Hugo generates HTML.

A future forum does not need access to the radio media filesystem.

Each machine knows as little as reasonably possible about the internals of the others.

That is an old idea.

And old ideas sometimes survive because they were good ideas.

We have spent several decades inventing increasingly sophisticated terminology for software architecture, but occasionally the answer is still:

make small things, give them clear jobs, and don’t unnecessarily glue everything together.

This Is Not Really Microservices

Before somebody wearing a Kubernetes conference T-shirt sends me an angry email:

Yes.

I know.

This isn’t technically what modern software architecture calls microservices.

A microservice architecture normally decomposes an application into small networked application components.

What I’m describing is closer to service isolation at the host level.

Instead of:

one giant server
    ├── website
    ├── radio
    ├── forum
    ├── database
    ├── chat
    ├── monitoring
    └── seventeen things I forgot installing

you get:

small server
    └── website

small server
    └── radio

small server
    └── forum

Each Linux kernel becomes a fairly hard architectural boundary.

The operating system itself is effectively the container.

A remarkably sophisticated container, admittedly.

It even comes with systemd.

Please direct complaints about that sentence to /dev/null.

The Good Part: Blast Radius

The biggest advantage is isolation.

Suppose I upgrade the forum software and discover that version 8.4.7 introduces the exciting new feature of not starting anymore.

That’s inconvenient.

But the radio is still playing.

The website is still serving pages.

Nothing else particularly cares.

On the everything-server:

              FAILURE
                 |
                 v

        +----------------+
        | web            |
        | radio          |
        | forum          |
        | database       |
        | backups        |
        +----------------+
                 |
                 v
          EVERYTHING DAY

On separate hosts:

[ web   ]   OK

[ radio ]   OK

[ forum ]   FIRE
               ^
               |
        problem lives here

Compare that to the everything-server:

sudo apt upgrade

Followed approximately eleven seconds later by:

why is EVERYTHING down

Separating workloads gives failures boundaries.

This matters particularly when some services are more important than others.

ATHENSNOISE has people actually listening to the radio stream. There are external users depending on that service being available.

I therefore don’t particularly want experimental web software sharing resources with it.

Production infrastructure becomes much less adventurous when actual humans begin depending on it.

This is Unix philosophy again, just expressed operationally.

The radio should not fail because the forum decided today was a good day to eat all available memory.

The web server should not care that a radio database migration went wrong.

Unrelated things should remain unrelated for as long as reasonably possible.

Upgrades Become Smaller

Another advantage is maintenance.

If every service lives on one machine, upgrading that machine becomes an infrastructure-wide event.

The web server needs maintenance?

You potentially affect the forum.

The forum needs a newer runtime?

You potentially affect the website.

Some application wants PostgreSQL 76 Enterprise Cloud Blockchain Edition?

Congratulations. Everybody gets to participate.

Separate hosts allow upgrades to happen independently.

I can upgrade the web server and leave the radio completely alone.

Then verify it.

Then wait.

Then upgrade something else later.

This matches the way I prefer doing production work anyway:

small steps and easy reversals.

Boring maintenance is good maintenance.

I don’t want server upgrades to feel like launching Apollo 13.

And again, the Unix instinct shows up.

A component that can be maintained independently is easier to reason about than one whose internal tentacles have quietly attached themselves to six other systems.

Security Boundaries Are Nice Too

Separate systems don’t magically make infrastructure secure.

You can absolutely build five insecure servers instead of one insecure server.

Congratulations on your distributed vulnerability architecture.

But separation can reduce the consequences of a compromise.

If a forum application gets exploited, the attacker hasn’t automatically landed on the same host containing everything else.

There are still credentials, network permissions and configuration to think about, obviously.

But the architecture gives you another boundary.

The forum server doesn’t need access to the radio media library.

The web server doesn’t need database credentials for the forum.

The radio server doesn’t need to know anything about Hugo.

Each machine can expose only the ports it actually requires.

Less shared state.

Less shared configuration.

Less opportunity for one stupid mistake to spread everywhere.

I am highly qualified in stupid mistakes, so I consider this a useful feature.

A good boundary doesn’t make bad software safe.

It just gives bad software fewer rooms in the house.

The Expensive Resource Is My Brain

There is another advantage which is harder to put into a benchmark.

Mental load.

A cloud architect might look at a 2 GB VPS using 400 MB of RAM and call it waste.

And technically, yes.

There is unused RAM sitting there doing absolutely nothing.

Shocking.

But if that machine’s entire conceptual documentation is:

Purpose: Forum

I consider that fairly valuable.

If the forum breaks:

forum broken
    ↓
investigate forum server

is a pleasant mental model.

Compare that with:

forum broken
    ↓
is PostgreSQL involved?
    ↓
did PHP eat the RAM?
    ↓
why is Apache unhappy?
    ↓
why has Liquidsoap restarted?
    ↓
who touched Docker?
    ↓
why am I debugging the radio now?

CPUs are cheap.

RAM is cheap.

My Saturday afternoon is not.

This is something infrastructure discussions often miss because server resources are easy to measure and human cognitive load isn’t.

You can calculate memory utilization to two decimal places.

There isn’t usually a Grafana panel for:

How many unrelated systems do I need to understand before touching this one?

Maybe there should be.

For a small project maintained by one person or a very small team, understandability is a resource.

Probably one of the most important ones.

The Unix tradition understood this rather well.

Small, understandable components are not merely aesthetically pleasing.

They reduce the amount of state you need to hold in your head.

Scaling Is Surprisingly Sensible

Small independent servers also allow scaling by workload.

Static Hugo website getting hammered?

Scale the web server.

Radio getting more listeners?

Scale the radio server or its bandwidth.

Forum suddenly becomes popular?

Scale the forum.

You aren’t forced to upgrade everything because one application needs another gigabyte of RAM.

This is important because different services consume resources differently.

A static website barely notices CPU load.

A radio server cares about network throughput and streaming processes.

A forum may care about database performance and memory.

Treating those workloads as identical because they happen to belong to the same domain doesn’t make much architectural sense.

Again, components have different jobs.

Let them have different resources.

And Now The Bad News

Unfortunately we have invented another machine.

Every server comes with a complimentary collection of responsibilities:

• updates
• SSH configuration
• firewall rules
• backups
• monitoring
• logs
• DNS
• TLS certificates
• user accounts
• service configuration
• security updates
• recovery documentation

One server has one /etc.

Five servers have five /etcs.

Linux has stubbornly refused to synchronize them telepathically.

This is where the architecture can become annoying.

Unix philosophy may encourage small components.

It does not volunteer to maintain them for you.

Maintenance Multiplies

The biggest disadvantage is administrative overhead.

Three servers means three machines to patch.

Four means four.

Ten means you have accidentally become a hosting company.

This scales surprisingly quickly.

You start with:

web
radio
forum

Perfectly reasonable.

Six months later:

web
radio
forum
monitoring
git
chat
analytics
backup
dev
staging

And suddenly Saturday afternoon involves SSHing into ten Debian installations because OpenSSL released a security update.

At some point automation stops being optional.

Even simple shell scripts, Ansible or another lightweight configuration system starts making sense.

Otherwise configuration drift appears.

Server A has Fail2ban configured one way.

Server B has another configuration.

Server C somehow still allows password SSH authentication because you forgot about it.

Server D hasn’t been updated since the Byzantine Empire.

Distributed systems distribute your mistakes too.

This is the point where “simple independent machines” can quietly mutate into “why do I have fourteen SSH tabs open?”

The philosophy only works while the number of moving parts remains comprehensible.

Cost Creep Is Real

Tiny VPS instances are cheap.

That’s part of the attraction.

Another $5 here.

Another $12 there.

Maybe another $10 for something else.

Individually none of these seem expensive.

Then one day you look at the monthly infrastructure bill and discover you’ve accidentally recreated AWS using Linode and pocket change.

This needs discipline.

The question should always be:

Does this service actually benefit from isolation?

Not:

Could this have its own server?

Everything could have its own server.

I could deploy a dedicated VPS whose only purpose is hosting robots.txt.

That doesn’t mean I should.

Probably.

And this matters philosophically too.

“Do one thing well” is not the same as:

every single thing requires a separate kernel.

cat does not need its own datacenter.

Unix philosophy is about useful boundaries, not ritualistic atomization.

Wasted Resources

There is also inevitable resource inefficiency.

My web server is a perfect example.

Most of the time it is barely doing anything.

Apache serves static HTML.

The CPU occasionally wakes up, handles a request and presumably goes back to contemplating its career choices.

If everything lived on one larger server, unused RAM and CPU could be shared dynamically between workloads.

With separate VPS instances, unused capacity on one machine doesn’t help another.

That’s the trade-off.

You sacrifice some efficiency for isolation and operational simplicity.

For small infrastructure this can still be perfectly acceptable because the absolute cost is low.

A wasted 400MB of RAM on a tiny VPS isn’t exactly going to bankrupt the project.

And sometimes that unused capacity isn’t really waste.

It is headroom.

It is predictability.

It is the assurance that one service doesn’t suddenly decide everybody else has had enough memory for today.

Network Dependencies

Putting services on different machines also means they communicate over networks instead of localhost.

Networks are wonderful.

Networks are also unreliable bastards.

A database connection across machines introduces latency.

Firewall rules matter.

DNS matters.

Routing matters.

TLS may matter.

Provider networking matters.

And now an application can fail even though both applications involved are perfectly healthy.

This is why I wouldn’t split tightly coupled components simply for architectural purity.

If an application expects a database sitting beside it and gains nothing meaningful from separating it, leave them together.

Architecture should solve problems.

It should not create interesting diagrams.

This is where the Unix analogy has limits.

Unix pipes are cheap.

Networks are not Unix pipes with longer cables.

Every boundary has a cost.

Use one when the clarity and isolation it gives you are worth that cost.

Backups Become More Important, Not Less

More machines also means more recovery plans.

Every host should answer some basic questions:

What needs backing up?

Where is the backup?

What doesn’t need backing up?

How do I rebuild this machine?

What external storage needs reconnecting?

What order should services start?

How do I know recovery actually worked?

The upside is that these recovery procedures become smaller.

Recovering the web server means recovering the web server.

It doesn’t simultaneously involve the radio, forum and every database known to mankind.

Again:

Smaller blast radius.

Smaller recovery domain.

More individual procedures.

That’s basically the entire architecture summarized in three sentences.

The system as a whole has more pieces.

Each piece should ideally be easier to explain.

That is the bargain.

When Does This Make Sense?

I think this architecture works particularly well for small self-hosted infrastructure where:

• workloads are clearly different
• services don’t need extremely tight integration
• downtime should be isolated
• VPS instances are inexpensive
• administration is handled by one person or a small team
• simplicity matters more than maximum hardware utilization
• each host can have a clear reason for existing

It probably makes considerably less sense when you have hundreds of services and hundreds of servers.

At that point congratulations, you’ve discovered why orchestration systems exist.

But there’s a rather large territory between:

one Raspberry Pi under the television

and:

global distributed cloud infrastructure

That middle ground is where many self-hosters actually live.

And I’m increasingly convinced that a handful of boring independent Linux boxes is a perfectly respectable architecture there.

There is sometimes a strange assumption in modern infrastructure culture that if you are not progressing toward clusters, orchestration and distributed abstraction layers, you’re somehow doing an immature version of infrastructure.

I don’t buy that.

Sometimes three small Unix systems are not a stepping stone toward the real architecture.

Sometimes they are the real architecture.

The Important Part: Don’t Become The Thing You’re Avoiding

The danger is turning micro-server architecture into another religion.

That’s how we ended up here in the first place.

If one machine can comfortably and safely run three tightly related services, let it.

If separating a workload provides useful fault isolation, separate it.

If Docker makes deployment easier, use Docker.

If a normal Debian package works perfectly fine, use the Debian package.

If you need Kubernetes, use Kubernetes.

If you don’t need Kubernetes, for the love of Tux don’t deploy Kubernetes because somebody on Medium drew a colorful diagram.

Unix philosophy isn’t:

split everything until the architecture looks impressive.

It’s:

keep components understandable, composable and loosely coupled where that actually helps.

That applies whether the component is a command-line tool, a daemon, a container or an entire VPS.

The goal isn’t maximum technological sophistication.

The goal is infrastructure you understand.

Infrastructure you can repair.

Infrastructure you can afford.

Infrastructure you can explain six months later without consulting an archaeological team.

And infrastructure that doesn’t require twelve dashboards before you can determine why the homepage is returning a 500 error.

Where I’m Heading

ATHENSNOISE.ROCKS already accidentally follows this philosophy.

The web frontend is one small machine.

The radio infrastructure is another.

External radio storage has its own role.

A future forum could reasonably become another small VPS rather than being piled onto an existing production server.

Other services may eventually follow.

Or they may not.

That’s the useful part.

There doesn’t need to be a grand migration to MICRO SERVER ARCHITECTURE™ accompanied by a conference presentation and a consultant charging €900 per day.

A new machine gets introduced when there is a good reason for that machine to exist.

Ideally that reason should fit in one sentence.

If it doesn’t, perhaps the architecture needs another look.

Otherwise:

Don’t add it.

Very revolutionary stuff.

Bottom Line

Micro-server hosting trades resource efficiency for isolation.

It trades one large maintenance domain for several smaller ones.

It reduces blast radius but increases the number of things you have to maintain.

It can make upgrades safer while simultaneously giving you more upgrades to perform.

It can keep infrastructure beautifully understandable or slowly turn your SSH config into a small telephone directory.

Used sensibly, though, I like it.

Not because it is new.

Probably the opposite.

A handful of cheap Linux VPS instances, each with a clearly defined purpose, feels remarkably close to an old Unix instinct:

small things, clear jobs, narrow interfaces.

Web server serves web pages.

Radio server serves radio.

Forum server serves arguments about music.

Everything knows its job.

Each machine has a reason to exist.

And when something inevitably catches fire, at least the whole building doesn’t have to burn down with it.

That’s not particularly modern architecture.

If anything, it’s rather old-fashioned.

Which may be exactly why I trust it.