How a Minecraft server sleeps without losing your world

Most Minecraft servers are empty most of the time. Mine certainly was. I wanted a host where an empty server costs nothing, but where the server is still there, with the world exactly as left, when a friend joins. This is how Saryno does that, and what went wrong on the way.

The idea

Every server on Saryno is its own small virtual machine. A virtual machine is just memory and a disk, and both can be saved to files. So when the last player leaves, I can write the whole machine, memory included, to disk and stop it. A stopped machine uses no CPU and no RAM. When a player connects, I load it back and the game carries on as if nothing happened.

I run virtual machines with Cloud Hypervisor, which can take a snapshot of a running machine and restore it later. The restore does not have to read all the memory first. It can load pages on demand as the guest touches them, so the machine is running again almost immediately and fills in the rest as it goes.

The part that is easy to get wrong: the world

A snapshot of memory is not a safe copy of the world on its own. A Minecraft server holds changes in memory and writes them out in the background. If I freeze it at the wrong moment, the files on disk and the memory can disagree.

So before every snapshot, a helper inside the machine tells the game to stop its automatic saving and flush everything to disk (save-off, then save-all flush), and only continues when the game confirms it. If the game does not confirm, the snapshot waits and tries again later. A game that is still loading, for example, is not snapshotted half-started.

The bug: the server that thought it had hung

The first time I tested this properly, wakes were slow and some servers crashed. The logs held a line I had not seen before:

The server has stopped responding! This is (probably) not a Paper bug.

That is Paper's watchdog. It checks that the main thread keeps ticking. If a tick takes longer than the timeout, the watchdog assumes the server has hung, prints a thread dump and can stop it.

My servers had not hung. They had been asleep. The virtual machine's clock is part of the snapshot, so on wake the machine believed it was still the minute it fell asleep. A helper then set the clock to the real time, which could be hours later. From the watchdog's point of view, the main thread had not ticked for hours, so as far as I can tell it declared the server dead before the main thread had a chance to tick again.

The fix was to switch the watchdog off for these machines (-Ddisable.watchdog=true). The watchdog protects you from a truly hung server, but in a machine that sleeps it cannot tell sleep from a hang. I added a second change as well: if the game was running when the machine froze and is gone when it wakes, the helper starts it again instead of leaving a dead machine.

How fast is it?

Fast, but not instant. On my test machines, restoring a snapshot takes a fraction of a second, and the machine reports itself up two to five seconds after the request. A server answering a command in the console takes a little longer. The first wake after a long sleep can be slower, because the snapshot is no longer in the host's memory cache and has to be read from disk. A cold start with no snapshot, such as a brand-new server, takes about half a minute, because it boots Java and loads the world from scratch.

So the honest summary is: a few seconds, not a blink.

Why only the free tier sleeps

Sleeping is how the free tier stays free. If a server never sleeps, it holds its memory all day, and that costs real money. Paid memory stays awake with no time limit, and you can leave a server on 24 hours a day. The free server sleeps when nobody is online and can stay awake for up to two hours at a time.

What I would tell anyone building this

• Treat the clock as part of the state. Anything with timers, timeouts or watchdogs will react to a jump in time.
• Make the game confirm that it saved before you freeze it, and refuse to snapshot when it does not.
• Measure wakes with cold caches. A test where the snapshot was just written will flatter you.

If you want to try it, make a free server.