AzuraCast And The Loginbot

Last night started with an extremely reasonable idea.

I wanted AzuraCast to send me a Telegram message whenever somebody successfully logged into the backend.

That's it.

A tiny security convenience.

User logs in. Telegram goes ding. I know about it.

This should probably have been my first warning.

Enter the Telegram Bot

I created an AthensNoise Telegram bot through BotFather, confirmed that the server could actually send messages to it, and then I started figuring out where exactly to hook it into AzuraCast.

The important distinction was that I didn't want:

Someone attempted to log in.

I wanted:

Someone actually logged in.

Because the internet contains enough automated login attempts already. I do not need Telegram providing live commentary every time some script in a datacenter discovers that the word admin exists.

So I traced AzuraCast's login routing and found the proper post-authentication events:

account:login
account:login:2fa

Perfect.

Hook into the successful login path, collect the useful information and send the alert.

Naturally, our first middleware implementation managed to upset Slim because the closure returned null.

Slim, quite correctly, informed me that this was not how middleware works.

Computers remain annoyingly literal.

I replaced that with a proper MiddlewareInterface implementation, checked the PHP syntax, checked the plugin bootstrap, checked cURL, checked the environment containing the Telegram credentials and tried again.

Incognito window.

Login.

Phone:

🔐 ATHENSNOISE — AZURACAST LOGIN

User, IP, browser.

Beautiful.

The whole chain was working end-to-end.

At this point a sensible person would probably have declared victory and gone to bed.

Unfortunately there was still an AzuraCast server running nearby.

Meanwhile, the Radio Had Fallen Down the Stairs

During all this, the station started throwing Liquidsoap errors about media files under paths such as:

/var/azuracast/stations/athensnoise.rocks/media/Old Folders/...

with variations of:

Nonexistent file or ill-formed URI

This was slightly concerning because those files were not supposed to have spontaneously ceased existing.

My first reaction was essentially:

The media volume is down.

Which turned out to be almost right, but in a much more interesting way.

Actual Server Triage

This is where the night became useful.

Instead of immediately performing the traditional sysadmin ritual of:

restart thing
restart bigger thing
restart Docker
stare at screen

I started working outward from the symptom.

First question:

Does the operating system actually see the disk?

Yes.

The external library was alive, mounted as /mnt/Library, backed by the expected ext4 device, and the music was sitting there quite happily.

So this was not a dead disk.

Good.

Second question:

Can Docker see it?

Also yes.

But then came the much more important third question:

Where does Docker think AzuraCast's station media actually lives?

And there was the problem.

The external music library was mounted into Docker at /mnt/Library, while AzuraCast's station media directory lived inside its own separate azuracast_station_data volume.

Both existed.

Both worked.

They just weren't connected.

Which is a very computer way of being completely functional and completely broken simultaneously.

Nothing Is Missing. AzuraCast Just Can't Get There.

That was the important moment in the triage.

The files had not disappeared.

The disk had not failed.

Docker had not eaten the MP3 collection.

Liquidsoap was telling the truth: from its view of the filesystem, the URI genuinely didn't exist.

The host knew where the music was.

The container knew where the external disk was.

AzuraCast's station media path knew absolutely nothing about either of those facts.

So I repaired the media bind/path relationship instead of trying to “fix” a perfectly healthy disk.

And suddenly the errors started making sense.

I enjoy this stage of debugging because the server stops appearing haunted.

It becomes merely badly configured.

Much more respectable.

But Liquidsoap Remembers

With the media path repaired and the affected playlists fixed, the station could see the tracks again.

Except the upcoming queue still contained entries generated while everything was broken.

So even though reality had improved, Liquidsoap was still working from yesterday's version of reality.

Which, honestly, is relatable.

A normal station restart wasn't enough to completely shake the stale queue loose, so I finally used AzuraCast's own CLI:

azuracast:station-queues:clear

Queue cleared.

Fresh scheduling.

Jingles returned.

The mixtape podcast returned.

Radio behaving like a radio again.

The Useful Part

The amusing thing is that the Telegram bot was probably the smallest part of the evening.

But it put me in exactly the right place to catch everything else.

And for once the resulting troubleshooting wasn't random poking.

I actually did proper triage:

symptom
↓
application logs
↓
Liquidsoap paths
↓
host filesystem
↓
mount state
↓
Docker mounts
↓
AzuraCast volumes
↓
playlist state
↓
station queue

Each test removed one possibility.

By the time I changed anything substantial, I knew why I was changing it.

This is apparently called “system administration”.

I've previously preferred the technical term:

Fuck around carefully until the machine explains itself.

And Somewhere Around 2AM...

The Telegram bot was happily reporting successful AzuraCast logins.

The external media library was correctly connected again.

The playlists could see their tracks.

The stale queue had been thrown into the sea.

Liquidsoap had stopped complaining.

The jingles were playing again.

And I'd gone from:

“It would be nice if Telegram told me when somebody logs in.”

to inspecting Docker volume topology and manually clearing AzuraCast station queues.

Which is pretty much how every “small improvement” to AthensNoise works.

You add one button.

Six hours later you understand a completely different subsystem.

Still, I came out of it knowing considerably more about how AzuraCast is wired together than I did the evening before.

And now the server messages me when somebody successfully logs in.

So technically:

feature completed.

Everything between those two points was apparently complimentary.