Hey, sorry I didnt reply sooner - hopefully that’s settled down the past couple of days? There was a bug causing the server to crash. No restrictions on vpn traffic in place :)

Hey welcome along 😊

Good timing, I’ve literally just updated to 0.19.18 which should have the fix in!

No, no meaningful cpu spikes I could make out anywhere, although admittedly I was focusing on the lemmy server container mostly

I noticed in the logs before every timeout there were lots of “verify” words appearing, and in each iteration of that statement there were more and more verify words. Honestly had no idea what it meant at the point, only that I didn’t recognise it from looking at lemmy logs previously, it always appeared before a crash, and it felt suspicious.

Here’s an example from some logs before a crash:

2026-03-15T21:47:22.670586Z  INFO HTTP request{http.method=POST http.scheme="https" http.host=lemmy.zip http.target=/inbox otel.kind="server" request_id=2cc6dc65-571d-4a69-9733-5e80e455c00b}:receive:community:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify: activitypub_federation::fetch: 
Fetching remote object https://hexbear.net/comment/7004776

thread 'actix-server worker 18' has overflowed its stack

fatal runtime error: stack overflow

I pinged some logs over to Nutomic on matrix, who thought it might have been related to nested comments, and then I noticed Dessalines had made the linked thread, which matched pretty much with what I was seeing behaviour and logs-wise.

Usefully the logs link the object it’s fetching, and 9 times out of 10 its a deeply nested hexbear thread! Or someone from another instance commenting on a nested hexbear thread. Nutomic confirmed the behaviour based on the logs in the issue, and I’m pulling the logs when I get chance to see what other threads are causing it to crash, although hopefully the fix will make it’s way into 0.19.18 beta 3 so I can stop worrying about it!

Likely related to this issue here: https://github.com/LemmyNet/lemmy/issues/6427

Something introduced in 0.19.16 is causing the server to restart. We can go 12+ hours with no problems, then 10 restarts in 3 hours. Seemingly random, but hopefully with the devs on it we can get a resolution.

Did you manage to get past cloudflare in the end? It’s on full lockdown mode like the lemmy site is, but I can probably back it off for this game tbh

I managed to accidentally lock all the chats for a while (hopefully now resolved) - I am also rewriting the chat logic to try and iron out issues.

Thanks, have reported to the blorp dev and will report to voyager dev. They work on the platform directly but the apps don’t show the links.

I was sad you lost your fastest reaction time!

still lots of people with a press left, but not sure if people just missed it ending!

I have a lot of work to do on it, but do people really want it back so quickly?

I can add a 1 hour cooldown though for a round 2 :)

Failsafe accidentally triggered, took me a sec to undo :)

Without a doubt the most important press

Try now - have tweaked some stuff to see if its a masto authorised fetch thing going on.

There is a leaderboard on there with a bunch of stats including that one :)

Go on, press it… you know you want to!

No, not yet, but I am actually pulling together something around the login as I’ve ended up using it in a couple of places with things I’m working on, so will let you know when thats up!

Failing on getting your user account, possibly because the connection gets rejected. Going to set a user agent (“TheButton/1.0”) and try again.

Have now set a user agent so you should be good to try again :)

20:09:12 [WARNING] Failed to get NodeInfo for dubvee.org: Expecting value: line 1 column 1 (char 0)

20:09:19 [WARNING] Could not resolve AP profile for ptz@dubvee.org, rejecting

❤️ im glad people enjoy them :)