FARK.com status page

Fark.com status page

This is the status page for Fark's behind-the-scenes geekery. If there's a major outage or problem, we'll try to post something here about it (unless we're too up to our eyeballs in trying to actually fix it)... so check here if Fark's hosed for an extended length of time. Normally this page will be blank. :-) For more minor issues, see Mike's blog, or much more likely, Mike's twitter, Drew's twitter, Fark twitter or all Fark staff/mod/admin twitters together.

Our normal hosting provider (since July 2019) is DigitalOcean. They have their own status page. This outage page is hosted at Amazon AWS in case of DigitalOcean problems.

Fri Oct 2 09:30:10 EDT 2020: Looking into why posting new images broke overnight. This appears to be due to a software update to a 3rd party library changing an API slightly. I've updated our code to conform to the new API and it seems better now.

Tue Sep 29 23:39:55 EDT 2020: A load balancer config issue has been fixed, so tonight's poor performance under heavy load should not happen next time.

Fri Sep 4 23:52:36 EDT 2020: quick database reboot for a kernel patch

Sun Aug 30 10:02:34 EDT 2020: I'm seeing some IPv4 reachability problems from Spectrum Cable -- IPv6 is fine. Looks like a BGP issue, which would be out of our control and maybe out of theirs too. Keeping an eye on it...
UPDATE: This appears to have been a CenturyLink issue, which affected way more than just Spectrum or DigitalOcean.

Mon Aug 24 10:28:54 PDT 2020: We're aware of issues with some images not loading. The cloud service we use is doing scheduled maintenance (https://status.digitalocean.com/incidents/fx3y2q1cqcf0) and we hope they will complete it shortly.

Tue Aug 11 15:55:01 EDT 2020: There may be some slowdowns today as I repair some corrupted database indexes.

Fri Aug 21 11:14:24 EDT 2020: investigating stalled IO on database server.

Tue Aug 11 15:55:01 EDT 2020: There may be some slowdowns today as I repair some corrupted database indexes.

Sun Jul 26 12:48:19 EDT 2020: I think I have found the cause of the performance problems. I think.

Sat Jul 25 17:33:38 EDT 2020: Performance problems are being looked at

Sun Jun 28 14:43:13 EDT 2020: Quick database restart to do a software update (which for some weird reason moved the location of an important file...)

Sat Jun 20 08:41:15 EDT 2020: We're aware embedded Youtube clips in comments aren't working. It's an issue between us and Google, not anything y'all are doin' wrong. We're looking at it.

Tue Jun 9 23:58:53 EDT 2020: Rebooted everything for a kernel security patch.

Mon Jun 8 12:49:55 EDT 2020: Latest upgrade to the load balancers didn't work correctly. Rolling back to the previous release now.

Tue May 12 21:14:05 EDT 2020: Rebooted everything for a kernel security patch.

Thu Apr 30 15:30:00 EDT 2020: Microsoft has unblocked our emails, finally.

Wed Apr 29 21:10:00 EDT 2020: Looking into what appears to be internal DigitalOcean routing issues. It may not actually be causing problems with the site, but it's making my monitoring go nuts. EDIT: It's their DNS servers, according to their status page.

Wed Apr 22 16:42:22 EDT 2020: Database problems resolved.

Wed Apr 22 14:39:01 EDT 2020: We're experiencing issues with our database server and are working on the problem.

Wed Apr 15 22:14:40 EDT 2020: One of our memcached servers just... rebooted itself for no apparent reason. It's back now, but trying to figure out if it crashed or if the cloud provider rebooted it for host maintenance or something. It's looking like the latter.

Mon Apr 13 13:27:52 EDT 2020: We're aware that Microsoft is blocking emails from us. Still trying to find out why.

Thu Apr 9 09:59:25 EDT 2020: Just woke up to fix another database outdage. I think it was the same issue as April 3.

Fri Apr 3 08:55:25 EDT 2020: Just woke up to fix a two-hour database outage.

Wed Apr 1 13:07:25 EDT 2020: We had a momentary blip a few minutes ago. The h4xx0r p1g5 got into the server cache and made a mess. Should be all better now.

No obvious issues at the moment.

Live basic (very basic) health check: