Affected Components:

Oct 26, 20:20:08 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 1 hour and 49 minutes

Affected Components:

Oct 22, 03:49:24 GMT+0
Completed - Maintenance has completed successfully..

Oct 22, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 21, 21:37:06 GMT+0
Identified - We are planning for an impromptu security maintenance during that time to remove security workarounds and patch remaining security issues from multiple CVEs announced this past week..

Duration: 1 hour and 2 minutes

Affected Components:

Oct 18, 17:32:11 GMT+0
Completed - Maintenance has completed successfully..

Oct 18, 16:59:49 GMT+0
Identified - Taking a lot longer to update the host system. Increasing maintenance window again. Sorry for the delays..

Oct 18, 16:42:15 GMT+0
Identified - Need to increase maintenance a few to patch underlying host as well..

Oct 18, 16:30:01 GMT+0
Identified - Maintenance is now in progress.

Oct 18, 16:12:23 GMT+0
Identified - We are planning for a scheduled emergency maintenance event during that time..

Duration: 4 hours and 31 minutes

Affected Components:

Oct 12, 22:31:06 GMT+0
Completed - Maintenance has completed successfully..

Oct 12, 18:00:29 GMT+0
Identified - Maintenance is now in progress..

Oct 10, 19:55:43 GMT+0
Identified - We are planning for a scheduled maintenance during that time as well as in preparation for some maintenance on Jouleworks' systems. Services will be unavailable during the maintenance window. We're hoping to be done with these maintenance windows that require being down for hours at a time soon, but bare with us a little longer..

Duration: 25 minutes

Affected Components:

Oct 7, 20:54:15 GMT+0
Completed - Maintenance has completed successfully..

Oct 7, 20:29:27 GMT+0
Identified - Maintenance is now in progress..

Oct 7, 19:33:58 GMT+0
Identified - Take two for this maintenance..

Duration: 2 hours

Affected Components:

Oct 4, 05:09:11 GMT+0
Completed - Maintenance has completed successfully..

Oct 4, 04:59:30 GMT+0
Identified - Backups complete. Updating backend now..

Oct 4, 03:17:19 GMT+0
Identified - System updates done and database backup is complete. Performing a backup on our object store (media uploads)..

Oct 4, 03:00:01 GMT+0
Identified - Maintenance is now in progress..

Oct 4, 03:00:00 GMT+0
Identified - This is our weekly scheduled maintenance event to update all services and apply non-critical security patches outside of an emergency maintenance event. Expected down time is usually within 1 hour, but scheduled for a couple hours in the event of unforeseen complications during the patching process..

Duration: 24 minutes

Affected Components:

Sep 29, 22:24:53 GMT+0
Completed - Maintenance has completed successfully..

Sep 29, 22:01:10 GMT+0
Identified - Maintenance is now in progress..

Sep 29, 19:34:45 GMT+0
Identified - Fluxer has updated their mobile clients to utilize a new captcha system which is causing mobile to fail to verify during login. We need to update our instance to support it. We will go down for approximately 30 minutes, but the maintenance event is for 1 hour just in case..

Duration: 19 minutes

Affected Components:

Sep 27, 15:48:39 GMT+0
Completed - Maintenance has completed successfully..

Sep 27, 15:30:00 GMT+0
Identified - Going offline temporarily to patch an issue with registration prior to new invite wave..

Duration: 2 hours

Affected Components:

Sep 27, 04:40:47 GMT+0
Completed - Maintenance has completed successfully, waiting for services to slowly come back online..

Sep 27, 03:00:45 GMT+0
Identified - Maintenance is now in progress..

Sep 21, 15:59:35 GMT+0
Identified - This is our weekly scheduled maintenance event to update all services and apply non-critical security patches outside of an emergency maintenance event. Expected down time is usually within 1 hour, but scheduled for a couple hours in the event of unforeseen complications during the patching process..

Duration: 1 hour and 14 minutes

Affected Components:

Sep 21, 14:29:04 GMT+0
Resolved - This incident has been resolved..

Sep 21, 14:10:06 GMT+0
Monitoring - Our host made a patch last night to fix a networking issue that seemed to have been fixed this morning (which caused an issue with said patch), so we needed to wait on reverting that change. Looks like services are back, but we're monitoring for a little bit to be sure..

Sep 21, 13:15:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 15 hours and 36 minutes

Affected Components:

Sep 21, 03:33:22 GMT+0
Completed - Maintenance has completed successfully. Server is now slowly loading back up, give it around an hour for things to things stabilize..

Sep 21, 01:10:40 GMT+0
Identified - Physical server migration has completed. We are performing our own maintenance and should be back up within the hour. Will update if there are any changes..

Sep 20, 23:56:36 GMT+0
Identified - Our host has been working on the server setup and networking for the few hours but is almost done. I want to say we will be back up within a couple hours, but we're increasing our maintenance window 4 more hours to be safe..

Sep 20, 15:08:46 GMT+0
Identified - Still waiting on the update that the servers are on-route. Jouleworks is running into delays with the colocation in getting access to the servers. We are increasing our maintenance window to take into account these delays. They are publishing live updates here: <https://journal.jouleworks.net/journal/september-20th-move-log/>.

Sep 20, 11:57:20 GMT+0
Identified - We're bringing all our services down and performing backups prior to the server migration. We will be back online after the move is complete and we validate everything..

Sep 11, 03:01:29 GMT+0
Identified - Jouleworks will be migrating all physical servers from their New Jersey colocation to a new colocation in Maryland. We will be offline during this migration. Specific times subject to change..

Duration: 1 day, 9 hours and 7 minutes

Affected Components:

Sep 6, 17:06:35 GMT+0
Resolved - This incident has been resolved..

Sep 6, 15:54:00 GMT+0
Monitoring - We have managed to completely recover all data and restore services. We are monitoring to ensure all communities start online and service is stable..

Sep 6, 11:38:58 GMT+0
Identified - Resilvering has completed. We are now in the old server and are migrating the data to our new server now..

Sep 6, 02:40:42 GMT+0
Identified - We have confirmed our ZFS array is still mostly intact as the attacker only wiped 1 drive. We are currently waiting for it to resilver so we will have no data loss. Expected to be down for another 5 hours at least for the process to finish..

Sep 6, 00:36:27 GMT+0
Identified - A rogue agent gained access to the physical server hosting FluxFur and wiped it clean. Jouleworks has completely closed the exploited IPMI connection and is working on restoring services. We are currently working on restoring FluxFur from our latest backups. We are confident no data has been leaked..

Sep 5, 15:53:19 GMT+0
Identified - Jouleworks is still working on the issue with their colocation. No ETA or additional information at this time..

Sep 5, 08:00:00 GMT+0
Identified - Jouleworks identified an issue with the physical server FluxFur (and other services) are hosted on and has a ticket open with their colocation. No ETA at this time. Identified server went offline around 3 AM EDT and became unresponsive to IPMI..

Duration: 1 hour

Affected Components:

Sep 5, 00:49:10 GMT+0
Resolved - This incident has been resolved..

Sep 4, 23:48:58 GMT+0
Investigating - Some third-party services still seem to be experiencing issues such as email, external embeds, and KLIPY GIFs. We're looking into the cause..

Duration: 1 hour and 29 minutes

Affected Components:

Sep 4, 22:51:58 GMT+0
Resolved - This incident has been resolved. All communities should be fully initialized and online within the hour..

Sep 4, 22:20:56 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 4, 22:01:29 GMT+0
Identified - Discovered an issue with our network configuration which caused a loss of network connectivity. After fixing that, some of the micro services failed to return to a healthy state. We are currently resolving those issues..

Sep 4, 21:34:45 GMT+0
Identified - We discovered a potential issue with our networking setup on the host provider end. We have engaged Jouleworks and are working to fix it as soon as possible. All running FluxFur services on the server are fine, connections to the service are the issue..

Sep 4, 21:22:50 GMT+0
Investigating - We are currently investigating this incident..

Duration: 50 minutes

Affected Components:

Sep 3, 16:45:23 GMT+0
Completed - Maintenance has completed successfully. It will take a little while for all guilds to fully come back online, so please be patient if you don't see some guilds up immediately..

Sep 3, 16:33:02 GMT+0
Identified - Maintenance has completed. Waiting on all guilds (FluxFur servers) to finish coming back online..

Sep 3, 16:12:48 GMT+0
Identified - Jouleworks has completed their maintenance. We are now updating the FluxFur services..

Sep 3, 15:55:44 GMT+0
Identified - Maintenance is now in progress..

Sep 3, 15:03:47 GMT+0
Identified - Jouleworks Services will be intermittently unavailable as they deploy a core network patch, which affects FluxFur's connectivity. Afterwards, we will be updating our container images to the latest ones provided by Fluxer..