FluxFur - Jouleworks Server Hijack – Incident details

All systems operational

Jouleworks Server Hijack

Resolved
Major outage
Started 4 days agoLasted 1 day

Affected

Fluxer

Major outage from 8:00 AM to 5:06 PM

Gateway

Major outage from 8:00 AM to 5:06 PM

API

Major outage from 8:00 AM to 5:06 PM

Livekit

Major outage from 8:00 AM to 5:06 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We have managed to completely recover all data and restore services. We are monitoring to ensure all communities start online and service is stable.

  • Update
    UTC
    Update

    Resilvering has completed. We are now in the old server and are migrating the data to our new server now.

  • Update
    UTC
    Update

    We have confirmed our ZFS array is still mostly intact as the attacker only wiped 1 drive. We are currently waiting for it to resilver so we will have no data loss.

    Expected to be down for another 5 hours at least for the process to finish.

  • Update
    UTC
    Update

    A rogue agent gained access to the physical server hosting FluxFur and wiped it clean. Jouleworks has completely closed the exploited IPMI connection and is working on restoring services. We are currently working on restoring FluxFur from our latest backups.

    We are confident no data has been leaked.

  • Update
    UTC
    Update

    Jouleworks is still working on the issue with their colocation. No ETA or additional information at this time.

  • Identified
    UTC
    Identified

    Jouleworks identified an issue with the physical server FluxFur (and other services) are hosted on and has a ticket open with their colocation. No ETA at this time.

    Identified server went offline around 3 AM EDT and became unresponsive to IPMI.