Lightning Platform shuts down following credential breach

The security incident

Lightning.ai identified a security incident on October 15 at 11:00 PM UTC. The breach involved unauthorized access to credentials within the Lightning Platform. The company took immediate action to secure the environment by revoking all user access on October 16 at 1:00 AM UTC.

The incident specifically impacted several core services of the Lightning Studio Platform. These include Studio, Deployments, Pipelines, and Drive. While these services are offline, the company stated that the GPU cluster rental infrastructure and managed services like Kubernetes and SLURM were not involved in the breach.

The company is currently treating this as an active investigation. They have engaged an independent external cybersecurity firm to help monitor the environment and assist with the technical audit. The platform will remain inaccessible to all users while the teams focus on addressing the event and restoring services.

How the breach occurred

Based on the initial findings, the company believes that certain customer credentials were compromised. These credentials allowed unauthorized parties to gain access to specific environments. This access may have extended to various connected third-party services linked to those environments.

Lightning has already revoked all identified credentials that were used during the incident. They are continuing to monitor the environment to ensure no further unauthorized activity occurs. The investigation aims to determine the full scope of which specific accounts were targeted and what data may have been accessed.

The company has notified customers who have been identified as potentially affected so far. These individuals were provided with account-specific information to help them manage their own response efforts. If a user has not received a direct notification, their environment has not been identified as affected based on the current investigation.

Impact on users and services

The shutdown creates significant disruption for customers who rely on the Lightning Studio Platform for their AI workflows. Because the Platform is currently inaccessible, users cannot interact with their Deployments or Pipelines. This forced outage is a precautionary measure to prevent further unauthorized access to sensitive data.

The breach is particularly concerning for users who store secrets and API keys within their AI Studios. If these keys were compromised, attackers could potentially access other cloud accounts or third-party systems. Lightning has already revoked identified keys associated with cloud accounts that were used on their platform.

Users are being urged to review their third-party systems and access logs for any suspicious activity. The company emphasizes that the goal of the shutdown is to protect the integrity of the customer environments. They are working to restore access as quickly as possible once the environment is deemed secure.

Recommended security actions

Lightning.ai recommends that all users take specific security steps regardless of whether they were contacted about the incident. One primary recommendation is to rotate all secrets and API keys. This includes revoking credentials stored in AI Studios, such as those used for GitHub access.

Users should also update the credentials used for any third-party services that were accessed by their virtual machines. This proactive approach helps ensure that even if a key was leaked during the incident, it can no longer be used by an unauthorized party. The company advises checking all connected systems for signs of compromise.

The company has provided a dedicated support email for users who have questions or observe suspicious activity. They are prioritizing the restoration of the service while maintaining a focus on security. By following the recommended rotation of keys, users can mitigate the risk of ongoing unauthorized access to their external accounts.

Next steps for restoration

The company is currently in the middle of an active investigation to fully understand the breach. They are working with external experts to ensure a thorough review of the security incident. The primary goal is to identify all compromised credentials and ensure the platform is safe for public use again.

While the platform remains offline, the team is actively working to restore access to the service for all users. They recognize the disruption this causes and are attempting to balance speed with necessary security protocols. Updates will likely be provided as the investigation progresses and the platform becomes available.

For those affected, the next step involves a manual audit of their own cloud environments and third-party logs. Users should look for any unauthorized logins or actions that occurred during the incident window. Lightning will continue to monitor the situation and provide updates as more information becomes available from their investigation.