Lucidworks Platform Status Page

Aug 7, 2026 · resolved Aug 8

## Summary On August 7, 2026, Lucidworks was alerted to several application credentials being invalidated across Connected Search, Lucidworks AI, and Lucidworks Platform, and declared a Sev1 incident. Lucidworks found that a regular cleanup job had begun erroneously designating active integrations for deletion starting on August 5, 2026, resulting in credentials that were actively in use being incorrectly invalidated and recreated. Lucidworks later determined that this same issue also invalidated embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking. We disabled the cleanup job on August 7 to stop further disruptions, rotated credentials for confirmed impacted customers, and deployed a permanent fix on August 12. No further customer-facing errors have been detected since the fix was deployed. ## Root Cause A recent code change to the backend service that manages customer integrations with a third-party authentication and authorization provider altered how that service checked for outdated integrations across a customer's full set of applications. As a result, the service could only see a partial view of each customer's active integrations rather than the complete list. Due to this, it incorrectly treated valid, active integrations as no longer in use and recreated them, generating new backend credentials in the process. Recreating an integration this way had a secondary effect: it also invalidated the embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking, in addition to the backend Platform credentials. This behavior occurred in irregular, clustered bursts rather than on a steady, predictable schedule, which made the resulting customer impact appear intermittent and difficult to correlate to a single cause. Lucidworks determined the root cause by correlating the timing of the incorrect credential and embed token changes with a recent code change to the affected service, and confirmed the diagnosis by tracing how that service determined which integrations were still active. Lucidworks confirmed the fix by deploying a corrected version of the service and validating that credential and embed token regeneration had stopped. ## Lucidworks Actions Lucidworks has taken and will take the following actions as a result of this incident: * Disabled the automated integration-reconciliation process to immediately stop further credential and embed token disruptions * Deployed a corrected version of the affected service and re-enabled the reconciliation process after validating the fix * Rotated third-party authentication and authorization credentials for confirmed impacted customer environments * We will audit and add consistent customer, application, and request tracking to logs across affected services to reduce future diagnosis time * We will evaluate an underlying third-party library defect identified during this investigation and determine whether an upgrade can remove the need for its current workaround * We will review how integration recreation is handled to reduce the blast radius of future defects, so that a single error cannot invalidate embed tokens for unrelated functionality * We will review our proactive customer outreach process to ensure affected customers are notified consistently, regardless of whether a customer contacts Support directly themselves ## Recommended Client Actions Lucidworks previously recommended that affected customers verify that any embedded Agent Studio widgets, Analytics signal collection Beacons, or usage metrics tracking are functioning normally following this incident, and contact Lucidworks Support if authentication errors persist. Lucidworks communicated this recommendation directly to affected customers prior to publication of this report.

Aug 6, 2026 · resolved Aug 6

Connected Search and the Lucidworks Platform are fully operational, having experienced no service degradation.  The root cause was confirmed to be isolated to a single customer issuing search requests using invalid credentials. The issue was validated and resolved after confirming all backend systems and platform functionality remained operating normally. Other customers utilizing the Lucidworks Platform are experiencing no impact. Please contact Lucidworks Support (https://support.lucidworks.com/hc/en-us) for any questions or assistance needed.

Jul 27, 2026 · resolved Jul 27

# Summary On July 27, 2026, at 17:56 UTC, the Secure Sockets Layer/Transport Layer Security \(SSL/TLS\) certificate serving the Lucidworks SaaS Platform domains expired. Users attempting to access the SaaS Platform UI at [platform.lucidworks.com](http://platform.lucidworks.com) received certificate errors, and Beacon signal collection via [api.lucidworks.com](http://api.lucidworks.com) was interrupted. Query serving by underlying search services was not affected, and Lucidworks AI continued to function. Investigation confirmed that the certificate covering [platform.lucidworks.com](http://platform.lucidworks.com), [api.lucidworks.com](http://api.lucidworks.com), and an internal domain had reached its expiration date without a replacement in place. The certificate was managed through a manual renewal process. Advance expiration warnings did not reach the responsible team, in part due to a delivery failure in the alert-notification pipeline, which is currently under investigation with the relevant vendors. Lucidworks generated a replacement certificate, completed domain ownership validation, and deployed the new certificate to the affected load balancers. Lucidworks engineers confirmed the new certificate was serving and verified that all affected services had returned to normal operation. Following a period of stable monitoring, the incident was confirmed to be fully resolved at 19:24 UTC. # Root Cause TLS connections to [platform.lucidworks.com](http://platform.lucidworks.com), [api.lucidworks.com](http://api.lucidworks.com), and an internal Production domain are secured by a shared Subject Alternative Name \(SAN\) certificate. This certificate was historically issued through a third-party certificate authority and renewed through a manual rotation process. Other Lucidworks domains and services use separate certificates and were not affected. The certificate reached its expiration date on July 27, 2026, at 17:56 UTC without a replacement having been issued. When the certificate expired, TLS connections to the affected domains failed, rendering the SaaS Platform UI inaccessible and interrupting Beacon signal ingestion. Advance warning of the expiration existed but did not result in action. A synthetic monitoring test generated a 30-day advance alert on June 27, 2026, which was delivered to an internal alert channel but was not acted upon. The monitoring test continued to run and fire as designed, but beginning on July 1, 2026, alerts stopped being delivered from the monitoring platform to the downstream incident management system, so the scheduled follow-up notifications and pending outage escalations never reached the responsible team. This delivery failure was silent — no monitoring existed on the alert pipeline itself — and is under active investigation with the vendors of our monitoring platform and incident management system. Review of the response itself found no platform defect. When the expired certificate was identified, replacement issuance, validation, and deployment proceeded without error. As part of the replacement, Lucidworks changed certificate providers rather than renewing through the incumbent certificate authority. Because this was the first certificate issued through the new provider for these domains, Domain Name System \(DNS\) domain ownership validation was required before the certificate could be issued. This one-time validation requirement increased Lucidworks’ overall time to resolve the issue. Subsequent issuance through this provider will not require repeating this validation step. # Lucidworks Actions Lucidworks has taken or will take the following actions as a result of this incident: * Eliminate the manual rotation step entirely * Determine the root cause and fix the alert-delivery failure between the monitoring platform and the incident-management system, in coordination with both vendors * Escalate certificate expiration alerts to a paged alerting path that requires acknowledgment and is separated from lower-urgency notification traffic # Recommended Client Actions No client action is required. Signals sent to the Beacon endpoint during the incident window \(17:56–19:01 UTC on July 27, 2026\) were not ingested and are not recoverable. Analytics for this period may reflect a corresponding gap in data or dip in graphs. Clients with questions or observations not described in this report are encouraged to contact Lucidworks Support.

Jun 19, 2026 · resolved Jun 19

The issue affecting the Commerce Studio Rules interface has been resolved, and all previously affected services are now operating normally. Commerce Studio users can interact with the Rules interface and page through large datasets without experiencing page load errors. There remains no direct end-user impact on search experiences. The root cause of the incident was database contention triggered by interactions involving a large number of rules. To resolve the issue, we scaled up the database, which normalized performance and stabilized the system.

May 28, 2026 · resolved May 28

## Summary On May 28, 2026, between 16:18 UTC and 17:53 UTC, Lucidworks AI hosted LLMs were unavailable in the `us-southcarolina` region. Customers using hosted LLM inference \(`llama-3-8b-instruct`, `llama-3v2-3b-instruct`, and `phi-4-multimodal-instruct`\) received 500 or 429 errors when attempting to query these models. Other SaaS Platform services, including search, embedding models, and the Lucidworks Platform UI, were not affected. Lucidworks Engineering declared a Sev1 incident at 16:54 UTC and began remediation efforts. All hosted LLM models were fully restored and operational at 17:53 UTC. ## Root Cause The incident was caused by a routine Kubernetes patch upgrade on a cluster in the `us-southcarolina` region.  LWAI-hosted models are served via [Ray Serve](https://docs.ray.io/en/latest/serve/index.html), which uses both “head” and “worker” nodes as part of its deployment system for routing inference requests.  The Kubernetes upgrade cycled node pools, causing all cluster head nodes and worker nodes to restart simultaneously.  Under normal conditions, the Ray cluster can tolerate a head node restart because worker nodes continue serving requests.  However, the node pool upgrades utilize a surge strategy where our platform waits for pods to leave the old node \(be evicted\) but not for them to be running on the new node. The platform considers the “drain” successful once the pod is gone from the old node and moves on to the next one, even if the pod is stuck in an “Initializing” state on the new node. This meant that all node pools were cycled in rapid succession, and both head and worker pods were evicted before any had finished initializing on their replacement nodes, resulting in a complete cluster outage. Recovery was prolonged by multiple compounding factors. First, one of the replacement head nodes was in a degraded state and unable to pull container images, requiring manual intervention to delete the node. Second, the LLM container images \(6-11 GB in size\) experienced abnormally slow Docker image transfer, taking 30-52 minutes compared to the typical 2-4 minutes observed in normal operation. Additionally, in a separate operation, new models were being brought online to expand our LWAI offering, and this caused the Ray operator's blue-green deployment strategy to require the existing _and_ replacement LLM deployments to be healthy before switching traffic, which extended the outage until the slower image pulls completed on both blue and green deployments. Lucidworks Engineering deleted the degraded node, waited for image pulls to complete on replacement nodes, and verified that all hosted models were responding to queries. The incident was verified as resolved at 18:07 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Implement sequenced Kubernetes upgrade procedures for clusters hosting LLM workloads, ensuring each node pool is fully healthy before the next pool is upgraded. * Investigate Docker image pre-loading strategies \(such as pre-baked disks or image streaming\) to eliminate long container image pull times for large ML model images. * Open a support ticket with our cloud provider to investigate the abnormal Docker pull times. * Establish a notification protocol to coordinate Kubernetes maintenance windows with LLM service owners to avoid conflicts with ongoing deployments. * Improve tooling around our Ray clusters to allow Lucidworks Engineering to force a failover to a blue or green state instead of waiting for Ray to automatically resolve the new and old deployments. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

Is Lucidworks Platform down right now?

No — Lucidworks Platform is up. All systems operational as of Aug 19, 10:44 PM UTC.

What is Lucidworks Platform's current status?

Lucidworks Platform: All Systems Operational. Alert24 checks Lucidworks Platform's status page continuously and can notify you the moment it changes.

How do I get alerted when Lucidworks Platform goes down?

Alert24 monitors Lucidworks Platform and 3,700+ other cloud and SaaS providers. When an outage is detected it updates your status page automatically and pages your on-call team — no manual checks. Start free at alert24.net.