PKI Storm: Managing 100,000 Simultaneous Certificate Requests in Kubernetes Recovery Scenarios
In large enterprises, TLS certificates are required everywhere: for services, workstations, containers, internal APIs, and system integrations. When thousands of consumers exist, PKI must handle massive request flows rather than individual certificates. One organization processed approximately 320 million certificate issuance requests per year.
Automation relied on the ESAUS central management system, which validates requests against security policies and forwards them to external certificate authorities without issuing certificates itself. For Kubernetes with Istio, the Citadel component of ESAUS was configured as the certificate source instead of Istio’s built-in CA. Citadel validates requests and context before passing them through ESAUS to corporate CAs, integrating Kubernetes into the enterprise PKI.
This created a dependency: Kubernetes recovery speed became tied to external CA performance. After a data center outage, dozens of thousands of pods restarted simultaneously. Because services used mTLS, each pod needed its own certificate and private key. In normal operation requests were spread over time, but during recovery roughly 100,000 requests arrived in a short window.
At 70 RPS the theoretical processing time reached almost 24 minutes. Timeouts triggered retries, amplifying load in a thundering herd effect. The infrastructure recovered faster than the CA could issue certificates, revealing that average load figures (10–11 RPS) did not predict burst behavior.
Vertical or horizontal CA scaling was evaluated. Reaching 1,050 RPS would theoretically cut storm time to 1.5 minutes, yet such capacity would remain idle more than 99 % of the time. The team instead decoupled issuance from consumption by pre-generating certificates and keys during normal operation and storing them in a secure reserve.
The reserve resides in the new Unified Secret Storage (ЕХС) layer, which provides centralized protected storage, strict authentication, access controls, auditing, high read throughput, and object lifecycle management. In performance mode, ЕХС delivers up to 14,000 RPS for existing secrets from standby nodes, allowing the burst to be served from reads rather than real-time CA operations.
The approach also addressed Key Recovery Agent (KRA) protection by keeping the KRA private key outside the CA and releasing it only for approved recovery operations. Over time the same storage layer absorbed additional operational secrets beyond the emergency reserve, evolving into a broader corporate secret management platform.
Key lessons include recognizing that average load poorly predicts disaster behavior, that some bottlenecks are better solved by separating production and consumption than by scaling, and that every architectural mitigation transfers complexity rather than eliminating it.
Related articles
Yandex Black October: How Architectural Flaws and Capacity Mismanagement Turned Data Center Losses Into a 11-Day Fintech Coma
In October 2026 three Yandex data centers in Central Russia were physically destroyed within days, triggering the collapse of Yandex Cloud and multiple core services. Between the first and third site failures engineers had 74 hours to restore quorum and rebalance workloads yet failed to do so because Control Plane components remained tied to the lost zone and spare capacity had been eliminated to cut costs. Core banking and BNPL systems entered Read-Only mode after losing quorum, forcing a complete stop-the-world reconciliation that lasted until 22 October. Backups stored inside the same regional S3 buckets proved unusable, resulting in a seven-day RPO and primary-key collisions after restoration. The incident exposed that Metro-Cluster designs had been incorrectly treated as full Disaster Recovery and that aggressive AP microservice patterns created logical split-brain conditions outside the database layer.
Astra Group Releases Low-Code Astra AI Platform for Building Corporate AI Agents in Closed Environments
Astra Group has launched the Astra AI Platform, a low-code constructor designed for creating corporate AI applications that operate entirely within closed circuits without reliance on foreign clouds. The platform targets small and medium businesses as well as large corporations, enabling rapid deployment of AI secretaries, technical support assistants, and other agents in days rather than months. It forms part of the broader Astra AI ecosystem and includes two main constructors: one for machine learning model training and deployment, and another for LLM-based language assistants, document search, and process automation. Users can assemble scenarios from visual blocks, debug them in real time, or extend functionality with custom Python code when needed. The solution supports RAG-based knowledge bases, integration with external tools such as CRM and ERP systems, and the coordination of multiple agents for distributed task handling. One demonstrated use case involves processing ten pages of engineering drawings and specifications in five minutes to prepare data for downstream systems. The product is already listed in the Russian domestic software registry and available to customers.
Multi-Tenancy Isolation in VictoriaMetrics, VictoriaLogs and Grafana: Building Client Separation in a Shared Stack
Voltir runs a single shared monitoring stack consisting of VictoriaMetrics, VictoriaLogs, Grafana and vmalert for all customers instead of deploying separate instances per tenant. Isolation relies on numeric account IDs enforced at every layer: nginx validates tokens against the platform API before proxying writes, Grafana data sources hard-code tenant accounts, and vmalert rules receive automatic account filters. The team discovered that a single malformed rule or template could break alerting for every tenant because vmalert loads the entire rules directory at once. They also found that Grafana automatically created personal organizations granting Org Admin rights, accumulating hundreds of orphaned organizations and users that consumed disk space and resources. A dedicated cleanup module plus hourly reconciliation now removes rules, organizations, series and users when a tenant is deleted, while isolation tests verify both data and API objects across tenants.
The Real Costs of Using Claude and GPT for Russian Businesses Exposed
Accessing generative AI models like Claude and GPT has become straightforward for Russian companies, yet calculating their true expenses remains complex. Hidden factors such as intermediary commissions, currency conversion rates, taxes, and the often-overlooked costs of model reasoning tokens can double or triple initial estimates. Founder Vitaly Nagay of the Heli service analyzed these financial and organizational risks for Russian firms relying on foreign AI providers. In one example, a support department processing 20,000 monthly queries sees costs rise from $240 to $480 when reasoning tokens are included. Nagay recommends evaluating providers by dividing actual ruble charges by the model's official dollar pricing to reveal hidden markups. Additional concerns involve data control, document compliance, and access history when employees use personal subscriptions. Companies are advised to verify billing details, data storage policies, and issue dedicated API keys with spending limits before procurement.