Scaling Elasticsearch Clusters to Store Hundreds of TB

Scaling Elasticsearch Clusters to Store Hundreds of TB

Bitdefender is a cybersecurity leader delivering best-in-class threat prevention, detection, and response solutions worldwide.

We will present Karma, our framework built to ingest, process, and visualize security-related events and analytics data. Karma is a framework compound of a number of node.js microservices, multi-nodes Elasticsearch clusters, and Kibana plugins, which ingests a very high number of ops/sec, and storage of hundreds of terabytes. We will present the architecture and the scalability steps we took to support this vast amount of data.

Before Karma, each Bitdefender team had its own way of collecting events and analytics. Nowadays, all cyber security attacks are interconnected, and we successfully integrate our laboratories’ individual knowledge into a single correlation platform. Using Elasticsearch and Kibana capabilities, Karma represents the entry point for many of Bitdefender’s security initiatives.