logs
Terabytes of logs? Manage them for FREE with episilia
"Episilia has been helping us in a big way to
manage 2TB of logs per day in production"
- VP Devops, FreightTiger
Solutions |
Full value from every log line
Episilia enables high value solutions on streaming logs to eliminate security threats, monitor user experience and troubleshoot applications.
.png)
Eliminate Security Threats
With high speed indexing at 15MBps per core and PCRE regex out of the box, incoming logs can be analyzed as they arrive and eliminate every security threat to the business.
.png)
Monitor User Experience
High velocity data coming in from streaming media, logistics carriers and payment gateways can be monitored to provide for the right user experience and rectified in realtime.
.png)
Troubleshoot Applications
Detailed application logs can be stored for months to get an accurate picture of the entire application stack and troubleshoot issues across versions without any guesswork.
Teams with high volume logs rely on Episilia to cut costs by 3x or more
Testimonial
"At nurture.farm we generate terabytes of logs daily and we use them to identify and fix production issues. Episilia operates at high throughput and low resource utilization while integrating seamlessly with the K8S cluster. Centralized logging with Episilia helps our business IT and security teams with root cause analysis."
Abhinasha Karana
Technology Architect @ nurture farm
Under the hood |
Fast and Affordable
Episilia costs less because it is fast
Zero Bottlenecks
.png)
-
Peer to Peer collaborative architecture; no SPOF
-
Stateless components to scale horizontally
-
No central database; S3 is the primary datastore for logs
-
Indexer and search instances scale independent of each other
Minimal Data Footprint
-
Data structures exploit natural order and repeatable values in logs
-
Double optimized bloom filters with index size at 1.5% of log bytes
-
Logs and index LZ4 compressed. 10TB of logs zips to 1TB on disk
-
Flatbuffer format to de-serialize data with less cpu cycles
CPU and RAM Optimized
-
C++ with SIMD indexes logs at 10-20MBps/core or 1TB/core/day
-
10 concurrent search queries with just 4 Cores + 4GB RAM
-
3 Tier cache in RAM, Rocksdb and Local FS
-
Tail has a guaranteed latency of 3-5 seconds from log write to display
"We did pre-mature optimization in every step in building Episilia, for each cpu cycle and each byte."
- Episilia Team
The features |
Built for dev and ops teams
Logging without limits, Queries in realtime, and Simple to operate
Collecting logs
Sources: Kafka, HTTP, S3
-
Supports Open Telemetry and generic JSON message format for logs
-
Logs can be transferred via Kafka, HTTP, S3 to Episilia
-
Timestamp derived from logs or stamped on arrival
-
Independent of log collectors - Fluent Bit, Vector, Filebeat
Indexing
10-20 MBps per core
-
Indexing at 10-20 MBps per core; roughly 1TB per core per day
-
Index is 1.5% of data size, covers keyword + regex
-
Indexer instances can be started/stopped to scale with no data loss
-
Group related logs together by virtual app IDs
Tailing and Alerting
K8s to console ~ 3 seconds
-
Logs tailed to Episilia console in under 3 seconds from origin
-
Tail logs support keyword and regex filters
-
Alerts on logs typically delivered within 5 seconds from origin
-
Alerts delivered to Slack, Pagerduty, email etc
Searching
1-5 seconds, 1M results
-
All logs available to search any time; no need to pre-load
-
Search queries support keyword + PCRE compliant regex queries
-
A search query needs a max of 350 millicores and 300MB RAM
-
98% of search queries return in between 1-5 seconds
-
Large search results > 1M downloaded to a csv file for analysis
-
Logs fetched from S3 on-demand; no hard dependency on disk
-
Frequently read logs are cached to local disk and RocksDB
-
Search instances can be started/stopped to scale
-
Grafana Loki browser supported besides Episilia native console
-
Save queries; share queries and search results with your team
Storage and archival
No DB, only S3
-
Data and index files are stored only in S3
-
All metadata stored in S3; no dependency on a live database
-
Any S3 store - MinIO, Azure, GCP, DigitalOcean, Alibaba, OCP
-
S3 files in date/hour folders; simple copy to archive/restore
Open and extensible
Open file formats and APIs
-
Log data files in Hadoop sequence file open format
-
Files in S3 can be processed by Hive or any HDFS engine
-
Search, tail, alerts available as APIs to integrate into third party apps
-
Supports Loki query format for search queries
Simple ops
Self-throttling, auto scaling
-
Episilia cluster is stateless; no state in any live database or disk
-
Deployed via Helm charts or docker manifests
-
Standard K8s/container nodes with AVX compatible cores for SIMD
-
Any service can be started/restarted/stopped any time
-
Distributed cluster with co-operating nodes; no leader election
-
Local redpanda for service co-ordination and communication
-
Prometheus-compatible for monitoring
-
All services support pause/resume control to avoid stop/restart
-
In-built throttling on memory usage; no crashes on high loads
-
Fine grained metrics collected to analyze and tune throughput
"We reduced our logging infrastructure cost by 4X with Episilia. There is no comparison."
- Architect, Nurture Farm
Deploy it today |
Episilia is FREE
-
Fill the form to get a FREE license
-
Install using Helm charts, instructions here
-
Send logs to Episilia cluster and view them