Elasticsearch, Logstash and Kibana for aggregate logs, analyse logs and create visualisations for application & infra monitoring.

18 Oct Doc

Elasticsearch

• it is a distributed search and analytics engine
• built on apache lucene
• it is a search engine
  • log analytics, full-text search, security intelligence, business analytics, operational intelligence
• how does it work
  • send data in JSON to Elasticsearch
  • Elasticsearch stores original documents & adds searchable reference to the doc in the cluster index
  • can search and retrieve the document using Elasticsearch API
• integration
  • using Kibana (visualization tool) to visualize data
• features
  • process large volumes of data in parallel, quickly finding the best matches for your queries.
  • Help users find the right information within your application, website, or data lake catalog.

Logstash

Importance

• fulfill need for log analytics space
• need log management + analytics solution to monitor infra

Not using Elastisearch

1. It is not as good at being a data store as some other options like MongoDB, Hadoop, etc. For smaller use cases, it will perform fine. If you are streaming TB’s of data every day, you will find that it either chokes or loses data.