Amazon OpenSearch Service with ELK

What is the ELK Stack?

The ELK stack is an acronym used to describe a stack that comprises of three popular projects: Elasticsearch, Logstash, and Kibana. Often referred to as Elasticsearch, the ELK stack gives you the ability to aggregate logs from all your systems and applications, analyze these logs, and create visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics, and more.

E = Elasticsearch 

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Support for various languages, high performance, and schema-free JSON documents makes Elasticsearch an ideal choice for various log analytics and search use cases.

On January 21, 2021, Elastic NV announced that they would change their software licensing strategy and not release new versions of Elasticsearch and Kibana under the permissive Apache License, Version 2.0 (ALv2) license. Instead, new versions of the software will be offered under the Elastic license, with source code available under the Elastic License or SSPL. These licenses are not open source and do not offer users the same freedoms. For a secure, high-quality, fully open-source search and analytics suite, you can use the OpenSearch project, a community-driven, ALv2 licensed fork of open source Elasticsearch and Kibana.

L = Logstash

Logstash is an open-source data ingestion tool that allows you to collect data from a variety of sources, transform it, and send it to your desired destination. With pre-built filters and support for over 200 plugins, Logstash allows users to easily ingest data regardless of the data source or type. 

Logstash is a lightweight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination. It is most often used as a data pipeline for Elasticsearch, an open-source analytics and search engine. Because of its tight integration with Elasticsearch, powerful log processing capabilities, and over 200 pre-built open-source plugins that can help you easily index your data, Logstash is a popular choice for loading data into Elasticsearch.

Easily load unstructured data

Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs. 

Pre-built filters

Logstash offers pre-built filters, so you can readily transform common data types, index them in Elasticsearch, and start querying without having to build custom data transformation pipelines.

Flexible plugin architecture

With over 200 plugins already available on Github, it is likely that someone has already built the plugin you need to customize your data pipeline. But if none is available that suits your requirements, you can easily create one yourself.

K = Kibana

Kibana is a data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.

On January 21, 2021, Elastic NV announced that they would change their software licensing strategy and not release new versions of Elasticsearch and Kibana under the permissive Apache License, Version 2.0 (ALv2) license. Instead, new versions of the software will be offered under the Elastic license, with source code available under the Elastic License or SSPL. These licenses are not open source and do not offer users the same freedoms. To ensure that the open source community and our customers continue to have a secure, high-quality, fully open source search and analytics suite, we introduced the OpenSearch project, a community-driven, ALv2 licensed fork of open source Elasticsearch and Kibana. The OpenSearch suite consists of a search engine, OpenSearch, and a visualization and user interface, OpenSearch Dashboards.

You can run Apache 2.0 licensed Kibana versions (up until version 7.10.2) on-premises, on Amazon EC2, or on Amazon OpenSearch Service. OpenSearch Dashboards is an open-source alternative to Kibana, which is also available to self-manage. It was derived from the last open-source version of Kibana (7.10.2) and contains many advancements and is well supported via the OpenSearch Project. With on-premises or Amazon EC2 deployments, you are responsible for provisioning the infrastructure, installing Kibana or OpenSearch Dashboards software, and managing the infrastructure. With Amazon OpenSearch Service, Kibana or OpenSearch Dashboards are deployed automatically with your domain as a fully managed service, automatically taking care of all the heavy-lifting to manage the cluster.

Interactive charts

Kibana offers intuitive charts and reports that you can use to interactively navigate through large amounts of log data. You can dynamically drag time windows, zoom in and out of specific data subsets, and drill down on reports to extract actionable insights from your data.

Mapping support

Kibana comes with powerful geospatial capabilities so you can seamlessly layer in geographical information on top of your data and visualize results on maps.

Pre-built aggregations and filters

Using Kibana’s pre-built aggregations and filters, you can run a variety of analytics like histograms, top-N queries, and trends with just a few clicks.

Easily accessible dashboards

You can easily set up dashboards and reports and share them with others. All you need is a browser to view and explore the data.

How does the ELK stack work?

1. Logstash ingests, transforms and sends the data to the right destination.
2. Elasticsearch indexes, analyzes, and searches the ingested data.
3. Kibana visualizes the results of the analysis.

Why is the ELK stack important?

The ELK Stack fulfills a need in the log analytics space. As more and more of your IT infrastructure move to public clouds, you need a log management and analytics solution to monitor this infrastructure as well as process any server logs, application logs, and clickstreams. The ELK stack provides a simple yet robust log analysis solution for your developers and DevOps engineers to gain valuable insights on failure diagnosis, application performance, and infrastructure monitoring – at a fraction of the price.

How can I choose the right solution for the ELK stack?

You can choose to deploy and manage the ELK stack yourself with Apache 2.0 licensed versions of Elasticsearch and Kibana (up until version 7.10.2) or self-manage an open source alternative to the ELK stack with OpenSearch, OpenSearch Dashboards, and Logstash. But, would you prefer that your developers or DevOps engineers spend time on building innovative applications or on managing operational tasks such as deployment, upgrades, software installation and patching, backups, and monitoring? Also, scaling up and down to meet your business requirements or achieving security and compliance is a challenge with the self-managed option.

What are AWS offerings for the ELK stack?

Amazon OpenSearch Service offers the latest versions of OpenSearch, support for 19 versions of Elasticsearch (1.5 to 7.10 versions), and visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions). The service integrates with Logstash as well as other AWS services such as Amazon Kinesis Data Firehose, Amazon CloudWatch Logs, and AWS IoT to give you the flexibility to select the data ingestion tool that meets your use case requirements.

About Amazon OpenSearch Service

Integrations are a keyway to reduce operational costs because you are using built in integrations. Ingesting data into Elasticsearch or OpenSearch can be challenging since it involves a number of steps including collecting, converting, mapping, and loading data from different data sources to your Elasticsearch or OpenSearch index. You have to convert the raw data into a structured data format such as JSON or CSV, clean it, and map it to target data fields. You also have to batch and buffer the data for efficient loading so that the data is available immediately for querying without overloading your cluster’s compute and networking resources.

With Amazon OpenSearch Service, you can easily accomplish all of this, by leveraging the integrations with Amazon Kinesis Data Firehose, Logstash, Amazon CloudWatch, or AWS IoT, providing you the flexibility to select the ingestion tool that meets your use case requirements.

Data ingestion using Amazon Kinesis Data Firehose

With Amazon Kinesis Firehose, you can easily convert raw streaming data from your data sources into the formats required by your Elasticsearch or OpenSearch index and load it to Amazon OpenSearch Service, without having to build your own data processing pipelines.

To use this feature, simply select an AWS Lambda function from the Amazon Kinesis Firehose delivery stream configuration tab in the AWS Management Console. Amazon Kinesis Firehose will automatically apply the AWS Lambda function to every input data record and load the transformed data to your Amazon OpenSearch Service index.

Amazon Kinesis Firehose provides pre-built Lambda blueprints that can be used without any change or customized for converting common data sources such as Apache logs and system logs to JSON and CSV formats. You can also configure Amazon Kinesis Firehose to automatically retry failed jobs and back up the raw streaming data. 

Data ingestion using Logstash

Amazon OpenSearch Service supports integration with Logstash, an open-source data processing tool that collects data from sources, transforms it, and then loads it to Elasticsearch or OpenSearch. You can easily deploy Logstash on Amazon EC2 and set up your Amazon OpenSearch Service domain as the backend store for all logs coming through your Logstash implementation. Logstash supports a library of pre-built filters to easily perform common transformations such as parsing unstructured log data into structured data through pattern-matching; renaming, removing, replacing, and modifying fields in your data records; and aggregating metrics. 

Data ingestion using Amazon CloudWatch Logs

Amazon CloudWatch Logs lets you monitor and troubleshoot your systems and applications using your existing system, application, and custom log files. You can configure a CloudWatch Logs log group to stream data to your Amazon OpenSearch Service domain in near real-time through a CloudWatch Logs subscription. This integration is convenient if you are already using CloudWatch Logs to collect log data and would like to share that data with your Amazon OpenSearch Service users. 

Data ingestion using AWS IoT

AWS IoT is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. With AWS IoT, you can capture data from connected devices such as consumer appliances, embedded sensors, and TV set-top boxes. Using the AWS Management console, you can configure AWS IoT to load the data directly to Amazon OpenSearch Service, enabling you to provide your customers near real-time access to IoT data and metrics. 

How to choose the right ingestion mechanism

Choosing the right ingestion mechanism depends on your use case requirements such as data latency and data type. For large data volumes, we recommend using Amazon Kinesis Data Firehose, which is fully managed, automatically scales to match the throughput of your data, and requires no ongoing administration. It can also transform, compress, and batch the data before loading it to Amazon OpenSearch Service domain. Often, the choice also comes down to the services you are already using. For example, if you are already collecting application logs using Amazon CloudWatch Logs, you can simply load that data into your Amazon OpenSearch Service domain without much additional effort.