@DevOpsSummit Authors: Liz McMillan, Pat Romanski, Dalibor Siroky, Jignesh Solanki, Dana Gardner

Related Topics: @DevOpsSummit, Microservices Expo, @CloudExpo

@DevOpsSummit: Blog Feed Post

Custom Elasticsearch Index Templates By @Sematext | @DevOps Summit [#DevOps]

Logsene is built on top of Elasticsearch and exposes a subset of its API to the users

One of the great things about Logsene, our log management tool, is that you don't need to care about the back-end - you know, where you store your logs. You just pick a log shipper (here are Top 5 Log Shippers), point it to Logsene (here's How to Send Logs to Logsene) and you are done. Logsene takes care of everything for you - your logs stop filling up your disk, you don't have to worry about log compression and rotation, your logs get indexed so when you need to troubleshoot issues you have one place where you get see and search all your logs from all your applications, servers, and environments. This is all nice and dandy, but what if your logs are special and you want them analyzed in a specific way, and not the way Logsene's predefined index templates and analysis work?  To handle such use cases we've recently made it possible for Logsene users to define how their logs are analyzed. Let's look at an example.

Registering Log Index Template in Logsene
Logsene is built on top of Elasticsearch and exposes a subset of its API to the users. Because of that, all the great tools available for Elasticsearch work with Logsene.  For example, you can use Logstash to ship logs to Logsene and you can use Kibana to search and graph logs stored in Logsene. In fact, Kibana is the alternative UI available out of the box for Logsene users. Logsene users can now use Elasticsearch Index Templates functionality to define new templates for their indices in Logsene. Let's say that we want to have a new type of logs that contain a new type, let's call it messages, with one analyzed text field - message, and two non-analyzed text fields - tag and nick. Our index template for that might look as follows:

curl -XPUT 'logsene-receiver.sematext.com/_template/ aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate' -d '{
"template" : "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*",
"order" : 21,
"settings" : {
"index.analysis.analyzer.my_own_lowercase.type" : "custom",
"index.analysis.analyzer.my_own_lowercase.tokenizer" : "keyword",
"index.analysis.analyzer.my_own_lowercase.filter.0" : "lowercase",
"mappings" : {
"message" : {
"properties" : {
"message" : { "type" : "string" },
"tags" : { "type" : "string", "analyzer" : "my_own_lowercase" },
"nick" : { "type" : "string", "analyzer" : "my_own_lowercase" }

That "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" is a fake Logsene app token we are using in this example.  You should, of course, use your own Logsene app token instead.

There are a few things one should remember when registering an index template in Logsene:

  1. To register a template one should use HTTP PUT method and send it to logsene-receiver.sematext.com/_template/TOKEN_NAME. The TOKEN part is your Logsene app's token and the NAME part is the name of the template, which should be unique for your Logsene app.
  2. The template property inside the JSON request should be set to TOKEN* (yes, with the trailing asterisk), otherwise Logsene will reject the template.
  3. The order property must to be higher than 10 and should be unique for your templates.
  4. Only mappings and settings sections of the templates are allowed, with the limitation that settings section can only contain analysis definition.
  5. You can register multiple index templates by just using a different NAME.
  6. Very importantly, keep in mind that registered index templates do not come into effect immediately - they become active within the next 24 hours - specifically, at 00:00 UTC.

The above command should result in a response similar to the following one:


This means that the template was successfully registered. If an error occurs the response from Logsene will be different, for example:

{"error":"Error occured during template verification","errorId": "2739358978185","status":"400"}

Reading Defined Templates
Of course, once your templates are in you can also read them. Doing that is very simple, you just need your Logsene app token and a request that looks as follows:

curl -XGET 'logsene-receiver.sematext.com/_template/ aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*'

The response will contain all the templates defined for the application with the specified token and will look as follows:

{"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate":{"order":21, "template":"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*","settings":
{"index.analysis.analyzer.my_own_lowercase.tokenizer":"keyword", "index.analysis.analyzer.my_own_lowercase.type":"custom", "index.analysis.analyzer.my_own_lowercase.filter.0":"lowercase"}, "mappings":{"message":{"properties":{"message":{"type":"string"},"tags":

Of course, you can also read a single template by running a command like this:

curl -XGET 'logsene-receiver.sematext.com/_template /aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate'

Deleting Templates
Sometimes you may need to remove templates you've defined.  Doing that is as simple as running HTTP DELETE command against the _template REST end-point of Logsene and specifying the template name, like this:

curl -XDELETE 'logsene-receiver.sematext.com/_template/ 18fcf616-7c1a-4bb5-840e-deaf9ad73d00_MyTemplate'

If deletion was successful Logsene will respond with the following message:


If something went wrong, you will see an error:

{"error":"Only full template names can be used with deletes","errorId":"1944493021299","status":"400"}

If you still haven't had a chance to try out Logsene, go to http://sematext.com/logsene/index.html and create a free account (or just add new Logsene application if you already have an account). You can also try a live demo of Logsene to quickly look how it works on common data.  If you can't ship your logs to the cloud, you can also run Logsene On Premises or on your own cloud instances (e.g. on AWS EC2).

Filed under: Logging Tagged: devops, log analytics, logging, logsene

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

@DevOpsSummit Stories
As Marc Andreessen says software is eating the world. Everything is rapidly moving toward being software-defined – from our phones and cars through our washing machines to the datacenter. However, there are larger challenges when implementing software defined on a larger scale - when building software defined infrastructure. In his session at 16th Cloud Expo, Boyan Ivanov, CEO of StorPool, provided some practical insights on what, how and why when implementing "software-defined" in the datacenter.
ChatOps is an emerging topic that has led to the wide availability of integrations between group chat and various other tools/platforms. Currently, HipChat is an extremely powerful collaboration platform due to the various ChatOps integrations that are available. However, DevOps automation can involve orchestration and complex workflows. In his session at @DevOpsSummit at 20th Cloud Expo, Himanshu Chhetri, CTO at Addteq, will cover practical examples and use cases such as self-provisioning infrastructure/applications, self-remediation workflows, integrating monitoring and complimenting integrations between Atlassian tools and other top tools in the industry.
The need for greater agility and scalability necessitated the digital transformation in the form of following equation: monolithic to microservices to serverless architecture (FaaS). To keep up with the cut-throat competition, the organisations need to update their technology stack to make software development their differentiating factor. Thus microservices architecture emerged as a potential method to provide development teams with greater flexibility and other advantages, such as the ability to deliver applications at warp speed using infrastructure as a service (IaaS) and platform as a service (PaaS) environments.
The use of containers by developers -- and now increasingly IT operators -- has grown from infatuation to deep and abiding love. But as with any long-term affair, the honeymoon soon leads to needing to live well together ... and maybe even getting some relationship help along the way. And so it goes with container orchestration and automation solutions, which are rapidly emerging as the means to maintain the bliss between rapid container adoption and broad container use among multiple cloud hosts. This BriefingsDirect cloud services maturity discussion focuses on new ways to gain container orchestration, to better use serverless computing models, and employ inclusive management to keep the container love alive.
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, which can process our conversational commands and orchestrate the outcomes we request across our personal and professional realm of connected devices.