@DevOpsSummit Authors: Pat Romanski, Zakia Bouachraoui, Elizabeth White, Yeshim Deniz, Liz McMillan

Related Topics: @DevOpsSummit, Microservices Expo, @CloudExpo

@DevOpsSummit: Blog Feed Post

Custom Elasticsearch Index Templates By @Sematext | @DevOps Summit [#DevOps]

Logsene is built on top of Elasticsearch and exposes a subset of its API to the users

One of the great things about Logsene, our log management tool, is that you don't need to care about the back-end - you know, where you store your logs. You just pick a log shipper (here are Top 5 Log Shippers), point it to Logsene (here's How to Send Logs to Logsene) and you are done. Logsene takes care of everything for you - your logs stop filling up your disk, you don't have to worry about log compression and rotation, your logs get indexed so when you need to troubleshoot issues you have one place where you get see and search all your logs from all your applications, servers, and environments. This is all nice and dandy, but what if your logs are special and you want them analyzed in a specific way, and not the way Logsene's predefined index templates and analysis work?  To handle such use cases we've recently made it possible for Logsene users to define how their logs are analyzed. Let's look at an example.

Registering Log Index Template in Logsene
Logsene is built on top of Elasticsearch and exposes a subset of its API to the users. Because of that, all the great tools available for Elasticsearch work with Logsene.  For example, you can use Logstash to ship logs to Logsene and you can use Kibana to search and graph logs stored in Logsene. In fact, Kibana is the alternative UI available out of the box for Logsene users. Logsene users can now use Elasticsearch Index Templates functionality to define new templates for their indices in Logsene. Let's say that we want to have a new type of logs that contain a new type, let's call it messages, with one analyzed text field - message, and two non-analyzed text fields - tag and nick. Our index template for that might look as follows:

curl -XPUT 'logsene-receiver.sematext.com/_template/ aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate' -d '{
"template" : "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*",
"order" : 21,
"settings" : {
"index.analysis.analyzer.my_own_lowercase.type" : "custom",
"index.analysis.analyzer.my_own_lowercase.tokenizer" : "keyword",
"index.analysis.analyzer.my_own_lowercase.filter.0" : "lowercase",
"mappings" : {
"message" : {
"properties" : {
"message" : { "type" : "string" },
"tags" : { "type" : "string", "analyzer" : "my_own_lowercase" },
"nick" : { "type" : "string", "analyzer" : "my_own_lowercase" }

That "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" is a fake Logsene app token we are using in this example.  You should, of course, use your own Logsene app token instead.

There are a few things one should remember when registering an index template in Logsene:

  1. To register a template one should use HTTP PUT method and send it to logsene-receiver.sematext.com/_template/TOKEN_NAME. The TOKEN part is your Logsene app's token and the NAME part is the name of the template, which should be unique for your Logsene app.
  2. The template property inside the JSON request should be set to TOKEN* (yes, with the trailing asterisk), otherwise Logsene will reject the template.
  3. The order property must to be higher than 10 and should be unique for your templates.
  4. Only mappings and settings sections of the templates are allowed, with the limitation that settings section can only contain analysis definition.
  5. You can register multiple index templates by just using a different NAME.
  6. Very importantly, keep in mind that registered index templates do not come into effect immediately - they become active within the next 24 hours - specifically, at 00:00 UTC.

The above command should result in a response similar to the following one:


This means that the template was successfully registered. If an error occurs the response from Logsene will be different, for example:

{"error":"Error occured during template verification","errorId": "2739358978185","status":"400"}

Reading Defined Templates
Of course, once your templates are in you can also read them. Doing that is very simple, you just need your Logsene app token and a request that looks as follows:

curl -XGET 'logsene-receiver.sematext.com/_template/ aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*'

The response will contain all the templates defined for the application with the specified token and will look as follows:

{"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate":{"order":21, "template":"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*","settings":
{"index.analysis.analyzer.my_own_lowercase.tokenizer":"keyword", "index.analysis.analyzer.my_own_lowercase.type":"custom", "index.analysis.analyzer.my_own_lowercase.filter.0":"lowercase"}, "mappings":{"message":{"properties":{"message":{"type":"string"},"tags":

Of course, you can also read a single template by running a command like this:

curl -XGET 'logsene-receiver.sematext.com/_template /aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate'

Deleting Templates
Sometimes you may need to remove templates you've defined.  Doing that is as simple as running HTTP DELETE command against the _template REST end-point of Logsene and specifying the template name, like this:

curl -XDELETE 'logsene-receiver.sematext.com/_template/ 18fcf616-7c1a-4bb5-840e-deaf9ad73d00_MyTemplate'

If deletion was successful Logsene will respond with the following message:


If something went wrong, you will see an error:

{"error":"Only full template names can be used with deletes","errorId":"1944493021299","status":"400"}

If you still haven't had a chance to try out Logsene, go to http://sematext.com/logsene/index.html and create a free account (or just add new Logsene application if you already have an account). You can also try a live demo of Logsene to quickly look how it works on common data.  If you can't ship your logs to the cloud, you can also run Logsene On Premises or on your own cloud instances (e.g. on AWS EC2).

Filed under: Logging Tagged: devops, log analytics, logging, logsene

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

@DevOpsSummit Stories
So the dumpster is on fire. Again. The site's down. Your boss's face is an ever-deepening purple. And you begin debating whether you should join the #incident channel or call an ambulance to deal with his impending stroke. Yes, we know this is a developer's fault. There's plenty of time for blame later. Postmortems have a macabre name because they were once intended to be Viking-like funerals for someone's job. But we're civilized now. Sort of. So we call them post-incident reviews. Fires are never going to stop. We're human. We miss bugs. Or we fat finger a command - deleting dozens of servers and bringing down S3 in US-EAST-1 for hours - effectively halting the internet. These things happen.
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
This sixteen (16) hour course provides an introduction to DevOps, the cultural and professional movement that stresses communication, collaboration, integration and automation in order to improve the flow of work between software developers and IT operations professionals. Improved workflows will result in an improved ability to design, develop, deploy and operate software and services faster.
Authorization of web applications developed in the cloud is a fundamental problem for security, yet companies often build solutions from scratch, which is error prone and impedes time to market. This talk shows developers how they can (instead) build on-top of community-owned projects and frameworks for better security.Whether you build software for enterprises, mobile, or internal microservices, security is important. Standards like SAML, OIDC, and SPIFFE help you solve identity and authentication, but for them authorization is out of scope. When you need to control "who can do what" in your app, you are on your own.
The digital transformation is real! To adapt, IT professionals need to transform their own skillset to become more multi-dimensional by gaining both depth and breadth of a wide variety of knowledge and competencies. Historically, while IT has been built on a foundation of specialty (or "I" shaped) silos, the DevOps principle of "shifting left" is opening up opportunities for developers, operational staff, security and others to grow their skills portfolio, advance their careers and become "T"-shaped.