Try removing the json codec and adding a json filter: input { file { type => “json” path => “/logs/mylogs.log” } } filter{ json{ source => “message” } } output { file { path => “/logs/out.log” } } You do not need the json codec because you do not want decode the source JSON but … Read more
Curator would be an ideal match here. You can find the link here – https://github.com/elastic/curator A command like below should work just fine – curator –host <IP> delete indices –older-than 30 –prefix “twitter-” –time-unit days –timestring ‘%Y-%m-%d’ You can keep in this in the CRON for removing the indices occasionally. You can find some examples … Read more
If you are talking about the query field a syntax like this works: field:<10 Will find just records with a field value less than 10. Found this by experimentation one day — don’t know if it’s documented anywhere.
At my company we started with Graylog2 and recently installed Kibana3. My personal opinion is that Kibana3 is more suited towards non-dev, while Graylog isn’t. Kibana: Pretty dashboards Graphs, charts and images “panel” customization, adding parallel coordinate graphs for example Easy/flexible management of dashboards (they save directly into their own ES index) Easy deployment (just … Read more
When possible, I’d go with a conditional wrapper just like the one you’re using. Feel free to post that as an answer! If your application produces only a few different line formats, you can use multiple match patterns with the grok filter. By default, the filter will process up to the first successful match: grok … Read more
To check if field foo exists: 1) For numeric type fields use: if ([foo]) { … } 2) For types other than numeric like boolean, string use: if (“” in [foo]) { … }
This is easy in Kibana 5 search bar. Just add a filter !(_exists_:”your_variable”) you can toggle the filter or write the inverse query as _exists_:”your_variable” In Kibana 4 and Kibana 3 you can use this query which is now deprecated _missing_:”your_variable” NOTE: In Elasticsearch 7.x, Kibana now has a pull down to select KQL or … Read more
As of ES 5.x , they have given this feature out of the box with logstash plugin. This will periodically import data from database and push to ES server. One has to create a simple import file given below (which is also described here) and use logstash to run the script. Logstash supports running this … Read more
To export data to csv/excel from Kibana follow the following steps:- Click on Visualize Tab & select a visualization (if created). If not created create a visualziation. Click on caret symbol (^) which is present at the bottom of the visualization. Then you will get an option of Export:Raw Formatted as the bottom of the … Read more
For Kibana 4 go to this answer This is easy to do with a terms panel: If you want to select the count of distinct IP that are in your logs, you should specify in the field clientip, you should put a big enough number in length (otherwise, it will join different IP under the … Read more