ELK is a general-purpose no-sql stack that can be used for monitoring. We’ve successfully deployed one on production and used it for some aspects of our monitoring system. You can ship metrics into it (if you wish) and use it to monitor them, but its not specifically designed to do that. Nor does the non-commercial … Read more
TLDR: Put “type”: “nested”, in your nested type. Say we have a normal type, and another type nested in it: { “some_index”: { “mappings”: { “normal_type”: { “properties”: { “nested_type”: { “type”: “nested”, “properties”: { “address”: { “type”: “string” }, “country”: { “type”: “string” } } }, “first_name”: { “type”: “string” }, “last_name”: { “type”: … Read more
Even though you can force ES to refresh itself, you’ve correctly noticed that it might hurt performance. One solution around this and what people often do (myself included) is to give an illusion of real-time. In the end, it’s merely a UX challenge and not really a technical limitation. When redirecting to the list of … Read more
Auto scaling doesn’t make a lot of sense with ElasticSearch. Shard moving and re-allocation is not a light process, especially if you have a lot of data. It stresses IO and network, and can degrade the performance of ElasticSearch badly. (If you want to limit the effect you should throttle cluster recovery using settings like … Read more
kafka provides publish/subscribe messaging as a distributed commit log. Usually you install kafka on each host where you need to produce some data to be forwarded somewhere else and all those hosts will together form a cluster. The good thing here is that if for some reason network connectivity becomes unstable or goes down, your … Read more
This is doable using the update api. It does require that you define the id of each document, since the update api requires the id of the document to determine its presence. Given an index created with the following documents: PUT /cars/car/1 { “color”: “blue”, “brand”: “mercedes” } PUT /cars/car/2 { “color”: “blue”, “brand”: “toyota” … Read more
On Docker Toolbox If you are in docker toolbox try the docker client from terminal and then make the configs: docker-machine ssh sudo sysctl -w vm.max_map_count=262144 exit On Docker For Mac: The vm_max_map_count setting must be set within the xhyve virtual machine: $ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty If it asks you for a username and passwordLog in … Read more
By Default, the fields are case-insensitive because of the mapping elastic applied. Try below: PUT myindex/doc/1 { “name”:”TEST” } GET myindex/_mapping It should return : { “myindex”: { “mappings”: { “doc”: { “properties”: { “name”: { “type”: “text”, “fields”: { “keyword”: { “type”: “keyword”, “ignore_above”: 256 } } } } } } } } Now … Read more
This is because of a breaking change in ES 6.x: mapping types were removed (even if for backward compatibility you can still specify it in the path) thus de facto limiting the index to a single type. See here for more info.
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] ulimit -n 65536 or set nofile to 65536 in /etc/security/limits.conf max number of threads [1024] for user [xxx] is too low, increase to at least [2048] ulimit -u 2048 Or set the nproc value to 2048 or above in /etc/security/limits.conf … Read more