Curator를 사용해보자(인덱스 관리)

아래처럼..log가 점점 쌓아져간다~엘라스틱서치에서 인덱스가 늘어나고 있다는 뜻이다.

그래서 아래처럼 줄여보고자 한다. 어떻게?

Curator를 사용해보자!

https://github.com/elastic/curator

Curator?

indices들을 관리해준다!

시작하기!

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/getting-started.html

Curator Setting

sudo pip install elasticsearch-curator

Collecting elasticsearch-curator

/usr/lib/python2.6/site-packages/pip-7.1.2-py2.6.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

  InsecurePlatformWarning

  Downloading elasticsearch_curator-3.4.0-py2.py3-none-any.whl (43kB)

    100% |████████████████████████████████| 45kB 832kB/s

Collecting click>=3.3 (from elasticsearch-curator)

  Downloading click-6.2-py2.py3-none-any.whl (70kB)

    100% |████████████████████████████████| 73kB 5.6MB/s

Collecting elasticsearch<2.1.0,>=1.8.0 (from elasticsearch-curator)

  Downloading elasticsearch-2.0.0-py2.py3-none-any.whl (58kB)

    100% |████████████████████████████████| 61kB 741kB/s

Collecting urllib3<2.0,>=1.8 (from elasticsearch<2.1.0,>=1.8.0->elasticsearch-curator)

  Downloading urllib3-1.13-py2.py3-none-any.whl (86kB)

    100% |████████████████████████████████| 90kB 4.7MB/s

Installing collected packages: click, urllib3, elasticsearch, elasticsearch-curator

Successfully installed click-6.2 elasticsearch-2.0.0 elasticsearch-curator-3.4.0 urllib3-1.13

/usr/lib/python2.6/site-packages/pip-7.1.2-py2.6.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

  InsecurePlatformWarning

Curator Upgrade

pip install -U elasticsearch-curator

Requirement already up-to-date: elasticsearch-curator in /usr/lib/python2.6/site-packages

Requirement already up-to-date: click>=3.3 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)

Requirement already up-to-date: elasticsearch<2.1.0,>=1.8.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)

Requirement already up-to-date: urllib3<2.0,>=1.8 in /usr/lib/python2.6/site-packages (from elasticsearch<2.1.0,>=1.8.0->elasticsearch-curator)

Curator --help

curator --help

Usage: curator [OPTIONS] COMMAND [ARGS]...

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:

  --host TEXT         Elasticsearch host.

  --url_prefix TEXT   Elasticsearch http url prefix.

  --port INTEGER      Elasticsearch port.

  --use_ssl           Connect to Elasticsearch through SSL.

  --certificate TEXT  Path to certificate to use for SSL validation.

                      (OPTIONAL)

  --ssl-no-validate   Do not validate SSL certificate

  --http_auth TEXT    Use Basic Authentication ex: user:pass

  --timeout INTEGER   Connection timeout in seconds.

  --master-only       Only operate on elected master node.

  --dry-run           Do not perform any changes.

  --debug             Debug mode

  --loglevel TEXT     Log level

  --logfile TEXT      log file

  --logformat TEXT    Log output format [default|logstash].

  --quiet             Suppress command-line output.

  --version           Show the version and exit.

  --help              Show this message and exit.

Commands:

  alias       Index Aliasing

  allocation  Index Allocation

  bloom       Disable bloom filter cache

  close       Close indices

  delete      Delete indices or snapshots

  open        Open indices

  optimize    Optimize Indices

  replicas    Replica Count Per-shard

  seal        Seal indices (Synced flush: ES 1.6.0+ only)

  show        Show indices or snapshots

  snapshot    Take snapshots of indices (Backup)

Curator Command

1) 색인 리스트 보기

curator --host 엘라스틱서치ip show indices --timestring '%Y.%m.%d'

2) 색인 삭제

curator --host 엘라스틱서치ip delete indices --older-than 30 --time-unit days --timestring '%Y.%m.%d' --time-unit days --prefix logstash

2015-12-15 13:59:15,246 INFO      Job starting: delete indices

2015-12-15 13:59:15,265 INFO      Pruning Kibana-related indices to prevent accidental deletion.

2015-12-15 13:59:15,265 INFO      Action delete will be performed on the following indices: [u'logstash-2015.09.02', u'logstash-2015.09.03', u'logstash-2015.09.04', u'logstash-2015.09.05', u'logstash-2015.09.06', u'logstash-2015.09.07', u'logstash-2015.09.08', u'logstash-2015.09.09', u'logstash-2015.09.10', u'logstash-2015.09.11', u'logstash-2015.09.12', u'logstash-2015.09.13', u'logstash-2015.09.14', u'logstash-2015.09.15', u'logstash-2015.09.16', u'logstash-2015.09.17', u'logstash-2015.09.18', u'logstash-2015.09.19', u'logstash-2015.09.20', u'logstash-2015.09.21', u'logstash-2015.09.22', u'logstash-2015.09.23', u'logstash-2015.09.24', u'logstash-2015.09.25', u'logstash-2015.09.26', u'logstash-2015.09.27', u'logstash-2015.09.28', u'logstash-2015.09.29', u'logstash-2015.09.30', u'logstash-2015.10.01', u'logstash-2015.10.02', u'logstash-2015.10.03', u'logstash-2015.10.04', u'logstash-2015.10.05', u'logstash-2015.10.06', u'logstash-2015.10.07', u'logstash-2015.10.08', u'logstash-2015.10.09', u'logstash-2015.10.10', u'logstash-2015.10.11', u'logstash-2015.10.12', u'logstash-2015.10.13', u'logstash-2015.10.14', u'logstash-2015.10.15', u'logstash-2015.10.16', u'logstash-2015.10.17', u'logstash-2015.10.18', u'logstash-2015.10.19', u'logstash-2015.10.20', u'logstash-2015.10.21', u'logstash-2015.10.22', u'logstash-2015.10.23', u'logstash-2015.10.24', u'logstash-2015.10.25', u'logstash-2015.10.26', u'logstash-2015.10.27', u'logstash-2015.10.28', u'logstash-2015.10.29', u'logstash-2015.10.30', u'logstash-2015.10.31', u'logstash-2015.11.01', u'logstash-2015.11.02', u'logstash-2015.11.03', u'logstash-2015.11.04', u'logstash-2015.11.05', u'logstash-2015.11.06', u'logstash-2015.11.07', u'logstash-2015.11.08', u'logstash-2015.11.09', u'logstash-2015.11.10', u'logstash-2015.11.11', u'logstash-2015.11.12', u'logstash-2015.11.13', u'logstash-2015.11.14', u'logstash-2015.11.15']

2015-12-15 13:59:15,268 INFO      Deleting indices as a batch operation:

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.02

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.03

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.04

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.05

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.06

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.07

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.08

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.09

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.10

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.11

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.12

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.13

2015-12-15 13:59:15,268 INFO      ---deleting index logstash-2015.09.14

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.15

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.16

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.17

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.18

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.19

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.20

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.21

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.22

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.23

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.24

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.25

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.26

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.27

2015-12-15 13:59:15,269 INFO      ---deleting index logstash-2015.09.28

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.09.29

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.09.30

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.01

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.02

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.03

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.04

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.05

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.06

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.07

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.08

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.09

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.10

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.11

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.12

2015-12-15 13:59:15,270 INFO      ---deleting index logstash-2015.10.13

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.14

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.15

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.16

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.17

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.18

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.19

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.20

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.21

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.22

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.23

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.24

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.25

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.26

2015-12-15 13:59:15,271 INFO      ---deleting index logstash-2015.10.27

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.10.28

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.10.29

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.10.30

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.10.31

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.01

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.02

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.03

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.04

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.05

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.06

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.07

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.08

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.09

2015-12-15 13:59:15,272 INFO      ---deleting index logstash-2015.11.10

2015-12-15 13:59:15,273 INFO      ---deleting index logstash-2015.11.11

2015-12-15 13:59:15,273 INFO      ---deleting index logstash-2015.11.12

2015-12-15 13:59:15,273 INFO      ---deleting index logstash-2015.11.13

2015-12-15 13:59:15,273 INFO      ---deleting index logstash-2015.11.14

2015-12-15 13:59:15,273 INFO      ---deleting index logstash-2015.11.15

   - 끝 -