Elasticsearch——Logstash安装

概述

Logstash 是开源的服务器端数据处理管道，能够同时从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的“存储库”中。

官方地址：https://www.elastic.co/cn/products/logstash

下载地址：[https://www.elastic.co/cn/downloads/logstash

yum 安装教程https://www.elastic.co/guide/en/logstash/7.4/installing-logstash.html#_yum

1
yum install logstash -y

启动服务

1
systemctl start logstash.service

测试数据集

电影数据

https://grouplens.org/datasets/movielens/

使用最小的数据集Small: 100,000 ratings and 3,600 tag applications applied to 9,000 movies by 600 users. Last updated 9/2018

1
wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
1
uzip ml-latest-small.zip
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
input {
  file {
    path => "数据路径目录地址/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}
1
2
cd /usr/share/logstash/bin
./logstash -f /data/es/movielens/logstash.conf

参考

https://blog.csdn.net/supercmd/article/details/91048042