[logstash] 利用JDBC存取 mysql

9 月 15

joechenELK, GNU/Linux No Comments

REF: https://www.zghhome.cn/?p=350

Access data from mySQL and output to ES
test1
test2
test3
test4
test5
test6
test7

############# input ##############
input {
    jdbc {
        jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_connection_string => "jdbc:mysql://192.168.7.79:3306/note"
        jdbc_user => "dbuser"
        jdbc_password => "password"
        #parameters => {""}
        schedule => "* * * * *"
        statement => "select ID,post_date,post_title,ping_status from note_posts where id > :sql_last_value order by id"
        #statement => "select ID,post_date,post_title,ping_status from note_posts"
        # setting for last run
        clean_run => false
        record_last_run => true
        use_column_value => true
        tracking_column => id
        last_run_metadata_path => "/usr/share/logstash/logstash_mydb-note_last_run"
        type => "mydb-note"
    }
}

###################################
############# filter ##############
###################################

############# output ##############
output {
#    stdout { codec => "rubydebug" }
if [type] == "mydb-note" {
      if !("_grokparsefailure" in [tags]) {
        elasticsearch {
        hosts => [ "127.0.0.1:9200" ]
        index => [ "logstash-mydb-note" ]
        }
      } else {
        elasticsearch {
        hosts => [ "127.0.0.1:9200" ]
        index => [ "logstash-failure-mydb-note" ]
        } # ELK
      } # else
    } # if
}

[ES][Elasticsearch] 2.x v.s 5.x

8 月 29

joechenELK No Comments

http://wzktravel.github.io/2017/01/18/elasticsearch-upgrade-to-5-1-2-from-2-3-5/

[ELK] ELK stack 中文教學視頻(CHINA)

7 月 28

joechenELK No Comments

ELK stack 中文教學視頻(CHINA)

[Graylog2][timezone]

3 月 01

joechenELK No Comments

graylog 設定服務的時區(search畫面, 預設是UTC)
完成設定後, 要做 reconfigure 

$ sudo graylog-ctl set-timezone Asia/Taipei
$ sudo graylog-ctl reconfigure
$ sudo reboot

[Graylog2][Connection refused][port 4001]

2 月 28

joechenELK, 未分類 No Comments

REF: https://github.com/Graylog2/graylog2-images/issues/59

一直無法正常啟用 graylog2, 後來在系統日誌裡看到 port 4001 拒絕連線

mariussturm commented on 27 May 2015
Looks like the Etcd database got corrupted for some reason. Maybe the machine run out of diskspace or was hard resetted. If this is a single node installation you could try to delete /var/opt/graylog/data/etcd and afterwards run sudo graylog-ctl reconfigure

[ELK] GeoIP 注意事項

11 月 16

joechenELK, 未分類 No Comments

CentOS 套件庫的版本是2013年的, 會有些公用IP查不出來, 建議自行下載
https://dev.maxmind.com/geoip/legacy/geolite/#Downloads

Logstash 如果名稱不是 logstash-* 開頭的 index, GeoIP的 “geo_point” 會無法啟用
http://blog.csdn.net/yanggd1987/article/details/50469113

[Graylog] 升級 OVA 版本的 graylog2

10 月 12

joechenELK, 未分類 No Comments

REF: http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html

 

Upgrade Graylog

Warning

The Graylog omnibus package does not support unattended upgrading from Graylog 1.x to Graylog 2.1.x!

Always perform a full backup or snapshot of the appliance before proceeding. Only upgrade if the release notes say the next version is a drop-in replacement. Choose the Graylog version you want to install from the list of Omnibus packages . graylog_latest.deb always links to the newest version:

$ wget https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
$ sudo graylog-ctl stop
$ sudo dpkg -G -i graylog_latest.deb
$ sudo graylog-ctl reconfigure
$ sudo reboot

如果遇到 reconfigure 失敗或不成功的狀況, 可參考

https://github.com/Graylog2/graylog2-images/issues/59
delete /var/opt/graylog/data/etcd
run sudo graylog-ctl reconfigure

 

Older Entries