{"id":411,"date":"2020-05-31T12:44:09","date_gmt":"2020-05-31T12:44:09","guid":{"rendered":"https:\/\/carson.fenimorefamily.com\/?p=411"},"modified":"2020-06-02T04:23:41","modified_gmt":"2020-06-02T04:23:41","slug":"adding-in-custom-indices-to-elastiflow","status":"publish","type":"post","link":"https:\/\/carson.fenimorefamily.com\/?p=411","title":{"rendered":"Adding in Custom Indices to Elastiflow"},"content":{"rendered":"\n

Let’s say you have an elastiflow<\/a> docker instance set up. This stack pushes all flow info into an index named “elastiflow-<version>-<Year>.<Month>.<Day>”. What if you wanted to use the same ELK stack for both elastiflow AND other stuff? <\/p>\n\n\n\n

This is possible, of course!<\/p>\n\n\n\n

Clone the elastiflow git repo<\/a><\/p>\n\n\n\n

Cd into the repo<\/p>\n\n\n\n

Add a new input filter to logstash\/elastiflow\/conf.d\/10_input_syslog.conf . For example to bring in syslog:<\/p>\n\n\n\n

input {\n  udp {\n    host => \"0.0.0.0\"\n    port => 10514\n    codec => \"json\"\n    type => \"rsyslog\"\n    tags => [\"rsyslog\"]\n  }\n}\n\nfilter { }<\/code><\/pre>\n\n\n\n
Modify logstash\/elastiflow\/conf.d\/30_output_10_single.logstash.conf<\/p>\n\n\n\n
\noutput {\n  if \"rsyslog\" in [tags]  {\n    elasticsearch {\n      user => \"${ELASTIFLOW_ES_USER:elastic}\"\n      password => \"${ELASTIFLOW_ES_PASSWD:changeme}\"\n      hosts => [ \"172.10.4.1:9200\" ]\n      index => \"logstash-%{+YYYY.MM.dd}\"\n      template => \"${ELASTIFLOW_TEMPLATE_PATH:\/etc\/logstash\/elastiflow\/templates}\/logstash.template.json\"\n      template_name => \"logstash-1.0.0\"\n    \n    }\n  } else {\n    elasticsearch {\n      id => \"output_elasticsearch_single\"\n      hosts => [ \"${ELASTIFLOW_ES_HOST:127.0.0.1:9200}\" ]\n      ssl => \"${ELASTIFLOW_ES_SSL_ENABLE:false}\"\n      ssl_certificate_verification => \"${ELASTIFLOW_ES_SSL_VERIFY:false}\"\n      # If ssl_certificate_verification is true, uncomment cacert and set the path to the certificate.\n      #cacert => \"\/PATH\/TO\/CERT\"\n      user => \"${ELASTIFLOW_ES_USER:elastic}\"\n      password => \"${ELASTIFLOW_ES_PASSWD:changeme}\"\n      index => \"elastiflow-3.5.3-%{+YYYY.MM.dd}\"\n      template => \"${ELASTIFLOW_TEMPLATE_PATH:\/etc\/logstash\/elastiflow\/templates}\/elastiflow.template.json\"\n      template_name => \"elastiflow-3.5.3\"\n      template_overwrite => \"true\"\n    }\n  } \n} <\/code><\/pre>\n\n\n\n
Rebuild the image:<\/p>\n\n\n\n
docker build --tag logstash-elastiflow-custom:1.0 .<\/code><\/pre>\n\n\n\n
Now bring up your stack, e.g. “docker-compose up -d”<\/p>\n\n\n\n
Now let’s test it.  We can generate a new syslog message by, say, logging into the syslog server.  If we do this the server shows the following message:<\/p>\n\n\n\n
Mar 31 08:37:37 zoobie-2-1 sshd[2625]: Accepted publickey for magplus from 172.10.4.32 port 61811 ssh2: RSA SHA256:2dui2biubddjwbdjbd\n<\/code><\/pre>\n\n\n\n
If we go to kibana -> Management and create an index, we should see a new logstash index.  Add it to kibana.  Then view the index in the discover view.  It should look like this:<\/p>\n\n\n\n
node.hostname:elk.myhomenet.sys node.ipaddr:172.10.4.1 event.type:rsyslog event.host:syslogserver.myhomenet.sys @version:3.5.3 facility:auth @timestamp:Mar 31, 2020 @ 08:24:52.000 sysloghost:zoobie-2-1 severity:info programname:sshd procid:2575 logstash_host:syslogserver.myhomenet.sys tags:rsyslog message: Accepted publickey for magplus from 172.10.4.32 port 61736 ssh2: RSA SHA256:2dui2biubddjwbdjbd _id:IJ6xanIBxE6Ab_zHIO3i _type:_doc _index:logstash-2020.03.31 _score:0<\/code><\/pre>\n\n\n\n
And there you have it!<\/p>\n\n\n\n
NOTE: This example does not cover setting up syslog forwarding, which is required to get syslog into logstash.  For a good example of this go to this Digital Ocean tutorial on syslog and logstash<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Let’s say you have an elastiflow docker instance set up. This stack pushes all flow info into an index named “elastiflow-<version>-<Year>.<Month>.<Day>”. What if you wanted to use the same ELK stack for both elastiflow AND other stuff? This is possible, of course! Clone the elastiflow git repo Cd into the repo Add a new input … Continue reading Adding in Custom Indices to Elastiflow<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[],"class_list":["post-411","post","type-post","status-publish","format-standard","hentry","category-databases"],"_links":{"self":[{"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=\/wp\/v2\/posts\/411","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=411"}],"version-history":[{"count":6,"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=\/wp\/v2\/posts\/411\/revisions"}],"predecessor-version":[{"id":421,"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=\/wp\/v2\/posts\/411\/revisions\/421"}],"wp:attachment":[{"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=411"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=411"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/carson.fenimorefamily.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=411"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}