Searching IRC Activity with Logstash / Elasticsearch / Kibana

Edit: This post is pretty old and Elasticsearch/Logstash/Kibana have evolved a lot since it was written.

To make sure I understood how to find data using Kibana3, I started collecting input from IRC.

kibana-irc

I have a ZNC bouncer set up on my network. 192.168.1.10

http://wiki.znc.in/ZNC

I have it set to Keep Buffer, Prepend Timestamps.
Timestamp Format:
[text]
[%Y-%m-%d %H:%M:%S]
[/text]

I used the IRC input for Logstash to have something to search with. Since I’m obsessing over this, might as well make a dashboard showing what I really want to see!
[text]
channel: "#logstash" OR channel: "#elasticsearch" OR message: "elasticsearch" OR message: "logstash" OR message: "kibana" OR message: "splunk" OR message: "syslog" OR message: "graylog*" OR message: "nxlog"
[/text]

kibana-irc-search

Below is my logstash configuration.

logstash-irc.conf
[text]
input {
irc {
channels => "#chat"
host => "192.168.1.10"
nick => "ragingcomputer"
password => "username:password"
port => 6667
secure => true
user => "ragingcomputer"
}

irc {
channels => [ "##boxee-hacking", "#archlinux-arm", "#arduino", "#avr", "#boxeeplus", "#chumby", "#elasticsearch", "#ffmpeg", "#launchpad", "#linuxcnc", "#linuxmce", "#lockresearch", "#logstash", "#mythtv-users", "#raspbian", "#sickbeard", "#sparkfun", "#ubuntu-mythtv", "#videolan", "#archlinux", "#ubuntu", "#debian", "#perl", "#znc", "##windows", "#pfsense", "#owncloud", "#redis", "#zabbix", "#nagios", "#reddit-sysadmin", "#sensu", "#graylog2", "#plex", "#couchpotato", "#ossec", "#graphite", "#hadoop", "#icinga", "#pauldotcom" ]
host => "192.168.1.10"
nick => "ragingcomputer"
password => "username:password"
port => 6667
secure => true
user => "ragingcomputer"
}

irc {
channels => "#twitlive"
host => "192.168.1.10"
nick => "ragingcomputer"
password => "username:password"
port => 6667
secure => true
user => "ragingcomputer"
}

}

filter {
if [message] =~ /ACTIONs[201[0-9]-[0-9][0-9]-[0-9][0-9]s[0-9][0-9]:[0-9][0-9]:[0-9][0-9]]/ {
drop { }
}
if [message] =~ /^[201[0-9]-[0-9][0-9]-[0-9][0-9]s[0-9][0-9]:[0-9][0-9]:[0-9][0-9]]/ {
drop { }
}
if [nick] =~ /^*/ {
drop { }
}
}

output {

elasticsearch {
host => "127.0.0.1"
cluster => "logcatcher"
}
}
[/text]

2 thoughts on “Searching IRC Activity with Logstash / Elasticsearch / Kibana

  1. Hey,

    I have 2 questions:

    1) Are you connecting to ZNC instead of parsing logs ?
    2) Why do you have multiple irc setups using same ZNC server and same user just with a different channel list

    Thanks

    Like

  2. I am connecting to ZNC. I have multiple users connecting to multiple server networks.
    The Username for ZNC doesn’t seem to matter. The password is how you identify and authenticate to ZNC.
    The password is more than just the password. It’s username:password

    I was connecting directly to the IRC bouncer instead of watching logs because my IRC bouncer is on a different machine. I’m not sure real-time log searching is as important as I first thought so I’ll probably start rsync the log files to my logstash machine and watching log files so I can spin down that VM without losing anything.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: