I was asked to make sure that we have a centralized way to keep all our logs at Viafoura a few days back.  What did I do?  I went to something I knew worked, something that has been tested for a long time and something that is widely used in today’s market.

Thank you syslog-ng by Balabit IT Security!  I have been using this logging daemon in my own projects, businesses and through the corporate world for years now and love the ease with which you do simple things like:

Name a source

source s_net {
    tcp(ip(127.0.0.1) port(1000) authentication(required) encrypt(allow));
};

Name a destination

destination d_syslog {
    file(“/var/log/syslog”);
};

Create some filters

filter f_syslog3 {
     not facility(auth, authpriv, mail) and not filter(f_debug);
};

Put it all together!

log {
    source(s_net);
    filter(f_syslog3);
    destination(d_syslog);
};

Usually at this point, I would have left it at that and called it a day.  There was a further requirement of my day’s work (this actually ended up taking a few days to solve completely)…  I had to ensure that all our devs could easily search through the beautiful logs in near realtime and find out why their code was breaking functionality, something that nearly never happens here, but we still have to build for it!

Who would come to the rescue now?  Well, Loggly of course! You see, the first time I used Loggly, they were still in their infancy, which meant that they didn’t have many of the features that I personally wanted (at my old company).  Nowadays, after listening to the gripes from their customers and doing their best to address them, they have a nicely polished product that serves the majority of our SysOps and Dev teams here at Viafoura.  What follows is a detailed guide/script* to help you do some neat things on Loggly (assuming that you’ve done your homework before the fact and actually implemented syslog-ng properly).

*Notes: This script can be reused (obviously), but works well in my environment.  It will not work out of the box and has a lot of assumptions to do with my network (as you can guess at Amazon) that your setup probably does not have.  There really are some neat conventions that I’ve followed to make all of this work, but, alas, can’t divulge them.

#!/bin/bash

################################################################################
#
# A script used to add myself as a particular device to loggly, so that I am
# able to send logs there.
#
# Author:   Victor Anjos
# Date:     March 5, 2013
# Version:  1.0
#
################################################################################
#

# some much needed variables
INPUT_STEM=https://viafoura.loggly.com/api/inputs/PLACEHOLDER/adddevice
INPUT1=<INPUT1>
INPUT2=<INPUT2>
INPUT3=<INPUT3>
INPUT4=<INPUT4>
INPUT5=<INPUT5>
INPUT6=<INPUT6>
INPUT7=<INPUT7>
INPUT8=<INPUT8>
INPUT9=<INPUT9>
CURLOPTIONS=” -X POST -u <YOURUSERNAME>:<YOURPASSWORD> -d name=DEVICENAME”

# stuff them into an array to be used later in loop
INPUT_ARRAY[0]=$INPUT1
INPUT_ARRAY[1]=$INPUT2
INPUT_ARRAY[2]=$INPUT3
INPUT_ARRAY[3]=$INPUT4
INPUT_ARRAY[4]=$INPUT5
INPUT_ARRAY[5]=$INPUT6
INPUT_ARRAY[6]=$INPUT7
INPUT_ARRAY[7]=$INPUT8
INPUT_ARRAY[8]=$INPUT_NGINXLOG

# might need these later
sec_grp=($(curl -s 169.254.169.254/2011-05-01/meta-data/security-groups)
av_zone=$(curl -s 169.254.169.254/2011-05-01/meta-data/placement/availability-zone)

# let’s get the environment
case $sec_grp in
*”Live”*)
ENVIRONMENT=live ;;
*”Staging”*)
ENVIRONMENT=staging ;;
*”Dev”*)
ENVIRONMENT=dev ;;
*)
ENVIRONMENT=unknown ;;
esac

# get the machine’s role
case $sec_grp in
ROLE1*)
role=role1
inputs=’1 3 7′ ;;
ROLE2*)
role=role2
inputs=’1 3 5 7′ ;;
ROLE3*)
role=role3
inputs=’1 3 5 7′ ;;
…
ROLEn*)
role=rolen
inputs=’1 3 5 6 7 8′ ;;
*)
role=NULL ;;
esac

# our machine’s hostname
HOSTNAME=$(echo ${sec_grp}-$av_zone)

# starting off the SPECIAL addition…
special_add=$(echo -e “#####\n# NGINX SPECIFIC LOGGLY STUFF\n#####\n”)

# at this point, if we’re a (CERTAIN TYPE OF) machine, let’s figure out which logs we have…
if [ $role == "rolex" ]; then
<do some special stuff for this special case>
fi

# we don’t really want to send DEV logs — let’s save a little money
if [ $ENVIRONMENT == "live" ]; then
# let’s add in the SPECIAL logs to track if it is not already there
if [ $(echo "$special_add" | wc -l) -ge 5 ]; then
# done this way so that tee doesn’t beat our other command and thus sets up and empty or half-filled file
(head -$(($(cat /etc/syslog-ng/syslog-ng.conf | wc -l) – 4)) /etc/syslog-ng/syslog-ng.conf; echo “$special_add”; tail -5 /etc/syslog-ng/syslog-ng.conf) | sudo tee ~/syslog-ng.conf
sudo cp ~/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf
fi

# add this machine to loggly
for level in $inputs; do
URL=$(echo $INPUT_STEM | sed -e “s/PLACEHOLDER/${INPUT_ARRAY[$level]}/”)
COPTS=$(echo $CURLOPTIONS | sed -e “s/DEVICENAME/$HOSTNAME/”)
curl $COPTS $URL
done
elif [ $ENVIRONMENT == "dev" -a $role == "rolex" ]; then # want to send stuff from our dev server
# add this machine to loggly
for level in $inputs; do
URL=$(echo $INPUT_STEM | sed -e “s/PLACEHOLDER/${INPUT_ARRAY[$level]}/”)
COPTS=$(echo $CURLOPTIONS | sed -e “s/DEVICENAME/$HOSTNAME/”)
curl $COPTS $URL
done
fi

In a few days, I may give you guys the extra little bits required for this to upstart properly, fix itself and remove old nodes.  That’s a good start though for anyone delving into Loggly for the first time!

As always, any suggestions can be left in our comment section below and I’ll try to incorporate/stay on top of things!

Have a great day and checkout Viafoura Presents Big Data Week in Toronto and Big Data Week itself!