LinuxDynasty

Since I do currently work for a streaming company, that would imply that we should have some type of monitoring for our RTSP streams . You will 1st need to get the openRTSP command.

• You will need to download the openRTSP command from www.live555.com/openRTSP/
• Or if you are running Zenoss on top of Debian you can just run a apt-get install livemedia-utils

Once you do that, all you have to do next is to download my script and have a valid server to point at and a path to test… Example below.. Download

./check_rtsp.py -d remote_server -p /iphone/2012
OK /iphone/2012, test completed successfull against remote_server |status=0

or with stats…

./check_rtsp.py -d remote_server -p /iphone/2012 -s
OK /iphone/2012, test completed successfull against remote_server |status=0 num_packets_received=16 num_packets_lost=0 elapsed_measurement_time=3.000073 kBytes_received_total=16.309000
measurement_sampling_interval_ms=1000 kbits_per_second_min=29.245485 kbits_per_second_ave=43.489608
kbits_per_second_max=51.737449 packet_loss_percentage_min=0.000000 packet_loss_percentage_ave=0.000000
packet_loss_percentage_max=0.000000 inter_packet_gap_ms_min=0.018000 inter_packet_gap_ms_ave=161.139313
inter_packet_gap_ms_max=901.439000 subsession=video/H264 num_packets_received=61 num_packets_lost=0
elapsed_measurement_time=3.000073 kBytes_received_total=27.630000 measurement_sampling_interval_ms=1000
kbits_per_second_min=0.000000 kbits_per_second_ave=73.678207 kbits_per_second_max=122.923442
packet_loss_percentage_min=0.000000 packet_loss_percentage_ave=0.000000 packet_loss_percentage_max=0.000000
inter_packet_gap_ms_min=0.009000 inter_packet_gap_ms_ave=24.794672 inter_packet_gap_ms_max=528.923000

So you can trend the different stats that openRTSP provides in Zenoss..

Hey guys, here is another Python script by me . The reason for this script, is to give you the capabilities to monitor and graph Bind 9 stats. This script will be executed through SNMP.. We will be using the UCD-MIB 1.3.6.1.4.1.2021.8.1

The stats are from named.stats... Here is an example of named.stats..
cat /var/cache/bind/named.stats
+++ Statistics Dump +++ (1325089261)
++ Incoming Requests ++
            83318718 QUERY
++ Incoming Queries ++
            54293282 A
                1929 NS
                3186 CNAME
                  27 SOA
            13645272 PTR
                7921 MX
                 781 TXT
            15224426 AAAA
                 200 SRV
                   1 NAPTR
                1271 A6
                  95 DS
                   2 NSEC
                  66 SPF
              140257 ANY
                   2 Others

You can download the script here .. Download

Continue reading “How To Monitor and get Bind 9 stats using Zenoss” »

We needed a way to monitor all of our NodeJS instances in Zenoss and not just a simple TCP connection. Since we were already using the Cluster plugin for NodeJS ‘learnboost.github.com/cluster/’ and we are using the stats plugin for Cluster ‘learnboost.github.com/cluster/docs/stats.html’. We decided to write a quick python script to connect to the stats plugin and parse that data to represent it to Zenoss.

You have 2 options… Run this script by itself using Zenoss to connect to the host or if you are using a socket connection or connecting to localhost only then use SNMP.

Download Here .. Download

• ( IN SNMP ) Options are modifiable ‘exec CheckClusterLiveStats /usr/local/bin/check_clusterlive_stats.py -t tcp -c localhost:8888‘
• Or directly in Zenoss ‘/usr/local/bin/check_clusterlive_stats.py -t tcp -c remotehost:8888′

This script can be used to connect to a TCP socket or to a local file socket. Example output from Zenoss Test Command…

Using SNMP Preparing Command...
Executing command /usr/local/zenoss/libexec/check_snmp -H 127.0.0.1 -o .1.3.6.1.4.1.8072.1.3.2.3.1.1.21.67.104.101.99.107.67.108.117.115.116.101.114.76.105.118.101.83.116.97.116.115 -C readonly -R OK against 127.0.0.1
SNMP OK - "Deaths Has not increased, Deaths count is 0 NodeJS cluster OK |restarts=0 workers=4 deaths=0 connections_total=193463 connections_active=0" |

Using Command only...

Preparing Command...
Executing command /usr/local/zenoss/libexec/check_clusterlive_stats.py -t tcp -c remotehost:8888
"Deaths Has not increased, Deaths count is 0 NodeJS cluster OK |restarts=0 workers=4 deaths=0 connections_total=193463 connections_active=0" |

The command above you can run with out snmp. The reason we use snmp because we only allow the stats plugin to run on the localhost interface. So we use snmp to execute the script and in return, it returns the results.

Here is the help output..
check_clusterlive_stats.py -h
Usage: check_clusterlive_stats.py arg --contype=socket|tcp --connection=pathto socket|host:port

Options:
  -h, --help            show this help message and exit
  -t CONTYPE, --contype=CONTYPE
                        socket or tcp
  -c CONNECTION, --connection=CONNECTION
                        /tmp/cluster-repl.sock or localhost:8888
  -w TIMEOUT, --timeout=TIMEOUT
                        int of how long you want this script to wait until it
                        times out

 

We needed a way to monitor a few different type of HTTP Status Codes, specifically… ( 400, 404, 500, 503, 504, 200 ). This list is modifiable within the script. Though I’m thinking of turning this list into an option, since everyone might not one to gather the count just for those mentioned above.

So In order to get this data, you will need to use SNMP and install logtail. To make this script work you will need to have an snmpd exec statement in the snmpd.conf file and a crontab entry for the script to run as often as you like.

This script has been tested, using Nginx and Ruby On Rails.
You can download the script here.. Download

• On cron, remember this is modifiable ‘* * * * * /usr/local/bin/get_http_codes.py -d “/var/log/nginx” -f “www-access.log” >/tmp/errorcount.txt’
• In snmpd.conf without quotes ‘exec GetHttpStats /bin/cat /tmp/errorcount.txt‘

In Zenoss you will have to use the check_snmp nagios command or the inherent snmp check from zenoss. The oid you need to use is the UCDavis OID. For instance I’m using 2 exec statements in snmpd.conf, so the OID I am using for this check is this one ‘.1.3.6.1.4.1.2021.8.1.101.2‘ The command I’m using in zenoss is like this ‘check_snmp -H ${dev/manageIp} -C readonly -P 2c -o .1.3.6.1.4.1.2021.8.1.101.2‘

The current options for get_http_codes.py is ‘-d‘ which is the directory where the log file is located and ‘-f ‘ the name of the logfile. This script uses logtail so that I can always get the difference from the last time I scanned the log file.

The purpose of this script is so that you can graph or create a threshold of how many of the below Http Error Codes you are getting between every check. For instance we run this check every 30 seconds in Cron and in Zenoss and we divide the results by 30 so we can get how many of the error codes are happening per second.
Example of the output of the script is ‘SNMP OK – “Nginx Codes OK|count200=181 count400=1 count404=0 count500=0 count503=0 count504=0″ | ‘

If you have any feature request or have any questions, please leave a comment. Thank you

Recently I had to aggregate the amount of 200, 400, 404, 500, 503,  and 504 HTTP 1.1 Codes from all of our Nginx and Ruby On Rails systems. So I decided to write a quick plugin for Zenoss that all you need to do is pass the devices or Device Class you want to aggregate the data from and the DataPoint name.

Also this makes it simple to use the Zenoss Thresholds and create Aggregate Reports based on the datapoints you use with out having to do some fancy python coding… Enjoy

 

Example..

python AllenZenossAggregate.py -h
Usage: AllenZenossAggregate.py -d "nginx-1" -d "nginx-2" -p "nginx_codes_count200"
OK Aggregate for devices  nginx-1 nginx-2 is 34|aggregate=34
     AllenZenossAggregate.py -o "/Server/Linux/Nginx" -p "nginx_codes_count200"
OK Aggregate for /Server/Linux/Nginx class is 22|aggregate=22
 Options:
  -h, --help            show this help message and exit
  -d DEVICE, --device=DEVICE
                        The device you want to grab the datapoints from.
  -o ORGANIZER, --organizer=ORGANIZER
                        The Class you want to get your list of devices from.
  -p DPOINTS, --dpoints=DPOINTS
                        Name of DataPoint nginx_codes_count200

Download

 

I have not been doing to many interesting projects as of the past year. This is the reason for my lack of updates. I am now starting to do some interesting stuff, so let the updates roll.

RabbitMQ was recently deployed in the company I currently work for. At the last minute ( as always ) they came to me and ask me to please add RabbitMQ monitoring to Zenoss. They said here is the url and now please monitor for a few stats ( using the RabbitMQ Status Plugin ). So I said to myself, I could easily just write a quick shell script to get the 3 stats that they needed and add them into Zenoss. After thinking about it…. In the near future they might ask for more than those 3 stats. So I decided to write a quick python script ( Zenoss Compatible ) to get all the stats from that status page and input them into Zenoss…

Prerequisites

1. Zenoss 2.5 and above ( I have not tested on 3.+ or <2 .4 )
2. lxml Python module    ”easy_install lxml” as the zenoss user
3. “Zenoss_Template_Manager.py” Optional, Only needed if you do not want to add all the datapoints manually… I DON’T!!!!
4. RabbitMQ Installed
5. RabbitMQ Status Plugin from www.lshift.net/blog/2009/11/30/introducing-rabbitmq-status-plugin
6. “check_rabbitmq.py“

Zenoss Template Manager = Download

CheckRabbitMQ.py = Download

Once you have all the above, we are ready to go..

1. copy both Python Scripts above into the /opt/zenoss/libexec/ folder  ( If you are using RedHat/CentOS ) and make sure they are executable.

As the Zenoss user run the script ….
/opt/zenoss/libexec/check_rabbitmq.py -u ‘rabbitmq-server:55672′ -a ‘mon-user mon-passwd’ |sed -re “s/^OK|/ /g” |sed -re “s/([A-Za-z0-9_.]+*)?=[0-9]+/-p “1,G”/g” |xargs /opt/zenoss/libexec/Zenoss_Template_Manager.py -o “/Devices/Server/Linux/RabbitMQ” -c ‘/opt/zenoss/libexec/check_rabbitmq.py -u “rabbitmq-server:55672″ -a “mon-user mon-passwd”‘ –template=RabbitMQ –dsource=RabbitMQStats -V $1

If you need to know how to use the Zenoss_Template_Manager.y script, check here www.linuxdynasty.org/howto-add-multiple-datapoints-to-zenoss-using-the-zenoss-api.html

So let me explain what the sed statements above are doing…

• “sed -re “s/^OK|/ /g”" This sed statement is removing the OK| from the beginning of the line
• “sed -re “s/([A-Za-z0-9_.]+*)?=[0-9]+/-p “1,G”/g”
  This 1st part of the sed statement is matching any letter,number,underscore, and period, any number of times until it reaches the equal “=”  ”[A-Za-z0-9_.]+*)?=”
  The  2nd part of this statement is going to match the “=” and any number of integers after it. “[0-9]+”
  Now we need to make the substitution…. So we are going to substitute, every match with a -p,  then a space and then the 1st group match in escaped quotes, then a comma and G for GAUGE.

The 2 sed statements above will do that for every match it finds. If you were to add each datapoint by hand, it would look like this….
/opt/zenoss/libexec/Zenoss_Template_Manager.py -o “/Devices/Server/Linux//RabbitMQ” -c ‘/opt/zenoss/libexec/check_rabbitmq.py -u “rabbitmq-server:55672″ -a “mon-user mon-passwd”‘ –template=RabbitMQ –dsource=RabbitMQStats -V -p “queue.conversion.event.tracking_msg_unack,G” -p “connections,G” -p “erlang_processes_used,G” -p “erlang_processes_avail,G” -p “file_descriptors_used,G” -p “file_descriptors_avai,G” -p “binary_memory,G” -p “memory_used,G” -p “memory_avail,G”

I hope the above scripts will save someone time and frustration……

 

We needed to integrate the Splunk Alerts into Zenoss, because even though Splunk can indeed send out alerts. Splunk does not have any clue about what an “Escalation Process” is. With Zenoss you can create an “Escalation Process”.

I have 2 ways to send events to Zenoss from Splunk..

1. Write a Script that uses the snmptrap command.
2. Write a Script that uses the Zenoss zensendevent command.

I decided to go with the Zenoss zensendevent command ( Which is a python script with no external dependencies, which can be copied from the Zenoss Server at $ZENHOME/bin/zensendevent ).

Now it’s time to get the ball rolling..

1. On the Splunk Server I copied the zensendevent script from the Zenoss Server to Splunk on /opt/splunk/bin/scripts/zensendevent.
2. I then created a shell script called Splunk2Zenoss.sh. ( This script takes the Saved Splunk Search and passes it over to Zenoss )This script will also be located in /opt/splunk/bin/scripts/Splunk2Zenoss.sh
3. You will then need to modify the options in the script. (For instance the severity of the alert, the zenoss server, the event mapping, event key, login and passwd )
4. I then created the saved search in Splunk and make sure to check the Trigger Shell Script option. ( Make sure to put the script name in here )

Continue reading “HowTo Send Splunk Alerts To Zenoss, And make them Look Like Splunk” »

I first made the Download Section for members only, just cause I wanted an idea on who is downloading the software I have written. Now, I no longer care who downloads it, as long as it helps out the people who did download it. That is all that matters. Though the forums section, does require you to be a member. I also recognize, most people are members of enough websites, that becoming a member of another one, can be over bearing.

I’ve come to realize, that CIM is the new SNMP, but on steroids. Most new SAN, NAS, Network, and Operating Systems now support CIM/WBEM. To me it is easier to gather statistics and information through CIM, then it is through SNMP. In this article I am going to give you a script that will allow you to query the 3par for Disk IO stats. You will be able to grab Disk IO stats on a per Volume, per Port, or per Disk basis.You will also be able to search for a Volume, Port, or Disk, instead of just dumping all the Volumes, Ports, or Disk.

Before you download this script, you will need to download pywbem from sourceforge and install it. .

get3ParIOstats.py == Download

All the data that you get from the script, must be saved as a COUNTER and not a GAUGE.

Here is an example of searching for statistics by DISK for DISK 2:6:1..

python get3parIOstats.py -u "3par" -a 'login passwd' -s '2:6:1' --diskOK|2:6:1_ReadIOs=67236384 2:6:1_WriteIOs=28457131 2:6:1_TotalIOs=95693515

Continue reading “HowTo get 3par disk IO stats into Zenoss” »