graphite和collectd以及自定义脚本

先说graphite,这个比较复杂点,是python写的。

首先我们了解一下graphite具体是如何工作的。

  1. carbon – 一个Twisted守护进程,监听时间序列数据
  2. whisper – 一个简单的数据库库,用来存储时间序列数据,在设计上类似于RRD
  3. graphite webapp – Django webapp,使用 Cairo来根据需要呈现图形

Graphite-High-Level-458x600

官方的安装文档:http://graphite.wikidot.com/installation

但是这里缺少了一些依赖包。 最好是python2.6 或者以上,虽然官方说2.4也可以,但是出现各种问题最后。

安装graphite

先导入epel源,我的centos6.4版本,然后yum安装下列包

yum install -y bitmap bitmap-fonts-compat Django django-tagging fontconfig cairo python-devel python-memcached python-twisted  pycairo mod_python python-ldap python-simplejson memcached python-zope-interface mod_wsgi python-sqlite2 python-rrdtool git python-pip

然后git下载如下几个项目

git clone https://github.com/graphite-project/graphite-web.git
git clone https://github.com/graphite-project/carbon.git
git clone https://github.com/graphite-project/whisper.git
git clone git://github.com/graphite-project/ceres

安装上述几个项目

cd graphite-web
python setup.py install
cd ..

cd carbon
python setup.py install
cd ..

cd whisper
python setup.py install
cd ..

cd ceres
python setup.py install

默认安装路径为/opt/graphite

安装完成后可以是用如下命令检查是否安装成功,不成功的根据提示来安装对应的包

python check-dependencies.py

其中特别注意pyparsing的版本必须是1.5.7,因为更高的版本是针对Python3的

pip-python install pyparsing==1.5.7

我的python插件和版本如下

Conch==8.2.0
Django==1.4.5
PyXML==0.8.4
SOAPpy==0.11.6
Twisted==13.0.0
Twisted-Core==8.2.0
Twisted-Lore==8.2.0
Twisted-Mail==8.2.0
Twisted-Names==8.2.0
Twisted-News==8.2.0
Twisted-Runner==8.2.0
Twisted-Web==8.2.0
Twisted-Words==8.2.0
ceres==0.10.0
distribute==0.6.46
django-tagging==0.3.1
fpconst==0.7.3
iniparse==0.3.1
iotop==0.3.2
mock==1.0.1
mod-python==3.3.1
nagios-plugins-rhev==1.0.0
nose==1.2.1
paramiko==1.7.5
pexpect==2.3
pssh==2.3.1
py-rrdtool==0.2.1
pyOpenSSL==0.10
pycrypto==2.0.1
pycurl==7.19.0
pygpgme==0.1
pyparsing==1.5.7
pysqlite==2.3.5
python-dateutil==1.4.1
python-ldap==2.3.10
python-memcached==1.43
pytz==2013b
pyxdg==0.18
simplejson==2.0.9
txAMQP==0.6.2
urlgrabber==3.9.1
whisper==0.9.9
yum-metadata-parser==1.1.2
zope.interface==4.0.5

配置graphite

cd /opt/graphite/conf/
cp carbon.conf.example carbon.conf
cp storage-schemas.conf.example storage-schemas.conf
cp graphite.wsgi.example graphite.wsgi

复制examples下的apache配置文件到对应的conf.d目录下

cp /opt/graphite/examples/example-graphite-vhost.conf /etc/http/conf.d/graphite.conf

记得给graphite.conf中的WSGISocketPrefix的目录给创建好就可。 当然你可以不用apache来启动,用python自带的web server也可以

python /opt/graphite/bin/run-graphite-devel-server.py /opt/graphite &

初始化数据库和本地配置

cd /opt/graphite/webapp/graphite
python manage.py syncdb
chown apache:apache /opt/graphite/storage/
cd /opt/graphite/webapp/graphite
cp local_settings.py.example local_settings.py

记得修改local_settings.py中时区为Asia/Shanghai, 其他为默认就可以了。

启动apache和carbon

/etc/init.d/httpd reload
python /opt/graphite/carbon-cache.py start

好了,这个时候graphite是安装完成了,但是graphite是只负责绘图,不负责收集数据,收集数据使用collectd来完成的。collectd分为服务器端和客户端。在这里,我们服务器端放在跟graphite同一台机器上。安装方法一样

安装collectd

安装依赖包

yum install rrdtool rrdtool-devel rrdtool-perl libgcrypt-devel gcc make gcc-c++

默认epel源里有collectd,但是版本太低,我们还是从官方网站上下载后手动进行安装

wget http://collectd.org/files/collectd-5.3.0.tar.bz2
tar jxvf collectd-5.3.0.tar.bz2
cd collectd-5.3.0
./configure --prefix=/opt/server/collectd --enable-all-plugins
make
make install

安装启动脚本
vim /etc/init.d/collectd

#!/bin/bash
#
# collectd    Startup script for the Collectd statistics gathering daemon
# chkconfig: - 99 01
# description: Collectd is a statistics gathering daemon used to collect \
#   system information ie. cpu, memory, disk, network
# processname: collectd
# config: /etc/collectd.conf
# config: /etc/sysconfig/collectd
# pidfile: /var/run/collectd.pid

# Source function library.
. /etc/init.d/functions
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/server/collectd/sbin

RETVAL=0
ARGS=""
prog="collectdmon"
service="collectd"
CONFIG=/opt/server/collectd/etc/collectd.conf
COLLECTD=/opt/server/collectd/sbin/collectd
COLLECTDMONPID=/var/run/collectdmon.pid

if [ -r /etc/sysconfig/$service ]; then
        . /etc/sysconfig/$service
fi

check_config() {
        if test ! -r "$CONFIG"; then
                return 2
        fi
        if ! $COLLECTD -t -C "$CONFIG"; then
                return 1
        fi
        return 0
}

start () {
        echo -n $"Starting collectd: "
        check_config
        rc="$?"
        if test "$rc" -ne 0; then
                RETVAL=6
                echo $"not starting due to configuration error"
                failure $"not starting $service due to configuration error"
        else
                daemon $prog -P $COLLECTDMONPID -c $COLLECTD -- -C "$CONFIG" $ARGS
                RETVAL=$?
                echo
                [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$service
        fi
}
stop () {
        echo -n $"Stopping collectd: "
        killproc $prog
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$service
}
# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status $prog
        ;;
  restart|reload)
        check_config
        rc="$?"
        if test "$rc" -ne 0; then
                RETVAL=6
                echo $"not restarting due to configuration error"
                failure $"not restarting $service due to configuration error"
        else
                stop
                start
        fi
        ;;
  condrestart)
        [ -f /var/lock/subsys/$prog ] && restart || :
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|condrestart}"
        exit 1
esac

exit $?

# vim:syntax=sh

修改collectd配置文件
首先看客户端的配置文件

LoadPlugin network

<Plugin network>
#   # client setup:
    Server "10.1.1.240" "25827"
# 10.1.1.240 为collectd服务器端的地址, 25827为监听的端口
#   <Server "239.192.74.66" "25826">
#       SecurityLevel Encrypt
#       Username "user"
#       Password "secret"
#       Interface "eth0"
#
    TimeToLive "128"
</Plugin>

启动collectd就可以了

服务器端配置文件

LoadPlugin network

<Plugin network>
#       # server setup:
        Listen "10.1.1.240" "25827"
</Plugin>

如果启动没有问题的话,那理论上在服务器端的/opt/server/collectd/var/lib/collectd 目录下可以看到client的目录,和默认load的模块比如cpu的一些rrd文件。

现在我们只能collectd来收集数据,grapite还没有跟collectd连通并进行绘图。我们需要有一个插件将collectd传递给carbon

安装collecd-carbon

mkdir /opt/server/collectd/plugin/
cd /opt/server/collectd/plugin/
git cloen git://github.com/indygreg/collectd-carbon

修改collecd.conf配置文件加载collectd-carbon

<LoadPlugin "python">
    Globals true
</LoadPlugin>

<Plugin "python">
    # carbon_writer.py is at path /opt/collectd-plugins/carbon_writer.py
    ModulePath "/opt/server/collectd/plugin/collectd-carbon/"
    Import "carbon_writer"
    <Module "carbon_writer">
        LineReceiverHost "10.1.1.240"
        LineReceiverPort 2003
        LineReceiverProtocol "tcp"
        DifferentiateCountersOverTime true
        LowercaseMetricNames true
        TypesDB "/opt/server/collectd/share/collectd/types.db"
    </Module>
</Plugin>

重新启动collectd就可以了,然后就会往服务器端的2003端口写入数据了。

最后,自定义脚本

虽然collectd默认自带了很多的脚本, 但是还是没法完全覆盖我们自己的要求,所以需要自己编写脚本去执行结果,并进行画图。
下面这个是我的事例脚本,很简单,就是吐出2个值,一个是上2秒的qps和最近10000次请求的超时时间的请求数量。

#!/bin/bash

#count the qps and error

id=$HOSTNAME

while true; do
  TIME=$(date -d "2 seconds ago" +%Y:%H:%M:%S)
  start_run=$(date +%s)
  abc_qps=$(tail -n50000 /opt/server/nginx/logs/abc.access.log | grep -c "$TIME")
  abc_err=$(tail -n10000 /opt/server/nginx/logs/abc.access.log | awk -F'|' '{if($14 > 0.08) print}'| wc -l)
    echo 'PUTVAL '"$id"'/qps/gauge-abc_qps '"$start_run"':'"$abc_qps"
    echo 'PUTVAL '"$id"'/timeout/gauge-abc_err '"$start_run"':'"$abc_err"
    sleep 10
done

其中最主要的就是输出时候的定义
id/qps/gauge-abc_qps 分别对应下面这个

<instance-id>/<plugin>-<plugin_instance>/<type>-<type_instance>

其中instance-id我这里用主机名来表示
plugin是qps和timeout两种
gauge是表示type的一种。总计有很多种type:

Generic Graph Types

    :blank
    :countr
    :bytes_txrx
    :recv_sent
    :requests
    :cache

Plugin-specific Graph Types

    :haproxy_sessions
    :haproxy_traffic
    :haproxy_status
    :apache_bytes
    :apache_requests
    :apache_scoreboard
    charge
    charge_percent
    cpu
    current
    df
    disk
    disk_octets
    disk_merged
    disk_ops
    disk_time
    dns_opscode
    dns_qtype
    dns_rcode
    dns_octets
    email
    email_size
    spam_score
    spam_check
    entropy
    fanspeed
    frequency
    frequency_offset
    gauge
    hddtemp
    if_errors
    if_packets
    irq
    load
    load_percent
    mails
    memory
    old_memory
    memcached_connections
    memcached_items
    memcached_octets
    memcached_ops
    mysql_commands
    mysql_handler
    mysql_octets
    mysql_qcache
    mysql_threads
    nfs_procedure
    nfs3_procedures
    nginx_connections
    nginx_requests
    opcode
    operations
    partition
    passenger_instances
    passenger_processes
    passenger_queued
    passenger_requests
    percent
    ping
    processes
    ps_rss
    ps_cputime
    ps_count
    ps_pagefaults
    ps_state
    queue_length
    qtype
    rcode
    swap
    ols_swap
    temperature
    timeleft
    time_offset
    if_octets
    cpufreq
    multimeter
    users
    voltage
    vs_threads
    vs_memory
    vs_processes

写好扩展脚本,把它放入到/opt/server/collectd/plugin/目录,加入可执行权限。

修改collectd.conf文件加载扩展脚本

LoadPlugin exec
Exec "apache" "/opt/server/collectd/plugin/qps_err.sh"
# apache为执行这个脚本的用户名
#       NotificationExec "user:group" "/path/to/exec"

最终的展示效果如下
graphite

关于 Timo
XNIX SA & MYSQL DBA

发表评论

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / 更改 )

Twitter picture

You are commenting using your Twitter account. Log Out / 更改 )

Facebook photo

You are commenting using your Facebook account. Log Out / 更改 )

Google+ photo

You are commenting using your Google+ account. Log Out / 更改 )

Connecting to %s

%d 博主赞过: