[로그통합]Graylog2 & MongDB & Elasticsearch

So the much anticipated Graylog2 version 0.10 was released on the 14th, so I finally found some time to build a test box with all the new software and see how it compares to get up and running.

So let’s get started.

Only non-default repo I have installed is:

1

2

3

4

5

6

7

8

cat /etc/yum.repos.d/epel.repo

[epel]

name=Extra Packages for Enterprise Linux 6 - $basearch

#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch

mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch

failovermethod=priority

enabled=0

We are downloading the following software packages:

1

2

3

4

5

6

7

8

9

10

11

12

# These are the versions of software I am using

ruby - 2.0.0-p0

rubygems - 2.0.0

graylog2-server - 0.10.0

graylog2-web-interface - 0.10.2

graylog2-radio - 1.0.0

elastic-search - 0.20.4

mongodb - 2.2.3

rabbitmq - 3.0.2

# From Yum (Note this is from my template so I may have missed a package here)

yum install -y gcc gcc-c++ gd gd-devel glibc glibc-common glibc-devel glibc-headers make automake httpd httpd-devel java-1.7.0-openjdk java-1.7.0-openjdk-devel wget tar vim nc libcurl-devel openssl-devel zlib-devel zlib patch readline readline-devel libffi-devel curl-devel

Download all the software: (http://www.graylog2.org/download)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

mkdir /opt/installs

cd /opt/installs

# Graylog2-Server

wget http://download.graylog2.org/graylog2-server/graylog2-server-0.10.0.tar.gz

# Graylog2-Web-Interface

wget http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-0.10.2.tar.gz

# Graylog2-Radio

wget http://download.graylog2.org/graylog2-radio/graylog2-radio-1.0.0.tar.gz

# Elastic-Search

wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.4.tar.gz

A one liner for the lazy:

1

2

3

4

wget http://download.graylog2.org/graylog2-server/graylog2-server-0.10.0.tar.gz && 

wget http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-0.10.2.tar.gz && 

wget http://download.graylog2.org/graylog2-radio/graylog2-radio-1.0.0.tar.gz && 

wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.4.tar.gz

Just to make sure we got everything:

1

2

3

4

5

6

7

8

>ls -alh

total 133M

drwxr-xr-x  2 root root 4.0K Feb 26 11:36 .

drwxr-xr-x. 3 root root 4.0K Dec 17 11:49 ..

-rw-r--r--  1 root root  16M Jan 29 03:05 elasticsearch-0.20.4.tar.gz

-rw-r--r--  1 root root 7.3M Feb 21 11:48 graylog2-radio-1.0.0.tar.gz

-rw-r--r--  1 root root  41M Feb 14 10:39 graylog2-server-0.10.0.tar.gz

-rw-r--r--  1 root root 2.0M Feb 19 16:08 graylog2-web-interface-0.10.2.tar.gz

Extract everything:

1

cat *.t* | tar -zxvf - -i

Great now let’s get Ruby installed:

1

cd /usr/local

Great now let’s get Libyaml installed:

libyaml-0.1.4-1.el6.rf.x86_64.rpm

libyaml-devel-0.1.4-1.el6.rf.x86_64.rpm

1

http://pkgs.repoforge.org/ ### 패키지 사이트

Install RVM (Ruby Version Manager):

1

curl -L https://get.rvm.io | bash -s stable --ruby

Once it’s done you need to source rvm to use it:

1

source /usr/local/rvm/scripts/rvm

Check to make sure ruby is working and it can find the binary:

1

2

ruby -v

ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]

Install Ruby Gem Bundle:

1

gem install bundle

Update System Gems:

1

gem update

Installing Elastic Search:

1

2

cd /usr/local

cp -R installs/elasticsearch-0.20.4 elasticsearch

Install the ES service wrapper:

1

2

3

4

5

6

7

8

curl -k -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz

mv *servicewrapper*/service elasticsearch/bin/

rm -Rf *servicewrapper*

/usr/local/elasticsearch/bin/service/elasticsearch install

# Returns

Detected RHEL or Fedora:

Installing the ElasticSearch daemon..

Give yourself a control script:

1

ln -s `readlink -f elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl

Give the ES cluster a unique name:

1

sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /usr/local/elasticsearch/config/elasticsearch.yml

Fire it up and test it:

1

2

3

4

/etc/init.d/elasticsearch start

# Give it a couple seconds before you try as Elastic isn't instant

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Should get a response similar to:

1

2

3

4

5

6

7

8

9

10

11

12

{

  "cluster_name" : "graylog2",

  "status" : "green",

  "timed_out" : false,

  "number_of_nodes" : 1,

  "number_of_data_nodes" : 1,

  "active_primary_shards" : 0,

  "active_shards" : 0,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0

}

Add ElasticSearch to boot:

1

2

chkconfig --add elasticsearch

chkconfig elasticsearch on

Setup 10gen Repo and Install MongoDB:

1

In a terminal session, begin by downloading the latest release. In most cases you will want to download the 64-bit version of MongoDB.

1

2

3

4

5

curl http://downloads.mongodb.org/linux/mongodb-linux-x86_64-2.4.3.tgz > mongodb.tgz

If you need to run the 32-bit version, use the following command. curl http://downloads.mongodb.org/linux/mongodb-linux-i686-2.4.3.tgz > mongodb.tgz Once you’ve downloaded the release, issue the following command to extract the files from the archive

tar -zxvf mongodb.tgz

Optional You may use the following command to copy the extracted folder into a more generic location.

cp -R -n mongodb-linux-????-??-??/ mongodb

Create the mongo db directory:

1

2

# Just in case you run the daemon manually, you will need this directory.

mkdir -p /usr/local/mongodb-linux-????-??-??

ln -s /usr/local/mongdb-linux-????-??-?? mongodb

Mongo configure file create and start:

  
mongodb.conf

1

vi /etc/mongodb.conf ### 파일 복사

mkdir /usr/local/mongodb/log ###,로그디렉토리 생성

mkdir /usr/local/mongodb/data ### 데이터 디렉토리 생성

Wait for it to start, it takes a couple seconds:

1

2

3

4

Starting mongod: forked process: 14099

all output going to: /var/log/mongo/mongod.log

# You may have to press enter here for it to show the next line

child process started successfully, parent exiting            [  OK  ]

Setup MongoDB and auth

1

mongo

1

2

3

MongoDB shell version: 2.2.2

connecting to: test

>

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

>use admin

switched to db admin

>db.addUser('admin', 'CHANEGME')  <--- DO NOT USE @ symbols

{

"user" : "admin",

"readOnly" : false,

"pwd" : "18efdb7d5f4c750bb1849d606e044960",

"_id" : ObjectId("512d05edfe07b13209a37262")

}

>db.auth('admin', 'CHANGEME')

1

>use graylog2

switched to db graylog2

>db.addUser('grayloguser', 'CHANGEME')  <--- DO NOT USE @ symbols

{

"user" : "grayloguser",

"readOnly" : false,

"pwd" : "dcd2fc9ca8bd61c6e5c1f88fc9c38872",

"_id" : ObjectId("512d0699fe07b13209a37263")

}

>db.auth('grayloguser', 'CHANGME')

1

>exit

Set MongoDB to start on boot:

1

chkconfig mongod on

Intall MongoDB Gem:

1

gem install bson_ext mongo json

Install Graylog2-Server:

1

2

cp -R /opt/installs/graylog2-server-0.10.0 /usr/local/graylog2-server

cd /usr/local/graylog2-server

1

2

cp graylog2.conf.example /etc/graylog2.conf

cp elasticsearch.yml.example /etc/graylog2-elasticsearch.yml

Configure Graylog2.conf:

1

vim /etc/graylog2.conf

Configure at least these variables in /etc/graylog2.conf:

is_master = true
- Set only one graylog2-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t. Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.
elasticsearch_config_file = /etc/graylog2-elasticsearch.yml
- This is the path to the ElasticSearch configuration file for the built-in ElasticSearch node of graylog2-server. Your graylog2-server node will act as a node in your ElasticSearch cluster, but not store any data itself. It will distribute the writes to other nodes in the ElasticSearch cluster.
elasticsearch_max_docs_per_index = 20000000
- How many log messages to keep per index. This setting multiplied withelasticsearch_max_number_of_indices results in the maximum number of messages in your Graylog2 setup. It is always better to have several more smaller indices than just a few larger ones.
elasticsearch_max_number_of_indices = 20
- How many indices to have in total. If this number is reached, the oldest index will be deleted.
elasticsearch_shards = 4
- The number of shards for your indices. A good setting here highly depends on the number of nodes in your ElasticSearch cluster. If you have one node, set it to 1. Read more about this in the knowledge base article about configuring and tuning ElasticSearch.
elasticsearch_replicas = 0
- The number of replicas for your indices. A good setting here highly depends on the number of nodes in your ElasticSearch cluster. If you have one node, set it to 0. Read more about this in the knowledge base article about configuring and tuning ElasticSearch.
recent_index_ttl_minutes = 60
- Graylog2 keeps a so called recent index that includes only the newest log messages. This allows fast overview pages in the web interface. The messages you see in the “show recent messages” view are from this index. If you have thousands of messages per minute, set it to 1 minute because there are so many new messages coming in. If you have just a few messages per minute, set it to a higher values to still have a good overview without having to click on “show all messages”.
mongodb_*
- Enter your MongoDB connection and authentication information here. Make sure that you connect the web interface to the same database. You don’t need to configure mongodb_user and mongodb_password ifmongodb_useauth is set to false.

The most important ones (the rest you can leave as default if you want):

processor_wait_strategy = blocking (this helps load on the server by quite a bit)
mongodb_password = whatever password you set earlier
Take note of the AMQP Section we will come back to this later

1

2

# To save

:wq

Start up Graylog2-Server:

1

cd /usr/local/graylog2-server

1

java -jar graylog2-server.jar --debug

Note: You may see a couple snappy error messages the first time you start graylog2. Just ignore them the first time as it’s having to setup all the indexes.

Wait until you get this message:

1

2013-02-26 13:23:08,167 INFO : org.graylog2.Core - Graylog2 up and running

Open another SSH session and test it out and make sure it’s working:

1

echo "<86>Dec 24 17:05:01 foo-bar CRON[10049]: pam_unix(cron:session):" | nc -w 1 -u localhost 514

You should see a whole bunch of output if it was successful. If not you will receive something like:

1

2

DEBUG: org.graylog2.inputs.syslog.SyslogProcessor -

Skipping incomplete message

Exit WebBrick:

1

Hit Ctrl + C and end the graylog2 webbrick server

Drop an init.d script to make it easy:

1

vim /etc/init.d/graylog2-server

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

#!/bin/sh

#

# graylog2-server:   graylog2 message collector

#

# chkconfig: - 98 02

# description:  This daemon listens for syslog and GELF messages and stores them in mongodb

#

CMD=$1

NOHUP=`which nohup`

JAVA_HOME=/usr/java/latest

JAVA_CMD=/usr/bin/java

GRAYLOG2_SERVER_HOME=/usr/local/graylog2-server

start() {

    echo "Starting graylog2-server ..."

    $NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &

}

stop() {

        PID=`cat /tmp/graylog2.pid`

    echo "Stopping graylog2-server ($PID) ..."

        kill $PID

}

restart() {

    echo "Restarting graylog2-server ..."

        stop

        start

}

case "$CMD" in

    start)

        start

        ;;

    stop)

        stop

        ;;

    restart)

        restart

        ;;

    *)

        echo "Usage $0 {start|stop|restart}"

        RETVAL=1

esac

Load it up:

1

2

3

4

chmod +x /etc/init.d/graylog2-server

chkconfig --add graylog2-server

chkconfig graylog2-server on

/etc/init.d/graylog2-server start

Install Graylog2 Web Interface:

1

cp -R /opt/installs/graylog2-web-interface-0.10.2 /usr/local/graylog2-web-interface

1

cd /usr/local/graylog2-web-interface

Fix the Gemfile:

Note: The current version of Graylog2-Web-Interface uses json-1.5.5, unfortunately there is a royal pain in the butt issue of installing Ruby 2.0.0 and not having the ruby development headers to compile it. To work around this issue we need to change the required version of json in the gemfile inside the graylog2-web-interface folder.

I also made some changes to the version of the mongo driver it’s using, as it was set to use a much older version (pretty sure the mongo, bson_ext part is optional)

Edit Gemfile inside graylog2-web-interface:

1

vim Gemfile

1

2

3

4

5

6

7

8

9

10

11

12

13

14

# Change

gem 'json', '~> 1.5.5'

gem 'bson', "~> 1.3.1"

gem 'bson_ext', "~> 1.3.1", :platforms => :ruby

# To

gem 'json', '~> 1.7.7'

gem 'bson', "~> 1.8.2"

gem 'bson_ext', "~> 1.8.2", :platforms => :ruby

# Add

gem "mongo", "~> 1.8.2"

# Save and quit

:wq

This is what my /usr/local/graylog2-web-interface/Gemfile looks like:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

source :rubygems

gem 'rack', '~> 1.4.1'

gem 'rake', '~> 0.9.2'

gem 'rails', '~> 3.2.12'

gem 'json', '~> 1.7.7'

gem 'chronic', '~> 0.6.7'

gem 'pony', '~> 1.1'  # unusual version number

gem 'graylog2-declarative_authorization', '~> 0.5.2', :require => 'declarative_authorization'

gem "mongo", "~> 1.8.2"

gem 'mongoid', '2.4.5'

gem "tire", "~> 0.5.1"

gem 'bson', "~> 1.8.2"

gem 'bson_ext', "~> 1.8.2", :platforms => :ruby

gem 'home_run', '~> 1.0.2', :platforms => :ruby

gem 'SystemTimer', '~> 1.2.3', :require => 'system_timer', :platforms => :ruby_18

gem 'rails_autolink', '~> 1.0.4'

gem 'kaminari', '~> 0.12.4'

gem 'jquery-rails', '~> 2.1'

gem 'therubyracer', '~> 0.10.2'

gem 'net-ldap', '~> 0.3.1'

group :development, :test do

  # might be useful to generate fake data in development

  gem 'machinist_mongo', '~> 1.2.0', :require => 'machinist/mongoid'

  gem 'faker', '~> 0.9.5'

end

group :development do

  # gem 'ruby-prof', '~> 0.10.5'  # works nice with NewRelic RPM Developer Mode

  gem 'passenger', '~> 3.0.17'

end

group :test do

  gem 'ci_reporter', '~> 1.6.4'

  gem 'shoulda', '~> 2.11.3'

  gem 'shoulda-activemodel', '0.0.2', :require => 'shoulda/active_model'  # fixed version - too hacky

  gem 'mocha', '~> 0.9.12'

  gem 'database_cleaner', '~> 0.6.0'

  gem 'timecop', '~> 0.3.5'

end

# Needed for the new asset pipeline

group :assets do

  gem 'sass-rails',   "~> 3.2.5"

  gem 'coffee-rails', "~> 3.2.2"

  gem 'uglifier',     ">= 1.0.3"

end

1

2

bundle update mongo json bson_ext

bundle install --without=development

curl -L https://get.rvm.io | bash -s stable --autolibs=enabled --ruby --rails --trace   ### 안되면 이걸로 실행

rvm install 1.9.2

Setup a Graylog2 User:

1

2

useradd graylog2 -d /usr/local/graylog2-web-interface -G rvm

chown -R graylog2:graylog2 /usr/local/graylog2-web-interface

Give root RVM access:

1

2

usermod -g rvm root

source /etc/profile.d/rvm.sh

Setup the web interface’s mongo config:

1

vim /usr/local/graylog2-web-interface/config/mongoid.yml

1

2

3

4

5

6

production:

  host: localhost

  port: 27017

  username: grayloguser

  password: PASSWORD

  database: graylog2

Start Graylog2 and setup your first user:

1

2

3

4

5

cd /usr/local/graylog2-web-interface

su graylog2

# Note there is bug with the current mongo driver that gives a notice about bson_ext not being loaded.  It will be fixed in a future version.

RAILS_ENV=production script/rails server

# Note: If mongo isn’t running, you used an @ symbol in your password or if you installed mongo from source or some other method, check /etc/mongod.conf and see where it is putting the database. You may have to create /data/db directory, you will get an error that looks like this:

1

/usr/local/rvm/gems/ruby-2.0.0-p0/gems/mongo-1.8.2/lib/mongo/util/uri_parser.rb:285:in `parse_hosts': MongoDB URI must include username, password,  (Mongo::MongoArgumentError)

Open a browser and see if the web ui is working:

1

2

3

4

5

Browse to http://IPADDRESS:3000 and setup your first user and login.

Once done, CTRL-C to stop the server

Then type EXIT to go back to root.

Install and configure Passenger:

Note: There is currently a bug with passenger-3.0.19 and ruby-2.0.0 if you run passenger-install-apache2-module you will get compilation errors. See this github issue

1

gem install passenger file-tail

If they’ve fixed the passenger bug and you install a version > 3.0.19 then the below doesn’t apply otherwise you can use these steps as a workaround.

To work around the current issue with passenger-3.0.19 you can edit the gempacktask.rb inside the passenger gem:

1

vim /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19/build/gempackagetask.rb

Delete everything in it and paste the following into it (quick way to delete everything inside vim type: 1000 dd:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

#!/usr/bin/env ruby

# Passenger note: this file is copied from Rake 0.8.1. The task names

# have been changed.

# Define a package task library to aid in the definition of GEM

# packages.

require 'rubygems'

require 'rake'

require 'build/packagetask'

require 'rubygems/user_interaction'

if /^2\./ =~ RUBY_VERSION

  require 'rubygems/package'

else

  require 'rubygems/builder'

end

module Rake

  # Create a package based upon a Gem spec.  Gem packages, as well as

  # zip files and tar/gzipped packages can be produced by this task.

  #

  # In addition to the Rake targets generated by PackageTask, a

  # GemPackageTask will also generate the following tasks:

  #

  # [<b>"<em>package_dir</em>/<em>name</em>-<em>version</em>.gem"</b>]

  #   Create a Ruby GEM package with the given name and version.

  #

  # Example using a Ruby GEM spec:

  #

  #   require 'rubygems'

  #

  #   spec = Gem::Specification.new do |s|

  #     s.platform = Gem::Platform::RUBY

  #     s.summary = "Ruby based make-like utility."

  #     s.name = 'rake'

  #     s.version = PKG_VERSION

  #     s.requirements << 'none'

  #     s.require_path = 'lib'

  #     s.autorequire = 'rake'

  #     s.files = PKG_FILES

  #     s.description = <<EOF

  #   Rake is a Make-like program implemented in Ruby. Tasks

  #   and dependencies are specified in standard Ruby syntax.

  #   EOF

  #   end

  #

  #   Rake::GemPackageTask.new(spec) do |pkg|

  #     pkg.need_zip = true

  #     pkg.need_tar = true

  #   end

  #

  class GemPackageTask < PackageTask

    # Ruby GEM spec containing the metadata for this package.  The

    # name, version and package_files are automatically determined

    # from the GEM spec and don't need to be explicitly provided.

    attr_accessor :gem_spec

    # Create a GEM Package task library.  Automatically define the gem

    # if a block is given.  If no block is supplied, then +define+

    # needs to be called to define the task.

    def initialize(gem_spec)

      init(gem_spec)

      yield self if block_given?

      define if block_given?

    end

    # Initialization tasks without the "yield self" or define

    # operations.

    def init(gem)

      super(gem.name, gem.version)

      @gem_spec = gem

      @package_files += gem_spec.files if gem_spec.files

    end

    # Create the Rake tasks and actions specified by this

    # GemPackageTask.  (+define+ is automatically called if a block is

    # given to +new+).

    def define

      super

      task :package => ['package:gem']

      desc "Build the gem file #{gem_file}"

      task 'package:gem' => ["#{package_dir}/#{gem_file}"]

      file "#{package_dir}/#{gem_file}" => [package_dir] + @gem_spec.files do

        when_writing("Creating GEM") {

          if /^2\./ =~ RUBY_VERSION

            Gem::Package.build(gem_spec)

          else

            Gem::Builder.new(gem_spec).build

          end

          verbose(true) {

            mv gem_file, "#{package_dir}/#{gem_file}"

          }

        }

      end

    end

    def gem_file

      if @gem_spec.platform == Gem::Platform::RUBY

        "#{package_name}.gem"

      else

        "#{package_name}-#{@gem_spec.platform}.gem"

      end

    end

  end

end

Now let’s install the apache module:

1

2

3

4

5

6

passenger-install-apache2-module

# You should get all green FOUND messages, if you don't make sure you have all the yum packages installed.

Copy the LoadModule text at the end once it finishes

Press Enter to continue.

Create passenger.conf for httpd:

1

vim /etc/httpd/conf.d/passenger.conf

Paste the text from the end of the passenger-apache2-module install:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

LoadModule passenger_module /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19/ext/apache2/mod_passenger.so

PassengerRoot /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19

PassengerRuby /usr/local/rvm/wrappers/ruby-2.0.0-p0/ruby

<VirtualHost *:80>

    DocumentRoot /opt/graylog2-web-interface/public

    RailsEnv 'production'

    <Directory /opt/graylog2-web/public>

        Allow from all

        Options -MultiViews

    </Directory>

    ErrorLog /var/log/httpd/graylog2_error.log

    LogLevel warn

    CustomLog /var/log/httpd/graylog2_access.log combined

</VirtualHost>

Set Permissions and Add Apache to startup and load:

1

2

3

chown -R apache:apache /usr/local/graylog2-web-interface

chkconfig httpd on

/etc/init.d/httpd start

Open a browser and see if the web ui is working:

1

Browse to http://IPADDRESS and login

Technically, you’ve got a working version of Graylog2 setup and working at this point. If you wanted you could start forwarding syslog messages to the Graylog2 Server on port 514 and off you go.

If you would like to setup an AMQP broker so that messages can be queued and add a few useful features (restarting graylog2-server without losing syslog messages), read further and see how it’s setup.

Setup AMQP (RabbitMQ):

I’m going to be using RabbitMQ as our AMQP broker, you can use others if you wish.

Install RabbitMQ:

1

2

3

cd /opt/installs

yum install -y erlang --enablerepo=epel

rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v3.0.2/rabbitmq-server-3.0.2-1.noarch.rpm

rabbitmq-server-3.1.0-1.noarch.rpm

Get RabbitMQAdmin:

1

2

3

4

5

6

7

8

# Download the script

wget http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_0_2/bin/rabbitmqadmin

# Move to /usr/local/bin

mv rabbitmqadmin /usr/local/bin

# Set permissions

chmod o+x /usr/local/bin/rabbitmqadmin

Start Rabbit and add to Startup:

1

2

/etc/init.d/rabbitmq-server start

chkconfig rabbitmq-server on

Create RabbitMQ Users and Exchanges for Graylog to use, here is the link to the official docs for reference:RabbitMQ Management

1

2

3

4

5

6

7

8

9

# Enable Management Plugins

rabbitmq-plugins enable rabbitmq_management

# Restart RabbitMQ

/etc/init.d/rabbitmq-server restart

# Setup the exchange

rabbitmqadmin declare exchange name=messages type=topic

-> exchange declared (You should get this as a successful notice)

Edit the graylog2.conf to enable AMQP:

1

vim /etc/graylog2.conf

1

2

3

4

5

6

7

8

9

# AMQP

amqp_enabled = true

amqp_host = localhost

amqp_port = 5672

amqp_username = guest

amqp_password = guest

amqp_virtualhost = /

# Save and exit

1

2

# Restart Graylog2-Server

/etc/init.d/graylog2-server restart

Monitor the Graylog2.log file for this next part, as you will immediately see java errors if something isn’t setup right (ignore the plugin errors):

1

tail -f /var/log/graylog2.log

Go to the Graylog2 Web Interface > Settings > AMQP:

Add new configurations with these parameters there

Exchange: messages
Routing key: syslogudp
TTL: 720000
Type: syslog

If no errors in the graylog2.log file and the web ui says added successfully. Look under Server Health on the web ui home page:

Setup the new Graylog2-Radio: Official Documentation

1

2

3

4

5

6

7

cd /opt/installs

cp -R graylog2-radio-1.0.0 /usr/local/graylog2-radio

cd /usr/local/graylog2-radio

mv graylog2-radio.conf.example /etc/graylog2-radio.conf

mv graylog2-radio-inputs.conf.example /etc/graylog2-radio-inputs.conf

cp bin/radioctl /etc/init.d/graylog2-radio

chmod o+x /etc/init.d/graylog2-radio

Edit the init script, add the comments at the top for chkconfig and change a couple paths:

1

vim /etc/init.d/graylog2-radio

1

2

3

4

5

6

7

8

9

10

#!/bin/bash

#

# graylog2-radio: graylog2 radio AMQP collector

#

# chkconfig: - 98 02

# description: This daemon listens for syslog and GELF messages and stores them in RabbitMQ

#

RADIO_JAR=/usr/local/graylog2-radio/graylog2-radio.jar

LOG_FILE=/var/log/graylog2-radio.log

Edit the Graylog2-Radio Inputs:

1

vim /etc/graylog2-radio-inputs.conf

1

2

3

4

5

udp messages 0.0.0.0 12501

#udp systemlogs 0.0.0.0 12502

#tcp otherlogs 0.0.0.0 12503

# Save and exit

Start Graylog2-Radio:

1

2

/etc/init.d/graylog2-radio start

Hit enter

You can monitor the log file to see if its working at:

1

tail -f /var/log/graylog2-radio.log

If everything is working correctly:

1

2

15:24:06.059 [main] INFO org.graylog2.radio.Radio - Started REST API at <http://127.0.0.1:12888/>

15:24:06.062 [main] INFO org.graylog2.radio.Radio - Spawning input [UDP /0.0.0.0:12501@messages]

Add Graylog2-Radio to Startup:

1

2

chkconfig --add graylog2-radio

chkconfig graylog2-radio on

Change rsyslog to forward to Graylog2 port:

1

vim /etc/rsyslog.conf

At the very bottom of the file:

1

*.* @IPOFGRAYLOG:12501

Of course this is by far the simplest method of getting all this working. Some other things you can do is add more exchanges for your applications, just don’t forget to setup the corresponding nodes in the graylog2-web-interface and inside the graylog2-radio inputs config file.

I ran through this entire setup 3 times and I “think” I got all the errors, but if you run into anything please let me know!

Side note: Stream Dashboards seem broken in 0.10.2 of the web-interface

RabbitMQ 설치 관련 오류

yum install libcom 설치

RabbitMQ 설치 오류 메시지 처리 방법

Database Hang (/var/log/rabbitmq/startup_log)
- /var/lib/rabbitmq/mnsia 디렉토리 삭제 후 재시작

Network Startup Failed

- rabbitmq could not start tcp listener
- netstat -nlp 로 tcp 5672 사용 데몬 확인

As in most cases I'm sure, this issue was caused by a misunderstanding on my part. RHEL provides an amqp server called Qpid which was already running. Since I thought that epmd was running on 5672, I missed the listing for ampq/Qpid in
etstat (not a fan of symbolic ports in the output now).

Stopping the Qpid service corrects the problem.

# service qpidd stop & chkconfig --del qpidd

Client SYSLOG 연동 : 기본 시스로그로 연동 시 graylog2에서 host field 값에 프로세스명이 찍히는 버그 발생. 아래의 rsyslog 설치 후 셋팅 진행하도록 workarround.

/etc/rsyslog.conf:

$template GRAYLOG2,"<%PRI%>%HOSTNAME% %TIMESTAMP% %syslogtag% %APP-NAME% %msg%\n"
$ActionForwardDefaultTemplate GRAYLOG2
*.* @graylog-server

/etc/sysconfig/rsyslogd.conf

SYSLOGD_OPTIONS="-c2 -m 0 -r514"

[로그통합]로그 파싱 관련 (0)	2013.05.21
[로그통합]JBOSS parsing (0)	2013.05.20
[로그통합]Weblogic logs using Logstash and Graylog2 (0)	2013.05.20
[로그통합]logstash + graylog2 (0)	2013.05.20

All Days 무한도전

카테고리

달력

공지사항

태그목록

최근에 올라온 글