Splunk Distributed node installations

Splunk Distributed node installations

Environment Setup

·         Download Ubuntu Linux 12.04.3 LTS or CentOS 6.7 (Any linux flavor)

·         Download Latest Splunk Enterprise tar ball ( For simplicity)

·         Download latest Universal Splunk Forwarder tar ball.

Pre-requisites

·         Do the necessary network settings and assign static IP if preferred.

·         Assign the hostname in network and hosts files.

Step 1> Identify the nodes with their roles and assign hostname and name for simplicity and easy management. In the below scenario I’m setting up 1 Search Head, 2 Indexers and 2 forwarder nodes.

Step 2> Install Splunk Enterprise binaries on the Search Head, Indexer_01 and Indexer_02 assigned machines. ( Extract the Splunk tar ball to the /opt folder as explained in the single node installation)

[root@splunk_standalone sbk_files]# tar -zxvf splunk-6.3.1-f3e41e4b37b2-Linux-x86_64.tgz -C /opt

Search Head Machine

[root@search_head splunk]# pwd

/opt/splunk

[root@search_head splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  6 04:36 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  6 04:06 var

[root@search_head splunk]#

Indexer_01 Machine

[root@indexer_01 splunk]# pwd

/opt/splunk

[root@indexer_01 splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  6 04:35 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  6 04:07 var

[root@indexer_01 splunk]#

Indexer_02 Machine

[root@indexer_02 splunk]# pwd

/opt/splunk

[root@indexer_02 splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  8 22:15 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  8 22:08 var

[root@indexer_02 splunk]#

Step 3> Install Universal Forwarder binaries into u_fwd_01 and u_fwd_02 names assigned machines. (Extract the Splunk tar ball to the /opt folder as explained in the single node installation)

[root@splunk_standalone sbk_files]# tar -zxvf splunk-6.3.1-f3e41e4b37b2-Linux-x86_64.tgz -C /opt

U_Fwd_01 Machine

[root@u_fwd_01 splunkforwarder]# pwd

/opt/splunkforwarder

[root@u_fwd_01 splunkforwarder]# ll

total 132

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 bin

-r--r--r--.  1  506  506    57 Oct 30  2015 copyright.txt

drwxr-xr-x. 13  506  506  4096 May  6 04:08 etc

drwxr-xr-x.  2  506  506  4096 Oct 30  2015 include

drwxr-xr-x.  4  506  506  4096 Oct 30  2015 lib

-r--r--r--.  1  506  506 62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506   509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 share

-r--r--r--.  1  506  506 31876 Oct 30  2015 splunkforwarder-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root  4096 May  6 04:08 var

[root@u_fwd_01 splunkforwarder]#

U_Fwd_02 Machine

[root@u_fwd_02 splunkforwarder]# pwd

/opt/splunkforwarder

[root@u_fwd_02 splunkforwarder]# ll

total 132

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 bin

-r--r--r--.  1  506  506    57 Oct 30  2015 copyright.txt

drwxr-xr-x. 13  506  506  4096 May  8 22:13 etc

drwxr-xr-x.  2  506  506  4096 Oct 30  2015 include

drwxr-xr-x.  4  506  506  4096 Oct 30  2015 lib

-r--r--r--.  1  506  506 62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506   509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 share

-r--r--r--.  1  506  506 31876 Oct 30  2015 splunkforwarder-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root  4096 May  8 22:13 var

[root@u_fwd_02 splunkforwarder]#

Configuring the instances into distributed deployment

Step 4> Enable/ Add Search Peer(Indexers) to the Search Head

·        Login into Search_head web UI.

·        Settings--> DISTRIBUTED ENV --> Distributed search

·        Search Peer-->add new --> Give inderxer details

·        Note:: Search peer is nothing but the indexer.

Step 5> Enable Receiver (Indexer is receiver here)

·        Login into Indexer Web UI

·        Setting --> DATA --> Forwarding and receiving

·        Receive data --> configure receiving --> add port number

·        Note:: this port will be open for forwarder to send data.

Step 6> Configure universal forwarder

·        Login into CLI of FORWARDER as it does not have WebUI.

·        Under $SPLUNK_HOME/etc/system/local     all conf files are placed details explained below

·        Configure the universal forwarder to forward to a specific receiving indexer, also known as the "receiver" :

splunk add forward-server <host>:<port> -auth <username>:<password>

Specify the DATA need to be monitored in input.conf

Restart the splunk forwarder to take effect of the new conf setting.

You must see below in the splunkd.log file which confirm the indexer was accepted through the port and monitoring of the data path we gave was successful.

In the Serach Web page you should see the data searched which will be same as data searched on Indexer (if the role in inderxer is set to search_head also)

In the inderxer web UI you should also see the monitored data in the data summary section.

Configure the universal forwarder

·          inputs.conf (for data inputs).

·          outputs.conf (for data outputs).

·          server.conf.

·          deploymentclient.conf.