Splunk Indexer Cluster Deployment

Splunk Indexer Cluster Deployment

Environment Setup

·         Download Ubuntu Linux 12.04.3 LTS or CentOS 6.7 (Any linux flavor)

·         Download Latest Splunk Enterprise tar ball ( For simplicity)

·         Download latest Universal Splunk Forwarder tar ball.

Pre-requisites

·         Do the necessary network settings and assign static IP if preferred.

·         Assign the hostname in network and hosts files.

Overview

1.      Identify requirement

Index cluster, with RF 2 and SF 2

For this number of nodes required will be RF +2

2indexer, 1 master node and 1 search head.

2.      Install the Splunk Enterprise cluster instances on your network

Install Splunk Enterprise on 4 nodes that is 2RF so two search peer nodes(indexers) and 1 master and 1search head.

 You need at least the replication factor number of peer nodes, but you might want to add more peers to boost indexing capacity, as mentioned in step 1d.

 You also need two more instances, one for the master node and the other for the search head.

3.      Enable clustering on the instances:

a. Enable the master node. See "Enable the master node".

Important: When the master starts up for the first time, it will block indexing on the peers until you have enabled and restarted the replication factor number of peers.

b. Enable the peer nodes. See "Enable the peer nodes".

c. Enable the cluster search head. It's easier to set up a search head for a cluster than for a non-clustered group of indexers. See "Enable the search head".

Step 1> Identify the nodes with their roles and assign hostname and name for simplicity and easy management. In the below scenario I’m setting up 1 Search Head, 2 Indexers, 1 Master Node and 1 forwarder node.

Step 2> Install Splunk Enterprise binaries on the Search Head, Indexer_01 and Indexer_02 assigned machines. (Extract the Splunk tar ball to the /opt folder as explained in the single node installation)

[root@splunk_standalone sbk_files]# tar -zxvf splunk-6.3.1-f3e41e4b37b2-Linux-x86_64.tgz -C /opt

Search Head Machine

[root@search_head splunk]# pwd

/opt/splunk

[root@search_head splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  6 04:36 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  6 04:06 var

[root@search_head splunk]#

Indexer_01 Machine

[root@indexer_01 splunk]# pwd

/opt/splunk

[root@indexer_01 splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  6 04:35 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  6 04:07 var

[root@indexer_01 splunk]#

Indexer_02 Machine

[root@indexer_02 splunk]# pwd

/opt/splunk

[root@indexer_02 splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  8 22:15 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  8 22:08 var

[root@indexer_02 splunk]#

Master Node Machine

[root@c_master_node splunk]# pwd

/opt/splunk

[root@c_master_node splunk]# ll

total 1800

drwxr-xr-x.  4  506  506    4096 Oct 30  2015 bin

-r--r--r--.  1  506  506      57 Oct 30  2015 copyright.txt

drwxr-xr-x. 16  506  506    4096 May  9 01:53 etc

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 include

drwxr-xr-x.  6  506  506    4096 Oct 30  2015 lib

-r--r--r--.  1  506  506   62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506     509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506    4096 Oct 30  2015 share

-r--r--r--.  1  506  506 1737206 Oct 30  2015 splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root    4096 May  9 01:49 var

[root@c_master_node splunk]#

Step 3> Install Universal Forwarder binaries into c_u_fwd_01 name assigned machine. (Extract the Splunk tar ball to the /opt folder as explained in the single node installation)

[root@splunk_standalone sbk_files]# tar -zxvf splunk-6.3.1-f3e41e4b37b2-Linux-x86_64.tgz -C /opt

C_U_Fwd_01 Machine

[root@u_fwd_01 splunkforwarder]# pwd

/opt/splunkforwarder

[root@u_fwd_01 splunkforwarder]# ll

total 132

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 bin

-r--r--r--.  1  506  506    57 Oct 30  2015 copyright.txt

drwxr-xr-x. 13  506  506  4096 May  6 04:08 etc

drwxr-xr-x.  2  506  506  4096 Oct 30  2015 include

drwxr-xr-x.  4  506  506  4096 Oct 30  2015 lib

-r--r--r--.  1  506  506 62027 Oct 30  2015 license-eula.txt

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 openssl

-r--r--r--.  1  506  506   509 Oct 30  2015 README-splunk.txt

drwxr-xr-x.  3  506  506  4096 Oct 30  2015 share

-r--r--r--.  1  506  506 31876 Oct 30  2015 splunkforwarder-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest

drwx--x--x.  6 root root  4096 May  6 04:08 var

[root@u_fwd_01 splunkforwarder]#

Configuring the instances into distributed deployment

Enable the master

To enable an indexer as the master node:

1. Click Settings in the upper right corner of Splunk Web.

2. In the Distributed environment group, click Clustering.

3. Select Enable clustering.

4. Select Master node and click Next.

5. There are a few fields to fill out:

•  Replication Factor.The replication factor determines how many copies of data the cluster maintains. The default is 3. For more information on the replication factor, see "Replication factor". Choose the right replication factor now. It is inadvisable to increase the replication factor later, once the cluster has significant amounts of data.
•  Search Factor. The search factor determines how many immediately searchable copies of data the cluster maintains. The default is 2. For more information on the search factor, see "Search factor". Choose the right search factor now. It is highly inadvisable to increase the search factor later, once the cluster has significant amounts of data.
•  Security Key. This is the key that authenticates communication between the master and the peers and search heads. The key must be the same across all cluster instances. If you leave the field empty here, leave it empty on the peers and search heads as well.

6. Click Enable master node.

7. The message appears, "You must restart Splunk for the master node to become active. You can restart Splunk from Server Controls." Click Go to Server Controls to go to the Settings page where you can initiate the restart.

Enable the peer

To enable an indexer as a peer node:

1. Click Settings in the upper right corner of Splunk Web.

2. In the Distributed environment group, click Clustering.

3. Select Enable clustering.

4. Select Peer node and click Next.

5. There are a few fields to fill out:

•  Master IP address or Hostname. Enter the master's IP address or hostname. For example: https://10.152.31.202.
•  Master port. Enter the master's management port. For example: 8089.
•  Peer replication port. This is the port on which the peer receives replicated data streamed from the other peers. You can specify any available, unused port for this purpose. This port must be different from the management port and receiving port.
•  Security key. This is the key that authenticates communication between the master and the peers and search heads. The key must be the same across all cluster instances. If the master has a security key, you must enter it here.

6. Click Enable peer node.

7. The message appears, "You must restart Splunk for the peer node to become active." Click Go to Server Controls to go to the Settings page where you can initiate the restart.

8. Repeat this process for all the cluster's peer nodes.

Enable the search head

To enable a Splunk instance as a cluster search head:

1. Click Settings in the upper right corner of Splunk Web.

2. In the Distributed environment group, click Clustering.

3. Select Enable clustering.

4. Select Search head node and click Next.

5. There are a few fields to fill out:

•  Master IP address or Hostname. Enter the master's IP address or hostname. For example: https://10.152.31.202.
•  Master port. Enter the master's management port. For example: 8089.
•  Security key. This is the key that authenticates communication between the master and the peers and search heads. The key must be the same across all cluster instances. If the master has a security key, you must enter it here.

6. Click Enable search head node.

7. The message appears, "You must restart Splunk for the search node to become active. You can restart Splunk from Server Controls" ClickGo to Server Controls to go to the Settings page where you can initiate the restart.

Use forwarders to get your data into the indexer cluster

Configure the connection from forwarder to peer node

There are three steps to setting up connections between forwarders and peer nodes:

1. Configure the peer nodes to receive data from forwarders.

2. Configure the forwarders to send data to the peer nodes.

3. Enable indexer acknowledgment for each forwarder. This step is required to ensure end-to-end data fidelity. If that is not a requirement for your deployment, you can skip this step

Example: A load-balancing forwarder with indexer acknowledgment

Here's a sample outputs.conf configuration for a forwarder that's using load balancing to send data in sequence to three peers in a cluster. It assumes that each of the peers has previously been configured to use 9997 for its receiving port:

[tcpout]
defaultGroup=my_LB_peers

[tcpout:my_LB_peers]
autoLBFrequency=40
server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997
useACK=true