Wednesday, March 30, 2016

MongoDB Replicaset installation on VMs

MongoDB Replicaset installation on VMs

1.       Create 3 VMs and install RHEL
[root@node2 ~]# uname -a
Linux node2.rs05.mongodb 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@node2 ~]#

2.       Add static IP ( Private IP in our case) to all the 3VMs
[root@node2 ~]# more /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.0.45
NETMAST=255.255.255.0
GATEWAY=192.168.0.1
DNS1=192.168.0.1
[root@node2 ~]#

3.       Assign hostname to all the 3VMs in /etc/sysconfig/network file.
[root@node2 ~]# hostname
node2.rs05.mongodb
[root@node2 ~]#

4.       Map 3VMs static IP and hostname details in the /etc/hosts file
[root@node1 ~]# more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.44    node1.rs05.mongodb
192.168.0.45    node2.rs05.mongodb
192.168.0.46    node3.rs05.mongodb
[root@node1 ~]#

5.       Disable the firewall (iptables)  /etc/sysconfig/iptables
[root@node1 ~]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@node1 ~]#

6.       Download MongoDB tar package and extract it and rename as mongodb
[root@node1 ~]# ll
total 44
-rw-------. 1 root root  1385 Mar 13 23:40 anaconda-ks.cfg
-rw-r--r--. 1 root root 28054 Mar 13 23:40 install.log
-rw-r--r--. 1 root root  7572 Mar 13 23:40 install.log.syslog
drwxr-xr-x. 3 root root  4096 Mar 14 03:20 mongo


7.       Create new mongodb folder under /usr/local/mongodb and copy mongo contents into it.
[root@node1 local]# ll mongodb/
total 100
drwxr-xr-x. 2 root root  4096 Mar 14 03:28 bin
-rw-r--r--. 1 root root 34520 Mar 14 03:28 GNU-AGPL-3.0
-rw-r--r--. 1 root root 16726 Mar 14 03:28 MPL-2
-rw-r--r--. 1 root root  1359 Mar 14 03:28 README
-rw-r--r--. 1 root root 35910 Mar 14 03:28 THIRD-PARTY-NOTICES
[root@node1 local]#

8.       Create configuration file under /etc/mongo.conf and all db, log, replicaset and mongo.pid details.
[root@node1 local]# more /etc/mongodb.conf
logpath=/var/log/mongodb/mongod.log
logappend=true
fork=true
port=27017
dbpath=/var/lib/mongodb
pidfilepath=/var/run/mongodb/mongod.pid
replSet=RS05
oplogSize=1024
[root@node1 local]#

9.       Add mongodb binaries into PATH variable  in bash profile file
[root@node1 local]# more ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/mongodb/bin

export PATH
[root@node1 local]#

10.   Start the mongod instance in all the VMs (nodes)
[root@node1 local]# mongod -f /etc/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 4631
child process started successfully, parent exiting
[root@node1 local]#

11.   Start the mongo in node one and add the remaining 2 nodes into the replica set.
$mongo
> config={_id:"RS05", members:[{_id:0, host:"node1.rs05.mongodb:27017", priority:3}, {_id:1, host:"node2.rs05.mongodb:27017", priority:2}, {_id:2, host:"node3.rs05.mongodb:27017", priority:1}]}
> rs.initiate(config)
>rs.status()
RS05:PRIMARY> rs.status()
{
        "set" : "RS05",
        "date" : ISODate("2016-03-15T09:32:19.938Z"),
        "myState" : 1,
        "term" : NumberLong(4),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "node1.rs05.mongodb:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 180,
                        "optime" : {
                                "ts" : Timestamp(1458034175, 2),
                                "t" : NumberLong(4)
                        },
                        "optimeDate" : ISODate("2016-03-15T09:29:35Z"),
                        "electionTime" : Timestamp(1458034175, 1),
                        "electionDate" : ISODate("2016-03-15T09:29:35Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "node2.rs05.mongodb:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 179,
                        "optime" : {
                                "ts" : Timestamp(1458034175, 2),
                                "t" : NumberLong(4)
                        },
                        "optimeDate" : ISODate("2016-03-15T09:29:35Z"),
                        "lastHeartbeat" : ISODate("2016-03-15T09:32:19.432Z"),
                        "lastHeartbeatRecv" : ISODate("2016-03-15T09:32:19.544Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "node1.rs05.mongodb:27017",
                        "configVersion" : 1
                },
                {
                        "_id" : 2,
                        "name" : "node3.rs05.mongodb:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 179,
                        "optime" : {
                                "ts" : Timestamp(1458034175, 2),
                                "t" : NumberLong(4)
                        },
                        "optimeDate" : ISODate("2016-03-15T09:29:35Z"),
                        "lastHeartbeat" : ISODate("2016-03-15T09:32:19.432Z"),
                        "lastHeartbeatRecv" : ISODate("2016-03-15T09:32:19.545Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "node1.rs05.mongodb:27017",
                        "configVersion" : 1
                }
        ],
        "ok" : 1
}
RS05:PRIMARY>


12.   Now the replica set is ready to play.

Wednesday, March 23, 2016

MongoDB Standalone Installation

MongoDB Standalone Installation best practice and reference guide.

Install MongoDB Community Edition on Red Hat Enterprise 6.6 using .rpm package

MongoDB provides officially supports packages in their own repository.

Ø  mongodb-org             A metapackage that will automatically install the four component packages listed below.
Ø  mongodb-org-server Contains the mongod daemon and associated configuration and init scripts.
Ø  mongodb-org-mongos   Contains the mongos daemon.
Ø  mongodb-org-shell    Contains the mongo shell.
Ø  mongodb-org-tools   Contains the following MongoDB tools: mongoimport bsondump, mongodump, mongoexport, mongofiles, mongooplog, mongoperf, mongorestore, mongostat, and mongotop.


The default /etc/mongod.conf configuration file supplied by the packages has bind_ip set to 127.0.0.1 by default. Modify this setting as needed for your environment before initializing a replica set.
The mongodb-org package includes various init scripts, including the init script /etc/rc.d/init.d/mongod. You can use these scripts to stop, start, and restart daemon processes.
            Ex: sudo service mongod start/stop/restart


Installing MongoDB
1.      Configure yum (package management system), here I’m using 2mongodb-org-2.6
Create a repo file /etc/yum.repos.d/mongodb-org-2.6.repo
And below details in the repo file
[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/
gpgcheck=0
enabled=1
2.      Install MongoDB by issuing repo install cmd
Sudo yum install –y mongodb-org

*Recommended to install complete package, if you want install each package individually then specify each component individually and append the version number to the package name.
*You can configure SELinux to allow MongoDB to start on Red Hat Linux-based systems, by updated SELINUX variable in /etc/selinux/config but not recommended.


Default Data Directories in MongoDB
Ø  Data Files are stored in /var/lib/mongo
Ø  Log files are stored in /var/log/mongodb
If you change the user that runs the MongoDB process, you must modify the access control rights to the /var/lib/mongo and /var/log/mongodb directories to give this user access to these directories.


Simple Start Stop and Restart CMDs
$service mongod start                    Check log to see service started and default port is in listening mode
$sudo service mongod stop                Check log to see if service is shut down and port released.
You can follow the state of the process for errors or important messages by watching the output in the /var/log/mongodb/mongod.log file







Thursday, March 17, 2016

Coudera CDH installation using remote repo

Hadoop.CDH.Installation.RemoteRepository
1.     Overview
This topic describes how to create a remote RPM packages/parcels repository and direct hosts in your Cloudera Manager deployment to use that repository.

Once you have created a parcels repository, go to Configuring the Cloudera Manager Server to Use the Parcel URL. After completing these steps, you have established the environment required to install a previous version of Cloudera Manager or install Cloudera Manager to hosts that are not connected to the Internet. Proceed with the installation process, being sure to target the newly created repository.

2.     Creating a Permanent Remote Repository
The repository is typically hosted using HTTP on a host inside your network. If you already have a web server in your organization, you can reuse it and put the parcel files into it.

Below are the detailed steps to setup a permanent remote repository:

1.      Logon the server you want to setup the web server and run below commands to install Apache httpd web server
yum install httpd
systemctl start httpd
systemctl enable httpd
·    RPM Packages
 Download the RPM packages for your OS distribution from:
 Move the RPM packages files to the web server directory, and modify file permissions
mkdir -p /var/www/html/cdh5/packages
tar -xvf cm5.*-centos7.tar.gz -C /var/www/html/cdh5/parcels
chmod -R ugo+rX /var/www/html/cdh5  (might not require)
3.      After moving the files and changing permissions, visit http://hostname:80/cdh5/parcels to verify that you can access the RPM packages. Apache may have been configured to not show indexes, which is also acceptable.

·    Parcels
a.      Download the parcel and manifest.json files for your OS distribution from:
·         CDH 5 - Impala, Spark, and Search are included in the CDH parcel
·         Accumulo - - https://archive.cloudera.com/accumulo-c5/parcels/
·         GPL Extras - https://archive.cloudera.com/gplextras5/parcels/
b.     Move the .parcel and manifest.json files to the web server directory, and modify file permissions
mkdir -p /var/www/html/cdh5/parcels
mv CDH-5.*-el7.parcel /var/www/html/cdh5/parcels
mv manifest.json /var/www/html/cdh5/parcels
chmod -R ugo+rX /var/www/html/cdh5  (might not require)
·         After moving the files and changing permissions, visit http://hostname:80/cdh5/parcels to verify that you can access the parcel. Apache may have been configured to not show indexes, which is also acceptable.

3.     Configuring the Cloudera Manager Server to Use the Parcel URL
1.      Use one of the following methods to open the parcel settings page:
·         Navigation bar
·   
     
·         Click the Configuration button.
·         Menu
·         Select Administration > Settings
·         Select Category > Parcels
2.      In the Remote Parcel Repository URLs list, click  to open an additional row.
3.      Enter the path to the parcel. For example, http://hostname:port/cdh5/parcels/.
4.      Click Save Changes to commit the changes.



4.     Reference
Creating and Using a Remote Parcel Repository for Cloudera Manager



Wednesday, March 2, 2016

An Introduction to Big Data

An Introduction to Big Data
What is BigData?
Big data is a buzzword, or one can say it’s a catch-phrase, which can be used to describe a huge volume of structured, unstructured, text, images, audio, video, log files, emails, simulations, 3D models, military surveillance, e-commerce and so on that is so massive that it's difficult to process using traditional database and software techniques. In most enterprise scenarios the data is too big or it moves too fast or it exceeds current processing capacity. Big data is nothing but a synonym of a huge and complex data that it becomes very tiresome, difficult or slow to collect, store, sort, process, retrieve and analyze it with the help of any existing relational database management tools or traditional data processing techniques. Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time.
Some examples of Big Data:
·         An airline jet collects 10 terabytes of sensor data for every 30 minutes of flying time.
·         Twitter has over 500 million registered users.
1.      The USA, whose 141.8 million accounts represents 27.4 percent of all Twitter users, good enough to finish well ahead of Brazil, Japan, the UK and Indonesia.
2.      79% of US Twitter users are more like to recommend brands they follow.
3.      67% of US Twitter users are more likely to buy from brands they follow.
4.      57% of all companies that use social media for business use Twitter.
How fast data is increasing:
Carefully look the picture which explains us “what happens in every 60 seconds on the internet“. By this we can understand how much data being generated in a second, a minute, a day or a year and how exponentially it is generating. As per the analysis by TechNewsDaily we might generate more than 8 Zettabytes of data by 2015.




Characteristics of Big Data:
Data scientists break big data into four dimensions: volume, variety, velocity and veracity.
·         Volume: BIG DATA depends upon how large it is. It could amount to hundreds of terabytes or even petabytes of information.
·         Velocity: The increasing rate at which data flows into an organization.
·         Variety: A common theme in big data systems is that the source data is diverse and doesn’t fall into neat relational structures.
·         Veracity: Big Data Veracity refers to the biases, noise and abnormality in data. Is the data that is being stored, and mined meaningful to the problem being analyzed.




Big Data Problems:
Traditional systems build within the company for handling the relational databases may not be able to support/scale as data generating with high volume, velocity and variety of data.
·         Volume: As an example, Terabytes of posts generated on Facebook or 400 billion annual twitter tweets could mean Big Data! This enormous amount of data will be stored somewhere to analyze and come up with data science reports for different solutions and problem solving approaches.
·         Velocity: Big data requires fast processing. Time factor plays a very crucial role in several organizations. For instance, millions of records are generated in the stock market which needs to be stored and processed with the same speed as its coming into the system.
·         Variety: There is no specific format of Big Data. It could be in any form such as structured, unstructured, text, images, audio, video, log files, emails, simulations, 3D models, etc. Until now we have been working with only structured data. It might be difficult to handle the quality and quantity of unstructured or semi-structured data that we are generating on a daily basis.
How Big Data handles the above problems:
·         Distributed File System (DFS): In DFS, we can divide a large set of data files into smaller blocks and load these blocks into multiple number of machines which will then be ready for parallel processing. For example, if we have 1 Terabyte of data to read with 1 machine and 4 Input/Output channels with each channel’s reading speed id 100MB/sec, the whole 1 TB data will be read in 45 minutes. On the other hand, if we have 10 different machines, we can divide 1 TB of data into 10 machines and then the data can be read in parallel which reduces the total time to only 4.5 minutes.
·         Parallel Processing: When data resides on N number of servers and holds the power of N servers, then the data can be processed in parallel for analysis, which helps the user to reduce the wait time to generate the final report or analyzed data.
·         Fault Tolerance: The Fault tolerance feature of Big Data frameworks (like Hadoop) is the one of the main reason for using this framework to run jobs. Even when running jobs on a large cluster where individual nodes or network components may experience high rates of failure, BigData frameworks can guide jobs toward a successful completion as the data is replicated into multiple nodes/slaves.
·         Use of Commodity Hardware: Most of the Big Data tools and frameworks need commodity hardware for its working which reduces the cost of the total infrastructure and very easy to add more clusters as data size increase.