Coudera CDH installation using remote repo

Hadoop.CDH.Installation.RemoteRepository

1.     Overview

This topic describes how to create a remote RPM packages/parcels repository and direct hosts in your Cloudera Manager deployment to use that repository.

Once you have created a parcels repository, go to Configuring the Cloudera Manager Server to Use the Parcel URL. After completing these steps, you have established the environment required to install a previous version of Cloudera Manager or install Cloudera Manager to hosts that are not connected to the Internet. Proceed with the installation process, being sure to target the newly created repository.

2.     Creating a Permanent Remote Repository

The repository is typically hosted using HTTP on a host inside your network. If you already have a web server in your organization, you can reuse it and put the parcel files into it.

Below are the detailed steps to setup a permanent remote repository:

1.      Logon the server you want to setup the web server and run below commands to install Apache httpd web server

yum install httpd

systemctl start httpd

systemctl enable httpd

·    RPM Packages

 Download the RPM packages for your OS distribution from:

·         CDH 5 http://archive.cloudera.com/cm5/repo-as-tarball/

 Move the RPM packages files to the web server directory, and modify file permissions

mkdir -p /var/www/html/cdh5/packages

tar -xvf cm5.*-centos7.tar.gz -C /var/www/html/cdh5/parcels

chmod -R ugo+rX /var/www/html/cdh5  (might not require)

3.      After moving the files and changing permissions, visit http://hostname:80/cdh5/parcels to verify that you can access the RPM packages. Apache may have been configured to not show indexes, which is also acceptable.

·    Parcels

a.      Download the parcel and manifest.json files for your OS distribution from:

·         CDH 5 - Impala, Spark, and Search are included in the CDH parcel

·         CDH - https://archive.cloudera.com/cdh5/parcels/

·         Accumulo - - https://archive.cloudera.com/accumulo-c5/parcels/

·         GPL Extras - https://archive.cloudera.com/gplextras5/parcels/

b.     Move the .parcel and manifest.json files to the web server directory, and modify file permissions

mkdir -p /var/www/html/cdh5/parcels

mv CDH-5.*-el7.parcel /var/www/html/cdh5/parcels

mv manifest.json /var/www/html/cdh5/parcels

chmod -R ugo+rX /var/www/html/cdh5  (might not require)

·         After moving the files and changing permissions, visit http://hostname:80/cdh5/parcels to verify that you can access the parcel. Apache may have been configured to not show indexes, which is also acceptable.

3.     Configuring the Cloudera Manager Server to Use the Parcel URL

1.      Use one of the following methods to open the parcel settings page:

·         Navigation bar

·   

     

·         Click the Configuration button.

·         Menu

·         Select Administration > Settings

·         Select Category > Parcels

2.      In the Remote Parcel Repository URLs list, click  to open an additional row.

3.      Enter the path to the parcel. For example, http://hostname:port/cdh5/parcels/.

4.      Click Save Changes to commit the changes.

4.     Reference

Creating and Using a Remote Parcel Repository for Cloudera Manager

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_parcel_repo.html

An Introduction to Big Data

An Introduction to Big Data

What is BigData?

Big data is a buzzword, or one can say it’s a catch-phrase, which can be used to describe a huge volume of structured, unstructured, text, images, audio, video, log files, emails, simulations, 3D models, military surveillance, e-commerce and so on that is so massive that it's difficult to process using traditional database and software techniques. In most enterprise scenarios the data is too big or it moves too fast or it exceeds current processing capacity. Big data is nothing but a synonym of a huge and complex data that it becomes very tiresome, difficult or slow to collect, store, sort, process, retrieve and analyze it with the help of any existing relational database management tools or traditional data processing techniques. Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time.

Some examples of Big Data:

·         An airline jet collects 10 terabytes of sensor data for every 30 minutes of flying time.

·         Twitter has over 500 million registered users.

1.      The USA, whose 141.8 million accounts represents 27.4 percent of all Twitter users, good enough to finish well ahead of Brazil, Japan, the UK and Indonesia.

2.      79% of US Twitter users are more like to recommend brands they follow.

3.      67% of US Twitter users are more likely to buy from brands they follow.

4.      57% of all companies that use social media for business use Twitter.

How fast data is increasing:

Carefully look the picture which explains us “what happens in every 60 seconds on the internet“. By this we can understand how much data being generated in a second, a minute, a day or a year and how exponentially it is generating. As per the analysis by TechNewsDaily we might generate more than 8 Zettabytes of data by 2015.

Characteristics of Big Data:

Data scientists break big data into four dimensions: volume, variety, velocity and veracity.

·         Volume: BIG DATA depends upon how large it is. It could amount to hundreds of terabytes or even petabytes of information.

·         Velocity: The increasing rate at which data flows into an organization.

·         Variety: A common theme in big data systems is that the source data is diverse and doesn’t fall into neat relational structures.

·         Veracity: Big Data Veracity refers to the biases, noise and abnormality in data. Is the data that is being stored, and mined meaningful to the problem being analyzed.

Big Data Problems:

Traditional systems build within the company for handling the relational databases may not be able to support/scale as data generating with high volume, velocity and variety of data.

·         Volume: As an example, Terabytes of posts generated on Facebook or 400 billion annual twitter tweets could mean Big Data! This enormous amount of data will be stored somewhere to analyze and come up with data science reports for different solutions and problem solving approaches.

·         Velocity: Big data requires fast processing. Time factor plays a very crucial role in several organizations. For instance, millions of records are generated in the stock market which needs to be stored and processed with the same speed as its coming into the system.

·         Variety: There is no specific format of Big Data. It could be in any form such as structured, unstructured, text, images, audio, video, log files, emails, simulations, 3D models, etc. Until now we have been working with only structured data. It might be difficult to handle the quality and quantity of unstructured or semi-structured data that we are generating on a daily basis.

How Big Data handles the above problems:

·         Distributed File System (DFS): In DFS, we can divide a large set of data files into smaller blocks and load these blocks into multiple number of machines which will then be ready for parallel processing. For example, if we have 1 Terabyte of data to read with 1 machine and 4 Input/Output channels with each channel’s reading speed id 100MB/sec, the whole 1 TB data will be read in 45 minutes. On the other hand, if we have 10 different machines, we can divide 1 TB of data into 10 machines and then the data can be read in parallel which reduces the total time to only 4.5 minutes.

·         Parallel Processing: When data resides on N number of servers and holds the power of N servers, then the data can be processed in parallel for analysis, which helps the user to reduce the wait time to generate the final report or analyzed data.

·         Fault Tolerance: The Fault tolerance feature of Big Data frameworks (like Hadoop) is the one of the main reason for using this framework to run jobs. Even when running jobs on a large cluster where individual nodes or network components may experience high rates of failure, BigData frameworks can guide jobs toward a successful completion as the data is replicated into multiple nodes/slaves.

·         Use of Commodity Hardware: Most of the Big Data tools and frameworks need commodity hardware for its working which reduces the cost of the total infrastructure and very easy to add more clusters as data size increase.

Cloudera CDH insttaion using Isilon storage

Hadoop.CDH.Installation.Isilon

1.     Introduction

This document describes how to create a Hadoop environment utilizing the Cloudera CDH and EMC Isilon Scale-Out NAS for HDFS accessible shared storage.

The nodes in Isilon OneFS system work together as peers in a shared-nothing hardware architecture with no single point of failure. Each nodes acts as a Hadoop name node and data node, the name node daemon is a distributed process that runs on all the nodes in the cluster. A compute client can connect to any node through HDFS.

As nodes are added, the file system expands dynamically and redistributes data.

2.     Environment

This installation guide is appropriate for the below environment.

·         Cloudera CDH 5.7.X

·         VMware vSphere 5.5 or later

·         RHEL/CentOS 6.7

·         Internet Explorer 10 or later

·         Isilon OneFS 8.0.0 or later

·         Installation

1.     Overview

Below is the overview of the installation process that this document will describe.

·    Confirm prerequisites

·    Install Isilon OneFS

·    Configure Isilon OneFS

·    Use Cloudera Manager to deploy CDH cluster upon Isilon OneFS

·    Validate CDH deployment

·         Confirm Prerequisites

1.     Prepare VMware virtualized environment

Before you start the installation process, a VMware virtualized environment must be ready to provision the virtual machines required for this deployment. ESXi 5.5 and later revision is recommend for the virtualized environment.

·    Prepare Cloudera Manager Server and Cluster Nodes

Please prepare the Cloudera Manager server and cluster nodes based on the instructions in note "Hadoop.CDH.Installation.PathB"

3.Isilon OneFS

For low-capacity, non-performance testing of Isilon, the EMC Isilon OneFS Simulator can be used instead of a cluster of physical Isilon appliances.

4.Networking

·         10 Gbe Ethernet is required

·         If using EMC Isilon Simulator, at least two static IP addresses are required, one for node ext-1 interface, another for the SmartConnect service IP), each additional Isilon node will require an additional IP address.

·         At a minimum, you will need to allocate one IP address per Access Zone per Isilon node.

·         # of IP addresses = 2 * (# of Isilon Nodes) * (# of Access Zones)

3.     Install Isilon OneFS

In this document, Isilon Simulator 8.0.0.1 will be used to setup a free and non-production use Isilon OneFS 8.0.0 cluster environment for the deployment of CDH 5.7.X.

You can download Isilon Simulator from this link: http://www.emc.com/products-solutions/trial-software-download/isilon.htm

For the detailed installation process of Isilon Simulator, please refer the instruction in note "Isilon.OneFS.Simulator.Installation"

4.     Configure Isilon OneFS

·         Add the Isilon Simulator nodes hostname and IP information to the named server configuration file

·         Add below content to file /var/named/named-forward.zone

cdh-isilon      IN      A       172.16.1.20

isilon02        IN      A       172.16.1.21

·         Add below content to file /var/named/named-reverse.zone

20              IN      PTR     cdh-isilon.bigdata.emc.local.

21              IN      PTR     isilon02.bigdata.emc.local.

2.Add license to activate Isilon Simulator HDFS module, the license key listed in below command will be expired by 7/17/2015. Before add the license, you need to change the current date in Isilon Simulator node to apply this license

date 1501010001

isi license licenses activate ACCEL-34PS2-32FWX-RNIWX-LLADX

isi license licenses list

3.Run Isilon Hadoop Tools scripts on Isilon Simulator node to create required users and directories

·         Download Isilon Hadoop Tools script from https://github.com/claudiofahey/isilon-hadoop-tools/releases

·         Upload isilon_create_users.sh and isilon_create_directories.sh on Isilon Simulator node

·         Run these two scripts

bash ./isilon_create_users.sh --dist cdh

bash ./isilon_create_directories.sh --dist cdh --fixperm

4.Run below commands to map the hdfs user to the Isilon super user, this will allow the hdfs user to chown all files

isi zone zones modify System --user-mapping-rules="hdfs=>root"

isi services hdfs disable

isi services hdfs enable

5.     Use Cloudera Manager to deploy CDH cluster upon Isilon OneFS

·         Login Cloudera Manager Admin Console http://cdh-manager.bigdata.emc.local:7180

The default port of CMS is 7180

The default user account is: admin/admin

After login, accept the EULA and click "Continue"

2.Choose Cloudera Manager Edition

From the Welcome to Cloudera Manager page, you can select the edition of Cloudera Manager to install

3.Specify hosts for your CDH cluster installation

Specify the hosts for your CDH cluster

cdh-master01.bigdata.emc.local

cdh-worker01.bigdata.emc.local

cdh-worker02.bigdata.emc.local

cdh-worker03.bigdata.emc.local

4.Cluster Installation - Select Repository

·         Use Parcels

·         Parcel Directory

/opt/cloudera/parcels

·         Local Parcel Repository Path

/opt/cloudera/parcel-repo

·         Remote Parcel Repository URLs

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/accumulo

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/cdh5

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/impala

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/kafka

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/keytrustee

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/search

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/spark

http://repo.bigdata.emc.local/cloudera-cdh5-parcel/cdrom/sqoop

·         Select the specific release of the Cloudera Manager Agent you want to install on your hosts

http://repo.bigdata.emc.local/cloudera-manager-rpm/ 

·         Use Packages (Preferred)

·         Select the version of CDH

·         Select the specific release of CDH you want to install on your hosts (Custom Repository )

http://repo.bigdata.emc.local/cloudera-cdh5-rpm/

·         Select the specific release of the Cloudera Manager Agent you want to install on your hosts (Custom Repository )

http://repo.bigdata.emc.local/cloudera-manager-rpm/ 

·    Cluster Installation - JDK Installation Options

·         Install Oracle Java SE Development Kit (JDK)

Unchecked (already installed in VM template)

·         Install Java Unlimited Strength Encryption Policy Files

Unchecked (already installed in VM template)

·    Cluster Installation - Enable Single User Mode

·         Single User Mode

Unchecked

7.Cluster Installation - Provide SSH login credentials

·         Password

·         Private Key

8.Cluster Installation - Installing

Click "Continue"

9.Cluster Installation - Detecting CDH versions on all hosts

Click "Continue"

10. Cluster Installation -Inspect hosts for correctness

Click "Finish"

11. Cluster Setup - Choose the CDH 5 services that you want to install on your cluster

·    Custom Services

·         Isilon

·         YARN

·         ZooKeeper

Note:

Do not select HDFS or Cloudera Navigator.

Isilon takes the place of the usual HDFS service. Cloudera Navigator is not currently supported with Isilon HDFS.

·    Cluster Setup - Customize Role Assignments

·    cdh-master01.bigdata.emc.local

·         YARN Resource Manager

·         Isilon Gateway

·    cdh-worker01~03.bigdata.emc.local

·         YARN Node Manager

·         HBase Region Server (If Installed)

·         Impala Daemon (If Installed)

Click "Continue"

·    Cluster Setup - Database Setup

Check "Use Embedded Database" and click "Continue"

14. Cluster Setup - Review Changes

·    Default File System URI

hdfs://isilon02.bigdata.emc.local:8020

hdfs://cdh-isilon.bigdata.emc.local:8020

·    WebHDFS URL

http://isilon02.bigdata.emc.local:8082/webhdfs/v1

http://cdh-isilon.bigdata.emc.local:8082/webhdfs/v1

Click "Continue"

·    Cluster Setup - First Run Command

Click "Continue"

16. Cluster Setup - Congratulations!

Click "Finish"

6.     Validate CDH deployment

1.     Login CDH Master Compute node and run below command to validate Hadoop Cluster

clear &&

hdfs dfs -ls / &&

hdfs dfs -put -f /etc/hosts /tmp &&

hdfs dfs -ls /tmp &&

hdfs dfs -cat /tmp/hosts &&

hdfs dfs -rm -skipTrash /tmp/hosts &&

hdfs dfs -ls /tmp &&

(sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 1000 ||

sudo -u hdfs yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 1000) &&

echo "Done"

4.     References:

1.      EMC Isilon Hadoop Starter Kit for Cloudera with VMware Big Data Extensions

http://hsk-docs.readthedocs.io/projects/hsk-cdh/en/latest/hsk.html

1.      EMC Isilon Best Practices for Hadoop Data Storage

http://www.emc.com/collateral/white-paper/h12877-wp-emc-isilon-hadoop-best-practices.pdf