Big Data - Terms and definition

What is Big Data?
Big Data refers to large sets of data that cannot be analyzed with traditional tools. It stands for data
related to large-scale processing architectures. Big Data refers to the data sets whose size makes it difficult for commonly used data capturing software tools to interpret, manage, and process them within a reasonable time frame.

What is Hadoop?
Hadoop is the software framework that is developed by Apache to support distributed processing of
data. Hadoop is an open-source framework built on the Java environment. It assists in the
processing of large data sets in a distributed computing environment.

MapReduce
The MapReduce is a core component of Hadoop, and is responsible for processing jobs in
distributed mode.

HDFS
The Hadoop Distributed File System (HDFS) is a distributed file system that shares some of
the features of other distributed file systems. It is used for storing and retrieving
unstructured data.

NameNode and DataNode
Hadoop is a master and slave architecture that includes the NameNode as the master and the
DataNode as the slave.

Pig
The Apache Pig is a platform which helps to analyze large datasets that includes high-level
language required to express data analysis programs. Pig is one of the components of the
Hadoop eco-system.

Hive
Hive is an open-source data warehousing system used to analyze a large amount of dataset
that is stored in Hadoop files. It has three key functions like summarization of data, query,
and analysis.

HBase
It is a distributed, column-oriented database built on top of HDFS (Hadoop Distributed
Filesystem). HBase can scale horizontally to thousands of commodity servers and petabytes
by indexing the storage.

ZooKeeper
It is used for performing region assignment. ZooKeeper is a centralized management service
for maintaining and configuring information, naming, providing distributed synchronization,
and group services.

Cloudera
It is a commercial tool for deploying Hadoop in an enterprise setup.

Sqoop
It is a tool that extracts data derived from non-Hadoop sources and formats them such that
the data can be used by Hadoop later

For Big Data Training- Contact eastbaytrng@gmail.com or (925) 322 3131

Big Plans for Big Data

Big Plans for Big Data
Big data's time is nearly here and many companies are preparing to pull the trigger on these projects in the coming months and years.

In a separate study released last month, Fortune 1000 senior business and technology executives surveyed by NewVantage Partners reported that their corporate investments in big data are projected to grow from 35 percent to 75 percent by 2017 for investments greater than $10 million, and by a remarkable 6 percent to 28 percent for investments greater than $50 million. In addition, 67 percent of executives report that they have big data initiatives running in production within the corporation, according to the report, Big Data Executive Survey 2014: An Update on the State of Big Data in the Large Corporate World.

http://newvantage.com/wp-content/uploads/2012/12/Big-Data-Survey-2014-Executive-Summary-110314.pdf

For Big Data training - Contact (925) 322 3131 or eastbaytrng@gmail.com

Why BIG Data?

Why Big Data?

Big Data is big deal - Harvard Magazine
Big data is big business.

International Data Corporation (IDC) forecasts that the Big Data technology and services market will grow at a 27% compound annual growth rate (CAGR) to $32.4 billion through 2017 - or about six times the growth rate of the overall information and communication technology (ICT) market.

The people who are buying 100 [terabytes] [of storage capacity] yesterday will buy 1,000 [TB] tomorrow.

Big Data archtect are paid $150/hour
Source: https://www.experfy.com/projects/category

Interview with a Data Scientist:
http://www.jobshadow.com/interview-with-a-data-scientist/

Big Data Salary - Sample from different companies
http://www.glassdoor.com/Salaries/big-data-engineer-salary-SRCH_KO0,17.htm

(Waav TCS $152k !!!)

Further details:
https://www.facebook.com/EastBayAcademy
Contact me or eastbaytrng@gmail.com for the new batch starting Jan-2015 via online.

Pre-requisite: If you know JAVA, useful. Otherwise we will give ~10-hours of Java training. Knowing any coding and interest in learning & consulting job is the KEY.

Big Data/Hadoop training is 30-hours - remote via Online.

** This is for somebody who is like me thinking of what is future of Peoplesoft after Work-Day and HCM Cloud & for people who want to switch technology using their existing skill-set. SQL and Java will be useful in Big Data.

Big Data and Hadoop Traininig !!

Big Data and Hadoop Training

RTSYS Inc

(925) 322 3131 email: eastbaytrng@gmail.com

Course Content:

1. > Understanding Big Data and Hadoop

Learning Objectives -  In this module, you will understand Big Data, the limitations of the existing solutions for Big Data problem, how Hadoop solves the Big Data problem, the common Hadoop ecosystem components, Hadoop Architecture, HDFS, Anatomy of File Write and Read, Rack Awareness.

                                                           
Topics -  Big Data, Limitations and Solutions of existing Data Analytics Architecture, Hadoop, Hadoop Features, Hadoop Ecosystem, Hadoop 2.x core components, Hadoop Storage: HDFS, Hadoop Processing: MapReduce Framework, Anatomy of File Write and Read, Rack Awareness.

2. > Hadoop Architecture and HDFS

Learning Objectives - In this module, you will learn the Hadoop Cluster Architecture, Important Configuration files in a Hadoop Cluster, Data Loading Techniques.

Topics - Hadoop 2.x Cluster Architecture - Federation and High Availability, A Typical Production Hadoop Cluster, Hadoop Cluster Modes, Common Hadoop Shell Commands, Hadoop 2.x Configuration Files, Password-Less SSH, MapReduce Job Execution, Data Loading Techniques: Hadoop Copy Commands, FLUME, SQOOP.

3. > Hadoop MapReduce Framework - I

Learning Objectives - In this module, you will understand Hadoop MapReduce framework and the working of MapReduce on data stored in HDFS. You will learn about YARN concepts in MapReduce.

Topics - MapReduce Use Cases, Traditional way Vs MapReduce way, Why MapReduce, Hadoop 2.x MapReduce Architecture, Hadoop 2.x MapReduce Components, YARN MR Application Execution Flow, YARN Workflow, Anatomy of MapReduce Program, Demo on MapReduce.

4. > Hadoop MapReduce Framework - II

Learning Objectives - In this module, you will understand concepts like Input Splits in MapReduce, Combiner & Partitioner and Demos on MapReduce using different data sets.

Topics - Input Splits, Relation between Input Splits and HDFS Blocks, MapReduce Job Submission Flow, Demo of Input Splits, MapReduce: Combiner & Partitioner, Demo on de-identifying Health Care Data set, Demo on Weather Data set.

5. > Advance MapReduce

Learning Objectives - In this module, you will learn Advance MapReduce concepts such as Counters, Distributed Cache, MRunit, Reduce Join, Custom Input Format, Sequence Input Format and how to deal with complex MapReduce programs.

Topics - Counters, Distributed Cache, MRunit, Reduce Join, Custom Input Format, Sequence Input Format.

6. > Pig

Learning Objectives - In this module, you will learn Pig, types of use case we can use Pig, tight coupling between Pig and MapReduce, and Pig Latin scripting.

Topics - About Pig, MapReduce Vs Pig, Pig Use Cases, Programming Structure in Pig, Pig Running Modes, Pig components, Pig Execution, Pig Latin Program, Data Models in Pig, Pig Data Types.

Pig Latin : Relational Operators, File Loaders, Group Operator, COGROUP Operator, Joins and COGROUP, Union, Diagnostic Operators, Pig UDF, Pig Demo on Healthcare Data set.

7. > Hive

Learning Objectives - This module will help you in understanding Hive concepts, Loading and Querying Data in Hive and Hive UDF. 

Topics - Hive Background, Hive Use Case, About Hive, Hive Vs Pig, Hive Architecture and Components, Metastore in Hive, Limitations of Hive, Comparison with Traditional Database, Hive Data Types and Data Models, Partitions and Buckets, Hive Tables(Managed Tables and External Tables), Importing Data, Querying Data, Managing Outputs, Hive Script, Hive UDF, Hive Demo on Healthcare Data set.

8. > Advance Hive and HBase

Learning Objectives - In this module, you will understand Advance Hive concepts such as UDF, dynamic Partitioning. You will also acquire in-depth knowledge of HBase, Hbase Architecture and its components.

Topics - Hive QL: Joining Tables, Dynamic Partitioning, Custom Map/Reduce Scripts, Hive : Thrift Server, User Defined Functions.

HBase: Introduction to NoSQL Databases and HBase, HBase v/s RDBMS, HBase Components, HBase Architecture, HBase Cluster Deployment.

9. > Advance HBase

Learning Objectives - This module will cover Advance HBase concepts. We will see demos on Bulk Loading , Filters. You will also learn what Zookeeper is all about, how it helps in monitoring a cluster, why HBase uses Zookeeper.

Topics - HBase Data Model, HBase Shell, HBase Client API, Data Loading Techniques, ZooKeeper Data Model, Zookeeper Service, Zookeeper, Demos on Bulk Loading, Getting and Inserting Data, Filters in HBase.

10. > Oozie and Hadoop Project

Learning Objectives - In this module, you will understand working of multiple Hadoop ecosystem components together in a Hadoop implementation to solve Big Data problems. We will discuss multiple data sets and specifications of the project. This module will also cover Flume & Sqoop demo and Apache Oozie Workflow Scheduler for Hadoop Jobs.

Topics - Flume and Sqoop Demo, Oozie, Oozie Components, Oozie Workflow, Scheduling with Oozie, Demo on Oozie Workflow, Oozie Co-ordinator, Oozie Commands, Oozie Web Console, Hadoop Project Demo.

Mini Project

At the end of the course, you will be working on a project where you be expected to perform Big Data Analytics using Map Reduce, Pig, Hive & HBase. You will get practical exposure about Data Loading techniques in Hadoop using Flume and SQOOP. You will understand how Oozie is used to schedule and manage Hadoop Jobs. You will also understand how the Hadoop Project environment is setup and how the Test environment is setup.

RTSYS Inc

(925) 322 3131 email: eastbaytrng@gmail.com