Mapr (HP) Hadoop Developer Certification Questions and Answers (Dumps and Practice Questions)

Question :
You have the following key-value pairs as output from your Map task:
(HadoopExam, 1)
(Is, 1)
(the, 1)
(best, 1)
(material, 1)
(provider, 1)
(for, 1)
(the, 1)
(Hadoop, 1)
How many keys will be passed to the Reducer's reduce() method?

1. 9
2. 8
3. Access Mostly Uused Products by 50000+ Subscribers
4. 6
5. 5

Correct Answer : Get Lastest Questions and Answer :
Explanation: Picking the appropriate size for the tasks for your job can radically change the performance of Hadoop. Increasing the number of tasks increases the
framework overhead, but increases load balancing and lowers the cost of failures. At one extreme is the 1 map/1 reduce case where nothing is distributed. The other extreme is to have
1,000,000 maps/ 1,000,000 reduces where the framework runs out of resources for the overhead.
Number of Maps : The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number
of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile,
so it is best if the maps take at least a minute to execute. Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for
the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the
input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size. Thus, if you expect 10TB of input data and have 128MB
DFS blocks, you'll end up with 82k maps, unless your mapred.map.tasks is even larger. Ultimately the InputFormat determines the number of maps. The number of map tasks can also be
increased manually using the JobConf's conf.setNumMapTasks(int num). This can be used to increase the number of map tasks, but will not set the number below that which Hadoop
determines via splitting the input data. Number of Reduces : The ideal reducers should be the optimal value that gets them closest to:

* A multiple of the block size * A task time between 5 and 15 minutes * Creates the fewest files possible
Anything other than that means there is a good chance your reducers are less than great. There is a tremendous tendency for users to use a REALLY high value ("More parallelism means
faster!") or a REALLY low value ("I don't want to blow my namespace quota!"). Both are equally dangerous, resulting in one or more of:
* Terrible performance on the next phase of the workflow * Terrible performance due to the shuffle * Terrible overall performance because you've overloaded the namenode with objects
that are ultimately useless * Destroying disk IO for no really sane reason * Lots of network transfers due to dealing with crazy amounts of CFIF/MFIF work Now, there are always
exceptions and special cases. One particular special case is that if following that advice makes the next step in the workflow do ridiculous things, then we need to likely 'be an
exception' in the above general rules of thumb. Currently the number of reduces is limited to roughly 1000 by the buffer size for the output files (io.buffer.size * 2 * numReduces
less than heapSize). This will be fixed at some point, but until it is it provides a pretty firm upper bound. The number of reduce tasks can also be increased in the same way as the
map tasks, via JobConf's conf.setNumReduceTasks(int num).When the number of reduce tasks is set to zero, no reduce tasks are executed for that job. The intermediate data produced by
the map phase is copied into HDFS as the output without modification. The intermediate data from each mapper becomes a single output file in HDFS. For more information about running
a job with zero reducersThe Reducer collects all the values associated with a given key together, and passes them to the reduce() method in a single call to that method. Because the
key 'the' appears twice in the list, the reduce() method will be called once for that key, with a list of the two values. Each other keys appear only once, so there will be a total
of six calls to the reduce() method, one for each unique key. For more information, see chapter 6 of Hadoop: The Definitive Guide, 3rd Edition in the Shuffle and Sort: The Reducer
Side section.

Watch the training from http://hadoopexam.com/index.html/#hadoop-training

Question : While processing the file using MapReduce framework, the output of the Mapper which we call as
intermediate key-value pairs, select the correct statement for this output of the mappers.

1. Intermediate key-value pairs are written to the HDFS of the machines running the map tasks, and then copied to the machines running the reduce tasks.
2. Intermediate key-value pairs are written to the local disks of the machines running the reduce tasks.
3. Access Mostly Uused Products by 50000+ Subscribers
4. Intermediate key-value pairs are written to the local disks of the machines running the map tasks, and then read by the machines running the reduce tasks.

Correct Answer : Get Lastest Questions and Answer :

Explanation: How is intermediate data organized? There are 2 kinds of intermediate data, intermediate data on the mapper and intermediate data on the reducer. Intermediate data on the
mapper is map from key to value. Intermediate data on reducer are ALL values of certain set of keys, partitioned by the key. If you want even more details than this then it goes into
the file system organization, than I would redirect you to read the source code of Hadoop. I am not sure you need to know that, you might but I am not you so I don't know your need.
What happens when there is too much data about a key to be held by a single file? Then you keep them in multiple files BUT when you call the reducer on a certain key, you must be
able to get access to all of them (all values of any key)All intermediate output generated by the mappers is written to local disk. Because it is intermediate data, storing it in
HDFS with replication would be excessive. When the data is ready for transfer, the reduce tasks will copy the data from the nodes that ran the map tasks.

Watch the training from http://hadoopexam.com/index.html/#hadoop-training

Question : HadoopExam stores everyday, the users IP address+location as a string in the file as well as
number of total clicks as an Integer (Incremented for each click) and this is quite huge file,
where the keys are strings (address+location), and the values are integers (clicks).
For each unique key, you want to identify the largest integer. In writing a MapReduce program to accomplish this,
using the combine is advantageous ?

1. Yes
2. No
3. Access Mostly Uused Products by 50000+ Subscribers
4. Yes, if configured while cluster setup

Correct Answer : Get Lastest Questions and Answer :

Explanation: Combiner: The pipeline showed earlier omits a processing step which can be used for optimizing bandwidth usage by your MapReduce job. Called the Combiner, this pass runs after
the Mapper and before the Reducer. Usage of the Combiner is optional. If this pass is suitable for your job, instances of the Combiner class are run on every node that has run map
tasks. The Combiner will receive as input all data emitted by the Mapper instances on a given node. The output from the Combiner is then sent to the Reducers, instead of the output
from the Mappers. The Combiner is a "mini-reduce" process which operates only on data generated by one machine.
Word count is a prime example for where a Combiner is useful. The Word Count program in listings 1--3 emits a (word, 1) pair for every instance of every word it sees. So if the same
document contains the word "cat" 3 times, the pair ("cat", 1) is emitted three times; all of these are then sent to the Reducer. By using a Combiner, these can be condensed into a
single ("cat", 3) pair to be sent to the Reducer. Now each node only sends a single value to the reducer for each word -- drastically reducing the total bandwidth required for the
shuffle process, and speeding up the job. The best part of all is that we do not need to write any additional code to take advantage of this! If a reduce function is both commutative
and associative, then it can be used as a Combiner as well. You can enable combining in the word count program by adding the following line to the driver:

conf.setCombinerClass(Reduce.class);
The Combiner should be an instance of the Reducer interface. If your Reducer itself cannot be used directly as a Combiner because of commutativity or associativity, you might still
be able to write a third class to use as a Combiner for your job.The only affect a combiner has is to reduce the number of records that are passed from the mappers to the reducers in
the shuffle and sort phase. For more information on combiners, see chapter 2 of Hadoop: The Definitive Guide, 3rd Edition in the Scaling Out: Combiner Functions section.The average
operation is commutative, but not associative, so it is not a natural fit for using a combiner. It would, however, be possible to define a custom intermediate data type that stores
both the computed average and the number of records that are represented by that average. The reducers could then use that information to unroll the averages computed by the
combiners and calculate a total average for each key. Note that in that case, the reducer could not be reused as the combiner; the combiner would have to be a separate
implementation. For more information on combiners, see chapter 2 of Hadoop: The Definitive Guide, 3rd Edition in the Scaling Out: Combiner Functions section.The maximum operation is
both associative and commutative, so it is a candidate for using a combiner.
Watch the training from http://hadoopexam.com/index.html/#hadoop-training

Mapr (HP) Hadoop Developer Certification Questions and Answers (Dumps and Practice Questions)

Related Questions