【Hadoop十五】Hadoop Counter-技术博客集

【Hadoop十五】Hadoop Counter
编程技术 / houtizong 发布于 3年前 180

1. 只有Map任务的Map Reduce Job

File System CountersFILE: Number of bytes read=3629530FILE: Number of bytes written=98312FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=8570654HDFS: Number of bytes written=1404469HDFS: Number of read operations=6HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=14522Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=14522Total vcore-seconds taken by all map tasks=14522Total megabyte-seconds taken by all map tasks=14870528Map-Reduce FrameworkMap input records=7452Map output records=7452Input split bytes=146Spilled Records=0Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=241CPU time spent (ms)=9750Physical memory (bytes) snapshot=184406016Virtual memory (bytes) snapshot=893657088Total committed heap usage (bytes)=89653248File Input Format Counters Bytes Read=8570508File Output Format Counters Bytes Written=1404469

2. 既有Map又有Reduce的MapReduce Job

File System CountersFILE: Number of bytes read=879582FILE: Number of bytes written=198227FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=2729649HDFS: Number of bytes written=265HDFS: Number of read operations=7HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Launched reduce tasks=1Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=7071Total time spent by all reduces in occupied slots (ms)=7804Total time spent by all map tasks (ms)=7071Total time spent by all reduce tasks (ms)=7804Total vcore-seconds taken by all map tasks=7071Total vcore-seconds taken by all reduce tasks=7804Total megabyte-seconds taken by all map tasks=7240704Total megabyte-seconds taken by all reduce tasks=7991296Map-Reduce FrameworkMap input records=20Map output records=1Map output bytes=167Map output materialized bytes=182Input split bytes=139Combine input records=1Combine output records=1Reduce input groups=1Reduce shuffle bytes=182Reduce input records=1Reduce output records=1Spilled Records=2Shuffled Maps =1Failed Shuffles=0Merged Map outputs=1GC time elapsed (ms)=122CPU time spent (ms)=3620Physical memory (bytes) snapshot=451244032Virtual memory (bytes) snapshot=1823916032Total committed heap usage (bytes)=288882688Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=2729510File Output Format Counters Bytes Written=265

上一篇：【Mahout二】基于Mahout CBayes算法的20newsgroup的脚本分析

下一篇：C/C++图片或文件的读写

请勿发布不友善或者负能量的内容。与人为善，比聪明更重要！

<div > &nbsp; &nbsp;1. 只有Map任务的Map Reduce Job <pre name="code" class="java">File System CountersFILE: Number of bytes read=3629530FILE: Number of bytes written=98312FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=8570654HDFS: Number of bytes written=1404469HDFS: Number of read operations=6HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=14522Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=14522Total vcore-seconds taken by all map tasks=14522Total megabyte-seconds taken by all map tasks=14870528Map-Reduce FrameworkMap input records=7452Map output records=7452Input split bytes=146Spilled Records=0Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=241CPU time spent (ms)=9750Physical memory (bytes) snapshot=184406016Virtual memory (bytes) snapshot=893657088Total committed heap usage (bytes)=89653248File Input Format Counters Bytes Read=8570508File Output Format Counters Bytes Written=1404469</pre> &nbsp; 2. 既有Map又有Reduce的MapReduce Job &nbsp; <pre name="code" class="java">File System CountersFILE: Number of bytes read=879582FILE: Number of bytes written=198227FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=2729649HDFS: Number of bytes written=265HDFS: Number of read operations=7HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Launched reduce tasks=1Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=7071Total time spent by all reduces in occupied slots (ms)=7804Total time spent by all map tasks (ms)=7071Total time spent by all reduce tasks (ms)=7804Total vcore-seconds taken by all map tasks=7071Total vcore-seconds taken by all reduce tasks=7804Total megabyte-seconds taken by all map tasks=7240704Total megabyte-seconds taken by all reduce tasks=7991296Map-Reduce FrameworkMap input records=20Map output records=1Map output bytes=167Map output materialized bytes=182Input split bytes=139Combine input records=1Combine output records=1Reduce input groups=1Reduce shuffle bytes=182Reduce input records=1Reduce output records=1Spilled Records=2Shuffled Maps =1Failed Shuffles=0Merged Map outputs=1GC time elapsed (ms)=122CPU time spent (ms)=3620Physical memory (bytes) snapshot=451244032Virtual memory (bytes) snapshot=1823916032Total committed heap usage (bytes)=288882688Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=2729510File Output Format Counters Bytes Written=265</pre> &nbsp; &nbsp; &nbsp; </div>

留言需要登陆哦

技术博客集 - 网站简介：
前后端技术：
后端基于Hyperf2.1框架开发,前端使用Bootstrap可视化布局系统生成
网站主要作用：
1.编程技术分享及讨论交流，内置聊天系统;
2.测试交流框架问题，比如：Hyperf、Laravel、TP、beego;
3.本站数据是基于大数据采集等爬虫技术为基础助力分享知识，如有侵权请发邮件到站长邮箱，站长会尽快处理;
4.站长邮箱：[email protected];

文章归档

文章标签

友情链接

首页
关于我们

Auther ·HouTiZong: 侯体宗的博客