Как узнать количество выполненных ведомых

Question

У меня есть вопрос о выделении карты/сокращения. У меня есть код wordcount, который я выполняю. (У меня есть 2 подчиненных и 1 мастер). Как узнать, что ведомый 2 выполнен? Как я могу сделать что-то вроде println в консоли, чтобы получить информацию, когда ведомое устройство выполнено?

Ну я попробовал это astuce в части функции карты:

String ip = InetAddress.getLocalHost().getHostAddress().toString(); 

я использую IP, и я посылаю его в качестве ключа (и значение new intWritable поэтому уменьшить получить IP из slave, который выполняет код карты и записывает его в выходной файл, но это было неудачно, потому что он написал IP-адрес ведомого 1 или ведомого 2, но не адреса 2 подчиненных устройств, и я использую:

DistributedCache.addCacheFile(link,job.getConfiguration()); 

для распределения всех f ile у всех подчиненных (не часть файла для каждого подчиненного).

и, наконец, я получаю:

узел менеджер журнала:

2015-05-17 22:35:52,855 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE) 
2015-05-17 22:35:54,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1431892621101_0005_01_000002 by user hduser 
2015-05-17 22:35:54,579 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1431892621101_0005 
2015-05-17 22:35:54,707 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser IP=192.168.1.11 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1431892621101_0005 CONTAINERID=container_1431892621101_0005_01_000002 
2015-05-17 22:35:55,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from NEW to INITING 
2015-05-17 22:35:55,102 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1431892621101_0005_01_000002 to application application_1431892621101_0005 
2015-05-17 22:35:55,145 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from INITING to RUNNING 
2015-05-17 22:35:55,642 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from NEW to LOCALIZING 
2015-05-17 22:35:55,658 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1431892621101_0005 
2015-05-17 22:35:55,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1431892621101_0005 
2015-05-17 22:35:55,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle 
2015-05-17 22:35:55,759 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1431892621101_0005 
2015-05-17 22:35:55,909 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.jar transitioned from INIT to DOWNLOADING 
2015-05-17 22:35:55,909 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.xml transitioned from INIT to DOWNLOADING 
2015-05-17 22:35:56,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1431892621101_0005_01_000002 
2015-05-17 22:35:56,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1431892621101_0005_01_000002.tokens. Credentials list: 
2015-05-17 22:36:00,700 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hduser 
2015-05-17 22:36:01,816 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1431892621101_0005_01_000002.tokens to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000002.tokens 
2015-05-17 22:36:01,865 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005 = file:/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005 
2015-05-17 22:36:22,696 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.jar(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/filecache/10/job.jar) transitioned from DOWNLOADING to LOCALIZED 
2015-05-17 22:36:22,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.xml(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/filecache/11/job.xml) transitioned from DOWNLOADING to LOCALIZED 
2015-05-17 22:36:22,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from LOCALIZING to LOCALIZED 
2015-05-17 22:36:23,752 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from LOCALIZED to RUNNING 
2015-05-17 22:36:23,902 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000002/default_container_executor.sh] 
2015-05-17 22:36:26,604 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1431892621101_0005_01_000002 
2015-05-17 22:36:29,051 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 28.3 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used 
2015-05-17 22:36:36,703 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 35.3 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used 
2015-05-17 22:36:42,008 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 66.4 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used 
2015-05-17 22:36:50,807 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 78.4 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:36:56,039 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 87.3 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:13,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 233.7 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:17,816 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE) 
2015-05-17 22:37:18,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1431892621101_0005_01_000002 
2015-05-17 22:37:18,094 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser IP=192.168.1.11 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1431892621101_0005 CONTAINERID=container_1431892621101_0005_01_000002 
2015-05-17 22:37:18,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from RUNNING to KILLING 
2015-05-17 22:37:18,149 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1431892621101_0005_01_000002 
2015-05-17 22:37:20,943 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1431892621101_0005_01_000002 is : 143 
2015-05-17 22:37:21,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 0B of 1 GB physical memory used; 0B of 2.1 GB virtual memory used 
2015-05-17 22:37:21,721 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 
2015-05-17 22:37:21,749 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1431892621101_0005 CONTAINERID=container_1431892621101_0005_01_000002 
2015-05-17 22:37:21,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 
2015-05-17 22:37:21,750 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1431892621101_0005_01_000002 from application application_1431892621101_0005 
2015-05-17 22:37:21,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1431892621101_0005 
2015-05-17 22:37:21,914 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000002 
2015-05-17 22:37:22,521 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE) 
2015-05-17 22:37:22,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1431892621101_0005_01_000003 by user hduser 
2015-05-17 22:37:22,639 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser IP=192.168.1.11 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1431892621101_0005 CONTAINERID=container_1431892621101_0005_01_000003 
2015-05-17 22:37:22,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1431892621101_0005_01_000003 to application application_1431892621101_0005 
2015-05-17 22:37:22,708 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from NEW to LOCALIZING 
2015-05-17 22:37:22,709 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1431892621101_0005 
2015-05-17 22:37:22,709 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1431892621101_0005 
2015-05-17 22:37:22,709 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle 
2015-05-17 22:37:22,710 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1431892621101_0005 
2015-05-17 22:37:22,712 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from LOCALIZING to LOCALIZED 
2015-05-17 22:37:23,245 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from LOCALIZED to RUNNING 
2015-05-17 22:37:23,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000003/default_container_executor.sh] 
2015-05-17 22:37:24,119 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1431892621101_0005_01_000003 
2015-05-17 22:37:24,119 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1431892621101_0005_01_000002 
2015-05-17 22:37:25,044 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1431892621101_0005_01_000002] 
2015-05-17 22:37:27,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 27.4 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:32,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 43.8 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:39,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 75 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:46,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 104.5 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:48,835 INFO org.apache.hadoop.mapred.ShuffleHandler: Setting connection close header... 
2015-05-17 22:37:52,871 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 147.5 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:37:56,163 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 151.8 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:38:33,363 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 157.2 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:38:40,907 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 157.1 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:38:57,551 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 161.5 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:39:02,314 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 164.0 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:39:07,150 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 164.0 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:39:10,542 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 164.0 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:39:13,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 161.4 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used 
2015-05-17 22:39:16,564 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE) 
2015-05-17 22:39:16,794 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1431892621101_0005_01_000003 
2015-05-17 22:39:16,795 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from RUNNING to KILLING 
2015-05-17 22:39:16,795 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1431892621101_0005_01_000003 
2015-05-17 22:39:16,796 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser IP=192.168.1.11 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1431892621101_0005 CONTAINERID=container_1431892621101_0005_01_000003 
2015-05-17 22:39:18,989 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1431892621101_0005_01_000003 is : 143 
2015-05-17 22:39:19,847 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1431892621101_0005 CONTAINERID=container_1431892621101_0005_01_000003 
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1431892621101_0005_01_000003 from application application_1431892621101_0005 
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1431892621101_0005 
2015-05-17 22:39:19,954 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000003 
2015-05-17 22:39:19,983 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 0B of 1 GB physical memory used; 0B of 2.1 GB virtual memory used 
2015-05-17 22:39:22,984 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1431892621101_0005_01_000003 
2015-05-17 22:39:29,850 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1431892621101_0005_01_000003] 
2015-05-17 22:39:30,206 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 
2015-05-17 22:39:30,264 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1431892621101_0005 
2015-05-17 22:39:30,272 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005 
2015-05-17 22:39:30,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 
2015-05-17 22:39:30,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1431892621101_0005, with delay of 10800 seconds 

Я надеюсь, что я найду, например, количество nodemanager экземпляров, которые выполняют код в этой части прецедента, но я не сделал ,

Как узнать количество ведомых, выполняемых в моем коде? Заранее спасибо.

Answer 1

DataNode журналы:

enter code here 
2015-05-17 22:15:53,749 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741854_1030 src: /192.168.1.10:53005 dest: /192.168.1.10:50010 
2015-05-17 22:16:02,962 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 329ms (threshold=300ms) 
2015-05-17 22:16:19,755 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 303ms (threshold=300ms) 
2015-05-17 22:17:00,801 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53005, dest: /192.168.1.10:50010, bytes: 57614344, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1317607313_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741854_1030, duration: 58914540985 
2015-05-17 22:17:00,823 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741854_1030, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:17:07,237 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741854_1030 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741854 for deletion 
2015-05-17 22:17:07,325 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741854_1030 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741854 
2015-05-17 22:24:31,972 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741855_1031 src: /192.168.1.10:53023 dest: /192.168.1.10:50010 
2015-05-17 22:25:29,693 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53023, dest: /192.168.1.10:50010, bytes: 57614412, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741855_1031, duration: 57499963690 
2015-05-17 22:25:29,712 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741855_1031, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:25:30,882 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741856_1032 src: /192.168.1.10:53028 dest: /192.168.1.10:50010 
2015-05-17 22:25:31,043 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53028, dest: /192.168.1.10:50010, bytes: 131, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741856_1032, duration: 26027070 
2015-05-17 22:25:31,045 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741856_1032, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:25:31,162 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741857_1033 src: /192.168.1.10:53030 dest: /192.168.1.10:50010 
2015-05-17 22:25:31,285 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53030, dest: /192.168.1.10:50010, bytes: 13, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741857_1033, duration: 62104463 
2015-05-17 22:25:31,285 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741857_1033, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:25:32,384 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741858_1034 src: /192.168.1.10:53032 dest: /192.168.1.10:50010 
2015-05-17 22:25:32,642 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53032, dest: /192.168.1.10:50010, bytes: 88165, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741858_1034, duration: 198784861 
2015-05-17 22:25:32,642 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741858_1034, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:26:03,671 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741859_1035 src: /192.168.1.11:44921 dest: /192.168.1.10:50010 
2015-05-17 22:26:03,870 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44921, dest: /192.168.1.10:50010, bytes: 104903, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741859_1035, duration: 141461930 
2015-05-17 22:26:03,871 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741859_1035, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:26:11,276 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741860_1036 src: /192.168.1.11:44928 dest: /192.168.1.10:50010 
2015-05-17 22:26:18,166 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44928, dest: /192.168.1.10:50010, bytes: 33241, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741860_1036, duration: 6865081449 
2015-05-17 22:26:18,166 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741860_1036, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:26:18,320 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741861_1037 src: /192.168.1.11:44935 dest: /192.168.1.10:50010 
2015-05-17 22:26:18,374 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44935, dest: /192.168.1.10:50010, bytes: 347, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741861_1037, duration: 50009828 
2015-05-17 22:26:18,375 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741861_1037, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:26:18,841 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741862_1038 src: /192.168.1.11:44938 dest: /192.168.1.10:50010 
2015-05-17 22:26:18,900 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44938, dest: /192.168.1.10:50010, bytes: 33241, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741862_1038, duration: 53415933 
2015-05-17 22:26:18,900 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741862_1038, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:26:19,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039 src: /192.168.1.11:44940 dest: /192.168.1.10:50010 
2015-05-17 22:26:19,329 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44940, dest: /192.168.1.10:50010, bytes: 104903, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039, duration: 157935747 
2015-05-17 22:26:19,331 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:26:23,457 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741856_1032 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741856 for deletion 
2015-05-17 22:26:23,515 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741856_1032 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741856 
2015-05-17 22:26:23,515 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741857_1033 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741857 for deletion 
2015-05-17 22:26:23,603 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741857_1033 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741857 
2015-05-17 22:26:23,677 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741858_1034 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741858 for deletion 
2015-05-17 22:26:23,678 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741859_1035 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741859 for deletion 
2015-05-17 22:26:23,680 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741860_1036 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741860 for deletion 
2015-05-17 22:26:23,680 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741858_1034 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741858 
2015-05-17 22:26:23,681 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741859_1035 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741859 
2015-05-17 22:26:23,682 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741855_1031 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741855 for deletion 
2015-05-17 22:26:23,683 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741860_1036 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741860 
2015-05-17 22:26:23,688 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741855_1031 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741855 
2015-05-17 22:31:18,177 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741864_1040 src: /192.168.1.10:53045 dest: /192.168.1.10:50010 
2015-05-17 22:31:18,411 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53045, dest: /192.168.1.10:50010, bytes: 2, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-572045485_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741864_1040, duration: 120795963 
2015-05-17 22:31:18,412 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741864_1040, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:33:03,737 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741865_1041 src: /192.168.1.10:53053 dest: /192.168.1.10:50010 
2015-05-17 22:33:03,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53053, dest: /192.168.1.10:50010, bytes: 2, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-947295319_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741865_1041, duration: 138647149 
2015-05-17 22:33:03,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741865_1041, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:34:37,656 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741866_1042 src: /192.168.1.10:53059 dest: /192.168.1.10:50010 
2015-05-17 22:35:35,213 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53059, dest: /192.168.1.10:50010, bytes: 57614412, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741866_1042, duration: 57228181985 
2015-05-17 22:35:35,282 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741866_1042, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:35:36,863 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043 src: /192.168.1.10:53064 dest: /192.168.1.10:50010 
2015-05-17 22:35:36,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53064, dest: /192.168.1.10:50010, bytes: 126, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043, duration: 24430351 
2015-05-17 22:35:36,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:35:37,104 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741868_1044 src: /192.168.1.10:53066 dest: /192.168.1.10:50010 
2015-05-17 22:35:37,266 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53066, dest: /192.168.1.10:50010, bytes: 26, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741868_1044, duration: 145913793 
2015-05-17 22:35:37,267 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741868_1044, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:35:38,798 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741869_1045 src: /192.168.1.10:53068 dest: /192.168.1.10:50010 
2015-05-17 22:35:39,276 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53068, dest: /192.168.1.10:50010, bytes: 88156, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741869_1045, duration: 190623489 
2015-05-17 22:35:39,278 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741869_1045, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:35:48,707 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046 src: /192.168.1.11:44948 dest: /192.168.1.10:50010 
2015-05-17 22:35:48,956 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44948, dest: /192.168.1.10:50010, bytes: 104894, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046, duration: 178516552 
2015-05-17 22:35:48,958 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:37:19,135 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741871_1047 src: /192.168.1.11:44955 dest: /192.168.1.10:50010 
2015-05-17 22:37:51,956 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741872_1048 src: /192.168.1.10:53088 dest: /192.168.1.10:50010 
2015-05-17 22:38:40,341 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 42910ms (threshold=300ms) 
2015-05-17 22:38:40,842 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 493ms (threshold=300ms) 
2015-05-17 22:38:41,809 WARN org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 42670ms 
GC pool 'PS Scavenge' had collection(s): count=1 time=42530ms 
2015-05-17 22:38:41,904 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 495ms (threshold=300ms) 
2015-05-17 22:38:50,682 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:382ms (threshold=300ms) 
2015-05-17 22:38:58,498 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:346ms (threshold=300ms) 
2015-05-17 22:39:03,675 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 407ms (threshold=300ms) 
2015-05-17 22:39:15,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53088, dest: /192.168.1.10:50010, bytes: 37785075, op: HDFS_WRITE, cliID: DFSClient_attempt_1431892621101_0005_r_000000_0_-581669036_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741872_1048, duration: 82970475818 
2015-05-17 22:39:15,346 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741872_1048, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 
2015-05-17 22:39:16,181 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741852_1028 
2015-05-17 22:39:16,236 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741851_1027 
2015-05-17 22:39:16,313 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741853_1029 
2015-05-17 22:39:19,173 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44955, dest: /192.168.1.10:50010, bytes: 33639, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741871_1047, duration: 119891168848 
2015-05-17 22:39:19,174 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741871_1047, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:39:20,113 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741873_1049 src: /192.168.1.11:44963 dest: /192.168.1.10:50010 
2015-05-17 22:39:20,135 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44963, dest: /192.168.1.10:50010, bytes: 350, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741873_1049, duration: 20566418 
2015-05-17 22:39:20,137 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741873_1049, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:39:20,455 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741874_1050 src: /192.168.1.11:44966 dest: /192.168.1.10:50010 
2015-05-17 22:39:20,528 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44966, dest: /192.168.1.10:50010, bytes: 33639, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741874_1050, duration: 68722185 
2015-05-17 22:39:20,528 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741874_1050, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:39:20,655 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741875_1051 src: /192.168.1.11:44968 dest: /192.168.1.10:50010 
2015-05-17 22:39:20,779 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44968, dest: /192.168.1.10:50010, bytes: 104894, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741875_1051, duration: 117317991 
2015-05-17 22:39:20,850 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741875_1051, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 
2015-05-17 22:39:21,674 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046 
2015-05-17 22:39:21,790 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039 
2015-05-17 22:39:21,874 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043 
2015-05-17 22:39:25,480 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741866_1042 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741866 for deletion 
2015-05-17 22:39:25,481 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741867_1043 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741867 for deletion 
2015-05-17 22:39:25,481 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741868_1044 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741868 for deletion 
2015-05-17 22:39:25,481 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741869_1045 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741869 for deletion 
2015-05-17 22:39:25,482 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741870_1046 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741870 for deletion 
2015-05-17 22:39:25,482 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741871_1047 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741871 for deletion 
2015-05-17 22:39:25,542 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741866_1042 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741866 
2015-05-17 22:39:25,542 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741867_1043 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741867 
2015-05-17 22:39:25,542 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741868_1044 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741868 
2015-05-17 22:39:25,554 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741869_1045 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741869 
2015-05-17 22:39:25,554 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741870_1046 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741870 
2015-05-17 22:39:25,554 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741871_1047 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_107374187