Pig работает с резьбой

Question

Я пытаюсь загрузить данные из локальной директории свиньи с помощью команды

записи = нагрузки «/ дома/hduser/Загрузки/1901», как (год: chararray, температура: int, quality: int);

dump records; Это прекрасно работает.

пока я попробовать следующую команду

filtered_records = Фильтр запись от температуры! = 9999 AND (качество == 0 или качество == 1 или качество == 4 или качество == 5 или качество == 9);

dump filt_records;

Это показывает следующее сообщение с какой-нить

2015-05-05 19:59:50,998 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for records 
2015-05-05 19:59:50,999 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for records 
2015-05-05 19:59:51,102 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId= 
2015-05-05 19:59:51,294 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: Store(file:/tmp/temp138869802/tmp-1519406150:org.apache.pig.builtin.BinStorage) - 1-90 Operator Key: 1-90) 
2015-05-05 19:59:51,380 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 
2015-05-05 19:59:51,393 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 
2015-05-05 19:59:51,457 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:51,460 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:51,465 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 
2015-05-05 19:59:52,893 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 
2015-05-05 19:59:53,042 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:53,043 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 
2015-05-05 19:59:53,047 [Thread-4] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 
2015-05-05 19:59:53,259 [Thread-4] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:53,323 [Thread-4] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:53,377 [Thread-4] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 
2015-05-05 19:59:53,385 [Thread-4] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 
2015-05-05 19:59:53,549 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 
2015-05-05 19:59:53,860 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:53,869 [Thread-13] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 
2015-05-05 19:59:53,870 [Thread-13] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 
2015-05-05 19:59:53,953 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:53,957 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:53,996 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:54,024 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:54,323 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 
2015-05-05 19:59:54,893 [Thread-13] INFO org.apache.hadoop.mapred.TaskRunner - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 
2015-05-05 19:59:54,901 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:54,901 [Thread-13] INFO org.apache.hadoop.mapred.LocalJobRunner - 
2015-05-05 19:59:54,902 [Thread-13] INFO org.apache.hadoop.mapred.TaskRunner - Task attempt_local_0001_m_000000_0 is allowed to commit now 
2015-05-05 19:59:54,907 [Thread-13] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:54,957 [Thread-13] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp138869802/tmp-1519406150 
2015-05-05 19:59:54,958 [Thread-13] INFO org.apache.hadoop.mapred.LocalJobRunner - 
2015-05-05 19:59:54,958 [Thread-13] INFO org.apache.hadoop.mapred.TaskRunner - Task 'attempt_local_0001_m_000000_0' done. 
2015-05-05 19:59:58,829 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 
2015-05-05 19:59:58,830 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Successfully stored result in: "file:/tmp/temp138869802/tmp-1519406150" 
2015-05-05 19:59:58,833 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Records written : 0 
2015-05-05 19:59:58,837 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Bytes written : 0 
2015-05-05 19:59:58,837 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Spillable Memory Manager spill count : 0 
2015-05-05 19:59:58,837 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Proactive spill count : 0 
2015-05-05 19:59:58,837 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 
2015-05-05 19:59:58,920 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 
2015-05-05 19:59:58,930 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 
2015-05-05 19:59:58,933 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 

Я не вижу выхода. помогите мне исправить проблему.

Answer 1

Я думаю, вы должны определить свой загрузчик, например:

records = load '/home/hduser/Downloads/1901' Using PigStorage(',') as (year:chararray,temperature:int, quality:int); 

Чтобы правильно есть данные в таблице записей в соответствии с счем вы определили. (Пожалуйста, используйте правильную функцию загрузчика, если вы используете csv или tsv, чем обновляете символ PigStorage для этого, что вы используете в своем файле данных)