-3
Я только что установил Hive в свой кластер Hadoop и загрузил свои данные в таблицу Hive. Когда я выпускаю select *, он отлично работает, но когда я выдаю select * from table where column1 in (select max(column1) from table);, он замерзает. Пожалуйста, помогите мне. Вот мой улей журналлюбой запрос на улей, кроме select * hangs
2017-02-17 07:42:28,116 INFO [main]: SessionState (SessionState.java:printInfo(951)) -
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
2017-02-17 07:42:28,438 WARN [main]: util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-17 07:42:28,560 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(589)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2017-02-17 07:42:28,710 INFO [main]: metastore.ObjectStore (ObjectStore.java:initialize(289)) - ObjectStore, initialize called
2017-02-17 07:42:30,831 INFO [main]: metastore.ObjectStore (ObjectStore.java:getPMF(370)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2017-02-17 07:42:33,354 INFO [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is DERBY
.....
2017-02-17 07:43:04,861 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:04,927 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:04,953 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: select consume_date,hour_id,fromdate,company_name,b03 from consumes where b03 in (select max(b03) from consumes)
2017-02-17 07:43:05,527 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed
2017-02-17 07:43:05,528 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=parse start=1487346184927 end=1487346185528 duration=601 from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:05,530 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:05,576 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:analyzeInternal(10127)) - Starting Semantic Analysis
2017-02-17 07:43:05,579 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:genResolvedParseTree(10074)) - Completed phase 1 of Semantic Analysis
2017-02-17 07:43:05,579 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1552)) - Get metadata for source tables
2017-02-17 07:43:05,579 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 0: get_table : db=default tbl=consumes
2017-02-17 07:43:05,580 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=linux ip=unknown-ip-addr cmd=get_table : db=default tbl=consumes
2017-02-17 07:43:06,076 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1704)) - Get metadata for subqueries
2017-02-17 07:43:06,092 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1728)) - Get metadata for destination tables
2017-02-17 07:43:06,096 ERROR [main]: hdfs.KeyProviderCache (KeyProviderCache.java:createKeyProviderURI(87)) - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
2017-02-17 07:43:06,129 INFO [main]: ql.Context (Context.java:getMRScratchDir(330)) - New scratch dir is hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:06,131 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:genResolvedParseTree(10078)) - Completed getting MetaData in Semantic Analysis
2017-02-17 07:43:06,252 INFO [main]: parse.BaseSemanticAnalyzer (CalcitePlanner.java:canCBOHandleAst(388)) - Not invoking CBO because the statement has too few joins
2017-02-17 07:43:06,450 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1552)) - Get metadata for source tables
2017-02-17 07:43:06,451 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 0: get_table : db=default tbl=consumes
2017-02-17 07:43:06,454 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=linux ip=unknown-ip-addr cmd=get_table : db=default tbl=consumes
2017-02-17 07:43:06,488 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1704)) - Get metadata for subqueries
2017-02-17 07:43:06,488 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1728)) - Get metadata for destination tables
2017-02-17 07:43:06,631 INFO [main]: common.FileUtils (FileUtils.java:mkdir(501)) - Creating directory if it doesn't exist: hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10000/.hive-staging_hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:06,759 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:genFileSinkPlan(6653)) - Set stats collection dir : hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10000/.hive-staging_hive_2017-02-17_07-43-04_926_1561320960043112851-1/-ext-10002
2017-02-17 07:43:06,839 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for FS(16)
2017-02-17 07:43:06,840 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for SEL(15)
2017-02-17 07:43:06,841 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(457)) - Processing for JOIN(13)
2017-02-17 07:43:06,841 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for RS(10)
2017-02-17 07:43:06,841 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(402)) - Processing for FIL(9)
2017-02-17 07:43:06,846 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(707)) - Pushdown Predicates of FIL For Alias : consumes
2017-02-17 07:43:06,846 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(710)) - b03 is not null
2017-02-17 07:43:06,847 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(382)) - Processing for TS(0)
2017-02-17 07:43:06,847 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(707)) - Pushdown Predicates of TS For Alias : consumes
2017-02-17 07:43:06,847 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(710)) - b03 is not null
2017-02-17 07:43:06,849 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for RS(12)
2017-02-17 07:43:06,849 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(402)) - Processing for FIL(11)
2017-02-17 07:43:06,850 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(707)) - Pushdown Predicates of FIL For Alias :
2017-02-17 07:43:06,850 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(710)) - _col0 is not null
2017-02-17 07:43:06,850 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for GBY(8)
2017-02-17 07:43:06,851 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(707)) - Pushdown Predicates of GBY For Alias :
2017-02-17 07:43:06,851 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(710)) - _col0 is not null
2017-02-17 07:43:06,851 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for SEL(7)
2017-02-17 07:43:06,851 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(707)) - Pushdown Predicates of SEL For Alias : sq_1
2017-02-17 07:43:06,851 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(710)) - _col0 is not null
2017-02-17 07:43:06,852 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for SEL(6)
2017-02-17 07:43:06,852 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(707)) - Pushdown Predicates of SEL For Alias : sq_1
2017-02-17 07:43:06,852 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:logExpr(710)) - _col0 is not null
2017-02-17 07:43:06,852 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for GBY(5)
2017-02-17 07:43:06,853 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for RS(4)
2017-02-17 07:43:06,853 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for GBY(3)
2017-02-17 07:43:06,853 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(655)) - Processing for SEL(2)
2017-02-17 07:43:06,853 INFO [main]: ppd.OpProcFactory (OpProcFactory.java:process(382)) - Processing for TS(1)
2017-02-17 07:43:06,863 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=partition-retrieving from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
2017-02-17 07:43:06,863 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=partition-retrieving start=1487346186863 end=1487346186863 duration=0 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner>
2017-02-17 07:43:06,880 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneJoinOperator(975)) - JOIN 13 oldExprs: {0=[Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[KEY.reducesinkkey0], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12]], 1=[]}
2017-02-17 07:43:06,880 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneJoinOperator(1080)) - JOIN 13 newExprs: {0=[Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col5], Column[KEY.reducesinkkey0]], 1=[]}
2017-02-17 07:43:06,881 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneReduceSinkOperator(817)) - RS 10 oldColExprMap: {VALUE._col10=Column[BLOCK__OFFSET__INSIDE__FILE], VALUE._col11=Column[INPUT__FILE__NAME], VALUE._col12=Column[ROW__ID], KEY.reducesinkkey0=Column[b03], VALUE._col2=Column[fromdate], VALUE._col3=Column[todate], VALUE._col4=Column[company_code], VALUE._col5=Column[company_name], VALUE._col0=Column[consume_date], VALUE._col1=Column[hour_id], VALUE._col6=Column[b04], VALUE._col7=Column[b27], VALUE._col8=Column[b31], VALUE._col9=Column[b32]}
2017-02-17 07:43:06,881 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneReduceSinkOperator(866)) - RS 10 newColExprMap: {KEY.reducesinkkey0=Column[b03], VALUE._col2=Column[fromdate], VALUE._col5=Column[company_name], VALUE._col0=Column[consume_date], VALUE._col1=Column[hour_id]}
2017-02-17 07:43:06,881 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneReduceSinkOperator(817)) - RS 12 oldColExprMap: {KEY.reducesinkkey0=Column[_col0]}
2017-02-17 07:43:06,881 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneReduceSinkOperator(866)) - RS 12 newColExprMap: {KEY.reducesinkkey0=Column[_col0]}
2017-02-17 07:43:06,883 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneReduceSinkOperator(817)) - RS 4 oldColExprMap: {VALUE._col0=Column[_col0]}
2017-02-17 07:43:06,883 INFO [main]: optimizer.ColumnPrunerProcFactory (ColumnPrunerProcFactory.java:pruneReduceSinkOperator(866)) - RS 4 newColExprMap: {VALUE._col0=Column[_col0]}
2017-02-17 07:43:06,948 INFO [main]: ql.Context (Context.java:getMRScratchDir(330)) - New scratch dir is hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:06,956 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=getInputSummary from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:06,984 INFO [main]: exec.Utilities (Utilities.java:run(2615)) - Cannot get size of hdfs://hadoopmaster:9000/user/hive/warehouse/consumes. Safely ignored.
2017-02-17 07:43:06,987 INFO [main]: exec.Utilities (Utilities.java:run(2615)) - Cannot get size of hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10003. Safely ignored.
2017-02-17 07:43:06,988 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=getInputSummary start=1487346186956 end=1487346186988 duration=32 from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:06,990 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=clonePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,123 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,123 INFO [main]: exec.Utilities (Utilities.java:serializePlan(938)) - Serializing MapredWork via kryo
2017-02-17 07:43:07,321 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=serializePlan start=1487346187123 end=1487346187321 duration=198 from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,321 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,321 INFO [main]: exec.Utilities (Utilities.java:deserializePlan(965)) - Deserializing MapredWork via kryo
2017-02-17 07:43:07,387 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=deserializePlan start=1487346187321 end=1487346187387 duration=66 from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,387 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=clonePlan start=1487346186990 end=1487346187387 duration=397 from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,400 INFO [main]: ql.Context (Context.java:getMRScratchDir(330)) - New scratch dir is hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:07,401 INFO [main]: ql.Context (Context.java:getMRScratchDir(330)) - New scratch dir is hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:07,406 INFO [main]: physical.LocalMapJoinProcFactory (LocalMapJoinProcFactory.java:process(139)) - Setting max memory usage to 0.9 for table sink not followed by group by
2017-02-17 07:43:07,447 INFO [main]: physical.NullScanTaskDispatcher (NullScanTaskDispatcher.java:dispatch(175)) - Looking for table scans where optimization is applicable
2017-02-17 07:43:07,451 INFO [main]: physical.NullScanTaskDispatcher (NullScanTaskDispatcher.java:dispatch(199)) - Found 0 null table scans
2017-02-17 07:43:07,452 INFO [main]: physical.NullScanTaskDispatcher (NullScanTaskDispatcher.java:dispatch(175)) - Looking for table scans where optimization is applicable
2017-02-17 07:43:07,452 INFO [main]: physical.NullScanTaskDispatcher (NullScanTaskDispatcher.java:dispatch(199)) - Found 0 null table scans
2017-02-17 07:43:07,453 INFO [main]: physical.NullScanTaskDispatcher (NullScanTaskDispatcher.java:dispatch(175)) - Looking for table scans where optimization is applicable
2017-02-17 07:43:07,453 INFO [main]: physical.NullScanTaskDispatcher (NullScanTaskDispatcher.java:dispatch(199)) - Found 0 null table scans
2017-02-17 07:43:07,473 INFO [main]: parse.CalcitePlanner (SemanticAnalyzer.java:analyzeInternal(10213)) - Completed plan generation
2017-02-17 07:43:07,473 INFO [main]: ql.Driver (Driver.java:compile(436)) - Semantic Analysis Completed
2017-02-17 07:43:07,473 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=semanticAnalyze start=1487346185530 end=1487346187473 duration=1943 from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:07,519 INFO [main]: exec.ListSinkOperator (Operator.java:initialize(332)) - Initializing operator OP[32]
2017-02-17 07:43:07,521 INFO [main]: exec.ListSinkOperator (Operator.java:initialize(372)) - Initialization Done 32 OP
2017-02-17 07:43:07,521 INFO [main]: exec.ListSinkOperator (Operator.java:initializeChildren(429)) - Operator 32 OP initialized
2017-02-17 07:43:07,529 INFO [main]: ql.Driver (Driver.java:getSchema(240)) - Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:consume_date, type:string, comment:null), FieldSchema(name:hour_id, type:int, comment:null), FieldSchema(name:fromdate, type:string, comment:null), FieldSchema(name:company_name, type:string, comment:null), FieldSchema(name:b03, type:decimal(18,8), comment:null)], properties:null)
2017-02-17 07:43:07,529 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=compile start=1487346184861 end=1487346187529 duration=2668 from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:07,530 INFO [main]: ql.Driver (Driver.java:checkConcurrency(160)) - Concurrency mode is disabled, not creating a lock manager
2017-02-17 07:43:07,530 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:07,530 INFO [main]: ql.Driver (Driver.java:execute(1328)) - Starting command(queryId=linux_20170217074304_4798207d-cb6e-4a87-8292-3baebe3907d4): select consume_date,hour_id,fromdate,company_name,b03 from consumes where b03 in (select max(b03) from consumes)
2017-02-17 07:43:07,531 INFO [main]: ql.Driver (SessionState.java:printInfo(951)) - Query ID = linux_20170217074304_4798207d-cb6e-4a87-8292-3baebe3907d4
2017-02-17 07:43:07,531 INFO [main]: ql.Driver (SessionState.java:printInfo(951)) - Total jobs = 3
2017-02-17 07:43:07,534 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=TimeToSubmit start=1487346184861 end=1487346187534 duration=2673 from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:07,534 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:07,534 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=task.MAPRED.Stage-2 from=org.apache.hadoop.hive.ql.Driver>
2017-02-17 07:43:07,552 INFO [main]: ql.Driver (SessionState.java:printInfo(951)) - Launching Job 1 out of 3
2017-02-17 07:43:07,554 INFO [main]: ql.Driver (Driver.java:launchTask(1651)) - Starting task [Stage-2:MAPRED] in serial mode
2017-02-17 07:43:07,555 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - Number of reduce tasks determined at compile time: 1
2017-02-17 07:43:07,555 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - In order to change the average load for a reducer (in bytes):
2017-02-17 07:43:07,555 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - set hive.exec.reducers.bytes.per.reducer=<number>
2017-02-17 07:43:07,556 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - In order to limit the maximum number of reducers:
2017-02-17 07:43:07,562 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - set hive.exec.reducers.max=<number>
2017-02-17 07:43:07,565 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - In order to set a constant number of reducers:
2017-02-17 07:43:07,567 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - set mapreduce.job.reduces=<number>
2017-02-17 07:43:07,568 INFO [main]: ql.Context (Context.java:getMRScratchDir(330)) - New scratch dir is hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:07,575 INFO [main]: mr.ExecDriver (ExecDriver.java:execute(288)) - Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
2017-02-17 07:43:07,577 INFO [main]: exec.Utilities (Utilities.java:getInputPaths(3397)) - Processing alias sq_1:consumes
2017-02-17 07:43:07,580 INFO [main]: exec.Utilities (Utilities.java:getInputPaths(3414)) - Adding input file hdfs://hadoopmaster:9000/user/hive/warehouse/consumes
2017-02-17 07:43:07,580 INFO [main]: exec.Utilities (Utilities.java:isEmptyPath(2698)) - Content Summary not cached for hdfs://hadoopmaster:9000/user/hive/warehouse/consumes
2017-02-17 07:43:07,651 INFO [main]: exec.Utilities (Utilities.java:createDummyFileForEmptyPartition(3497)) - Changed input file hdfs://hadoopmaster:9000/user/hive/warehouse/consumes to empty file hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10006/0
2017-02-17 07:43:07,651 INFO [main]: ql.Context (Context.java:getMRScratchDir(330)) - New scratch dir is hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1
2017-02-17 07:43:07,665 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:07,666 INFO [main]: exec.Utilities (Utilities.java:serializePlan(938)) - Serializing MapWork via kryo
2017-02-17 07:43:08,663 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=serializePlan start=1487346187665 end=1487346188663 duration=998 from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:08,669 INFO [main]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
2017-02-17 07:43:08,702 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:08,703 INFO [main]: exec.Utilities (Utilities.java:serializePlan(938)) - Serializing ReduceWork via kryo
2017-02-17 07:43:08,745 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=serializePlan start=1487346188702 end=1487346188745 duration=43 from=org.apache.hadoop.hive.ql.exec.Utilities>
2017-02-17 07:43:08,747 ERROR [main]: mr.ExecDriver (ExecDriver.java:execute(400)) - yarn
2017-02-17 07:43:08,836 INFO [main]: client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at hadoopmaster/192.168.23.132:8050
2017-02-17 07:43:09,138 INFO [main]: client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at hadoopmaster/192.168.23.132:8050
2017-02-17 07:43:09,146 INFO [main]: exec.Utilities (Utilities.java:getBaseWork(390)) - PLAN PATH = hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10007/a922e0ae-c541-4b92-8f9d-088bde0d1475/map.xml
2017-02-17 07:43:09,147 INFO [main]: exec.Utilities (Utilities.java:getBaseWork(390)) - PLAN PATH = hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10007/a922e0ae-c541-4b92-8f9d-088bde0d1475/reduce.xml
2017-02-17 07:43:09,454 WARN [main]: mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2017-02-17 07:43:12,706 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=getSplits from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
2017-02-17 07:43:12,707 INFO [main]: exec.Utilities (Utilities.java:getBaseWork(390)) - PLAN PATH = hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10007/a922e0ae-c541-4b92-8f9d-088bde0d1475/map.xml
2017-02-17 07:43:12,707 INFO [main]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(517)) - Total number of paths: 1, launching 1 threads to check non-combinable ones.
2017-02-17 07:43:12,729 INFO [main]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(439)) - CombineHiveInputSplit creating pool for hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10006/0; using filter path hdfs://hadoopmaster:9000/tmp/hive/linux/d79925b9-fb4a-41c8-b45e-cc42db800405/hive_2017-02-17_07-43-04_926_1561320960043112851-1/-mr-10006/0
2017-02-17 07:43:12,768 INFO [main]: input.FileInputFormat (FileInputFormat.java:listStatus(283)) - Total input paths to process : 1
2017-02-17 07:43:12,771 INFO [main]: input.CombineFileInputFormat (CombineFileInputFormat.java:createSplits(413)) - DEBUG: Terminated node allocation with : CompletedNodes: 0, size left: 0
2017-02-17 07:43:12,773 INFO [main]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(494)) - number of splits 1
2017-02-17 07:43:12,775 INFO [main]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(587)) - Number of all splits 1
2017-02-17 07:43:12,775 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=getSplits start=1487346192706 end=1487346192775 duration=69 from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
2017-02-17 07:43:12,857 INFO [main]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of splits:1
2017-02-17 07:43:12,951 INFO [main]: mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting tokens for job: job_1487346076570_0001
2017-02-17 07:43:13,435 INFO [main]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(273)) - Submitted application application_1487346076570_0001
2017-02-17 07:43:13,505 INFO [main]: mapreduce.Job (Job.java:submit(1294)) - The url to track the job: http://hadoopmaster:8088/proxy/application_1487346076570_0001/
2017-02-17 07:43:13,510 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - Starting Job = job_1487346076570_0001, Tracking URL = http://hadoopmaster:8088/proxy/application_1487346076570_0001/
2017-02-17 07:43:13,514 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1487346076570_0001
2017-02-17 07:43:41,582 INFO [SIGINT handler]: CliDriver (SessionState.java:printInfo(951)) - Interrupting... Be patient, this might take some time.
2017-02-17 07:43:41,584 INFO [SIGINT handler]: CliDriver (SessionState.java:printInfo(951)) - Press Ctrl+C again to kill JVM
2017-02-17 07:43:41,841 INFO [SIGINT handler]: impl.YarnClientImpl (YarnClientImpl.java:killApplication(395)) - Killed application application_1487346076570_0001
2017-02-17 07:43:42,058 INFO [SIGINT handler]: CliDriver (SessionState.java:printInfo(951)) - Exiting the JVM
2017-02-17 07:43:42,102 INFO [Thread-11]: impl.YarnClientImpl (YarnClientImpl.java:killApplication(395)) - Killed application application_1487346076570_0001
У меня есть 2 ошибки 1) ERROR [главная]: hdfs.KeyProviderCache (KeyProviderCache.java:createKeyProviderURI(87)) - Не удалось найти URI с помощью клавиши [dfs.encryption. key.provider.uri], чтобы создать keyProvider !! 2) ERROR [главная]: mr.ExecDriver (ExecDriver.java:execute(400)) - пряжа
tnx столько для вашего ответа, но с этим запросом также у меня такая же ошибка :( – hiveuser