HMaster启动后自动停止,且HRegionServer进程依旧存在

HMaster启动后自动停止,且HRegionServer进程依旧存在。
查看HMaster日志显示:
2017-11-01 16:50:19,536 FATAL [hbs14:16000.activeMasterManager] master.HMaster: Failed to become active master
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:118)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99)
at com.sun.proxy.$Proxy16.setSafeMode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2419)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1036)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1020)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:525)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:971)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:429)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:693)
at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:189)
at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1803)
at java.lang.Thread.run(Thread.java:745)
2017-11-01 16:50:19,538 FATAL [hbs14:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:118)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99)
at com.sun.proxy.$Proxy16.setSafeMode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2419)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1036)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1020)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:525)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:971)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:429)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:693)
at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:189)
at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1803)
at java.lang.Thread.run(Thread.java:745)
2017-11-01 16:50:19,538 INFO [hbs14:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2017-11-01 16:50:22,508 INFO [master/hbs14/127.0.0.1:16000] ipc.RpcServer: Stopping server on 16000
2017-11-01 16:50:22,508 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2017-11-01 16:50:22,509 INFO [master/hbs14/127.0.0.1:16000] regionserver.HRegionServer: Stopping infoServer
2017-11-01 16:50:22,515 INFO [master/hbs14/127.0.0.1:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2017-11-01 16:50:22,515 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-11-01 16:50:22,515 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-11-01 16:50:22,517 INFO [master/hbs14/127.0.0.1:16000] regionserver.HRegionServer: stopping server hbs14,16000,1509526217638
2017-11-01 16:50:22,517 INFO [master/hbs14/127.0.0.1:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x35f76c597c20001
2017-11-01 16:50:22,532 INFO [master/hbs14/127.0.0.1:16000] zookeeper.ZooKeeper: Session: 0x35f76c597c20001 closed
2017-11-01 16:50:22,532 INFO [master/hbs14/127.0.0.1:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x35f76c597c20001
2017-11-01 16:50:22,535 INFO [master/hbs14/127.0.0.1:16000] regionserver.HRegionServer: stopping server hbs14,16000,1509526217638; all regions closed.
2017-11-01 16:50:22,536 INFO [master/hbs14/127.0.0.1:16000] hbase.ChoreService: Chore service for: hbs14,16000,1509526217638 had [] on shutdown
2017-11-01 16:50:22,555 INFO [master/hbs14/127.0.0.1:16000] ipc.RpcServer: Stopping server on 16000
2017-11-01 16:50:22,581 INFO [master/hbs14/127.0.0.1:16000] zookeeper.ZooKeeper: Session: 0x35f76c597c20000 closed
2017-11-01 16:50:22,581 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x35f76c597c20000
2017-11-01 16:50:22,581 INFO [master/hbs14/127.0.0.1:16000] regionserver.HRegionServer: stopping server hbs14,16000,1509526217638; zookeeper connection closed.
2017-11-01 16:50:22,581 INFO [master/hbs14/127.0.0.1:16000] regionserver.HRegionServer: master/hbs14/127.0.0.1:16000 exiting

已邀请:

SuperbDong

赞同来自:

可以先用jstack看下退出卡在哪里了

qgxiaozhan

赞同来自:

 org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:971) 看样子是hdfs还在安全模式。

要回复问题请先登录注册