can not load page: MasterNotRunningException

[chenhsiu]

I have setup conf/hbase-site.xml properly, but get MasterNotRunningException as follows. Any clues?

! @6e7mlmi15 - Internal server error, for request [GET /] ->

play.core.ActionInvoker$$anonfun$receive$1$$anon$1: Execution exception [[MasterNotRunningException: Retried 10 times]]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:134) [play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:115) [play_2.9.1-2.0.4.jar:2.0.4]
at akka.actor.Actor$class.apply(Actor.scala:318) [akka-actor-2.0.2.jar:2.0.2] at play.core.ActionInvoker.apply(Invoker.scala:113) [play_2.9.1-2.0.4.jar:2.0.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:626) [akka-actor-2.0.2.jar:2.0.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197) [akka-actor-2.0.2.jar:2.0.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:179) [akka-actor-2.0.2.jar:2.0.2]
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516) [akka-actor-2.0.2.jar:2.0.2] at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259) [akka-actor-2.0.2.jar:2.0.2]
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) [akka-actor-2.0.2.jar:2.0.2]
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479) [akka-actor-2.0.2.jar:2.0.2]
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [akka-actor-2.0.2.jar:2.0.2]
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:120) ~[hbase-0.90.5.jar:0.90.5]
at models.HBase$class.withAdmin(HBase.scala:42) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at models.hbase090.HBase090.withAdmin(HBase090.scala:9) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at models.hbase090.HBase090.eachRegionServer(HBase090.scala:11) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at models.Region$.all(Region.scala:71) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at controllers.Application$$anonfun$index$1.apply(Application.scala:13) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at controllers.Application$$anonfun$index$1.apply(Application.scala:12) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at play.api.mvc.Action$$anon$1.apply(Action.scala:170) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1$$anonfun$6.apply(Invoker.scala:126) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1$$anonfun$6.apply(Invoker.scala:126) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.utils.Threads$.withContextClassLoader(Threads.scala:17) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:125) [play_2.9.1-2.0.4.jar:2.0.4]
... 11 common frames omitted

[chenhsiu]

my logs/log4j.log is empty, how can I turn on the debug log?

[meniku]

you can try to change the loglevel from "WARN" to "INFO" in the file "conf/log4j.xml" (line 22)

I recently received a E-Mail with a similar problem. I gave the following suggestions, but did never receive an answer if anything helped:

A few generic questions without looking at the logfiles:

1. Are you compiling Hannibal right on the host where you run it or do you use the "build_package" script? Please let me know if that's the case, maybe there is still some problem with the script.
2. Are you running the latest Hannibal version from the master branch? Maybe the problem is already fixed ;-) To update, simply run 'git pull' inside your Hannibal directory.

From the logfile output, I can't exactly tell you what's wrong. I see two possible reasons:
3. the HBase configuration could be wrong (doublecheck that you have the correct zookeeper-quorum in conf/hbase-site.xml).
4. There could be a problem connecting to Zookeeper. Please try to issue this command from your hannibal-host: 'echo "stat" | nc zookeeper-hostname 2181' (for each of your zookeeper-nodes as configured in hbase-site.xml, replace 'zookeeper-hostname' with the correct hostname). The command should immediately return some statistics about zookeeper.

[gdtm86]

Hi Nils,

I am seeing a similar issue.

-Guru Medasani

On Tue, May 7, 2013 at 7:58 AM, NiKu notifications@github.com wrote:

you can try to change the loglevel from "WARN" to "INFO" in the file
"conf/log4j.xml" (line 22)

I recently received a E-Mail with a similar problem. I gave the following
suggestions, but did never receive an answer if anything helped:

A few generic questions without looking at the logfiles:

1. Are you compiling Hannibal right on the host where you run it or do you
   use the "build_package" script? Please let me know if that's the case,
   maybe there is still some problem with the script.
2. Are you running the latest Hannibal version from the master branch?
   Maybe the problem is already fixed ;-) To update, simply run 'git pull'
   inside your Hannibal directory.

From the logfile output, I can't exactly tell you what's wrong. I see two
possible reasons:
3. the HBase configuration could be wrong (doublecheck that you have the
correct zookeeper-quorum in conf/hbase-site.xml).
4. There could be a problem connecting to Zookeeper. Please try to issue
this command from your hannibal-host: 'echo "stat" | nc zookeeper-hostname
2181' (for each of your zookeeper-nodes as configured in hbase-site.xml,
replace 'zookeeper-hostname' with the correct hostname). The command should
immediately return some statistics about zookeeper.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-17540333
.

[chenhsiu]

@meniku

1. I use the create_package script
2. yes. the latest one
3. I see following logs in log4j.log, after turn on INFO debug level

2013-05-07 16:12:10,228 INFO [application-akka.actor.default-dispatcher-1] ZooKeeper - Session: 0x23e564781076263 closed
2013-05-07 16:12:10,229 INFO [application-akka.actor.default-dispatcher-1] ZooKeeper - Initiating client connection, connectString=spn-d-hdd1.sjdc:2181,spn-d-hdn2.sjdc:2181,spn-d-hdn1.sjdc:2181 sessionTimeout=120000 watcher=hconnection
2013-05-07 16:12:10,230 INFO [application-akka.actor.default-dispatcher-1-EventThread] ClientCnxn - EventThread shut down
2013-05-07 16:12:10,231 INFO [application-akka.actor.default-dispatcher-1-SendThread()] ClientCnxn - Opening socket connection to server spn-d-hdd1.sjdc/10.31.56.35:2181
2013-05-07 16:12:10,232 INFO [application-akka.actor.default-dispatcher-1-SendThread(spn-d-hdd1.sjdc:2181)] ClientCnxn - Socket connection established to spn-d-hdd1.sjdc/10.31.56.35:2181, initiating session
2013-05-07 16:12:10,234 INFO [application-akka.actor.default-dispatcher-1-SendThread(spn-d-hdd1.sjdc:2181)] ClientCnxn - Session establishment complete on server spn-d-hdd1.sjdc/10.31.56.35:2181, sessionid = 0x23e564781076264, negotiated timeout = 120000

[chenhsiu]

@meniku BTW, does hannibal support Kerberos authentication?

[gdtm86]

Chris,

Which HBase version are you using?

Best Regards,
Guru Medasani

On Tue, May 7, 2013 at 11:15 AM, Chris Huang notifications@github.comwrote:

@meniku https://github.com/meniku BTW, does hannibal support Kerberos
authentication?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-17552503
.

[meniku]

@gdtm86 From the exception trace I can see that Hannibal is compiled for HBase 0.90

I wonder whether setting the hbase.master in conf/hbase-site.xml could help?

<property>
<name>hbase.master</name>
<value>master-server:60000</value>
</property>

[gdtm86]

I tied this and still no luck. Seeing the same errors.

-Guru

On Wed, May 8, 2013 at 7:58 AM, NiKu notifications@github.com wrote:

@gdtm86 https://github.com/gdtm86 From the exception trace I can see
that Hannibal is compiled for HBase 0.90

I wonder whether setting the hbase.master in conf/hbase-site.xml could
help?

hbase.master
master-server:60000

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-17603767
.

[chenhsiu]

@gdtm86 I use 0.92
@meniku does hannibal support Kerberos authentication? My Hadoop/HBase cluster has Kerberos enabled. Is that the reason?

[meniku]

@chenhsiu your stacktrace indicates that you compile Hannibal for HBase 0.90, for the packaged version, please use the following command to create the package

 HANNIBAL_HBASE_VERSION=0.92 ./create_package

I actually don't know whether Hannibal supports Kerberos authentication since I never used it. From a quick look in the HBase Documentation it seams that you need to setup your hbase-site.xml properly.

[meniku]

I just had a similiar issue when I built Hannibal for the wrong version. It did not work directly after I changed the version with HANNIBAL_HBASE_VERSION. I had to do a clean prior to rebuild Hannibal.

The following made it work for me:

./sbt clean
HANNIBAL_HBASE_VERSION=0.92 ./create_package

Does this help?

[keyboardfann]

Hi, I test it on HBase 0.94 without security and HANNIBAL work fine.
I test it on HBase 0.94 with security HANNIBAL can't work.

Could you give me some idea ?

Below is the information:

TEST CASE 1:
hbase-site.xml without SecureRpcEngine config, hannibal service can start . When it want to update RegionInfo, it show MasterNotRunningException and hmaster log show Incorrect header or version mismatch. It seems RPC version mismatch.

hbase-site.xml :

[root@ci-test-1 ~]# cat hannibal/conf/hbase-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.cluster.distibuted</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ci-test-1.client.org:8020/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ci-test-1.client.org</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
</configuration>

[hbase@ci-test-1 hannibal]$ ./start

starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /home/hbase/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/home/hbase/hannibal/)

(Starting server. Type Ctrl+D to exit logs, the server will remain in background)

Play server process ID is 23711
[debug] c.j.b.BoneCPDataSource - JDBC URL = jdbc:h2:data/metrics.h2, Username = null, partitions = 1, max (per partition) = 30, min (per partition) = 5, helper threads = 3, idle max age = 10 min, idle test period = 1 min
[warn] c.j.b.BoneCPConfig - JDBC username was not set in config!
[warn] c.j.b.BoneCPConfig - JDBC password was not set in config!
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from ehcache.xml found in the classpath: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from URL: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from InputStream
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xmlns:xsi
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xsi:noNamespaceSchemaLocation
[debug] n.s.e.CacheManager - Creating new CacheManager with default config
[debug] n.s.e.u.PropertyUtil - propertiesString is null.
[debug] n.s.e.CacheManager - No disk store path defined. Skipping disk store path conflict test.
[debug] n.s.e.c.ConfigurationHelper - No CacheManagerEventListenerFactory class specified. Skipping...
[debug] n.s.e.Cache - No BootstrapCacheLoaderFactory class specified. Skipping...
[debug] n.s.e.Cache - CacheWriter factory not configured. Skipping...
[debug] n.s.e.c.ConfigurationHelper - No CacheExceptionHandlerFactory class specified. Skipping...
[debug] n.s.e.s.MemoryStore - Initialized net.sf.ehcache.store.NotifyingMemoryStore for play
[debug] n.s.e.Cache - Initialised cache: play
[debug] n.s.e.c.ConfigurationHelper - CacheDecoratorFactory not configured for defaultCache. Skipping for 'play'.
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@257807a, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Wed Dec 04 11:06:37 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 94ms... (Wed Dec 04 11:06:37 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Wed Dec 04 11:06:39 UTC 2013)
[ERROR] [12/04/2013 11:14:24.139] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] Retried 10 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:122)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.Region$.updateCache(Region.scala:92)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$1.apply$mcV$sp(UpdateMetricsActor.scala:66)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:65)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

[info] application - start initialize Logfile... (Wed Dec 04 11:14:24 UTC 2013)
[ERROR] [12/04/2013 11:22:07.549] [application-akka.actor.default-dispatcher-1] [akka://application/user/updateMetricsActor] Retried 10 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:122)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.LogFile$$anonfun$discoverLogFileUrlPattern$1.apply$mcV$sp(LogFile.scala:119)
    at scala.util.control.Breaks.breakable(Breaks.scala:35)
    at models.LogFile$.discoverLogFileUrlPattern(LogFile.scala:118)
    at models.LogFile$.init(LogFile.scala:156)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$3.apply$mcV$sp(UpdateMetricsActor.scala:85)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:84)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

HBase Master log:

[2013-12-04 11:26:17,384][IPC Reader 9 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7792 got version 3 expected version 4
[2013-12-04 11:26:18,409][IPC Reader 0 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7793 got version 3 expected version 4
[2013-12-04 11:26:19,412][IPC Reader 1 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7794 got version 3 expected version 4
[2013-12-04 11:26:20,420][IPC Reader 2 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7795 got version 3 expected version 4
[2013-12-04 11:26:22,433][IPC Reader 3 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7796 got version 3 expected version 4
[2013-12-04 11:26:24,445][IPC Reader 4 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7797 got version 3 expected version 4
[2013-12-04 11:26:28,486][IPC Reader 5 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7798 got version 3 expected version 4
[2013-12-04 11:26:32,532][IPC Reader 6 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7799 got version 3 expected version 4
[2013-12-04 11:26:40,602][IPC Reader 7 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7800 got version 3 expected version 4

TEST CASE 2:
hbase-site.xml with SecureRpcEngine config, hannibal can't start and show SecureRpcEngine class not found.

[root@ci-test-1 ~]# cat hannibal/conf/hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.cluster.distibuted</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ci-test-1.client.org:8020/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ci-test-1.client.org</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
   <property>
    <name>hbase.rpc.engine</name>
    <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
  </property>
</configuration>

[hbase@tmh6-ci-test-1 hannibal]$ ./start

starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /home/hbase/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/home/hbase/hannibal/)
(Starting server. Type Ctrl+D to exit logs, the server will remain in background)
Play server process ID is 25281
[debug] c.j.b.BoneCPDataSource - JDBC URL = jdbc:h2:data/metrics.h2, Username = null, partitions = 1, max (per partition) = 30, min (per partition) = 5, helper threads = 3, idle max age = 10 min, idle test period = 1 min
[warn] c.j.b.BoneCPConfig - JDBC username was not set in config!
[warn] c.j.b.BoneCPConfig - JDBC password was not set in config!
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from ehcache.xml found in the classpath: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from URL: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from InputStream
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xmlns:xsi
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xsi:noNamespaceSchemaLocation
[debug] n.s.e.CacheManager - Creating new CacheManager with default config
[debug] n.s.e.u.PropertyUtil - propertiesString is null.
[debug] n.s.e.CacheManager - No disk store path defined. Skipping disk store path conflict test.
[debug] n.s.e.c.ConfigurationHelper - No CacheManagerEventListenerFactory class specified. Skipping...
[debug] n.s.e.Cache - No BootstrapCacheLoaderFactory class specified. Skipping...
[debug] n.s.e.Cache - CacheWriter factory not configured. Skipping...
[debug] n.s.e.c.ConfigurationHelper - No CacheExceptionHandlerFactory class specified. Skipping...
[debug] n.s.e.s.MemoryStore - Initialized net.sf.ehcache.store.NotifyingMemoryStore for play
[debug] n.s.e.Cache - Initialised cache: play
[debug] n.s.e.c.ConfigurationHelper - CacheDecoratorFactory not configured for defaultCache. Skipping for 'play'.
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@78497062, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Wed Dec 04 11:28:40 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 133ms... (Wed Dec 04 11:28:41 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Wed Dec 04 11:28:42 UTC 2013)
[ERROR] [12/04/2013 11:28:44.272] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.SecureRpcEngine
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.SecureRpcEngine
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProtocolEngine(HBaseRPC.java:114)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:682)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:110)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.Region$.updateCache(Region.scala:92)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$1.apply$mcV$sp(UpdateMetricsActor.scala:66)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:65)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.SecureRpcEngine
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
    ... 24 more

[tzzhc4]

Was there every an answer about Hannibal supporting Kerberos?

[meniku]

@tzzhc4
Sorry that I never answered this. Unfortunately I have no idea how to set up a test environment for Kerberos. Maybe it would be better if someone else could test this, maybe someone who already has Kerberos running. Possible the best thing would be if we would have a vagrant setup with kerberos enabled?

@keyboardfann My ideas:

case 1

The log message:

[2013-12-04 11:26:17,384][IPC Reader 9 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7792 got version 3 expected version 4
...

Could indicate that the used hadoop version 0.20.205.0 by Hannibal is maybe too old. Could you try to change the used Version in project/Build.scala from

      case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-core" % "0.20.205.0",
        "org.apache.hbase" % "hbase" % "0.94.3"
      )

to

      case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-common" % "0.22.0",
        "org.apache.hbase" % "hbase" % "0.94.14"
      )

and then rebuild the project? I think this is the solution to your first problem and this ticket. Thank you for providing the log-messages.

case 2

I really don't know why the class SecureRpcEngine can not be found. The only thing I found out is, that your second configuration looks not the way it is documented for HBase 0.94.

Instead of

...
   <property>
    <name>hbase.rpc.engine</name>
    <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
  </property>
...

it is documented like

...
      <property>
        <name>hbase.security.authentication</name>
        <value>kerberos</value>
      </property> 

      <property>
        <name>hbase.rpc.protection</name>
        <value>privacy</value>
      </property>
...

Are you sure that your configuration is correct?

[gshively]

The problem with the resolution of SecureRpcEngine seems to be that HBase 0.94 has a security profile. Seems as though the SecureRpcEngine is defined only when the security profile is defined. I tried to add the 0.94.14-security to the project/Build.scala, where it couldn't find it. But changing back to the 0.94.14 worked - and if I replace the hbase-0.94.14.jar with my hbase-0.94.14-security.jar - SecureRpcEngine is resolved.

I also had to upgrade the hadoop-core, which I did to 1.0.4 prior to seeing your message. Looks like I'm still getting errors, but not sure if it is with my JAAS config or something else.