org.apache.accumulo.tserver.HoldTimeoutException: Commits are held

accumulo-user | mohit.kaushik | 11 months ago
  1. 0

    Re:Tablet servers died (exceeded maximum hold time)

    accumulo-user | 11 months ago | mohit.kaushik
    org.apache.accumulo.tserver.HoldTimeoutException: Commits are held
  2. 0

    Users - Mutation Rejected exception with server Error 1

    nabble.com | 3 months ago
    org.apache.accumulo.tserver.HoldTimeoutException: Commits are held
  3. 0

    Re:Tablet servers died (exceeded maximum hold time)

    accumulo-user | 11 months ago | mohit.kaushik
    org.apache.accumulo.tserver.HoldTimeoutException: Commits are held
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    I was running continuous ingest on a 7 node cluster (5 slaves) and after enabling HDFS agitation, my clients died. {code:title=ingest.err} Thread "org.apache.accumulo.test.continuous.ContinuousIngest" died java.lang.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.start.Main$1.run(Main.java:137) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.UndeclaredThrowableException at $Proxy9.addMutation(Unknown Source) at org.apache.accumulo.test.continuous.ContinuousIngest.main(ContinuousIngest.java:212) ... 6 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.trace.instrument.TraceProxy$2.invoke(TraceProxy.java:43) ... 8 more Caused by: org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0 security codes: {} # server errors 1 # exceptions 0 at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:258) at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:43) ... 12 more {code} {code:title=ingest.out} UUID 1392844086463 f822a6a9-9592-4b3a-ab3b-1c172be20b96 FLUSH 1392844135523 49047 6165 1000000 1000000 FLUSH 1392844165594 30071 7787 2000000 1000000 FLUSH 1392844195875 30281 7816 3000000 1000000 FLUSH 1392844226787 30912 8086 4000000 1000000 FLUSH 1392844257194 30407 7989 5000000 1000000 FLUSH 1392844287518 30324 7743 6000000 1000000 FLUSH 1392844325833 38315 10933 7000000 1000000 FLUSH 1392844364708 38875 7916 8000000 1000000 FLUSH 1392844395818 31110 8104 9000000 1000000 2014-02-19 13:16:57,444 [impl.TabletServerBatchWriter] ERROR: Server side error on tserver1:10011: org.apache.thrift.TApplicationException: Internal error processing applyUpdates 2014-02-19 13:16:57,446 [impl.TabletServerBatchWriter] ERROR: Failed to send tablet server tserver1:10011 its batch : Error on server tserver1:10011 org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server tserver1:10011 at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:937) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$1600(TabletServerBatchWriter.java:616) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:801) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:765) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47) at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.thrift.TApplicationException: Internal error processing applyUpdates at org.apache.thrift.TApplicationException.read(TApplicationException.java:108) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:431) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:417) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:899) ... 11 more {code} {code:title=tserver.log} 2014-02-19 13:16:56,156 [util.TServerUtils$THsHaServer] WARN : Got an IOException in internalRead! java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) at sun.nio.ch.IOUtil.read(IOUtil.java:171) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243) at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141) at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:515) at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:355) at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:202) at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:198) at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154) {code} Note that this last message was not propagated to the monitor for some reason, but that is likely a different issue. (I had been seeing other WARN messages show up earlier.)

    Apache's JIRA Issue Tracker | 3 years ago | Mike Drob
    org.apache.accumulo.tserver.HoldTimeoutException: Commits are held

    Root Cause Analysis

    1. org.apache.accumulo.tserver.HoldTimeoutException

      Commits are held

      at org.apache.accumulo.tserver.TabletServerResourceManager.waitUntilCommitsAreEnabled()
    2. org.apache.accumulo
      TabletServer$ThriftClientHandler.closeUpdate
      1. org.apache.accumulo.tserver.TabletServerResourceManager.waitUntilCommitsAreEnabled(TabletServerResourceManager.java:493)
      2. org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.flush(TabletServer.java:795)
      3. org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.closeUpdate(TabletServer.java:952)
      3 frames
    3. Java RT
      Method.invoke
      1. sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      2. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      3. java.lang.reflect.Method.invoke(Method.java:606)
      3 frames
    4. org.apache.accumulo
      RpcWrapper$1.invoke
      1. org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
      2. org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:47)
      2 frames
    5. com.sun.proxy
      $Proxy19.closeUpdate
      1. com.sun.proxy.$Proxy19.closeUpdate(Unknown Source)
      1 frame
    6. Core
      TabletClientService$Processor$closeUpdate.getResult
      1. org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.getResult(TabletClientService.java:2446)
      2. org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.getResult(TabletClientService.java:2430)
      2 frames
    7. Apache Thrift
      TBaseProcessor.process
      1. org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      2. org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      2 frames
    8. org.apache.accumulo
      TimedProcessor.process
      1. org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
      1 frame
    9. Apache Thrift
      AbstractNonblockingServer$FrameBuffer.invoke
      1. org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
      1 frame
    10. org.apache.accumulo
      CustomNonBlockingServer$1.run
      1. org.apache.accumulo.server.rpc.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:78)
      1 frame
    11. Java RT
      ThreadPoolExecutor$Worker.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      2 frames
    12. org.apache.accumulo
      LoggingRunnable.run
      1. org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
      1 frame
    13. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)
      1 frame