java.io.IOException

JIRA | Andrew Jackson | 1 year ago
  1. 0

    Hi, We're using the AMQPUrlReceiver and in three of four of our crawlers, H3 has stopped picking up messages. In at least two of the three 'hung' crawler jobs, there was an error like this: {code} 2015-09-14 17:13:54.154 SEVERE thread-57 org.archive.crawler.frontier.BdbMultipleWorkQueues.put() URI enqueueing failed; OperationStatus.KEYEXIST http://www.google-analytics.com/r/__utm.gif?utmwv=5.6.6&utms=1&utmn=1086570298&utmhn=sedoparking.com&utmcs=UTF-8&utmsr=1024x768&utmvp=1280x985&utmsc=32-bit&utmul=c&utmje=0&utmfl=-&utmdt=firstclassclaims.co.uk%C2%A0-%C2%A0firstclassclaims%20Resources%20and%20Information.&utmhid=1211709328&utmr=http%3A%2F%2Ffirstclassclaims.co.uk%2F&utmp=1021%2F1&utmht=1442234170949&utmac=UA-19309218-3&utmcc=__utma%3D1.549524138.1442234171.1442234171.1442234171.1%3B%2B__utmz%3D1.1442234171.1.1.utmcsr%3Dfirstclassclaims.co.uk%7Cutmccn%3D(referral)%7Cutmcmd%3Dreferral%7Cutmcct%3D%2F%3B&utmjid=2020018054&utmredir=1&utmu=qhCAAAAAAAAAAAAAAAAAAAAE~ java.lang.RuntimeException at org.archive.crawler.frontier.BdbMultipleWorkQueues.put(BdbMultipleWorkQueues.java:349) at org.archive.crawler.frontier.BdbWorkQueue.insertItem(BdbWorkQueue.java:144) at org.archive.crawler.frontier.WorkQueue.insert(WorkQueue.java:404) at org.archive.crawler.frontier.WorkQueue.enqueue(WorkQueue.java:150) at org.archive.crawler.frontier.WorkQueueFrontier.sendToQueue(WorkQueueFrontier.java:393) at org.archive.crawler.frontier.WorkQueueFrontier.processScheduleAlways(WorkQueueFrontier.java:333) at org.archive.crawler.frontier.AbstractFrontier.receive(AbstractFrontier.java:554) at org.archive.crawler.util.SetBasedUriUniqFilter.addForce(SetBasedUriUniqFilter.java:104) at org.archive.crawler.frontier.WorkQueueFrontier.processScheduleIfUnique(WorkQueueFrontier.java:376) at org.archive.crawler.frontier.WorkQueueFrontier.schedule(WorkQueueFrontier.java:356) at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:236) at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140) at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) {code} and then, a little later, an error like this: {code} DefaultExceptionHandler: Consumer org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer@7c7c4c4a (amq.ctag-lTrPs85xlBDjbVldRAlCqA) method handleDelivery for channel AMQChannel(amqp://guest@192.168.45.26:5672/,1) threw an exception for channel AMQChannel(amqp://guest@192.168.45.26:5672/,1): com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:907) at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:253) at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140) at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 2015-09-14 17:13:56.286 WARNING thread-31 org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent() failed to pause flow on amqp channel java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124) at com.rabbitmq.client.impl.ChannelN.flow(ChannelN.java:1061) at com.rabbitmq.client.impl.ChannelN.flow(ChannelN.java:61) at org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent(AMQPUrlReceiver.java:337) at org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent(AMQPUrlReceiver.java:59) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303) at org.archive.crawler.framework.CrawlController.sendCrawlStateChangeEvent(CrawlController.java:327) at org.archive.crawler.framework.CrawlController.completePause(CrawlController.java:413) at org.archive.crawler.framework.CrawlController.requestCrawlStart(CrawlController.java:364) at org.archive.crawler.framework.CrawlJob$1.run(CrawlJob.java:430) Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: {#method<connection.close>(reply-code=540, reply-text=NOT_IMPLEMENTED - active=false, class-id=20, method-id=20), null, ""} at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118) ... 10 more {code} I believe an uncaught exception in a thread will at least take out that thread, and possibly a whole threadpool. In which case, the error we are seeing will presumably take out the StarterRestarter and prevent any closed RabbitMQ connection from being re-opened. Firstly, does this sound about right? Secondly, should the BDB error we are seeing ever happen? Thirdly, should we switch to catching RuntimeExceptions in the UrlConsumer. Thanks, Andy Jackson

    JIRA | 1 year ago | Andrew Jackson
    java.io.IOException
  2. 0

    Hi, We're using the AMQPUrlReceiver and in three of four of our crawlers, H3 has stopped picking up messages. In at least two of the three 'hung' crawler jobs, there was an error like this: {code} 2015-09-14 17:13:54.154 SEVERE thread-57 org.archive.crawler.frontier.BdbMultipleWorkQueues.put() URI enqueueing failed; OperationStatus.KEYEXIST http://www.google-analytics.com/r/__utm.gif?utmwv=5.6.6&utms=1&utmn=1086570298&utmhn=sedoparking.com&utmcs=UTF-8&utmsr=1024x768&utmvp=1280x985&utmsc=32-bit&utmul=c&utmje=0&utmfl=-&utmdt=firstclassclaims.co.uk%C2%A0-%C2%A0firstclassclaims%20Resources%20and%20Information.&utmhid=1211709328&utmr=http%3A%2F%2Ffirstclassclaims.co.uk%2F&utmp=1021%2F1&utmht=1442234170949&utmac=UA-19309218-3&utmcc=__utma%3D1.549524138.1442234171.1442234171.1442234171.1%3B%2B__utmz%3D1.1442234171.1.1.utmcsr%3Dfirstclassclaims.co.uk%7Cutmccn%3D(referral)%7Cutmcmd%3Dreferral%7Cutmcct%3D%2F%3B&utmjid=2020018054&utmredir=1&utmu=qhCAAAAAAAAAAAAAAAAAAAAE~ java.lang.RuntimeException at org.archive.crawler.frontier.BdbMultipleWorkQueues.put(BdbMultipleWorkQueues.java:349) at org.archive.crawler.frontier.BdbWorkQueue.insertItem(BdbWorkQueue.java:144) at org.archive.crawler.frontier.WorkQueue.insert(WorkQueue.java:404) at org.archive.crawler.frontier.WorkQueue.enqueue(WorkQueue.java:150) at org.archive.crawler.frontier.WorkQueueFrontier.sendToQueue(WorkQueueFrontier.java:393) at org.archive.crawler.frontier.WorkQueueFrontier.processScheduleAlways(WorkQueueFrontier.java:333) at org.archive.crawler.frontier.AbstractFrontier.receive(AbstractFrontier.java:554) at org.archive.crawler.util.SetBasedUriUniqFilter.addForce(SetBasedUriUniqFilter.java:104) at org.archive.crawler.frontier.WorkQueueFrontier.processScheduleIfUnique(WorkQueueFrontier.java:376) at org.archive.crawler.frontier.WorkQueueFrontier.schedule(WorkQueueFrontier.java:356) at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:236) at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140) at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) {code} and then, a little later, an error like this: {code} DefaultExceptionHandler: Consumer org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer@7c7c4c4a (amq.ctag-lTrPs85xlBDjbVldRAlCqA) method handleDelivery for channel AMQChannel(amqp://guest@192.168.45.26:5672/,1) threw an exception for channel AMQChannel(amqp://guest@192.168.45.26:5672/,1): com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:907) at org.archive.crawler.frontier.AMQPUrlReceiver$UrlConsumer.handleDelivery(AMQPUrlReceiver.java:253) at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:140) at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:76) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 2015-09-14 17:13:56.286 WARNING thread-31 org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent() failed to pause flow on amqp channel java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124) at com.rabbitmq.client.impl.ChannelN.flow(ChannelN.java:1061) at com.rabbitmq.client.impl.ChannelN.flow(ChannelN.java:61) at org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent(AMQPUrlReceiver.java:337) at org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent(AMQPUrlReceiver.java:59) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303) at org.archive.crawler.framework.CrawlController.sendCrawlStateChangeEvent(CrawlController.java:327) at org.archive.crawler.framework.CrawlController.completePause(CrawlController.java:413) at org.archive.crawler.framework.CrawlController.requestCrawlStart(CrawlController.java:364) at org.archive.crawler.framework.CrawlJob$1.run(CrawlJob.java:430) Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: {#method<connection.close>(reply-code=540, reply-text=NOT_IMPLEMENTED - active=false, class-id=20, method-id=20), null, ""} at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118) ... 10 more {code} I believe an uncaught exception in a thread will at least take out that thread, and possibly a whole threadpool. In which case, the error we are seeing will presumably take out the StarterRestarter and prevent any closed RabbitMQ connection from being re-opened. Firstly, does this sound about right? Secondly, should the BDB error we are seeing ever happen? Thirdly, should we switch to catching RuntimeExceptions in the UrlConsumer. Thanks, Andy Jackson

    JIRA | 1 year ago | Andrew Jackson
    java.io.IOException
  3. 0

    [rabbitmq-discuss] ShutdownSignalException second 'channel.open'

    Google Groups | 5 years ago | Yogesh Ketkar
    com.rabbitmq.client.ShutdownSignalException: connection error; reason: {#method<connection.close>(reply-code=503, reply-text=COMMAND_INVALID - second 'channel.open' seen, class-id=20, method-id=10), null, "[B@105691e"}
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    java client reconnect error: attempt to reuse consumer tag

    Google Groups | 3 years ago | James Bellenger
    com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering consumer amq.ctag-LOiutBNDPk8Dl4gUWxJgdg
  6. 0

    RabbitMQ cluster fail-over issue

    Stack Overflow | 3 years ago | Pand005
    com.rabbitmq.client.ShutdownSignalException: connection error; reason: {#method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0), null, ""}

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. com.rabbitmq.client.ShutdownSignalException

      connection error; reason: {#method<connection.close>(reply-code=540, reply-text=NOT_IMPLEMENTED - active=false, class-id=20, method-id=20), null, ""}

      at com.rabbitmq.utility.ValueOrException.getValue()
    2. RabbitMQ Java Client
      ChannelN.flow
      1. com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
      2. com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
      3. com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
      4. com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
      5. com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
      6. com.rabbitmq.client.impl.ChannelN.flow(ChannelN.java:1061)
      7. com.rabbitmq.client.impl.ChannelN.flow(ChannelN.java:61)
      7 frames
    3. org.archive.crawler
      AMQPUrlReceiver.onApplicationEvent
      1. org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent(AMQPUrlReceiver.java:337)
      2. org.archive.crawler.frontier.AMQPUrlReceiver.onApplicationEvent(AMQPUrlReceiver.java:59)
      2 frames
    4. Spring Context
      AbstractApplicationContext.publishEvent
      1. org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)
      2. org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
      2 frames
    5. org.archive.crawler
      CrawlJob$1.run
      1. org.archive.crawler.framework.CrawlController.sendCrawlStateChangeEvent(CrawlController.java:327)
      2. org.archive.crawler.framework.CrawlController.completePause(CrawlController.java:413)
      3. org.archive.crawler.framework.CrawlController.requestCrawlStart(CrawlController.java:364)
      4. org.archive.crawler.framework.CrawlJob$1.run(CrawlJob.java:430)
      4 frames