java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException

Apache's JIRA Issue Tracker | Rudra Sharma | 11 months ago
  1. 0

    When deploying to a nimbus a topology which is larger in size >17MB, we get an exception. In storm 0.9.3 this could be mitigated by using the following config on the storm.yaml to increse the buffer size to handle the topology size. i.e. 50MB would be nimbus.thrift.max_buffer_size: 50000000 This configuration does not resolve the issue in the master branch of storm and we cannot deploy topologies which are large in size. Here is the log on the client side when attempting to deploy to the nimbus node: java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) [siem-ng-storm-deployer-cloud.jar:] at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) [siem-ng-storm-deployer-cloud.jar:] Caused by: org.apache.thrift7.transport.TTransportException at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] ... 4 more Here is the log on the server side (nimbus.log): 2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar 2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar 2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from client: /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar 2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] Exception while invoking! org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger than max length (16384000)! at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137) at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.thrift7.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    Apache's JIRA Issue Tracker | 11 months ago | Rudra Sharma
    java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
  2. 0

    When deploying to a nimbus a topology which is larger in size >17MB, we get an exception. In storm 0.9.3 this could be mitigated by using the following config on the storm.yaml to increse the buffer size to handle the topology size. i.e. 50MB would be nimbus.thrift.max_buffer_size: 50000000 This configuration does not resolve the issue in the master branch of storm and we cannot deploy topologies which are large in size. Here is the log on the client side when attempting to deploy to the nimbus node: java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) [siem-ng-storm-deployer-cloud.jar:] at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) [siem-ng-storm-deployer-cloud.jar:] Caused by: org.apache.thrift7.transport.TTransportException at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] ... 4 more Here is the log on the server side (nimbus.log): 2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar 2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar 2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from client: /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar 2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] Exception while invoking! org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger than max length (16384000)! at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137) at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.thrift7.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    Apache's JIRA Issue Tracker | 11 months ago | Rudra Sharma
    java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Storm dprc thrift7.transport.TTransportException: Frame size (1213486160) larger than max length (1048576)!

    Stack Overflow | 12 months ago | hai046
    org.apache.thrift7.transport.TTransportException: Frame size (1213486160) larger than max length (1048576)!
  5. 0

    Not able to submit the storm-starter-0.9.4 topology with the storm 0.9.4 version

    narkive.com | 1 year ago
    org.apache.thrift7.transport.TTransportException

    4 unregistered visitors

    Root Cause Analysis

    1. org.apache.thrift7.transport.TTransportException

      No message provided

      at org.apache.thrift7.transport.TIOStreamTransport.read()
    2. org.apache.thrift7
      TServiceClient.receiveBase
      1. org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      2. org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      3. org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      4. org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      5. org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      6. org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      7. org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      8. org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      9. org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      9 frames
    3. backtype.storm.generated
      Nimbus$Client.submitTopology
      1. backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      2. backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      2 frames
    4. backtype.storm
      StormSubmitter.submitTopology
      1. backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      2. backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      3. backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155)[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      3 frames
    5. com.trustwave.siem
      TopologyDeployer.main
      1. com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149)[siem-ng-storm-deployer-cloud.jar:]
      2. com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87)[siem-ng-storm-deployer-cloud.jar:]
      2 frames