org.apache.cassandra.exceptions.RepairException: [repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c on logdata/onlinedata, (-143721749331492309,-139544903266258032]] Sync failed between /10.9.9.241 and /10.6.8.212

Apache's JIRA Issue Tracker | Andreas Schnitzerling | 2 years ago
  1. 0

    During repair, connections are closing and throwing exceptions. CPU is running on 100%, when error occurs. My test-configuration is one node w/ 2.1.2 and 11 nodes w/ 2.0.11. If I make repair either on 2.1 or 2.0 I get such an error. But if I have 2.0 everywhere istalled, no error. 2.0 nodes make "endless" repair in that circumstance. Seems to be incompatibility between 2.0 and 2.1. {panel:title=system.log} ERROR [STREAM-OUT-/10.6.8.212] 2014-11-18 12:28:34,948 StreamSession.java:472 - [Stream #0866dc80-6f16-11e4-bc5c-5fe413b6852c] Streaming error occurred java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:1.7.0_55] at sun.nio.ch.SocketDispatcher.write(Unknown Source) ~[na:1.7.0_55] at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source) ~[na:1.7.0_55] at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_55] at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[na:1.7.0_55] at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:346) [apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:326) [apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at java.lang.Thread.run(Unknown Source) [na:1.7.0_55] ERROR [AntiEntropySessions:1] 2014-11-18 12:28:34,948 RepairSession.java:303 - [repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c] session completed with the following error org.apache.cassandra.exceptions.RepairException: [repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c on logdata/onlinedata, (-143721749331492309,-139544903266258032]] Sync failed between /10.9.9.241 and /10.6.8.212 at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:389) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:126) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_55] at java.lang.Thread.run(Unknown Source) [na:1.7.0_55] {panel} Since in windows only parallel repair is possible, is there a way to throttle CPU-consumption? I reduced rpc_X_threads to 4 and concurrent_reads/writes to 4. But no change. On other nodes is C* 2.0.10 and nothing in their system.log.

    Apache's JIRA Issue Tracker | 2 years ago | Andreas Schnitzerling
    org.apache.cassandra.exceptions.RepairException: [repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c on logdata/onlinedata, (-143721749331492309,-139544903266258032]] Sync failed between /10.9.9.241 and /10.6.8.212
  2. 0

    Cassandra exceptions during repair

    Stack Overflow | 3 years ago | Caggy
    org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
  3. 0

    Cassandra nodetool repair freezes entire cluster

    Stack Overflow | 2 years ago | Brent Haines
    org.apache.cassandra.exceptions.RepairException: [repair #5f37c2e0-052b-11e4-92f5-b9bfa38ef354 on apps/ids_by_text, (-7683110849073497716,-7679039947314690170]] Sync failed between /10.0.2.166 and /10.0.2.163
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    nodetool repair stalled

    Google Groups | 3 years ago | Paolo Crosato
    org.apache.cassandra.exceptions.RepairException: [repair #728f4860-79d3-11e3-8c98-a1357a0d9222 on OpsCenter/rollups300, (4515884230644880127,4556138740897423021]] Sync failed between /10.255.235.18 and /10.255.235.19
  6. 0

    delay/stall processing reads

    Google Groups | 3 years ago | Thunder Stumpges
    org.apache.cassandra.exceptions.RepairException: [repair #179e84f0-775f-11e3-9c6a-538c9374b226 on test_video/videosearchresult, (9124197757189535229,9176696932386753491]] Validation failed in /10.20.2.11

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.cassandra.exceptions.RepairException

      [repair #e10d0240-6f15-11e4-bc5c-5fe413b6852c on logdata/onlinedata, (-143721749331492309,-139544903266258032]] Sync failed between /10.9.9.241 and /10.6.8.212

      at org.apache.cassandra.repair.RepairSession.syncComplete()
    2. org.apache.cassandra
      MessageDeliveryTask.run
      1. org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
      2. org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:389)[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
      3. org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:126)[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
      4. org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)[apache-cassandra-2.1.2-SNAPSHOT.jar:2.1.2-SNAPSHOT]
      4 frames
    3. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)[na:1.7.0_55]
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)[na:1.7.0_55]
      3. java.lang.Thread.run(Unknown Source)[na:1.7.0_55]
      3 frames