java.io.IOException: Too many open files

DataStax JIRA | Carlos Scheidecker | 12 months ago
  1. 0

    Apache Tomcat Exception - Too many open files

    Stack Overflow | 7 months ago | Nithin K Anil
    java.io.IOException: Too many open files
  2. 0

    Same dev environment with same application on Tomcat 8.0.23. However the dev nodes have been upgraded to 3.0.0 and later to 3.0.1. The Cassandra driver is version 3.0.0-beta1. It seems that connections crash, do not get cleared and it leads to a memory leak stack overflow condition. Attached is an error log file from tomcat. There are 3 Cassandra nodes: .15, .16 and .17. 13:12:33.904 [cluster1-nio-worker-0] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.15:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat 13:12:33.905 [cluster1-nio-worker-0] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.15:9042-1, inFlight=0, closed=false] heartbeat query succeeded 14-Dec-2015 13:12:33.941 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:49.487 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 13:12:49.504 [cluster1-nio-worker-3] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.15:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat 13:12:49.506 [cluster1-nio-worker-3] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.15:9042-2, inFlight=0, closed=false] heartbeat query succeeded 13:12:49.508 [cluster1-nio-worker-1] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.16:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat 13:12:49.511 [cluster1-nio-worker-1] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.16:9042-1, inFlight=0, closed=false] heartbeat query succeeded 13:12:49.522 [cluster1-nio-worker-2] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.17:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat 13:12:49.525 [cluster1-nio-worker-2] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.17:9042-1, inFlight=0, closed=false] heartbeat query succeeded For the node configuration in terms of connections I have the default which is unlimited: The maximum number of concurrent client connections. The default is -1, which means unlimited. native_transport_max_concurrent_connections: -1 The maximum number of concurrent client connections per source ip. The default is -1, which means unlimited. native_transport_max_concurrent_connections_per_ip: -1 I had also checked Tomcat limits and I have no limits on opened files with it: cat /proc/30948/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 256854 256854 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 256854 256854 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us This app was working just fine with latest Cassandra 2.x driver. After upgrading to Cassandra 3.0.0 and later 3.0.1 we had been having the same issues. It happens every 1 or 2 days. On my @Repository class here is the connect() method which is the same as before: private void connect() { if (session == null) { String contactpoints = env.getProperty("cassandra.contactpoints"); PlainTextAuthProvider plainAuthProvider = new PlainTextAuthProvider( env.getProperty("cassandra.username"), env.getProperty("cassandra.password")); // Probably needs to set Pooling Options etc // Cluster.builder().getConfiguration().getPoolingOptions() // Cluster.builder().withReconnectionPolicy(policy) // Cluster.builder().withRetryPolicy(policy) cluster = Cluster.builder() .addContactPoints(contactpoints.split(",")) .withAuthProvider(plainAuthProvider).build(); // connect() without keyspace // Session cs = cluster.connect(); keyspace = env.getProperty("cassandra.oauthKeyspace"); //session = cluster.connect(keyspace); // conect without specifying keyspace session = cluster.connect(); } }

    DataStax JIRA | 12 months ago | Carlos Scheidecker
    java.io.IOException: Too many open files
  3. 0

    Unifi 3.0.8 Update /Problems with NVR - Ubiquiti Networks Community

    ubnt.com | 1 year ago
    java.io.IOException: Too many open files
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Re: NIO Connector: Too many open files

    apache.org | 1 year ago
    java.io.IOException: Too many open files

  1. rp 5 times, last 8 months ago
6 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.IOException

    Too many open files

    at sun.nio.ch.ServerSocketChannelImpl.accept0()
  2. Java RT
    ServerSocketChannelImpl.accept
    1. sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    2. sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    3. sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    3 frames
  3. Grizzly HTTP
    NioEndpoint$Acceptor.run
    1. org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
    1 frame
  4. Java RT
    Thread.run
    1. java.lang.Thread.run(Thread.java:745)
    1 frame