net.sf.ehcache.distribution.RemoteCacheException: Error doing put to remote peer. Message was: Error unmarshaling return header; nested exception is: java.net.SocketTimeoutException: Read timed out

Terracotta Project Issue Tracker | Sourceforge Tracker | 7 years ago
  1. 0

    When CAS sends the PGT back to a clustered client, that cluster must perform synchronous replication, as the client will retrieve the PGT from the storage within milliseconds. However, when one of the nodes (configured for manual peer discovery because multicast does not work at that site) goes down, Ehcache throws the following exception: SEVERE: Servlet.service() for servlet default threw exception net.sf.ehcache.distribution.RemoteCacheException: Error doing put to remote peer. Message was: Connection refused to host: AAA.BBB.CCC.DDD; nested exception is: java.net.ConnectException: Connection refused at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:136) at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementPut(RMISynchronousCacheReplicator.java:119) at net.sf.ehcache.event.RegisteredEventListeners.notifyElementPut(RegisteredEventListeners.java:102) at net.sf.ehcache.Cache.put(Cache.java:699) at net.sf.ehcache.Cache.put(Cache.java:632) at org.jasig.cas.client.proxy.EhcacheBackedProxyGrantingTicketStorageImpl.save(EhcacheBackedProxyGrantingTicketStorageImpl.java:34) at org.jasig.cas.client.util.CommonUtils.readAndRespondToProxyReceptorRequest(CommonUtils.java:184) at org.jasig.cas.client.validation.Cas20ProxyReceivingTicketValidationFilter.preFilter(Cas20ProxyReceivingTicketValidationFilter.java:160) at org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:150) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:619) This exception propagates all the way to Tomcat (it's re-thrown in 3.1.12), which causes the application server to report HTTP result code 500. When CAS receives that code, it does not include the PGTIOU in the /serviceValidate response even though all the live client nodes have received the PGT. I think it should be OK for EhcacheBackedProxyGrantingTicketStorageImpl to catch the RemoteCacheException when replicating synchronously, log it, and NOT propagate it to allow CAS to operate normally. After all, one of the reasons to have a cluster is to be able to take down a node and continue operating the cluster.

    Apereo Issues | 6 years ago | Adam Rybicki
    net.sf.ehcache.distribution.RemoteCacheException: Error doing put to remote peer. Message was: Connection refused to host: AAA.BBB.CCC.DDD; nested exception is: java.net.ConnectException: Connection refused
  2. 0

    This generates stack trace such as: {noformat} 2014-08-01 07:15:53,459 main WARN [net.sf.ehcache.CacheManager] Cache com.atlassian.jira.task.TaskManagerImpl.taskMaprequested bootstrap but a CacheException occured. Error bootstrapping from remote peer. Message was: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.atlassian.jira.i18n.BackingI18n net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.atlassian.jira.i18n.BackingI18n at net.sf.ehcache.distribution.RMIBootstrapCacheLoader.doLoad(RMIBootstrapCacheLoader.java:176) at net.sf.ehcache.distribution.RMIBootstrapCacheLoader.load(RMIBootstrapCacheLoader.java:77) at net.sf.ehcache.Cache.bootstrap(Cache.java:1364) at net.sf.ehcache.CacheManager.initializeEhcache(CacheManager.java:1388) at net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:1437) at net.sf.ehcache.CacheManager.addConfiguredCaches(CacheManager.java:839) at net.sf.ehcache.CacheManager.doInit(CacheManager.java:498) at net.sf.ehcache.CacheManager.init(CacheManager.java:392) at net.sf.ehcache.CacheManager.<init>(CacheManager.java:265) at net.sf.ehcache.CacheManager.newInstance(CacheManager.java:1123) at net.sf.ehcache.CacheManager.newInstance(CacheManager.java:1099) at com.atlassian.jira.CacheManagerRegistrar.getCacheManagerDelegate(CacheManagerRegistrar.java:130) {noformat}

    Atlassian JIRA | 2 years ago | Trevor Campbell [Atlassian]
    net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.atlassian.jira.i18n.BackingI18n
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    This generates stack trace such as: {noformat} 2014-08-01 07:15:53,459 main WARN [net.sf.ehcache.CacheManager] Cache com.atlassian.jira.task.TaskManagerImpl.taskMaprequested bootstrap but a CacheException occured. Error bootstrapping from remote peer. Message was: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.atlassian.jira.i18n.BackingI18n net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.atlassian.jira.i18n.BackingI18n at net.sf.ehcache.distribution.RMIBootstrapCacheLoader.doLoad(RMIBootstrapCacheLoader.java:176) at net.sf.ehcache.distribution.RMIBootstrapCacheLoader.load(RMIBootstrapCacheLoader.java:77) at net.sf.ehcache.Cache.bootstrap(Cache.java:1364) at net.sf.ehcache.CacheManager.initializeEhcache(CacheManager.java:1388) at net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:1437) at net.sf.ehcache.CacheManager.addConfiguredCaches(CacheManager.java:839) at net.sf.ehcache.CacheManager.doInit(CacheManager.java:498) at net.sf.ehcache.CacheManager.init(CacheManager.java:392) at net.sf.ehcache.CacheManager.<init>(CacheManager.java:265) at net.sf.ehcache.CacheManager.newInstance(CacheManager.java:1123) at net.sf.ehcache.CacheManager.newInstance(CacheManager.java:1099) at com.atlassian.jira.CacheManagerRegistrar.getCacheManagerDelegate(CacheManagerRegistrar.java:130) {noformat}

    Atlassian JIRA | 2 years ago | Trevor Campbell [Atlassian]
    net.sf.ehcache.distribution.RemoteCacheException: Error bootstrapping from remote peer. Message was: error unmarshalling return; nested exception is: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.atlassian.jira.i18n.BackingI18n

    Root Cause Analysis

    1. net.sf.ehcache.distribution.RemoteCacheException

      Error doing put to remote peer. Message was: Error unmarshaling return header; nested exception is: java.net.SocketTimeoutException: Read timed out

      at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification()
    2. Ehcache
      Cache.put
      1. net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicatePutNotification(RMISynchronousCacheReplicator.java:135)
      2. net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementUpdated(RMISynchronousCacheReplicator.java:170)
      3. net.sf.ehcache.event.RegisteredEventListeners.notifyElementUpdated(RegisteredEventListeners.java:103)
      4. net.sf.ehcache.Cache.put(Cache.java:574)
      5. net.sf.ehcache.Cache.put(Cache.java:522)
      5 frames