hudson.AbortException: Timeout while waiting for sufficient resources: 1,024 cpu units, 3,200mb free memory

Google Groups | lu...@vvoosh.com (JIRA) | 5 months ago
  1. 0

    [JIRA] (JENKINS-36468) Tasks fail to run on ECS after upgrade to 1.4

    Google Groups | 5 months ago | lu...@vvoosh.com (JIRA)
    hudson.AbortException: Timeout while waiting for sufficient resources: 1,024 cpu units, 3,200mb free memory
  2. 0

    With the latest plugin version (1.4) my tasks fail to run on the ecs slaves. They sit waiting for executors forever. I made sure I had two servers with plenty of resources connected to ECS. It looks like the task is assigned to a node but then immediately disconnects. After downgrading to version 1.3 it all started working perfectly again. I tested it with the same number of servers, same tasks, same jenkins versions, etc. See the logs below for both versions. Version 1.4 failures: {quote} Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 3,767mb of free memory. 3,000mb are required Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/ceb4eece-2abe-4f19-aad3-8d3cbd18a7ee has 1,024 units of free cpu. 512 units are required Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e94238750e85 Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure to run task with definition arn:aws:ecs:eu-west-1:AWS_ACC:task-definition/jenkins-slave:78 on ECS cluster arn:aws:ecs:eu-west-1:AWS_ACC:cluster/JenkinsSlaves Jul 06, 2016 8:37:33 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService runEcsTask WARNING: Slave JenkinsSlavesECS-16e94238750e85 - Failure reason=RESOURCE:MEMORY, arn=arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent ECS Slave BackendTests failed to launch java.lang.IllegalStateException: ECS Slave JenkinsSlavesECS-16e9422efc9bc3 (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/b6621ca4-b397-46ad-9659-9c6fe1633b32) not connected since 7/6/16 8:37 AM seconds at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent ECS Slave BackendTests failed to launch hudson.AbortException: Failed to run slave container JenkinsSlavesECS-16e94238750e85 at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:157) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:250) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.816 Jul 06, 2016 8:37:43 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: -0.184 Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources .......... Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Resources found in instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601: [{Name: CPU,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 1024,StringSetValue: []}, {Name: MEMORY,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 3767,StringSetValue: []}, {Name: PORTS,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: [22, 2376, 2375, 51678]}, {Name: PORTS_UDP,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: []}] Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 3,767mb of free memory. 3,000mb are required Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources INFO: Instance arn:aws:ecs:eu-west-1:AWS_ACC:container-instance/e949c164-1e0c-4e9c-85c3-36b035b5b601 has 1,024 units of free cpu. 512 units are required Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e9448c0da1df Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Slave JenkinsSlavesECS-16e9448c0da1df - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da Jul 06, 2016 8:37:43 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call WARNING: ECS Slave JenkinsSlavesECS-16e9448c0da1df (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/d8ad05da-902d-4c10-8b58-c55f612bb0da) not connected since 7/6/16 8:37 AM seconds Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained INFO: Started termination Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins$21 onAttained INFO: Completed termination Jul 06, 2016 8:37:43 AM jenkins.model.Jenkins _cleanUpDisconnectComputers INFO: Starting node disconnection {quote} After downgrade to version 1.3, working well: {quote} Jul 06, 2016 8:40:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning ECS Slave BackendTests from JenkinsSlavesECS with 1 executors. Remaining excess workload: 0.303 Jul 06, 2016 8:40:06 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Created Slave: JenkinsSlavesECS-16e965bf8c0ffe Jul 06, 2016 8:40:07 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: Slave JenkinsSlavesECS-16e965bf8c0ffe - Slave Task Started : arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c Jul 06, 2016 8:40:11 AM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #1 from /XX.XX.XX.XX:YY Jul 06, 2016 8:40:15 AM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call INFO: ECS Slave JenkinsSlavesECS-16e965bf8c0ffe (ecs task arn:aws:ecs:eu-west-1:AWS_ACC:task/efa27aa4-599c-4da8-89fb-a8e21740882c) connected Jul 06, 2016 8:40:16 AM hudson.slaves.NodeProvisioner$2 run INFO: ECS Slave BackendTests provisioning successfully completed. We have now 3 computer(s) {quote}

    Jenkins JIRA | 5 months ago | Luis Silva
    hudson.AbortException: Failed to run slave container JenkinsSlavesECS-16e94238750e85
  3. 0

    We're running Jenkins 1.598 with subversion plugin 2.5. One of our two build slaves is running ubuntu 11.10; the other (jslave05) is a newly created box running ubuntu 14.04. Our repository is weird and won't work with subversion 1.8, which is installed on the slave. That said, my understanding is that subversion-plugin doesn't use the svn version on slaves and instead checks out using SVNKit. We have subversion-plugin set to 1.6. When a build is run for the second time on jslave05 (i.e. when the workspace exists already and it just needs to update it) it fails immediately with the following stacktrace. Removing the workspace and rebuilding seems to fix it for the next run, but the next after that will fail. {code} Started by user mdonahue [EnvInject] - Loading node environment variables. Building remotely on jslave05 (tools workdev) in workspace /home/jenkins/hudson_builds/workspace/Build_Project-dev-PRJ-21533 FATAL: null java.lang.NullPointerException at java.util.EnumMap.<init>(EnumMap.java:138) at org.tmatesoft.svn.core.internal.db.SVNSqlJetDb.<init>(SVNSqlJetDb.java:61) at org.tmatesoft.svn.core.internal.db.SVNSqlJetDb.open(SVNSqlJetDb.java:119) at org.tmatesoft.svn.core.internal.wc17.db.SVNWCDb.openDb(SVNWCDb.java:4827) at org.tmatesoft.svn.core.internal.wc17.db.SVNWCDb.parseDir(SVNWCDb.java:1901) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.detectWcGeneration(SvnOperationFactory.java:1696) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.getImplementation(SvnOperationFactory.java:1342) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1248) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNWCClient.doInfo(SVNWCClient.java:2485) at hudson.scm.subversion.UpdateUpdater$TaskImpl.parseSvnInfo(UpdateUpdater.java:125) at hudson.scm.subversion.UpdateUpdater$TaskImpl.getSvnCommandToUse(UpdateUpdater.java:87) at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:130) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:991) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:972) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:948) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2685) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to jslave05(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:752) at hudson.FilePath.act(FilePath.java:978) at hudson.FilePath.act(FilePath.java:967) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:897) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:833) at hudson.scm.SCM.checkout(SCM.java:484) at hudson.model.AbstractProject.checkout(AbstractProject.java:1270) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:531) at hudson.model.Run.execute(Run.java:1718) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) {code} I also see the following message in the jenkins log on the master: {code} Feb 27, 2015 11:48:52 AM WARNING hudson.model.AbstractBuild$AbstractBuildExecution reportError Publisher hudson.tasks.junit.JUnitResultArchiver aborted due to exception hudson.AbortException: Test reports were found but none of them are new. Did tests run? For example, /home/jenkins/hudson_builds/workspace/Build_PROJECT-dev-PRJ-21533/project/app/build/test/test-results/TEST-Firefox_310_Linux.ProjectTestCase.xml is 1 hr 10 min old at hudson.tasks.junit.TestResult.parse(TestResult.java:178) at hudson.tasks.junit.TestResult.parse(TestResult.java:146) at hudson.tasks.junit.TestResult.<init>(TestResult.java:122) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:119) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:92) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2685) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to jslave05(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:752) at hudson.FilePath.act(FilePath.java:978) at hudson.FilePath.act(FilePath.java:967) at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:89) at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:121) at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:138) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:74) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:721) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:670) at hudson.model.Run.execute(Run.java:1743) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) {code}

    Jenkins JIRA | 2 years ago | Mason Donahue
    hudson.AbortException: Test reports were found but none of them are new. Did tests run? For example, /home/jenkins/hudson_builds/workspace/Build_PROJECT-dev-PRJ-21533/project/app/build/test/test-results/TEST-Firefox_310_Linux.ProjectTestCase.xml is 1 hr 10 min old
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Running a multimodule build with the following parameters {code} mvn clean install -Dmaven.test.skip=true {code} will cause {code} [INFO] --- jasmine-maven-plugin:1.3.1.4:test (default) @ lh-ress-ui --- [INFO] Skipping Jasmine Specs [JENKINS] Zeichne Testergebnisse auf hudson.AbortException: Test reports were found but none of them are new. Did tests run? For example, /home/jenkinslocal/jenkins-jobs/workspace/lufthansa-reactor-master-SNAPSHOT-NEW/lufthansa-ress-ui/target/jasmine/TEST-jasmine.xml is 44 Jahre old at hudson.tasks.junit.TestResult.parse(TestResult.java:213) at hudson.maven.reporters.SurefireArchiver.postExecute(SurefireArchiver.java:148) at hudson.maven.Maven3Builder$MavenExecutionListener.recordMojoEnded(Maven3Builder.java:628) at hudson.maven.Maven3Builder$MavenExecutionListener.mojoSucceeded(Maven3Builder.java:610) at hudson.maven.Maven3Builder$JenkinsEventSpy.onEvent(Maven3Builder.java:306) at org.apache.maven.eventspy.internal.EventSpyDispatcher.onEvent(EventSpyDispatcher.java:108) at org.apache.maven.eventspy.internal.EventSpyExecutionListener.mojoSucceeded(EventSpyExecutionListener.java:131) at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:87) at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:227) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152) at org.jvnet.hudson.maven3.launcher.Maven31Launcher.main(Maven31Launcher.java:132) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238) at jenkins.maven3.agent.Maven31Main.launch(Maven31Main.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at hudson.maven.Maven3Builder.call(Maven3Builder.java:134) at hudson.maven.Maven3Builder.call(Maven3Builder.java:69) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [INFO] {code} If setting {code} -DskipTests=true {code} is added, everything works normal but then also the unit tests will not be executed.

    Jenkins JIRA | 2 years ago | Frank Mueller
    hudson.AbortException: Test reports were found but none of them are new. Did tests run? For example, /home/jenkinslocal/jenkins-jobs/workspace/lufthansa-reactor-master-SNAPSHOT-NEW/lufthansa-ress-ui/target/jasmine/TEST-jasmine.xml is 44 Jahre old
  6. 0

    builds aborted due to time out in agent connection?

    GitHub | 2 years ago | adriaanm
    hudson.AbortException: no workspace for scala-2.11.x-validate-test #107

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. hudson.AbortException

      Timeout while waiting for sufficient resources: 1,024 cpu units, 3,200mb free memory

      at com.cloudbees.jenkins.plugins.amazonecs.ECSService.waitForSufficientClusterResources()
    2. com.cloudbees.jenkins
      ECSCloud$ProvisioningCallback.call
      1. com.cloudbees.jenkins.plugins.amazonecs.ECSService.waitForSufficientClusterResources(ECSService.java:199)
      2. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:237)
      3. com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:219)
      3 frames
    3. jenkins.util
      ContextResettingExecutorService$2.call
      1. jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      1 frame
    4. Java RT
      ThreadPoolExecutor$Worker.run
      1. java.util.concurrent.FutureTask.run(FutureTask.java:262)
      2. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      3. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      3 frames