com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:203) 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:141) 2015-11-06 12:46:09.768484500 at okio.Okio$2.read(Okio.java:137) 2015-11-06 12:46:09.768484500 at okio.AsyncTimeout$2.read(AsyncTimeout.java:217) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285) 2015-11-06 12:46:09.768487500 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) 2015-11-06 12:46:09.768487500 at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.Call.getResponse(Call.java:274) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) 2015-11-06 12:46:09.768497500 at com.squareup.okhttp.Call.execute(Call.java:81) 2015-11-06 12:46:09.768497500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225) 2015-11-06 12:46:09.768498500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source) 2015-11-06 12:46:09.768498500 at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93) 2015-11-06 12:46:09.768500500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364) 2015-11-06 12:46:09.768508500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62) 2015-11-06 12:46:09.768510500 at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

Atlassian JIRA | Andre Borzzatto [Atlassian] | 1 year ago
  1. 0

    It's reproducible by adding attachments on issues. It's also intermittent. By the time JIRA tries to sync with Blobstore, the following errors appears: {code} 2015-11-07 01:52:19.881416500 2015-11-07 01:52:19,881 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout 2015-11-07 01:52:20.084498500 2015-11-07 01:52:20,084 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.091263500 2015-11-07 01:52:20,091 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout 2015-11-07 01:52:20.234413500 2015-11-07 01:52:20,234 blobstore-client-http:thread-614 WARN sysadmin 111x144165x4 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout 2015-11-07 01:52:20.293592500 2015-11-07 01:52:20,293 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.436838500 2015-11-07 01:52:20,436 blobstore-client-http:thread-614 WARN sysadmin 111x144165x4 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.486762500 2015-11-07 01:52:20,486 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.696227500 2015-11-07 01:52:20,696 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed {code} {code} 2015-11-06 12:46:09.768474500 2015-11-06 12:46:09,764 ajp-nio-127.0.0.104-8009-exec-137 WARN sysadmin 764x38599x5 pp5zew 104.192.142.137,104.192.141.18 /rest/jddap/1.0/attachment [c.a.j.issue.thumbnail.DefaultThumbnailManager] Error writing to thumbnail file: /data/jirastudio/jira/home/data/attachments/NEE/10000/NEE-7175/thumbs/_thumb_24718.png7966518938032165601.tmp 2015-11-06 12:46:09.768476500 com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:203) 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:141) 2015-11-06 12:46:09.768484500 at okio.Okio$2.read(Okio.java:137) 2015-11-06 12:46:09.768484500 at okio.AsyncTimeout$2.read(AsyncTimeout.java:217) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285) 2015-11-06 12:46:09.768487500 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) 2015-11-06 12:46:09.768487500 at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.Call.getResponse(Call.java:274) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) 2015-11-06 12:46:09.768497500 at com.squareup.okhttp.Call.execute(Call.java:81) 2015-11-06 12:46:09.768497500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225) 2015-11-06 12:46:09.768498500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source) 2015-11-06 12:46:09.768498500 at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93) 2015-11-06 12:46:09.768500500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364) 2015-11-06 12:46:09.768508500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62) 2015-11-06 12:46:09.768510500 at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2015-11-06 12:46:09.768513500 at java.lang.Thread.run(Thread.java:745) 2015-11-06 12:46:09.768513500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:107) 2015-11-06 12:46:09.768515500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:101) 2015-11-06 12:46:09.768515500 at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216) 2015-11-06 12:46:09.768516500 at com.atlassian.fugue.Either$Left.fold(Either.java:508) 2015-11-06 12:46:09.768516500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$15.apply(BlobStoreAttachmentStore.java:367) 2015-11-06 12:46:09.768519500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$15.apply(BlobStoreAttachmentStore.java:363) 2015-11-06 12:46:09.768520500 at com.atlassian.util.concurrent.Promises$Of$1.apply(Promises.java:263) 2015-11-06 12:46:09.768520500 at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:185) 2015-11-06 12:46:09.768521500 at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319) 2015-11-06 12:46:09.768522500 at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) {code} h3. This is the public report of an internal incident.

    Atlassian JIRA | 1 year ago | Andre Borzzatto [Atlassian]
    com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:203) 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:141) 2015-11-06 12:46:09.768484500 at okio.Okio$2.read(Okio.java:137) 2015-11-06 12:46:09.768484500 at okio.AsyncTimeout$2.read(AsyncTimeout.java:217) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285) 2015-11-06 12:46:09.768487500 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) 2015-11-06 12:46:09.768487500 at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.Call.getResponse(Call.java:274) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) 2015-11-06 12:46:09.768497500 at com.squareup.okhttp.Call.execute(Call.java:81) 2015-11-06 12:46:09.768497500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225) 2015-11-06 12:46:09.768498500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source) 2015-11-06 12:46:09.768498500 at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93) 2015-11-06 12:46:09.768500500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364) 2015-11-06 12:46:09.768508500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62) 2015-11-06 12:46:09.768510500 at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2. 0

    It's reproducible by adding attachments on issues. It's also intermittent. By the time JIRA tries to sync with Blobstore, the following errors appears: {code} 2015-11-07 01:52:19.881416500 2015-11-07 01:52:19,881 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout 2015-11-07 01:52:20.084498500 2015-11-07 01:52:20,084 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.091263500 2015-11-07 01:52:20,091 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout 2015-11-07 01:52:20.234413500 2015-11-07 01:52:20,234 blobstore-client-http:thread-614 WARN sysadmin 111x144165x4 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout 2015-11-07 01:52:20.293592500 2015-11-07 01:52:20,293 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.436838500 2015-11-07 01:52:20,436 blobstore-client-http:thread-614 WARN sysadmin 111x144165x4 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.486762500 2015-11-07 01:52:20,486 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed 2015-11-07 01:52:20.696227500 2015-11-07 01:52:20,696 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed {code} {code} 2015-11-06 12:46:09.768474500 2015-11-06 12:46:09,764 ajp-nio-127.0.0.104-8009-exec-137 WARN sysadmin 764x38599x5 pp5zew 104.192.142.137,104.192.141.18 /rest/jddap/1.0/attachment [c.a.j.issue.thumbnail.DefaultThumbnailManager] Error writing to thumbnail file: /data/jirastudio/jira/home/data/attachments/NEE/10000/NEE-7175/thumbs/_thumb_24718.png7966518938032165601.tmp 2015-11-06 12:46:09.768476500 com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:203) 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:141) 2015-11-06 12:46:09.768484500 at okio.Okio$2.read(Okio.java:137) 2015-11-06 12:46:09.768484500 at okio.AsyncTimeout$2.read(AsyncTimeout.java:217) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285) 2015-11-06 12:46:09.768487500 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) 2015-11-06 12:46:09.768487500 at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.Call.getResponse(Call.java:274) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) 2015-11-06 12:46:09.768497500 at com.squareup.okhttp.Call.execute(Call.java:81) 2015-11-06 12:46:09.768497500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225) 2015-11-06 12:46:09.768498500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source) 2015-11-06 12:46:09.768498500 at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93) 2015-11-06 12:46:09.768500500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364) 2015-11-06 12:46:09.768508500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62) 2015-11-06 12:46:09.768510500 at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2015-11-06 12:46:09.768513500 at java.lang.Thread.run(Thread.java:745) 2015-11-06 12:46:09.768513500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:107) 2015-11-06 12:46:09.768515500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:101) 2015-11-06 12:46:09.768515500 at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216) 2015-11-06 12:46:09.768516500 at com.atlassian.fugue.Either$Left.fold(Either.java:508) 2015-11-06 12:46:09.768516500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$15.apply(BlobStoreAttachmentStore.java:367) 2015-11-06 12:46:09.768519500 at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$15.apply(BlobStoreAttachmentStore.java:363) 2015-11-06 12:46:09.768520500 at com.atlassian.util.concurrent.Promises$Of$1.apply(Promises.java:263) 2015-11-06 12:46:09.768520500 at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:185) 2015-11-06 12:46:09.768521500 at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319) 2015-11-06 12:46:09.768522500 at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) {code} h3. This is the public report of an internal incident.

    Atlassian JIRA | 1 year ago | Andre Borzzatto [Atlassian]
    com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:203) 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:141) 2015-11-06 12:46:09.768484500 at okio.Okio$2.read(Okio.java:137) 2015-11-06 12:46:09.768484500 at okio.AsyncTimeout$2.read(AsyncTimeout.java:217) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285) 2015-11-06 12:46:09.768487500 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) 2015-11-06 12:46:09.768487500 at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.Call.getResponse(Call.java:274) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) 2015-11-06 12:46:09.768497500 at com.squareup.okhttp.Call.execute(Call.java:81) 2015-11-06 12:46:09.768497500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225) 2015-11-06 12:46:09.768498500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source) 2015-11-06 12:46:09.768498500 at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93) 2015-11-06 12:46:09.768500500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364) 2015-11-06 12:46:09.768508500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62) 2015-11-06 12:46:09.768510500 at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  3. 0

    h3. Summary If an exception occurs during the processing of a thumbnail (such as a corrupt jpg), JIRA will not record the failure, and the next time the resource is requested, it will generate it (and fail) again. Typically on those issue that failed due to the condition: {noformat} 2015-09-11 21:24:41.272460500 2015-09-11 21:24:41,267 ajp-nio-127.0.0.104-8009-exec-1063 WARN sysadmin 1283x137820x1 t77wfs 202.68.74.170,104.192.141.33 /browse/DD-108 [c.a.j.issue.thumbnail.DefaultThumbnailManager] Error writing to thumbnail file: /data/jirastudio/jira/home/data/attachments/TEST/10000/TEST-108/thumbs/_thumb_45662.png1391332809470942590.tmp 2015-09-11 21:24:41.272467500 com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 45662: java.awt.image.RasterFormatException: (y + height) is outside of Raster 2015-09-11 21:24:41.272795500 at sun.awt.image.ByteInterleavedRaster.createWritableChild(ByteInterleavedRaster.java:1248) 2015-09-11 21:24:41.272796500 at sun.awt.image.ByteInterleavedRaster.createChild(ByteInterleavedRaster.java:1208) 2015-09-11 21:24:41.272797500 at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.readImageAsRasterAndReplaceColorProfile(Unknown Source) 2015-09-11 21:24:41.272800500 at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.read(Unknown Source) 2015-09-11 21:24:41.272800500 at com.atlassian.core.util.thumbnail.Thumber.scaleImage(Thumber.java:231) 2015-09-11 21:24:41.272800500 at com.atlassian.core.util.thumbnail.Thumber.createThumbnail(Thumber.java:291) 2015-09-11 21:24:41.272812500 at com.atlassian.core.util.thumbnail.Thumber.retrieveOrCreateThumbNail(Thumber.java:270) {noformat} h3. Expected Result This should be tracked even in the event of catastrophic failure (such as OOME taking down the application), and thus the attempt should be recorded and committed before the attempt, not after. When it fails, it should disable the thumbnail from being generated again by marking {{thumbnailable}} on the {{fileattachment}} table to 0 h3. Actual Result It will take a very long time to load the issue, and fail the page load. Multiple reload will still hit into the same issue. Potential piling up the memory if it keeps loaded. h3. Workaround Identify from the error message which image is causing the error, for example in the above error it is image id 45662. Shut down JIRA, take a backup of the database and then run: {noformat} update fileattachment set thumbnailable = 0 where id in (<image ids you identified above>) {noformat}

    Atlassian JIRA | 1 year ago | Harold Murn
    com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 45662: java.awt.image.RasterFormatException: (y + height) is outside of Raster 2015-09-11 21:24:41.272795500 at sun.awt.image.ByteInterleavedRaster.createWritableChild(ByteInterleavedRaster.java:1248) 2015-09-11 21:24:41.272796500 at sun.awt.image.ByteInterleavedRaster.createChild(ByteInterleavedRaster.java:1208) 2015-09-11 21:24:41.272797500 at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.readImageAsRasterAndReplaceColorProfile(Unknown Source)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    h3. Summary If an exception occurs during the processing of a thumbnail (such as a corrupt jpg), JIRA will not record the failure, and the next time the resource is requested, it will generate it (and fail) again. Typically on those issue that failed due to the condition: {noformat} 2015-09-11 21:24:41.272460500 2015-09-11 21:24:41,267 ajp-nio-127.0.0.104-8009-exec-1063 WARN sysadmin 1283x137820x1 t77wfs 202.68.74.170,104.192.141.33 /browse/DD-108 [c.a.j.issue.thumbnail.DefaultThumbnailManager] Error writing to thumbnail file: /data/jirastudio/jira/home/data/attachments/TEST/10000/TEST-108/thumbs/_thumb_45662.png1391332809470942590.tmp 2015-09-11 21:24:41.272467500 com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 45662: java.awt.image.RasterFormatException: (y + height) is outside of Raster 2015-09-11 21:24:41.272795500 at sun.awt.image.ByteInterleavedRaster.createWritableChild(ByteInterleavedRaster.java:1248) 2015-09-11 21:24:41.272796500 at sun.awt.image.ByteInterleavedRaster.createChild(ByteInterleavedRaster.java:1208) 2015-09-11 21:24:41.272797500 at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.readImageAsRasterAndReplaceColorProfile(Unknown Source) 2015-09-11 21:24:41.272800500 at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.read(Unknown Source) 2015-09-11 21:24:41.272800500 at com.atlassian.core.util.thumbnail.Thumber.scaleImage(Thumber.java:231) 2015-09-11 21:24:41.272800500 at com.atlassian.core.util.thumbnail.Thumber.createThumbnail(Thumber.java:291) 2015-09-11 21:24:41.272812500 at com.atlassian.core.util.thumbnail.Thumber.retrieveOrCreateThumbNail(Thumber.java:270) {noformat} h3. Expected Result This should be tracked even in the event of catastrophic failure (such as OOME taking down the application), and thus the attempt should be recorded and committed before the attempt, not after. When it fails, it should disable the thumbnail from being generated again by marking {{thumbnailable}} on the {{fileattachment}} table to 0 h3. Actual Result It will take a very long time to load the issue, and fail the page load. Multiple reload will still hit into the same issue. Potential piling up the memory if it keeps loaded. h3. Workaround Identify from the error message which image is causing the error, for example in the above error it is image id 45662. Shut down JIRA, take a backup of the database and then run: {noformat} update fileattachment set thumbnailable = 0 where id in (<image ids you identified above>) {noformat}

    Atlassian JIRA | 1 year ago | Harold Murn
    com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 45662: java.awt.image.RasterFormatException: (y + height) is outside of Raster 2015-09-11 21:24:41.272795500 at sun.awt.image.ByteInterleavedRaster.createWritableChild(ByteInterleavedRaster.java:1248) 2015-09-11 21:24:41.272796500 at sun.awt.image.ByteInterleavedRaster.createChild(ByteInterleavedRaster.java:1208) 2015-09-11 21:24:41.272797500 at com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReader.readImageAsRasterAndReplaceColorProfile(Unknown Source)

    Root Cause Analysis

    1. com.atlassian.jira.issue.attachment.AttachmentReadException

      Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:203) 2015-11-06 12:46:09.768483500 at java.net.SocketInputStream.read(SocketInputStream.java:141) 2015-11-06 12:46:09.768484500 at okio.Okio$2.read(Okio.java:137) 2015-11-06 12:46:09.768484500 at okio.AsyncTimeout$2.read(AsyncTimeout.java:217) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291) 2015-11-06 12:46:09.768486500 at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285) 2015-11-06 12:46:09.768487500 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) 2015-11-06 12:46:09.768487500 at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80) 2015-11-06 12:46:09.768491500 at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95) 2015-11-06 12:46:09.768492500 at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760) 2015-11-06 12:46:09.768494500 at com.squareup.okhttp.Call.getResponse(Call.java:274) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230) 2015-11-06 12:46:09.768495500 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201) 2015-11-06 12:46:09.768497500 at com.squareup.okhttp.Call.execute(Call.java:81) 2015-11-06 12:46:09.768497500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225) 2015-11-06 12:46:09.768498500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source) 2015-11-06 12:46:09.768498500 at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93) 2015-11-06 12:46:09.768500500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364) 2015-11-06 12:46:09.768508500 at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119) 2015-11-06 12:46:09.768509500 at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62) 2015-11-06 12:46:09.768510500 at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2015-11-06 12:46:09.768512500 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run()
    2. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)
      1 frame
    3. com.atlassian.jira
      BlobStoreAttachmentStore$3.apply
      1. com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:107)
      1 frame