org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro

Pentaho BI Platform Tracking | Rohit K | 2 years ago
  1. 0

    I am configuring the avro to be read by PDI 5.2. The PDI is able to browse the file residing on the cluster.However any attempt to preview or to transfer it to any other location resulting the same error message Done with shims settings http://wiki.pentaho.com/display/BAD/...for+YARN+Shims and http://funpdi.blogspot.in/2013/03/pe...nd-hadoop.html Followed both aforementioned links The error message is: org.pentaho.di.core.exception.KettleFileException: Exception reading line: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro at org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:157) at org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:95) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.getCSV(HadoopFileInputDialog.java:2563) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.get(HadoopFileInputDialog.java:2507) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.access$300(HadoopFileInputDialog.java:117) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog$5.handleEvent(HadoopFileInputDialog.java:483) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.open(HadoopFileInputDialog.java:680) at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:124) at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8720) at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3027) at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:744) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1310) at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7931) at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9202) at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:648) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:878) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:559) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:789) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at java.io.DataInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at org.apache.commons.vfs.util.MonitorInputStream.read(Unknown Source) at org.pentaho.di.core.compress.CompressionInputStream.read(CompressionInputStream.java:36) at java.io.InputStream.read(Unknown Source) at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) at sun.nio.cs.StreamDecoder.implRead(Unknown Source) at sun.nio.cs.StreamDecoder.read(Unknown Source) at sun.nio.cs.StreamDecoder.read0(Unknown Source) at sun.nio.cs.StreamDecoder.read(Unknown Source) at java.io.InputStreamReader.read(Unknown Source) at org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:106) ... 28 more Please suggest that what is remaining in order to read/preview/write data to hadoop cluster? Thanks Rohit

    Pentaho BI Platform Tracking | 2 years ago | Rohit K
    org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro
  2. 0

    I am configuring the avro to be read by PDI 5.2. The PDI is able to browse the file residing on the cluster.However any attempt to preview or to transfer it to any other location resulting the same error message Done with shims settings http://wiki.pentaho.com/display/BAD/...for+YARN+Shims and http://funpdi.blogspot.in/2013/03/pe...nd-hadoop.html Followed both aforementioned links The error message is: org.pentaho.di.core.exception.KettleFileException: Exception reading line: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro at org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:157) at org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:95) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.getCSV(HadoopFileInputDialog.java:2563) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.get(HadoopFileInputDialog.java:2507) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.access$300(HadoopFileInputDialog.java:117) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog$5.handleEvent(HadoopFileInputDialog.java:483) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.open(HadoopFileInputDialog.java:680) at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:124) at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8720) at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3027) at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:744) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1310) at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7931) at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9202) at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:648) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:878) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:559) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:789) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at java.io.DataInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at org.apache.commons.vfs.util.MonitorInputStream.read(Unknown Source) at org.pentaho.di.core.compress.CompressionInputStream.read(CompressionInputStream.java:36) at java.io.InputStream.read(Unknown Source) at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) at sun.nio.cs.StreamDecoder.implRead(Unknown Source) at sun.nio.cs.StreamDecoder.read(Unknown Source) at sun.nio.cs.StreamDecoder.read0(Unknown Source) at sun.nio.cs.StreamDecoder.read(Unknown Source) at java.io.InputStreamReader.read(Unknown Source) at org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:106) ... 28 more Please suggest that what is remaining in order to read/preview/write data to hadoop cluster? Thanks Rohit

    Pentaho BI Platform Tracking | 2 years ago | Rohit K
    FiveNOAA_00000.avro
  3. 0

    ConnectTimeoutException connecting to Hortonworks VM in Eclipse for MapReduce job?

    Stack Overflow | 2 years ago | Peter Andrus
    org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 file=/user/hue/smallClaimData.txt
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    BlockMissingException when reading file from HDFS

    Stack Overflow | 2 years ago | Kshitiz Sharma
    org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1205966836-127.0.0.1-1406828172945:blk_1073742119_1304 file=/notice.html
  6. 0

    Missing block in Hadoop

    Stack Overflow | 2 years ago | user3033194
    org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-971868671-192.168.50.2-1406571670535:blk_1073743276_2475 file=/wikidumps/enwiki-20130904-pages-meta-history3.xml-p000032706p000037161

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.hdfs.BlockMissingException

      Could not obtain block: BP-647747408-172.31.42.71-1418062088466:blk_1073741892_1068 file=/dragonflytest/FiveNOAA_00000.avro

      at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode()
    2. Apache Hadoop HDFS
      DFSInputStream.read
      1. org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:878)
      2. org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:559)
      3. org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:789)
      4. org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836)
      4 frames
    3. Java RT
      BufferedInputStream.read
      1. java.io.DataInputStream.read(Unknown Source)
      2. java.io.BufferedInputStream.fill(Unknown Source)
      3. java.io.BufferedInputStream.read(Unknown Source)
      3 frames
    4. Commons VFS
      MonitorInputStream.read
      1. org.apache.commons.vfs.util.MonitorInputStream.read(Unknown Source)
      1 frame
    5. org.pentaho.di
      CompressionInputStream.read
      1. org.pentaho.di.core.compress.CompressionInputStream.read(CompressionInputStream.java:36)
      1 frame
    6. Java RT
      InputStreamReader.read
      1. java.io.InputStream.read(Unknown Source)
      2. sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
      3. sun.nio.cs.StreamDecoder.implRead(Unknown Source)
      4. sun.nio.cs.StreamDecoder.read(Unknown Source)
      5. sun.nio.cs.StreamDecoder.read0(Unknown Source)
      6. sun.nio.cs.StreamDecoder.read(Unknown Source)
      7. java.io.InputStreamReader.read(Unknown Source)
      7 frames
    7. org.pentaho.di
      HadoopFileInputDialog$5.handleEvent
      1. org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:106)
      2. org.pentaho.di.trans.steps.textfileinput.TextFileInput.getLine(TextFileInput.java:95)
      3. org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.getCSV(HadoopFileInputDialog.java:2563)
      4. org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.get(HadoopFileInputDialog.java:2507)
      5. org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.access$300(HadoopFileInputDialog.java:117)
      6. org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog$5.handleEvent(HadoopFileInputDialog.java:483)
      6 frames
    8. RWT
      Display.readAndDispatch
      1. org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
      2. org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
      3. org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
      4. org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
      4 frames
    9. org.pentaho.di
      TransGraph.mouseDoubleClick
      1. org.pentaho.di.ui.trans.steps.hadoopfileinput.HadoopFileInputDialog.open(HadoopFileInputDialog.java:680)
      2. org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:124)
      3. org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8720)
      4. org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3027)
      5. org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:744)
      5 frames
    10. RWT
      Display.readAndDispatch
      1. org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
      2. org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
      3. org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
      4. org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
      5. org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
      5 frames
    11. org.pentaho.di
      Spoon.main
      1. org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1310)
      2. org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7931)
      3. org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9202)
      4. org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:648)
      4 frames
    12. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      4. java.lang.reflect.Method.invoke(Unknown Source)
      4 frames
    13. org.pentaho.commons
      Launcher.main
      1. org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
      1 frame