org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character.

JIRA | Dylan Tong | 2 years ago
  1. 0

    I'm working with a prospective customer (Equinix) who encountered an issue with the latest Hadoop connector when operating MongoDB 2.8 with the latest Hive distribution. While the issue appears to be specific to Hive code involving the generation of a malformed job.xml, it appears like the issue is triggered by our connector when data is inserted into the Hive table and written back to MongoDB (the reverse doesn't appear to cause an issue). The issue appears to be similar to the one described in the forums here: https://groups.google.com/forum/#!topic/mongodb-user/lKbha0SzMP8 I have provided repro steps from the customer as well as a detailed error log below. I'm also working on getting connected with our Hortonworks partners to get their assistance; nonetheless, I think this is something we should track as the issue seems to be triggered by basic functionality. Let me know if you like direct contact with the Equinix engineers to discuss the issue. REPRO STEPS as described by client: insert the data onto eqx_eg_accounts_view to test this. CREATE EXTERNAL TABLE accounts_external ( account _id STRING, account_name STRING ) STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES('mongo.columns.mapping'='{"account_id":"account_id", "account_name":"account_name"}') TBLPROPERTIES('mongo.uri'='mongodb://sv2lxgsed01:27017/data-terminal.accounts'); INSERT OVERWRITE TABLE accounts SELECT a.account_id, a.account_name from eqx_eg_accounts_view a where a.account_id = 10000; If we insert the data in the Mongo collection ACCOUNTS and select in Hive, we were able to see the data, but while trying to insert the data onto this External table in Hive as above, it throws error. Error during job, obtaining debugging information... FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask Select account_id, account_name from accounts_external; hive> insert into table individuals values(123,"test1",30); Query ID = hive_20141215235858_01f9d535-307e-4413-ad46-b432b6d3316f Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1418635895251_0095, Tracking URL = http://sv2lxgsed02.corp.equinix.com:8088/proxy/application_1418635895251_0095/ Kill Command = /usr/hdp/2.2.0.0-2041/hadoop/bin/hadoop job -kill job_1418635895251_0095 Hadoop job information for Stage-0: number of mappers: 0; number of reducers: 0 2014-12-15 23:58:48,006 Stage-0 map = 0%, reduce = 0% Ended Job = job_1418635895251_0095 with errors Error during job, obtaining debugging information... FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-0: HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec Error Log: Log Type: stderr Log Upload Time: 15-Dec-2014 23:58:48 Log Length: 325 [Fatal Error] job.xml:832:51: Character reference "&#0" is an invalid XML character. log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Log Type: stdout Log Upload Time: 15-Dec-2014 23:58:48 Log Length: 0 Log Type: syslog Log Upload Time: 15-Dec-2014 23:58:48 Log Length: 2868 2014-12-15 23:58:44,933 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1418635895251_0095_000001 2014-12-15 23:58:45,283 FATAL [main] org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character. at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2354) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2423) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2376) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2283) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1110) at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:51) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1421) 2014-12-15 23:58:45,285 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2519) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2376) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2283) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1110) at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:51) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1421) Caused by: org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character. at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2354) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2423) ... 5 more 2014-12-15 23:58:45,288 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1

    JIRA | 2 years ago | Dylan Tong
    org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character.
  2. 0

    I'm working with a prospective customer (Equinix) who encountered an issue with the latest Hadoop connector when operating MongoDB 2.8 with the latest Hive distribution. While the issue appears to be specific to Hive code involving the generation of a malformed job.xml, it appears like the issue is triggered by our connector when data is inserted into the Hive table and written back to MongoDB (the reverse doesn't appear to cause an issue). The issue appears to be similar to the one described in the forums here: https://groups.google.com/forum/#!topic/mongodb-user/lKbha0SzMP8 I have provided repro steps from the customer as well as a detailed error log below. I'm also working on getting connected with our Hortonworks partners to get their assistance; nonetheless, I think this is something we should track as the issue seems to be triggered by basic functionality. Let me know if you like direct contact with the Equinix engineers to discuss the issue. REPRO STEPS as described by client: insert the data onto eqx_eg_accounts_view to test this. CREATE EXTERNAL TABLE accounts_external ( account _id STRING, account_name STRING ) STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES('mongo.columns.mapping'='{"account_id":"account_id", "account_name":"account_name"}') TBLPROPERTIES('mongo.uri'='mongodb://sv2lxgsed01:27017/data-terminal.accounts'); INSERT OVERWRITE TABLE accounts SELECT a.account_id, a.account_name from eqx_eg_accounts_view a where a.account_id = 10000; If we insert the data in the Mongo collection ACCOUNTS and select in Hive, we were able to see the data, but while trying to insert the data onto this External table in Hive as above, it throws error. Error during job, obtaining debugging information... FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask Select account_id, account_name from accounts_external; hive> insert into table individuals values(123,"test1",30); Query ID = hive_20141215235858_01f9d535-307e-4413-ad46-b432b6d3316f Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1418635895251_0095, Tracking URL = http://sv2lxgsed02.corp.equinix.com:8088/proxy/application_1418635895251_0095/ Kill Command = /usr/hdp/2.2.0.0-2041/hadoop/bin/hadoop job -kill job_1418635895251_0095 Hadoop job information for Stage-0: number of mappers: 0; number of reducers: 0 2014-12-15 23:58:48,006 Stage-0 map = 0%, reduce = 0% Ended Job = job_1418635895251_0095 with errors Error during job, obtaining debugging information... FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-0: HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec Error Log: Log Type: stderr Log Upload Time: 15-Dec-2014 23:58:48 Log Length: 325 [Fatal Error] job.xml:832:51: Character reference "&#0" is an invalid XML character. log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Log Type: stdout Log Upload Time: 15-Dec-2014 23:58:48 Log Length: 0 Log Type: syslog Log Upload Time: 15-Dec-2014 23:58:48 Log Length: 2868 2014-12-15 23:58:44,933 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1418635895251_0095_000001 2014-12-15 23:58:45,283 FATAL [main] org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character. at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2354) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2423) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2376) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2283) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1110) at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:51) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1421) 2014-12-15 23:58:45,285 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2519) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2376) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2283) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1110) at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:51) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1421) Caused by: org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character. at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2354) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2423) ... 5 more 2014-12-15 23:58:45,288 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1

    JIRA | 2 years ago | Dylan Tong
    org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character.
  3. 0

    Elastic search Hive integration issues

    GitHub | 2 years ago | Vinaypandit
    org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///hadoop/yarn/local/usercache/root/appcache/application_1421441970386_0016/container_1421441970386_0016_01_000001/job.xml; lineNumber: 846; columnNumber: 51; Character reference "&#0" is an invalid XML character.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Hive- Elasticsearch Write Operation

    GitHub | 2 years ago | sandesh2014
    org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///hadoop/yarn/local/usercache/hdfs/appcache/application_1427797951492_0011/container_1427797951492_0011_01_000001/job.xml; lineNumber: 843; columnNumber: 51; Character reference "&#0" is an invalid XML character.
  6. 0

    GitHub comment 41#86869182

    GitHub | 2 years ago | ManikandanV
    org.apache.hadoop.conf.Configuration: error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1427431806640_0012/container_1427431806640_0012_01_000001/job.xml; lineNumber: 665; columnNumber: 51; Character reference "&#0" is an invalid XML character.

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.conf.Configuration

      error parsing conf job.xml org.xml.sax.SAXParseException; systemId: file:///opt/hadoop/yarn/local/usercache/hive/appcache/application_1418635895251_0095/container_1418635895251_0095_01_000001/job.xml; lineNumber: 832; columnNumber: 51; Character reference "&#0" is an invalid XML character.

      at org.apache.xerces.parsers.DOMParser.parse()
    2. Xerces2-j
      DocumentBuilderImpl.parse
      1. org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
      2. org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
      2 frames
    3. Java RT
      DocumentBuilder.parse
      1. javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
      1 frame
    4. Hadoop
      Configuration.get
      1. org.apache.hadoop.conf.Configuration.parse(Configuration.java:2354)
      2. org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2423)
      3. org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2376)
      4. org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2283)
      5. org.apache.hadoop.conf.Configuration.get(Configuration.java:1110)
      5 frames
    5. hadoop-mapreduce-client-common
      MRWebAppUtil.initialize
      1. org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:51)
      1 frame
    6. hadoop-mapreduce-client-app
      MRAppMaster.main
      1. org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1421)
      1 frame