org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44

JIRA | Cetin Sert | 10 years ago
  1. 0

    While running heritrix 1.13.1 rev 51?? through 5176 on Solaris 10 x64 with (I think I have seen the same issue on a Debian x86 machine, but I will have to check first.) ---------------------- java version "1.6.0" Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode) ---------------------- using a duplication-reductive* modules configuration as in (additionally: arc-writer skip-identical-digests: true) ---------------------- Select Fetchers Processors that fetch documents using various protocols org.archive.crawler.processor.recrawl.PersistLoadProcessor* Down Remove Info org.archive.crawler.fetcher.FetchDNS Up Down Remove Info org.archive.crawler.fetcher.FetchHTTP Up Down Remove Info org.archive.crawler.processor.recrawl.FetchHistoryProcessor* Up Remove Info Select Extractors Processors that extracts links from URIs org.archive.crawler.extractor.ExtractorHTTP Down Remove Info org.archive.crawler.extractor.ExtractorHTML Up Down Remove Info org.archive.crawler.extractor.ExtractorCSS Up Down Remove Info org.archive.crawler.extractor.ExtractorJS Up Down Remove Info org.archive.crawler.extractor.ExtractorSWF Up Remove Info Select Writers Processors that write documents to archive files org.archive.crawler.writer.ARCWriterProcessor Remove Info Select Post Processors Processors that do cleanup and feed the Frontier with new URIs org.archive.crawler.postprocessor.CrawlStateUpdater Down Remove Info org.archive.crawler.postprocessor.LinksScoper Up Down Remove Info org.archive.crawler.postprocessor.FrontierScheduler Up Down Remove Info org.archive.crawler.processor.recrawl.PersistStoreProcessor* Up Remove Info ---------------------- and this custom decide rule ---------------------- /* GeoLocationMatchesRegExpDecideRule * * $Id: GeoLocationMatchesRegExpDecideRule.java 4649 2006-09-25 17:16:55Z sert@lexik.ids-mannheim.de $ * * Created on Mar 7, 2007 * Switched to TextUtils on May 18, 2007 * * Copyright (C) 2007 IDS */ package org.archive.crawler.deciderules; import java.util.logging.Logger; import java.util.logging.Level; import java.util.regex.*; import javax.management.AttributeNotFoundException; import org.archive.crawler.settings.SimpleType; import org.archive.util.TextUtils; import net.sf.javainetlocator.*; /** * Rule applies configured decision to any CrawlURIs whose geographic location * matches the supplied regexp. * * @author sert * @version $Date: 2007-03-08 11:57:00 +0200 (Do, 18 May 2007) $, $Revision: 0006 $ */ public class GeoLocationMatchesRegExpDecideRule extends PredicatedDecideRule { private static final long serialVersionUID = 6441410666674319295L; private static final Logger logger = Logger.getLogger(GeoLocationMatchesRegExpDecideRule.class.getName()); private static final String hostnamePattern = "(?<=\\w+://).+?(?=(/|$))"; public static final String ATTR_REGEXP = "regexp"; /** * Usual constructor. * @param name */ public GeoLocationMatchesRegExpDecideRule(String name) { super(name); setDescription("GeoLocationMatchesRegExpDecideRule. Applies the configured " + "decision to URIs whose geographic location country names match the supplied regular expression. " + "For example: DE|AT|CH - " + "sert@lexik.ids-mannheim.de - revision 0006"); addElementToDefinition(new SimpleType(ATTR_REGEXP, "Java regular" + "expression to match.", "")); } /** * Evaluate whether given object's string version * matches configured regexp * * @param object * @return true if regexp is matched */ protected boolean evaluate(Object object) { try { String countryCodePattern = getRegexp(object); String url = object.toString(); String countryCodeMatch = getGeoLocation(url); boolean result = (countryCodePattern == null || countryCodeMatch == null)? false: TextUtils.matches(countryCodePattern, countryCodeMatch); if (logger.isLoggable(Level.FINE)) { logger.fine("Tested geolocation '" + countryCodeMatch + "' match with regex '" + countryCodePattern + " and result was " + result); } return result; } catch (ClassCastException e) { // if not CrawlURI, always disregard return false; } } private String getGeoLocation(String url) { Matcher m = TextUtils.getMatcher(hostnamePattern, url); if (m.find()) { try { return InetAddressLocator.getLocale(m.group()).getCountry(); } catch (Exception e) { } } TextUtils.recycleMatcher(m); return null; } /** * Get the regular expression string to match the URI against. * * @param o the object for which the regular expression should be * matched against. * @return the regular expression to match against. */ protected String getRegexp(Object o) { try { return (String) getAttribute(o, ATTR_REGEXP); } catch (AttributeNotFoundException e) { logger.severe(e.getMessage()); return null; // Basically the filter is inactive if this occurs. } } } ---------------------- results in the following exception after terminating the first job of a fresh heritrix instance and upon starting a second job based on the first one. ---------------------- Time: Mai. 18, 2007 07:37:58 GMT Level: SEVERE Message: On crawl: default-b-2 Unable to setup crawl modules Exception: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 Cause: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Stacktrace: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:324) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) ... 7 more ---------------------- Shutting the instance down and restarting a fresh one and repeating the same task of creating a new job based on the last successfully crawled job throws no exceptions and proceeds as expected. What may be going wrong here? Thanks in advance for your reply. Best Regards, Cetin Sert Institut für Deutsche Sprache Mannheim, Germany

    JIRA | 10 years ago | Cetin Sert
    org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44
  2. 0

    While running heritrix 1.13.1 rev 51?? through 5176 on Solaris 10 x64 with (I think I have seen the same issue on a Debian x86 machine, but I will have to check first.) ---------------------- java version "1.6.0" Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode) ---------------------- using a duplication-reductive* modules configuration as in (additionally: arc-writer skip-identical-digests: true) ---------------------- Select Fetchers Processors that fetch documents using various protocols org.archive.crawler.processor.recrawl.PersistLoadProcessor* Down Remove Info org.archive.crawler.fetcher.FetchDNS Up Down Remove Info org.archive.crawler.fetcher.FetchHTTP Up Down Remove Info org.archive.crawler.processor.recrawl.FetchHistoryProcessor* Up Remove Info Select Extractors Processors that extracts links from URIs org.archive.crawler.extractor.ExtractorHTTP Down Remove Info org.archive.crawler.extractor.ExtractorHTML Up Down Remove Info org.archive.crawler.extractor.ExtractorCSS Up Down Remove Info org.archive.crawler.extractor.ExtractorJS Up Down Remove Info org.archive.crawler.extractor.ExtractorSWF Up Remove Info Select Writers Processors that write documents to archive files org.archive.crawler.writer.ARCWriterProcessor Remove Info Select Post Processors Processors that do cleanup and feed the Frontier with new URIs org.archive.crawler.postprocessor.CrawlStateUpdater Down Remove Info org.archive.crawler.postprocessor.LinksScoper Up Down Remove Info org.archive.crawler.postprocessor.FrontierScheduler Up Down Remove Info org.archive.crawler.processor.recrawl.PersistStoreProcessor* Up Remove Info ---------------------- and this custom decide rule ---------------------- /* GeoLocationMatchesRegExpDecideRule * * $Id: GeoLocationMatchesRegExpDecideRule.java 4649 2006-09-25 17:16:55Z sert@lexik.ids-mannheim.de $ * * Created on Mar 7, 2007 * Switched to TextUtils on May 18, 2007 * * Copyright (C) 2007 IDS */ package org.archive.crawler.deciderules; import java.util.logging.Logger; import java.util.logging.Level; import java.util.regex.*; import javax.management.AttributeNotFoundException; import org.archive.crawler.settings.SimpleType; import org.archive.util.TextUtils; import net.sf.javainetlocator.*; /** * Rule applies configured decision to any CrawlURIs whose geographic location * matches the supplied regexp. * * @author sert * @version $Date: 2007-03-08 11:57:00 +0200 (Do, 18 May 2007) $, $Revision: 0006 $ */ public class GeoLocationMatchesRegExpDecideRule extends PredicatedDecideRule { private static final long serialVersionUID = 6441410666674319295L; private static final Logger logger = Logger.getLogger(GeoLocationMatchesRegExpDecideRule.class.getName()); private static final String hostnamePattern = "(?<=\\w+://).+?(?=(/|$))"; public static final String ATTR_REGEXP = "regexp"; /** * Usual constructor. * @param name */ public GeoLocationMatchesRegExpDecideRule(String name) { super(name); setDescription("GeoLocationMatchesRegExpDecideRule. Applies the configured " + "decision to URIs whose geographic location country names match the supplied regular expression. " + "For example: DE|AT|CH - " + "sert@lexik.ids-mannheim.de - revision 0006"); addElementToDefinition(new SimpleType(ATTR_REGEXP, "Java regular" + "expression to match.", "")); } /** * Evaluate whether given object's string version * matches configured regexp * * @param object * @return true if regexp is matched */ protected boolean evaluate(Object object) { try { String countryCodePattern = getRegexp(object); String url = object.toString(); String countryCodeMatch = getGeoLocation(url); boolean result = (countryCodePattern == null || countryCodeMatch == null)? false: TextUtils.matches(countryCodePattern, countryCodeMatch); if (logger.isLoggable(Level.FINE)) { logger.fine("Tested geolocation '" + countryCodeMatch + "' match with regex '" + countryCodePattern + " and result was " + result); } return result; } catch (ClassCastException e) { // if not CrawlURI, always disregard return false; } } private String getGeoLocation(String url) { Matcher m = TextUtils.getMatcher(hostnamePattern, url); if (m.find()) { try { return InetAddressLocator.getLocale(m.group()).getCountry(); } catch (Exception e) { } } TextUtils.recycleMatcher(m); return null; } /** * Get the regular expression string to match the URI against. * * @param o the object for which the regular expression should be * matched against. * @return the regular expression to match against. */ protected String getRegexp(Object o) { try { return (String) getAttribute(o, ATTR_REGEXP); } catch (AttributeNotFoundException e) { logger.severe(e.getMessage()); return null; // Basically the filter is inactive if this occurs. } } } ---------------------- results in the following exception after terminating the first job of a fresh heritrix instance and upon starting a second job based on the first one. ---------------------- Time: Mai. 18, 2007 07:37:58 GMT Level: SEVERE Message: On crawl: default-b-2 Unable to setup crawl modules Exception: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 Cause: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Stacktrace: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:324) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) ... 7 more ---------------------- Shutting the instance down and restarting a fresh one and repeating the same task of creating a new job based on the last successfully crawled job throws no exceptions and proceeds as expected. What may be going wrong here? Thanks in advance for your reply. Best Regards, Cetin Sert Institut für Deutsche Sprache Mannheim, Germany

    JIRA | 10 years ago | Cetin Sert
    org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44
  3. 0

    While running heritrix 1.13.1 rev 51?? through 5176 on Solaris 10 x64 with (I think I have seen the same issue on a Debian x86 machine, but I will have to check first.) ---------------------- java version "1.6.0" Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode) ---------------------- using a duplication-reductive* modules configuration as in (additionally: arc-writer skip-identical-digests: true) ---------------------- Select Fetchers Processors that fetch documents using various protocols org.archive.crawler.processor.recrawl.PersistLoadProcessor* Down Remove Info org.archive.crawler.fetcher.FetchDNS Up Down Remove Info org.archive.crawler.fetcher.FetchHTTP Up Down Remove Info org.archive.crawler.processor.recrawl.FetchHistoryProcessor* Up Remove Info Select Extractors Processors that extracts links from URIs org.archive.crawler.extractor.ExtractorHTTP Down Remove Info org.archive.crawler.extractor.ExtractorHTML Up Down Remove Info org.archive.crawler.extractor.ExtractorCSS Up Down Remove Info org.archive.crawler.extractor.ExtractorJS Up Down Remove Info org.archive.crawler.extractor.ExtractorSWF Up Remove Info Select Writers Processors that write documents to archive files org.archive.crawler.writer.ARCWriterProcessor Remove Info Select Post Processors Processors that do cleanup and feed the Frontier with new URIs org.archive.crawler.postprocessor.CrawlStateUpdater Down Remove Info org.archive.crawler.postprocessor.LinksScoper Up Down Remove Info org.archive.crawler.postprocessor.FrontierScheduler Up Down Remove Info org.archive.crawler.processor.recrawl.PersistStoreProcessor* Up Remove Info ---------------------- and this custom decide rule ---------------------- /* GeoLocationMatchesRegExpDecideRule * * $Id: GeoLocationMatchesRegExpDecideRule.java 4649 2006-09-25 17:16:55Z sert@lexik.ids-mannheim.de $ * * Created on Mar 7, 2007 * Switched to TextUtils on May 18, 2007 * * Copyright (C) 2007 IDS */ package org.archive.crawler.deciderules; import java.util.logging.Logger; import java.util.logging.Level; import java.util.regex.*; import javax.management.AttributeNotFoundException; import org.archive.crawler.settings.SimpleType; import org.archive.util.TextUtils; import net.sf.javainetlocator.*; /** * Rule applies configured decision to any CrawlURIs whose geographic location * matches the supplied regexp. * * @author sert * @version $Date: 2007-03-08 11:57:00 +0200 (Do, 18 May 2007) $, $Revision: 0006 $ */ public class GeoLocationMatchesRegExpDecideRule extends PredicatedDecideRule { private static final long serialVersionUID = 6441410666674319295L; private static final Logger logger = Logger.getLogger(GeoLocationMatchesRegExpDecideRule.class.getName()); private static final String hostnamePattern = "(?<=\\w+://).+?(?=(/|$))"; public static final String ATTR_REGEXP = "regexp"; /** * Usual constructor. * @param name */ public GeoLocationMatchesRegExpDecideRule(String name) { super(name); setDescription("GeoLocationMatchesRegExpDecideRule. Applies the configured " + "decision to URIs whose geographic location country names match the supplied regular expression. " + "For example: DE|AT|CH - " + "sert@lexik.ids-mannheim.de - revision 0006"); addElementToDefinition(new SimpleType(ATTR_REGEXP, "Java regular" + "expression to match.", "")); } /** * Evaluate whether given object's string version * matches configured regexp * * @param object * @return true if regexp is matched */ protected boolean evaluate(Object object) { try { String countryCodePattern = getRegexp(object); String url = object.toString(); String countryCodeMatch = getGeoLocation(url); boolean result = (countryCodePattern == null || countryCodeMatch == null)? false: TextUtils.matches(countryCodePattern, countryCodeMatch); if (logger.isLoggable(Level.FINE)) { logger.fine("Tested geolocation '" + countryCodeMatch + "' match with regex '" + countryCodePattern + " and result was " + result); } return result; } catch (ClassCastException e) { // if not CrawlURI, always disregard return false; } } private String getGeoLocation(String url) { Matcher m = TextUtils.getMatcher(hostnamePattern, url); if (m.find()) { try { return InetAddressLocator.getLocale(m.group()).getCountry(); } catch (Exception e) { } } TextUtils.recycleMatcher(m); return null; } /** * Get the regular expression string to match the URI against. * * @param o the object for which the regular expression should be * matched against. * @return the regular expression to match against. */ protected String getRegexp(Object o) { try { return (String) getAttribute(o, ATTR_REGEXP); } catch (AttributeNotFoundException e) { logger.severe(e.getMessage()); return null; // Basically the filter is inactive if this occurs. } } } ---------------------- results in the following exception after terminating the first job of a fresh heritrix instance and upon starting a second job based on the first one. ---------------------- Time: Mai. 18, 2007 07:37:58 GMT Level: SEVERE Message: On crawl: default-b-2 Unable to setup crawl modules Exception: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 Cause: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Stacktrace: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:324) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) ... 7 more ---------------------- Shutting the instance down and restarting a fresh one and repeating the same task of creating a new job based on the last successfully crawled job throws no exceptions and proceeds as expected. What may be going wrong here? Thanks in advance for your reply. Best Regards, Cetin Sert Institut für Deutsche Sprache Mannheim, Germany

    JIRA | 10 years ago | Cetin Sert
    java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    While running heritrix 1.13.1 rev 51?? through 5176 on Solaris 10 x64 with (I think I have seen the same issue on a Debian x86 machine, but I will have to check first.) ---------------------- java version "1.6.0" Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode) ---------------------- using a duplication-reductive* modules configuration as in (additionally: arc-writer skip-identical-digests: true) ---------------------- Select Fetchers Processors that fetch documents using various protocols org.archive.crawler.processor.recrawl.PersistLoadProcessor* Down Remove Info org.archive.crawler.fetcher.FetchDNS Up Down Remove Info org.archive.crawler.fetcher.FetchHTTP Up Down Remove Info org.archive.crawler.processor.recrawl.FetchHistoryProcessor* Up Remove Info Select Extractors Processors that extracts links from URIs org.archive.crawler.extractor.ExtractorHTTP Down Remove Info org.archive.crawler.extractor.ExtractorHTML Up Down Remove Info org.archive.crawler.extractor.ExtractorCSS Up Down Remove Info org.archive.crawler.extractor.ExtractorJS Up Down Remove Info org.archive.crawler.extractor.ExtractorSWF Up Remove Info Select Writers Processors that write documents to archive files org.archive.crawler.writer.ARCWriterProcessor Remove Info Select Post Processors Processors that do cleanup and feed the Frontier with new URIs org.archive.crawler.postprocessor.CrawlStateUpdater Down Remove Info org.archive.crawler.postprocessor.LinksScoper Up Down Remove Info org.archive.crawler.postprocessor.FrontierScheduler Up Down Remove Info org.archive.crawler.processor.recrawl.PersistStoreProcessor* Up Remove Info ---------------------- and this custom decide rule ---------------------- /* GeoLocationMatchesRegExpDecideRule * * $Id: GeoLocationMatchesRegExpDecideRule.java 4649 2006-09-25 17:16:55Z sert@lexik.ids-mannheim.de $ * * Created on Mar 7, 2007 * Switched to TextUtils on May 18, 2007 * * Copyright (C) 2007 IDS */ package org.archive.crawler.deciderules; import java.util.logging.Logger; import java.util.logging.Level; import java.util.regex.*; import javax.management.AttributeNotFoundException; import org.archive.crawler.settings.SimpleType; import org.archive.util.TextUtils; import net.sf.javainetlocator.*; /** * Rule applies configured decision to any CrawlURIs whose geographic location * matches the supplied regexp. * * @author sert * @version $Date: 2007-03-08 11:57:00 +0200 (Do, 18 May 2007) $, $Revision: 0006 $ */ public class GeoLocationMatchesRegExpDecideRule extends PredicatedDecideRule { private static final long serialVersionUID = 6441410666674319295L; private static final Logger logger = Logger.getLogger(GeoLocationMatchesRegExpDecideRule.class.getName()); private static final String hostnamePattern = "(?<=\\w+://).+?(?=(/|$))"; public static final String ATTR_REGEXP = "regexp"; /** * Usual constructor. * @param name */ public GeoLocationMatchesRegExpDecideRule(String name) { super(name); setDescription("GeoLocationMatchesRegExpDecideRule. Applies the configured " + "decision to URIs whose geographic location country names match the supplied regular expression. " + "For example: DE|AT|CH - " + "sert@lexik.ids-mannheim.de - revision 0006"); addElementToDefinition(new SimpleType(ATTR_REGEXP, "Java regular" + "expression to match.", "")); } /** * Evaluate whether given object's string version * matches configured regexp * * @param object * @return true if regexp is matched */ protected boolean evaluate(Object object) { try { String countryCodePattern = getRegexp(object); String url = object.toString(); String countryCodeMatch = getGeoLocation(url); boolean result = (countryCodePattern == null || countryCodeMatch == null)? false: TextUtils.matches(countryCodePattern, countryCodeMatch); if (logger.isLoggable(Level.FINE)) { logger.fine("Tested geolocation '" + countryCodeMatch + "' match with regex '" + countryCodePattern + " and result was " + result); } return result; } catch (ClassCastException e) { // if not CrawlURI, always disregard return false; } } private String getGeoLocation(String url) { Matcher m = TextUtils.getMatcher(hostnamePattern, url); if (m.find()) { try { return InetAddressLocator.getLocale(m.group()).getCountry(); } catch (Exception e) { } } TextUtils.recycleMatcher(m); return null; } /** * Get the regular expression string to match the URI against. * * @param o the object for which the regular expression should be * matched against. * @return the regular expression to match against. */ protected String getRegexp(Object o) { try { return (String) getAttribute(o, ATTR_REGEXP); } catch (AttributeNotFoundException e) { logger.severe(e.getMessage()); return null; // Basically the filter is inactive if this occurs. } } } ---------------------- results in the following exception after terminating the first job of a fresh heritrix instance and upon starting a second job based on the first one. ---------------------- Time: Mai. 18, 2007 07:37:58 GMT Level: SEVERE Message: On crawl: default-b-2 Unable to setup crawl modules Exception: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 Cause: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Stacktrace: org.archive.crawler.framework.exceptions.FatalConfigurationException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:324) at org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282) at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656) at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377) at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846) at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142) at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44 at org.archive.crawler.util.BdbUriUniqFilter.(BdbUriUniqFilter.java:102) at org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154) at org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320) ... 7 more ---------------------- Shutting the instance down and restarting a fresh one and repeating the same task of creating a new job based on the last successfully crawled job throws no exceptions and proceeds as expected. What may be going wrong here? Thanks in advance for your reply. Best Regards, Cetin Sert Institut für Deutsche Sprache Mannheim, Germany

    JIRA | 10 years ago | Cetin Sert
    java.io.IOException: (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44
  6. 0

    java.io.IOException: Failed to obtain node lock

    GitHub | 5 years ago | splix
    java.io.IOException: Failed to obtain node lock

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.io.IOException

      (JE 3.2.23) Lock expired. Locker -1_StartNextJob_BasicLocker: waited for lock on database=_jeNameMap node=44 type=WRITE grant=WAIT_PROMOTION timeoutMillis=5000 startTime=1179473873000 endTime=1179473878005 Owners: [, ] Waiters: [] Transaction -1_StartNextJob_BasicLocker owns 44 Transaction -1_StartNextJob_BasicLocker waits for node 44

      at org.archive.crawler.util.BdbUriUniqFilter.<init>()
    2. org.archive.crawler
      CrawlJobHandler$3.run
      1. org.archive.crawler.util.BdbUriUniqFilter.<init>(BdbUriUniqFilter.java:102)
      2. org.archive.crawler.frontier.BdbFrontier.createAlreadyIncluded(BdbFrontier.java:154)
      3. org.archive.crawler.frontier.WorkQueueFrontier.initialize(WorkQueueFrontier.java:320)
      4. org.archive.crawler.frontier.BdbFrontier.initialize(BdbFrontier.java:282)
      5. org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:656)
      6. org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:377)
      7. org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:846)
      8. org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1142)
      9. org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1125)
      9 frames
    3. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:619)
      1 frame