org.apache.http.client.CircularRedirectException: Circular redirect to 'http://.../robots.txt/'

opensearchserve | a90a | 2 years ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Hanged Crawl

    opensearchserve | 2 years ago | a90a
    org.apache.http.client.CircularRedirectException: Circular redirect to 'http://.../robots.txt/'

    Root Cause Analysis

    1. org.apache.http.client.CircularRedirectException

      Circular redirect to 'http://.../robots.txt/'

      at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI()
    2. Apache HttpClient
      DefaultRedirectStrategy.getRedirect
      1. org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:174)
      2. org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:217)
      2 frames
    3. com.jaeksoft.searchlib
      ThreadAbstract.run
      1. com.jaeksoft.searchlib.crawler.web.spider.HttpAbstract.getRedirectLocation(HttpAbstract.java:222)
      2. com.jaeksoft.searchlib.crawler.web.spider.HttpDownloader.getDownloadItem(HttpDownloader.java:94)
      3. com.jaeksoft.searchlib.crawler.web.spider.HttpDownloader.request(HttpDownloader.java:123)
      4. com.jaeksoft.searchlib.crawler.web.spider.HttpDownloader.request(HttpDownloader.java:159)
      5. com.jaeksoft.searchlib.crawler.web.spider.HttpDownloader.get(HttpDownloader.java:175)
      6. com.jaeksoft.searchlib.crawler.web.spider.Crawl.download(Crawl.java:283)
      7. com.jaeksoft.searchlib.crawler.web.robotstxt.RobotsTxtCache.getRobotsTxt(RobotsTxtCache.java:107)
      8. com.jaeksoft.searchlib.crawler.web.spider.Crawl.checkRobotTxtAllow(Crawl.java:236)
      9. com.jaeksoft.searchlib.crawler.web.process.WebCrawlThread.crawl(WebCrawlThread.java:181)
      10. com.jaeksoft.searchlib.crawler.web.process.WebCrawlThread.runner(WebCrawlThread.java:126)
      11. com.jaeksoft.searchlib.process.ThreadAbstract.run(ThreadAbstract.java:291)
      11 frames
    4. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      3. java.lang.Thread.run(Unknown Source)
      3 frames