Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.frontier
Class RecoveryJournal

    java.lang.Object
      org.archive.crawler.io.CrawlerJournal
          org.archive.crawler.frontier.RecoveryJournal

All Implemented Interfaces:
  FrontierJournal

    public class RecoveryJournal

extends CrawlerJournal
implements FrontierJournal

Helper class for managing a simple Frontier change-events journal which is useful for recovering from crawl problems. By replaying the journal into a new Frontier, its state (at least with respect to URIs alreadyIncluded and in pending queues) will match that of the original Frontier, allowing a pseudo-resume of a previous crawl, at least as far as URI visitation/coverage is concerned.

Author:
  gojomo

Field Summary
`static java.lang.String` | `F_ADD`
`static java.lang.String` | `F_DISREGARD`
`static java.lang.String` | `F_EMIT`
`static java.lang.String` | `F_FAILURE`
`static java.lang.String` | `F_RESCHEDULE`
`static java.lang.String` | `F_SUCCESS`

Fields inherited from class org.archive.crawler.io.CrawlerJournal
`accumulatingBuffer, GZIP_SUFFIX, gzipFile, lines, LOG_ERROR, LOG_TIMESTAMP, out, timestamp_interval`

Fields inherited from interface org.archive.crawler.frontier.FrontierJournal
`LOGNAME_RECOVER`

Constructor Summary
`RecoveryJournal(java.lang.String path, java.lang.String filename)` Create a new recovery journal at the given location

Method Summary
`void` | `added(CrawlURI curi)`
`void` | `emitted(CrawlURI curi)` Note that a CrawlURI was emitted for processing.
`void` | `finishedDisregard(CrawlURI curi)`
`void` | `finishedFailure(CrawlURI curi)`
`void` | `finishedFailure(java.lang.String u)`
`void` | `finishedFailure(UURI uuri)`
`void` | `finishedSuccess(CrawlURI curi)`
`protected void` | `finishedSuccess(java.lang.String uuri)`
`void` | `finishedSuccess(UURI uuri)`
`static void` | `importRecoverLog(java.io.File source, CrawlController controller, boolean retainFailures)` Utility method for scanning a recovery journal and applying it to a Frontier.
`void` | `rescheduled(CrawlURI curi)`

Methods inherited from class org.archive.crawler.io.CrawlerJournal
`checkpoint, close, considerTimestamp, getBufferedInput, getBufferedReader, initialize, noteLine, seriousError, writeLine, writeLine, writeLine, writeLine`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Methods inherited from interface org.archive.crawler.frontier.FrontierJournal
`checkpoint, close, seriousError`

Field Detail

### F_ADD

    public static final java.lang.String F_ADD

See Also:
  Constant Field Values

### F_EMIT

    public static final java.lang.String F_EMIT

See Also:
  Constant Field Values

### F_DISREGARD

    public static final java.lang.String F_DISREGARD

See Also:
  Constant Field Values

### F_RESCHEDULE

    public static final java.lang.String F_RESCHEDULE

See Also:
  Constant Field Values

### F_SUCCESS

    public static final java.lang.String F_SUCCESS

See Also:
  Constant Field Values

### F_FAILURE

    public static final java.lang.String F_FAILURE

See Also:
  Constant Field Values

Constructor Detail

### RecoveryJournal

    public RecoveryJournal(java.lang.String path,
                           java.lang.String filename)
                    throws java.io.IOException

  Create a new recovery journal at the given location

  Parameters:
    `path` - Directory to make the recovery journal in.
    `filename` - Name to use for recovery journal file.
  Throws:
    `java.io.IOException`

Method Detail

### added

    public void added(CrawlURI curi)

  Specified by:
    `added` in interface `FrontierJournal`
  Parameters:
    `curi` - CrawlURI that has been scheduled to be added to the Frontier.

### finishedSuccess

    public void finishedSuccess(CrawlURI curi)

  Specified by:
    `finishedSuccess` in interface `FrontierJournal`
  Parameters:
    `curi` - CrawlURI that finished successfully.

### finishedSuccess

    public void finishedSuccess(UURI uuri)

  Specified by:
    `finishedSuccess` in interface `FrontierJournal`
  Parameters:
    `uuri` - UURI that finished successfully.

### finishedSuccess

    protected void finishedSuccess(java.lang.String uuri)

### emitted

    public void emitted(CrawlURI curi)

  Description copied from interface: `FrontierJournal`
  Note that a CrawlURI was emitted for processing. If not followed by a finished or rescheduled notation in the journal, the CrawlURI was still in-process when the journal ended.

  Specified by:
    `emitted` in interface `FrontierJournal`
  Parameters:
    `curi` - CrawlURI emitted.

### finishedDisregard

    public void finishedDisregard(CrawlURI curi)

  Specified by:
    `finishedDisregard` in interface `FrontierJournal`
  Parameters:
    `curi` - CrawlURI finished disregarded (uncounted failure).

### finishedFailure

    public void finishedFailure(CrawlURI curi)

  Specified by:
    `finishedFailure` in interface `FrontierJournal`
  Parameters:
    `curi` - CrawlURI finished unsuccessfully.

### finishedFailure

    public void finishedFailure(UURI uuri)

  Specified by:
    `finishedFailure` in interface `FrontierJournal`
  Parameters:
    `uuri` - UURI that finished unsuccessfully

### finishedFailure

    public void finishedFailure(java.lang.String u)

### rescheduled

    public void rescheduled(CrawlURI curi)

  Specified by:
    `rescheduled` in interface `FrontierJournal`
  Parameters:
    `curi` - CrawlURI that was returned to the Frontier for another try.

### importRecoverLog

    public static void importRecoverLog(java.io.File source,
                                        CrawlController controller,
                                        boolean retainFailures)
                                 throws java.io.IOException

  Utility method for scanning a recovery journal and applying it to a Frontier.

  Parameters:
    `source` - Recover log path.
    `frontier` - Frontier reference.
    `retainFailures` -
  Throws:
    `java.io.IOException`
  See Also:
    `Frontier.importRecoverLog(String, boolean)`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.