Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.framework
Class AbstractTracker

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.AbstractTracker

All Implemented Interfaces:
  java.io.Serializable, java.lang.Runnable, javax.management.DynamicMBean, CrawlStatusListener, StatisticsTracking

Direct Known Subclasses:
  StatisticsTracker

    public abstract class AbstractTracker

extends ModuleType
implements StatisticsTracking, CrawlStatusListener, java.io.Serializable

A partial implementation of the StatisticsTracking interface.

It covers the thread handling. (Launching, pausing etc.) Included in this is keeping track of the total time spent (actually) crawling. Several methods to access the time started, finished etc. are provided.

To handle the thread work the class implements the CrawlStatusListener and uses it's events to pause, resume and stop logging of statistics. The run() method will call logActivity() at intervals specified in the crawl order.

Implementation of logActivity (the actual logging) as well as listening for CrawlURIDisposition events is not addressed.

Author:
  Kristinn Sigurdsson
See Also:
  `StatisticsTracking`, `StatisticsTracker`, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`static java.lang.String` | `ATTR_STATS_INTERVAL` Attribute name for logging interval in seconds setting
`protected CrawlController` | `controller` A reference to the CrawlContoller of the crawl that we are to track statistics for.
`protected long` | `crawlerEndTime`
`protected long` | `crawlerPauseStarted`
`protected long` | `crawlerStartTime`
`protected long` | `crawlerTotalPausedTime`
`static java.lang.Integer` | `DEFAULT_STATISTICS_REPORT_INTERVAL` Default period between logging stat values
`protected long` | `lastLogPointTime` Timestamp of when this logger last wrote something to the log
`protected boolean` | `shouldrun`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.framework.StatisticsTracking
`SEED_DISPOSITION_DISREGARD, SEED_DISPOSITION_FAILURE, SEED_DISPOSITION_NOT_PROCESSED, SEED_DISPOSITION_RETRY, SEED_DISPOSITION_SUCCESS`

Constructor Summary
`AbstractTracker(java.lang.String name, java.lang.String description)`

Method Summary
`long` | `crawlDuration()` Returns how long the current crawl has been running (excluding any time spent paused/suspended/stopped) since it began.
`void` | `crawlEnded(java.lang.String sExitMessage)` Called when a CrawlController has ended a crawl and is about to exit.
`void` | `crawlEnding(java.lang.String sExitMessage)` Called when a CrawlController is ending a crawl (for any reason)
`void` | `crawlPaused(java.lang.String statusMessage)` Called when a CrawlController is actually paused (all threads are idle).
`void` | `crawlPausing(java.lang.String statusMessage)` Called when a CrawlController is going to be paused.
`void` | `crawlResuming(java.lang.String statusMessage)` Called when a CrawlController is resuming a crawl that had been paused.
`void` | `crawlStarted(java.lang.String message)` Called on crawl start.
`protected void` | `dumpReports()` Dump reports, if any, on request or at crawl end.
`protected void` | `finalCleanup()` Cleanup resources used, at crawl end.
`long` | `getCrawlEndTime()` If crawl has ended it will return the time it ended (given by `System.currentTimeMillis()` at that time).
`long` | `getCrawlerTotalElapsedTime()` Total amount of time spent actively crawling so far.
`long` | `getCrawlPauseStartedTime()` Get the time when the the crawl was last paused/suspended (as given by `System.currentTimeMillis()` at that time).
`long` | `getCrawlStartTime()` Get the starting time of the crawl (as given by `System.currentTimeMillis()` when the crawl started).
`long` | `getCrawlTotalPauseTime()` Returns the number of milliseconds that the crawl spent paused or otherwise in a nonactive state.
`protected int` | `getLogWriteInterval()` The number of seconds to wait between writing snapshot data to log file.
`void` | `initialize(CrawlController c)` Sets up the Logger (including logInterval) and registers with the CrawlController for CrawlStatus and CrawlURIDisposition events.
`protected void` | `logNote(java.lang.String note)`
`void` | `noteStart()` Notify tracker that crawl has begun.
`protected void` | `progressStatisticsEvent(java.util.EventObject e)` A method for logging current crawler state.
`java.lang.String` | `progressStatisticsLegend()`
`void` | `run()` Start thread.
`protected void` | `tallyCurrentPause()` For a current pause (if any), add paused time to total and reset

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Methods inherited from interface org.archive.crawler.framework.StatisticsTracking
`activeThreadCount, averageDepth, congestionRatio, currentProcessedDocsPerSec, currentProcessedKBPerSec, deepestUri, getProgressStatistics, getProgressStatisticsLine, getSeedRecordsSortedByStatusCode, processedDocsPerSec, processedKBPerSec, successfullyFetchedCount, totalBytesCrawled, totalBytesWritten, totalCount`

Methods inherited from interface org.archive.crawler.event.CrawlStatusListener
`crawlCheckpoint`

Field Detail

### DEFAULT_STATISTICS_REPORT_INTERVAL

    public static final java.lang.Integer DEFAULT_STATISTICS_REPORT_INTERVAL

  Default period between logging stat values

### ATTR_STATS_INTERVAL

    public static final java.lang.String ATTR_STATS_INTERVAL

  Attribute name for logging interval in seconds setting

  See Also:
    Constant Field Values

### controller

    protected transient CrawlController controller

  A reference to the CrawlContoller of the crawl that we are to track statistics for.

### crawlerStartTime

    protected long crawlerStartTime

### crawlerEndTime

    protected long crawlerEndTime

### crawlerPauseStarted

    protected long crawlerPauseStarted

### crawlerTotalPausedTime

    protected long crawlerTotalPausedTime

### lastLogPointTime

    protected long lastLogPointTime

  Timestamp of when this logger last wrote something to the log

### shouldrun

    protected boolean shouldrun

Constructor Detail

### AbstractTracker

    public AbstractTracker(java.lang.String name,
                           java.lang.String description)

Parameters:
  `name` -
  `description` -

Method Detail

### initialize

    public void initialize(CrawlController c)
                    throws FatalConfigurationException

  Sets up the Logger (including logInterval) and registers with the CrawlController for CrawlStatus and CrawlURIDisposition events.

  Specified by:
    `initialize` in interface `StatisticsTracking`
  Parameters:
    `c` - A crawl controller instance.
  Throws:
    `FatalConfigurationException` - Not thrown here. For overrides that go to settings system for configuration.
  See Also:
    `CrawlStatusListener`, `CrawlURIDispositionListener`

### run

    public void run()

  Start thread. Will call logActivity() at intervals specified by logInterval

  Specified by:
    `run` in interface `java.lang.Runnable`

### progressStatisticsLegend

    public java.lang.String progressStatisticsLegend()

  Specified by:
    `progressStatisticsLegend` in interface `StatisticsTracking`
  Returns:
    legend for progress-statistics lines/log

### noteStart

    public void noteStart()

  Notify tracker that crawl has begun. Must be called outside tracker's own thread, to ensure it is noted before other threads start interacting with tracker.

  Specified by:
    `noteStart` in interface `StatisticsTracking`

### progressStatisticsEvent

    protected void progressStatisticsEvent(java.util.EventObject e)

  A method for logging current crawler state. This method will be called by run() at intervals specified in the crawl order file. It is also invoked when pausing or stopping a crawl to capture the state at that point. Default behavior is call to `CrawlController.logProgressStatistics(java.lang.String)` so CrawlController can act on progress statistics event.

  It is recommended that for implementations of this method it be carefully considered if it should be synchronized in whole or in part

  Parameters:
    `e` - Progress statistics event.

### getCrawlStartTime

    public long getCrawlStartTime()

  Get the starting time of the crawl (as given by `System.currentTimeMillis()` when the crawl started).

  Returns:
    time fo the crawl's start

### getCrawlEndTime

    public long getCrawlEndTime()

  If crawl has ended it will return the time it ended (given by `System.currentTimeMillis()` at that time).
  If crawl is still going on it will return the same as `System.currentTimeMillis()` at the time of the call.

  Returns:
    The time of the crawl ending or the current time if the crawl has not ended.

### getCrawlTotalPauseTime

    public long getCrawlTotalPauseTime()

  Returns the number of milliseconds that the crawl spent paused or otherwise in a nonactive state.

  Returns:
    the number of msec. that the crawl was paused or otherwise suspended.

### getCrawlPauseStartedTime

    public long getCrawlPauseStartedTime()

  Get the time when the the crawl was last paused/suspended (as given by `System.currentTimeMillis()` at that time). Will be 0 if the crawl is not currently paused.

  Returns:
    time of the crawl's last pause/suspend or 0 if the crawl is not currently paused.

### getCrawlerTotalElapsedTime

    public long getCrawlerTotalElapsedTime()

  Description copied from interface: `StatisticsTracking`
  Total amount of time spent actively crawling so far.

  Returns the total amount of time (in milliseconds) that has elapsed from the start of the crawl and until the current time or if the crawl has ended until the the end of the crawl minus any time spent paused.

  Specified by:
    `getCrawlerTotalElapsedTime` in interface `StatisticsTracking`
  Returns:
    Total amount of time (in msec.) spent crawling so far.

### getLogWriteInterval

    protected int getLogWriteInterval()

  The number of seconds to wait between writing snapshot data to log file.

  Returns:
    the number of seconds to wait between writing snapshot data to log file.

### crawlPausing

    public void crawlPausing(java.lang.String statusMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is going to be paused.

  Specified by:
    `crawlPausing` in interface `CrawlStatusListener`
  Parameters:
    `statusMessage` - Should be `STATUS_WAITING_FOR_PAUSE`. Passed for convenience
  See Also:
    `CrawlStatusListener.crawlPausing(java.lang.String)`

### logNote

    protected void logNote(java.lang.String note)

### crawlPaused

    public void crawlPaused(java.lang.String statusMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is actually paused (all threads are idle).

  Specified by:
    `crawlPaused` in interface `CrawlStatusListener`
  Parameters:
    `statusMessage` - Should be `CrawlJob.STATUS_PAUSED`. Passed for convenience

### crawlResuming

    public void crawlResuming(java.lang.String statusMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is resuming a crawl that had been paused.

  Specified by:
    `crawlResuming` in interface `CrawlStatusListener`
  Parameters:
    `statusMessage` - Should be `CrawlJob.STATUS_RUNNING`. Passed for convenience

### tallyCurrentPause

    protected void tallyCurrentPause()

  For a current pause (if any), add paused time to total and reset

### crawlEnding

    public void crawlEnding(java.lang.String sExitMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController is ending a crawl (for any reason)

  Specified by:
    `crawlEnding` in interface `CrawlStatusListener`
  Parameters:
    `sExitMessage` - Type of exit. Should be one of the STATUS constants in defined in CrawlJob.
  See Also:
    `CrawlJob`

### crawlEnded

    public void crawlEnded(java.lang.String sExitMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController has ended a crawl and is about to exit.

  Specified by:
    `crawlEnded` in interface `CrawlStatusListener`
  Parameters:
    `sExitMessage` - Type of exit. Should be one of the STATUS constants in defined in CrawlJob.
  See Also:
    `CrawlStatusListener.crawlEnded(java.lang.String)`

### crawlStarted

    public void crawlStarted(java.lang.String message)

  Description copied from interface: `CrawlStatusListener`
  Called on crawl start.

  Specified by:
    `crawlStarted` in interface `CrawlStatusListener`
  Parameters:
    `message` - Start message.

### dumpReports

    protected void dumpReports()

  Dump reports, if any, on request or at crawl end.

### finalCleanup

    protected void finalCleanup()

  Cleanup resources used, at crawl end.

### crawlDuration

    public long crawlDuration()

  Description copied from interface: `StatisticsTracking`
  Returns how long the current crawl has been running (excluding any time spent paused/suspended/stopped) since it began.

  Specified by:
    `crawlDuration` in interface `StatisticsTracking`
  Returns:
    The length of time - in msec - that this crawl has been running.
  See Also:
    `StatisticsTracking.crawlDuration()`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.