Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.frontier
Class WorkQueueFrontier

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.frontier.AbstractFrontier
                          org.archive.crawler.frontier.WorkQueueFrontier

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean, CoreAttributeConstants, FetchStatusCodes, UriUniqFilter.HasUriReceiver, CrawlStatusListener, Frontier, Reporter

Direct Known Subclasses:
  BdbFrontier

    public abstract class WorkQueueFrontier

extends AbstractFrontier
implements FetchStatusCodes, CoreAttributeConstants, UriUniqFilter.HasUriReceiver, java.io.Serializable

A common Frontier base using several queues to hold pending URIs. Uses in-memory map of all known 'queues' inside a single database. Round-robins between all queues.

Author:
  Gordon Mohr, Christian Kohlschuetter
See Also:
  Serialized Form

Nested Class Summary
`class` | `WorkQueueFrontier.WakeTask`

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Nested classes/interfaces inherited from interface org.archive.crawler.framework.Frontier
`Frontier.FrontierGroup`

Field Summary
`static java.lang.String` | `ALL_NONEMPTY`
`static java.lang.String` | `ALL_QUEUES`
`protected java.util.Map<java.lang.String,WorkQueue>` | `allQueues` All known queues.
`protected UriUniqFilter` | `alreadyIncluded` those UURIs which are already in-process (or processed), and thus should not be rescheduled
`static java.lang.String` | `ATTR_BALANCE_REPLENISH_AMOUNT` amount to replenish budget on each activation (duty cycle)
`static java.lang.String` | `ATTR_COST_POLICY` cost assignment policy to use (by class name)
`static java.lang.String` | `ATTR_ERROR_PENALTY_AMOUNT` whether to hold queues INACTIVE until needed for throughput
`static java.lang.String` | `ATTR_HOLD_QUEUES` whether to hold queues INACTIVE until needed for throughput
`static java.lang.String` | `ATTR_QUEUE_TOTAL_BUDGET` total expenditure to allow a queue before 'retiring' it
`static java.lang.String` | `ATTR_SNOOZE_DEACTIVATE_MS` When a snooze target for a queue is longer than this amount, and there are already ready queues, deactivate rather than snooze the current queue -- so other more responsive sites get a chance in active rotation.
`static java.lang.String` | `ATTR_TARGET_READY_QUEUES_BACKLOG` target size of ready queues backlog
`(package private) java.lang.String[]` | `AVAILABLE_COST_POLICIES` all policies available to be chosen
`protected static java.lang.Integer` | `DEFAULT_BALANCE_REPLENISH_AMOUNT`
`protected static java.lang.String` | `DEFAULT_COST_POLICY`
`protected static java.lang.Integer` | `DEFAULT_ERROR_PENALTY_AMOUNT`
`protected static java.lang.Boolean` | `DEFAULT_HOLD_QUEUES`
`protected static java.lang.Long` | `DEFAULT_QUEUE_TOTAL_BUDGET`
`static java.lang.Long` | `DEFAULT_SNOOZE_DEACTIVATE_MS`
`protected static java.lang.Integer` | `DEFAULT_TARGET_READY_QUEUES_BACKLOG`
`protected java.util.Queue<java.lang.String>` | `inactiveQueues` All 'inactive' queues, not yet in active rotation.
`protected org.apache.commons.collections.Bag` | `inProcessQueues` all per-class queues from whom a URI is outstanding
`protected WorkQueue` | `longestActiveQueue`
`protected WorkQueueFrontier.WakeTask` | `nextWake` Task for next wake
`protected java.util.concurrent.BlockingQueue<java.lang.String>` | `readyClassQueues` All per-class queues whose first item may be handed out.
`protected java.util.concurrent.Semaphore` | `readyFiller` single-thread access to ready-filling code
`protected static java.lang.String[]` | `REPORTS`
`protected java.util.Queue<java.lang.String>` | `retiredQueues` 'retired' queues, no longer considered for activation.
`protected java.util.SortedSet<WorkQueue>` | `snoozedClassQueues` All per-class queues held in snoozed state, sorted by wake time.
`static java.lang.String` | `STANDARD_REPORT`
`protected int` | `targetSizeForReadyQueues` Target (minimum) size to keep readyClassQueues
`protected java.util.Timer` | `wakeTimer` Timer for tasks which wake head item of snoozedClassQueues

Fields inherited from class org.archive.crawler.frontier.AbstractFrontier
`ACCEPTABLE_FORCE_QUEUE, ATTR_DELAY_FACTOR, ATTR_FORCE_QUEUE, ATTR_MAX_DELAY, ATTR_MAX_HOST_BANDWIDTH_USAGE, ATTR_MAX_OVERALL_BANDWIDTH_USAGE, ATTR_MAX_RETRIES, ATTR_MIN_DELAY, ATTR_PAUSE_AT_FINISH, ATTR_PAUSE_AT_START, ATTR_PREFERENCE_EMBED_HOPS, ATTR_QUEUE_ASSIGNMENT_POLICY, ATTR_RECOVERY_ENABLED, ATTR_RETRY_DELAY, ATTR_SOURCE_TAG_SEEDS, controller, DEFAULT_ATTR_RECOVERY_ENABLED, DEFAULT_DELAY_FACTOR, DEFAULT_FORCE_QUEUE, DEFAULT_MAX_DELAY, DEFAULT_MAX_HOST_BANDWIDTH_USAGE, DEFAULT_MAX_OVERALL_BANDWIDTH_USAGE, DEFAULT_MAX_RETRIES, DEFAULT_MIN_DELAY, DEFAULT_PAUSE_AT_FINISH, DEFAULT_PAUSE_AT_START, DEFAULT_PREFERENCE_EMBED_HOPS, DEFAULT_RETRY_DELAY, DEFAULT_SOURCE_TAG_SEEDS, disregardedUriCount, failedFetchCount, IGNORED_SEEDS_FILENAME, lastMaxBandwidthKB, liveDisregardedUriCount, liveFailedFetchCount, liveQueuedUriCount, liveSucceededFetchCount, nextOrdinal, processedBytesAfterLastEmittedURI, queuedUriCount, shouldPause, shouldTerminate, succeededFetchCount, totalProcessedBytes`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Fields inherited from interface org.archive.crawler.datamodel.FetchStatusCodes
`S_BLOCKED_BY_CUSTOM_PROCESSOR, S_BLOCKED_BY_QUOTA, S_BLOCKED_BY_RUNTIME_LIMIT, S_BLOCKED_BY_USER, S_CONNECT_FAILED, S_CONNECT_LOST, S_DEEMED_CHAFF, S_DEFERRED, S_DELETED_BY_USER, S_DNS_SUCCESS, S_DOMAIN_PREREQUISITE_FAILURE, S_DOMAIN_UNRESOLVABLE, S_GETBYNAME_SUCCESS, S_OTHER_PREREQUISITE_FAILURE, S_OUT_OF_SCOPE, S_PREREQUISITE_UNSCHEDULABLE_FAILURE, S_PROCESSING_THREAD_KILLED, S_ROBOTS_PRECLUDED, S_ROBOTS_PREREQUISITE_FAILURE, S_RUNTIME_EXCEPTION, S_SERIOUS_ERROR, S_TIMEOUT, S_TOO_MANY_EMBED_HOPS, S_TOO_MANY_LINK_HOPS, S_TOO_MANY_RETRIES, S_UNATTEMPTED, S_UNFETCHABLE_URI, S_UNQUEUEABLE`

Fields inherited from interface org.archive.crawler.datamodel.CoreAttributeConstants
`A_ANNOTATIONS, A_CONTENT_DIGEST, A_CONTENT_TYPE, A_CREDENTIAL_AVATARS_KEY, A_DELAY_FACTOR, A_DISTANCE_FROM_SEED, A_DNS_FETCH_TIME, A_DNS_SERVER_IP_LABEL, A_ETAG_HEADER, A_FETCH_BEGAN_TIME, A_FETCH_COMPLETED_TIME, A_FETCH_HISTORY, A_FORCE_RETIRE, A_HERITABLE_KEYS, A_HTML_BASE, A_HTTP_BIND_ADDRESS, A_HTTP_PROXY_HOST, A_HTTP_PROXY_PORT, A_HTTP_TRANSACTION, A_LAST_MODIFIED_HEADER, A_LOCALIZED_ERRORS, A_META_ROBOTS, A_MINIMUM_DELAY, A_MIRROR_PATH, A_PREREQUISITE_URI, A_REFERENCE_LENGTH, A_RETRY_DELAY, A_RRECORD_SET_LABEL, A_RUNTIME_EXCEPTION, A_SOURCE_TAG, A_STATUS, HEADER_TRUNC, LENGTH_TRUNC, TIMER_TRUNC, TRUNC_SUFFIX`

Fields inherited from interface org.archive.crawler.framework.Frontier
`ATTR_NAME`

Constructor Summary
`WorkQueueFrontier(java.lang.String name, java.lang.String description)` Create the CommonFrontier

Method Summary
`protected void` | `appendQueueReports(java.io.PrintWriter w, java.util.Iterator iterator, int total, int max)` Append queue report to general Frontier report.
`protected CrawlURI` | `asCrawlUri(CandidateURI caUri)`
`long` | `averageDepth()`
`protected abstract void` | `closeQueue()`
`float` | `congestionRatio()`
`void` | `considerIncluded(UURI u)` Notify Frontier that it should consider the given UURI as if already scheduled.
`void` | `crawlEnded(java.lang.String sExitMessage)` Called when a CrawlController has ended a crawl and is about to exit.
`protected abstract UriUniqFilter` | `createAlreadyIncluded()` Create a UriUniqFilter that will serve as record of already seen URIs.
`long` | `deepestUri()`
`void` | `deleted(CrawlURI curi)` Force logging, etc.
`long` | `deleteURIs(java.lang.String uriMatch)` Delete all scheduled URIs matching the given regex.
`long` | `deleteURIs(java.lang.String uriMatch, java.lang.String queueMatch)` Delete all scheduled URIs matching the given regex, in queues with names matching the second given regex.
`long` | `discoveredUriCount()` (non-Javadoc)
`void` | `finished(CrawlURI curi)` Note that the previously emitted CrawlURI has completed its processing (for now).
`protected void` | `forget(CrawlURI curi)` Forget the given CrawlURI.
`Frontier.FrontierGroup` | `getGroup(CrawlURI curi)` Get the 'frontier group' (usually queue) for the given CrawlURI.
`protected abstract WorkQueue` | `getQueueFor(CrawlURI curi)` Return the work queue for the given CrawlURI's classKey.
`protected abstract WorkQueue` | `getQueueFor(java.lang.String classKey)` Return the work queue for the given classKey, or null if no such queue exists.
`java.lang.String[]` | `getReports()` Get an array of report names offered by this Reporter.
`void` | `initialize(CrawlController c)` Initializes the Frontier, given the supplied CrawlController.
`protected abstract void` | `initQueue()`
`protected void` | `initQueuesOfQueues()` Set up the various queues-of-queues used by the frontier.
`boolean` | `isEmpty()` Frontier is empty only if all queues are empty and no URIs are in-process
`void` | `kickUpdate()` Accomodate any changes in settings.
`CrawlURI` | `next()` Return the next CrawlURI to be processed (and presumably visited/fetched) by a a worker thread.
`void` | `receive(CandidateURI caUri)` Accept the given CandidateURI for scheduling, as it has passed the alreadyIncluded filter.
`void` | `reportTo(java.lang.String name, java.io.PrintWriter writer)` This method compiles a human readable report on the status of the frontier at the time of the call.
`void` | `schedule(CandidateURI caUri)` Arrange for the given CandidateURI to be visited, if it is not already scheduled/completed.
`protected void` | `sendToQueue(CrawlURI curi)` Send a CrawlURI to the appropriate subqueue.
`java.lang.String` | `singleLineLegend()` Return a legend for the single-line summary report as a String.
`void` | `singleLineReportTo(java.io.PrintWriter w)` Make a single-line summary report to the passed-in writer
`(package private) void` | `wakeQueues()` Wake any queues sitting in the snoozed queue whose time has come.
`protected abstract boolean` | `workQueueDataOnDisk()` Returns `true` if the WorkQueue implementation of this Frontier stores its workload on disk instead of relying on serialization mechanisms.

Methods inherited from class org.archive.crawler.frontier.AbstractFrontier
`applySpecialHandling, canonicalize, canonicalize, crawlCheckpoint, crawlEnding, crawlPaused, crawlPausing, crawlResuming, crawlStarted, decrementQueuedCount, disregardedUriCount, doJournalAdded, doJournalDisregarded, doJournalEmitted, doJournalFinishedFailure, doJournalFinishedSuccess, doJournalRescheduled, failedFetchCount, finishedUriCount, getClassKey, getFrontierJournal, getQueueAssignmentPolicy, getServer, importRecoverLog, incrementDisregardedUriCount, incrementFailedFetchCount, incrementQueuedUriCount, incrementQueuedUriCount, incrementSucceededFetchCount, isDisregarded, loadSeeds, log, logLocalizedErrors, needsRetrying, noteAboutToEmit, overMaxRetries, pause, politenessDelayFor, preNext, queuedUriCount, reportTo, retryDelayFor, saveIgnoredItems, scratchDirFor, singleLineReport, start, succeededFetchCount, tally, terminate, totalBytesWritten, unpause`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement, listUsedFiles`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Methods inherited from interface org.archive.crawler.framework.Frontier
`getInitialMarker, getURIsList`

Field Detail

### ATTR_SNOOZE_DEACTIVATE_MS

    public static final java.lang.String ATTR_SNOOZE_DEACTIVATE_MS

  When a snooze target for a queue is longer than this amount, and there are already ready queues, deactivate rather than snooze the current queue -- so other more responsive sites get a chance in active rotation. (As a result, queue's next try may be much further in the future than the snooze target delay.)

  See Also:
    Constant Field Values

### DEFAULT_SNOOZE_DEACTIVATE_MS

    public static java.lang.Long DEFAULT_SNOOZE_DEACTIVATE_MS

### ATTR_HOLD_QUEUES

    public static final java.lang.String ATTR_HOLD_QUEUES

  whether to hold queues INACTIVE until needed for throughput

  See Also:
    Constant Field Values

### DEFAULT_HOLD_QUEUES

    protected static final java.lang.Boolean DEFAULT_HOLD_QUEUES

### ATTR_BALANCE_REPLENISH_AMOUNT

    public static final java.lang.String ATTR_BALANCE_REPLENISH_AMOUNT

  amount to replenish budget on each activation (duty cycle)

  See Also:
    Constant Field Values

### DEFAULT_BALANCE_REPLENISH_AMOUNT

    protected static final java.lang.Integer DEFAULT_BALANCE_REPLENISH_AMOUNT

### ATTR_ERROR_PENALTY_AMOUNT

    public static final java.lang.String ATTR_ERROR_PENALTY_AMOUNT

  whether to hold queues INACTIVE until needed for throughput

  See Also:
    Constant Field Values

### DEFAULT_ERROR_PENALTY_AMOUNT

    protected static final java.lang.Integer DEFAULT_ERROR_PENALTY_AMOUNT

### ATTR_QUEUE_TOTAL_BUDGET

    public static final java.lang.String ATTR_QUEUE_TOTAL_BUDGET

  total expenditure to allow a queue before 'retiring' it

  See Also:
    Constant Field Values

### DEFAULT_QUEUE_TOTAL_BUDGET

    protected static final java.lang.Long DEFAULT_QUEUE_TOTAL_BUDGET

### ATTR_COST_POLICY

    public static final java.lang.String ATTR_COST_POLICY

  cost assignment policy to use (by class name)

  See Also:
    Constant Field Values

### DEFAULT_COST_POLICY

    protected static final java.lang.String DEFAULT_COST_POLICY

### ATTR_TARGET_READY_QUEUES_BACKLOG

    public static final java.lang.String ATTR_TARGET_READY_QUEUES_BACKLOG

  target size of ready queues backlog

  See Also:
    Constant Field Values

### DEFAULT_TARGET_READY_QUEUES_BACKLOG

    protected static final java.lang.Integer DEFAULT_TARGET_READY_QUEUES_BACKLOG

### alreadyIncluded

    protected transient UriUniqFilter alreadyIncluded

  those UURIs which are already in-process (or processed), and thus should not be rescheduled

### allQueues

    protected transient java.util.Map<java.lang.String,WorkQueue> allQueues

  All known queues.

### readyClassQueues

    protected java.util.concurrent.BlockingQueue<java.lang.String> readyClassQueues

  All per-class queues whose first item may be handed out. Linked-list of keys for the queues.

### targetSizeForReadyQueues

    protected int targetSizeForReadyQueues

  Target (minimum) size to keep readyClassQueues

### readyFiller

    protected transient java.util.concurrent.Semaphore readyFiller

  single-thread access to ready-filling code

### inactiveQueues

    protected java.util.Queue<java.lang.String> inactiveQueues

  All 'inactive' queues, not yet in active rotation. Linked-list of keys for the queues.

### retiredQueues

    protected java.util.Queue<java.lang.String> retiredQueues

  'retired' queues, no longer considered for activation. Linked-list of keys for queues.

### inProcessQueues

    protected org.apache.commons.collections.Bag inProcessQueues

  all per-class queues from whom a URI is outstanding

### snoozedClassQueues

    protected java.util.SortedSet<WorkQueue> snoozedClassQueues

  All per-class queues held in snoozed state, sorted by wake time.

### wakeTimer

    protected transient java.util.Timer wakeTimer

  Timer for tasks which wake head item of snoozedClassQueues

### nextWake

    protected transient WorkQueueFrontier.WakeTask nextWake

  Task for next wake

### longestActiveQueue

    protected WorkQueue longestActiveQueue

### AVAILABLE_COST_POLICIES

    java.lang.String[] AVAILABLE_COST_POLICIES

  all policies available to be chosen

### STANDARD_REPORT

    public static java.lang.String STANDARD_REPORT

### ALL_NONEMPTY

    public static java.lang.String ALL_NONEMPTY

### ALL_QUEUES

    public static java.lang.String ALL_QUEUES

### REPORTS

    protected static java.lang.String[] REPORTS

Constructor Detail

### WorkQueueFrontier

    public WorkQueueFrontier(java.lang.String name,
                             java.lang.String description)

  Create the CommonFrontier

  Parameters:
    `name` -
    `description` -

Method Detail

### initialize

    public void initialize(CrawlController c)
                    throws FatalConfigurationException,
                           java.io.IOException

  Initializes the Frontier, given the supplied CrawlController.

  Specified by:
    `initialize` in interface `Frontier`
  Overrides:
    `initialize` in class `AbstractFrontier`
  Parameters:
    `c` - The CrawlController that created the Frontier.
  Throws:
    `FatalConfigurationException` - If provided settings are illegal or otherwise unusable.
    `java.io.IOException` - If there is a problem reading settings or seeds file from disk.
  See Also:
    `Frontier.initialize(org.archive.crawler.framework.CrawlController)`

### initQueuesOfQueues

    protected void initQueuesOfQueues()

  Set up the various queues-of-queues used by the frontier. Override in implementing subclasses to reduce or eliminate risk of queues growing without bound.

### crawlEnded

    public void crawlEnded(java.lang.String sExitMessage)

  Description copied from interface: `CrawlStatusListener`
  Called when a CrawlController has ended a crawl and is about to exit.

  Specified by:
    `crawlEnded` in interface `CrawlStatusListener`
  Overrides:
    `crawlEnded` in class `AbstractFrontier`
  Parameters:
    `sExitMessage` - Type of exit. Should be one of the STATUS constants in defined in CrawlJob.
  See Also:
    `CrawlJob`

### createAlreadyIncluded

    protected abstract UriUniqFilter createAlreadyIncluded()
                                                    throws java.io.IOException

  Create a UriUniqFilter that will serve as record of already seen URIs.

  Returns:
    A UURISet that will serve as a record of already seen URIs
  Throws:
    `java.io.IOException`

### schedule

    public void schedule(CandidateURI caUri)

  Arrange for the given CandidateURI to be visited, if it is not already scheduled/completed.

  Specified by:
    `schedule` in interface `Frontier`
  Parameters:
    `caUri` - The URI to schedule.
  See Also:
    `Frontier.schedule(org.archive.crawler.datamodel.CandidateURI)`

### receive

    public void receive(CandidateURI caUri)

  Accept the given CandidateURI for scheduling, as it has passed the alreadyIncluded filter. Choose a per-classKey queue and enqueue it. If this item has made an unready queue ready, place that queue on the readyClassQueues queue.

  Specified by:
    `receive` in interface `UriUniqFilter.HasUriReceiver`
  Parameters:
    `caUri` - CandidateURI.

### asCrawlUri

    protected CrawlURI asCrawlUri(CandidateURI caUri)

  Overrides:
    `asCrawlUri` in class `AbstractFrontier`

### sendToQueue

    protected void sendToQueue(CrawlURI curi)

  Send a CrawlURI to the appropriate subqueue.

  Parameters:
    `curi` -

### kickUpdate

    public void kickUpdate()

  Accomodate any changes in settings.

  Specified by:
    `kickUpdate` in interface `Frontier`
  Overrides:
    `kickUpdate` in class `AbstractFrontier`
  See Also:
    `Frontier.kickUpdate()`

### getQueueFor

    protected abstract WorkQueue getQueueFor(CrawlURI curi)

  Return the work queue for the given CrawlURI's classKey. URIs are ordered and politeness-delayed within their 'class'. If the requested queue is not found, a new instance is created.

  Parameters:
    `curi` - CrawlURI to base queue on
  Returns:
    the found or created ClassKeyQueue

### getQueueFor

    protected abstract WorkQueue getQueueFor(java.lang.String classKey)

  Return the work queue for the given classKey, or null if no such queue exists.

  Parameters:
    `classKey` - key to look for
  Returns:
    the found WorkQueue

### next

    public CrawlURI next()
                  throws java.lang.InterruptedException,
                         EndedException

  Return the next CrawlURI to be processed (and presumably visited/fetched) by a a worker thread. Relies on the readyClassQueues having been loaded with any work queues that are eligible to provide a URI.

  Specified by:
    `next` in interface `Frontier`
  Returns:
    next CrawlURI to be processed. Or null if none is available.
  Throws:
    `java.lang.InterruptedException`
    `EndedException`
  See Also:
    `Frontier.next()`

### wakeQueues

    void wakeQueues()

  Wake any queues sitting in the snoozed queue whose time has come.

### finished

    public void finished(CrawlURI curi)

  Note that the previously emitted CrawlURI has completed its processing (for now). The CrawlURI may be scheduled to retry, if appropriate, and other related URIs may become eligible for release via the next next() call, as a result of finished(). (non-Javadoc)

  Specified by:
    `finished` in interface `Frontier`
  Parameters:
    `curi` - The URI that has finished processing.
  See Also:
    `Frontier.finished(org.archive.crawler.datamodel.CrawlURI)`

### forget

    protected void forget(CrawlURI curi)

  Forget the given CrawlURI. This allows a new instance to be created in the future, if it is reencountered under different circumstances.

  Parameters:
    `curi` - The CrawlURI to forget

### discoveredUriCount

    public long discoveredUriCount()

  (non-Javadoc)

  Specified by:
    `discoveredUriCount` in interface `Frontier`
  Returns:
    Number of discovered URIs.
  See Also:
    `Frontier.discoveredUriCount()`

### deleteURIs

    public long deleteURIs(java.lang.String uriMatch)

  Delete all scheduled URIs matching the given regex.

  Specified by:
    `deleteURIs` in interface `Frontier`
  Parameters:
    `match` - regex of URIs to delete
  Returns:
    Number of items deleted.

### deleteURIs

    public long deleteURIs(java.lang.String uriMatch,
                           java.lang.String queueMatch)

  Delete all scheduled URIs matching the given regex, in queues with names matching the second given regex.

  Specified by:
    `deleteURIs` in interface `Frontier`
  Parameters:
    `uriMatch` - regex of URIs to delete
    `queueMatch` - regex of queues to affect, or null for all
  Returns:
    Number of items deleted.

### getReports

    public java.lang.String[] getReports()

  Description copied from interface: `Reporter`
  Get an array of report names offered by this Reporter. A name in brackets indicates a free-form String, in accordance with the informal description inside the brackets, may yield a useful report.

  Specified by:
    `getReports` in interface `Reporter`
  Returns:
    String array of report names, empty if there is only one report type

### singleLineReportTo

    public void singleLineReportTo(java.io.PrintWriter w)

  Description copied from interface: `Reporter`
  Make a single-line summary report to the passed-in writer

  Specified by:
    `singleLineReportTo` in interface `Reporter`
  Parameters:
    `w` - Where to write to.

### singleLineLegend

    public java.lang.String singleLineLegend()

  Description copied from interface: `Reporter`
  Return a legend for the single-line summary report as a String.

  Specified by:
    `singleLineLegend` in interface `Reporter`
  Returns:
    String single-line summary legend

### reportTo

    public void reportTo(java.lang.String name,
                         java.io.PrintWriter writer)

  This method compiles a human readable report on the status of the frontier at the time of the call.

  Specified by:
    `reportTo` in interface `Reporter`
  Parameters:
    `name` - Name of report.
    `writer` - Where to write to.

### appendQueueReports

    protected void appendQueueReports(java.io.PrintWriter w,
                                      java.util.Iterator iterator,
                                      int total,
                                      int max)

  Append queue report to general Frontier report.

  Parameters:
    `w` - StringBuffer to append to.
    `iterator` - An iterator over
    `total` -
    `max` -

### deleted

    public void deleted(CrawlURI curi)

  Force logging, etc. of operator- deleted CrawlURIs

  Specified by:
    `deleted` in interface `Frontier`
  Parameters:
    `curi` - Deleted CrawlURI.
  See Also:
    `Frontier.deleted(org.archive.crawler.datamodel.CrawlURI)`

### considerIncluded

    public void considerIncluded(UURI u)

  Description copied from interface: `Frontier`
  Notify Frontier that it should consider the given UURI as if already scheduled.

  Specified by:
    `considerIncluded` in interface `Frontier`
  Parameters:
    `u` - UURI instance to add to the Already Included set.

### initQueue

    protected abstract void initQueue()
                               throws java.io.IOException

  Throws:
    `java.io.IOException`

### closeQueue

    protected abstract void closeQueue()
                                throws java.io.IOException

  Throws:
    `java.io.IOException`

### workQueueDataOnDisk

    protected abstract boolean workQueueDataOnDisk()

  Returns `true` if the WorkQueue implementation of this Frontier stores its workload on disk instead of relying on serialization mechanisms. TODO: rename! (this is a very misleading name) or kill (don't see any implementations that return false)

  Returns:
    a constant boolean value for this class/instance

### getGroup

    public Frontier.FrontierGroup getGroup(CrawlURI curi)

  Description copied from interface: `Frontier`
  Get the 'frontier group' (usually queue) for the given CrawlURI.

  Specified by:
    `getGroup` in interface `Frontier`
  Parameters:
    `curi` - CrawlURI to find matching group
  Returns:
    FrontierGroup for the CrawlURI

### averageDepth

    public long averageDepth()

  Specified by:
    `averageDepth` in interface `Frontier`

### congestionRatio

    public float congestionRatio()

  Specified by:
    `congestionRatio` in interface `Frontier`

### deepestUri

    public long deepestUri()

  Specified by:
    `deepestUri` in interface `Frontier`

### isEmpty

    public boolean isEmpty()

  Description copied from class: `AbstractFrontier`
  Frontier is empty only if all queues are empty and no URIs are in-process

  Specified by:
    `isEmpty` in interface `Frontier`
  Overrides:
    `isEmpty` in class `AbstractFrontier`
  Returns:
    True if queues are empty.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.