Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.frontier
Class WorkQueue

    java.lang.Object
      org.archive.crawler.frontier.WorkQueue

All Implemented Interfaces:
  java.io.Serializable, java.lang.Comparable, CrawlSubstats.HasCrawlSubstats, Frontier.FrontierGroup, Reporter

Direct Known Subclasses:
  BdbWorkQueue

    public abstract class WorkQueue

extends java.lang.Object
implements Frontier.FrontierGroup, java.lang.Comparable, java.io.Serializable, Reporter

A single queue of related URIs to visit, grouped by a classKey (typically "hostname:port" or similar)

Author:
  gojomo, Christian Kohlschuetter
See Also:
  Serialized Form

Field Summary
`protected java.lang.String` | `classKey` The classKey
`(package private) static long` | `serialVersionUID`
`protected CrawlSubstats` | `substats` Substats for all CrawlURIs in this group

Constructor Summary
`WorkQueue(java.lang.String pClassKey)`

Method Summary
`void` | `clearHeld()` Clear isHeld to false
`int` | `compareTo(java.lang.Object obj)`
`protected abstract void` | `deleteItem(WorkQueueFrontier frontier, CrawlURI item)` Removes the given item from the queue.
`long` | `deleteMatching(WorkQueueFrontier frontier, java.lang.String match)` Delete URIs matching the given pattern from this queue.
`protected abstract long` | `deleteMatchingFromQueue(WorkQueueFrontier frontier, java.lang.String match)` Delete URIs matching the given pattern from this queue.
`void` | `dequeue(WorkQueueFrontier frontier)` Remove the peekItem from the queue and adjusts the count.
`void` | `enqueue(WorkQueueFrontier frontier, CrawlURI curi)` Add the given CrawlURI, noting its addition in running count.
`int` | `expend(int amount)` Decrease the internal running budget by the given amount.
`java.lang.String` | `getClassKey()`
`UURI` | `getContextUURI(WorkQueueFrontier wqf)`
`long` | `getCount()`
`java.lang.String[]` | `getReports()` Get an array of report names offered by this Reporter.
`int` | `getSessionBalance()` Return current session 'activity budget balance'
`CrawlSubstats` | `getSubstats()`
`long` | `getTotalExpenditure()` Return the tally of all expenditures on this queue
`long` | `getWakeTime()`
`int` | `incrementSessionBalance(int amount)` Increase the internal running budget to be used before deactivating the queue
`protected abstract void` | `insertItem(WorkQueueFrontier frontier, CrawlURI curi, boolean expectedPresent)` Insert the given curi, whether it is already present or not.
`boolean` | `isHeld()` Whether the queue is already in a lifecycle stage -- such as ready, in-progress, snoozed -- and thus should not be redundantly inserted to readyClassQueues
`boolean` | `isOverBudget()` Check whether queue has temporarily or permanently exceeded its budget.
`boolean` | `isRetired()`
`void` | `noteError(int penalty)` Note an error and assess an extra penalty.
`CrawlURI` | `peek(WorkQueueFrontier frontier)` Return the topmost queue item -- and remember it, such that even later higher-priority inserts don't change it.
`protected abstract CrawlURI` | `peekItem(WorkQueueFrontier frontier)` Returns first item from queue (does not delete)
`int` | `refund(int amount)` A URI should not have been charged against queue (eg it was disregarded); return the amount expended
`void` | `reportTo(java.io.PrintWriter writer)` Make a default report to the passed-in Writer.
`void` | `reportTo(java.lang.String name, java.io.PrintWriter writer)` Make a report of the given name to the passed-in Writer, If null, give the default report.
`protected void` | `resume(WorkQueueFrontier frontier)` Resumes this WorkQueue.
`void` | `setActive(WorkQueueFrontier frontier, boolean b)`
`void` | `setHeld()` Set isHeld to true
`void` | `setRetired(boolean b)` Set the retired status of this queue.
`void` | `setSessionBalance(int balance)` Set the session 'activity budget balance' to the given value
`void` | `setTotalBudget(long budget)` Set the total expenditure level allowable before queue is considered inherently 'over-budget'.
`void` | `setWakeTime(long l)`
`java.lang.String` | `singleLineLegend()` Return a legend for the single-line summary report as a String.
`java.lang.String` | `singleLineReport()` Return a short single-line summary report as a String.
`void` | `singleLineReportTo(java.io.PrintWriter writer)` Make a single-line summary report to the passed-in writer
`protected void` | `suspend(WorkQueueFrontier frontier)` Suspends this WorkQueue.
`void` | `unpeek()` Forgive the peek, allowing a subsequent peek to return a different item.
`void` | `update(WorkQueueFrontier frontier, CrawlURI curi)` Update the given CrawlURI, which should already be present.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### serialVersionUID

    static final long serialVersionUID

See Also:
  Constant Field Values

### classKey

    protected final java.lang.String classKey

  The classKey

### substats

    protected CrawlSubstats substats

  Substats for all CrawlURIs in this group

Constructor Detail

### WorkQueue

    public WorkQueue(java.lang.String pClassKey)

Method Detail

### deleteMatching

    public long deleteMatching(WorkQueueFrontier frontier,
                               java.lang.String match)

  Delete URIs matching the given pattern from this queue.

  Parameters:
    `frontier` -
    `match` -
  Returns:
    count of deleted URIs

### enqueue

    public void enqueue(WorkQueueFrontier frontier,
                        CrawlURI curi)

  Add the given CrawlURI, noting its addition in running count. (It should not already be present.)

  Parameters:
    `frontier` - Work queues manager.
    `curi` - CrawlURI to insert.

### peek

    public CrawlURI peek(WorkQueueFrontier frontier)

  Return the topmost queue item -- and remember it, such that even later higher-priority inserts don't change it. TODO: evaluate if this is really necessary

  Parameters:
    `frontier` - Work queues manager
  Returns:
    topmost queue item, or null

### dequeue

    public void dequeue(WorkQueueFrontier frontier)

  Remove the peekItem from the queue and adjusts the count.

  Parameters:
    `frontier` - Work queues manager.

### setSessionBalance

    public void setSessionBalance(int balance)

  Set the session 'activity budget balance' to the given value

  Parameters:
    `balance` - to use

### getSessionBalance

    public int getSessionBalance()

  Return current session 'activity budget balance'

  Returns:
    session balance

### setTotalBudget

    public void setTotalBudget(long budget)

  Set the total expenditure level allowable before queue is considered inherently 'over-budget'.

  Parameters:
    `budget` -

### isOverBudget

    public boolean isOverBudget()

  Check whether queue has temporarily or permanently exceeded its budget.

  Returns:
    true if queue is over its set budget(s)

### getTotalExpenditure

    public long getTotalExpenditure()

  Return the tally of all expenditures on this queue

  Returns:
    total amount expended on this queue

### incrementSessionBalance

    public int incrementSessionBalance(int amount)

  Increase the internal running budget to be used before deactivating the queue

  Parameters:
    `amount` - amount to increment
  Returns:
    updated budget value

### expend

    public int expend(int amount)

  Decrease the internal running budget by the given amount.

  Parameters:
    `amount` - tp decrement
  Returns:
    updated budget value

### refund

    public int refund(int amount)

  A URI should not have been charged against queue (eg it was disregarded); return the amount expended

  Parameters:
    `amount` - to return
  Returns:
    updated budget value

### noteError

    public void noteError(int penalty)

  Note an error and assess an extra penalty.

  Parameters:
    `penalty` - additional amount to deduct

### setWakeTime

    public void setWakeTime(long l)

  Parameters:
    `l` -

### getWakeTime

    public long getWakeTime()

  Returns:
    wakeTime

### getClassKey

    public java.lang.String getClassKey()

  Returns:
    classKey, the 'identifier', for this queue.

### clearHeld

    public void clearHeld()

  Clear isHeld to false

### isHeld

    public boolean isHeld()

  Whether the queue is already in a lifecycle stage -- such as ready, in-progress, snoozed -- and thus should not be redundantly inserted to readyClassQueues

  Returns:
    isHeld

### setHeld

    public void setHeld()

  Set isHeld to true

### unpeek

    public void unpeek()

  Forgive the peek, allowing a subsequent peek to return a different item.

### compareTo

    public final int compareTo(java.lang.Object obj)

  Specified by:
    `compareTo` in interface `java.lang.Comparable`

### update

    public void update(WorkQueueFrontier frontier,
                       CrawlURI curi)

  Update the given CrawlURI, which should already be present. (This is not checked.) Equivalent to an enqueue without affecting the count.

  Parameters:
    `frontier` - Work queues manager.
    `curi` - CrawlURI to update.

### getCount

    public long getCount()

  Returns:
    Returns the count.

### insertItem

    protected abstract void insertItem(WorkQueueFrontier frontier,
                                       CrawlURI curi,
                                       boolean expectedPresent)
                                throws java.io.IOException

  Insert the given curi, whether it is already present or not. Hook for subclasses.

  Parameters:
    `frontier` - WorkQueueFrontier.
    `curi` - CrawlURI to insert.
  Throws:
    `java.io.IOException` - if there was a problem while inserting the item

### deleteMatchingFromQueue

    protected abstract long deleteMatchingFromQueue(WorkQueueFrontier frontier,
                                                    java.lang.String match)
                                             throws java.io.IOException

  Delete URIs matching the given pattern from this queue.

  Parameters:
    `frontier` - WorkQueues manager.
    `match` - the pattern to match
  Returns:
    count of deleted URIs
  Throws:
    `java.io.IOException` - if there was a problem while deleting

### deleteItem

    protected abstract void deleteItem(WorkQueueFrontier frontier,
                                       CrawlURI item)
                                throws java.io.IOException

  Removes the given item from the queue. This is only used to remove the first item in the queue, so it is not necessary to implement a random-access queue.

  Parameters:
    `frontier` - Work queues manager.
  Throws:
    `java.io.IOException` - if there was a problem while deleting the item

### peekItem

    protected abstract CrawlURI peekItem(WorkQueueFrontier frontier)
                                  throws java.io.IOException

  Returns first item from queue (does not delete)

  Returns:
    The peeked item, or null
  Throws:
    `java.io.IOException` - if there was a problem while peeking

### suspend

    protected void suspend(WorkQueueFrontier frontier)
                    throws java.io.IOException

  Suspends this WorkQueue. Closes all connections to resources etc.

  Parameters:
    `frontier` -
  Throws:
    `java.io.IOException`

### resume

    protected void resume(WorkQueueFrontier frontier)
                   throws java.io.IOException

  Resumes this WorkQueue. Eventually opens connections to resources etc.

  Parameters:
    `frontier` -
  Throws:
    `java.io.IOException`

### setActive

    public void setActive(WorkQueueFrontier frontier,
                          boolean b)

### getReports

    public java.lang.String[] getReports()

  Description copied from interface: `Reporter`
  Get an array of report names offered by this Reporter. A name in brackets indicates a free-form String, in accordance with the informal description inside the brackets, may yield a useful report.

  Specified by:
    `getReports` in interface `Reporter`
  Returns:
    String array of report names, empty if there is only one report type

### reportTo

    public void reportTo(java.io.PrintWriter writer)

  Description copied from interface: `Reporter`
  Make a default report to the passed-in Writer. Should be equivalent to reportTo(null, writer)

  Specified by:
    `reportTo` in interface `Reporter`
  Parameters:
    `writer` - to receive report

### singleLineReportTo

    public void singleLineReportTo(java.io.PrintWriter writer)

  Description copied from interface: `Reporter`
  Make a single-line summary report to the passed-in writer

  Specified by:
    `singleLineReportTo` in interface `Reporter`
  Parameters:
    `writer` - to receive report

### singleLineLegend

    public java.lang.String singleLineLegend()

  Description copied from interface: `Reporter`
  Return a legend for the single-line summary report as a String.

  Specified by:
    `singleLineLegend` in interface `Reporter`
  Returns:
    String single-line summary legend

### singleLineReport

    public java.lang.String singleLineReport()

  Description copied from interface: `Reporter`
  Return a short single-line summary report as a String.

  Specified by:
    `singleLineReport` in interface `Reporter`
  Returns:
    String single-line summary report

### reportTo

    public void reportTo(java.lang.String name,
                         java.io.PrintWriter writer)

  Description copied from interface: `Reporter`
  Make a report of the given name to the passed-in Writer, If null, give the default report.

  Specified by:
    `reportTo` in interface `Reporter`
  Parameters:
    `writer` -
  Throws:
    `java.io.IOException`

### getSubstats

    public CrawlSubstats getSubstats()

  Specified by:
    `getSubstats` in interface `CrawlSubstats.HasCrawlSubstats`

### setRetired

    public void setRetired(boolean b)

  Set the retired status of this queue.

  Parameters:
    `b` - new value for retired status

### isRetired

    public boolean isRetired()

### getContextUURI

    public UURI getContextUURI(WorkQueueFrontier wqf)

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.