Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.frontier
Class BdbMultipleWorkQueues

    java.lang.Object
      org.archive.crawler.frontier.BdbMultipleWorkQueues

    public class BdbMultipleWorkQueues

extends java.lang.Object

A BerkeleyDB-database-backed structure for holding ordered groupings of CrawlURIs. Reading the groupings from specific per-grouping (per-classKey/per-Host) starting points allows this to act as a collection of independent queues.

For how the bdb keys are made, see `calculateInsertKey(CrawlURI)`.

TODO: refactor, improve naming.

Author:
  gojomo

Nested Class Summary
`class` | `BdbMultipleWorkQueues.BdbFrontierMarker` Marker for remembering a position within the BdbMultipleWorkQueues.

Constructor Summary
`BdbMultipleWorkQueues(com.sleepycat.je.Environment env, com.sleepycat.bind.serial.StoredClassCatalog classCatalog, boolean recycle)` Create the multi queue in the given environment.

Method Summary
`void` | `addCap(byte[] origin)` Add a dummy 'cap' entry at the given insertion key.
`(package private) static com.sleepycat.je.DatabaseEntry` | `calculateInsertKey(CrawlURI curi)` Calculate the insertKey that places a CrawlURI in the desired spot.
`(package private) static byte[]` | `calculateOriginKey(java.lang.String classKey)` Calculate the 'origin' key for a virtual queue of items with the given classKey.
`void` | `close()` clean up
`void` | `delete(CrawlURI item)` Delete the given CrawlURI from persistent store.
`long` | `deleteMatchingFromQueue(java.lang.String match, java.lang.String queue, com.sleepycat.je.DatabaseEntry headKey)` Delete all CrawlURIs matching the given expression.
`protected void` | `forAllPendingDo(org.apache.commons.collections.Closure c)` Utility method to perform action for all pending CrawlURI instances.
`CrawlURI` | `get(com.sleepycat.je.DatabaseEntry headKey)` Get the next nearest item after the given key.
`protected com.sleepycat.je.DatabaseEntry` | `getFirstKey()`
`java.util.List` | `getFrom(FrontierMarker m, int maxMatches)`
`FrontierMarker` | `getInitialMarker(java.lang.String regexpr)` Get a marker for beginning a scan over all contents
`protected com.sleepycat.je.OperationStatus` | `getNextNearestItem(com.sleepycat.je.DatabaseEntry headKey, com.sleepycat.je.DatabaseEntry result)`
`void` | `put(CrawlURI curi, boolean overwriteIfPresent)` Put the given CrawlURI in at the appropriate place.
`(package private) void` | `sync()` Method used by BdbFrontier during checkpointing.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### BdbMultipleWorkQueues

    public BdbMultipleWorkQueues(com.sleepycat.je.Environment env,
                                 com.sleepycat.bind.serial.StoredClassCatalog classCatalog,
                                 boolean recycle)
                          throws com.sleepycat.je.DatabaseException

  Create the multi queue in the given environment.

  Parameters:
    `env` - bdb environment to use
    `classCatalog` - Class catalog to use.
    `recycle` - True if we are to reuse db content if any.
  Throws:
    `com.sleepycat.je.DatabaseException`

Method Detail

### deleteMatchingFromQueue

    public long deleteMatchingFromQueue(java.lang.String match,
                                        java.lang.String queue,
                                        com.sleepycat.je.DatabaseEntry headKey)
                                 throws com.sleepycat.je.DatabaseException

  Delete all CrawlURIs matching the given expression.

  Parameters:
    `match` -
    `queue` -
    `headKey` -
  Returns:
    count of deleted items
  Throws:
    `com.sleepycat.je.DatabaseException`
    `com.sleepycat.je.DatabaseException`

### getFrom

    public java.util.List getFrom(FrontierMarker m,
                                  int maxMatches)
                           throws com.sleepycat.je.DatabaseException

  Parameters:
    `m` - marker
    `maxMatches` -
  Returns:
    list of matches starting from marker position
  Throws:
    `com.sleepycat.je.DatabaseException`

### getInitialMarker

    public FrontierMarker getInitialMarker(java.lang.String regexpr)

  Get a marker for beginning a scan over all contents

  Parameters:
    `regexpr` -
  Returns:
    a marker pointing to the first item

### getFirstKey

    protected com.sleepycat.je.DatabaseEntry getFirstKey()
                                                  throws com.sleepycat.je.DatabaseException

  Returns:
    the key to the first item in the database
  Throws:
    `com.sleepycat.je.DatabaseException`

### get

    public CrawlURI get(com.sleepycat.je.DatabaseEntry headKey)
                 throws com.sleepycat.je.DatabaseException

  Get the next nearest item after the given key. Relies on external discipline -- we'll look at the queues count of how many items it has -- to avoid asking for something from a range where there are no associated items -- otherwise could get first item of next 'queue' by mistake.

  TODO: hold within a queue's range

  Parameters:
    `headKey` - Key prefix that demarks the beginning of the range in `pendingUrisDB` we're interested in.
  Returns:
    CrawlURI.
  Throws:
    `com.sleepycat.je.DatabaseException`

### getNextNearestItem

    protected com.sleepycat.je.OperationStatus getNextNearestItem(com.sleepycat.je.DatabaseEntry headKey,
                                                                  com.sleepycat.je.DatabaseEntry result)
                                                           throws com.sleepycat.je.DatabaseException

  Throws:
    `com.sleepycat.je.DatabaseException`

### put

    public void put(CrawlURI curi,
                    boolean overwriteIfPresent)
             throws com.sleepycat.je.DatabaseException

  Put the given CrawlURI in at the appropriate place.

  Parameters:
    `curi` -
  Throws:
    `com.sleepycat.je.DatabaseException`

### calculateOriginKey

    static byte[] calculateOriginKey(java.lang.String classKey)

  Calculate the 'origin' key for a virtual queue of items with the given classKey. This origin key will be a prefix of the keys for all items in the queue.

  Parameters:
    `classKey` - String key to derive origin byte key from
  Returns:
    a byte array key

### calculateInsertKey

    static com.sleepycat.je.DatabaseEntry calculateInsertKey(CrawlURI curi)

  Calculate the insertKey that places a CrawlURI in the desired spot. First bytes are always classKey (usu. host) based -- ensuring grouping by host -- terminated by a zero byte. Then 8 bytes of data ensuring desired ordering within that 'queue' are used. The first byte of these 8 is priority -- allowing 'immediate' and 'soon' items to sort above regular. Next 1 byte is 'cost'. Last 6 bytes are ordinal serial number, ensuring earlier-discovered URIs sort before later. NOTE: Dangers here are: (1) priorities or costs over 2^7 (signed byte comparison) (2) ordinals over 2^48 Package access & static for testing purposes.

  Parameters:
    `curi` -
  Returns:
    a DatabaseEntry key for the CrawlURI

### delete

    public void delete(CrawlURI item)
                throws com.sleepycat.je.DatabaseException

  Delete the given CrawlURI from persistent store. Requires the key under which it was stored be available.

  Parameters:
    `item` -
  Throws:
    `com.sleepycat.je.DatabaseException`

### sync

    void sync()

  Method used by BdbFrontier during checkpointing.

  The backing bdbje database has been marked deferred write so we save on writes to disk. Means no guarantees disk will have whats in memory unless a sync is called (Calling sync on the bdbje Environment is not sufficent).

  Package access only because only Frontiers of this package would ever need access.

  See Also:
    Deferred Write Databases

### close

    public void close()

  clean up

### addCap

    public void addCap(byte[] origin)

  Add a dummy 'cap' entry at the given insertion key. Prevents 'seeks' to queue heads from holding lock on last item of 'preceding' queue. See: http://sourceforge.net/tracker/index.php?func=detail&aid=1262665&group_id=73833&atid=539102

  Parameters:
    `origin` - key at which to insert the cap

### forAllPendingDo

    protected void forAllPendingDo(org.apache.commons.collections.Closure c)
                            throws com.sleepycat.je.DatabaseException

  Utility method to perform action for all pending CrawlURI instances.

  Parameters:
    `c` - Closure action to perform
  Throws:
    `com.sleepycat.je.DatabaseException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.