Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.framework
Class CrawlScope

    java.lang.Object
      javax.management.Attribute
          org.archive.crawler.settings.Type
              org.archive.crawler.settings.ComplexType
                  org.archive.crawler.settings.ModuleType
                      org.archive.crawler.framework.Filter
                          org.archive.crawler.framework.CrawlScope

All Implemented Interfaces:
  java.io.Serializable, javax.management.DynamicMBean

Direct Known Subclasses:
  ClassicScope, DecidingScope

    public class CrawlScope

extends Filter

A CrawlScope instance defines which URIs are "in" a particular crawl. It is essentially a Filter which determines, looking at the totality of information available about a CandidateURI/CrawlURI instamce, if that URI should be scheduled for crawling. Dynamic information inherent in the discovery of the URI -- such as the path by which it was discovered -- may be considered. Dynamic information which requires the consultation of external and potentially volatile information -- such as current robots.txt requests and the history of attempts to crawl the same URI -- should NOT be considered. Those potentially high-latency decisions should be made at another step.

Author:
  gojomo
See Also:
  Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType
`ComplexType.MBeanAttributeInfoIterator`

Field Summary
`static java.lang.String` | `ATTR_NAME`
`static java.lang.String` | `ATTR_REREAD_SEEDS_ON_CONFIG` Whether every configu change should trigger a rereading of the original seeds spec/file.
`static java.lang.String` | `ATTR_SEEDS`
`static java.lang.Boolean` | `DEFAULT_REREAD_SEEDS_ON_CONFIG`
`protected java.util.Set<SeedListener>` | `seedListeners`

Fields inherited from class org.archive.crawler.framework.Filter
`ATTR_ENABLED`

Fields inherited from class org.archive.crawler.settings.ComplexType
`definition, definitionMap`

Constructor Summary
`CrawlScope()` Default constructor.
`CrawlScope(java.lang.String name)` Constructs a new CrawlScope.

Method Summary
`boolean` | `addSeed(CandidateURI curi)` Add a new seed to scope.
`void` | `addSeedListener(SeedListener sl)`
`protected void` | `checkClose(java.util.Iterator iter)` Convenience method to close SeedFileIterator, if appropriate.
`java.io.File` | `getSeedfile()`
`void` | `initialize(CrawlController controller)` Initialize is called just before the crawler starts to run.
`protected boolean` | `isSameHost(UURI a, UURI b)`
`protected boolean` | `isSeed(java.lang.Object o)` Check if a URI is in the seeds.
`void` | `kickUpdate()` Take note of a situation (such as settings edit) where involved reconfiguration (such as reading from external files) may be necessary.
`void` | `listUsedFiles(java.util.List<java.lang.String> list)` Those Modules that use files on disk should list them all when this method is called.
`void` | `refreshSeeds()` Refresh seeds.
`java.util.Iterator<UURI>` | `seedsIterator()` Gets an iterator over all configured seeds.
`java.util.Iterator<UURI>` | `seedsIterator(java.io.Writer ignoredItemWriter)` Gets an iterator over all configured seeds.
`java.lang.String` | `toString()`

Methods inherited from class org.archive.crawler.framework.Filter
`accepts, getFilterOffPosition, innerAccepts, returnTrueIfMatches`

Methods inherited from class org.archive.crawler.settings.ModuleType
`addElement`

Methods inherited from class org.archive.crawler.settings.ComplexType
`addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, unsetAttribute`

Methods inherited from class org.archive.crawler.settings.Type
`addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient`

Methods inherited from class javax.management.Attribute
`getName`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

### ATTR_NAME

    public static final java.lang.String ATTR_NAME

See Also:
  Constant Field Values

### ATTR_SEEDS

    public static final java.lang.String ATTR_SEEDS

See Also:
  Constant Field Values

### ATTR_REREAD_SEEDS_ON_CONFIG

    public static final java.lang.String ATTR_REREAD_SEEDS_ON_CONFIG

  Whether every configu change should trigger a rereading of the original seeds spec/file.

  See Also:
    Constant Field Values

### DEFAULT_REREAD_SEEDS_ON_CONFIG

    public static final java.lang.Boolean DEFAULT_REREAD_SEEDS_ON_CONFIG

### seedListeners

    protected java.util.Set<SeedListener> seedListeners

Constructor Detail

### CrawlScope

    public CrawlScope(java.lang.String name)

  Constructs a new CrawlScope.

  Parameters:
    `name` - the name is ignored since it always have to be the value of the constant ATT_NAME.

### CrawlScope

    public CrawlScope()

  Default constructor.

Method Detail

### initialize

    public void initialize(CrawlController controller)

  Initialize is called just before the crawler starts to run. The settings system is up and initialized so can be used. This initialize happens after `ComplexType.earlyInitialize(CrawlerSettings)`.

  Parameters:
    `controller` - Controller object.

### toString

    public java.lang.String toString()

  Overrides:
    `toString` in class `Filter`

### refreshSeeds

    public void refreshSeeds()

  Refresh seeds.

### getSeedfile

    public java.io.File getSeedfile()

  Returns:
    Seed list file or null if problem getting settings file.

### isSeed

    protected boolean isSeed(java.lang.Object o)

  Check if a URI is in the seeds.

  Parameters:
    `o` - the URI to check.
  Returns:
    true if URI is a seed.

### isSameHost

    protected boolean isSameHost(UURI a,
                                 UURI b)

  Parameters:
    `a` - First UURI of compare.
    `b` - Second UURI of compare.
  Returns:
    True if UURIs are of same host.

### listUsedFiles

    public void listUsedFiles(java.util.List<java.lang.String> list)

  Description copied from class: `ModuleType`
  Those Modules that use files on disk should list them all when this method is called.

  Each file (as a string name with full path) should be added to the provided list.

  Modules that do not use any files can safely ignore this method.

  Overrides:
    `listUsedFiles` in class `ModuleType`
  Parameters:
    `list` - The list to add files to.

### kickUpdate

    public void kickUpdate()

  Take note of a situation (such as settings edit) where involved reconfiguration (such as reading from external files) may be necessary.

  Overrides:
    `kickUpdate` in class `Filter`

### seedsIterator

    public java.util.Iterator<UURI> seedsIterator()

  Gets an iterator over all configured seeds. Subclasses which cache seeds in memory can override with more efficient implementation.

  Returns:
    Iterator, perhaps over a disk file, of seeds

### seedsIterator

    public java.util.Iterator<UURI> seedsIterator(java.io.Writer ignoredItemWriter)

  Gets an iterator over all configured seeds. Subclasses which cache seeds in memory can override with more efficient implementation.

  Parameters:
    `ignoredItemWriter` - optional writer to get ignored seed items report
  Returns:
    Iterator, perhaps over a disk file, of seeds

### checkClose

    protected void checkClose(java.util.Iterator iter)

  Convenience method to close SeedFileIterator, if appropriate.

  Parameters:
    `iter` - Iterator to check if SeedFileIterator needing closing

### addSeed

    public boolean addSeed(CandidateURI curi)

  Add a new seed to scope. By default, simply appends to seeds file, though subclasses may handle differently.

  This method is *not* sufficient to get the new seed scheduled in the Frontier for crawling -- it only affects the Scope's seed record (and decisions which flow from seeds).

  Parameters:
    `curi` - CandidateUri to add
  Returns:
    true if successful, false if add failed for any reason

### addSeedListener

    public void addSeedListener(SeedListener sl)

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.