Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV NEXT | FRAMES NO FRAMES

## Uses of Interface
org.archive.crawler.datamodel.FetchStatusCodes

Packages that use FetchStatusCodes
org.archive.crawler.datamodel |
org.archive.crawler.fetcher |
org.archive.crawler.framework |
org.archive.crawler.frontier |
org.archive.crawler.postprocessor |
org.archive.crawler.prefetch |
org.archive.crawler.processor |
org.archive.crawler.writer |

Uses of FetchStatusCodes in org.archive.crawler.datamodel

Classes in org.archive.crawler.datamodel that implement FetchStatusCodes
`class` | `CrawlSubstats` Collector of statististics for a 'subset' of a crawl, such as a server (host:port), host, or frontier group (eg queue).
`class` | `CrawlURI` Represents a candidate URI and the associated state it collects as it is crawled.

Uses of FetchStatusCodes in org.archive.crawler.fetcher

Classes in org.archive.crawler.fetcher that implement FetchStatusCodes
`class` | `FetchDNS` Processor to resolve 'dns:' URIs.
`class` | `FetchHTTP` HTTP fetcher that uses Apache Jakarta Commons HttpClient library.

Uses of FetchStatusCodes in org.archive.crawler.framework

Classes in org.archive.crawler.framework that implement FetchStatusCodes
`class` | `ToeThread` One "worker thread"; asks for CrawlURIs, processes them, repeats unless told otherwise.
`class` | `WriterPoolProcessor` Abstract implementation of a file pool processor.

Uses of FetchStatusCodes in org.archive.crawler.frontier

Classes in org.archive.crawler.frontier that implement FetchStatusCodes
`class` | `AbstractFrontier` Shared facilities for Frontier implementations.
`class` | `AdaptiveRevisitFrontier` A Frontier that will repeatedly visit all encountered URIs.
`class` | `BdbFrontier` A Frontier using several BerkeleyDB JE Databases to hold its record of known hosts (queues), and pending URIs.
`class` | `DomainSensitiveFrontier` Deprecated. As of release 1.10.0. Replaced by `BdbFrontier` and `QuotaEnforcer`.
`class` | `WorkQueueFrontier` A common Frontier base using several queues to hold pending URIs.

Uses of FetchStatusCodes in org.archive.crawler.postprocessor

Classes in org.archive.crawler.postprocessor that implement FetchStatusCodes
`class` | `CrawlStateUpdater` A step, late in the processing of a CrawlURI, for updating the per-host information that may have been affected by the fetch.
`class` | `FrontierScheduler` 'Schedule' with the Frontier CandidateURIs being carried by the passed CrawlURI.
`class` | `LinksScoper` Determine which extracted links are within scope.

Uses of FetchStatusCodes in org.archive.crawler.prefetch

Classes in org.archive.crawler.prefetch that implement FetchStatusCodes
`class` | `PreconditionEnforcer` Ensures the preconditions for a fetch -- such as DNS lookup or acquiring and respecting a robots.txt policy -- are satisfied before a URI is passed to subsequent stages.
`class` | `Preselector` If set to recheck the crawl's scope, gives a yes/no on whether a CrawlURI should be processed at all.
`class` | `QuotaEnforcer` A simple quota enforcer.
`class` | `RuntimeLimitEnforcer` A processor to enforce runtime limits on crawls.

Uses of FetchStatusCodes in org.archive.crawler.processor

Classes in org.archive.crawler.processor that implement FetchStatusCodes
`class` | `BeanShellProcessor` A processor which runs a BeanShell script on the CrawlURI.
`class` | `CrawlMapper` A simple crawl splitter/mapper, dividing up CandidateURIs/CrawlURIs between crawlers by diverting some range of URIs to local log files (which can then be imported to other crawlers).
`class` | `HashCrawlMapper` Maps URIs to one of N crawler names by applying a hash to the URI's (possibly-transformed) classKey.
`class` | `LexicalCrawlMapper` A simple crawl splitter/mapper, dividing up CandidateURIs/CrawlURIs between crawlers by diverting some range of URIs to local log files (which can then be imported to other crawlers).

Uses of FetchStatusCodes in org.archive.crawler.writer

Classes in org.archive.crawler.writer that implement FetchStatusCodes
`class` | `ARCWriterProcessor` Processor module for writing the results of successful fetches (and perhaps someday, certain kinds of network failures) to the Internet Archive ARC file format.
`class` | `WARCWriterProcessor` Experimental WARCWriterProcessor.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV NEXT | FRAMES NO FRAMES

Copyright © 2003-2008 Internet Archive. All Rights Reserved.