Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler.framework
Class CrawlController

    java.lang.Object
      org.archive.crawler.framework.CrawlController

All Implemented Interfaces:
  java.io.Serializable, Reporter

Direct Known Subclasses:
  CrawlJob.MBeanCrawlController

    public class CrawlController

extends java.lang.Object
implements java.io.Serializable, Reporter

CrawlController collects all the classes which cooperate to perform a crawl and provides a high-level interface to the running crawl. As the "global context" for a crawl, subcomponents will often reach each other through the CrawlController.

Author:
  Gordon Mohr
See Also:
  Serialized Form

Field Summary
`static java.lang.String` | `CURRENT_LOG_SUFFIX` suffix to use on active logs
`java.util.logging.Logger` | `localErrors` This logger is for job-scoped logging, specifically errors which happen and are handled within a particular processor.
`static char` | `MANIFEST_CONFIG_FILE` abbrieviation label for config files in manifest
`static char` | `MANIFEST_LOG_FILE` abbrieviation label for log files in manifest
`static java.lang.String` | `MANIFEST_REPORT`
`static char` | `MANIFEST_REPORT_FILE` abbrieviation label for report files in manifest
`static java.lang.String` | `PROCESSORS_REPORT`
`protected java.util.ArrayList<CrawlURIDispositionListener>` | `registeredCrawlURIDispositionListeners`
`java.util.logging.Logger` | `reports` Logger to hold job summary report.
`protected static java.lang.String[]` | `REPORTS`
`java.util.logging.Logger` | `runtimeErrors` This logger contains unexpected runtime errors.
`protected StatisticsTracking` | `statistics`
`java.util.logging.Logger` | `uriErrors` Special log for URI format problems, wherever they may occur.
`java.util.logging.Logger` | `uriProcessing` Crawl progress logger.

Constructor Summary
`CrawlController()` Default constructor

Method Summary
`void` | `acquireContinuePermission()` Proceed only if allowed, giving CrawlController a chance to enforce single-thread mode.
`void` | `addCrawlStatusListener(CrawlStatusListener cl)` Register for CrawlStatus events.
`void` | `addCrawlURIDispositionListener(CrawlURIDispositionListener cl)` Register for CrawlURIDisposition events.
`void` | `addOrderToManifest()` Add order file contents to manifest.
`void` | `addToManifest(java.lang.String file, char type, boolean bundle)` Add a file to the manifest of files used/generated by the current crawl.
`boolean` | `atFinish()` Evaluate if the crawl should stop because it is finished, without actually stopping the crawl.
`void` | `beginCrawlStop()` Start the process of stopping the crawl.
`void` | `checkFinish()` Evaluate if the crawl should stop because it is finished.
`(package private) void` | `checkpoint()` Run checkpointing.
`protected void` | `checkpointBdb(java.io.File checkpointDir)` Checkpoint bdb.
`protected void` | `checkpointBigMaps(java.io.File cpDir)`
`void` | `closeLogFiles()` Close all log files and remove handlers from loggers.
`(package private) void` | `completePause()`
`protected void` | `completeStop()` Called when the last toethread exits.
`protected FatalConfigurationException` | `convertToFatalConfigurationException(java.lang.Exception e)`
`protected void` | `copySettings(java.io.File checkpointDir)` Copy off the settings.
`void` | `fireCrawledURIDisregardEvent(CrawlURI curi)` Allows an external class to raise a CrawlURIDispostion crawledURIDisregard event that will be broadcast to all listeners that have registered with the CrawlController.
`void` | `fireCrawledURIFailureEvent(CrawlURI curi)` Allows an external class to raise a CrawlURIDispostion crawledURIFailure event that will be broadcast to all listeners that have registered with the CrawlController.
`void` | `fireCrawledURINeedRetryEvent(CrawlURI curi)` Allows an external class to raise a CrawlURIDispostion crawledURINeedRetry event that will be broadcast to all listeners that have registered with the CrawlController.
`void` | `fireCrawledURISuccessfulEvent(CrawlURI curi)` Allows an external class to raise a CrawlURIDispostion crawledURISuccessful event that will be broadcast to all listeners that have registered with the CrawlController.
`void` | `freeReserveMemory()`
`int` | `getActiveToeCount()`
`EnhancedEnvironment` | `getBdbEnvironment()`
`protected java.lang.String` | `getBdbLogFileName(long index)`
`
`<K,V> java.util.Map<K,V>`
` | `getBigMap(java.lang.String dbName, java.lang.Class<? super K> keyClass, java.lang.Class<? super V> valueClass)` Call this method to get instance of the crawler BigMap implementation.
`protected boolean` | `getCheckpointCopyBdbjeLogs()`
`Checkpoint` | `getCheckpointRecover()` Get recover checkpoint.
`static Checkpoint` | `getCheckpointRecover(CrawlOrder order)`
`java.io.File` | `getCheckpointsDisk()`
`com.sleepycat.bind.serial.StoredClassCatalog` | `getClassCatalog()` Deprecated. use EnhancedEnvironment's getClassCatalog() instead
`java.io.File` | `getDisk()` Get the 'working' directory of the current crawl.
`ProcessorChain` | `getFirstProcessorChain()` Get the first processor chain.
`Frontier` | `getFrontier()`
`java.io.File` | `getLogsDir()`
`CrawlOrder` | `getOrder()`
`ProcessorChain` | `getPostprocessorChain()` Get the postprocessor chain.
`ProcessorChainList` | `getProcessorChainList()` Get the list of processor chains.
`java.lang.String[]` | `getReports()` Get an array of report names offered by this Reporter.
`CrawlScope` | `getScope()`
`java.io.File` | `getScratchDisk()`
`ServerCache` | `getServerCache()`
`java.io.File` | `getSettingsDir(java.lang.String key)` Return fullpath to the directory named by `key` in settings.
`SettingsHandler` | `getSettingsHandler()`
`java.lang.Object` | `getState()`
`java.io.File` | `getStateDisk()`
`StatisticsTracking` | `getStatistics()`
`int` | `getToeCount()`
`ToePool` | `getToePool()`
`void` | `initialize(SettingsHandler sH)` Starting from nothing, set up CrawlController and associated classes to be ready for a first crawl.
`void` | `installThreadContextSettingsHandler()` Utility method to install this crawl's SettingsHandler into the 'global' (for this thread) holder, so that any subsequent deserialization operations in this thread can find it.
`boolean` | `isCheckpointing()`
`boolean` | `isCheckpointRecover()`
`static boolean` | `isCheckpointRecover(CrawlOrder order)`
`boolean` | `isPaused()` Tell if the controller is paused
`boolean` | `isPausing()`
`boolean` | `isRunning()`
`void` | `kickUpdate()` While many settings will update automatically when the SettingsHandler is modified, some settings need to be explicitly changed to reflect new settings.
`void` | `killThread(int threadNumber, boolean replace)` Kills a thread.
`void` | `logProgressStatistics(java.lang.String msg)` Log to the progress statistics log.
`void` | `logUriError(org.apache.commons.httpclient.URIException e, UURI u, java.lang.CharSequence l)` Log a URIException from deep inside other components to the crawl's shared log.
`void` | `multiThreadMode()` Go to back to regular multi thread mode, where all ToeThreads may proceed at once
`java.lang.String` | `oneLineReportThreads()`
`protected void` | `processBdbLogs(java.io.File checkpointDir, java.lang.String lastBdbCheckpointLog)`
`void` | `progressStatisticsEvent(java.util.EventObject e)` Called whenever progress statistics logging event.
`void` | `releaseContinuePermission()` Relinquish continue permission at end of processing (allowing another thread to proceed if in single-thread mode).
`protected void` | `reportManifestTo(java.io.PrintWriter writer)`
`protected void` | `reportProcessorsTo(java.io.PrintWriter writer)` Compiles and returns a human readable report on the active processors.
`void` | `reportTo(java.io.PrintWriter writer)` Make a default report to the passed-in Writer.
`void` | `reportTo(java.lang.String name, java.io.PrintWriter writer)` Make a report of the given name to the passed-in Writer, If null, give the default report.
`void` | `requestCrawlCheckpoint()` Request a checkpoint.
`void` | `requestCrawlPause()` Stop the crawl temporarly.
`void` | `requestCrawlResume()` Resume crawl from paused state
`void` | `requestCrawlStart()` Operator requested crawl begin
`void` | `requestCrawlStop()` Operator requested for crawl to stop.
`void` | `requestCrawlStop(java.lang.String message)` Operator requested for crawl to stop.
`protected void` | `restoreStatisticsTracker(MapType loggers, java.lang.String replaceName)`
`protected void` | `rotateLogFiles(java.lang.String generationSuffix)`
`protected void` | `runFrontierRecover(java.lang.String recoverPath)`
`protected void` | `sendCheckpointEvent(java.io.File checkpointDir)` Send the checkpoint event.
`protected void` | `sendCrawlStateChangeEvent(java.lang.Object newState, java.lang.String message)` Send crawl change event to all listeners.
`protected void` | `setBdbjeBkgrdThreads(com.sleepycat.je.EnvironmentConfig config, java.util.List threads, java.lang.String setting)`
`void` | `setOrder(CrawlOrder o)`
`protected void` | `setupCheckpointRecover()` Does setup of checkpoint recover.
`java.lang.String` | `singleLineLegend()` Return a legend for the single-line summary report as a String.
`java.lang.String` | `singleLineReport()` Return a short single-line summary report as a String.
`void` | `singleLineReportTo(java.io.PrintWriter writer)` Make a single-line summary report to the passed-in writer
`void` | `singleThreadMode()` Go to single thread mode, where only one ToeThread may proceed at a time.
`void` | `toeEnded()` Note that a ToeThread ended, possibly completing the crawl-stop.
`void` | `toePaused()` Note that a ToeThread reached paused condition, possibly completing the crawl-pause.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### MANIFEST_CONFIG_FILE

    public static final char MANIFEST_CONFIG_FILE

  abbrieviation label for config files in manifest

  See Also:
    Constant Field Values

### MANIFEST_REPORT_FILE

    public static final char MANIFEST_REPORT_FILE

  abbrieviation label for report files in manifest

  See Also:
    Constant Field Values

### MANIFEST_LOG_FILE

    public static final char MANIFEST_LOG_FILE

  abbrieviation label for log files in manifest

  See Also:
    Constant Field Values

### CURRENT_LOG_SUFFIX

    public static final java.lang.String CURRENT_LOG_SUFFIX

  suffix to use on active logs

  See Also:
    Constant Field Values

### uriProcessing

    public transient java.util.logging.Logger uriProcessing

  Crawl progress logger. No exceptions. Logs summary result of each url processing.

### runtimeErrors

    public transient java.util.logging.Logger runtimeErrors

  This logger contains unexpected runtime errors. Would contain errors trying to set up a job or failures inside processors that they are not prepared to recover from.

### localErrors

    public transient java.util.logging.Logger localErrors

  This logger is for job-scoped logging, specifically errors which happen and are handled within a particular processor. Examples would be socket timeouts, exceptions thrown by extractors, etc.

### uriErrors

    public transient java.util.logging.Logger uriErrors

  Special log for URI format problems, wherever they may occur.

### reports

    public transient java.util.logging.Logger reports

  Logger to hold job summary report. Large state reports made at infrequent intervals (e.g. job ending) go here.

### statistics

    protected StatisticsTracking statistics

### registeredCrawlURIDispositionListeners

    protected transient java.util.ArrayList<CrawlURIDispositionListener> registeredCrawlURIDispositionListeners

### PROCESSORS_REPORT

    public static final java.lang.String PROCESSORS_REPORT

See Also:
  Constant Field Values

### MANIFEST_REPORT

    public static final java.lang.String MANIFEST_REPORT

See Also:
  Constant Field Values

### REPORTS

    protected static final java.lang.String[] REPORTS

Constructor Detail

### CrawlController

    public CrawlController()

  Default constructor

Method Detail

### initialize

    public void initialize(SettingsHandler sH)
                    throws InitializationException

  Starting from nothing, set up CrawlController and associated classes to be ready for a first crawl.

  Parameters:
    `sH` - Settings handler.
  Throws:
    `InitializationException`

### installThreadContextSettingsHandler

    public void installThreadContextSettingsHandler()

  Utility method to install this crawl's SettingsHandler into the 'global' (for this thread) holder, so that any subsequent deserialization operations in this thread can find it.

  Parameters:
    `sH` -

### setupCheckpointRecover

    protected void setupCheckpointRecover()
                                   throws java.io.IOException

  Does setup of checkpoint recover. Copies bdb log files into state dir.

  Throws:
    `java.io.IOException`

### getCheckpointCopyBdbjeLogs

    protected boolean getCheckpointCopyBdbjeLogs()

### getBdbEnvironment

    public EnhancedEnvironment getBdbEnvironment()

  Returns:
    the shared EnhancedEnvironment

### getClassCatalog

    public com.sleepycat.bind.serial.StoredClassCatalog getClassCatalog()

  Deprecated. use EnhancedEnvironment's getClassCatalog() instead

### addCrawlStatusListener

    public void addCrawlStatusListener(CrawlStatusListener cl)

  Register for CrawlStatus events.

  Parameters:
    `cl` - a class implementing the CrawlStatusListener interface
  See Also:
    `CrawlStatusListener`

### addCrawlURIDispositionListener

    public void addCrawlURIDispositionListener(CrawlURIDispositionListener cl)

  Register for CrawlURIDisposition events.

  Parameters:
    `cl` - a class implementing the CrawlURIDispostionListener interface
  See Also:
    `CrawlURIDispositionListener`

### fireCrawledURISuccessfulEvent

    public void fireCrawledURISuccessfulEvent(CrawlURI curi)

  Allows an external class to raise a CrawlURIDispostion crawledURISuccessful event that will be broadcast to all listeners that have registered with the CrawlController.

  Parameters:
    `curi` - - The CrawlURI that will be sent with the event notification.
  See Also:
    `CrawlURIDispositionListener.crawledURISuccessful(CrawlURI)`

### fireCrawledURINeedRetryEvent

    public void fireCrawledURINeedRetryEvent(CrawlURI curi)

  Allows an external class to raise a CrawlURIDispostion crawledURINeedRetry event that will be broadcast to all listeners that have registered with the CrawlController.

  Parameters:
    `curi` - - The CrawlURI that will be sent with the event notification.
  See Also:
    `CrawlURIDispositionListener.crawledURINeedRetry(CrawlURI)`

### fireCrawledURIDisregardEvent

    public void fireCrawledURIDisregardEvent(CrawlURI curi)

  Allows an external class to raise a CrawlURIDispostion crawledURIDisregard event that will be broadcast to all listeners that have registered with the CrawlController.

  Parameters:
    `curi` - - The CrawlURI that will be sent with the event notification.
  See Also:
    `CrawlURIDispositionListener.crawledURIDisregard(CrawlURI)`

### fireCrawledURIFailureEvent

    public void fireCrawledURIFailureEvent(CrawlURI curi)

  Allows an external class to raise a CrawlURIDispostion crawledURIFailure event that will be broadcast to all listeners that have registered with the CrawlController.

  Parameters:
    `curi` - - The CrawlURI that will be sent with the event notification.
  See Also:
    `CrawlURIDispositionListener.crawledURIFailure(CrawlURI)`

### runFrontierRecover

    protected void runFrontierRecover(java.lang.String recoverPath)
                               throws javax.management.AttributeNotFoundException,
                                      javax.management.MBeanException,
                                      javax.management.ReflectionException,
                                      FatalConfigurationException

  Throws:
    `javax.management.AttributeNotFoundException`
    `javax.management.MBeanException`
    `javax.management.ReflectionException`
    `FatalConfigurationException`

### getLogsDir

    public java.io.File getLogsDir()

  Returns:
    The logging directory or null if problem reading the settings.

### getSettingsDir

    public java.io.File getSettingsDir(java.lang.String key)
                                throws javax.management.AttributeNotFoundException

  Return fullpath to the directory named by `key` in settings. If directory does not exist, it and all intermediary dirs will be created.

  Parameters:
    `key` - Key to use going to settings.
  Returns:
    Full path to directory named by `key`.
  Throws:
    `javax.management.AttributeNotFoundException`

### restoreStatisticsTracker

    protected void restoreStatisticsTracker(MapType loggers,
                                            java.lang.String replaceName)
                                     throws FatalConfigurationException

  Throws:
    `FatalConfigurationException`

### convertToFatalConfigurationException

    protected FatalConfigurationException convertToFatalConfigurationException(java.lang.Exception e)

### rotateLogFiles

    protected void rotateLogFiles(java.lang.String generationSuffix)
                           throws java.io.IOException

  Throws:
    `java.io.IOException`

### closeLogFiles

    public void closeLogFiles()

  Close all log files and remove handlers from loggers.

### getStatistics

    public StatisticsTracking getStatistics()

  Returns:
    Object this controller is using to track crawl statistics

### sendCrawlStateChangeEvent

    protected void sendCrawlStateChangeEvent(java.lang.Object newState,
                                             java.lang.String message)

  Send crawl change event to all listeners.

  Parameters:
    `newState` - State change we're to tell listeners' about.
    `message` - Message on state change.
  See Also:
    `for special case event sending telling listeners to checkpoint.`

### sendCheckpointEvent

    protected void sendCheckpointEvent(java.io.File checkpointDir)
                                throws java.lang.Exception

  Send the checkpoint event. Has its own method apart from `sendCrawlStateChangeEvent(Object, String)` because checkpointing throws an Exception (Didn't want to have to wrap all of the sendCrawlStateChangeEvent in try/catches).

  Parameters:
    `checkpointDir` - Where to write checkpoint state to.
  Throws:
    `java.lang.Exception`

### requestCrawlStart

    public void requestCrawlStart()

  Operator requested crawl begin

### completeStop

    protected void completeStop()

  Called when the last toethread exits.

### completePause

    void completePause()

### requestCrawlCheckpoint

    public void requestCrawlCheckpoint()
                                throws java.lang.IllegalStateException

  Request a checkpoint. Sets a checkpointing thread running.

  Throws:
    `java.lang.IllegalStateException` - Thrown if crawl is not in paused state (Crawl must be first paused before checkpointing).

### isCheckpointing

    public boolean isCheckpointing()

  Returns:
    True if checkpointing.

### checkpoint

    void checkpoint()
              throws java.lang.Exception

  Run checkpointing. CrawlController takes care of managing the checkpointing/serializing of bdb, the StatisticsTracker, and the CheckpointContext. Other modules that want to revive themselves on checkpoint recovery need to save state during their `CrawlStatusListener.crawlCheckpoint(File)` invocation and then in their #initialize if a module, or in their #initialTask if a processor, check with the CrawlController if its checkpoint recovery. If it is, read in their old state from the pointed to checkpoint directory.

  Default access only to be called by Checkpointer.

  Throws:
    `java.lang.Exception`

### copySettings

    protected void copySettings(java.io.File checkpointDir)
                         throws java.io.IOException

  Copy off the settings.

  Parameters:
    `checkpointDir` - Directory to write checkpoint to.
  Throws:
    `java.io.IOException`

### checkpointBdb

    protected void checkpointBdb(java.io.File checkpointDir)
                          throws com.sleepycat.je.DatabaseException,
                                 java.io.IOException,
                                 java.lang.RuntimeException

  Checkpoint bdb. I used do a call to log cleaning as suggested in je-2.0 javadoc but takes way too much time (20minutes for a crawl of 1million items). Assume cleaner is keeping up. Below was log cleaning loop .

      int totalCleaned = 0;
       for (int cleaned = 0; (cleaned = this.bdbEnvironment.cleanLog()) != 0;
        totalCleaned += cleaned) {
            LOGGER.fine("Cleaned " + cleaned + " log files.");
       }

  I also used to do a sync. But, from Mark Hayes, sync and checkpoint are effectively same thing only sync is not configurable. He suggests doing one or the other:

  MS: Reading code, Environment.sync() is a checkpoint. Looks like I don't need to call a checkpoint after calling a sync?

  MH: Right, they're almost the same thing -- just do one or the other, not both. With the new API, you'll need to do a checkpoint not a sync, because the sync() method has no config parameter. Don't worry -- it's fine to do a checkpoint even though you're not using.

  Parameters:
    `checkpointDir` - Directory to write checkpoint to.
  Throws:
    `com.sleepycat.je.DatabaseException`
    `java.io.IOException`
    `java.lang.RuntimeException` - Thrown if failed setup of new bdb environment.

### processBdbLogs

    protected void processBdbLogs(java.io.File checkpointDir,
                                  java.lang.String lastBdbCheckpointLog)
                           throws java.io.IOException

  Throws:
    `java.io.IOException`

### getBdbLogFileName

    protected java.lang.String getBdbLogFileName(long index)

### setBdbjeBkgrdThreads

    protected void setBdbjeBkgrdThreads(com.sleepycat.je.EnvironmentConfig config,
                                        java.util.List threads,
                                        java.lang.String setting)

### getCheckpointRecover

    public Checkpoint getCheckpointRecover()

  Get recover checkpoint. Returns null if we're NOT in recover mode. Looks at ATTR_RECOVER_PATH and if its a directory, assumes checkpoint recover. If checkpoint mode, returns Checkpoint instance if checkpoint was VALID (else null).

  Returns:
    Checkpoint instance if we're in recover checkpoint mode and the pointed-to checkpoint was valid.
  See Also:
    `isCheckpointRecover()`

### getCheckpointRecover

    public static Checkpoint getCheckpointRecover(CrawlOrder order)

### isCheckpointRecover

    public static boolean isCheckpointRecover(CrawlOrder order)

### isCheckpointRecover

    public boolean isCheckpointRecover()

  Returns:
    True if we're in checkpoint recover mode. Call `getCheckpointRecover()` to get at Checkpoint instance that has info on checkpoint directory being recovered from.

### requestCrawlStop

    public void requestCrawlStop()

  Operator requested for crawl to stop.

### requestCrawlStop

    public void requestCrawlStop(java.lang.String message)

  Operator requested for crawl to stop.

  Parameters:
    `message` -

### beginCrawlStop

    public void beginCrawlStop()

  Start the process of stopping the crawl.

### requestCrawlPause

    public void requestCrawlPause()

  Stop the crawl temporarly.

### isPaused

    public boolean isPaused()

  Tell if the controller is paused

  Returns:
    true if paused

### isPausing

    public boolean isPausing()

### isRunning

    public boolean isRunning()

### requestCrawlResume

    public void requestCrawlResume()

  Resume crawl from paused state

### getActiveToeCount

    public int getActiveToeCount()

  Returns:
    Active toe thread count.

### getOrder

    public CrawlOrder getOrder()

  Returns:
    The order file instance.

### getServerCache

    public ServerCache getServerCache()

  Returns:
    The server cache instance.

### setOrder

    public void setOrder(CrawlOrder o)

  Parameters:
    `o` -

### getFrontier

    public Frontier getFrontier()

  Returns:
    The frontier.

### getScope

    public CrawlScope getScope()

  Returns:
    This crawl scope.

### getProcessorChainList

    public ProcessorChainList getProcessorChainList()

  Get the list of processor chains.

  Returns:
    the list of processor chains.

### getFirstProcessorChain

    public ProcessorChain getFirstProcessorChain()

  Get the first processor chain.

  Returns:
    the first processor chain.

### getPostprocessorChain

    public ProcessorChain getPostprocessorChain()

  Get the postprocessor chain.

  Returns:
    the postprocessor chain.

### getDisk

    public java.io.File getDisk()

  Get the 'working' directory of the current crawl.

  Returns:
    the 'working' directory of the current crawl.

### getScratchDisk

    public java.io.File getScratchDisk()

  Returns:
    Scratch disk location.

### getStateDisk

    public java.io.File getStateDisk()

  Returns:
    State disk location.

### getToeCount

    public int getToeCount()

  Returns:
    The number of ToeThreads
  See Also:
    `ToePool.getToeCount()`

### getToePool

    public ToePool getToePool()

  Returns:
    The ToePool

### oneLineReportThreads

    public java.lang.String oneLineReportThreads()

  Returns:
    toepool one-line report

### kickUpdate

    public void kickUpdate()

  While many settings will update automatically when the SettingsHandler is modified, some settings need to be explicitly changed to reflect new settings. This includes, number of toe threads and seeds.

### getSettingsHandler

    public SettingsHandler getSettingsHandler()

  Returns:
    The settings handler.

### killThread

    public void killThread(int threadNumber,
                           boolean replace)

  Kills a thread. For details see `ToePool.killThread(int, boolean)`.

  Parameters:
    `threadNumber` - Thread to kill.
    `replace` - Should thread be replaced.
  See Also:
    `ToePool.killThread(int, boolean)`

### addToManifest

    public void addToManifest(java.lang.String file,
                              char type,
                              boolean bundle)

  Add a file to the manifest of files used/generated by the current crawl. TODO: Its possible for a file to be added twice if reports are force generated midcrawl. Fix.

  Parameters:
    `file` - The filename (with absolute path) of the file to add
    `type` - The type of the file
    `bundle` - Should the file be included in a typical bundling of crawler files.
  See Also:
    `MANIFEST_CONFIG_FILE`, `MANIFEST_LOG_FILE`, `MANIFEST_REPORT_FILE`

### checkFinish

    public void checkFinish()

  Evaluate if the crawl should stop because it is finished.

### atFinish

    public boolean atFinish()

  Evaluate if the crawl should stop because it is finished, without actually stopping the crawl.

  Returns:
    true if crawl is at a finish-possible state

### singleThreadMode

    public void singleThreadMode()

  Go to single thread mode, where only one ToeThread may proceed at a time. Also acquires the single lock, so no further threads will proceed past an acquireContinuePermission. Caller mush be sure to release lock to allow other threads to proceed one at a time.

### multiThreadMode

    public void multiThreadMode()

  Go to back to regular multi thread mode, where all ToeThreads may proceed at once

### acquireContinuePermission

    public void acquireContinuePermission()

  Proceed only if allowed, giving CrawlController a chance to enforce single-thread mode.

### releaseContinuePermission

    public void releaseContinuePermission()

  Relinquish continue permission at end of processing (allowing another thread to proceed if in single-thread mode).

### freeReserveMemory

    public void freeReserveMemory()

### toePaused

    public void toePaused()

  Note that a ToeThread reached paused condition, possibly completing the crawl-pause.

### toeEnded

    public void toeEnded()

  Note that a ToeThread ended, possibly completing the crawl-stop.

### addOrderToManifest

    public void addOrderToManifest()

  Add order file contents to manifest. Write configuration files and any files managed by CrawlController to it - files managed by other classes, excluding the settings framework, are responsible for adding their files to the manifest themselves. by calling addToManifest. Call before writing out reports.

### logUriError

    public void logUriError(org.apache.commons.httpclient.URIException e,
                            UURI u,
                            java.lang.CharSequence l)

  Log a URIException from deep inside other components to the crawl's shared log.

  Parameters:
    `e` - URIException encountered
    `u` - CrawlURI where problem occurred
    `l` - String which could not be interpreted as URI without exception

### getReports

    public java.lang.String[] getReports()

  Description copied from interface: `Reporter`
  Get an array of report names offered by this Reporter. A name in brackets indicates a free-form String, in accordance with the informal description inside the brackets, may yield a useful report.

  Specified by:
    `getReports` in interface `Reporter`
  Returns:
    String array of report names, empty if there is only one report type

### reportTo

    public void reportTo(java.io.PrintWriter writer)

  Description copied from interface: `Reporter`
  Make a default report to the passed-in Writer. Should be equivalent to reportTo(null, writer)

  Specified by:
    `reportTo` in interface `Reporter`
  Parameters:
    `writer` - to receive report

### singleLineReport

    public java.lang.String singleLineReport()

  Description copied from interface: `Reporter`
  Return a short single-line summary report as a String.

  Specified by:
    `singleLineReport` in interface `Reporter`
  Returns:
    String single-line summary report

### reportTo

    public void reportTo(java.lang.String name,
                         java.io.PrintWriter writer)

  Description copied from interface: `Reporter`
  Make a report of the given name to the passed-in Writer, If null, give the default report.

  Specified by:
    `reportTo` in interface `Reporter`
    `writer` - to receive report

### reportManifestTo

    protected void reportManifestTo(java.io.PrintWriter writer)

  Parameters:
    `writer` - Where to write report to.

### reportProcessorsTo

    protected void reportProcessorsTo(java.io.PrintWriter writer)

  Compiles and returns a human readable report on the active processors.

  Parameters:
    `writer` - Where to write to.
  See Also:
    `Processor.report()`

### singleLineReportTo

    public void singleLineReportTo(java.io.PrintWriter writer)

  Description copied from interface: `Reporter`
  Make a single-line summary report to the passed-in writer

  Specified by:
    `singleLineReportTo` in interface `Reporter`
  Parameters:
    `writer` - to receive report

### singleLineLegend

    public java.lang.String singleLineLegend()

  Description copied from interface: `Reporter`
  Return a legend for the single-line summary report as a String.

  Specified by:
    `singleLineLegend` in interface `Reporter`
  Returns:
    String single-line summary legend

### getBigMap

    public <K,V> java.util.Map<K,V> getBigMap(java.lang.String dbName,
                                              java.lang.Class<? super K> keyClass,
                                              java.lang.Class<? super V> valueClass)
                                 throws java.lang.Exception

  Call this method to get instance of the crawler BigMap implementation. A "BigMap" is a Map that knows how to manage ever-growing sets of key/value pairs. If we're in a checkpoint recovery, this method will manage reinstantiation of checkpointed bigmaps.

  Parameters:
    `dbName` - Name to give any associated database. Also used as part of name serializing out bigmap. Needs to be unique to a crawl.
    `keyClass` - Class of keys we'll be using.
    `valueClass` - Class of values we'll be using.
  Returns:
    Map that knows how to carry large sets of key/value pairs or if none available, returns instance of HashMap.
  Throws:
    `java.lang.Exception`

### checkpointBigMaps

    protected void checkpointBigMaps(java.io.File cpDir)
                              throws java.lang.Exception

  Throws:
    `java.lang.Exception`

### progressStatisticsEvent

    public void progressStatisticsEvent(java.util.EventObject e)

  Called whenever progress statistics logging event.

  Parameters:
    `e` - Progress statistics event.

### logProgressStatistics

    public void logProgressStatistics(java.lang.String msg)

  Log to the progress statistics log.

  Parameters:
    `msg` - Message to write the progress statistics log.

### getState

    public java.lang.Object getState()

  Returns:
    CrawlController state.

### getCheckpointsDisk

    public java.io.File getCheckpointsDisk()

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.