Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.archive.crawler
Class Heritrix

    java.lang.Object
      org.archive.crawler.Heritrix

All Implemented Interfaces:
  javax.management.DynamicMBean, javax.management.MBeanRegistration

    public class Heritrix

extends java.lang.Object
implements javax.management.DynamicMBean, javax.management.MBeanRegistration

Main class for Heritrix crawler. Heritrix is usually launched by a shell script that backgrounds heritrix that redirects all stdout and stderr emitted by heritrix to a log file. So that startup messages emitted subsequent to the redirection of stdout and stderr show on the console, this class prints usage or startup output such as where the web UI can be found, etc., to a STARTLOG that the shell script is waiting on. As soon as the shell script sees output in this file, it prints its content and breaks out of its wait. See ${HERITRIX_HOME}/bin/heritrix.

Heritrix can also be embedded or launched by webapp initialization or by JMX bootstrapping. So far I count 4 methods of instantiation:

1. From this classes main -- the method usually used;
2. From the Heritrix UI (The local-instances.jsp) page;
3. A creation by a JMX agent at the behest of a remote JMX client; and
4. A container such as tomcat or jboss.

Author:
  gojomo, Kristinn Sigurdsson, Stack

Field Summary
`static java.lang.String` | `DEFAULT_ENCODING` Default encoding.

Constructor Summary
`Heritrix()` Constructor.
`Heritrix(boolean jmxregister)`
`Heritrix(java.lang.String name, boolean jmxregister)` Constructor.
`Heritrix(java.lang.String name, boolean jmxregister, CrawlJobHandler cjh)` Constructor.

Method Summary
`protected CrawlJob` | `addCrawlJob(CrawlJob job)`
`protected java.lang.String` | `addCrawlJob(java.io.File order, java.lang.String name, java.lang.String description, java.lang.String seeds)`
`java.lang.String` | `addCrawlJob(java.lang.String orderPathOrUrl, java.lang.String name, java.lang.String description, java.lang.String seeds)` This method is called when we have an order file to hand that we want to base a job on.
`protected java.lang.String` | `addCrawlJob(java.net.URL url, java.net.HttpURLConnection connection, java.lang.String name, java.lang.String description, java.lang.String seeds)`
`protected CrawlJob` | `addCrawlJobBasedOn(java.io.File orderFile, java.lang.String name, java.lang.String description, java.lang.String seeds)`
`java.lang.String` | `addCrawlJobBasedOn(java.lang.String jobUidOrProfile, java.lang.String name, java.lang.String description, java.lang.String seeds)`
`protected java.lang.String` | `addCrawlJobBasedonJar(java.io.File jarFile, java.lang.String name, java.lang.String description, java.lang.String seeds)` Undo jar file and use as basis for a new job.
`protected static javax.management.ObjectName` | `addGuiPort(javax.management.ObjectName name)`
`protected static javax.management.ObjectName` | `addVitals(javax.management.ObjectName name)` Add vital stats to passed in ObjectName.
`protected javax.management.openmbean.OpenMBeanInfoSupport` | `buildMBeanInfo()` Build up the MBean info for Heritrix main.
`protected java.lang.String` | `checkForEmptyPlaceHolder(java.lang.String str)` If passed str has placeholder for the empty string, return the empty string else return orginal.
`protected static void` | `configureTrustStore()` Configure our trust store.
`protected static void` | `containerInitialization()` Run setup tasks for this 'container'.
`protected static void` | `copyToSystemProperty(java.lang.String key, java.lang.String value)` Copy the given key-value into System properties, as long as there is no existing value.
`protected static CrawlJob` | `createCrawlJob(CrawlJobHandler handler, java.io.File crawlOrderFile, java.lang.String name)`
`protected CrawlJob` | `createCrawlJobBasedOn(java.io.File orderFile, java.lang.String name, java.lang.String description, java.lang.String seeds)`
`protected static void` | `deregisterJndi(javax.management.ObjectName name)`
`void` | `destroy()` Do inverse of construction.
`protected static java.lang.String` | `doCmdLineArgs(java.lang.String[] args)`
`protected java.lang.String` | `doOneCrawl(java.lang.String crawlOrderFile)` Launch the crawler without a web UI and run the passed crawl only.
`protected java.lang.String` | `doOneCrawl(java.lang.String crawlOrderFile, CrawlStatusListener listener)` Launch the crawler without a web UI and run passed crawl only.
`SinkHandlerLogRecord` | `getAlert(java.lang.String id)`
`java.util.Vector` | `getAlerts()`
`int` | `getAlertsCount()`
`java.lang.Object` | `getAttribute(java.lang.String attribute_name)`
`javax.management.AttributeList` | `getAttributes(java.lang.String[] attributeNames)`
`static java.io.File` | `getConfdir()` Get the configuration directory.
`static java.io.File` | `getConfdir(boolean fail)` Get the configuration directory.
`protected java.lang.String` | `getCrawlendReport(java.lang.String jobUid, java.lang.String reportName)` Return named crawl end report for job with passed uid.
`protected static java.io.File` | `getHeritrixHome()` Exploit `-Dheritrix.home` if available to us.
`static java.lang.String` | `getHeritrixOut()`
`static SimpleHttpServer` | `getHttpServer()`
`static java.util.Map` | `getInstances()`
`static javax.management.ObjectName` | `getJmxObjectName()`
`static javax.management.ObjectName` | `getJmxObjectName(java.lang.String name)`
`static javax.management.ObjectName` | `getJmxObjectName(java.lang.String name, java.lang.String type)`
`protected static javax.management.ObjectName` | `getJndiContainerName()`
`protected static javax.naming.Context` | `getJndiContext()`
`CrawlJobHandler` | `getJobHandler()` Get the job handler
`static java.io.File` | `getJobsdir()`
`javax.management.MBeanInfo` | `getMBeanInfo()`
`javax.management.ObjectName` | `getMBeanName()`
`static javax.management.MBeanServer` | `getMBeanServer()` Get MBeanServer.
`java.util.Vector` | `getNewAlerts()`
`int` | `getNewAlertsCount()`
`protected java.lang.String` | `getNoJmxName()`
`protected static java.io.InputStream` | `getPropertiesInputStream()`
`protected static java.lang.Thread` | `getShutdownThread(boolean sysexit, int exitCode, java.lang.String name)`
`static Heritrix` | `getSingleInstance()`
`java.lang.String` | `getStatus()`
`protected static java.io.File` | `getSubDir(java.lang.String subdirName)` Get and check for existence of expected subdir.
`protected static java.io.File` | `getSubDir(java.lang.String subdirName, boolean fail)` Get and optionally check for existence of subdir.
`static java.lang.String` | `getVersion()` Get the heritrix version.
`static java.io.File` | `getWarsdir()`
`java.lang.String` | `interrupt(java.lang.String threadName)`
`java.lang.Object` | `invoke(java.lang.String operationName, java.lang.Object[] params, java.lang.String[] signature)`
`static boolean` | `isCommandLine()`
`protected static boolean` | `isDevelopment()`
`static boolean` | `isSingleInstance()`
`boolean` | `isStarted()`
`protected static boolean` | `isValidLoginPasswordString(java.lang.String str)` Test string is valid login/password string.
`java.lang.String` | `launch()` Launch the crawler for a web UI.
`java.lang.String` | `launch(java.lang.String crawlOrderFile, boolean runMode)` Launch the crawler for a web UI.
`protected static java.util.Properties` | `loadProperties()` Load the heritrix.properties file.
`static void` | `main(java.lang.String[] args)` Launch program.
`protected javax.management.openmbean.TabularData` | `makeJobsTabularData(java.util.List jobs)`
`protected static void` | `patchLogging()` If the user hasn't altered the default logging parameters, tighten them up somewhat: some of our libraries are way too verbose at the INFO or WARNING levels.
`static void` | `performHeritrixShutDown()` Exit program.
`static void` | `performHeritrixShutDown(int exitCode)` Exit program.
`void` | `postDeregister()`
`void` | `postRegister(java.lang.Boolean registrationDone)`
`void` | `preDeregister()`
`static void` | `prepareHeritrixShutDown()` Prepars for program shutdown.
`javax.management.ObjectName` | `preRegister(javax.management.MBeanServer server, javax.management.ObjectName name)`
`void` | `readAlert(java.lang.String id)`
`protected static void` | `registerContainerJndi()`
`protected static void` | `registerHeritrix(Heritrix h, java.lang.String name, boolean jmxregister)` Register Heritrix with JNDI, JMX, and with the static hashtable of all Heritrix instances known to this JVM.
`protected static void` | `registerJndi(javax.management.ObjectName name)`
`static javax.management.MBeanServer` | `registerMBean(javax.management.MBeanServer server, java.lang.Object objToRegister, javax.management.ObjectName objName)`
`static javax.management.MBeanServer` | `registerMBean(javax.management.MBeanServer server, java.lang.Object objToRegister, java.lang.String name, java.lang.String type)`
`static javax.management.MBeanServer` | `registerMBean(java.lang.Object objToRegister, java.lang.String name, java.lang.String type)`
`void` | `removeAlert(java.lang.String id)`
`static void` | `resetAuthentication(java.lang.String newUsername, java.lang.String newPassword)` Replace existing administrator login info with new info.
`protected static java.lang.String` | `selftest(java.lang.String oneSelfTestName, int port)` Run the selftest
`void` | `setAttribute(javax.management.Attribute attribute)`
`javax.management.AttributeList` | `setAttributes(javax.management.AttributeList attributes)`
`static void` | `shutdown()`
`static void` | `shutdown(int exitCode)` Shutdown all running heritrix instances and the JVM.
`void` | `start()` Start Heritrix.
`void` | `startCrawling()`
`protected static java.lang.String` | `startEmbeddedWebserver(java.util.Collection<java.lang.String> hosts, int port, java.lang.String adminLoginPassword)` Start up the embedded Jetty webserver instance.
`protected static java.lang.String` | `startEmbeddedWebserver(int port, boolean lho, java.lang.String adminLoginPassword)` Deprecated. Use startEmbeddedWebserver(hosts, port, adminLoginPassword)
`void` | `stop()` Stop Heritrix.
`void` | `stopCrawling()`
`protected static void` | `unregisterHeritrix(Heritrix h)`
`static void` | `unregisterMBean(javax.management.MBeanServer server, javax.management.ObjectName name)`
`static void` | `unregisterMBean(javax.management.MBeanServer server, java.lang.String name, java.lang.String type)`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### DEFAULT_ENCODING

    public static final java.lang.String DEFAULT_ENCODING

  Default encoding. Used for content when fetching if none specified.

  See Also:
    Constant Field Values

Constructor Detail

### Heritrix

    public Heritrix()
             throws java.io.IOException

  Constructor. Does not register the created instance with JMX. Assumed this constructor is used by such as JMX agent creating an instance of Heritrix at the commmand of a remote client (In this case Heritrix will be registered by the invoking agent).

  Throws:
    `java.io.IOException`

### Heritrix

    public Heritrix(boolean jmxregister)
             throws java.io.IOException

Throws:
  `java.io.IOException`

### Heritrix

    public Heritrix(java.lang.String name,
                    boolean jmxregister)
             throws java.io.IOException

  Constructor.

  Parameters:
    `name` - If null, we bring up the default Heritrix instance.
    `jmxregister` - True if we are to register this instance with JMX agent.
  Throws:
    `java.io.IOException`

### Heritrix

    public Heritrix(java.lang.String name,
                    boolean jmxregister,
                    CrawlJobHandler cjh)
             throws java.io.IOException

  Constructor.

  Parameters:
    `name` - If null, we bring up the default Heritrix instance.
    `jmxregister` - True if we are to register this instance with JMX agent.
    `cjh` - CrawlJobHandler to use.
  Throws:
    `java.io.IOException`

Method Detail

### containerInitialization

    protected static void containerInitialization()
                                           throws java.io.IOException

  Run setup tasks for this 'container'. Idempotent.

  Throws:
    `java.io.IOException`

### destroy

    public void destroy()

  Do inverse of construction. Used by anyone who does a 'new Heritrix' when they want to cleanup the instance. Of note, there may be Heritrix threads still hanging around after the call to destroy completes. They'll eventually go down after they've finished their cleanup routines. In particular, if you are watching Heritrix via JMX, you can see the Heritrix instance JMX bean unregister ahead of the CrawlJob JMX bean that its hosting.

### main

    public static void main(java.lang.String[] args)
                     throws java.lang.Exception

  Launch program. Optionally will launch a web server to host UI. Will also register Heritrix MBean with first found JMX Agent (Usually the 1.5.0 JVM Agent).

  Parameters:
    `args` - Command line arguments.
  Throws:
    `java.lang.Exception`

### doCmdLineArgs

    protected static java.lang.String doCmdLineArgs(java.lang.String[] args)
                                             throws java.lang.Exception

  Throws:
    `java.lang.Exception`

### getHeritrixOut

    public static java.lang.String getHeritrixOut()

  Returns:
    The file we dump stdout and stderr into.

### getHeritrixHome

    protected static java.io.File getHeritrixHome()
                                           throws java.io.IOException

  Exploit `-Dheritrix.home` if available to us. Is current working dir if no heritrix.home property supplied.

  Returns:
    Heritrix home directory.
  Throws:
    `java.io.IOException`

### getJobsdir

    public static java.io.File getJobsdir()
                                   throws java.io.IOException

  Returns:
    The directory into which we put jobs. If the system property 'heritrix.jobsdir' is set, we will use its value in place of the default 'jobs' directory in the current working directory.
  Throws:
    `java.io.IOException`

### getSubDir

    protected static java.io.File getSubDir(java.lang.String subdirName)
                                     throws java.io.IOException

  Get and check for existence of expected subdir. If development flag set, then look for dir under src dir.

  Parameters:
    `subdirName` - Dir to look for.
  Returns:
    The extant subdir. Otherwise null if we're running in a webapp context where there is no conf directory available.
  Throws:
    `java.io.IOException` - if unable to find expected subdir.

### getSubDir

    protected static java.io.File getSubDir(java.lang.String subdirName,
                                            boolean fail)
                                     throws java.io.IOException

  Get and optionally check for existence of subdir. If development flag set, then look for dir under src dir.

  Parameters:
    `subdirName` - Dir to look for.
    `fail` - True if we are to fail if directory does not exist; false if we are to return false if the directory does not exist.
  Returns:
    The extant subdir. Otherwise null if we're running in a webapp context where there is no subdir directory available.
  Throws:
    `java.io.IOException` - if unable to find expected subdir.

### isValidLoginPasswordString

    protected static boolean isValidLoginPasswordString(java.lang.String str)

  Test string is valid login/password string. A valid login/password string has the login and password compounded w/ a ':' delimiter.

  Parameters:
    `str` - String to test.
  Returns:
    True if valid password/login string.

### isDevelopment

    protected static boolean isDevelopment()

### loadProperties

    protected static java.util.Properties loadProperties()
                                                  throws java.io.IOException

  Load the heritrix.properties file. Adds any property that starts with `HERITRIX_PROPERTIES_PREFIX` or `ARCHIVE_PACKAGE` into system properties (except logging '.level' directives).

  Returns:
    Loaded properties.
  Throws:
    `java.io.IOException`

### copyToSystemProperty

    protected static void copyToSystemProperty(java.lang.String key,
                                               java.lang.String value)

  Copy the given key-value into System properties, as long as there is no existing value.

  Parameters:
    `key` - property key
    `value` - property value

### getPropertiesInputStream

    protected static java.io.InputStream getPropertiesInputStream()
                                                           throws java.io.IOException

  Throws:
    `java.io.IOException`

### patchLogging

    protected static void patchLogging()
                                throws java.lang.SecurityException,
                                       java.io.IOException

  If the user hasn't altered the default logging parameters, tighten them up somewhat: some of our libraries are way too verbose at the INFO or WARNING levels. This might be a problem running inside in someone else's container. Container's seem to prefer commons logging so we ain't messing them doing the below.

  Throws:
    `java.io.IOException`
    `java.lang.SecurityException`

### configureTrustStore

    protected static void configureTrustStore()

  Configure our trust store. If system property is defined, then use it for our truststore. Otherwise use the heritrix truststore under conf directory if it exists.

  If we're not launched from the command-line, we will not be able to find our truststore. The truststore is nor normally used so rare should this be a problem (In case where we don't use find our trust store, we'll use the 'default' -- either the JVMs or the containers).

### selftest

    protected static java.lang.String selftest(java.lang.String oneSelfTestName,
                                               int port)
                                        throws java.lang.Exception

  Run the selftest

  Parameters:
    `oneSelfTestName` - Name of a test if we are to run one only rather than the default running all tests.
    `port` - Port number to use for web UI.
  Returns:
    Status of how selftest startup went.
  Throws:
    `java.lang.Exception`

### doOneCrawl

    protected java.lang.String doOneCrawl(java.lang.String crawlOrderFile)
                                   throws InitializationException,
                                          javax.management.InvalidAttributeValueException

  Launch the crawler without a web UI and run the passed crawl only. Specialized version of `launch()`.

  Parameters:
    `crawlOrderFile` - The crawl order to crawl.
  Returns:
    Status string.
  Throws:
    `InitializationException`
    `javax.management.InvalidAttributeValueException`

### doOneCrawl

    protected java.lang.String doOneCrawl(java.lang.String crawlOrderFile,
                                          CrawlStatusListener listener)
                                   throws InitializationException,
                                          javax.management.InvalidAttributeValueException

  Launch the crawler without a web UI and run passed crawl only. Specialized version of `launch()`.

  Parameters:
    `crawlOrderFile` - The crawl order to crawl.
    `listener` - Register this crawl status listener before starting crawl (You can use this listener to notice end-of-crawl).
  Returns:
    Status string.
  Throws:
    `InitializationException`
    `javax.management.InvalidAttributeValueException`

### launch

    public java.lang.String launch()
                            throws java.lang.Exception

  Launch the crawler for a web UI. Crawler hangs around waiting on jobs.

  Returns:
    A status string describing how the launch went.
  Throws:
    `java.lang.Exception`
    `java.lang.Exception`

### launch

    public java.lang.String launch(java.lang.String crawlOrderFile,
                                   boolean runMode)
                            throws java.lang.Exception

  Launch the crawler for a web UI. Crawler hangs around waiting on jobs.

  Parameters:
    `crawlOrderFile` - File to crawl. May be null.
    `runMode` - Whether crawler should be set to run mode.
  Returns:
    A status string describing how the launch went.
  Throws:
    `java.lang.Exception`

### startEmbeddedWebserver

    protected static java.lang.String startEmbeddedWebserver(int port,
                                                             boolean lho,
                                                             java.lang.String adminLoginPassword)
                                                      throws java.lang.Exception

  Deprecated. Use startEmbeddedWebserver(hosts, port, adminLoginPassword)

  Start up the embedded Jetty webserver instance. This is done when we're run from the command-line.

  Parameters:
    `port` - Port number to use for web UI.
    `adminLoginPassword` - Compound of login and password.
  Returns:
    Status on webserver startup.
  Throws:
    `java.lang.Exception`

### startEmbeddedWebserver

    protected static java.lang.String startEmbeddedWebserver(java.util.Collection<java.lang.String> hosts,
                                                             int port,
                                                             java.lang.String adminLoginPassword)
                                                      throws java.lang.Exception

  Start up the embedded Jetty webserver instance. This is done when we're run from the command-line.

  Parameters:
    `hosts` - a list of IP addresses or hostnames to bind to, or an empty collection to bind to all available network interfaces
    `port` - Port number to use for web UI.
    `adminLoginPassword` - Compound of login and password.
  Returns:
    Status on webserver startup.
  Throws:
    `java.lang.Exception`

### resetAuthentication

    public static void resetAuthentication(java.lang.String newUsername,
                                           java.lang.String newPassword)

  Replace existing administrator login info with new info.

  Parameters:
    `newUsername` - new administrator login username
    `newPassword` - new administrator login password

### createCrawlJob

    protected static CrawlJob createCrawlJob(CrawlJobHandler handler,
                                             java.io.File crawlOrderFile,
                                             java.lang.String name)
                                      throws javax.management.InvalidAttributeValueException

  Throws:
    `javax.management.InvalidAttributeValueException`

### addCrawlJob

    public java.lang.String addCrawlJob(java.lang.String orderPathOrUrl,
                                        java.lang.String name,
                                        java.lang.String description,
                                        java.lang.String seeds)
                                 throws java.io.IOException,
                                        FatalConfigurationException

  This method is called when we have an order file to hand that we want to base a job on. It leaves the order file in place and just starts up a job that uses all the order points to for locations for logs, etc.

  Parameters:
    `orderPathOrUrl` - Path to an order file or to a seeds file.
    `name` - Name to use for this job.
    `description` -
    `seeds` -
  Returns:
    A status string.
  Throws:
    `java.io.IOException`
    `FatalConfigurationException`

### addCrawlJob

    protected java.lang.String addCrawlJob(java.net.URL url,
                                           java.net.HttpURLConnection connection,
                                           java.lang.String name,
                                           java.lang.String description,
                                           java.lang.String seeds)
                                    throws java.io.IOException,
                                           FatalConfigurationException

  Throws:
    `java.io.IOException`
    `FatalConfigurationException`

### addCrawlJob

    protected java.lang.String addCrawlJob(java.io.File order,
                                           java.lang.String name,
                                           java.lang.String description,
                                           java.lang.String seeds)
                                    throws FatalConfigurationException,
                                           java.io.IOException

  Throws:
    `FatalConfigurationException`
    `java.io.IOException`

### addCrawlJobBasedonJar

    protected java.lang.String addCrawlJobBasedonJar(java.io.File jarFile,
                                                     java.lang.String name,
                                                     java.lang.String description,
                                                     java.lang.String seeds)
                                              throws java.io.IOException,
                                                     FatalConfigurationException

  Undo jar file and use as basis for a new job.

  Parameters:
    `jarFile` - Pointer to file that holds jar.
    `name` - Name to use for new job.
    `description` -
    `seeds` -
  Returns:
    Message.
  Throws:
    `java.io.IOException`
    `FatalConfigurationException`

### addCrawlJobBasedOn

    public java.lang.String addCrawlJobBasedOn(java.lang.String jobUidOrProfile,
                                               java.lang.String name,
                                               java.lang.String description,
                                               java.lang.String seeds)

### addCrawlJobBasedOn

    protected CrawlJob addCrawlJobBasedOn(java.io.File orderFile,
                                          java.lang.String name,
                                          java.lang.String description,
                                          java.lang.String seeds)
                                   throws FatalConfigurationException

  Throws:
    `FatalConfigurationException`

### createCrawlJobBasedOn

    protected CrawlJob createCrawlJobBasedOn(java.io.File orderFile,
                                             java.lang.String name,
                                             java.lang.String description,
                                             java.lang.String seeds)
                                      throws FatalConfigurationException

  Throws:
    `FatalConfigurationException`

### addCrawlJob

    protected CrawlJob addCrawlJob(CrawlJob job)

### startCrawling

    public void startCrawling()

### stopCrawling

    public void stopCrawling()

### getVersion

    public static java.lang.String getVersion()

  Get the heritrix version.

  Returns:
    The heritrix version. May be null.

### getJobHandler

    public CrawlJobHandler getJobHandler()

  Get the job handler

  Returns:
    The CrawlJobHandler being used.

### getConfdir

    public static java.io.File getConfdir()
                                   throws java.io.IOException

  Get the configuration directory.

  Returns:
    The conf directory under HERITRIX_HOME or null if none can be found.
  Throws:
    `java.io.IOException`

### getConfdir

    public static java.io.File getConfdir(boolean fail)
                                   throws java.io.IOException

  Get the configuration directory.

  Parameters:
    `fail` - Throw IOE if can't find directory if true, else just return null.
  Returns:
    The conf directory under HERITRIX_HOME or null (or an IOE) if can't be found.
  Throws:
    `java.io.IOException`

### getHttpServer

    public static SimpleHttpServer getHttpServer()

  Returns:
    Returns the httpServer. May be null if one was not started.

### getWarsdir

    public static java.io.File getWarsdir()
                                   throws java.io.IOException

  Returns:
    Returns the directory under which reside the WAR files we're to load into the servlet container.
  Throws:
    `java.io.IOException`

### prepareHeritrixShutDown

    public static void prepareHeritrixShutDown()

  Prepars for program shutdown. This method does it's best to prepare the program so that it can exit normally. It will kill the httpServer and terminate any running job.
  It is advisible to wait a few (~1000) millisec after calling this method and before calling performHeritrixShutDown() to allow as many threads as possible to finish what they are doing.

### performHeritrixShutDown

    public static void performHeritrixShutDown()

  Exit program. Recommended that prepareHeritrixShutDown() be invoked prior to this method.

### performHeritrixShutDown

    public static void performHeritrixShutDown(int exitCode)

  Exit program. Recommended that prepareHeritrixShutDown() be invoked prior to this method.

  Parameters:
    `exitCode` - Code to pass System.exit.

### shutdown

    public static void shutdown(int exitCode)

  Shutdown all running heritrix instances and the JVM. Assumes stop has already been called.

  Parameters:
    `exitCode` - Exit code to pass system exit.

### getShutdownThread

    protected static java.lang.Thread getShutdownThread(boolean sysexit,
                                                        int exitCode,
                                                        java.lang.String name)

### shutdown

    public static void shutdown()

### registerHeritrix

    protected static void registerHeritrix(Heritrix h,
                                           java.lang.String name,
                                           boolean jmxregister)
                                    throws javax.management.MalformedObjectNameException,
                                           javax.management.InstanceAlreadyExistsException,
                                           javax.management.MBeanRegistrationException,
                                           javax.management.NotCompliantMBeanException

  Register Heritrix with JNDI, JMX, and with the static hashtable of all Heritrix instances known to this JVM. If launched from cmdline, register Heritrix MBean if an agent to register ourselves with. Usually this method will only have effect if we're running in a 1.5.0 JDK and command line options such as '-Dcom.sun.management.jmxremote.port=8082 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false' are supplied. See Monitoring and Management Using JMX for more on the command line options and how to connect to the Heritrix bean using the JDK 1.5.0 jconsole tool. We register currently with first server we find (TODO: Make configurable).

  If we register successfully with a JMX agent, then part of the registration will include our registering ourselves with JNDI.

  Finally, add the heritrix instance to the hashtable of all the Heritrix instances floating in the current VM. This latter registeration happens whether or no there is a JMX agent to register with. This is a list we keep out of convenience so its easy iterating over all all instances calling stop when main application is going down.

  Parameters:
    `h` - Instance of heritrix to register.
    `name` - Name to use for this Heritrix instance.
    `jmxregister` - True if we are to register this instance with JMX.
  Throws:
    `java.lang.NullPointerException`
    `javax.management.MalformedObjectNameException`
    `javax.management.NotCompliantMBeanException`
    `javax.management.MBeanRegistrationException`
    `javax.management.InstanceAlreadyExistsException`

### unregisterHeritrix

    protected static void unregisterHeritrix(Heritrix h)
                                      throws javax.management.InstanceNotFoundException,
                                             javax.management.MBeanRegistrationException,
                                             java.lang.NullPointerException

  Throws:
    `javax.management.InstanceNotFoundException`
    `javax.management.MBeanRegistrationException`
    `java.lang.NullPointerException`

### getMBeanServer

    public static javax.management.MBeanServer getMBeanServer()

  Get MBeanServer. Currently uses first MBeanServer found. This will definetly not be whats always wanted. TODO: Make which server settable. Also, if none, put up our own MBeanServer.

  Returns:
    An MBeanServer to register with or null.

### registerMBean

    public static javax.management.MBeanServer registerMBean(java.lang.Object objToRegister,
                                                             java.lang.String name,
                                                             java.lang.String type)
                                                      throws javax.management.InstanceAlreadyExistsException,
                                                             javax.management.MBeanRegistrationException,
                                                             javax.management.NotCompliantMBeanException

  Throws:
    `javax.management.InstanceAlreadyExistsException`
    `javax.management.MBeanRegistrationException`
    `javax.management.NotCompliantMBeanException`

### registerMBean

    public static javax.management.MBeanServer registerMBean(javax.management.MBeanServer server,
                                                             java.lang.Object objToRegister,
                                                             java.lang.String name,
                                                             java.lang.String type)
                                                      throws javax.management.InstanceAlreadyExistsException,
                                                             javax.management.MBeanRegistrationException,
                                                             javax.management.NotCompliantMBeanException

  Throws:
    `javax.management.InstanceAlreadyExistsException`
    `javax.management.MBeanRegistrationException`
    `javax.management.NotCompliantMBeanException`

### registerMBean

    public static javax.management.MBeanServer registerMBean(javax.management.MBeanServer server,
                                                             java.lang.Object objToRegister,
                                                             javax.management.ObjectName objName)
                                                      throws javax.management.InstanceAlreadyExistsException,
                                                             javax.management.MBeanRegistrationException,
                                                             javax.management.NotCompliantMBeanException

  Throws:
    `javax.management.InstanceAlreadyExistsException`
    `javax.management.MBeanRegistrationException`
    `javax.management.NotCompliantMBeanException`

### unregisterMBean

    public static void unregisterMBean(javax.management.MBeanServer server,
                                       java.lang.String name,
                                       java.lang.String type)

### unregisterMBean

    public static void unregisterMBean(javax.management.MBeanServer server,
                                       javax.management.ObjectName name)

### getNoJmxName

    protected java.lang.String getNoJmxName()

  Returns:
    Name to use when no JMX agent available.

### getJmxObjectName

    public static javax.management.ObjectName getJmxObjectName()
                                                        throws javax.management.MalformedObjectNameException,
                                                               java.lang.NullPointerException

  Throws:
    `javax.management.MalformedObjectNameException`
    `java.lang.NullPointerException`

### getJmxObjectName

    public static javax.management.ObjectName getJmxObjectName(java.lang.String name)
                                                        throws javax.management.MalformedObjectNameException,
                                                               java.lang.NullPointerException

  Throws:
    `javax.management.MalformedObjectNameException`
    `java.lang.NullPointerException`

### getJmxObjectName

    public static javax.management.ObjectName getJmxObjectName(java.lang.String name,
                                                               java.lang.String type)
                                                        throws javax.management.MalformedObjectNameException,
                                                               java.lang.NullPointerException

  Throws:
    `javax.management.MalformedObjectNameException`
    `java.lang.NullPointerException`

### isCommandLine

    public static boolean isCommandLine()

  Returns:
    Returns true if Heritrix was launched from the command line. (When launched from command line, we do stuff like put up a web server to manage our web interface and we register ourselves with the first available jmx agent).

### isStarted

    public boolean isStarted()

  Returns:
    True if heritrix has been started.

### getStatus

    public java.lang.String getStatus()

### getAlertsCount

    public int getAlertsCount()

### getNewAlertsCount

    public int getNewAlertsCount()

### getAlerts

    public java.util.Vector getAlerts()

### getNewAlerts

    public java.util.Vector getNewAlerts()

### getAlert

    public SinkHandlerLogRecord getAlert(java.lang.String id)

### readAlert

    public void readAlert(java.lang.String id)

### removeAlert

    public void removeAlert(java.lang.String id)

### start

    public void start()

  Start Heritrix. Used by JMX and webapp initialization for starting Heritrix. Not by the cmdline launched Heritrix. Idempotent. If start is called by JMX, then new instance of Heritrix is automatically registered w/ JMX Agent. If started by webapp, need to register the new Heritrix instance.

### stop

    public void stop()

  Stop Heritrix. Used by JMX and webapp initialization for stopping Heritrix.

### interrupt

    public java.lang.String interrupt(java.lang.String threadName)

### buildMBeanInfo

    protected javax.management.openmbean.OpenMBeanInfoSupport buildMBeanInfo()

  Build up the MBean info for Heritrix main.

  Returns:
    Return created mbean info instance.

### getAttribute

    public java.lang.Object getAttribute(java.lang.String attribute_name)
                                  throws javax.management.AttributeNotFoundException

  Specified by:
    `getAttribute` in interface `javax.management.DynamicMBean`
  Throws:
    `javax.management.AttributeNotFoundException`

### setAttribute

    public void setAttribute(javax.management.Attribute attribute)
                      throws javax.management.AttributeNotFoundException

  Specified by:
    `setAttribute` in interface `javax.management.DynamicMBean`
  Throws:
    `javax.management.AttributeNotFoundException`

### getAttributes

    public javax.management.AttributeList getAttributes(java.lang.String[] attributeNames)

  Specified by:
    `getAttributes` in interface `javax.management.DynamicMBean`

### setAttributes

    public javax.management.AttributeList setAttributes(javax.management.AttributeList attributes)

  Specified by:
    `setAttributes` in interface `javax.management.DynamicMBean`

### invoke

    public java.lang.Object invoke(java.lang.String operationName,
                                   java.lang.Object[] params,
                                   java.lang.String[] signature)
                            throws javax.management.ReflectionException

  Specified by:
    `invoke` in interface `javax.management.DynamicMBean`
  Throws:
    `javax.management.ReflectionException`

### getCrawlendReport

    protected java.lang.String getCrawlendReport(java.lang.String jobUid,
                                                 java.lang.String reportName)
                                          throws java.io.IOException

  Return named crawl end report for job with passed uid. Crawler makes reports when its finished its crawl. Use this method to get a String version of one of these files.

  Parameters:
    `jobUid` - The unique ID for the job whose reports you want to see (Must be a completed job).
    `reportName` - Name of report minus '.txt' (e.g. crawl-report).
  Returns:
    String version of the on-disk report.
  Throws:
    `java.io.IOException`

### makeJobsTabularData

    protected javax.management.openmbean.TabularData makeJobsTabularData(java.util.List jobs)
                                                                  throws javax.management.openmbean.OpenDataException

  Throws:
    `javax.management.openmbean.OpenDataException`

### checkForEmptyPlaceHolder

    protected java.lang.String checkForEmptyPlaceHolder(java.lang.String str)

  If passed str has placeholder for the empty string, return the empty string else return orginal. Dumb jmx clients can't pass empty string so they'll pass a representation of empty string such as ' ' or '-'. Convert such strings to empty string.

  Parameters:
    `str` - String to check.
  Returns:
    Original `str` or empty string if `str` contains a placeholder for the empty-string (e.g. '-', or ' ').

### getMBeanInfo

    public javax.management.MBeanInfo getMBeanInfo()

  Specified by:
    `getMBeanInfo` in interface `javax.management.DynamicMBean`

### getMBeanName

    public javax.management.ObjectName getMBeanName()

  Returns:
    Name this instance registered in JMX (Only available after JMX registration).

### preRegister

    public javax.management.ObjectName preRegister(javax.management.MBeanServer server,
                                                   javax.management.ObjectName name)
                                            throws java.lang.Exception

  Specified by:
    `preRegister` in interface `javax.management.MBeanRegistration`
  Throws:
    `java.lang.Exception`

### addVitals

    protected static javax.management.ObjectName addVitals(javax.management.ObjectName name)
                                                    throws java.net.UnknownHostException,
                                                           javax.management.MalformedObjectNameException,
                                                           java.lang.NullPointerException

  Add vital stats to passed in ObjectName.

  Parameters:
    `name` - ObjectName to add to.
  Returns:
    name with host, guiport, and jmxport added.
  Throws:
    `java.net.UnknownHostException`
    `javax.management.MalformedObjectNameException`
    `java.lang.NullPointerException`

### addGuiPort

    protected static javax.management.ObjectName addGuiPort(javax.management.ObjectName name)
                                                     throws javax.management.MalformedObjectNameException,
                                                            java.lang.NullPointerException

  Throws:
    `javax.management.MalformedObjectNameException`
    `java.lang.NullPointerException`

### postRegister

    public void postRegister(java.lang.Boolean registrationDone)

  Specified by:
    `postRegister` in interface `javax.management.MBeanRegistration`

### preDeregister

    public void preDeregister()
                       throws java.lang.Exception

  Specified by:
    `preDeregister` in interface `javax.management.MBeanRegistration`
  Throws:
    `java.lang.Exception`

### postDeregister

    public void postDeregister()

  Specified by:
    `postDeregister` in interface `javax.management.MBeanRegistration`

### registerContainerJndi

    protected static void registerContainerJndi()
                                         throws javax.management.MalformedObjectNameException,
                                                java.lang.NullPointerException,
                                                java.net.UnknownHostException,
                                                javax.naming.NamingException

  Throws:
    `javax.management.MalformedObjectNameException`
    `java.lang.NullPointerException`
    `java.net.UnknownHostException`
    `javax.naming.NamingException`

### registerJndi

    protected static void registerJndi(javax.management.ObjectName name)
                                throws java.lang.NullPointerException,
                                       javax.naming.NamingException

  Throws:
    `java.lang.NullPointerException`
    `javax.naming.NamingException`

### deregisterJndi

    protected static void deregisterJndi(javax.management.ObjectName name)
                                  throws java.lang.NullPointerException,
                                         javax.naming.NamingException

  Throws:
    `java.lang.NullPointerException`
    `javax.naming.NamingException`

### getJndiContext

    protected static javax.naming.Context getJndiContext()
                                                  throws javax.naming.NamingException

  Returns:
    Jndi context for the crawler or null if none found.
  Throws:
    `javax.naming.NamingException`

### getJndiContainerName

    protected static javax.management.ObjectName getJndiContainerName()
                                                               throws javax.management.MalformedObjectNameException,
                                                                      java.lang.NullPointerException,
                                                                      java.net.UnknownHostException

  Returns:
    Jndi container name -- the name to use for the 'container' that can host zero or more heritrix instances (Return a JMX ObjectName. We use ObjectName because then we're sync'd with JMX naming and ObjectName has nice parsing).
  Throws:
    `java.lang.NullPointerException`
    `javax.management.MalformedObjectNameException`
    `java.net.UnknownHostException`

### getInstances

    public static java.util.Map getInstances()

  Returns:
    Return all registered instances of Heritrix (Rare are there more than one).

### isSingleInstance

    public static boolean isSingleInstance()

  Returns:
    True if only one instance of Heritrix.

### getSingleInstance

    public static Heritrix getSingleInstance()

  Returns:
    Returns single instance or null if no instance or multiple.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2003-2008 Internet Archive. All Rights Reserved.