##### Overview

- License
- System Requirements
- Downloads
- User Manual
- Developer Manual
- Javadocs
- FAQ
- Wiki
- Browse/Submit a Bug
- Related Projects
  - Archive Access
  - Heritrix Cluster Controller (hcc)
  - cmdline-jmxclient
  - Deduplicator
  - Hadoop DFS Writer Processor

##### Project Documentation

- About Heritrix
- Project Info
- Project Reports
- Development Process

## Introduction

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

## Webmasters!

Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.

If you notice our crawler behaving poorly -- The Internet Archive uses archive.org_bot as User Agent when crawling -- please send us email at:

## Getting Started

See the User Manual.

## News and Status

### Release 2.0.0 official 2008-02-20

The official Heritrix 2.0.0 release is now available.

Release notes, with instructions to download and install

Download from Sourceforge files area

Four notable differences in Heritrix 2 are:

(1) A more rigorous separation of the Web UI from the 'crawl engine', giving greater flexibility to control crawlers remotely.

(2) A new settings system, easing module development and offering new opportunities for dynamic configuration construction.

(3) A new mechanism for custom override settings for sets of related URIs, extending beyond Heritrix 1.x's domain-centric overrides.

(4) A new system for ordering URIs within a single URI-queue, and for allocating frontier effort among different URI-queues, based on assigned integer 'precedence' values.

A tutorial of starting a basic crawl in the changed web UI is available.

Documentation for 2.0 is still limited but will gradually improve on the project wiki. (There are now direct help links from individual web UI settings to actual or potential wiki pages.)

Questions and discussion are as always welcome on the project discussion list.

Bugs and enhancement requests may be entered in the JIRA project issue tracker.

### Release 1.12.1 05/06/2007

Release 1.12.1 is a bug fix release. See the Release Notes and list of fixed issues for details.

Additional notes about 1.12.1 which may be updated with information post-release are available on the Heritrix wiki.

### Release 1.12.0 03/16/2007

Release 1.12.0 is the first of several planned releases enhancing Heritrix with "smart crawler" functionality. In this release, the theme has been offering new options to reduce the amount of duplicate content crawled and stored when recrawling sites at regular intervals. A number of other enhancements and bug fixes are also included. See the Release Notes for details.

### HDFS Writer Processor 01/25/2007

The HDFS Writer Processor extension enables storing crawled content directly into HDFS, the Hadoop Distributed FileSystem. For details, see the README.txt. To download, see HDFS Writer Processor

### Release 1.10.2 01/15/2007

This is primarily a bug-fix release, with a couple of new features, provided before a number of significant changes to the Heritrix project that will require developer and crawl operator adjustments. Post-1.10.2, Heritrix source code control, issue tracking, and build process will migrate to new systems. Also, updates to core classes, especially with regard to the settings architecture, will noticeably break backward compatibility with 1.10.2 and prior crawler settings files and formats. See Release Notes for details.

### Release 1.10.1 09/27/2006

Bug fix release. See Release Notes for detail.

### Deduplicator (add-on for Heritrix) 0.2.0 release 09/14/2006

The Deduplicator is a add-on module for Heritrix that allows sequential snapshot crawls to leverage information about previous iterations to avoid storing (or even downloading) duplicate data. See the mailing list announcement for details.

### Release 1.10.0 09/11/2006

Release 1.10.0 adds new configuration options, experimental new protocol and format support, and lots of fixes (43 tracked bugs have been fixed and 35 feature requests added). Requires JDK 1.5.x. See Release Notes for detail.

### Release 1.8.0 05/05/2006

Release 1.8.0 offers a number of improvements, including 13 requested enhancements and fixes for 18 reported bugs. See Heritrix Release Notes for detail and Known Limitations.

### Release 1.6.0 12/01/2005

Release 1.6.0 offers improved remote control and monitoring via JMX, a crawl-checkpointing facility, and experimental support for bloom filter already-included testing, partitioning a crawl across multiple independent crawlers, and per-host/domain/queue-grouping collection quotas. Performance and stability in large crawls is also improved. Among tracked issues, it includes 39 requested enhancements and fixes 96 reported bugs. See Heritrix Release Notes for detail and Known Limitations: e.g. Again you will need to tweak your old order files to make them work with the new release.

### Release 1.4.0 04/28/2005

Much improved memory usage, new experimental scoping/filter model, and a new revisiting frontier. Over 90 bugs fixed. See Heritrix Release Notes for detail and Known Limitations: e.g. You cannot use your old order files with the new release.

### Release 1.2.0 11/16/2004

Added IP-based politeness, configurable URI-canonicalization, and mid-fetch abort. Lots of Bug fixes. See Heritrix Release Notes for detail and Known Limitations (In particular, https fetching requires SUN JDK and UI throws OOME if jobs run in series).

### Release 1.0.4 09/22/2004

Bug fix. Crawl.log and ARC metadata lines could have whitespace in URIs and mimetype fields. See Heritrix Release Notes for detail and Known Limitations.

### Release 1.0.2 09/14/2004

Bug fixes. See Heritrix Release Notes for detail and known limitations.

### Release 1.0.0 08/06/2004

Added new prefix ('SURT') scope and filter, compression of recovery log, mass adding of URIs to running crawler, crawling via a http proxy, adding of headers to request, improved out-of-the-box defaults, hash of content to crawl log and to arcreader output, and many bug fixes. See Heritrix Release Notes for detail and known limitations.

### 1.0.0 first release candidate, 0.10.0 06/04/2004

Release for second heritrix workshop, Copenhagen 06/2004 (1.0.0 first release candidate). Added site-first prioritization, fixed link extraction of multibyte URIs, added metadata to arcs as xml, changed arc naming template, new user and developer manuals, added basic/digest auth and http post/get login facility, and added help to UI. Bug fixes. See Heritrix Release Notes for detail and known limitations.

### Release 0.8.1 05/28/2004

Fixes to build with maven rc2+.

### Release 0.8.0 05/24/2004

Release (and branch heritrix-0_8 made at the heritrix-0_7_1 tag) because of concurrentmodificationexceptions if tens of seeds supplied and to fix domain-scope leakage. Also, made continuous build publically available, incorporated integration selftest into build, made it a maven-build only (ant-build no longer supported), added day/night configurations (refinements), ameliorated too-many-open files, added exploit of http-header content-type charset creating character streams, and heritrix now crawls ssl sites. UI improvements include red start by bad configuration, precompilation, and delineation of advanced settings. See Heritrix Release Notes for detail.

### Release 0.6.0 03/25/2004

Release made in advance of radical frontier changes. Added bandwidth throttle, operator 'diary', settable robots expiration, crawler cookie pre-population, and changing of certain options mid-crawl. Many UI improvements including UI display of critical exceptions, UI desccription of job-order options, and improved reporting. Optimizations. Updated httpclient lib to 2.0 release and jmx libs to 1.2.1. See Heritrix Release Notes for detail.

### Point Release 0.4.1 02/12/2004

Released heritrix-0.4.1 to fix URIRegExpFilter retains memory.

### Release 0.4.0 02/10/2004

Release made for heritrix workshop, San Francisco, 02/2004. New MBEAN-based configuration, extensive UI revamp, first unit tests and integration selftest framework added, pooling of ARCWriters, new cmd-line start scripts, httpclient lib update (2.0RC3) and bugfixes. See Heritrix Release Notes for detail.

### First Release 01/05/2004

Today we made our first 'official' heritrix release, heritrix-0.2.0.