10. Release 1.2.0 - 2004-11-16
Prev |  | Next

## 10. Release 1.2.0 - 2004-11-16

Abstract

Added IP-based politeness, configurable URI-canonicalization, and mid-fetch abort. Lots of Bug fixes.

### 10.1. Known Limitations

#### 10.1.1. IBM JVM

The IBM JVM generally is more performant than SUN JVMs. It also emits more detailed heap dumps. That said, new Heritrix 1.2.0 features may not work on the IBM JVM.

##### 10.1.1.1. HTTPS

Heritrix 1.2.0 uses the new HttpClient 3.0x library which allows the setting of socket read timeouts. Connections to https sites fail if using the IBM JVM.

The IBM JVM 141 (cxia321411-20030930) NPEs setting the NoTcpDelay.

    java.lang.NullPointerException
       at com.ibm.jsse.bf.setTcpNoDelay(Unknown Source)
       at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:683)
       at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1328)

Using the IBM JVM 142, its saying SSL connection not open when we go to use inputstreams:

    java.net.SocketException: Socket is not connected
       at java.net.Socket.getInputStream(Socket.java:726)     at com.ibm.jsse.bs.getInputStream(Unknown Source)
       at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:715)
       at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1328)

Newer versions of the httpclient library may address this (Current version is alpha2).

#### 10.1.2. Jobs don't show in UI when a bunch are run back-to-back

If more than one job waiting in the queue of pending jobs, then the second job often won't show in the UI; The UI says its running but its not possible to see a status bar on the running job. See [ 1024120 ] Lost crawl job after terminate running job with jobs pending. For now, the workaround is to study the running job by viewing the crawl job logs on disk (Oddly, the 3rd queued up job will start to show in the UI again).

#### 10.1.3. Running more than one job in series throws OOME

OutOfMemoryExceptions are frequent when jobs are run in series. [ 1055592 ] terminated crawl still hogging memory, causing OOM. For now, restart Heritrix between the running of jobs.

### 10.2. Changes

Table 7. Changes

ID | Type | Summary | Open Date | By
--- | --- | --- | --- | ---
1067095 | Add | Hang in http fetcher when mid-fetch aborts | 2004-11-15 | stack-sf
1066804 | Add | Allow specification of heritrix_out.log filename | 2004-11-15 | stack-sf
903845 | Add | IP-based politeness | 2004-10-28 | gojomo
1054849 | Add | Recover from crawl initialized with a recovery log | 2004-10-26 | stack-sf
1054851 | Add | Import gzipped or non-gzipped recovery log | 2004-10-26 | stack-sf
1050378 | Add | Add bdb alreadyseen option to hostsqueuesfrontier | 2004-10-19 | stack-sf
973881 | Add | Force generation of report files | 2004-06-16 | stack-sf
1010883 | Add | Scripts to generate end-of-job reports | 2004-08-17 | danavery
988277 | Add | [Need feedback] "Done with ARC file" event | 2004-07-09 | stack-sf
1044977 | Add | Logging of scope-rejected URIs | 2004-10-11 | stack-sf
902970 | Add | HTTPClient should use supplied IP / avoid DNS lookup | 2004-02-23 | stack-sf
903093 | Add | Setting of Integer.MAX_VALUE is ugly | 2004-02-23 | stack-sf
900004 | Add | canonicalization of URIs for alreadyIncluded testing | 2004-02-18 | stack-sf
941072 | Add | Allow operator-configured mid-HTTP-fetch filters | 2004-04-23 | stack-sf
1037891 | Add | Cmdline defaults in properties file | 2004-09-30 | stack-sf
1037304 | Add | Upgrade httpclient to 3.0.x | 2004-09-29 | stack-sf
994141 | Add | Update build to use maven 1.0 | 2004-07-19 | stack-sf
1002336 | Add | Figure what profiler to use | 2004-08-02 | stack-sf
1064887 | Fix | http and https prerequisites contention | 2004-11-11 | stack-sf
1062604 | Fix | Seed to SURT coversion issuesI | 22004-11-08 | stack-sf
11061795 | Fix | ServerCache HashMaps access thread-safetyI | 22004-11-06 | gojomo
1060589 | Fix | Can't open logs of old jobs post-restart in UII | 22004-11-04 | stack-sf
1058565 | Fix | Non-default 'logs' location doesn't show in web UI | 2004-11-010 | stack-sf
1058568 | Fix | IMG 'lowsrc' may not be extracted | 2004-11-010 | stack-sf
1055854 | Fix | completed crawls show as 'aborted by user' | 2004-10-270 | gojomo
1059237 | Fix | MultiThreadedHttpConnectionManager https already connected | 2004-11-020 | stack-sf
1052578 | Fix | recovery log of recovered crawl insufficient to recover | 2004-10-220 | stack-sf
908690 | Fix | Some dates are GMT, others are not * | 2004-03-020 | gojomo
958096 | Fix | Flushing CrawlServers problematic * | 2004-05-210 | gojomo
1052570 | Fix | Threads contend for scratch files (after kill/readFully/Gap) | 2004-10-220 | gojomo
1033701 | Fix | incorrect number of total active threads * | 2004-09-230 | gojomo
1000840 | Fix | diskincludedfrontier performance is awful | 2004-07-30 | gojomo
1043251 | Fix | better/longer dns retries on lookup failure | 2004-10-08 | gojomo
1051072 | Fix | ExtractorHTML takes forever on worst-case HTML | 2004-10-20 | gojomo
1051916 | Fix | ExtractorJS takes forever on worst-case JS | 2004-10-21 | gojomo
1050238 | Fix | jdk required (doc implies jre) | 2004-10-19 | stack-sf
1038135 | Fix | prerequisite hysteresis/robots ahead of dns | 2004-09-30 | gojomo
1015728 | Fix | Crawl upper time/size bounds ignored | 2004-08-24 | gojomo
1002356 | Fix | timing issue on crawl-start & run-time stat | 2004-08-02 | gojomo
1002332 | Fix | inactiveQueuesMemoryLoadTarget mechanism behaves poorly | 2004-08-02 | gojomo
1045016 | Fix | DNS URIs don't get override settings | 2004-10-11 | gojomo
998184 | Fix | Gzipped recover log corrupt at end; last < 32K unrecoverable | 2004-07-26 | gojomo
998272 | Fix | No crawl if host-queues-memory-capacity = 0 | 2004-07-26 | stack-sf
1002335 | Fix | frontier report unusable in big crawls; frontier info needed | 2004-08-02 | gojomo
984390 | Fix | Build fails: "rws" mode and Mac OS X interact badly | 2004-07-02 | stack-sf
1000929 | Fix | fatal runtimeexceptions in frontier give no info in web UI | 2004-07-30 | gojomo
964625 | Fix | seed parser *too* lenient | 2004-06-01 | johnerik
980051 | Fix | Auth unsupported logged to console | 2004-06-25 | stack-sf
1002146 | Fix | bad queue keys: shouldn't be URIs; should be handled better | 2004-08-02 | stack-sf
1046696 | Fix | UURIFactory.validateEscaping() -> IllegalArgumentException | 2004-10-13 | stack-sf
1045736 | Fix | ARCReader crashes if zero-length gzip record | 2004-10-12 | stack-sf
1002144 | Fix | [UURI] Catch bad-encoding earlier | 2004-08-02 | stack-sf
1036680 | Fix | PathDepthFilter innerAccepts SEVERE log: "Failed getPath..." | 2004-09-28 | stack-sf
1045847 | Fix | Unnecessary toString() in ExtractorHTML.processScriptCode() | 2004-10-12 | gojomo
1044527 | Fix | Domain names in 'overrides' are not in alphabetical order | 2004-10-11 | kristinn_sig
1012639 | Fix | If CC timesout selftest, no build failed message | 2004-08-19 | stack-sf
1012642 | Fix | selftest hanging because no crawl stop event | 2004-08-19 | stack-sf
931565 | Fix | CrawlStateUpdater - NullPointerException | 2004-04-08 | stack-sf
973294 | Fix | NoSuchElementException in URI queues halts crawling | 2004-06-15 | gojomo
1033657 | Fix | [UURI] >2047 AFTER escaping (Stops crawl) | 2004-09-23 | stack-sf
1010966 | Fix | crawl.log has URIs with spaces in them | 2004-08-17 | stack-sf
963970 | Fix | unfetchable URI schemes should never be queued | 2004-05-31 | gojomo
1031607 | Fix | KeyedQueue server<->key mismatch noted: pfbuser<->mprsrv.agr | 2004-09-20 | stack-sf
1031525 | Fix | NPE reading override | 2004-09-20 | stack-sf
1031168 | Fix | Wrong handling of date in ARCRecordMetaData | 2004-09-20 | johnerik

Prev |  | Next
9. Release 1.4.0 - 2005-04-28 | Home | 11. Release 1.0.4 - 2004-09-22