22.7. Extensibility
السابق | فصل 22. Zend_Search | التالي

## 22.7. Extensibility

### 22.7.1. Text Analysis

`Zend_Search_Lucene_Analysis_Analyzer` class is used by indexer to tokenize document text fields.

`Zend_Search_Lucene_Analysis_Analyzer::getDefault()` and `Zend_Search_Lucene_Analysis_Analyzer::setDefault()` methods are used to get and set default analyzer.

Thus you can assign your own text analyzer or choose it from the set of predefined analyzers: `Zend_Search_Lucene_Analysis_Analyzer_Common_Text` and `Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive` (default). Both of them interpret token as a sequence of letters. `Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive` converts tokens to lower case.

To switch between analyzers:

    <?php

    Zend_Search_Lucene_Analysis_Analyzer::setDefault(
        new Zend_Search_Lucene_Analysis_Analyzer_Common_Text());
    ...
    $index->addDocument($doc);

    ?>

`Zend_Search_Lucene_Analysis_Analyzer_Common` is designed to be a parent for all user defined analyzers. User should only define the `tokenize()` method, which takes input data as a string and returns an array of tokens.

The `tokenize()` method should apply the `normalize()` method to all tokens. It will allow you to use token filters with your analyzer.

Here is an example of a custom Analyzer, which takes words with digits as terms:

مثال 22.1. Custom text Analyzer.

    <?php
    /** Here is a custom text analyser, which treats words with digits as one term */

    /** Zend_Search_Lucene_Analysis_Analyzer_Common */
    require_once 'Zend/Search/Lucene/Analysis/Analyzer/Common.php';

    class My_Analyzer extends Zend_Search_Lucene_Analysis_Analyzer_Common
    {
        /**
         * Tokenize text to a terms
         * Returns array of Zend_Search_Lucene_Analysis_Token objects
         *
         * @param string $data
         * @return array
         */
        public function tokenize($data)
        {
            $tokenStream = array();

            $position = 0;
            while ($position < strlen($data)) {
                // skip white space
                while ($position < strlen($data) && !ctype_alpha($data{$position}) && !ctype_digit($data{$position})) {
                    $position++;
                }

                $termStartPosition = $position;

                // read token
                while ($position < strlen($data) && (ctype_alpha($data{$position}) || ctype_digit($data{$position}))) {
                    $position++;
                }

                // Empty token, end of stream.
                if ($position == $termStartPosition) {
                    break;
                }

                $token = new Zend_Search_Lucene_Analysis_Token(substr($data,
                                                 $termStartPosition,
                                                 $position-$termStartPosition),
                                          $termStartPosition,
                                          $position);
                $tokenStream[] = $this->normalize($token);
            }

            return $tokenStream;
        }
    }

    Zend_Search_Lucene_Analysis_Analyzer::setDefault(
        new My_Analyzer());

    ?>

### 22.7.2. Scoring Algorithms

The score of query `q` for document `d` is defined as follows:

`score(q,d) = sum( tf(t in d) * idf(t) * getBoost(t.field in d) * lengthNorm(t.field in d) ) * coord(q,d) * queryNorm(q)`

tf(t in d) - `Zend_Search_Lucene_Search_Similarity::tf($freq)` - a score factor based on a term or phrase's frequency in a document.

idf(t) - `Zend_Search_Lucene_Search_SimilaritySimilarity::tf($term, $reader)` - a score factor for a simple term for the specified index.

getBoost(t.field in d) - boost factor for the term field.

lengthNorm($term) - the normalization value for a field given the total number of terms contained in a field. This value is stored within the index. These values, together with field boosts, are stored in an index and multiplied into scores for hits on each field by the search code.

Matches in longer fields are less precise, so implementations of this method usually return smaller values when numTokens is large, and larger values when numTokens is small.

coord(q,d) - `Zend_Search_Lucene_Search_Similarity::coord($overlap, $maxOverlap)` - a score factor based on the fraction of all query terms that a document contains.

The presence of a large portion of the query terms indicates a better match with the query, so implementations of this method usually return larger values when the ratio between these parameters is large and smaller values when the ratio between them is small.

queryNorm(q) - the normalization value for a query given the sum of the squared weights of each of the query terms. This value is then multiplied into the weight of each query term.

This does not affect ranking, but rather just attempts to make scores from different queries comparable.

Scoring algorithm can be customized by defining your own Similarity class. To do this extend Zend_Search_Lucene_Search_Similarity class as defined below, then use `Zend_Search_Lucene_Search_Similarity::setDefault($similarity);` method to set it as default.

    <?php

    class MySimilarity extends Zend_Search_Lucene_Search_Similarity {
        public function lengthNorm($fieldName, $numTerms) {
            return 1.0/sqrt($numTerms);
        }

        public function queryNorm($sumOfSquaredWeights) {
            return 1.0/sqrt($sumOfSquaredWeights);
        }

        public function tf($freq) {
            return sqrt($freq);
        }

        /**
         * It's not used now. Computes the amount of a sloppy phrase match,
         * based on an edit distance.
         */
        public function sloppyFreq($distance) {
            return 1.0;
        }

        public function idfFreq($docFreq, $numDocs) {
            return log($numDocs/(float)($docFreq+1)) + 1.0;
        }

        public function coord($overlap, $maxOverlap) {
            return $overlap/(float)$maxOverlap;
        }
    }

    $mySimilarity = new MySimilarity();
    Zend_Search_Lucene_Search_Similarity::setDefault($mySimilarity);

    ?>

### 22.7.3. Storage Containers

An abstract class `Zend_Search_Lucene_Storage_Directory` defines directory functionality.

The `Zend_Search_Lucene` constructor uses either a string or a `Zend_Search_Lucene_Storage_Directory` object as an input.

`Zend_Search_Lucene_Storage_Directory_Filesystem` class implements directory functionality for file system.

If a string is used as an input for the `Zend_Search_Lucene` constructor, then the index reader (`Zend_Search_Lucene` object) treats it as a file system path and instantiates `Zend_Search_Lucene_Storage_Directory_Filesystem` objects by themselves.

You can define your own directory implementation by extending the `Zend_Search_Lucene_Storage_Directory` class.

`Zend_Search_Lucene_Storage_Directory` methods:

    <?php

    abstract class Zend_Search_Lucene_Storage_Directory {
    /**
     * Closes the store.
     *
     * @return void
     */
    abstract function close();

    /**
     * Creates a new, empty file in the directory with the given $filename.
     *
     * @param string $name
     * @return void
     */
    abstract function createFile($filename);

    /**
     * Removes an existing $filename in the directory.
     *
     * @param string $filename
     * @return void
     */
    abstract function deleteFile($filename);

    /**
     * Returns true if a file with the given $filename exists.
     *
     * @param string $filename
     * @return boolean
     */
    abstract function fileExists($filename);

    /**
     * Returns the length of a $filename in the directory.
     *
     * @param string $filename
     * @return integer
     */
    abstract function fileLength($filename);

    /**
     * Returns the UNIX timestamp $filename was last modified.
     *
     * @param string $filename
     * @return integer
     */
    abstract function fileModified($filename);

    /**
     * Renames an existing file in the directory.
     *
     * @param string $from
     * @param string $to
     * @return void
     */
    abstract function renameFile($from, $to);

    /**
     * Sets the modified time of $filename to now.
     *
     * @param string $filename
     * @return void
     */
    abstract function touchFile($filename);

    /**
     * Returns a Zend_Search_Lucene_Storage_File object for a given $filename in the directory.
     *
     * @param string $filename
     * @return Zend_Search_Lucene_Storage_File
     */
    abstract function getFileObject($filename);

    }

    ?>

`getFileObject($filename)` method of `Zend_Search_Lucene_Storage_Directory` class returns a `Zend_Search_Lucene_Storage_File` object.

`Zend_Search_Lucene_Storage_File` abstract class implements file abstraction and index file reading primitives.

You must also extend `Zend_Search_Lucene_Storage_File` for your Directory implementation.

Only two methods of `Zend_Search_Lucene_Storage_File` must be overloaded in your implementation:

    <?php

    class MyFile extends Zend_Search_Lucene_Storage_File {
        /**
         * Sets the file position indicator and advances the file pointer.
         * The new position, measured in bytes from the beginning of the file,
         * is obtained by adding offset to the position specified by whence,
         * whose values are defined as follows:
         * SEEK_SET - Set position equal to offset bytes.
         * SEEK_CUR - Set position to current location plus offset.
         * SEEK_END - Set position to end-of-file plus offset. (To move to
         * a position before the end-of-file, you need to pass a negative value
         * in offset.)
         * Upon success, returns 0; otherwise, returns -1
         *
         * @param integer $offset
         * @param integer $whence
         * @return integer
         */
        public function seek($offset, $whence=SEEK_SET) {
            ...
        }

        /**
         * Read a $length bytes from the file and advance the file pointer.
         *
         * @param integer $length
         * @return string
         */
        protected function _fread($length=1) {
            ...
        }
    }

    ?>

السابق | أعلى | التالي
22.6. Character set. | البداية | 22.8. Interoperating with Java Lucene