solr/solr-ref-guide/src/de-duplication.adoc

= De-Duplication
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

If duplicate, or near-duplicate documents are a concern in your index, de-duplication may be worth implementing.

Preventing duplicate or near duplicate documents from entering an index or tagging documents with a signature/fingerprint for duplicate field collapsing can be efficiently achieved with a low collision or fuzzy hash algorithm. Solr natively supports de-duplication techniques of this type via the `Signature` class and allows for the easy addition of new hash/signature implementations. A Signature can be implemented in a few ways:

* MD5Signature: 128-bit hash used for exact duplicate detection.
* Lookup3Signature: 64-bit hash used for exact duplicate detection. This is much faster than MD5 and smaller to index.
* https://cwiki.apache.org/confluence/display/solr/TextProfileSignature[TextProfileSignature]: Fuzzy hashing implementation from Apache Nutch for near duplicate detection. It's tunable but works best on longer text.

Other, more sophisticated algorithms for fuzzy/near hashing can be added later.

[IMPORTANT]
====
Adding in the de-duplication process will change the `allowDups` setting so that it applies to an update term (with `signatureField` in this case) rather than the unique field Term.

Of course the `signatureField` could be the unique field, but generally you want the unique field to be unique. When a document is added, a signature will automatically be generated and attached to the document in the specified `signatureField`.
====

== Configuration Options

There are two places in Solr to configure de-duplication: in `solrconfig.xml` and in `schema.xml`.

=== In solrconfig.xml

The `SignatureUpdateProcessorFactory` has to be registered in `solrconfig.xml` as part of an <<update-request-processors.adoc#,Update Request Processor Chain>>, as in this example:

[source,xml]
----
<updateRequestProcessorChain default="true">
  <processor class="solr.LogUpdateProcessorFactory" />
  <processor class="solr.processor.SignatureUpdateProcessorFactory">
    <str name="signatureField">id</str>
    <str name="fields">name,features,cat</str>
    <str name="signatureClass">solr.processor.Lookup3Signature</str>
  </processor>
  <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
----

The `SignatureUpdateProcessorFactory` takes several properties:

`signatureClass`::
A Signature implementation for generating a signature hash. The default is `org.apache.solr.update.processor.Lookup3Signature`.
+
The full classpath of the implementation must be specified. The available options are described above, the associated classpaths to use are:

* `org.apache.solr.update.processor.Lookup3Signature`
* `org.apache.solr.update.processor.MD5Signature`
* `org.apache.solr.update.process.TextProfileSignature`

`fields`::
The fields to use to generate the signature hash in a comma separated list. By default, all fields on the document will be used.

`signatureField`::
The name of the field used to hold the fingerprint/signature. The field should be defined in `schema.xml`. The default is `signatureField`.

`enabled`::
Set to *false* to disable de-duplication processing. The default is *true*.

`overwriteDupes`::
If *true*, the default, when a document exists that already matches this signature, it will be overwritten.  If you are using `overwriteDupes=true` the `signatureField` must be `indexed="true"` in your Schema.

.Using `SignatureUpdateProcessorFactory` in SolrCloud
[WARNING]
====
// https://issues.apache.org/jira/browse/SOLR-3473

There are 2 important things to keep in mind when using `SignatureUpdateProcessorFactory` with SolrCloud:

. The `overwriteDupes=true` setting does not work _except_ in the special case of using the uniqueKey field as the `signatureField`.  Attempting De-duplication on any other `signatureField` will not work correctly because of how updates are forwarded to replicas
. When using the uniqueKey field as the `signatureField`, `SignatureUpdateProcessorFactory` must be run prior to the `<<update-request-processors.adoc#update-processors-in-solrcloud,DistributedUpdateProcessor>>` to ensure that documents can be routed to the correct shard leader based on the (generated) uniqueKey field.

(Using any other `signatureField` with `overwriteDupes=false` -- to generate a Signature for each document with out De-duplication -- has no limitations.)
====