solr/solr-ref-guide/src/blob-store-api.adoc - lucene-solr - Git at Google

 = Blob Store API
 // Licensed to the Apache Software Foundation (ASF) under one
 // or more contributor license agreements.  See the NOTICE file
 // distributed with this work for additional information
 // regarding copyright ownership.  The ASF licenses this file
 // to you under the Apache License, Version 2.0 (the
 // "License"); you may not use this file except in compliance
 // with the License.  You may obtain a copy of the License at
 //
 //   http://www.apache.org/licenses/LICENSE-2.0
 //
 // Unless required by applicable law or agreed to in writing,
 // software distributed under the License is distributed on an
 // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 // KIND, either express or implied.  See the License for the
 // specific language governing permissions and limitations
 // under the License.

 The Blob Store REST API provides REST methods to store, retrieve or list files in a Lucene index.

 It can be used to upload a jar file which contains standard Solr components such as RequestHandlers, SearchComponents,
 or other custom code you have written for Solr. Schema components _do not_ yet support the Blob Store.

 When using the blob store, note that the API does not delete or overwrite a previous object if a new one is uploaded with the same name.
 It always adds a new version of the blob to the index.
 Because the `.system` collection is a standard Solr collection, deleting blobs is the same as deleting documents.

 *The blob store is only available when running in SolrCloud mode.* Solr in standalone mode does not support use of a blob store.

 The blob store API is implemented as a requestHandler. A special collection named ".system" is used to store the blobs. This collection can be created in advance, but if it does not exist it will be created automatically.

 == About the .system Collection

 Before uploading blobs to the blob store, a special collection must be created and it must be named `.system`. Solr will automatically create this collection if it does not already exist, but you can also create it manually if you choose.

 The BlobHandler is automatically registered in the .system collection. The `solrconfig.xml`, Schema, and other configuration files for the collection are automatically provided by the system and don't need to be defined specifically.

 If you do not use the `-shards` or `-replicationFactor` options, then defaults of numShards=1 and replicationFactor=3 (or maximum nodes in the cluster) will be used.

 You can create the `.system` collection with the <<collection-management.adoc#create,CREATE command>> of the Collections API, as in this example:

 [.dynamic-tabs]
 --
 [example.tab-pane#v1create]
 ====
 [.tab-label]*V1 API*

 [source,bash]
 ----
 curl http://localhost:8983/solr/admin/collections?action=CREATE&name=.system&replicationFactor=2&numShards=2
 ----

 Note that this example will create the .system collection across 2 shards with a replication factor of 2; you may need to customize this for your Solr implementation.
 ====

 [example.tab-pane#v2create]
 ====
 [.tab-label]*V2 API*

 [source,bash]
 ----
 curl -X POST -H 'Content-type: application/json' -d '{"create": {"name": ".system", "numShards": "2", "replicationFactor": "2"}}' http://localhost:8983/api/collections
 ----

 Note that this example will create the .system collection across 2 shards with a replication factor of 2; you may need to customize this for your Solr implementation.
 ====
 --

 IMPORTANT: The `bin/solr` script cannot be used to create the `.system` collection.

 == Upload Files to Blob Store

 After the `.system` collection has been created, files can be uploaded to the blob store with a request similar to the following:

 [.dynamic-tabs]
 --
 [example.tab-pane#v1upload]
 ====
 [.tab-label]*V1 API*

 [source,bash]
 ----
 curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @{filename} http://localhost:8983/solr/.system/blob/{blobname}
 ----

 For example, to upload a file named "test1.jar" as a blob named "test", you would make a POST request like:

 [source,bash]
 ----
 curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test1.jar http://localhost:8983/solr/.system/blob/test
 ----
 ====

 [example.tab-pane#v2upload]
 ====
 [.tab-label]*V2 API*

 [source,bash]
 ----
 curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @{filename} http://localhost:8983/api/collections/.system/blob/{blobname}
 ----

 For example, to upload a file named "test1.jar" as a blob named "test", you would make a POST request like:

 [source,bash]
 ----
 curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test1.jar http://localhost:8983/api/collections/.system/blob/test
 ----
 ====
 --

 Note that by default, the blob store has a limit of 5Mb for any blob. This can be increased if necessary
 by changing the value for the `maxSize` setting in `solrconfig.xml` for the `.system` collection.
 See the section <<configuring-solrconfig-xml.adoc#,Configuring solrconfig.xml>> for information about how to modify `solrconfig.xml` for any collection.

 A GET request will return the list of blobs and other details:

 [.dynamic-tabs]
 --
 [example.tab-pane#v1getblob]
 ====
 [.tab-label]*V1 API*

 For all blobs:

 [source,bash]
 ----
 curl http://localhost:8983/solr/.system/blob?omitHeader=true
 ----

 For a single blob:

 [source,bash]
 ----
 curl http://localhost:8983/solr/.system/blob/test?omitHeader=true
 ----

 Output:

 [source,json]
 ----
 {
   "response":{"numFound":1,"start":0,"docs":[
       {
         "id":"test/1",
         "md5":"20ff915fa3f5a5d66216081ae705c41b",
         "blobName":"test",
         "version":1,
         "timestamp":"2015-02-04T16:45:48.374Z",
         "size":13108}]
   }
 }
 ----
 ====

 [example.tab-pane#v2getblob]
 ====
 [.tab-label]*V2 API*

 For all blobs:

 [source,bash]
 ----
 curl http://localhost:8983/api/collections/.system/blob?omitHeader=true
 ----

 For a single blob:

 [source,bash]
 ----
 curl http://localhost:8983/api/collections/.system/blob/test?omitHeader=true
 ----

 Output:

 [source,json]
 ----
 {
   "response":{"numFound":1,"start":0,"docs":[
       {
         "id":"test/1",
         "md5":"20ff915fa3f5a5d66216081ae705c41b",
         "blobName":"test",
         "version":1,
         "timestamp":"2015-02-04T16:45:48.374Z",
         "size":13108}]
   }
 }
 ----
 ====
 --

 The filestream response writer can retrieve a blob for download, as in:

 [.dynamic-tabs]
 --
 [example.tab-pane#v1retrieveblob]
 ====
 [.tab-label]*V1 API*

 For a specific version of a blob, include the version to the request:
 [source,bash]
 ----
 curl http://localhost:8983/solr/.system/blob/{blobname}/{version}?wt=filestream > {outputfilename}
 ----

 For the latest version of a blob, the `\{version}` can be omitted:

 [source,bash]
 ----
 curl http://localhost:8983/solr/.system/blob/{blobname}?wt=filestream > {outputfilename}
 ----
 ====

 [example.tab-pane#v2retrieveblob]
 ====
 [.tab-label]*V2 API*
 For a specific version of a blob, include the version to the request:

 [source,bash]
 ----
 curl http://localhost:8983/api/collections/.system/blob/{blobname}/{version}?wt=filestream > {outputfilename}
 ----

 For the latest version of a blob, the `\{version}` can be omitted:

 [source,bash]
 ----
 curl http://localhost:8983/api/collections/.system/blob/{blobname}?wt=filestream > {outputfilename}
 ----
 ====
 --

 == Use a Blob in a Handler or Component

 To use the blob as the class for a request handler or search component, you create a request handler in `solrconfig.xml` as usual. You will need to define the following parameters:

 `class`:: the fully qualified class name. For example, if you created a new request handler class called CRUDHandler, you would enter `org.apache.solr.core.CRUDHandler`.
 `runtimeLib`:: Set to true to require that this component should be loaded from the classloader that loads the runtime jars.

 For example, to use a blob named test, you would configure `solrconfig.xml` like this:

 [source,xml]
 ----
 <requestHandler name="/myhandler" class="org.apache.solr.core.myHandler" runtimeLib="true" version="1">
 </requestHandler>
 ----

 If there are parameters available in the custom handler, you can define them in the same way as any other request handler definition.

 NOTE: Blob store can only be used to dynamically load components configured in `solrconfig.xml`. Components specified in `schema.xml` cannot be loaded from blob store.

 == Deleting Blobs

 Once loaded to the blob store, blobs are handled very similarly to usual indexed documents in Solr.
 To delete blobs, you can use the same approaches used to delete individual documents from the index,
 namely Delete By ID and Delete By Query.

 For example, to delete a blob with the id `test/1`, you would issue a command like this:

 [source,text]
 curl -H 'Content-Type: application/json' -d '{"delete": {"id": "test/1"}}' http://localhost:8983/solr/.system/update?commit=true

 Be sure to tell Solr to perform a <<updatehandlers-in-solrconfig.adoc#commits,commit>> as part of the request
  (`commit=true` in the above example) to see the change immediately.
 If you do not instruct Solr to perform a commit, Solr will use the `.system` collection autoCommit settings,
 which may not be the expected behavior.

 You can also use the delete by query syntax, as so:

 [source,text]
 curl -H 'Content-Type: application/json' -d '{"delete": {"query": "id:test/1"}}' http://localhost:8983/solr/.system/update?commit=true

 For more on deleting documents generally, see the section <<uploading-data-with-index-handlers.adoc#sending-json-update-commands,Sending JSON Update Commands>>.
	= Blob Store API
	// Licensed to the Apache Software Foundation (ASF) under one
	// or more contributor license agreements. See the NOTICE file
	// distributed with this work for additional information
	// regarding copyright ownership. The ASF licenses this file
	// to you under the Apache License, Version 2.0 (the
	// "License"); you may not use this file except in compliance
	// with the License. You may obtain a copy of the License at
	//
	// http://www.apache.org/licenses/LICENSE-2.0
	//
	// Unless required by applicable law or agreed to in writing,
	// software distributed under the License is distributed on an
	// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	// KIND, either express or implied. See the License for the
	// specific language governing permissions and limitations
	// under the License.

	The Blob Store REST API provides REST methods to store, retrieve or list files in a Lucene index.

	It can be used to upload a jar file which contains standard Solr components such as RequestHandlers, SearchComponents,
	or other custom code you have written for Solr. Schema components _do not_ yet support the Blob Store.

	When using the blob store, note that the API does not delete or overwrite a previous object if a new one is uploaded with the same name.
	It always adds a new version of the blob to the index.
	Because the `.system` collection is a standard Solr collection, deleting blobs is the same as deleting documents.

	The blob store is only available when running in SolrCloud mode. Solr in standalone mode does not support use of a blob store.

	The blob store API is implemented as a requestHandler. A special collection named ".system" is used to store the blobs. This collection can be created in advance, but if it does not exist it will be created automatically.

	== About the .system Collection

	Before uploading blobs to the blob store, a special collection must be created and it must be named `.system`. Solr will automatically create this collection if it does not already exist, but you can also create it manually if you choose.

	The BlobHandler is automatically registered in the .system collection. The `solrconfig.xml`, Schema, and other configuration files for the collection are automatically provided by the system and don't need to be defined specifically.

	If you do not use the `-shards` or `-replicationFactor` options, then defaults of numShards=1 and replicationFactor=3 (or maximum nodes in the cluster) will be used.

	You can create the `.system` collection with the <<collection-management.adoc#create,CREATE command>> of the Collections API, as in this example:

	[.dynamic-tabs]
	--
	[example.tab-pane#v1create]
	====
	[.tab-label]V1 API

	[source,bash]
	----
	curl http://localhost:8983/solr/admin/collections?action=CREATE&name=.system&replicationFactor=2&numShards=2
	----

	Note that this example will create the .system collection across 2 shards with a replication factor of 2; you may need to customize this for your Solr implementation.
	====

	[example.tab-pane#v2create]
	====
	[.tab-label]V2 API

	[source,bash]
	----
	curl -X POST -H 'Content-type: application/json' -d '{"create": {"name": ".system", "numShards": "2", "replicationFactor": "2"}}' http://localhost:8983/api/collections
	----

	Note that this example will create the .system collection across 2 shards with a replication factor of 2; you may need to customize this for your Solr implementation.
	====
	--

	IMPORTANT: The `bin/solr` script cannot be used to create the `.system` collection.

	== Upload Files to Blob Store

	After the `.system` collection has been created, files can be uploaded to the blob store with a request similar to the following:

	[.dynamic-tabs]
	--
	[example.tab-pane#v1upload]
	====
	[.tab-label]V1 API

	[source,bash]
	----
	curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @{filename} http://localhost:8983/solr/.system/blob/{blobname}
	----

	For example, to upload a file named "test1.jar" as a blob named "test", you would make a POST request like:

	[source,bash]
	----
	curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test1.jar http://localhost:8983/solr/.system/blob/test
	----
	====

	[example.tab-pane#v2upload]
	====
	[.tab-label]V2 API

	[source,bash]
	----
	curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @{filename} http://localhost:8983/api/collections/.system/blob/{blobname}
	----

	For example, to upload a file named "test1.jar" as a blob named "test", you would make a POST request like:

	[source,bash]
	----
	curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @test1.jar http://localhost:8983/api/collections/.system/blob/test
	----
	====
	--

	Note that by default, the blob store has a limit of 5Mb for any blob. This can be increased if necessary
	by changing the value for the `maxSize` setting in `solrconfig.xml` for the `.system` collection.
	See the section <<configuring-solrconfig-xml.adoc#,Configuring solrconfig.xml>> for information about how to modify `solrconfig.xml` for any collection.

	A GET request will return the list of blobs and other details:

	[.dynamic-tabs]
	--
	[example.tab-pane#v1getblob]
	====
	[.tab-label]V1 API

	For all blobs:

	[source,bash]
	----
	curl http://localhost:8983/solr/.system/blob?omitHeader=true
	----

	For a single blob:

	[source,bash]
	----
	curl http://localhost:8983/solr/.system/blob/test?omitHeader=true
	----

	Output:

	[source,json]
	----
	{
	"response":{"numFound":1,"start":0,"docs":[
	{
	"id":"test/1",
	"md5":"20ff915fa3f5a5d66216081ae705c41b",
	"blobName":"test",
	"version":1,
	"timestamp":"2015-02-04T16:45:48.374Z",
	"size":13108}]
	}
	}
	----
	====

	[example.tab-pane#v2getblob]
	====
	[.tab-label]V2 API

	For all blobs:

	[source,bash]
	----
	curl http://localhost:8983/api/collections/.system/blob?omitHeader=true
	----

	For a single blob:

	[source,bash]
	----
	curl http://localhost:8983/api/collections/.system/blob/test?omitHeader=true
	----

	Output:

	[source,json]
	----
	{
	"response":{"numFound":1,"start":0,"docs":[
	{
	"id":"test/1",
	"md5":"20ff915fa3f5a5d66216081ae705c41b",
	"blobName":"test",
	"version":1,
	"timestamp":"2015-02-04T16:45:48.374Z",
	"size":13108}]
	}
	}
	----
	====
	--

	The filestream response writer can retrieve a blob for download, as in:

	[.dynamic-tabs]
	--
	[example.tab-pane#v1retrieveblob]
	====
	[.tab-label]V1 API

	For a specific version of a blob, include the version to the request:
	[source,bash]
	----
	curl http://localhost:8983/solr/.system/blob/{blobname}/{version}?wt=filestream > {outputfilename}
	----

	For the latest version of a blob, the `\{version}` can be omitted:

	[source,bash]
	----
	curl http://localhost:8983/solr/.system/blob/{blobname}?wt=filestream > {outputfilename}
	----
	====

	[example.tab-pane#v2retrieveblob]
	====
	[.tab-label]V2 API
	For a specific version of a blob, include the version to the request:

	[source,bash]
	----
	curl http://localhost:8983/api/collections/.system/blob/{blobname}/{version}?wt=filestream > {outputfilename}
	----

	For the latest version of a blob, the `\{version}` can be omitted:

	[source,bash]
	----
	curl http://localhost:8983/api/collections/.system/blob/{blobname}?wt=filestream > {outputfilename}
	----
	====
	--

	== Use a Blob in a Handler or Component

	To use the blob as the class for a request handler or search component, you create a request handler in `solrconfig.xml` as usual. You will need to define the following parameters:

	`class`:: the fully qualified class name. For example, if you created a new request handler class called CRUDHandler, you would enter `org.apache.solr.core.CRUDHandler`.
	`runtimeLib`:: Set to true to require that this component should be loaded from the classloader that loads the runtime jars.

	For example, to use a blob named test, you would configure `solrconfig.xml` like this:

	[source,xml]
	----
	<requestHandler name="/myhandler" class="org.apache.solr.core.myHandler" runtimeLib="true" version="1">
	</requestHandler>
	----

	If there are parameters available in the custom handler, you can define them in the same way as any other request handler definition.

	NOTE: Blob store can only be used to dynamically load components configured in `solrconfig.xml`. Components specified in `schema.xml` cannot be loaded from blob store.

	== Deleting Blobs

	Once loaded to the blob store, blobs are handled very similarly to usual indexed documents in Solr.
	To delete blobs, you can use the same approaches used to delete individual documents from the index,
	namely Delete By ID and Delete By Query.

	For example, to delete a blob with the id `test/1`, you would issue a command like this:

	[source,text]
	curl -H 'Content-Type: application/json' -d '{"delete": {"id": "test/1"}}' http://localhost:8983/solr/.system/update?commit=true

	Be sure to tell Solr to perform a <<updatehandlers-in-solrconfig.adoc#commits,commit>> as part of the request
	(`commit=true` in the above example) to see the change immediately.
	If you do not instruct Solr to perform a commit, Solr will use the `.system` collection autoCommit settings,
	which may not be the expected behavior.

	You can also use the delete by query syntax, as so:

	[source,text]
	curl -H 'Content-Type: application/json' -d '{"delete": {"query": "id:test/1"}}' http://localhost:8983/solr/.system/update?commit=true

	For more on deleting documents generally, see the section <<uploading-data-with-index-handlers.adoc#sending-json-update-commands,Sending JSON Update Commands>>.