<

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x

Atomically updating documents in Solr using SolrJ

Published on
10,509 Points
7,409 Views
1 Endorsement
Last Modified:
In Solr 4.0 it is possible to atomically (or partially) update individual fields in a document. This article will show the operations possible for atomic updating as well as setting up your Solr instance to be able to perform the actions. One major motivation for using atomic updating is being able to change a part of the document without the need to regenerate the entire document. So if your document is created from many different data sources where fetching the data might be expensive, atomic updating might be worth looking into.
 

Getting started

First you must be using Solr 4.0. Older versions do not support atomic updates. And second, all fields in your schema.xml file must be set to stored. So if your schema file looked like this:

<field name="id" type="number" indexed="true" stored="true" required="true" />
<field name="title" type="text_en" indexed="true" stored="false"/>
<field name="submit_date" type="date" indexed="true" stored="false" />
<field name="views" type="number" indexed="true" stored="false" />

Open in new window

all stored="false" attributes must be changed to stored="true" so that your schema file looks like this:

<field name="id" type="number" indexed="true" stored="true" required="true" />
<field name="title" type="text_en" indexed="true" stored="true"/>
<field name="submit_date" type="date" indexed="true" stored="true" />
<field name="views" type="number" indexed="true" stored="true" />

Open in new window

Out of the box, atomic updating should work correctly if your schema file is configured correctly. If atomic updating does not work, it might be helpful to view the caveats and limitations section of the Solr wiki for other helpful information.
 


Atomically updating fields in SolrJ

Once your instance of Solr is up and running and configured correctly, you can atomically update a document using SolrJ.  When atomically updating a field, it is possible to perform four actions:

  • set (two operations in one command) - set a value or remove it if null is used as the value.
  • add – adds an additional value to a multi-valued field.
  • inc - increments a number field by a certain value or decrements it when a negative value is used.
Below is an example of a method to increment a field in a document by a number given the document's ID.
public void incrementDocumentByValue(String solrServerUrl, String documentId, 
   String fieldToUpdate, int incrementValue)
{
   // create the SolrJ client
   HttpSolrServer client = new HttpSolrServer(solrServerUrl);
 
   // create a new document
   SolrInputDocument document = new SolrInputDocument();
         
   // create a map of the operation to be performed
   Map<String, Object> operation = new HashMap<>();

   // insert the operation into the map
   operation.put("inc", new Integer(incrementValue));

   // unique identifier
   SolrSearchUtil.addToDocument(document, "id", documentId);

   // add the atomic update
   SolrSearchUtil.addToDocument(document, fieldToUpdate, operation);

   // send it to the solr server and shutdown
   client.add( document );
   client.shutdown();
}

Open in new window

The important part of the code above is the operation variable, which is java.util.Map where the key is the operation to be performed and the value is the number you want to increment the document's field by. So if we wanted to increase the views of a document by 5, the code might look like:

incrementDocumentByValue("localhost:8983/solr", "unique-id-0001", "views", 5);

Open in new window

 

Cost considerations

Being able to atomically update fields is not free and comes with a cost in the form of storage and perhaps performance. By setting all fields to be stored, the size of the index will definitely increase. From my experience, the increase can be anywhere from 5% to 50% depending on the data that is stored in those fields. If you have a large index of 10 gigabytes, expect your atomically updatable index to be a 11 to 15 gigabytes in size.

As for query time, it would make sense that that a larger index would mean Solr would have more work to do when searching for data in that index, but from my experience the query time is not so far off from a regular index where most of the fields aren't set to be stored. Using the Solr Admin query interface queries would sometimes return quicker on the atomically updatable index than the regular index. I tried looking for research or other people's experience with performance regarding an atomically updatable index, but I wasn't able to find anything, so if you run into anything please let me know.

Atomically updating documents can be useful in a right situation. Setting up your index to be compatible with atomic updating is relatively simple, but your index will be larger than an index that is not compatible with atomic updating. Using SolrJ, updating individual fields of a document is very simple.
1
Comment
0 Comments

Featured Post

Become a Certified Penetration Testing Engineer

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

Join & Write a Comment

A query can call a function, and a function can call Excel, even though we are in Access. This is Part 2, and steps you through the VBA that "wraps" Excel functionality so we can use its worksheet functions in Access. The declaration statement de…
There are many cases found where ScanPST.exe fails to repair corrupt Outlook PST File. When user tries to repair PST using Inbox Repair tool and it throws below error: •      Inbox Repair tool does not recognize the file •      ScanPST.exe hangs in betwee…
Other articles by this author

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month