Solr Digest, Spring-Summer 2011, Part 1

No, Solr Digests are not dead, we’ve just been crazily busy at Sematext (yes, we are hiring!). Since our last Solr Digest not one, but 2 new Solr releases have been made: 3.2 in June, 3.3 in July and version 3.4 is imminent – voting is already in progress, so you can expect a new release pretty soon. Also, there were a number of interesting developments on the trunk (future 3.x and 4.0 versions). Therefore, we will be publishing two Solr Digests this time. This first Digest covers general developments in Solr world, while the sequel will be more focused on two features drawing a lot of attention: Solr Cloud and Near Real Time search.

Let’s get started with a short overview of announced news in 3.2 and 3.3. First, 3.2 brought us:

• Ability to specify overwrite and commitWithin as request parameters when using the JSON update format
• TermQParserPlugin, useful when generating filter queries from terms returned by field faceting or terms component
• DebugComponent now supports using a NamedList to model Explanation objects in its responses instead of Explanation.toString
• Improvements to the UIMA and Carrot2 integrations
• Highlighting performance improvements
• A test-framework jar for easy testing of Solr extensions
• Bugfixes and improvements from Apache Lucene 3.2

With 3.3 we got:

• Grouping / Field Collapsing
• A new, automaton-based suggest/autocomplete implementation offering an order of magnitude smaller RAM consumption
• KStemFilterFactory, an optimized implementation of a less aggressive stemmer for English
• Solr defaults to a new, more efficient merge policy (TieredMergePolicy). See Mike’s cool Lucene segment merging video
• Important bugfixes, including extremely high RAM usage in spellchecking
• Bugfixes and improvements from Apache Lucene 3.3

Let’s now look at other interesting stuff. We’ll start with DataImportHandler and its bug fixes. As you’ll notice, there are quite a few of them (and we didn’t even list them all!) so we advise using all available patches.

Already committed features

• A bug-fix for DataImportHandler – “replication reserves commit-point forever if using replicateAfter=startup”. SOLR-2469 brought a fix to version 3.2 and future 4.0 (trunk). This problem caused unnecessary (and huge) buildup in the number of index files on the slaves.
• Another bug-fix for DataImportHandler – DIH does not commit if only Deletes are processed. When using special commands $deleteDocById and/or $deleteDocByQuery, when there were no updates of documents, commit wasn’t called by the DIH. Fix is available in 3.4 and 4.0.
• Also – DataImportHandler multi-threaded option throws exception. The problem would happen when threads attribute was used. The fix for this is available in 3.4 and 4.0. Related to this is another fixed issue – DIH multi threaded mode does not resolves attributes correctly also available in 3.4 and 4.0.
• Join feature got committed to the trunk (future 4.0 version). It can also perform cross-core joins now, which can be very useful. However, this feature also initiated some heated discussions which can be seen in SOLR-2272. The root cause was the fact that this feature was committed only to the Solr while Lucene got none of it. Of course, it might get refactored and included in Lucene too in the future, but this discussion shows the divisons which still existed between Solr and Lucene communities back then.
• While we’re talking about Join feature, it might be worth mentioning a patch in SOLR-2604 which back-ports it to 3.x version. Be careful though, it was created for version 3.2 more than two months ago, so a few more adjustments after applying this patch might be needed.
• Function Queries got new if(), exists(), and(), or(), not(), xor() and def() functions. The fix is committed to trunk so you’ll be able to use it in 4.0.
• As can be seen from the Solr 3.3 announcement, one of the longest living Solr issue is finally closed for good . SOLR-236 – Field Collapsing – along with SOLR-2524 finally bring field collapsing to 3_x and future 4.0 versions.
• Since grouping/field collapsing was added to Solr, we should be able to use faceting in combination with it. Issue SOLR-2665 – Solr Post Group Faceting – brought exactly that to 3.4 and 4.0.
• Ever wanted to have more control over what gets stored in the cache? SOLR-2429 will bring exactly that starting with the next Solr release – 3.4. It is simple to use, just add cache=false to your queries like this: fq={!frange l=10 u=100 cache=false}mul(popularity,price).  Note that with this new functionality you can prevent either a filter or a query to be cached, while document caching still remains out of request-time control.
• If you’re using JMX to observe the state of your Solr installation, you might have encountered a problem when reloading Solr cores – it appears that JMX beans didn’t survive those reloads in the past versions. The fix is created and is available in future 3_x and trunk releases.

Interesting features in development

• To achieve case-insensitive search with wildcard queries you could use a patch suplied under issue SOLR-2438. It has to be said that this isn’t committed to svn and it is hard to say whether it ever will be since there is a similar issue SOLR-219 on which work started 4 years ago.
• Multithreaded faceting might bring some performance improvements. At the moment, initial patch exists, but more work will be needed here and it still isn’t clear how big improvement we could expect in real-world conditions, but it is worth keeping an eye on this issue.
• We all know that Solr’s Spatial support has its limitations. One of them is specifying bounding box which isn’t based on point distance, effectively making it limited to a circular shape. Under SOLR-2609 we might get support for exactly this.
• For anyone interested in which direction Spatial support might evolve, we suggest checking Lucene Spatial Playground. It continues the great work done in SOLR-2155 which provided extension to initial GeoSpatial support in Solr by adding multivalued spatial fields. At some point, SOLR-2155 might get the goodness from LSP. Also, another thing to check would be a thread on Lucene Spatial Future.

Interesting new features

• Support for Lucene’s Surround Parser is added to Solr in issue SOLR-2703. The patch is already committed to the trunk.
• Solr will get the ability to use configuration like analyzer type=”phrase”. Lucene’s Query Parsers recently got a simpler way to use different analyzer based on the query string. One example is usage of double quotes where one can decide that instead of current meaning in Lucene/Solr world – specifying a phrase to be searched for – it should have a meaning like in Google’s search engine – find this exact wording. Patch for this exists and can be applied on the trunk (it depends on Lucene trunk).
• SOLR-2593 aims to provide a new Solr core admin action – ‘split’ – for splitting index. It would be used in case some core got too big or in any other case you might find it necessary.  Lucene already has a similar function.

Miscellaneous

• Oracle released Java 7 about a month ago, but we advize against using it yet. JVM crashes and index corruption are issues likely to be encoutered with it. For more information, visit this URL
• As anticipated for some time, Java 5 support got axed from Lucene 4.0 (trunk). You can expect similar stuff for Solr too.
• Solr’s build system has been reworked now. Among other things, this implies changes in directory structure in Solr project. For example, solr/src/ doesn’t exist any more and its old subdirs /java and /test are now in solr/core/. The changes are already applied to the trunk and 3_x which holds the next 3.4 version. For more details, see SOLR-2452.
• A handy Solr architecture diagram can be found in ML thread
• Solr’s Admin UI is being refreshed with the work in JIRA issue SOLR-2399 (we already wrote about it) and its spin-off SOLR-2667. Some of this stuff is already committed (on the trunk), so you may want to inspect the changes. More details can be found in the wiki where you can also get the sneak-peak of the upcoming changes.

And that would be all for part one of the Solr Spring-Summer 2011 Digest edition from @sematext. Part two of the Spring-Summer Digest is coming in a few days – stay tuned!

Solr Digest, February-March 2011

We Sematexters have been very busy over the past few months, so we missed Solr’s February Digest. This one will therefore be a bit longer than usual.  Let’s get started…

First, some major news : Solr 3.1 is officially released! The details of the announcement can be found here. We covered most of the new features in our digests already, so we’ll keep it short:

• Numeric range facets (similar to date faceting)
• New spatial search, including spatial filtering, boosting and sorting capabilities
• Example Velocity driven search UI at http://localhost:8983/solr/browse
• A new termvector-based highlighter
• Extend dismax (edismax) query parser which addresses some missing features in the dismax query parser along with some extensions
• Several more components now support distributed mode: TermsComponent, SpellCheckComponent
• A new Auto Suggest component
• Ability to sort by functions
• JSON document indexing
• CSV response format
• Apache UIMA integration for metadata extraction
• Leverages Lucene 3.1 and it’s inherent optimizations and bug fixes as well as new analysis capabilities
• Numerous improvements, bug fixes, and optimizations

You can start your download .

Already committed features

• post.jar got improved – JIRA issue improve post.jar to handle non UTF-8 files removed some of its very old limitations
• jetty server included in Solr distribution didn’t support UTF-8. Now this is solved, fresh 3.1 version already contains this fix

Interesting features in development

• as part of SolrCloud, distributed indexing is being implemented in JIRA issue SOLR-2358. You can already see the work in progress in the initial patch, but you can also check SOLR-2341 which deals with shard distribution policies which will be available in Solr 4.0
• If you ever wanted to add custom fields (not existing in the index) to Solr responses, you couldn’t have done that from Solr components. There were other ways to achieve such functionality (for instance, customizing response writer class), but it looks like we’ll get such ability inside of components, too. No need to say how much more natural that would be. Anyway, issue Allow components to add fields to outgoing documents provides the umbrella for this new functionality. Although it is already closed, there are few sub-issues in which actual pieces of logic will be implemented.
• if you have problem with case sensitive searches in wildcard queries, you might take a look at a patch provided in JIRA issue Case Insensitive Search for Wildcard Queries
• although Solr got its first solid spatial implementation in version 3.1, many people found its limitations. One of them is surely a case where documents have multivalued spatial fields. We already wrote about SOLR-2155 in our December digest, but work under that issue hasn’t stopped and keeps evolving. It is likely that it will become a part of the standard Solr distribution and Lucene could get it incorporated, too. If you need spatial search you may want to watch this issue.

Interesting new features

• one common problem when using Solr’s default spellchecker or auto-suggest is filtering of suggestions based on what some user can see (for instance, depending on the region in which your user resides). JIRA issue Doc Filters for Auto-suggest, spell checking, terms component, etc. proposes a feature which would help here. Currently, no work was done there, though we believe we’ll get to see some progress in the future. While we are at it, in case you need such feature in Auto-suggest now, you might take a look at our in-house Search Auto-Complete solution, which you can see in action on search-lucene.com and search-hadoop.com.
• just like there are default components for SearchHandlers (which are used by default for every new search handler, unless overriden), update processors will get a similar feature. JIRA issue Let some UpdateProcessors be default without explicitly configuring them will take care that some important update processors are available by default to your UpdateRequestProcessorChain.
• one great new feature could be added to Solr – ability not to cache a filter. JIRA issue SOLR-2429 will deal with this. Many Solr users will be happy to optimize their cache performance when this feature is available some day.

Miscellaneous

• some interesting thoughts on spellchecker can be found in ML thread My spellchecker experiment and much more on that topic in the related blog
• should you use ASCIIFoldingFilter or MappingCharFilter when dealing with accents? Interesting discussion in thread Should ASCIIFoldingFilter be deprecated? could help you decide which one is right for you
• interesting idea for Solr’s admin UI can be found in this ML thread. Community’s reception was very good so we also got Solr Admin Interface, reworked issue as the home for this new work.
• anyone using Solr’s UIMA (Unstructured Information Management Architecture) contrib might be interested to know that its wiki page got a major improvement – more docs to read!
• we might be a bit late on this, but there is still some time left – Google’s Summer of Code applications can be submitted until 8th April. Check this ML thread for some detail.  And don’t forge that Sematext is sponsoring interns, too!
• new Solr/Lucene users should take a look at the Refcard provided by Erik Hatcher in ML thread [infomercial] Lucene Refcard at DZone
• some deep thoughts on Solr/Lucene’s release process by some of the key people can be found here Brainstorming on Improving the Release Process. Related to that is a JIRA issue Define Test Plan for 4.0 which will… eh, contain some info about Test plan for 4.0 release, obviously. Also, check the TestPlans wiki page that’s in the making.

Although there were some other interesting topics, we have to stop somewhere. Until next month, you’ll find us on Twitter.

Solr Digest, January 2011

Welcome to the second season of Sematext’s monthly Solr Digests. Once again, we compiled a list of most interesting topics in Solr world for the previous month:

Already committed features

• A bug related to using PHPSerialized response writer in sharded environment was fixed and committed in SOLR-2307. It affected all recent Solr versions (trunk, 3_x, 1.4.1,…) and the fix is committed to 3_x branch and trunk. In case you’re stuck with older version of Solr, you can manually try applying the patch, it should be doable.
• One old JIRA issue Enable sorting by Function Query is finally closed and committed to 3_x and trunk.
• A problem with race condition in StreamingUpdateSolrServer got its fixes before, however it appears that issue wasn’t fixed completely. Now another fix is committed to 3_x and trunk, so if you use this feature, we advise picking up the fix.

Interesting features in development

• Support for complex syntax (e.g. wildcards) in phrase queries is being brought to Lucene. In case you’re interested, you can take a look at LUCENE-1823 or LUCENE-1486 which was another try at similar functionality. These issues have been in development for a long time and still aren’t finished, although patches exist. Similar feature for Solr is developed under SOLR-1604, where you can also find some patches. However, we think it is a bit unclear if any of these issues will ever be committed to Lucene/Solr, so if you’re interested, check the progress on them occasionally and don’t hold your breath.

Interesting new features

• Solr might get improved per-field similarity integration into schema.xml. Currently, in Solr’s schema only global SimilarityProvider can be defined.

Miscellaneous

• Anyone having performance problems when using large start and rows parameters could benefit from looking at issue SOLR-2218. You can find some advice on how to deal with the problem using existing Solr capabilities.
• An interesting patch from issue Modify default solrconfig parameters via JMX aims to provide more flexibility in configuring Solr
• As usual, one common question is related to Solr/Lucene versions and release dates. In ML threads [Solr4.0] Release Date, Lucene 3.1 Release Proposal, Release schedule Lucene 4? and Is solr 4.0 ready for prime time? (or other ways to use geo distance in search) you can find more. In short, 3.1 is next version and might happen soon (March is being mention). 4.0 is a major release with many features not present in 3.1 and it is not likely that we will get it soon. Another ML thread provides insight into future release strategy.
• As usual, heated discussions are being held over Maven in Lucene/Solr world. If you’re interesting into reading what community thinks about Maven’s place in Lucene/Solr, we recommend reading (very long!) ML thread Let’s drop Maven Artifacts ! Discussions like this might eventually lead to Maven being dropped, and as a matter of fact, some sort of voting is already done in ML thread [POLL] Where do you get Lucene/Solr from?  Maven?  ASF Mirrors? where you can make your voice heard.
• A few interesting conferences are slowly approaching : Lucene Revolution 2011 is Coming – May 25 & 26 and Berlin Buzzwords 2011. Also, note that applications for Google’s Summer of Code must be submitted by the end of February.

And that’s all for January.