Skip to main content

Version 1.1: Search engine update

· 3 min read

In this series of posts, I will introduce the changes that are coming in version 1.1 (which in fact is already available as a pre-release).

For this first feature, I will let Naveen, our GSoC candidate, present it. Naveen has made an excellent job implementing various search engine features, including advanced filters, support for Okapi BM25 scoring algorithm, and fuzzy search. Here is what he has to say:

What got done​

The project consisted of three parts:

  1. Make search better by introducing additional search filters. (e.g., tags, notebook, type)
  2. Make the ranking of search results better by implementing the Okapi BM25 relevance function.
  3. Make fuzzy search possible.

Code contributions​

  1. All: Add search filtersJoplin's search had been using the Full-Text Search(FTS) offered by  SQLite directly. So though it was fast, it was not versatile. For  example, we can't restrict the search scope to a particular notebook or  search based on tags.The current search implementation fixes most of these problems. It  provides a better abstraction over FTS, supporting many new filters. The  documentation for the new search filters can be found here.
  2. All: Weigh notes using Okapi BM25 scoreJoplin used a ranking function based on the number of times the search query occurs in the note and how close they are.But there are better ways to rank notes, considering not just the  number of times a word appears, but how common it is. Words like "the"  is in most notes, while words like "zebra" are not common and should be  considered more relevant.The new search implementation uses Okapi BM25  as the ranking function. It ranks a set of documents based on the query  terms appearing in each document, regardless of their proximity.
  3. Desktop: Fuzzy searchWe've added support for fuzzy search. It doesn't replace the need to put  * at the end if you want to do a prefix search. But it does let you be a  bit more relaxed about the exact spelling. Searching for "tomatos" will  also give you results for "tomatoes". (This feature isn't released yet,  but will be soon)

It has been a pleasure working with Laurent and Caleb. They did an excellent job as mentors.

I hope these new features will be useful to many people. I plan to  continue as a member of the Joplin community and help in whatever way I  can.