3.4. Nouveau

Warning

Nouveau is an experimental feature. Future releases might change how the endpoints work and might invalidate existing indexes.

Nouveau indexes enable you to query a database by using the Lucene Query Parser Syntax. A nouveau index uses one, or multiple, fields from your documents. You can use a nouveau index to run queries to find documents based on the content they contain.

Warning

Nouveau cannot function unless it has a functioning Nouveau server. See Nouveau Server Installation for details.

To create a nouveau index, you add a JavaScript function to a design document in the database. An index builds after processing one search request or after the server detects a document update. The index function takes the following parameters:

  1. Field type - The type of the field, can be string, text, double or stored. See Field Types for more information.

  2. Field name - The name of the field you want to use when you query the index. If you set this parameter to default, then this field is queried if no field is specified in the query syntax.

  3. Data that you want to index, for example, doc.address.country.

  4. (Optional) The third parameter includes the following field: store.

By default, a nouveau index response returns 25 rows. The number of hits that are returned can be changed by using the limit parameter. Each response includes a bookmark field. You can include the value of the bookmark field in subsequent queries to fetch results from deeper in the result set.

Example design document that defines a nouveau index:

{
    "_id": "_design/nouveau_example",
    "nouveau": {
        "animals": {
            "index": "function(doc){ ... }"
        }
    }
}

A nouveau index will inherit the partitioning type from the options.partitioned field of the design document that contains it.

3.4.1. Field Types

Nouveau currently supports four field types, each of which has different semantics to the others.

Text

A text field is the most common field type, the field value is analyzed at index time to permit efficient querying by the individual words within it (and wildcards, and regex, etc). This field type is not appropriate for sorting, range queries and faceting.

String

A string field indexes the fields value as a single token without analysis (that is, no case-folding, no common suffixes are removed, etc). This field type is recommended for sorting and faceting. You can search on string fields but you must specify the keyword analyzer in the index definition for this field to ensure that your queries are not analyzed.

Double

A double field requires a number value and is appropriate for sorting, range queries and range faceting.

Stored

A stored field stores the field value into the index without analysis. The value is returned with search results but you cannot search, sort, range or facet over a stored field.

Warning

the type of any specific field is determined by the first index call. Attempts to index a different type into the same field will throw an exception and prevent the index from building.

3.4.2. Index functions

Attempting to index by using a data field that does not exist fails. To avoid this problem, use the appropriate guard clause.

Note

Your indexing functions operate in a memory-constrained environment where the document itself forms a part of the memory that is used in that environment. Your code’s stack and document must fit inside this memory. In other words, a document must be loaded in order to be indexed. Documents are limited to a maximum size of 64 MB.

The function that is contained in the index field is a JavaScript function that is called for each document in the database. The function takes the document as a parameter, extracts some data from it, and then calls the function that is defined in the index field to index that data.

The index function takes four parameters, where the third parameter is optional.

  1. The first parameter is the type of the field.

  2. The second parameter is the name of the field you intend to use when querying the index, and which is specified in the Lucene syntax portion of subsequent queries. An example appears in the following query:

    q=color:red

    The Lucene field name color is the first parameter of the index function.

    If the special value "default" is used when you define the name, you do not have to specify a field name at query time. The effect is that the query can be simplified:

    q=red

  3. The third parameter is the data to be indexed. Keep the following information in mind when you index your data:

    • This data must be only a string, number, or boolean. Other types will cause an error to be thrown by the index function call.

    • If an error is thrown when running your function, for this reason or others, the document will not be added to that search index.

  4. The fourth, optional, parameter is a JavaScript object with the following fields:

    Index function (optional parameter)

    • store - If true, the value is returned in the search result; otherwise, the value is not returned. Values are true or false. Default is false.

    Note

    If you do not set the store parameter, the index data results for the document are not returned in response to a query.

Example search index function:

function(doc) {
    if (typeof(doc.min_length) == 'number') {
        index("double", "min_length", doc.min_length, {"store": true});
    }
    if (typeof(doc.diet) == 'string') {
        index("string", "diet", doc.diet, {"store": true});
    }
    if (typeof(doc.latin_name) == 'string') {
        index("string", "latin_name", doc.latin_name, {"store": true});
    }
    if (typeof(doc.class) == 'string') {
        index("string", "class", doc.class, {"store": true});
    }
}

3.4.2.1. Index guard clauses

Runtime errors in the index function cause the document not to be indexed at all. The most common runtime errors are described below;

Example of failing to check whether the indexed value exists:

Warning

example of bad code

index("double", "min_length", doc.min_length, {"store": true});

For documents without a min_length value, this index call will pass undefined as the value. This will be rejected by nouveau’s validation function and the document will not be indexed.

Example of failing to check whether the nested indexed value exists:

Warning

example of bad code

if (doc.foo.bar) {
    index("string", "bar", doc.foo.bar, {"store": true});
}

This bad example fails in a different way if doc.foo doesn’t exist; the evaluation of doc.foo.bar throws an exception.

if (doc.foo && typeof(doc.foo) == 'object' && typeof(doc.foo.bar == 'string')) {
    index("string", "bar", doc.foo.bar, {"store": true});
}

This example correctly checks that doc.foo is an object and its bar entry is a string.

Example of checking the index value exists but disallowing valid false values:

Warning

example of bad code

if (doc.min_length) {
  index("double", "min_length", doc.min_length, {"store": true});
}

We correct the previous mistake so documents without min_length are indexed (assuming there are other index calls for values that do exist) but we’ve acccidentally prevented the indexing of the min_length field if the doc.min_length happens to be 0.

if (typeof(doc.min_length == 'number')) {
  index("double", "min_length", doc.min_length, {"store": true});
}

This good example ensures we index any document where min_length is a number.

3.4.3. Analyzers

Analyzers convert textual input into tokens which can be searched on. Analyzers typically have different rules for how they break up input into tokens, they might convert all text to lower case, they might omit whole words (typically words so common they are unlikely to be useful for searching), they might omit parts of words (removing ing suffixes in English, for example):

We expose a large number of Lucene’s analyzers. We invent one ourselves (simple_asciifolding);

  • arabic

  • armenian

  • basque

  • bulgarian

  • catalan

  • chinese

  • cjk

  • classic

  • czech

  • danish

  • dutch

  • email

  • english

  • finnish

  • french

  • galician

  • german

  • hindi

  • hungarian

  • indonesian

  • irish

  • italian

  • japanese

  • keyword

  • latvian

  • norwegian

  • persian

  • polish

  • portugese

  • romanian

  • russian

  • simple

  • simple_asciifolding

  • spanish

  • standard

  • swedish

  • thai

  • turkish

  • whitespace

Example analyzer document:

{
    "_id": "_design/analyzer_example",
    "nouveau": {
        "INDEX_NAME": {
            "index": "function (doc) { ... }",
            "default_analyzer": "$ANALYZER_NAME"
        }
    }
}

3.4.3.1. Field analyzers

You may optionally specify a different analyzer for a specific field.

Example of defining different analyzers for different fields:

{
    "_id": "_design/analyzer_example",
    "nouveau": {
        "INDEX_NAME": {
            "default_analyzer": "english",
            "field_analyzers": {
                "spanish": "spanish",
                "german": "german"
            },
            "index": "function (doc) { ... }"
        }
    }
}

3.4.3.2. Testing analyzer tokenization

You can test the results of analyzer tokenization by posting sample data to the _nouveau_analyze endpoint.

Example of using HTTP to test the keyword analyzer:

POST /_nouveau_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"keyword", "text":"ablanks@renovations.com"}

Example of using the command line to test the keyword analyzer:

curl 'https://$HOST:5984/_nouveau_analyze' -H 'Content-Type: application/json'
    -d '{"analyzer":"keyword", "text":"ablanks@renovations.com"}'

Result of testing the keyword analyzer:

{
    "tokens": [
        "ablanks@renovations.com"
    ]
}

Example of using HTTP to test the standard analyzer:

POST /_nouveau_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"standard", "text":"ablanks@renovations.com"}

Example of using the command line to test the standard analyzer:

curl 'https://$HOST:5984/_nouveau_analyze' -H 'Content-Type: application/json'
    -d '{"analyzer":"standard", "text":"ablanks@renovations.com"}'

Result of testing the standard analyzer:

{
    "tokens": [
        "ablanks",
        "renovations.com"
    ]
}

3.4.4. Queries

After you create a search index, you can query it.

  • Issue a partition query using: GET /$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/_nouveau/$INDEX_NAME

  • Issue a global query using: GET /$DATABASE/_design/$DDOC/_nouveau/$INDEX_NAME

Specify your search by using the q parameter.

Example of using HTTP to query a partitioned index:

GET /$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/_nouveau/$INDEX_NAME?include_docs=true&q=*:*&limit=1 HTTP/1.1
Content-Type: application/json

Example of using HTTP to query a global index:

GET /$DATABASE/_design/$DDOC/_nouveau/$INDEX_NAME?include_docs=true&q=*:*&limit=1 HTTP/1.1
Content-Type: application/json

Example of using the command line to query a partitioned index:

curl https://$HOST:5984/$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/
_nouveau/$INDEX_NAME?include_docs=true\&q=*:*\&limit=1 \

Example of using the command line to query a global index:

curl https://$HOST:5984/$DATABASE/_design/$DDOC/_nouveau/$INDEX_NAME?
include_docs=true\&q=*:*\&limit=1 \

3.4.4.1. Query Parameters

A full list of query parameters can be found in the API Reference.

Note

Do not combine the bookmark and update options. These options constrain the choice of shard replicas to use for the response. When used together, the options might cause problems when contact is attempted with replicas that are slow or not available.

3.4.4.2. Relevance

When more than one result might be returned, it is possible for them to be sorted. By default, the sorting order is determined by ‘relevance’.

Relevance is measured according to Apache Lucene Scoring. As an example, if you search a simple database for the word example, two documents might contain the word. If one document mentions the word example 10 times, but the second document mentions it only twice, then the first document is considered to be more ‘relevant’.

If you do not provide a sort parameter, relevance is used by default. The highest scoring matches are returned first.

If you provide a sort parameter, then matches are returned in that order, ignoring relevance.

If you want to use a sort parameter, and also include ordering by relevance in your search results, use the special fields -<score> or <score> within the sort parameter.

3.4.4.3. POSTing search queries

Instead of using the GET HTTP method, you can also use POST. The main advantage of POST queries is that they can have a request body, so you can specify the request as a JSON object. Each parameter in the query string of a GET request corresponds to a field in the JSON object in the request body.

Example of using HTTP to POST a search request:

POST /db/_design/ddoc/_nouveau/searchname HTTP/1.1
Content-Type: application/json

Example of using the command line to POST a search request:

curl 'https://$HOST:5984/db/_design/ddoc/_nouveau/searchname' -X POST -H 'Content-Type: application/json' -d @search.json

Example JSON document that contains a search request:

{
    "q": "index:my query",
    "sort": "foo",
    "limit": 3
}

3.4.5. Query syntax

The CouchDB search query syntax is based on the Lucene syntax. Search queries take the form of name:value unless the name is omitted, in which case they use the default field, as demonstrated in the following examples:

Example search query expressions:

// Birds
class:bird

// Animals that begin with the letter "l"
l*

// Carnivorous birds
class:bird AND diet:carnivore

// Herbivores that start with letter "l"
l* AND diet:herbivore

// Medium-sized herbivores
min_length:[1 TO 3] AND diet:herbivore

// Herbivores that are 2m long or less
diet:herbivore AND min_length:[-Infinity TO 2]

// Mammals that are at least 1.5m long
class:mammal AND min_length:[1.5 TO Infinity]

// Find "Meles meles"
latin_name:"Meles meles"

// Mammals who are herbivore or carnivore
diet:(herbivore OR omnivore) AND class:mammal

// Return all results
*:*

Queries over multiple fields can be logically combined, and groups and fields can be further grouped. The available logical operators are case-sensitive and are AND, +, OR, NOT and -. Range queries can run over strings or numbers.

If you want a fuzzy search, you can run a query with ~ to find terms like the search term. For instance, look~ finds the terms book and took.

Note

If the lower and upper bounds of a range query are both strings that contain only numeric digits, the bounds are treated as numbers not as strings. For example, if you search by using the query mod_date:["20170101" TO "20171231"], the results include documents for which mod_date is between the numeric values 20170101 and 20171231, not between the strings “20170101” and “20171231”.

You can alter the importance of a search term by adding ^ and a positive number. This alteration makes matches containing the term more or less relevant, proportional to the power of the boost value. The default value is 1, which means no increase or decrease in the strength of the match. A decimal value of 0 - 1 reduces importance. making the match strength weaker. A value greater than one increases importance, making the match strength stronger.

Wildcard searches are supported, for both single (?) and multiple (*) character searches. For example, dat? would match date and data, whereas dat* would match date, data, database, and dates. Wildcards must come after the search term.

Use *:* to return all results.

The following characters require escaping if you want to search on them:

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /

To escape one of these characters, use a preceding backslash character (\).

The response to a search query contains an order field for each of the results. The order field is an array where the first element is the field or fields that are specified in the sort parameter. See the sort parameter. If no sort parameter is included in the query, then the order field contains the Lucene relevance score.

3.4.5.1. Faceting

Nouveau Search also supports faceted searching, enabling discovery of aggregate information about matches quickly and easily. You can match all documents by using the special ?q=*:* query syntax, and use the returned facets to refine your query.

Example of search query:

function(doc) {
    index("string", "type", doc.type);
    index("double", "price", doc.price);
}

To use facets, all the documents in the index must include all the fields that have faceting enabled. If your documents do not include all the fields, you receive a bad_request error with the following reason, “The field_name does not exist.” If each document does not contain all the fields for facets, create separate indexes for each field. If you do not create separate indexes for each field, you must include only documents that contain all the fields. Verify that the fields exist in each document by using a single if statement.

Example if statement to verify that the required fields exist in each document:

if (typeof doc.town == "string" && typeof doc.name == "string") {
    index("string", "town", doc.town);
    index("string", "name", doc.name);
   }

3.4.5.2. Counts

Note

The counts option is only available when making global queries.

The counts facet syntax takes a list of fields, and returns the number of query results for each unique value of each named field.

Note

The count operation works only if the indexed values are strings. The indexed values cannot be mixed types. For example, if 100 strings are indexed, and one number, then the index cannot be used for count operations. You can check the type by using the typeof operator, and convert it by using the parseInt, parseFloat, or .toString() functions.

Example of a query using the counts facet syntax:

?q=*:*&counts=["type"]

Example response after using of the counts facet syntax:

{
    "total_rows":100000,
    "bookmark":"g...",
    "rows":[...],
    "counts":{
        "type":{
            "sofa": 10,
            "chair": 100,
            "lamp": 97
        }
    }
}

3.4.5.3. Ranges

Note

The ranges option is only available when making global queries.

The range facet syntax reuses the standard Lucene syntax for ranges to return counts of results that fit into each specified category. Inclusive range queries are denoted by brackets ([, ]). Exclusive range queries are denoted by curly brackets ({, }).

Note

The range operation works only if the indexed values are numbers. The indexed values cannot be mixed types. For example, if 100 strings are indexed, and one number, then the index cannot be used for range operations. You can check the type by using the typeof operator, and convert it by using the parseInt, parseFloat, or .toString() functions.

Example of a request that uses faceted search for matching ranges:

?q=*:*&ranges={"price":{"cheap":"[0 TO 100]","expensive":"{100 TO Infinity}"}}

Example results after a ranges check on a faceted search:

{
    "total_rows":100000,
    "bookmark":"g...",
    "rows":[...],
    "ranges": {
        "price": {
            "expensive": 278682,
            "cheap": 257023
        }
    }
}