3.4. Nouveau¶
Warning
Nouveau is an experimental feature. Future releases might change how the endpoints work and might invalidate existing indexes.
Nouveau indexes enable you to query a database by using the Lucene Query Parser Syntax. A nouveau index uses one, or multiple, fields from your documents. You can use a nouveau index to run queries to find documents based on the content they contain.
Warning
Nouveau cannot function unless it has a functioning Nouveau server. See Nouveau Server Installation for details.
To create a nouveau index, you add a JavaScript function to a design document in the
database. An index builds after processing one search request or after the server detects
a document update. The index
function takes the following parameters:
Field type - The type of the field, can be
string
,text
,double
orstored
. See Field Types for more information.Field name - The name of the field you want to use when you query the index. If you set this parameter to
default
, then this field is queried if no field is specified in the query syntax.Data that you want to index, for example,
doc.address.country
.(Optional) The third parameter includes the following field:
store
.
By default, a nouveau index response returns 25 rows. The number of hits that are returned
can be changed by using the limit
parameter. Each response includes a bookmark
field. You can include the value of the bookmark
field in subsequent queries to fetch
results from deeper in the result set.
Example design document that defines a nouveau index:
{
"_id": "_design/nouveau_example",
"nouveau": {
"animals": {
"index": "function(doc){ ... }"
}
}
}
A nouveau index will inherit the partitioning type from the options.partitioned
field
of the design document that contains it.
3.4.1. Field Types¶
Nouveau currently supports four field types, each of which has different semantics to the others.
- Text
A text field is the most common field type, the field value is analyzed at index time to permit efficient querying by the individual words within it (and wildcards, and regex, etc). This field type is not appropriate for sorting, range queries and faceting.
- String
A string field indexes the fields value as a single token without analysis (that is, no case-folding, no common suffixes are removed, etc). This field type is recommended for sorting and faceting. You can search on string fields but you must specify the
keyword
analyzer in the index definition for this field to ensure that your queries are not analyzed.- Double
A double field requires a number value and is appropriate for sorting, range queries and range faceting.
- Stored
A stored field stores the field value into the index without analysis. The value is returned with search results but you cannot search, sort, range or facet over a stored field.
Warning
the type of any specific field is determined by the first index call. Attempts to index a different type into the same field will throw an exception and prevent the index from building.
3.4.2. Index functions¶
Attempting to index by using a data field that does not exist fails. To avoid this problem, use the appropriate guard clause.
Note
Your indexing functions operate in a memory-constrained environment where the document itself forms a part of the memory that is used in that environment. Your code’s stack and document must fit inside this memory. In other words, a document must be loaded in order to be indexed. Documents are limited to a maximum size of 64 MB.
The function that is contained in the index field is a JavaScript function
that is called for each document in the database.
The function takes the document as a parameter,
extracts some data from it, and then calls the function that is defined
in the index
field to index that data.
The index
function takes four parameters, where the third parameter is optional.
The first parameter is the type of the field.
The second parameter is the name of the field you intend to use when querying the index, and which is specified in the Lucene syntax portion of subsequent queries. An example appears in the following query:
q=color:red
The Lucene field name
color
is the first parameter of theindex
function.If the special value
"default"
is used when you define the name, you do not have to specify a field name at query time. The effect is that the query can be simplified:q=red
The third parameter is the data to be indexed. Keep the following information in mind when you index your data:
This data must be only a string, number, or boolean. Other types will cause an error to be thrown by the index function call.
If an error is thrown when running your function, for this reason or others, the document will not be added to that search index.
The fourth, optional, parameter is a JavaScript object with the following fields:
Index function (optional parameter)
store - If
true
, the value is returned in the search result; otherwise, the value is not returned. Values aretrue
orfalse
. Default isfalse
.
Note
If you do not set the
store
parameter, the index data results for the document are not returned in response to a query.
Example search index function:
function(doc) {
if (typeof(doc.min_length) == 'number') {
index("double", "min_length", doc.min_length, {"store": true});
}
if (typeof(doc.diet) == 'string') {
index("string", "diet", doc.diet, {"store": true});
}
if (typeof(doc.latin_name) == 'string') {
index("string", "latin_name", doc.latin_name, {"store": true});
}
if (typeof(doc.class) == 'string') {
index("string", "class", doc.class, {"store": true});
}
}
3.4.2.1. Index guard clauses¶
Runtime errors in the index function cause the document not to be indexed at all. The most common runtime errors are described below;
Example of failing to check whether the indexed value exists:
Warning
example of bad code
index("double", "min_length", doc.min_length, {"store": true});
For documents without a min_length value, this index call will
pass undefined
as the value. This will be rejected by nouveau’s
validation function and the document will not be indexed.
Example of failing to check whether the nested indexed value exists:
Warning
example of bad code
if (doc.foo.bar) {
index("string", "bar", doc.foo.bar, {"store": true});
}
This bad example fails in a different way if doc.foo
doesn’t
exist; the evaluation of doc.foo.bar
throws an exception.
if (doc.foo && typeof(doc.foo) == 'object' && typeof(doc.foo.bar == 'string')) {
index("string", "bar", doc.foo.bar, {"store": true});
}
This example correctly checks that doc.foo
is an object and its
bar
entry is a string.
Example of checking the index value exists but disallowing valid false values:
Warning
example of bad code
if (doc.min_length) {
index("double", "min_length", doc.min_length, {"store": true});
}
We correct the previous mistake so documents without min_length are
indexed (assuming there are other index calls for values that do
exist) but we’ve acccidentally prevented the indexing of the
min_length
field if the doc.min_length
happens to be 0
.
if (typeof(doc.min_length == 'number')) {
index("double", "min_length", doc.min_length, {"store": true});
}
This good example ensures we index any document where min_length
is a number.
3.4.3. Analyzers¶
Analyzers convert textual input into tokens
which can be searched
on. Analyzers typically have different rules for how they break up
input into tokens, they might convert all text to lower case, they
might omit whole words (typically words so common they are unlikely to
be useful for searching), they might omit parts of words (removing
ing
suffixes in English, for example):
We expose a large number of Lucene’s analyzers. We invent one
ourselves (simple_asciifolding
);
arabic
armenian
basque
bulgarian
catalan
chinese
cjk
classic
czech
danish
dutch
email
english
finnish
french
galician
german
hindi
hungarian
indonesian
irish
italian
japanese
keyword
latvian
norwegian
persian
polish
portugese
romanian
russian
simple
simple_asciifolding
spanish
standard
swedish
thai
turkish
whitespace
Example analyzer document:
{
"_id": "_design/analyzer_example",
"nouveau": {
"INDEX_NAME": {
"index": "function (doc) { ... }",
"default_analyzer": "$ANALYZER_NAME"
}
}
}
3.4.3.1. Field analyzers¶
You may optionally specify a different analyzer for a specific field.
Example of defining different analyzers for different fields:
{
"_id": "_design/analyzer_example",
"nouveau": {
"INDEX_NAME": {
"default_analyzer": "english",
"field_analyzers": {
"spanish": "spanish",
"german": "german"
},
"index": "function (doc) { ... }"
}
}
}
3.4.3.2. Testing analyzer tokenization¶
You can test the results of analyzer tokenization by posting sample data to the
_nouveau_analyze
endpoint.
Example of using HTTP to test the keyword analyzer:
POST /_nouveau_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"keyword", "text":"ablanks@renovations.com"}
Example of using the command line to test the keyword analyzer:
curl 'https://$HOST:5984/_nouveau_analyze' -H 'Content-Type: application/json'
-d '{"analyzer":"keyword", "text":"ablanks@renovations.com"}'
Result of testing the keyword analyzer:
{
"tokens": [
"ablanks@renovations.com"
]
}
Example of using HTTP to test the standard analyzer:
POST /_nouveau_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"standard", "text":"ablanks@renovations.com"}
Example of using the command line to test the standard analyzer:
curl 'https://$HOST:5984/_nouveau_analyze' -H 'Content-Type: application/json'
-d '{"analyzer":"standard", "text":"ablanks@renovations.com"}'
Result of testing the standard analyzer:
{
"tokens": [
"ablanks",
"renovations.com"
]
}
3.4.4. Queries¶
After you create a search index, you can query it.
Issue a partition query using:
GET /$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/_nouveau/$INDEX_NAME
Issue a global query using:
GET /$DATABASE/_design/$DDOC/_nouveau/$INDEX_NAME
Specify your search by using the q
parameter.
Example of using HTTP to query a partitioned index:
GET /$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/_nouveau/$INDEX_NAME?include_docs=true&q=*:*&limit=1 HTTP/1.1
Content-Type: application/json
Example of using HTTP to query a global index:
GET /$DATABASE/_design/$DDOC/_nouveau/$INDEX_NAME?include_docs=true&q=*:*&limit=1 HTTP/1.1
Content-Type: application/json
Example of using the command line to query a partitioned index:
curl https://$HOST:5984/$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/
_nouveau/$INDEX_NAME?include_docs=true\&q=*:*\&limit=1 \
Example of using the command line to query a global index:
curl https://$HOST:5984/$DATABASE/_design/$DDOC/_nouveau/$INDEX_NAME?
include_docs=true\&q=*:*\&limit=1 \
3.4.4.1. Query Parameters¶
A full list of query parameters can be found in the API Reference.
Note
Do not combine the bookmark
and update
options. These options
constrain the choice of shard replicas to use for the response. When used
together, the options might cause problems when contact is attempted
with replicas that are slow or not available.
3.4.4.2. Relevance¶
When more than one result might be returned, it is possible for them to be sorted. By default, the sorting order is determined by ‘relevance’.
Relevance is measured according to Apache Lucene Scoring.
As an example, if you search a simple database for the word
example
, two documents might contain the word. If one document
mentions the word example
10 times, but the second document
mentions it only twice, then the first document is considered to be
more ‘relevant’.
If you do not provide a sort
parameter, relevance is used by default. The highest
scoring matches are returned first.
If you provide a sort
parameter, then matches are returned in that order, ignoring
relevance.
If you want to use a sort
parameter, and also include ordering by relevance in your
search results, use the special fields -<score>
or <score>
within the sort
parameter.
3.4.4.3. POSTing search queries¶
Instead of using the GET
HTTP method, you can also use POST
. The main advantage of
POST
queries is that they can have a request body, so you can specify the request as a
JSON object. Each parameter in the query string of a GET
request corresponds to a
field in the JSON object in the request body.
Example of using HTTP to POST a search request:
POST /db/_design/ddoc/_nouveau/searchname HTTP/1.1
Content-Type: application/json
Example of using the command line to POST a search request:
curl 'https://$HOST:5984/db/_design/ddoc/_nouveau/searchname' -X POST -H 'Content-Type: application/json' -d @search.json
Example JSON document that contains a search request:
{
"q": "index:my query",
"sort": "foo",
"limit": 3
}
3.4.5. Query syntax¶
The CouchDB search query syntax is based on the
Lucene syntax.
Search queries take the form of name:value
unless the name is omitted, in which case
they use the default field, as demonstrated in the following examples:
Example search query expressions:
// Birds
class:bird
// Animals that begin with the letter "l"
l*
// Carnivorous birds
class:bird AND diet:carnivore
// Herbivores that start with letter "l"
l* AND diet:herbivore
// Medium-sized herbivores
min_length:[1 TO 3] AND diet:herbivore
// Herbivores that are 2m long or less
diet:herbivore AND min_length:[* TO 2]
// Mammals that are at least 1.5m long
class:mammal AND min_length:[1.5 TO *]
// Find "Meles meles"
latin_name:"Meles meles"
// Mammals who are herbivore or carnivore
diet:(herbivore OR omnivore) AND class:mammal
// Return all results
*:*
Queries over multiple fields can be logically combined, and groups and fields can be
further grouped. The available logical operators are case-sensitive and are AND
,
+
, OR
, NOT
and -
. Range queries can run over strings or numbers.
If you want a fuzzy search, you can run a query with ~
to find terms like the search
term. For instance, look~
finds the terms book
and took
.
Note
If the lower and upper bounds of a range query are both strings that
contain only numeric digits, the bounds are treated as numbers not as
strings. For example, if you search by using the query
mod_date:["20170101" TO "20171231"]
, the results include documents
for which mod_date
is between the numeric values 20170101 and
20171231, not between the strings “20170101” and “20171231”.
You can alter the importance of a search term by adding ^
and a positive number. This
alteration makes matches containing the term more or less relevant, proportional to the
power of the boost value. The default value is 1, which means no increase or decrease in
the strength of the match. A decimal value of 0 - 1 reduces importance. making the match
strength weaker. A value greater than one increases importance, making the match strength
stronger.
Wildcard searches are supported, for both single (?
) and multiple (*
) character
searches. For example, dat?
would match date
and data
, whereas dat*
would
match date
, data
, database
, and dates
. Wildcards must come after the
search term.
Use *:*
to return all results.
The following characters require escaping if you want to search on them:
+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /
To escape one of these characters, use a preceding backslash character (\
).
The response to a search query contains an order
field for each of the results. The
order
field is an array where the first element is the field or fields that are
specified in the sort
parameter. See the
sort parameter. If no sort
parameter is included
in the query, then the order
field contains the Lucene relevance score.
3.4.5.1. Faceting¶
Nouveau Search also supports faceted searching, enabling discovery of aggregate
information about matches quickly and easily. You can match all documents by using the
special ?q=*:*
query syntax, and use the returned facets to refine your query.
Example of search query:
function(doc) {
index("string", "type", doc.type);
index("double", "price", doc.price);
}
To use facets, all the documents in the index must include all the fields that have
faceting enabled. If your documents do not include all the fields, you receive a
bad_request
error with the following reason, “The field_name
does not exist.” If
each document does not contain all the fields for facets, create separate indexes for each
field. If you do not create separate indexes for each field, you must include only
documents that contain all the fields. Verify that the fields exist in each document by
using a single if
statement.
The top_n
query parameter controls how many facets, per grouping, are returned,
defaulting to 10, to a maximum of 1000.
Example if statement to verify that the required fields exist in each document:
if (typeof doc.town == "string" && typeof doc.name == "string") {
index("string", "town", doc.town);
index("string", "name", doc.name);
}
3.4.5.2. Counts¶
Note
The counts
option is only available when making global queries.
The counts
facet syntax takes a list of fields, and returns the number of query
results for each unique value of each named field.
Note
The count
operation works only if the indexed values are strings.
The indexed values cannot be mixed types. For example,
if 100 strings are indexed, and one number,
then the index cannot be used for count
operations.
You can check the type by using the typeof
operator, and convert it
by using the parseInt
,
parseFloat
, or .toString()
functions.
Example of a query using the counts facet syntax:
?q=*:*&counts=["type"]
Example response after using of the counts facet syntax:
{
"total_rows":100000,
"bookmark":"g...",
"rows":[...],
"counts":{
"type":{
"sofa": 10,
"chair": 100,
"lamp": 97
}
}
}
3.4.5.3. Ranges¶
Note
The ranges
option is only available when making global queries.
The value of the range parameter is a JSON object where the fields names are double
fields, and the values of the fields are arrays of JSON objects. The objects
must have a label
, min
and max
value (of type string, double, double
respectively), and optional min_inclusive
and max_inclusive
properties
(defaulting to true
if not specified).
Example of a request that uses faceted search for matching ranges:
?q=*:*&ranges={"price":[{"label":"cheap","min":0,"max":"100","max_inclusive":false},{"label":"expensive","min":100}]}
Example results after a ranges check on a faceted search:
{
"total_rows":100000,
"bookmark":"g...",
"rows":[...],
"ranges": {
"price": {
"expensive": 278682,
"cheap": 257023
}
}
}