Mercator

An Enhanced Developer Experience created by Roxoft.

Get Started

Built on and for Redis

Mercator is a set of open source modules to enhance the Developer Experience for Redis by adding data access capabilities beyond Redis basic get/set operations. the preview page.

Mercator can be loaded on your existing Redis (version 6 and later) installation without any modification to the Redis Server code.

...

rxIndex

The rxIndexer is loaded on any Redis Node to add automatic full text indexing of the data on the Redis node.

No need to reload your data. rxIndexer starts indexing as soon as it loaded. Existing data may be indexed by invoking a re-index command.

No need to modify your application. rxIndexer just works on the standard dml commands.

...

rxIndexStore

rxIndexStore changes the role of a Redis Node to Index node.


rxIndexStore maintains the actual full text index. rxIndexStore works with rxIndexer and is not used directly by the Developer or System Administrator.


rxIndexStore support rxQuery by providing index retrieval operations.

...

rxQuery

rxQuery operates on the index build on the Index nodes.


rxQuery brings set operations to Redis using a schema free Query Language, e.g:

rxquery tesla y

will return all keys with a value containing tesla or y., and:

rxquery tesla & y

will return all keys with a value containing tesla and y.


rxQuery can return all keys with values that match a query expression or the key/value pairs can be returned.

Optionally the key can be delivered to a LUA script.


...

rxGraphDb

rxGraphDb enables REDIS to load RDF style data and turn REDIS in a graph database.


The data to be loaded is formatted as a JSON Array with object-predicate-subject triplets.


The object, subject and predicate are translated into a graph with edges and vertices.


The loaded data can be accessed and manipulated with standard REDIS commands and using the commands provided by the various Mercator modules.

...

rxQuery Graph traversal

rxQuery can execute set based queries as well as gremlin like graph traversal expressions.


Assuming a grapg db with family relations, the following query will return all parents and their childrenr:

rxquery g.v.in(vader,moeder).by(object)

Set based query operations and graph traversal may be combined in a single query.

rxquery g.v.( as(all).in(vader) | as(all).in(moeder) ).by(object)
...

rxRules

rxRules add business rules (or triggers) to REDIS.


Business Rules (or triggers) can be used to pre-create key sets for rxQuery and/or to add or modify the database graph.


Business Rules are executed after every 'Write&Apos; operation on a REDIS key.

The following sample rule just adds a key to a keyset when the rule evaluates to true, or removes the key from key set when the rule evaluates to false.

RULE.SET $$ISCOUNTRY has(type,country)

The following sample rule add or removes a property and add or removes the key from the names key set based on the outcome of the expression.

RULE.SET $$KNOWNPARENTS (as(aKey).hasin(father)|as(aKey).hasin(mother)).property(known_parent,yes)

Frequently Asked Question

General

Why Mercator?

In ancient times, navigation at sea was limited to daytime and following coast lines and visible landmarks.

World maps and positioning tools enable seafarers to go beyond the limits of their time. Gerardus Mercator was a well known cartographer who produced a new world map that formed the bases for all nautical scharts since then.


Likewise, Redis provides a basic set of commands to memorize and recall data.

Being developed of a high performance cache, recent developments attempt to transform Redis in a modern in-memory database. With the Mercator modules we support this developement. With the Mercatopr modules Redis is advanced to the next level as in in-memory no-sql database.


What features will Mercator add to my REDIS clusters?

Mercator will add the following features to your clusters:

  • Automatic full text indexing.
  • Schema free, almost natural, query language.
  • GraphDb capabilities.
  • Graph traversal.
  • Business Rules or Triggers.

    • How is Mercator Integrated in my REDIS cluster?

    Mercator is a set of REDIS modules which can be load using the REDIS MODULE LOAD command. The modules can be unloaded as needed using the REDIS MODULE UNLOAD command.
    The REDIS Server code is not changed. Some commands from the Mercator Suite may pre or post process standard REDIS commands.

    Which Redis versions are supported?

    Mercator is developed end verified on Redis 6. Newer will be certified soon after the initial release. Since Mercator is few to none source code dependencies, any dependencies are abstracted in a seperate 'C' module.

    Which programming language is used for Mercator?

    Mercator is primarely developed in C++ with some server dependency abstractions in C .


    The initial developement was in C, then an attempt was made to refactor the modules in Rust. However Rust required a steep learning curve and missed features available from C++. Hence, most code has been refactored to C++ .

    Rust is promoted by Redis Labs for Redis Module development, why is Mercator not developed in Rust?

    An earnest effort was taken to develop ysing Rust. And yes, Rust has great features. And Yes, some things are developed easier in C++.
    The good thing of Rust is the many ready to use modules. However the endless conversion from various string types became quite a frustration. In addition, having a class hierarchy proofed to be very helpful.
    So while Rust certainly may be considered for many projects, it was decided that Mercator would benefit from using C++.

    Redis is known for its unparalled speed. How is the performance affected by Mercator?

    It's true, the performance of REDIS stunning and some service providers has even improved those numbers..


    It's also true that the performance of REDIS greatly depends on your workload. A high workload with big keys and values can affect the responsiveness of REDIS, as do LUA scripts.


    Your server instance selection is mainly driven by memory size. However server instances with more memory comes often with more cpu cores.

    Redis however benefits from the added memory but is not fully taking advantage of the extra processing power since the command processor is single threaded.

    Mercator will paralize work were possible, this way adding minimal delays to regular command processing.

    Mercator commands will be execution is duplexed with regular commands avoiding Redis becoming unresponsive. While being duplexed the command is blocked, so that concurrent commands can be executed.

    With what Redis features is Mercator compatible?

    Feature Initial Release Planned
    Non Cluster Mode Yes
    Cluster Mode No Yes
    Non Cluster Mode
    Single databases Yes
    Multiple databases No Yes
    ACL No Yes
    Replication No Yes
    Master failover Yes
    Index on remote Redis cluster Yes
    Index on same Redis Node, different DB No Yes
    Cluster Mode
    Replication No Yes
    ACL No Yes
    Master failover Yes
    Index on remote Redis cluster Yes
    'One Cluster' No Yes

    To learn more about Redis configuration options with rxMercator: Watch our demonstration on configurations.


    Full Text Indexing

    Do I have to modify my application and reload my data?

    NO! Unlike other Redis Indexing modules, rxIndexer just taps into any supported write commands to index. Existing data can be indexed by executing:

    Which key types are indexed?

    The indexer post processes the following REDIS key types:

    Key Type Value or Field Value Type
    String String The string value is parsed into verbs. Each verb is added as a key reference to the index without a field name. The relevance for each verb is 1 / the number of verbs.
    Hash String Each field value is parsed into verbs. Each verb is added as a key reference to the index with the field name as classifier. The relevance for each verb is 1 / the number of verbs.
    Sample simple text value indexed
    Sample Redis Command
    SET FR "France stripes vertical red white blue euro french "
    France
    Verb Field Relevance Key
    france 0.222 fr
    stripes 0.111 fr
    vertical 0.111 fr
    euro 0.111 fr
    blue 0.111 fr
    white 0.111 fr
    red 0.111 fr
    String Text The value is divided into sentences, subsentences and enumerations. Each section is parsed into verb and each verb is added as a key reference to the index without a field name. The relevance for each verb is 1 / the number of verbs in the section.

    Enumeration are sections like : , ...
    Hash Text The each field value is divided into sentences, subsentences and enumerations. Each section is parsed into verb and each verb is added as a key reference to the index with the field name as classifier. The relevance for each verb is 1 / the number of verbs in the section.

    Enumeration are sections like : , ...
    Sample text indexed, all enumerations.
    Sample Redis Command
    SET FR 'country: France; pattern: stripes; orientation: vertical; colors: red white blue; currency: euro; languages: french; emblem: ""'
    Mercator Index Structure
    France
    Verb Field Relevance Key
    france country 1.0 fr
    france languages 1.0 fr
    stripes pattern 1.0 fr
    vertical orientation 1.0 fr
    euro currency 0.111 fr
    blue colors 0.333 fr
    white colors 0.333 fr
    red colors 0.333 fr
    Sample text indexed, mixed subsentence and enumerations.
    Sample Redis Command
    SET FR 'france; stripes; vertical; colors: red white blue; french; ""'
    Mercator Index Structure
    France
    Verb Field Relevance Key
    france 0.5 fr
    stripes 1.0 fr
    vertical 1.0 fr
    euro 1.0 fr
    blue colors 0.333 fr
    white colors 0.333 fr
    red colors 0.333 fr
    String JSON
    1. The JSON object is parsed, into field/value pairs using the build-in JSON parser.
    2. Each field value is parsed and processed as described above under string, text or JSON values.
    Hash JSON The each field value is processed as described above for a STRING key.
    Sample JSON value.
    Sample Redis Command
    SET FR '{"country": "France", "pattern": "stripes", "orientation": "vertical", "colors": "red white blue", "currency": "euro", "languages": "french", "emblem": ""'
    Mercator Index Structure
    France
    Verb Field Relevance Key
    france country 1.0 fr
    france languages 1.0 fr
    stripes pattern 1.0 fr
    vertical orientation 1.0 fr
    euro currency 0.111 fr
    blue colors 0.333 fr
    white colors 0.333 fr
    red colors 0.333 fr


    RXREINDEX
    Rxreindex is duplexed and will not block the command processing loop.
    Normal Redis commands, including 'WRITE' commands, can be executed will re-indexing is in progress.

    Full Text Indexing/ Query Language

    Why is a relevance factor calculated?

               

    Look at the three flags above. These flags share some of the same colors. (the Danish flag is defined here as a red flag with a white cross. So the main color is red). However when the flags database would be queried for red flags, which flags match? Surely, all flags has a degree of red, but the Danish flag is more red than the Maltese flag and the Maltese flag is more red than the France flag.
    In order to rate the answer to a query and to determine which answer are more relevant than others, during indexing a relevance factor is calculated. As described below:

    th>
    Sample Redis Command
    SET FR '{"country": "France", "pattern": "stripes", "orientation": "vertical", "colors": "red white blue", "currency": "euro", "languages": "french", "emblem": '" }'
    Mercator Index Structure
    France
    Verb Field Relevance Key
    blue flag-colors 0.333 france
    white flag-colors 0.333 france
    red flag-colors 0.333 france
    Sample Redis Command
    SET MT '{"country": "Malta", "pattern": "stripes", "orientation": "horizontal", "colors": "white,red", "icon": "cross", "icon-color": "silver gray", "icon-position": "top left", "emblem": "" }'
    Malta
    Verb Field Relevance Key
    red flag-colors 0.5 malta
    white flag-colors 0.333 malta
    Sample Redis Command
    HMSET hmset DK country Denmark pattern solid icon '40/60 cross' colors red icon-color 'white' icon-position "center left" emblem ""
    Denmark
    Verb Field Relevance Key
    red flag-colors 1.0 denmark



    Below some queries and results
    RXQUERY Results
    red
    Key Score
    denmark 1.0
    malta 0.5
    france 0.333
    white & red
    Key Score
    malta 1.0
    france 0.666
    white red

    By default, the missing operators are replaced by OR, can be changed programmatically.

    RXQUERY white red renders the same result as RXQUERY red | white.


    How are keys indexed?

    Every SET type command is intercepted the key and value are queued for indexing and the command is normally executed.
    The indexing queue is executed on a secondary thread. The indexer parses the values and creates REDIS Transactions (MULTI/EXEC) which are enqueued on the Redis Index Update queue.
    The Redis Index Update queue executes the index update transactions on the REDIS node designated as Index Node .

    What is a key set and how is it used?

    Any query is parsed into an reverse polish expression which can easily and efficient be executed.

    Subscribe to receive updates!

    Address

    16924 4th Ave NE, Shoreline WA

    Email

    rxmercator@gmail.com

    Phone

    +1 (206) 972-2764