Overview

Configuration is part of Motr cluster meta-data. Configuration client library (confc) provides API for accessing configuration data. Confc obtains configuration data from the configuration server (confd) and caches this data in local memory.

Confd tries to obtain requested configuration data from its own cache. In case of cache miss, confd loads data from the configuration database and updates the cache.

Confd is a user-space service. Confc library is implemented in user space and in the kernel. Applications access configuration data by linking with confc library and using its API.

Definitions

Confc (configuration client library, configuration client): a library that provides configuration consumers with API to query Motr configuration.
Confd (configuration server): a management service that provides configuration clients with information obtained from configuration database.
Configuration consumer: any software that uses confc API to access Motr configuration. Alternative name: application.
Configuration cache: configuration data stored in node’s memory. Confc library maintains such a cache and provides configuration consumers with access to its data. Confd also uses configuration cache for faster retrieval of information requested by configuration clients.
Configuration object: a data structure that contains configuration information. There are several types of configuration objects: filesystem, service, node, etc.
Identity of a configuration object is a pair of its type and identifier.
Configuration object is a stub if its status is not equal to M0_CS_READY. Stubs contain no meaningful configuration data.
A configuration object is said to be pinned if its reference counter is nonzero; otherwise it is unpinned.
Relation: a pointer from one configuration object to another configuration object or to a collection of objects. In former case it is one-to-one relation, in the latter case it is one-to-many relation.
Downlink: a relation whose destination is located further from the "root" configuration object than the origin.

Requirements

r.conf.confc.kernel Confc library must be implemented for the kernel.
r.conf.confc.user Confc library must be implemented for user space.
r.conf.confc.async Confc library provides asynchronous interfaces for accessing configuration data.
r.conf.cache.data-model The implementation should organize configuration information as outlined in section 4.1 of the HLD. The same data structures should be used for confc and confd caches, if possible. Configuration structures must be kept in memory.
r.conf.cache.pinning Pinning of an object protects existence of this object in the cache. Pinned object can be moved from a stub condition to "ready".
r.conf.cache.unique-objects Configuration cache must not contain multiple objects with the same identity.

Dependencies

We assume that the size of configuration database is very small compared with other meta-data. Confd can take advantage of this assumption and load the entire contents of configuration database into the cache.
Motr database library ("db/db.h") should provide a user-space interface for creating in-memory databases. This interface will be used by confd and user-space confc.
m0_rpc_item_get() and m0_rpc_item_put() should be implemented.

Confc implementation schedules a state transition in ->rio_replied(). The data of ->ri_reply will be consumed only when the new state is being entered to. The rpc item pointed to by ->ri_reply must not be freed (by rpc layer) until confc has consumed the data. Thus the need for m0_rpc_item_get().

Design Highlights

The application should not use relations of a configuration object to access other objects.

Rationale: relations may point to unpinned objects. Confc implementation may convert unpinned objects into stubs. The application shall not use stubs, since they contain no valid configuration data.

See also
Private Fields
The registry of cached configuration objects (m0_conf_cache::ca_registry) is not expected to be queried frequently. It makes sense to base its implementation on linked list data structure.

Functional Specification

Logical Specification

Components Overview

Confc and confd maintain independent in-memory caches of configuration data.

Configuration cache can be pre-loaded from an ASCII string.

If a confc cache does not have enough data to fulfill a request of configuration consumer, confc obtains the necessary data from the confd and adds new configuration data to the cache.

If a confd cache does not have enough data to fulfill a request of confc, confd loads the necessary data from the configuration database and updates the cache.

State Specification

Threading and Concurrency Model

confc
confd

Conformance

i.conf.confc.kernel The implementation of confc uses portable subset of Motr core API, which abstracts away the differences between kernel and user-space code.
i.conf.confc.user Confc library is implemented for user space.
i.conf.confc.async m0_confc_open() and m0_confc_readdir() are asynchronous calls.
i.conf.cache.data-model Configuration information is organized as outlined in section 4.1 of the HLD. One-to-many relationships are represented by m0_conf_dir objects. The same data structures are used for both confc and confd. Configuration structures are kept in memory.
i.conf.cache.pinning Confc "pins" configuration object by incrementing its reference counter. m0_confc_fini() asserts (M0_PRE()) that no objects are pinned when the cache is being destroyed.
i.conf.cache.unique-objects Uniqueness of configuration object identities is achieved by using a registry of cached objects (m0_conf_cache::ca_registry).

Unit Tests

Fault Injection mechanism (lib/finject.h) will be used to test handling of "rare" errors (e.g., allocation errors) and to disable some of external modules' functionality (e.g., to make m0_rpc_post() a noop).

Infrastructure Test Suite

Test:: m0_conf_cache operations will be tested.

Test:: Path operations will be tested. This includes checking validity of various paths.

Test:: Object operations will be tested. This includes allocation, comparison with on-wire representation, stub enrichment.

Test:: m0_confstr_parse() will be tested.

confc Test Suite

Test suite's init routine will create an "ast" thread (search
for `ast_thread' in sm/ut/sm.c).  This thread will process ASTs
as they are posted by confc functions.

Test:: path_walk() will be tested.

Test:: m0_confc_open*() and m0_confc_close() will be tested.

Test:: Cache operations will be tested. This includes cache_add(), object_enrich(), cache_grow(), and cache_preload().

Scalability

Current design imposes no restrictions on the size of configuration cache. If a configuration database is huge and the application is keen to know every aspect of cluster configuration, confc cache may eventually consume all available memory. Confc will be unable to allocate new objects, its state machines will end in S_FAILURE state, and m0_confc_ctx_error() will return -ENOMEM. The application may opt to get rid of configuration cache by issuing m0_confc_fini().

XXX

Todo:: Implement cache eviction.

References

For documentation links, please refer to this file : doc/motr-design-doc-list.rst

HLD of configuration caching
HLD of configuration.schema
Configuration one-pager