HA notifications support in RM.

Data Structures
struct	m0_rm_ha_tracker
struct	m0_rm_ha_subscriber
struct	m0_rm_ha_event

Enumerations
enum	m0_rm_ha_subscriber_state {   RM_HA_SBSCR_INIT, RM_HA_SBSCR_FS_OPEN, RM_HA_SBSCR_DITER_SEARCH, RM_HA_SBSCR_FAILURE,   RM_HA_SBSCR_FINAL }

Functions
static void	rm_ha_sbscr_state_set (struct m0_rm_ha_subscriber *sbscr, int state)
static void	rm_ha_sbscr_fail (struct m0_rm_ha_subscriber *sbscr, int rc)
static void	rm_ha_sbscr_ast_post (struct m0_rm_ha_subscriber *sbscr, void(*cb)(struct m0_sm_group *, struct m0_sm_ast *))
static bool	rm_ha_rms_is_located (struct m0_conf_obj *next, struct m0_rm_ha_subscriber *sbscr)
static bool	rm_ha_svc_filter (const struct m0_conf_obj *obj)
static void	rm_ha_sbscr_diter_next (struct m0_rm_ha_subscriber *sbscr)
static void	rm_ha_sbscr_diter_next_ast (struct m0_sm_group *grp, struct m0_sm_ast *ast)
static bool	rm_ha_diter_cb (struct m0_clink *clink)
static void	rm_ha_sbscr_fs_opened (struct m0_sm_group *grp, struct m0_sm_ast *ast)
static bool	rm_ha_sbscr_fs_open_cb (struct m0_clink *link)
static void	rm_ha_conf_open (struct m0_sm_group *grp, struct m0_sm_ast *ast)
static int	rm_remote_ep_to_rms_obj (struct m0_confc *confc, const char *rem_ep, struct m0_conf_obj **obj)
static bool	rm_ha_conf_expired_cb (struct m0_clink *cl)
M0_INTERNAL int	m0_rm_ha_subscriber_init (struct m0_rm_ha_subscriber *sbscr, struct m0_sm_group *grp, struct m0_confc *confc, const char *rem_ep, struct m0_rm_ha_tracker *tracker)
M0_INTERNAL void	m0_rm_ha_subscribe (struct m0_rm_ha_subscriber *sbscr)
M0_INTERNAL int	m0_rm_ha_subscribe_sync (struct m0_confc *confc, const char *rem_ep, struct m0_rm_ha_tracker *tracker)
M0_INTERNAL void	m0_rm_ha_subscriber_fini (struct m0_rm_ha_subscriber *sbscr)
M0_INTERNAL void	m0_rm_ha_tracker_init (struct m0_rm_ha_tracker *tracker, m0_chan_cb_t cb)
M0_INTERNAL void	m0_rm_ha_tracker_fini (struct m0_rm_ha_tracker *tracker)
M0_INTERNAL void	m0_rm_ha_unsubscribe (struct m0_rm_ha_tracker *tracker)
M0_INTERNAL void	m0_rm_ha_unsubscribe_lock (struct m0_rm_ha_tracker *tracker)

Variables
static struct m0_sm_state_descr	rm_ha_subscriber_states []
static const struct m0_sm_conf	rm_ha_sbscr_sm_conf
enum m0_rm_ha_subscriber_state	M0_XCA_DOMAIN

Detailed Description

Local RM owners communicate with remote RM owners operating in context of RM service located on remote network node. HA can make decision about RM service failure and notify all nodes about it. Locally, acceptance of such notification means HA state change of corresponding configuration object in confc instance to M0_NC_FAILED. Also, HA can make decision that RM service is recovered froom failure and is online again. Local RM owners should track these HA state changes and take appropriate actions on them.

Local RM owners create a subscription to HA notifications about remote RM services. Effectively, subscription means attaching clink to corresponding RM service configuration object channel (m0_conf_obj::co_chan). Subscription can be made synchronously through m0_rm_ha_subscribe_sync() function, or asynchronously using m0_rm_ha_subscriber structure and related functions. RM service configuration object is searched by user-provided remote endpoint in user-provided confc instance.

There are two types of remote owners in relation to local owners: debtors and creditors. They have different subscription policies.

Subscription to debtor's HA state change is created when the first request from the remote is received (

See also

rfom_debtor_subscribe()). RPC endpoint is extracted from RPC connection. This subscription is done automatically inside RM code. Result of subscription doesn't affect overall result of incoming request processing.

Subscription to creditor's HA state change isn't created automatically by RM code. The reason is that creditor remote structure initialised and managed by RM user and provided to m0_rm_owner_init(). So RM user should create subscription manually. Not all users are interested in tracking creditor failures (assume short-lived local owner that is created to make a few credit requests). Also, confc can be not ready at the point of local RM owner creation (for example, rconfc requests read lock when confc is not ready yet). If local RM owner observes creditor failure notified by HA, then it invokes "self-windup" procedure ending up in ROS_DEAD_CREDITOR or ROS_INSOLVENT state. The reason for such behaviour is that RM owner can't functioning properly without creditor. RM user can detect that fact by monitoring owner state and react properly. For example, switch to another creditor using m0_rm_owner_creditor_reset(). Also local RM owner can recover from ROS_DEAD_CREDITOR back to ROS_ACTIVE if HA state of creditor returns back to M0_NC_ONLINE.

Concrete actions taken upon remote owner HA state changes are implemented in rm_remote_death_ast()/rm_remote_online_ast() functions.

Note

Several remote RM owners can co-exist in one remote RM service. In this case several subscriptions (by the number of owners) will be created instead of one. It's a little overhead, but it eases implementation.

Enumeration Type Documentation

m0_rm_ha_subscriber_state

enum m0_rm_ha_subscriber_state

Enumerator
RM_HA_SBSCR_INIT
RM_HA_SBSCR_FS_OPEN
RM_HA_SBSCR_DITER_SEARCH
RM_HA_SBSCR_FAILURE
RM_HA_SBSCR_FINAL

Definition at line 87 of file rm_ha.h.

Function Documentation

m0_rm_ha_subscribe()

M0_INTERNAL void m0_rm_ha_subscribe

(

struct m0_rm_ha_subscriber *

sbscr

)

Start asynchronous subscription process.

User should wait until m0_rm_ha_subscriber::rhs_sm is in one of (RM_HA_SBSCR_FINAL, RM_HA_SBSCR_FAILURE) states. If subscriber state machine ends up in RM_HA_SBSCR_FINAL state, then subscription is successful and RM HA tracker clink is attached to found RM service configuration object.

Definition at line 355 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_subscribe_sync()

M0_INTERNAL int m0_rm_ha_subscribe_sync	(	struct m0_confc *	confc,
		const char *	rem_ep,
		struct m0_rm_ha_tracker *	tracker
	)

Subscribe to HA notifications about RM service failure synchronously. Arguments have the same meaning as in m0_rm_ha_subscriber_init().

Todo:

Ideally this function should be implemented through asynchronous m0_rm_ha_subscribe(). The problem is that m0_rm_ha_subscriber internally locks confc sm group. For global confc it is locality0 sm group. So this group can't be provided to m0_rm_ha_subscriber_init(). Usually users requesting synchronous operation don't have another option, except locality0 sm group.

Todo:

What if remote is already in M0_NC_FAILED state? We should check it somewhere.

Definition at line 360 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_subscriber_fini()

M0_INTERNAL void m0_rm_ha_subscriber_fini

(

struct m0_rm_ha_subscriber *

sbscr

)

Finalise RM HA subscriber.

Definition at line 394 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_subscriber_init()

M0_INTERNAL int m0_rm_ha_subscriber_init	(	struct m0_rm_ha_subscriber *	sbscr,
		struct m0_sm_group *	grp,
		struct m0_confc *	confc,
		const char *	rem_ep,
		struct m0_rm_ha_tracker *	tracker
	)

Initialise RM HA subscriber.

Parameters

grp	sm group to execute asynchronous operations during subscription
confc	confc to search for remote RM service configuration object
rem_ep	RPC endpoint of the remote RM service. Serves as search criteria
tracker	initialised tracker structure

Definition at line 340 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_tracker_fini()

M0_INTERNAL void m0_rm_ha_tracker_fini

(

struct m0_rm_ha_tracker *

tracker

)

Finalise RM HA tracker.

Precondition

tracker is not subscribed.

Definition at line 408 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_tracker_init()

M0_INTERNAL void m0_rm_ha_tracker_init	(	struct m0_rm_ha_tracker *	tracker,
		m0_chan_cb_t	cb
	)

Initialise RM HA tracker structure.

Parameters

cb	Used to initialise internal clink. This clink will be attached to RM service configuration object when subscription is done.

Definition at line 399 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_unsubscribe()

M0_INTERNAL void m0_rm_ha_unsubscribe

(

struct m0_rm_ha_tracker *

tracker

)

Removes remote's subscription to HA notifications. It is safe to call it even if subscription wasn't done before or failed.

Precondition

Confc with object being tracked should be locked.

The object may be a fake one. Need to put only "real" configuration object

See also

_confc_phony_cache_append()

Definition at line 415 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

m0_rm_ha_unsubscribe_lock()

M0_INTERNAL void m0_rm_ha_unsubscribe_lock

(

struct m0_rm_ha_tracker *

tracker

)

Locked version of m0_rm_ha_unsubscribe().

Definition at line 435 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_conf_expired_cb()

static bool rm_ha_conf_expired_cb ( struct m0_clink *  cl )

static

Definition at line 330 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_conf_open()

static void rm_ha_conf_open ( struct m0_sm_group *  grp, struct m0_sm_ast *  ast  )

static

Definition at line 249 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_diter_cb()

static bool rm_ha_diter_cb ( struct m0_clink *  clink )

static

Definition at line 192 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_rms_is_located()

static bool rm_ha_rms_is_located ( struct m0_conf_obj *  next, struct m0_rm_ha_subscriber *  sbscr  )

static

Definition at line 103 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_ast_post()

static void rm_ha_sbscr_ast_post ( struct m0_rm_ha_subscriber *  sbscr, void(*)(struct m0_sm_group *, struct m0_sm_ast *)  cb  )

static

Definition at line 94 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_diter_next()

static void rm_ha_sbscr_diter_next ( struct m0_rm_ha_subscriber *  sbscr )

static

Todo:

What if obj is already in M0_NC_FAILED state? We should check it somewhere.

Definition at line 129 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_diter_next_ast()

static void rm_ha_sbscr_diter_next_ast ( struct m0_sm_group *  grp, struct m0_sm_ast *  ast  )

static

Definition at line 184 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_fail()

static void rm_ha_sbscr_fail ( struct m0_rm_ha_subscriber *  sbscr, int  rc  )

static

Definition at line 87 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_fs_open_cb()

static bool rm_ha_sbscr_fs_open_cb ( struct m0_clink *  link )

static

Definition at line 232 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_fs_opened()

static void rm_ha_sbscr_fs_opened ( struct m0_sm_group *  grp, struct m0_sm_ast *  ast  )

static

Definition at line 201 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_sbscr_state_set()

static void rm_ha_sbscr_state_set ( struct m0_rm_ha_subscriber *  sbscr, int  state  )

static

Definition at line 79 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_ha_svc_filter()

static bool rm_ha_svc_filter ( const struct m0_conf_obj *  obj )

static

Definition at line 124 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

rm_remote_ep_to_rms_obj()

static int rm_remote_ep_to_rms_obj ( struct m0_confc *  confc, const char *  rem_ep, struct m0_conf_obj **  obj  )

static

Helper function to locate RMS conf object with the given endpoint.

Definition at line 272 of file rm_ha.c.

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

M0_XCA_DOMAIN

enum m0_rm_ha_subscriber_state M0_XCA_DOMAIN

rm_ha_sbscr_sm_conf

const struct m0_sm_conf rm_ha_sbscr_sm_conf

static

Initial value:

= {
        .scf_name      = "rm_ha_subscriber",
        .scf_nr_states = ARRAY_SIZE(rm_ha_subscriber_states),
        .scf_state     = rm_ha_subscriber_states
}

Definition at line 73 of file rm_ha.c.

rm_ha_subscriber_states

struct m0_sm_state_descr rm_ha_subscriber_states[]

static

Initial value:

= {
        [RM_HA_SBSCR_INIT] = {
                .sd_flags   = M0_SDF_INITIAL,
                .sd_name    = "Initial",
                .sd_allowed = M0_BITS(RM_HA_SBSCR_FS_OPEN)
        },
        [RM_HA_SBSCR_FS_OPEN] =  {
                .sd_name    = "Open_configuration",
                .sd_allowed = M0_BITS(RM_HA_SBSCR_FAILURE,
                                      RM_HA_SBSCR_DITER_SEARCH)
        },
        [RM_HA_SBSCR_DITER_SEARCH] = {
                .sd_name    = "Diter_search",
                .sd_allowed = M0_BITS(RM_HA_SBSCR_FAILURE, RM_HA_SBSCR_FINAL)
        },
        [RM_HA_SBSCR_FAILURE] = {
                .sd_flags   = M0_SDF_FAILURE | M0_SDF_FINAL,
                .sd_name    = "Failure",
                .sd_allowed = 0
        },
        [RM_HA_SBSCR_FINAL] = {
                .sd_flags   = M0_SDF_TERMINAL,
                .sd_name    = "Final",
                .sd_allowed = 0
        }
}

Definition at line 46 of file rm_ha.c.