control/

DAOS Control Plane (aka daos_server)

DAOS operates over two, closely integrated planes, Control and Data. The Data plane handles the heavy lifting transport operations while the Control plane orchestrates process and storage management, facilitating the operation of the Data plane.

DAOS Server implements the DAOS Control Plane and is written in Go. It is tasked with network and storage hardware provisioning and allocation in addition to instantiation and management of the DAOS IO Servers (Data Plane written in C) running on the same host. Users of DAOS will interact directly only with the Control Plane in the form of the DAOS Server and associated tools.

The DAOS Server implements the gRPC protocol to communicate with client gRPC applications and interacts with DAOS IO Servers through Unix domain sockets.

Multiple gRPC server modules are loaded by the control server. Currently included modules are security and management.

The Control Plane implements a replicated management service as part of the DAOS Server, responsible for handling distributed operations across the DAOS System.

The management tool is an example client application which can connect to both the agent to perform security functions (such as providing credentials and retrieving security contexts) and to the local management server to perform management functions (such as storage device discovery).

Documentation

• Management API
• Management internals
• Agent API
• Agent internals
• dRPC
• server package
• management tool package
• client package
• common package

Architecture

First a view of software component architecture:

There are operations that will be performed on individual nodes in parallel, such as hardware provisioning, which will execute through storage or network libraries.

Such broadcast commands (which will connect to a list of hosts) will usually be issued by the management tool, a gRPC client, and handled by the gRPC MgmtCtlServer running in daos_server. These commands will not traverse dRPC but will perform node-local functions such as hardware (network and storage) provisioning.

Other operations which will interact with the DAOS data plane through the replicated management service. Such operations will be triggered via a single control plane instance (access point) and use the replicated management service (running in the data plane) to perform a distributed operation such as creating a storage pool.

Commands which require connection to an access point will be forwarded to the data plane (iosrv), redirected by gRPC MgmtSvc over dRPC channel and handled by the mgmt module.

Development Requirements

• Go 1.10 or higher
• gRPC
• Protocol Buffers
• Dep for managing dependencies in vendor directory.

Development setup

• If changing vendor package versions, edit src/control/Gopkg.toml and then run dep ensure from src/control.
• (Optional) protoc protocol buffer compiler

Building the app

For instructions on building and running DAOS see the Quickstart guide.

Build with scons and binaries should be produced in install/bin directory.

Testing the app

Run the tests go test within each directory containing tests

Run unit tests locally

Checkout the DAOS source code:

git clone <https://github.com/daos-stack/daos.git>

Checkout the SPDK source code on branch v18.07.x:

git clone --single-branch --branch v18.07.x [email protected]:spdk/spdk.git

Continue installing SPDK with the procedure in the repository on branch v18.07.x: SPDK-v18.07.x

Setup environment variables:

DAOS_REPO="/path/to/daos_repo"
SPDK_REPO="/path/to/spdk_repo"
export CGO_LDFLAGS="-L${SPDK_REPO}/build/lib:${DAOS_REPO}/src/control/lib/spdk/src"
export CGO_CFLAGS="-I${SPDK_REPO}/include:${DAOS_REPO}/src/control/lib/spdk/include"
export LD_LIBRARY_PATH="${SPDK_REPO}/build/lib:${DAOS_REPO}/src/control/lib/spdk"

Build NVME libs:

cd ${DAOS_REPO}/src/control/lib/spdk
gcc ${CGO_LDFLAGS} ${CGO_CFLAGS} -Werror -g -Wshadow -Wall -Wno-missing-braces -c -fpic -Iinclude src/*.c -lspdk
gcc ${CGO_LDFLAGS} ${CGO_CFLAGS} -shared -o libnvme_control.so *.o

To run suite of control plane unit tests:

cd ${DAOS_REPO}/src/control
./run_go_tests.sh

To run the go-spdk tests:

cd ${DAOS_REPO}/src/control/lib/spdk
go test -v

Coding Guidelines

daos_server and daos_agent

Avoid calling os.Exit (or function with equivalent effects), except for assertion purposes. Fatal errors shall be returned back to main, who calls os.Exit accordingly.