Mesos Module —— Hook 实践

发表于 更新于 分类于 阅读次数： 本文字数： 1.1k 阅读时长 ≈ 4 分钟

在容器化解决方案中，除了如日中天的k8s，mesos+marathon+docker 也算是另一种主流。维护mesos过程中，难免会产生一些新的需求，本文笔者介绍一个mesos module hook的开发实践。

简介

mesos module 可以实现 mesos 的 plugin 扩展，丰富 mesos 的功能. mesos 默认提供了几种module功能的支持. 详见参考.

mesos module 分类：

kind description
Allocator Provides available resources to schedulers.
Anonymous Coexists with parent process without providing any callbacks.
Authenticatee Provides the client side of an RPC authentication.
Authenticator Provides the server side of an RPC authentication.
Authorizer Allows certain principals to perform specific actions.
ContainerLogger Container logger modules implement container log management.
Hook Provides means of tying into specific callbacks.
HttpAuthenticatee Provides the client side of an HTTP authentication.
HttpAuthenticator Provides the server side of an HTTP authentication.
Isolator Enables experimenting with specialized isolation and monitoring.
MasterContender Contender modules allow implementing custom leader election mechanisms.
MasterDetector Detector modules allow implementing custom master detection mechanisms.
QoSController QoS modules allow providing task performance guarantees.
ResourceEstimator Estimator modules allow predicting the resources total used.
SecretResolver Secret resolver modules allow interfacing with secure data providers.
TestModule Used internally for testing purposes only.

使用

module 以 .so 库的形式，在 mesos 启动时候加载进去。因此比较方便. 可以与 mesos 源码一起编译构建，也可独立构建.

独立编译时需要加入相应的mesos依赖库就好，主要有以下几个，hook.cpp , libprocess, google-protobuf, glog,libblkid等.

本例中还依赖了其他 linux 系统头文件，kenerl-headers, xfsprods-devel 等，在此不赘述.

启动 mesos master 或 slave 时,需指定 –modules与**–hooks**(或**–isolators**)，例如：**–modules=/path/to/your/modules.json和–hooks=demo_mesos_xfsquotahook**.

modules.json 文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 {
   "libraries": [
     {
       "name": "foo",
       "modules": [
         {
           "name": "org_apache_mesos_bar",
           "parameters": [
             {
               "key": "X",
               "value": "Y"
             }
           ]
         },
         {
           "name": "org_apache_mesos_baz"
         }
       ]
     }
   ]
 }```

### hook开发

可供 hook 的接口定义在 `mesos/scr/hook.hpp` 文件。hook 开发就是实现对应接口。不同的时机调用的上下文也不同，因此开发时需要注意下接口调用的时机和并发相关的问题。一般来讲，mesos 是基于 libprocess 实现d  Actor 并发模型，不需要特别关注这一块。


### hook运行原理

hook 的运行由 HookManager 管理，源码在 `src/hook/manager.cpp` 中。HookManager 维护了一个 HashMap.

```C++
static LinkedHashMap<string, Hook*> availableHooks;

  • load
    在 slave 启动的时候，会解析**–modules**参数,调用ModuleManager::load()对应的.so库. ModuleManager 负责 module 的 load , unload 等生命周期.

  • initialize
    slave launch 的时候，会调用hookManager::initialize(),委托 hookManager 进行初始化，所有hook的 event 也是由 hookManager 转发处理，对 slave 基本是透明的.
    hookManager 根据**–hooks**参数指定的module名，调用ModuleManager::creat()初始化，按顺序被添加进 availableHooks 中.

  • Run
    不同 hook 时机，由不同的调用者，如 slaveProcess 或 containerizerProcess 或 executor，会调用 hookManager 对应的方法，执行Hook操作.
    前置检验由hookManager::available() 完成.

本例中hookslavePreLaunchDockerTaskExecutorDecorator是在dockercontainerizerProcess::launchTask() 中被调用.注:hookmanager 维护同一个 event 多个hook module 的调用顺序，一般而言，先load，先调用，也就是以启动参数--hooks={1}，{2}... 顺序调用.

构建和测试

注： build module 时会使用当前的 mesos 中的库，比如libmesos.so，libprocess.so等，因此module 与 mesos 最好配对使用，保持版本一致性.

Demo

本例为实现 XFS Quota操作, Hook 如下接口.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// This task and executor decorator is called from within the slave after
// receiving a run task request from the master but before the docker
// containerizer launches the task. A module implementing the hook can
// inspect the arguments and return a `Failure` if the task should be
// rejected with a `TASK_FAILED`.
// The hook can return a set of environment variables individually for
// both, the executor and the task. Note that for custom executors,
// the task environment variables, in case of conflicts, *will*
// overwrite the executor variables.
//
// NOTE: The order of hooks matters for environment variables.
// If there is a conflict, the hook loaded last will take priority.
//
// NOTE: Unlike `slaveExecutorEnvironmentDecorator`, environment variables
// returned from this hook, in case of conflicts, *will* overwrite
// environment variables inside the `ExecutorInfo`.
virtual process::Future<Option<DockerTaskExecutorPrepareInfo>>
  slavePreLaunchDockerTaskExecutorDecorator(
      const Option<TaskInfo>& taskInfo,
      const ExecutorInfo& executorInfo,
      const std::string& containerName,
      const std::string& containerWorkDirectory,
      const std::string& mappedSandboxDirectory,
      const Option<std::map<std::string, std::string>>& env)
{
  return None();
}

关键代码,由三部分组成.

  • Hook实体类.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
class QuotaHook : public ::mesos::Hook {
public:
    virtual Future <Option<DockerTaskExecutorPrepareInfo>> slavePreLaunchDockerTaskExecutorDecorator(
            const Option <TaskInfo> &taskInfo,
            const ExecutorInfo &executorInfo,
            const string &containerName,
            const string &containerWorkDirectory,
            const string &mappedSandboxDirectory,
            const Option <map<string, string>> &env) {

        LOG(INFO) << "Executing 'slavePreLaunchDockerTaskExecutorDecorator' hook name " << containerName << " workdir "
                  << containerWorkDirectory << " mappedSandboxDirectory " << mappedSandboxDirectory;

        //parse params
        Try <QuotaFlags> flags = parse(env);
        if (!flags.isError() && flags.get().enable) {
           Try<Nothing> status =  quotaProcess(flags.get().path, containerName, flags.get().hardLimit);
           if (status.isError()){
               return Failure("[Error] Fail to set xfs quota because " + status.error());
           }
        }

      //TODO add labels in prepareInfo byproduct is OK?
        DockerTaskExecutorPrepareInfo prepareInfo;
        Environment *taskEnvironment = prepareInfo.mutable_taskenvironment();
        Environment::Variable *variable = taskEnvironment->add_variables();
        variable->set_name("QOUTAHOOK_TASK");
        variable->set_value("run");

        Environment *executorEnvironment = prepareInfo.mutable_executorenvironment();
        variable = executorEnvironment->add_variables();
        variable->set_name("QOUTAHOOK_EXECUTOR");
        variable->set_value("run");
        return prepareInfo;
    }
private:
//...此处省略具体实现
}
  • 静态实例化入口.
1
2
3
static Hook *createHook(const Parameters &parameters) {
    return new QuotaHook();
}
  • module 声明.
1
2
3
4
5
6
7
8
9
// Declares a Hook module named ''.
mesos::modules::Module<Hook> demo_mesos_xfsquotahook(
        MESOS_MODULE_API_VERSION,
        MESOS_VERSION,
        "Company",
        "xxxxx@yyyy.zzz",
        "XFS Quota Hook",
        nullptr,
        createHook);

参考资料