Modular Build Architecture ​

To satisfy build scenarios and objectives while achieving flexible customization and extensibility of the build system, openUBMC reconstructed the build framework into a task orchestration framework. Its core comprises task standardization, describable targets, and automated scheduling.

• Task standardization: The framework provides a base task class that all tasks inherit. This ensures consistent parameter passing, task creation logic, unified task execution, and product collection, providing a uniform scheduling paradigm for the core scheduler.
• Describable targets: Text files are used to describe the task topology required to achieve build goals, including the task classes to be executed, dependencies, and subtasks.
• Automated scheduling: The scheduler analyzes target description files to obtain a complete task topology. Based on task dependencies and status, it creates multi-process instances of task classes to implement parallel building.

The modular build framework consists of global configuration, base task classes, task implementations, target description files, a multi-process scheduling framework, performance testing tools, and a global status management process.

Global Configuration ​

• To pass parameters when the modular solution calls tasks, a Config object that records a public configuration list is defined.

Base Work Class ​

• The Work class defines run and install methods and provides fundamental capabilities required for specific task implementations. Initialization of the base class requires a global configuration object, which serves as the input for subsequent task execution.

import sys
import os
import logging
from colorama import Fore, Back, Style
from iutils.config import Config
from multiprocessing import Process
cwd = os.path.join(os.getcwd(), os.path.dirname(__file__))
sys.path.append(f"{cwd}/../")

class Work(Process):
    name = "WorkBase"
    description="Base class"
    work_name = ""

    def __init__(self, config: Config, work_name=""):
        Process.__init__(self)
        self.config = config
        self.work_name = work_name

    # Subclasses must implement the run method. An exception is raised if the method is not implemented.
    def run(self):
        logging.info(f"{__file__} error")
        raise Exception

    # Installation function that installs the output of each work into the config.rootfs directory.
    def install(self):
        pass

    # Standard interface for command execution. The specific implementation is omitted.
    def run_command(self, command, ignore_error=False, show_log=False, sudo=False, command_echo=True, timeout=600, command_key = None):
      pass

    # Color-coded error logs. The specific implementation is omitted.
    def log_error(self, msg, frame_cnt=1):
      pass

    # Color-coded warning logs. The specific implementation is omitted.
    def log_warn(self, msg, frame_cnt=1):
      pass

Task Implementation ​

• Task implementations must inherit from the Work class. As defined in the Work class, tasks inherit multi-process execution capabilities from the Process class. The scheduling framework implements multi-process scheduling based on these Process capabilities.
• The following is a simple example of an environment initialization file:

#!/usr/bin/env python
# coding:utf-8

import os
import sys
import subprocess
import jsonschema
import json
import yaml
import logging
cwd = os.path.join(os.getcwd(), os.path.dirname(__file__))
sys.path.append(f"{cwd}/../")
sys.path.append(f"{cwd}/../../")
sys.path.append(f"{cwd}/../../../")
from works.work import Work
from iutils.config import Config
class WorkPrepare(Work):
   def __init__(self, config: Config, work_name = ""):
       """ Task initialization receives a global Config object to unify global configuration. """
       super(WorkBuildCommon, self).__init__(config, work_name)
    def run(self):
        # Task execution where task logic is implemented
    def install(self):
        # Installation phase
# Provides the capability of independent script debugging and execution.
if __name__ == "__main__":
    cfg = Config()
    cfg.parse_args()
    wk = WorkEnvirPrepare(cfg)
    wk.run()

• Every task includes if __name__ == "__main__": code, allowing any script to be called independently. When calling a script, you need to enter the build directory and run python3 works/work_prepare.py.

Target Description File ​

• Target description files are in YAML format and describe content such as task names, task classes, subtasks, and dependent tasks.

# Task example, required when a task is a dependency
- name: work.prepare_env
  # Python class for the task, executed by an independent process
  klass: works.work_prepare.WorkPrepare

  # List of subtasks. Subsystems are executed recursively with identical logic after klass completion.
  subworks:
    - name: work.dependency.bison
      klass: works.dependency.work_install_bison.WorkInstallBison
    - name: work.dependency.flatbuffers
      klass: works.dependency.work_install_flatbuffers.WorkInstallflatbuffers
- name: work.build.conan_debug_rc
  klass: works.build.work_build_conan.WorkBuildConan
  # Task dependency list. You can wait for multiple tasks. The current task is executed only after all dependencies complete.
  wait:
    - work.prepare_env
  # Task private configuration. The build framework checks if the task has a set_xxx method, for example set_stage(self, value).
  # It then calls the set_xxx method to configure the task.
  work_config:
    stage:
      rc
  # Global task configuration. Unlike task private configuration, this affects the global Config object. It calls the set_xxx method of the configuration object to change its values.
  target_config:
    build_type: release

• As shown in the example comments, to extend the configuration capabilities of the framework, target_config and work_config are defined. The former configures the Config object, while the latter configures the Work object. Their functions are explained within the scheduling framework.

Multi-process Scheduling Framework ​

• A multi-process scheduling solution where each task runs in an independent process is adopted. Each task (including subsystems) has four execution phases: creating the task, waiting for dependencies, executing the task, and creating child processes. There are three states: running, completed, and error.

1. Creating the task: A task process is created.
2. Waiting for dependencies: Multi-processing means that states cannot be shared directly. To address this, a global status management process is created. Tasks poll this process for the state of tasks they depend on.
3. Executing the task: After all dependencies complete, the run method of the task begins execution. If any dependency task fails, the current task reports an error and exits.
4. Creating child processes: Each child process repeats the four phases of the parent process recursively until all tasks complete and the build target is achieved.

• During the phase of creating child processes, the Config object of the task is passed to child processes. This is the purpose of target_config, as the scheduling framework passes this object along the scheduling chain to child processes. work_config only configures the individual task. It may or may not modify the Config object based on the specific behavior of the task, without being restricted by the framework.

• Task states

  • Tasks can either succeed or fail. Each process reports its state externally. The current mechanism requires that if any task fails, other tasks are terminated, and the system exits with an overall error.
  • Because tasks reside in different processes, state synchronization relies on inter-process communication. A global state management process is created for this purpose. This process receives state reports from tasks over TCP port 23456, records them in a global task state table, and responds to state queries. The waiting process involves tasks sending TCP messages to poll the global state management process.