When developing service code, you must ensure that your features function correctly. Although these features may be only a small part of the entire system, they still require unit testing (UT), integration testing (IT), and some real-world environment testing. These tests verify the correctness of newly developed functional units, ensure that these features interact correctly with surrounding interfaces during integration, and confirm that they do not break existing features.

When you write test code, your goals should be to improve the testability of product service code and enhance the quality of developer testing (DT) case design, implementation, and execution. You should also aim to optimize the compilation, running, debugging, and loading times of DT projects while reducing maintenance costs. These efforts improve DT efficiency, lower costs, and increase development productivity.

The components of openUBMC are capable of independent testing during development. The tests you perform can significantly reduce defects after version integration and lower the cost of locating defects later. The test code you write is as important as the service code itself.

The current openUBMC DT testing framework mainly covers UT and IT scenarios:

UT focuses on the reliability of functions within a component. The primary targets are functions that the internal submodules of a component call between each other.
IT focuses on the reliability of interfaces between this component and other components rather than internal functions.

The openUBMC development tool, BMC Studio, provides the bingo test command for DT. The parameters for the command are as follows:

shell

Test component

optional arguments:
  -h, --help            show this help message and exit
  -bt BUILD_TYPE, --build_type BUILD_TYPE
                        Build type. Options: dt, debug, release. Default: debug.
  --stage STAGE         Component package release stage. Options: dev (developer), pre, rc, stable. Default: dev.
  -r REMOTE, --remote REMOTE
                        Conan repository alias. Run conan remote list to view configured Conan repositories. Default: openUBMC_dev.
  -s, --from_source     Build from source, include dependency component
  -nc, --no_cache       Does not use the cache package in the ~/.conan/data directory (deletes the cache before building).
  --user USER           Specifies the user field for the Conan package. Attempts to read the conan.user field from the /etc/bingo.conf file. Otherwise, it is managed by the program.
  -as, --asan           Enable address sanitizer
  -ut, --unit_test      Enable unit test
  -it, --integration_test
                        Enable integration test
  -f TEST_FILTER, --test_filter TEST_FILTER
                        Run unit test with a filter
  -a APP, --app APP     App in hica
  --wb, --without_build  Do not rebuild during testing.

The bingo test -ut and bingo test -it commands run unit tests and integration tests, respectively. These are the commands that developers use most frequently.

Unit Testing

UT, also known as module testing, verifies the correctness of program modules. You must write test cases for both your custom code and automatically generated code to ensure the correctness of the code logic.

Testing Framework

openUBMC uses the LuaUnit open-source software as the base testing framework for UT. It is a comprehensive and easy-to-use UT framework that provides features such as case management, scheduling, and assertions. The framework analyzes referenced files for all classes starting with Test and calls test functions named with the test_* prefix in sequence. openUBMC has integrated luaunit into its development tools. The build process automatically includes DT dependencies, so you do not need to configure them manually. You only need to import LuaUnit and write your test code:

lua

luaunit = require('luaunit')

LuaUnit provides a series of assertXX() assertion functions, such as assertNotNil() and assertEquals(), which offer more detailed information. Therefore, do not use the assert() function directly in openUBMC. The following sections briefly introduce several commonly used assertion functions:

Equality assertion:

lua

-- Equality assertion. If the assertion fails, the tool prints extra_msg. For table types, it performs a deep comparison of the number of elements, keys, and values.
assertEquals(actual, expected[, extra_msg])

-- Inequality assertion.
assertNotEquals(actual, expected[, extra_msg])

Value assertions

lua

-- Following Lua rules, the tool considers any value other than false and nil as true to pass the assertion.
assertEvalToTrue(value[, extra_msg])

-- Only false and nil pass the assertion. All other values fail.
assertEvalToFalse(value[, extra_msg])

-- Only true passes the assertion.
assertTrue(value[, extra_msg])

-- Only false passes the assertion.
assertFalse(value[, extra_msg])

-- Only nil passes the assertion.
assertNil(value[, extra_msg])

-- Any value that is not nil passes the assertion.
assertNotNil(value[, extra_msg])

-- The assertion is passed only if actual and expected refer to the same object. For basic types such as string, numbers, boolean, and nil, this is equivalent to assertEquals.
assertIs(actual, expected[, extra_msg])

-- The assertion passes only if actual and expected refer to different objects.
assertNotIs(actual, expected[, extra_msg])

Type assertions

lua

-- The assertion passes only if the variable type is number.
assertIsNumber(value[, extra_msg])

-- The assertion passes only if the type is string.
assertIsString(value[, extra_msg])

-- The assertion passes only if the type is table.
assertIsTable(value[, extra_msg])

-- The assertion passes only if the type is boolean.
assertIsBoolean(value[, extra_msg])

-- The assertion passes only if the type is nil.
assertIsNil(value[, extra_msg])

-- The assertion passes only if the type is function.
assertIsFunction(value[, extra_msg])

-- The assertion passes only if the type is userdata.
assertIsUserdata(value[, extra_msg])

-- The assertion passes only if the type is coroutine.
assertIsCoroutine(value[, extra_msg])

-- The assertion passes only if the type is thread.
assertIsThread(value[, extra_msg])

Scope of Unit Testing

UT focuses on verifying the correctness of the smallest testable parts of software, that is, functional units. Its core objective is to test the external interfaces of functional units to ensure their stability and reliability. Because the internal functions that a functional unit calls may change due to code refactoring or feature optimization, you do not need to test these internal functions directly with unit tests.

In a Lua module, UT targets functions that other Lua modules within the component can access. These functions are typically exported through the return statement of the module or defined within the returned table. If a function is local and not exported through return, you generally do not need to perform UT on it separately. You can test such functions indirectly through other exposed functions. However, if a return statement exports a local function, you must include it in the scope of UT.

Test Code Development

UT targets the interface level. The test code resides in the test/unit directory of the component repository. When you write unit test code, you must define corresponding test case files in the test/unit directory based on the modules to be tested. The rules for test case files are as follows:

Define the test file based on the feature being tested. The filename must start with test followed by the specific feature, such as test_operation.lua.

Define a global table object in the test file to hold the specific test functions. The names of these test functions must also start with test. The following is an example:

File to be tested operation.lua: This file implements service features and is stored in the src/lualib directory.

lua

-- This file implements service features, including arithmetic methods.
local calculator = {}

function calculator:multiply(v1, v2)
    return v1 * v2
end

function calculator:divide(v1, v2)
    if v2 == 0 then
        error("Division by zero")
    end
    return v1 / v2
end
return calculator

Corresponding test case file test_operation.lua: This is the UT file stored in the test/unit directory.

lua

-- Import the module to be tested.
local calculator = require 'operation'
-- Import the LuaUnit framework.
local lu = require 'luaunit'

-- Write test cases.
test_calculator = {}

-- Test.
function test_calculator:test_multiply()
    local re = calculator:multiply(0, 1)
    lu.assertEquals(re, 1)
end

function test_calculator:test_divide()
    local re = calculator:divide(0, 1)
    lu.assertEquals(re, 0)
    re = calculator:divide(1, 1)
    lu.assertEquals(re, 1)
    local ok = pcall(calculator.divide, 1, 0)
    lu.assertEquals(ok, false)
end

In openUBMC, the unit tests for a component run Lua scripts directly. The entry file is test.lua in the test/unit directory. This file performs the following actions: loads configuration files, imports dependency libraries, imports all test modules (test case files or directories), and starts the tests through LuaUnit. Most code in test.lua is fixed and does not require modification. Generally, you only need to use require to import the test module:

lua

require 'test_operation'

The following is an example of a complete test.lua file:

lua

loadfile(os.getenv('CONFIG_FILE'), 't', {package = package, os = os})()

local lu = require('luaunit')
local utils = require 'utils.core'
local logging = require 'mc.logging'

local current_file_dir = debug.getinfo(1).source:match('@?(.*)/')

utils.chdir(current_file_dir)
logging:setPrint(nil)
logging:setLevel(logging.INFO)

-- Import the test module.
require 'test_operation'

-- Start LuaUnit.
os.exit(lu.LuaUnit.run())

After importing the test module into the unit test entry file test.lua, run the unit test command in the component repository path:

shell

bingo test -ut

This command automatically executes the unit test entry file test.lua. Based on the test.lua file, the LuaUnit framework scans for all variables (functions or tables) starting with test or Test in the imported test modules and runs them.

Because UT focuses only on functional tests, it executes test methods by running Lua scripts directly. For function methods that need to call external component interfaces, unit tests cannot reach those interfaces. In such cases, you can use mocks to simulate interface calls.

Simply speaking, the process for implementing unit tests is as follows:

Determine the test scope: Identify the functions in the lua module that are accessible externally.
For a new module, create a test_xxx.lua file (where xxx is the name of the service code file) in the test/unit directory and write the corresponding DT code. The format of the unit test file is as follows: (1) Import the module to be tested. (2) Create a test table. Integrate cases of the same type into one table to form a test suite (a group of similar test cases). (3) Write the body of the test functions. Ensure that test cases include normal service flows and consider critical exception logic and boundary values. Aim for comprehensive test coverage.
Load the test file in the unit test entry script test.lua. For example:
lua
```
require 'test_operation'
```
Run the test command in the component path to execute the unit tests for the component.
shell
```
bingo test -ut
```

Test Coverage

The openUBMC engineering tool bingo provides unit test coverage statistics. The luacov coverage statistics framework automatically calculates the coverage of unit test cases. Like the luaunit development framework, luacov is integrated into the development tools and does not require manual configuration. After writing the unit tests and service code, you can run the following command in the component directory to calculate test coverage:

shell

bingo test -ut -cov

This command automatically detects UT code coverage and generates a visualization file luacov.report.html in the temp/coverage path. You can also view the code coverage statistics on the console. The following is an example of the statistics results:

shell

Line Coverage: 18.7%, Hits: 346 lines, Misses: 1509 lines
Coverage report luacov.report.html saved to /new_app/temp/coverage
DT report results output to /new_app/temp/coverage/dt_result.json
Total incremental coverage: 100.0%, C code coverage: 100.0%, Lua code coverage: 100.0%
Incremental coverage report saved to dt_result.json
================================ Build Menu ================================
Unit Test (ut):                      True
Integration Test (it):               False
Coverage:                            True
Address Sanitizer:                   False
============================================================================

The scope of component coverage statistics includes all Lua code in the src root directory of the component.

When writing unit test code, you should aim for the highest possible coverage and fully consider the boundary values of the test cases.

Integration Testing

While UT targets functional functions within a component, IT focuses on the reliability of interfaces between the current component and other components. You must perform IT on the interfaces provided externally through D-Bus to ensure their stability. For a component such as my_app, the configuration file test/integration/test_my_app.conf (named with the test_ prefix followed by the component name) typically defines Skynet environment variables and references the components that IT depends on. The test/integration/test_my_app.lua file (also named with the test_ prefix followed by the component name) serves as the entry file for running integration tests. You can write integration test code or reference integration test files in this file.

The test_new_app.config file is the entry configuration file for component IT. The IT command bingo test -it runs the integration tests based on the definitions in this configuration file. The description of test_new_app.config is as follows:

lua

-- Include the /root/BMC/new_app/temp/opt/bmc/libmc/config.cfg file.
include("$CONFIG_FILE")                

-- Initialize IT directories.
config:init_integration_test_dirs()

-- Define the entry file for IT.
config:set_start("test_new_app")

-- Define components related to IT. These components must have test dependencies defined in service.json.
config:include_app('new_app')
config:done()

-- Define SkyNet variables that can be used in the IT Lua files.
TEST_DATA_DIR = 'test/integration/.test_temp_data/'
test_apps_root = 'test/integration/apps/'

In the test_new_app.config configuration file, config:set_start("test_new_app") defines test_new_app.lua as the entry file for IT. The description of test_new_app.lua is as follows:

lua

-- Reference the services required for IT.
local skynet = require 'skynet'
require 'skynet.manager'
local log = require 'mc.logging'
local utils = require 'mc.utils'
local test_common = require 'test_common.utils'

-- Preparation phase for IT. Create a temporary directory for files needed during IT.
local function prepare_test_data()
    local test_data_dir = skynet.getenv('TEST_DATA_DIR')
   
    -- The test_data_dir variable is already defined in test_xxx.config.
    os.execute('mkdir -p ' .. test_data_dir)
    os.execute('mkdir -p ' .. '/tmp/test_dump')
end

-- Clear test data to prevent existing data in the temporary directory from affecting IT.
local function clear_test_data(exit_test)
    log:info('clear test data')
    local test_data_dir = skynet.getenv('TEST_DATA_DIR')
    if not exit_test then
        return utils.remove_file(test_data_dir)
    end

    skynet.timeout(0, function()
        skynet.sleep(20)
        skynet.abort()
        utils.remove_file(test_data_dir)
        utils.remove_file('/tmp/test_dump')
    end)
end

-- Service test code for IT. You can write your service test code here.
local function test_new_app()
    log:info('================ test start ================')
    -- todo

    log:info('================ test complete ================')
end

-- Entry function for IT.
skynet.start(function()
    clear_test_data(false)
    prepare_test_data()
    test_common.dbus_launch()
    skynet.uniqueservice('main')
    skynet.fork(function()
        local ok, err = pcall(test_new_app)
        clear_test_data(true)
        if not ok then
            error(err)
        end
    end)
end)

Simply speaking, the process for implementing integration tests is as follows:

Determine the test scope: Identify the rpc interfaces that the component exposes externally.
For a new module, IT may involve various scenarios, each with different configuration files and test scripts. You can create subdirectories in the test/integration directory to manage these scenarios.
Define the components that IT depends on in the service.json file.
Write the corresponding code in the integration test configuration file test_xxx.conf and the script file test_xxx.lua (which contains the Skynet start function) to call the test functions.
Run the test command to execute the integration tests for the component.
shell
```
bingo test -it
```

Unit Testing ​

Testing Framework ​

Scope of Unit Testing ​

Test Code Development ​

Test Coverage ​

Integration Testing ​

Unit Testing

Testing Framework

Scope of Unit Testing

Test Code Development

Test Coverage

Integration Testing