Build System and Directory Layout

The Directory Structure

<BoxLambda Root Directory>
├── boxlambda_setup.sh
├── CMakeLists.txt
├── sub/
│   └── <git submodules>
├── gw/
│   ├── CMakeLists.txt
│   ├── components/
│   │   ├── CMakeLists.txt
│   │   ├── wbuart32/
│   │   │   ├── rtl/
│   │   │   ├── CMakeLists.txt
│   │   │   └── Bender.yml
│   │   └── <other gw component directories>
│   └── projects/
│       ├── CMakeLists.txt
│       ├── ddr_test/
│       │   ├── constr/
│       │   ├── rtl/
│       │   ├── sim/
│       │   ├── CMakeLists.txt
│       │   └── Bender.yml
│       └── <other gw project directories>
├── sw/
│   ├── CMakeLists.txt
│   ├── components/
│   │   ├── CMakeLists.txt
│   │   └── <sw component directories>
│   └── projects/
│       ├── CMakeLists.txt
│       ├── ddr_test/
│       │   ├── CMakeLists.txt
│       │   └── <ddr_test sources>
│       └── <other sw project directories>
└── build/
    ├── sim/
    ├── arty-a7-35/ 
    └── arty-a7-100/

Note that the software tree (sw/) is separate from the gateware tree (gw/) and the build trees (build/) are separate from the sw and gw source trees.

The build trees are CMake build trees generated by the boxlambda_setup.sh script. As you can see in the tree diagram above, the script generates three build trees: one for simulation, one for the Arty-A7-35T, and one for the Arty A7-100T. The build trees are not under version control.

The Gateware Build Structure

Bender

The build system is CMake based but relies on Bender for gateware package dependency Management:

https://github.com/pulp-platform/bender

Central to Bender is the package manifest bender.yml. In the manifest, you specify the HDL sources that make up the package, dependencies, include paths, targets (e.g. synth, sim), and associated defines. A package directory is a directory containing a bender.yml file. When you run bender in that directory, you can ask it to generate a flat list of all the sources from the current package, and the packages it depends on. Optionally, it can generate that list, plus any defines associated with a given target, as a Tcl script. This makes integration with Vivado very easy.

Project View of the Build System

Project View of the Build System

Bender Targets

Currently, the build system uses the following Bender targets:

  • module_name: set when building a component separately (OOC)
  - target: ibex_wb_core
    files:
      - rtl/ibex_wb_core_wrapper.sv
  • vivado: set when synthesizing using Vivado.
  • verilator: set when building Verilator simulation model.
  • constraints: set when retrieving .xdc constraints files for this component or project.

Three Layers

The gateware system has three layers:

  1. The Project Layer (top): Hello World is an example project. A project is the top layer of the build system. The bender.yml manifest contains the top-level files of an SoC build, the project's .xdc constraints file, memory files used by the SoC, and a list of components the project depends on.
  2. The Component Layer (middle): Components are the middle layer of the build system. They are the building blocks of an SoC. A component's sources, defines, and dependencies are defined in a bender.yml manifest. A component gets its HDL sources from its rtl/ subdirectory and/or from sub/, the submodule layer. I'm considering each Wishbone Bus Master or Slave a component.
  3. The Submodule Layer (bottom): Submodules are the bottom layer of the build system. They are the Git Submodules that BoxLambda is referencing, as discussed previously.

Building Gateware

Assuming all Prerequisites are installed, navigate to the build tree of choice and type:

make <gw component or project name>_<action>

where action is one of the following:

  • lint: Lint-check the given gateware component or project.
  • sim: Build the Verilator simulation model (Vmodel) of the given gateware project. This action only exists in the sim/ build tree.
  • synth: Synthesize the given gateware component or project. This action only exists in the arty-a7 build trees. When you run the synth action on a component, Out-Of-Context (OOC) synthesis is performed. The component's input and output ports aren't hooked up to anything.
  • impl: Implement the given gateware project and generate its bitstream. This action only exists in the arty-a7 build trees.
  • load: Load the gateware project's bitstream file onto the connected target. This action only exists in the arty-a7 build trees.

Some examples:

cd <boxlambda_root_dir>/build/sim && make ddr_test_sim

cd <boxlambda_root_dir>/build/arty-a7-35 && make hello_world_synth

cd <boxlambda_root_dir>/build/arty-a7-100 && make hello_dbg_impl && make hello_dbg_load

The build directory tree mimics the source tree. When a build has been completed, a gateware project's Verilator model or the Vivado project files can be found under that project's directory. E.g.:

$ cd build/arty-a7-35/gw/projects/hello_world
$ make hello_world_synth
...
$ ls
CMakeFiles           hello_world.constraints_file_list  project.cache          project.runs    syn_util.rpt
CTestTestfile.cmake  hello_world.mem_file_list          project.dep            project.xpr
Makefile             hello_world.vivado_sources         project.hw             spram.mem
cmake_install.cmake  hello_world.vivado_sources.dep     project.ip_user_files  syn_timing.rpt

What happens when you run make hello_world_synth

When you run make hello_world_synth, the following happens:

  1. Make determines if (re)synthesis is needed. If synthesis is up-to-date, no further action is taken.
  2. Make runs a bender script command on the bender.yml file in the gw/projects/hello_world/ directory. The bender script command is wrapped in the scripts/bender_gen_vivado_source.sh shell script.
  3. The bender script command processes that bender.yml manifest, as well as the bender.yml manifests of any dependent components.
  4. The bender script command emits a list of all the HDL sources that make up the project.
  5. Similarly, the scripts/bender_gen_constraints_file_list.sh and scripts/bender_gen_mem_file_list.sh emits the .xdc constraints and .mem memory files for the project.
  6. Make feeds these file lists into a vivado_create_project.tcl script.
  7. The vivado_create_project.tcl script creates a Vivado project.
  8. Make kicks off the vivado_synth.tcl script which opens the Vivado project and starts synthesis.

When you run make hello_world_impl, the following happens:

  1. Make determines if (re)implementation and bitstream generation is needed. If the bitstream file is up-to-date, no further action is taken. Make will also run the hello_world_synth rule above because it's a dependency of hello_world_impl.
  2. Make kicks off the vivado_impl.tcl script which opens the Vivado project, picks up the synthesis checkpoint, and starts implementation.

Verilator Lint Waivers

Rather than add lint waivers to the source code of git submodules, the waivers are grouped into .vlt files that live in the corresponding gateware component or project subdirectory. This way we avoid making unnecessary code changes in the git submodules.

For example gw/components/ibex/lint.vlt

Gateware Component/Project Directory Layout

A component or project directory typically contains the following files and subdirectories:

<component/project>
├── CMakeLists.txt
├── Bender.yml: The component/project Bender manifest.
├── lint.vlt: Lint waivers.
├── rtl
│   └── <Boxlambda specific RTL sources for given component/project>
└── sim
    └── <C++ test bench code for given component/source>

Building Software

The software corresponding with a gateware project automatically gets compiled, converted to a memory file, and included in the gateware project as part of the build process. Software projects can also be built independently. From the build directory just type: make <sw project name>. For example:

$ cd sim/sw/projects/hello_world/ $ make hello_world ... $ ls CMakeFiles Makefile hello_world hello_world.map CTestTestfile.cmake cmake_install.cmake hello_world.hex hello_world.mem

Make All, Clean, and Regen

make all will lint check and build 'impl' all gateware projects.

Make clean in a build tree will remove all the generated files that the build system is aware of. The generated files the build system is not aware of, e.g. synthesis utilization report files, will not be removed, however. If you want to go back to a completely clean build tree, type make regen from the build directory. This command will completely remove and regenerate the build tree.

Creating additional build trees

You can easily create additional build trees from the BoxLambda root directory with the following command:

cmake --preset=sim|arty-a7-35|arty-a7-100 -B <build directory>

For example:

cmake --preset=sim -B build/sim2

The CMakeLists

The build system consists of a tree of CMakeLists.txt files: The top-level CMakeLists.txt adds the gw/ and sw/ subdirectories. The CMakeLists.txt files in those subdirectories add the components/ and projects/ subdirectories, etc., down to the individual GW and SW component and project directories.

A Gateware Component CMakeList

The build instructions for a gateware component are grouped into one CMake function: gw_component_rules(). A GW component-level CMakeLists.txt file contains just a call to this function, passing in the expected parameters:

gw_component_rules(
    TOP_MODULE <top module name> 
    COMPONENT_NAME <component name> 
)

For example:

gw_component_rules(
    TOP_MODULE wb_wbuart_wrap_wrap 
    COMPONENT_NAME wbuart32 
)

The component's sources, definitions, and dependencies are still defined in its bender.yml manifest. The CMake build system interfaces with Bender through a set of scripts to extract the necessary info and pass it on to Vivado or Verilator.

A Gateware Project CMakeList

The build instructions for a gateware project are also grouped into a CMake function: gw_project_rules(). This function has a few additional arguments compared to its component counterpart. A typical GW project CMakeLists.txt file looks like this:

gw_project_rules(
    TOP_MODULE <top module name>
    PROJECT_NAME <project name>
    MEM_FILE_TARGET <sw project name>
    MEM_FILE_OUT <name of memory file expected by the SoC build. Currently, all project builds use spram.mem.>
    VERILATOR_CPP_FLAGS <Verilator CPP flags, e.g. include paths>
    VERILATOR_LD_FLAGS <Verilator link flags, e.g. -lncurses>
)

#Add testcase.
add_test(NAME <test name>
    COMMAND <test command>
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
)

For example:

gw_project_rules(
    TOP_MODULE ibex_soc
    PROJECT_NAME hello_world
    MEM_FILE_TARGET hello_world
    MEM_FILE_OUT spram.mem
    VERILATOR_CPP_FLAGS "-I${PROJECT_SOURCE_DIR}/sub/wbuart32/bench/cpp/"
    VERILATOR_LD_FLAGS "-lncurses"
)

add_test(NAME hello_world_test
    COMMAND ./Vmodel
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
)

As is the case for GW components, the project's sources, definitions, dependencies, and constraint files are defined in its bender.yml manifest. The reference to the SW project delivering the memory file is not defined in the Bender manifest, however. The SW project name is passed in as the MEM_FILE_TARGET parameter in the gw_project_rules() call.

Any test cases are also added to the project's CMakeLists.txt file.

Software Build Structure

CMake is designed to build software. The necessary functions for creating libraries, executables, etc. are predefined. The only custom function added to the software CMakeLists tree is link_internal_create_mem_file(). This function implements the necessary instructions to link the given executable against the BoxLambda internal memory map and generate a memory file, to be used by the GW part of the build system.

A typical SW project CMakeLists.txt file looks like this:

add_executable(hello_world
    EXCLUDE_FROM_ALL
    ../../../sub/ibex_wb/soc/fpga/arty-a7-35/sw/examples/hello/hello.c
    ../../../sub/ibex_wb/soc/fpga/arty-a7-35/sw/libs/soc/gpio.c
    ../../../sub/ibex_wb/soc/fpga/arty-a7-35/sw/libs/soc/utils.c
)

target_compile_options(hello_world
    PRIVATE -g)

link_internal_create_mem_file(hello_world)

Implementation

CMakeLists organization.

CMakeLists Organization.

The actual gateware build recipes (Bender interaction, verilating, synthesizing...) are implemented by a set of bash and tcl scripts kept in the scripts/ directory:

    bender_gen_constraints_file_list.sh
    bender_gen_verilator_sources.sh
    bender_gen_vivado_sources.sh
    bender_get_cpp_files.sh
    bender_get_vlts.sh
    gen_mem_file_list.sh
    prg_bitstream.tcl
    verilator_lint_check.sh
    verilator_sim.sh
    vivado_create_project.tcl
    vivado_impl.tcl
    vivado_synth.tcl

Having the build recipes as scripts instead of CMake allows me to invoke and test them outside of the build system.

The CMake build instructions define the various targets and the relationships between them, and invoke the above build scripts when needed.

The CMake build definitions are located as close as possible to the part of the tree to which they apply, e.g. the gw_project_rules() function can be found in the gw/projects/CMakeLists.txt file. Gw_component_rules() can be found in the gw/components/CMakeLists.txt file. Gateware build instructions common to components and projects are located in the gw/CMakeLists.txt file.

Cross-Compilation

RISC-V cross-compilation is set up by passing in a toolchain file to CMake. The toolchain file is located in scripts/toolchain.cmake.

Bender Interaction Hack

GNU Make, CMake's backend, uses the modification date of dependencies to decide if a build rule should be triggered, e.g. an object gets rebuilt when the corresponding source code has a more recent modification date than the object file itself. With Bender, however, a component's or project's bender.yml file is just the tip of a tree. The Bender target and package dependencies also have to be considered. Simply listing the bender.yml file as a dependency is not good enough. Instead, I'm using the Bender script output as a dependency:

  1. The build system runs the bender script command.
  2. The output of that command is stored in a temporary file.
  3. That file is compared with the Bender script output file used by the previous build of the same target.
    • If it's different, the file is copied over, making it the Bender script output file to be used by the next build step. The Bender script output file is a dependency for synthesis, so synthesis will be triggered.
    • If the temporary file is the same as the Bender script output file used by the previous build of that target, the temporary file is discarded. Synthesis will not be triggered.

This mechanism is implemented in the scripts/bender_gen_vivado_sources.sh and scripts/bender_gen_verilator_sources.sh scripts. The same scripts also generate a DepFile: a dependency list of all the sources referenced in the Bender manifest. This DepFile is referenced by the synthesis target so synthesis (or verilation) will be triggered if any of the sources change.

CMake and Bender Interaction.

CMake and Bender Interaction.