13.9 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11
    __ __          __            __
   / //_/__  _____/ /_________  / /
  / ,< / _ \/ ___/ __/ ___/ _ \/ /
 / /| /  __(__  ) /_/ /  /  __/ /
/_/_|_\___/____/\__/_/ __\___/_/ _________
  / ___/____  / __/ /_/ __ )/  |/  / ____/
  \__ \/ __ \/ /_/ __/ __  / /|_/ / /
 ___/ / /_/ / __/ /_/ /_/ / /  / / /___
/____/\____/_/  \__/_____/_/  /_/\____/

13 14

16 17 18
# Kestrel SoftBMC Project
## SoC integration files
Copyright (c) 2020 - 2021 Raptor Engineering, LLC


# What is it?

Kestrel is the world's first full featured "soft" BMC SoC, designed for and tested on the ECP-5 series of FPGAs from Lattice using the NextPNR open source development flow on ppc64le.  It is currently able to IPL a POWER9 host, such as the Blackbird and Talos II systems from Raptor Computing Systems [^1], completely independent of the ASpeed hard ASIC BMC integrated on those systems.

As a fully open HDL and open firmware / open software design using open tooling on open ISA systems, Kestrel allows a very high level of assurance regarding security, owner control, and trustability of this key component on modern systems.

# How to build

Building the Kestrel BMC for the ECP-5 FPGA on the Versa board is quite straightforward.  We recommend the latest versions of NextPNR and related tooling be used, as we push these tools to their limits in many respects with this design.  Kestrel is primarily developed on Debian Buster on Talos II or Blackbird hosts (ppc64le).  Raptor strongly recommends that its development environment be used, as we will not be able to assist with issues encountered on other platforms.  In particular, Raptor will not provide assistance with issues specific to either x86_64 systems or proprietary FPGA toolchains.

Please see the [Quick Start Guide]( for setup instructions applicable to the standard Raptor development environment on POWER9.  After the subrepositories are set up, simply run:

34 35
    cd kestrel/litex/litex-boards/litex_boards/targets
    ./ --device=LFE5UM --cpu-type=microwatt --cpu-variant=standard+ghdl --build --nextpnr-seed 1

# Updating the bitstream with new firmware

The bitstream is automatically stuffed with valid ROM contents, however, if a developer wishes to update the rom, the following sequence can be used:

41 42 43
    cd kestrel/litex/litex-boards/litex_boards/targets/build/versa_ecp5/gateware
    ecpbram -i versa_ecp5.config -o versa_ecp5_stuffed.config -f rom.init -t rom_data.init
    ecppack versa_ecp5_stuffed.config --svf versa_ecp5.svf --bit versa_ecp5.bit --bootaddr 0

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176
# How to connect

## POWER9 hosts
### Interfaces
The Kestrel soft BMC will require, at minimum, the following signals to be connected to a POWER9 host to allow the host to IPL:
- I2C (Platform control)
- I2C (AVSBus)

Of the four interfaces above, only FSI will require soldering to attach to the on-board 3.3V translator.  On Blackbird, the three 3.3V FSI control lines (FSI_SOFT_CLK, FSI_SOFT_DAT, DBG_FSI0_DAT_EN) are available at R3200, R3202, and R7018 respectively (see photographs).  Alternatively, if soldering is not desired, the FSI debug connector (J3200) can be utilized with an external voltage level translator circuit between the connector and the FPGA.  Similarly, the Talos II mainboard should be able to be utilized as the host, but Raptor has not yet tested Kestrel on Talos II.

![RCS Blackbird top-side FSI tap points](images/rcs/blackbird/fsi_tap_top.png) ![RCS Blackbird bottom-side FSI tap points](images/rcs/blackbird/fsi_tap_bottom.png)

When connecting the LPC bus to J10105, be sure to keep all lines as short as possible and length matched.  Weak pull-ups (15kΩ) are required between the LPC data / frame / reset lines and +3.3V AUX (or 3.3V FPGA) power.  No pull-up should be used on the clock line.

The I2C platform control lines are available at J10103, while the I2C AVSBus lines are available at J6902.  Both headers are standard IDC stake pins.

In all cases, we recommend consulting the Blackbird (or Talos II) schematics along with the Versa ECP5 schematics while making the initial connections between the two boards.  Special attention to high speed signal routing and grounding will help ensure a successful result.

### ASpeed BMC deactivation
In addition, the ASpeed BMC will need to be deactivated.  This is slightly more difficult than might be expected, because the ASpeed BMC is pinstrapped in a way that it will actively interfere with the LPC lines if held in reset.  As a result, the preferred way to isolate it from the mainboard is to allow it to boot, then issue the following commmands at the ASpeed BMC root console:

    # Disable AST LPC pins
    devmem 0x1e6e20ac 32 0x00000000

    # Switch FSI GPIOs (H1,H2,H3) to input mode (Raptor Blackbird mainboards)
    echo gpio-fsi > /sys/bus/platform/drivers/fsi-master-acf/unbind
    devmem 0x1e780024 32 0x00010000

    # Disable AST I2C bus 13 pins
    devmem 0x1e6e2090 32 0x023f0000

## All hosts
Most, if not all, mainboards will require some form of storage for the bootloader and platform firmware.  Kestrel contains a high-speed SPI/QSPI master that can be used to attach any standard SPI Flash device.  We recommend that all wires be kept as short as possible between the Versa board and the external SPI device to minimize interference and maximize transfer rates.

We strongly recommend you reference the pin configuration file at litex-boards/litex_boards/platforms/ when attaching any devices or external mainboards to the Versa board.  This file contains the pin information that will, in conjunction with the Versa board schematics, allow correct wiring of the ECP5 to any attached hardware.

# How to use

## Overview
Due to FPGA size constraints on the Versa board, host I/O and control is limited to a serial console at this time.  Ethernet and automatic BMC firmware loading support, while entirely possible from a logical standpoint, will require either further Microwatt size optimization to free up resources on the ECP5 -45 FPGA used on the Versa board, or the use of a larger ECP5 device (see the Future Direction section for more details).

## Bitstream and firmware load
In most test and development applications, the soft BMC firmware will be loaded from the same host machine that is loading the ECP5 bitstream to the target board.  Consistent with the build process, we use POWER hosts to load and control the ECP5 during development.  If desired, it is also possible to flash the FPGA bitstream and BMC firmware to the on-board Flash memory of the Versa board, although that process is highly board-specific and is outside the intended scope of this document.

The recommended test setup is two console windows, one running the LiteX terminal and firmware loader on the target's FTDI serial interface:

    cd kestrel/litex/litex-boards/litex_boards/targets/firmware
    python3 kestrel/litex/litex/litex/tools/ --speed 115200 /dev/ttyUSB1 --kernel kestrel/litex/litex-boards/litex_boards/targets/firmware/firmware.bin

and the other running the OpenOCD programming tool:

    cd kestrel/litex/litex-boards/litex_boards/targets
    openocd --log_output openocd.log 3 -f "/usr/share/trellis/misc/openocd/ecp5-versa.cfg" -c "transport select jtag; init; svf build/versa_ecp5/gateware/versa_ecp5.svf; exit"

When it is desired to recompile the firmware and re-upload, exit the LiteX terminal with a rapid `Ctrl+C Ctrl+C`, re-run make in the firmware directory, and then restart the LiteX terminal.  From there, you may either us the `reboot` command at the BMC terminal, or re-run OpenOCD to reset all hardware inside the FPGA with a new bistream load.

## Host control and initial IPL
When the Kestrel firmware finishes loading the PNOR image from external Flash memory and completes its self-checks, a `FSP0>` prompt will appear on the serial console.  From this prompt, several commands are supported.  The most commonly used command is `poweron`, it will power on the board, start the IPL process, and attach a host serial console.  To escape the host serial console, the standard `~.<RETURN>` escape sequence is supported.  `chassisoff` will power down the host immediately.  Other supported commands are listed in the built-in help system, which is available with the `help` command.

## Example console output

If you've wired everything up correctly, and depending on exactly how your host POWER9 system's PNOR was configured along with the debug options active on the Kestrel BMC, you should see output similar to the following upon issuing the `poweron` command at the Kestrel console:

    Platform FPGA communication verified
    Commanding chassis power ON...
    Chassis power verified active
    1 CPU(s) installed
    Applying AVSBus workarounds...
            VDD/VCS 0: Enabling AVSBus CLK/MDAT pullups and selecting VIH/VIL 0x2 (0.65V/0.55V)
            VDN 0: Enabling AVSBus CLK/MDAT pullups and selecting VIH/VIL 0x2 (0.65V/0.55V)
            AVSBus workaround application complete!
    Enabling AVSbus PMBUS functionality...
            VDD 0: Placing device in AVSBus voltage command mode
            VCS 0: Placing device in AVSBus voltage command mode
            VDN 0: Placing device in AVSBus voltage command mode
            AVSBus PMBUS functionality enabled!
    initialize_fsi_master(): after setup: ctl 0x00031400, sta 0x10000000
    Starting IPL on side 0
    access_fsi_mem(): address 0x000800, data: 0xa0ff0800 sta: 0x41000001
    access_fsi_mem(): address 0x000900, data: 0x00000001 sta: 0x41000001
    access_fsi_mem(): address 0x000900, data: 0x00000001 sta: 0x41000001
    access_fsi_mem(): address 0x000900, data: 0x00000001 sta: 0x41000001
    access_fsi_mem(): address 0x002824, data: 0x00000000 sta: 0x41000001
    access_fsi_mem(): address 0x000808, data: 0x02040000 sta: 0x41000001
    access_fsi_mem(): address 0x000814, data: 0x04000510 sta: 0x41000001
    access_fsi_mem(): address 0x000808, data: 0x02040000 sta: 0x41000001
    access_fsi_mem(): address 0x000900, data: 0x00000001 sta: 0x41000001
    access_fsi_mem(): address 0x002c60, data: 0xeeeef304 sta: 0x41000001
    access_fsi_mem(): address 0x002c60, data: 0xeeeef30c sta: 0x41000001
    access_fsi_mem(): address 0x000870, data: 0x20000000 sta: 0x41000001
    access_fsi_mem(): address 0x001034, data: 0x60000000 sta: 0x41000001
    access_fsi_mem(): address 0x00102c, data: 0xffffffff sta: 0x41000001
    access_fsi_mem(): address 0x002820, data: 0x00000000 sta: 0x41000001
    access_fsi_mem(): address 0x002820, data: 0x00000000 sta: 0x41000001
    access_fsi_mem(): address 0x002804, data: 0x04c04000 sta: 0x41000001
    access_fsi_mem(): address 0x002804, data: 0x04c04000 sta: 0x41000001
    access_fsi_mem(): address 0x002804, data: 0x84c04000 sta: 0x41000001

    --== Welcome to SBE - CommitId[0x9cc77bc5] ==--
    SBE starting hostboot

    --== Welcome to Hostboot hostboot-8f8fa4c/hbicore.bin ==--

Note the FSI start sequence is the last task to run after main platform setup and that the console switches over to the host console directly afterward.  If you see no output past that point, there is likely an issue with the LPC or SPI PNOR bus wiring.  Alternately, it is possible the SPI PNOR device does not contain a useable firmware image.  Debug information in this situation can be obtained by escaping to the Kestrel console and running the `sbe_status` command.

# Status

## Known Issues

- Many IPMI commands are unimplemented in this initial release, causing the OCCs to fail to come online [^2] [^3].  These commands will be implemented over the coming weeks and months.
- NextPNR continues to indicate timing failure even though the design operates as intended.  This needs to be investigated and resolved.
- Microwatt is consuming around half of the entire FPGA on the Versa ECP5 -45 devices.  This is due to issues within GHDL/Yosys/NextPNR, apparently triggered by the Microwatt codebase, that need to be investigated and resolved.

## Future Direction

Kestrel is an active development project at Raptor Engineering, and as such features will continue to be implemented over time.  Our current focus is the remaining IPMI commands, followed by Microwatt size reduction in an effort to (re)enable Ethernet support.  Linux is also on our roadmap, although timing is currently unknown due to the aforementioned blocking factors.

Kestrel is intended to be used on an upcoming Raptor Computing Systems product incorporating a much larger ECP5 FPGA.  This product is not a standard PC form factor device, nor will it be a cost focused product, but it is designed partly as a development platform for Kestrel technologies.  Look for more information on this specialty product later in 2021!

# Licensing

Raptor Engineering has made the Kestrel project specific HDL and firmware open to the public under the GPL v3.  Should you require other licensing terms, we are able to extend a commercial license of the codebase as needed.  Please contact for more information and pricing details.

We welcome contributions and enhancements to Kestrel.  Due to the use of BSD and GPL licenses throughout the various Kestrel sub-repsitories, we may be able to merge pull requests for some modules without a CLA, while we may require a CLA for modifications to other modules.  We will make any CLA requirements clear in pull request comments, but as a general rule the Raptor GPL-licensed modules will require a CLA for contributions to be upstreamed.

[^2]: Minimum to IPL IPMI commands have been documented here:
[^3]: A sample of one command know to block OCC onlining (written in C) is here: