Every now and then I get asked if we have a process for getting REDHAWK onto some new platform, X. Usually the platform is based on a Xilinx Zynq part, which has a 32-bit ARM processor (or two) with a modest amount of FPGA fabric. I then add in meta-redhawk-sdr and Yocto to do the heavy lifting of cross-compiling. Rinse and repeat — we have built multiple working REDHAWK Nodes hosting a variety of transceiver chips whether it’s the Ettus Research e310, a MicroZed GPP, a ZedBoard Programmable Persona FEI, and so on.
This time, I was asked to target the Xilinx ZCU102 Evaluation Board, specifically while enabling access to the Analog Devices ADRV9371 HPC FMC transceiver attached to it. The ZCU102 is a quad-core 64-bit ARM with a relatively large, fast UltraScale+ FPGA attached to it. The ADRV9371 HPC FMC evaluation board is a multi-transceiver similar to the AD9361 used in the ZedBoard design, but much more sophisticated.
And to extend the request a bit farther, we want to target at least Yocto Rocko (2.4.x) for the sake of security and hardening.
So did we do it?
You can see the SCA Explorer panel on the right side of the image. In it, we have the ProgrammableNode running with a GPP and FEI_Programmable device (from our zedboard work). Under the FEI_Programmable, it lists 2 transmitters and 2 receivers (we did not expose the sniffer or observer interfaces). And on the left side is the properties panel for the GPP indicating our quad-core aarch64 ARM 64-bit processor is up and running.
While the FEI device is still a work in progress, we do already have IIO and the example AD9371 IIO stream app working:
root@zcu102-zynqmp-adrv9371:/usr/local/adi# ./ad9371-iiostream
* Acquiring IIO context
* Acquiring AD9371 streaming devices
* Configuring AD9371 for streaming
* Acquiring AD9371 phy RX channel 0
rf_bandwidth: 100000000
sampling_frequency: 122880000
* Acquiring AD9371 RX lo channel
* Acquiring AD9371 phy TX channel 0
rf_bandwidth: 75000000
sampling_frequency: 245760000
* Acquiring AD9371 TX lo channel
* Initializing AD9371 IIO streaming channels
* Enabling IIO streaming channels
* Creating non-cyclic IIO buffers with 1 MiS
* Starting IO streaming (press CTRL+C to cancel)
RX 1.05 MSmp, TX 1.05 MSmp
RX 2.10 MSmp, TX 2.10 MSmp
RX 3.15 MSmp, TX 3.15 MSmp
RX 4.19 MSmp, TX 4.19 MSmp
If you’re interested in utilizing this work, please feel free to contact us! Some of the highlights of the effort are described below.
The Reference Design
As of this writing, 13 April 2018 at 1040 hours, the platform as a whole is relatively new territory requiring specific development branches/tags of the Analog Devices HDL and Kernel repositories to get a working reference design. In fact the branch of HDL was deleted and replaced with a tag just a few hours before I began this article.
Then there’s the Linux build. One great resource was a forum post of a user having some success with PetaLinux 2017.4 and an external kernel. It’s a good read on the level of integration that can be accomplished with a recent version of PetaLinux now that it’s Yocto-based.
‘Pure’ Yocto Approach
Going the pure Yocto approach has a few advantages for us at Geon, and our customers. By sticking to that completely-open tool chain and keeping up with releases, our builds now can easily use a security layer for hardening our environments with OpenSCAP, for example. And at the time we began this effort, Xilinx’ PetaLinux layer did not support a version of Yocto new enough to directly integrate, so we stayed away from PetaLinux.
Note: on April 9th 2018.1 was released which is Rocko compatible.
The following sections are going to include some of the pitfalls/hurdles encountered along the way. Though not every step will be published, these are some helpful roadsigns similar to what we posted on FMCOMMS integration last year.
Meta-Xilinx
Integrating with Xilinx products in a pure Yocto environment generally requires adding their meta-xilinx layer to your environment and picking the right branch. However, the ZCU102 is clearly a bit of a new target for that layer.
The way the README is written on the master and rocko branches, for example, would make you think it’s only going to help you get an emulated ZCU102 going (QEMU Support vs. with QEMU Support). Another example is that the layer preferentially builds U-BOOT SPL, which does not support loading the required power management firmware (at this time). The layer provides the recipe to build that firmware, and makes it required by the machine definition, but one must patch and make other adjustments to help nudge things along. The machine definition also does not ensure that an FSBL is loaded into the boot partition. With a few small patches you can safely include the boot.bin (SPL) generated by this layer’s U-BOOT-XLNX recipe to the machine definition, but read on for an alternative.
The above experience encouraged me to look at their other published solution to ensure the FSBL, PMU, etc. all get built properly. So this layer then provides some scaffolding for us: a top-level hook for swapping to Analog Device’s kernel source vs. Xilinx as well as a basic machine definition one can use as a template.
Meta-Xilinx-Tools
Another layer that Xilinx maintains is called meta-xilinx-tools, which requires meta-xilinx. This layer uses Vivado as an external tool chain. It’s basically the concept used in PetaLinux, but without requiring all the wrappers, tooling, etc. that obfuscate, somewhat, the fact that you’re really just making a Yocto build. Further, what this means is it will unpack the system_top.hdf produced from Analog Devices’ HDL build and compile other packages like the FSBL (first stage boot loader), with the bitstream and U-Boot built in, as if you did it by hand from the SDK, for example.
I began with the rel-2017.4 branch which had a number of issues, some of which were corrected in the master branch shortly after I made my own patches. Symptoms of of these issues are for example having Xvfb errors for all the various instances that the recipes launch to control Vivado. None of those instances will get released or (per the error message) get communicated with because things like ps and xlsclients don’t exist in the recipe’s build environment. Older Yocto versions were more forgiving. In newer versions, the build environment is deliberately siloed to prevent host contamination and ensure that required packages are depended upon for a given package. To summarize: it makes the process repeatable between a variety of host environments. Because it’s important to recognize the why, I think it’s also important to look at different possible solutions.
The solution that was pushed to the meta-xilinx-tools master branch was to prepend the recipe’s PATH variable with the host’s /usr/bin (see here). That is the very definition of host contamination (and it’s actually encouraged in the README with regard to installing Xvfb on the host). It invites all kinds of opportunity Yocto-built native packages to be superceded by the host environment. A better solution is to make the recipe(s) (or class) in question DEPENDS on ??-native packages already provided by Yocto. These will be compiled for the host processor architecture and exposed in the build environment. For anything else minor, like ps, selectively expose the specific commands into the environment (export a function pointing back to /usr/bin/ps, for example). Given Xilinx have a solution, I’m holding off on a pull request at this time but wanted to point out the less heavy-handed approach using Yocto features.
As mentioned before, the Xilinx machine definition for the ZCU102 does not include the FSBL binary (boot.bin, usually). Interestingly the FSBL binary produced by this layer is called BOOT.bin. It will end up on a FAT partition, so the case of the file name does not matter…except that the meta-xilinx layer’s u-boot-zynq-uenv recipe will treat BOOT.bin as an FPGA bitstream and try to load for you because the file name does not exactly, case-sensitive match boot.bin. There is a simple solution for this issue: the FSBL is put together as one of the last steps before building the root file system (using an image class), so you can simply patch that class to fix the case, which will ultimately overwrite the U-BOOT-XLNX -deployed one and not get injected as a bitsream load command into the uEnv.txt file.
Analog Devices
We made this layer and described it a bit in the FMCOMMS series of posts. Our more recent internal iterations handle a broader variety of devices and do not have the earlier drawback of a non-bootable kernel.
If you want to create a layer like this, your goal should be to have a kernel recipe for Analog Devices so that you can gain their driver support for the FMC card of your choice as well as the device tree. If you’re using meta-xilinx-tools, you’ll need to tweak U-BOOT-XLNX’s task dependencies, etc., to pull in the generated device tree rather than use the one extracted from system_top.hdf. And finally, you’ll need a means to setup the preferred provider to point at your new kernel, among other things (you can do this in local.conf, but you may find it easier to just require the Xilinx machine and modify/remove as needed).
If you would like access to this layer, please contact us.
REDHAWK SDR
We’ve been supporting this layer for a few years now and made some significant strides last year to support nearly every core asset found at REDHAWK SDR. Today as part of this exercise, we’ve tagged a v2.0.8-0 release that was used in this build. It provides REDHAWK SDR 2.0.8 and has been tested within a Yocto Rocko 2.4.1 build environment.
FEI Programmable Device
The FEI Programmable Device is part of another layer of ours that includes a variety of Persona Devices that work with this top-level Programmable device. Under normal operation, this device operates like a normal FEI transmitter/receiver. Under persona operation, it can inject a variety of IP cores into the data stream, i.e., within in the FPGA fabric, similar to our zedboard and RF-NoC work. This allows for hardware-accelerated Component deployments in REDHAWK.
As with our Analog Devices integration layer, this layer (and Device) are internal projects at this time. Feel free to contact us for more information.
Conclusion
Though the work on the FEI Programmable Device is ongoing, what we have today is a consistent, working build process for installing a Yocto Rocko 2.4.1 -built operating system with security hardening and REDHAWK onto a new target: the Xilinx ZCU102 evaluation board. And we were able to do it while also integrating with Analog Devices’ ADRV9371 HPC FMC evaluation board with LibIIO installed.
Keep checking in on the website periodically or contact us for more information as we prepare this platform for hardware accelerated FEI receiver integration.
