RISC-V SoC + AI | Run a demo of the ncnn inference framework use Allwinner’s D1 “Nezha” Development Board

0.Introduction

D1 is Allwinner's first SoC based on the RISC-V ISA,which has a 64-bit Xuantie C906 core from T-Head. The "Nezha" Development Board is an AIoT development board based on the D1 SoC customized by Allwinner Online. It is a rare mass-produced development board that uses RISC-V SoC and can run the GNU/Linux operating system.

ncnn is a high-performance neural network forward computing framework launched by Tencent Youtu Lab that is extremely optimized for mobile device. It is also a relatively rare neural network framework that has been adapted and optimized for the RISC-V architecture.

This article is a tutorial that step-by-step shows step by step how to run a demo of the ncnn neural network inference framework on a brand new Allwinner D1 "Nezha" Development Board.

The completion of this article refers to the following materials, and I am very grateful to nihui, BedRock and other developers for sharing on the Internet!
[1] D1 Nezha - Online Documentation
[2] nihui: 在全志d1开发板上玩ncnn (in chinese)
[3] 腾讯优图ncnn新版本适配国产CPU,全志D1加持最高速度提升70倍!(in chinese)
This article is also allowed to be reprinted with the source of this article attached: https://verimake.com/d/35

1.Necessary Materials

  • “Nezha” Development Board
  • Type-C USB Cable
  • USB to TTL Converter
  • Internet-connected computer running GNU/Linux or Windows 10/11
  • (This article uses a Windows 10 computer as an example)

2.Power on!

Take out the Nezha and the USB to TTL converter, first connect the converter to the DEBUG port as shown in the figure below, and then plug the converter into the USB port of the computer. If the computer does not recognize the serial port, it may be because the driver for the converter is not installed, which can be downloaded here.

When using the initial firmware that comes with a new board, the DEBUG port will provide a Shell via the 115200 baud rate UART serial port when the board is powered on. We can use PuTTY, MobaXterm or other similar terminal emulators to access this serial port to operate the board in the shell. (If using PuTTY, you can refer to the following figure to set it up)

After starting the terminal emulator, temporarily we can only see a black box, because the development board has not been powered on.

Find the Type-C USB cable and connect the OTG` port on the board to the computer to power it on (although you can also connect to `POWER`port, you will need to use this `OTG port in the following steps, so it is recommended). After powering up the board, you will notice that the LEDs on the board light up and various prompts start popping up in the terminal emulator! After about 10 to 20 seconds, you can notice a line letting you press enter. So, press enter in the terminal emulator, then you can access the screen shown below, and can execute various GNU/Linux commands.

3.Re-flash the firmware

The firmware that comes with a newly opened development board is based on the Open v1.0 version of the D1 Tina SDK, and an illegal command error may occurs if run the ncnn programs (see here). According to a comment by BedRock in the Allwinner Online Developer Forum, before next steps, we can update the firmware to a new version based on Open v1.01 SDK

If you want to compile your own firmware or modify it, you can download the SDK by referring to the "Tina SDK Versions" section in the D1 documentation. However if you just want to run a ncnn for fun, you can just select one of the firmwares from the "Firmware Downloads" section. For example, I can choose the D1 Nezha HDMI test firmware 20210804 (a little penguin logo will display through HDMI after powering on) (the name is a bit long) as an example. This firmware is presumably designed to demonstrate the HDMI output function, so that we can see a little penguin when we connect a screen to the development board with an HDMI cable, but since it is compiled based on the v1.01 SDK, we can just as well use it to run the ncnn demo.

After downloading the .img` firmware file, refer to the ["Download" section of "Compile and download"](https://d1.docs.aw-ol.com/source/4_tinaversion/) in the D1 documentation, we can download the [Allwinner USB driver](https://www.aw-ol.com/downloads/resources/15). Then run `install.bat with administrator privileges to install the driver, and then install the download software to operate, so that the firmware can be downloaded into the development board. For example, in Windows computer, we need to download PhoenixSuit software to download the firmware. PS: you need to select "full disk erase upgrade" in the download software, as shown in the figure below, otherwise the download may not be successful.

After downloading successfully, we can restart the board.

4. Install and configure the cross-compiler toolchain

Note: The steps in Sections 4 and 5 require to do in GNU/Linux operating system. If you have installed GNU/Linux operating system on your computer, or use a virtual machine, you can follow the steps directly in the GNU/Linux operating system. If your computer is installed with a relatively new version of Windows 10 / 11, you can also search for related tutorials on the Internet to install and configure a WSL (Windows Subsystem Linux), and then follow the steps in WSL.

On the "Resource Download"page of the T-Head open chip community, we can find the V2.0.1 version of riscv64-linux-x86_64-20210512.tar.gz in "Toolchain-900 Series", download it to your computer, put it in , unzip it

tar -xf riscv64-linux-x86_64-20210512.tar.gz
After extracting the .tar.gz file, use vim` or other text editor to open`~/.bashrc, and add the path of the toolchain to the environment variable at the end, that is, the following content

# My PATH
export RISCV_ROOT_PATH=/<path where you want to put it>/riscv64-linux-x86_64-20210512

Save and close the editor, use the source command to refresh the Shell environment:

source ~/.bashrc

Then we have finished installing and configuring the cross-compilation toolchain.

5. Compile ncnn and prepare the demo program

Download the source code of ncnn on GitHub which released on the July 20, 2021 to your computer, extract it, enter the ncnn-20210720 directory, prepare and start compiling

mkdir build-c906
cd build-c906
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/c906.toolchain.cmake -DCMAKE_BUILD_TYPE=relwithdebinfo -DNCNN_OPENMP=OFF -DNCNN_THREADS=OFF -DNCNN_RUNTIME_CPU=OFF -DNCNN_RVV=ON -DNCNN_SIMPLEOCV=ON -DNCNN_BUILD_EXAMPLES=ON ..
make -j32

Compilation takes about a few minutes, after finished, all output files can be found in the build-c906 directory.
We can try two demos first, they are:

  • ① use benchncnn to do benchmark test
  • ② use NanoDet model to do object detection on your own picture (NanoDet is an super fast and high accuracy lightweight anchor-free object detection model)

In order to transfer files transferring to the development board, we can first create two folders, benchncnn_demo` and `nanodet_demo on the computer, then put the required files in ① and ② separately and get ready.

For ①, we copy the build-c906/benchmark/benchncnn` and `benchmark/*.param in the ncnn directory and paste them into benchncnn_demo.
For ②, we need to copy build-c906/examples/nanodet` in the ncnn directory and paste it into nanodet demo. Then we download the two files namede `nanodet_m.bin` and `nanodet_m.param in ncnn-assets, and put them along the pictures you want to detect into nanodet_demo.

The structure of the two directories is similar like this:

benchncnn_demo/
├ benchncnn
├ alexnet.param
├ blazeface.param
└ <a bunch of other models’ param>

nanodet_demo/
├ nanodet
├ nanodet_m.bin
├ nanodet_m.param
└ <the pictures you want to detect>

6.Use ADB to transfer files

ADB's full name is Android Debug Bridge, which is designed for Android. Since Allwinner has also adapted it to Tina Linux, so we can also use ADB to debug Tina Linux devices. Refer to the "Development & Production Tools" section in D1 documentation to download. After the download, add its path to the environment variable to use it.

When we run these two demos, we may only need to use the following commands. If you need to use other functions, you can also search for related tutorials on the Internet.

To check the connection between the devices and the computer, you can enter

adb devices

adb pushto transfer files or folders on the computer to the development board, the syntax is as follows

adb push <file path on computer> <target path on board>

adb pull To transfer files or folders on the development board to the computer, the syntax is as follows

adb pull <file path on board> <target path on computer>

It should be noted that the paths on the development board need to be absolute path.

7.Run demo

Transfer the benchncnn_demo` and `nanodet_demo` folders shown in Section 5 to the development board through the `adb push` command. Then enter the `benchncnn_demo directory on the development board, and then enter

./benchncnn 4 1 0 -1 0

You can start benchmarking. If it runs successfully, you will see a prompts like the one below, showing the milliseconds ncnn takes to use various models to inference a frame on the current platform.

(If you don't want to wait for the whole test, you can also press Ctrl + C to stop it)

Enter the nanonodet_demo directory on the development board, then enter

./nanodet <The image you want to detect>

To use NanoDet to detect objects in this image. After finishing, you can see the text version of the detection results in the command line. You can also use adb pull` to transfer the generated `image.png image file to the computer and then open it for viewing with bounder boxes, categories and confidence sources drawn on it. For example, there are cars detected on the Nanjing Yangtze River Bridge.

At this point, we have successfully run two demos of the neural network framework on a RISC-V development board running the GNU/Linux system. If you want to carry out follow-up experiments or research and development, you can learn more about D1 Nezha and ncnn .

Leave a Reply

Your email address will not be published. Required fields are marked *