Easiest ARM 64-bit developer boards to get hold of and integrate are the Raspberry Pis, so I used 4 of them in a 1U cluster to provide both Linux kernel development lab machines as well as infrastructure like terminal server and test servers.
In this cluster I have two infrastructure servers (Raspberry Pi 3)and 2 dev/test/build servers (Raspberry Pi 4). This was the easiest way to get the lab project going, other ARM boards like that from Gigabyte are hard to find in India. I may add Nvidia Tegra (e.g. Jetson) to the mix in future.
To pack everything into the rack, I am using a UCTRONICS Ultimate Rack for Raspberry Pi 4 to mount the Pi devices into 1U. Even thought it is for Pi 4, I could also mount Pi 3s in it. For each Pi, I am using REES52 Metal Aluminium Alloy Raspberry Pi 4 Armor Case Cooling Heatsink for Raspberry Pi. These heatsinks are available for both Pi 3 and Pi 4.
Serial console for Linux kernel development
The devtest systems also have serial console provision using USB to TTL serial cables connecting it to the required (serial communication) GPIO pins of Pi [Update: The PL2303 based cables didn’t work reliably. Instead, I have been using Waveshare (original) USB to TTL serial cables with FT232RNL chip. See this post for serial console setup details] For the documentation on GPIO pinout, refer to the Pi documentation here. The TTL side has 4 color coded cables with 2.54mm female jumper connectors which fits perfectly with GPIO pins.
No need to connect Vcc (red) as it is already provided. Rx (white) of the TTL side connects to the Tx (pin 8) of GPIO. Tx (green) of the TTL side connects to the Rx (pin 10) of GPIO. GND (black) is connected to GND (pin 14) on the GPIO port. [Update: With the Waveshare FT232RNL based serial cable, GND is black, Tx is green and Rx is blue.]
To enable it in Linux, I edited /boot/config.txt and changed the following line :
#serial console enable_uart=1
I also edited /boot/cmdline.txt and changed the kernel command line :
console=ttyAMA0,115200 console=tty0 root=PARTUUID=3188ead5-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash
ttyAMA0 is UART0 and is available via GPIO pin 8 and 10 (Tx and Rx). By default, Pi OS sets itself up with boot log suppressed using plymouth.ignore-serial-consoles in /boot/cmdline.txt. To see the boot log on serial console, that needs to be removed.
[dipankar@conan ~]$ telnet pits1 3000
Trying 192.168.10.101...
Connected to pits1.
Escape character is '^]'.
ser2net port tcp,3000 device serialdev, /dev/ttyUSB0, 115200n81,local [,115200N81,CLOCAL] (Debian GNU/Linux)
[ OK ] Started Show Plymouth Boot Screen.
[ OK ] Started Forward Password R…s to Plymouth Directory Watch.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Found device /dev/ttyAMA0.
[ OK ] Found device /dev/disk/by-partuuid/3188ead5-01.
Starting File System Check…isk/by-partuuid/3188ead5-01...
[ OK ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.
[ OK ] Finished File System Check…/disk/by-partuuid/3188ead5-01.
Mounting /boot...
[ OK ] Mounted /boot.
[ OK ] Reached target Local File Systems.
........
[Trimmed]
[ OK ] Started Authorization Manager.
Starting Modem Manager...
[ OK ] Started LSB: Switch to ond…(unless shift key is pressed).
[ OK ] Started OpenBSD Secure Shell server.
Debian GNU/Linux 11 aarch64-pi1 ttyAMA0
aarch64-pi1 login: dipankar
Password:
Linux aarch64-pi1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul 3 03:29:28 IST 2023 from 192.168.2.137 on pts/0
Wi-Fi is currently blocked by rfkill.
Use raspi-config to set the country before use.
dipankar@aarch64-pi1:~$ uname -a
Linux aarch64-pi1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
I am using the EEPROM based bootloader (default) on the RPis. To enable the bootloader console on serial port, I did the following using the rpi-eeprom-config tool :
dipankar@aarch64-pi1:~ $ sudo rpi-eeprom-config --edit
Updating bootloader EEPROM
image: /lib/firmware/raspberrypi/bootloader/default/pieeprom-2023-01-11.bin
config_src: /boot/pieeprom.upd
config: /tmp/tmp1cs0zp8u/boot.conf
################################################################################
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
################################################################################
*** To cancel this update run 'sudo rpi-eeprom-update -r' ***
*** INSTALLING /tmp/tmp1cs0zp8u/pieeprom.upd ***
CURRENT: Wed 11 Jan 2023 05:40:52 PM UTC (1673458852)
UPDATE: Wed 11 Jan 2023 05:40:52 PM UTC (1673458852)
BOOTFS: /boot
Using recovery.bin for EEPROM update
EEPROM updates pending. Please reboot to apply the update.
To cancel a pending update run "sudo rpi-eeprom-update -r".
It invokes nano editor on the config file to be written to the EEPROM. I set BOOT_UART to 1.
GNU nano 5.4 /tmp/tmp1cs0zp8u/boot.conf *
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
[ line 2/6 (33%), col 12/12 (100%), char 17/54 (31%) ]
^G Help ^O Write Out ^W Where Is ^K Cut ^T Execute ^C Location
^X Exit ^R Read File ^\ Replace ^U Paste ^J Justify ^_ Go To Line
This enables RPi bootloader messages on UART (serial console) :
RPi: BOOTLOADER release VERSION:8ba17717 DATE: 2023/01/11 TIME: 17:40:52 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1673458852 serial 3f26a675 boardrev d03115 stc 663091 PM_RSTS: 0x00000020 part 00000000 reset_info 00000000 uSD voltage 3.3V Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200 DDR 3200 1 0 64 152 XHCI-STOP xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb USBSTS 1 xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb xHC ports 5 slots 32 intrs 4 Boot mode: SD (01) order f4 USB2[1] 400202e1 connected USB2 root HUB port 1 init DEV [01:00] 2.16 000000:01 class 9 VID 2109 PID 3431 HUB init [01:00] 2.16 000000:01 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 OCR c0ff8000 [358] CID: 000353445344323536857df140290173 CSD: 400e00325b590007725f7f800a404000 SD: bus-width: 4 spec: 2 SCR: 0x02458043 0x00000000 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2 [Trimmed]
Software
I am using standard microSD cards to boot the devices. The process is very well documented – just burn the image on the microSD, put it in the slot and the device boots. Pi devices are very well supported with multiple linux distros, but so far I have only been using Pi OS which is debian based. Going forward I intend to also run Fedora in these devices.
Notes :
At the time of writing this post, I see intermittent errors from the pl2303 USB to serial adapter driver in the USB->serial cable I am using. While the serial console works, I am seeing a hang after /dev/ttyUSBX device is opened and closed multiple times by the serial console application (to be described in a future post). I will update the post with a more robust working solution if I find one later.
Update [1st Dec, 2024]:
I have somewhat upgraded the serial console USB hub to get more reliable serial consoles in the rack. I am now using an official Raspberry USB 3.0 hub with separate power which replaced the earlier generic USB hub without power.