{"id":180,"date":"2023-07-04T11:36:16","date_gmt":"2023-07-04T06:06:16","guid":{"rendered":"https:\/\/hill9.org\/journal\/?p=180"},"modified":"2024-12-05T12:59:55","modified_gmt":"2024-12-05T07:29:55","slug":"home-lab-systems-arm","status":"publish","type":"post","link":"https:\/\/hill9.org\/journal\/2023\/07\/04\/home-lab-systems-arm\/","title":{"rendered":"Home Lab Linux Systems – ARM"},"content":{"rendered":"\n

Easiest ARM 64-bit developer boards to get hold of and integrate are the Raspberry Pis, so I used 4 of them in a 1U cluster to provide both Linux kernel development lab machines as well as infrastructure like terminal server and test servers.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

In this cluster I have two infrastructure servers (Raspberry Pi 3)and 2 dev\/test\/build servers (Raspberry Pi 4). This was the easiest way to get the lab project going, other ARM boards like that from Gigabyte are hard to find in India. I may add Nvidia Tegra (e.g. Jetson) to the mix in future.<\/p>\n\n\n\n

To pack everything into the rack, I am using a UCTRONICS Ultimate Rack for Raspberry Pi 4<\/a> to mount the Pi devices into 1U. Even thought it is for Pi 4, I could also mount Pi 3s in it. For each Pi, I am using REES52 Metal Aluminium Alloy Raspberry Pi 4 Armor Case Cooling Heatsink for Raspberry Pi<\/a>. These heatsinks are available for both Pi 3 and Pi 4.<\/p>\n\n\n\n

Serial console<\/strong> for Linux kernel development<\/h2>\n\n\n\n

The devtest systems also have serial console provision using USB to TTL serial cables<\/a> connecting it to the required (serial communication) GPIO pins of Pi [Update: The PL2303 based cables didn’t work reliably. Instead, I have been using Waveshare (original) USB to TTL serial cables with FT232RNL chip<\/a>.<\/em><\/strong> See this post<\/a> for serial console setup details<\/em><\/strong>] For the documentation on GPIO pinout, refer to the Pi documentation here<\/a>. The TTL side has 4 color coded cables with 2.54mm female jumper connectors which fits perfectly with GPIO pins.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

No need to connect Vcc (red) as it is already provided. Rx (white) of the TTL side connects to the Tx (pin 8) of GPIO. Tx (green) of the TTL side connects to the Rx (pin 10) of GPIO. GND (black) is connected to GND (pin 14) on the GPIO port. [Update: With the Waveshare FT232RNL based serial cable, GND is black, Tx is green and Rx is blue.<\/strong><\/em>]<\/p>\n\n\n\n

To enable it in Linux, I edited \/boot\/config.txt<\/em> and changed the following line :<\/p>\n\n\n\n

#serial console\nenable_uart=1<\/pre>\n\n\n\n
I also edited \/boot\/cmdline.txt <\/em>and changed the kernel command line :<\/p>\n\n\n\n
console=ttyAMA0,115200 console=tty0 root=PARTUUID=3188ead5-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash<\/pre>\n\n\n\n
ttyAMA0<\/em> is UART0 and is available via GPIO pin 8 and 10 (Tx and Rx). By default, Pi OS sets itself up with boot log suppressed using plymouth.ignore-serial-consoles<\/em> in \/boot\/cmdline.txt<\/em>. To see the boot log on serial console, that needs to be removed.<\/p>\n\n\n\n
[dipankar@conan ~]$ telnet pits1 3000\nTrying 192.168.10.101...\nConnected to pits1.\nEscape character is '^]'.\n\nser2net port tcp,3000 device serialdev, \/dev\/ttyUSB0, 115200n81,local [,115200N81,CLOCAL] (Debian GNU\/Linux)\n\n[  OK  ] Started Show Plymouth Boot Screen.\n[  OK  ] Started Forward Password R\u2026s to Plymouth Directory Watch.\n[  OK  ] Reached target Local Encrypted Volumes.\n[  OK  ] Found device \/dev\/ttyAMA0.\n[  OK  ] Found device \/dev\/disk\/by-partuuid\/3188ead5-01.\n         Starting File System Check\u2026isk\/by-partuuid\/3188ead5-01...\n[  OK  ] Listening on Load\/Save RF \u2026itch Status \/dev\/rfkill Watch.\n[  OK  ] Finished File System Check\u2026\/disk\/by-partuuid\/3188ead5-01.\n         Mounting \/boot...\n[  OK  ] Mounted \/boot.\n[  OK  ] Reached target Local File Systems.\n........\n[Trimmed]\n[  OK  ] Started Authorization Manager.\n         Starting Modem Manager...\n[  OK  ] Started LSB: Switch to ond\u2026(unless shift key is pressed).\n[  OK  ] Started OpenBSD Secure Shell server.\n\nDebian GNU\/Linux 11 aarch64-pi1 ttyAMA0\n\naarch64-pi1 login: dipankar\nPassword: \nLinux aarch64-pi1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64\n\nThe programs included with the Debian GNU\/Linux system are free software;\nthe exact distribution terms for each program are described in the\nindividual files in \/usr\/share\/doc\/*\/copyright.\n\nDebian GNU\/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\npermitted by applicable law.\nLast login: Mon Jul  3 03:29:28 IST 2023 from 192.168.2.137 on pts\/0\n\nWi-Fi is currently blocked by rfkill.\nUse raspi-config to set the country before use.\n\ndipankar@aarch64-pi1:~$ uname -a\nLinux aarch64-pi1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU\/Linux<\/pre>\n\n\n\n
I am using the EEPROM based bootloader (default) on the RPis. To enable the bootloader console on serial port, I did the following using the rpi-eeprom-config<\/em> tool  :<\/p>\n\n\n\n
dipankar@aarch64-pi1:~ $ sudo rpi-eeprom-config --edit\nUpdating bootloader EEPROM\n image: \/lib\/firmware\/raspberrypi\/bootloader\/default\/pieeprom-2023-01-11.bin\nconfig_src: \/boot\/pieeprom.upd\nconfig: \/tmp\/tmp1cs0zp8u\/boot.conf\n################################################################################\n[all]\nBOOT_UART=1\nWAKE_ON_GPIO=1\nPOWER_OFF_ON_HALT=0\n\n\n################################################################################\n\n*** To cancel this update run 'sudo rpi-eeprom-update -r' ***\n\n*** INSTALLING \/tmp\/tmp1cs0zp8u\/pieeprom.upd  ***\n\n   CURRENT: Wed 11 Jan 2023 05:40:52 PM UTC (1673458852)\n    UPDATE: Wed 11 Jan 2023 05:40:52 PM UTC (1673458852)\n    BOOTFS: \/boot\nUsing recovery.bin for EEPROM update\n\nEEPROM updates pending. Please reboot to apply the update.\nTo cancel a pending update run \"sudo rpi-eeprom-update -r\".<\/pre>\n\n\n\n
It invokes nano<\/em> editor on the config file to be written to the EEPROM. I set BOOT_UART <\/em>to 1. <\/em><\/p>\n\n\n\n
 GNU nano 5.4               \/tmp\/tmp1cs0zp8u\/boot.conf *                       \n[all]\nBOOT_UART=1\nWAKE_ON_GPIO=1\nPOWER_OFF_ON_HALT=0\n\n\n\n\n             [ line 2\/6 (33%), col 12\/12 (100%), char 17\/54 (31%) ]\n^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location\n^X Exit      ^R Read File ^\\ Replace   ^U Paste     ^J Justify   ^_ Go To Line\n\n<\/pre>\n\n\n\n
This enables RPi bootloader messages on UART (serial console) :<\/p>\n\n\n\n
RPi: BOOTLOADER release VERSION:8ba17717 DATE: 2023\/01\/11 TIME: 17:40:52\nBOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1673458852 serial 3f26a675 boardrev d03115 stc 663091\nPM_RSTS: 0x00000020\npart 00000000 reset_info 00000000\nuSD voltage 3.3V\nInitialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200\nDDR 3200 1 0 64 152\n\nXHCI-STOP\nxHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb\nUSBSTS 1\nxHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb\nxHC ports 5 slots 32 intrs 4\nBoot mode: SD (01) order f4\nUSB2[1] 400202e1 connected\nUSB2 root HUB port 1 init\nDEV [01:00] 2.16 000000:01 class 9 VID 2109 PID 3431\nHUB init [01:00] 2.16 000000:01\nSD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276\nSD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276\nOCR c0ff8000 [358]\nCID: 000353445344323536857df140290173\nCSD: 400e00325b590007725f7f800a404000\nSD: bus-width: 4 spec: 2 SCR: 0x02458043 0x00000000\nSD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2\n[Trimmed]           <\/pre>\n\n\n\n
Software<\/strong><\/h2>\n\n\n\n
I am using standard microSD cards to boot the devices. The process is very well documented – just burn the image on the microSD, put it in the slot and the device boots. Pi devices are very well supported with multiple linux distros, but so far I have only been using Pi OS which is debian based.  Going forward I intend to also run Fedora in these devices.<\/p>\n\n\n\n
Notes :<\/strong><\/h2>\n\n\n\n
At the time of writing this post, I see intermittent errors from the pl2303 USB to serial adapter driver in the USB->serial cable I am using. While the serial console works, I am seeing a hang after \/dev\/ttyUSBX device is opened and closed multiple times by the serial console application (to be described in a future post). I will update the post with a more robust working solution if I find one later.<\/p>\n\n\n\n
Update [1st Dec, 2024]:<\/h2>\n\n\n\n
I have somewhat upgraded the serial console USB hub to get more reliable serial consoles in the rack. I am now using an official Raspberry USB 3.0 hub with separate power which replaced the earlier generic USB hub without power.<\/p>\n","protected":false},"excerpt":{"rendered":"
Easiest ARM 64-bit developer boards to get hold of and integrate are the Raspberry Pis, so I used 4 of them in a 1U cluster to provide both Linux kernel development lab machines as well as infrastructure like terminal server and test servers. In this cluster I have two infrastructure servers (Raspberry Pi 3)and 2…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[6,3,5,13],"tags":[],"class_list":["post-180","post","type-post","status-publish","format-standard","hentry","category-homelab","category-linux","category-oss","category-raspberry-pi"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts\/180","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/comments?post=180"}],"version-history":[{"count":34,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts\/180\/revisions"}],"predecessor-version":[{"id":628,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts\/180\/revisions\/628"}],"wp:attachment":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/media?parent=180"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/categories?post=180"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/tags?post=180"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}