As described in the earlier posts, I have connected to various COM ports of different systems using USB to serial cables, most of these use PL2303 adapter which is well-supported in Linux [Update: PL2303 didn’t work reliably, see the update section below on serial HW<\/strong><\/em>]. I am using one of the infrastructure RPis as the terminal server for this along with a USB hub where I can connect the USB end of all the serial cables. This provides all the serial ports at \/dev\/ttyUSB0, \/dev\/ttyUSB1…<\/em> etc. Instead of requiring to log into this terminal server and run something like minicom<\/em> to connect to the serial port, I am using ser2net<\/em> as a service to provide connection to the serial consoles from anywhere in my home network, for example from my development workstation. ser2net<\/em> is a network to serial proxy allowing a client to establish telnet connections to a serial port. I set up ser2net<\/em> by editing \/etc\/ser2net.yaml<\/em> configuration file (not <\/em><\/strong>\/etc\/ser2net.conf<\/em> as in earlier versions of ser2net<\/em>).<\/p>\n\n\n\n The 115200n81<\/em> indicates the baudrate, word length and parity of serial communication link. To connect to the first serial port of the terminal server, I do the following :<\/p>\n\n\n\n The pl2303 based cables from Amazon [REES52 USB to TTL serial cable] turned out to be unreliable in the long run. After using them for some time, I used get these error messages :<\/p>\n\n\n\n After a while, the console server had to be rebooted. Perhaps the quality of the cable or the version of PL2303 chip was bad. In any case, I have replaced them all by Waveshare (original) USB to TTL serial cable with FT232RNL chip<\/a>. The FT232 chip seems to be a lot more reliable than the PL2303 I was using in the older cable. For 2U servers with DB9 male connectors, I am using Waveshare (original) USB to DB9 female serial cable<\/a>. Both of these are considerably expensive than the the older cables, but they are original Waveshare cables and build quality is much better, in my opinion.<\/p>\n\n\n\n One other issue that crops up with both Debian and Fedora distributions (irrespective of arch) is how agetty <\/em>is run on the serial port (console). By default, agetty<\/em> is run to cycle through multiple baud rates if the connection is somehow broken. This is how it is set up in \/lib\/systemd\/system\/serial-getty@.service<\/em> :<\/p>\n\n\n\n This causes agetty<\/em> to set the serial port speed to 57600 baud if the connection is broken somehow at 115200 baud. After this the serial console will stop working as ser2net<\/em> is configured to use a fixed baud rate of 115200 in this example setup. With the baud rate mismatch, you will see non-printable characters on the serial console like below :<\/p>\n\n\n\n The solution is to run agetty<\/em> at only one speed in the target Linux systems – 115200.<\/p>\n\n\n\n This prevents the random change of serial port speed in the target system.<\/p>\n\n\n\n I implemented a PXE service in the home lab with the intention that I wouldn’t have to manage installation media and I could repetitively do automated provisioning in future. PXE service works in two parts :<\/p>\n\n\n\n tftp and DNS service using dnsmasq : <\/strong>While PXE boot can transfer files over multiple protocols like http and tftp, for simplicity, I chose tftp. PXE boot also uses DHCP request\/response to discover PXE servers in the network. I made the necessary changes in \/etc\/dnsmasq.conf<\/em> to set up both of these services for UEFI base x86 servers. This PXE service in my home lab currently provides Fedora only. Eventually, I intend to convert it to a multi-PXE installation server. I added the following lines to \/etc\/dnsmasq.conf<\/em> :<\/p>\n\n\n\n tftp-root<\/em> sets up the root directory for tftp service. Subsequently, PXE boot files for different vendor class can be set up in different directories under tftp-root<\/em> and only the relative path needs to be used in dnsmasq.conf<\/em>. Since tftp<\/em> service is provided by dnsmasq<\/em>, I disabled the standard tftp<\/em> service by removing the tftpd-hpa<\/em> package.<\/p>\n\n\n\n In my PXE boot setup, I am using pxe-service<\/em> directive to specify what files to be booted over PXE by the client system. The first parameter to the pxe-service<\/em> command is the client system architecture. There is a set of pre-defined client architecture (e.g. X86-64_EFI<\/em>) as per the man page of dnsmasq<\/em> – x86PC, PC98, syslinux<\/em> supports interactive menus which can be used to offer choice of different boot options like distributions to the user. The menu can be defined in the file $tftp-root\/pxelinux.cfg\/default<\/em>. Here is a snippet of my menu configuration which implements multi-OS \/ multi-architecture boot (Fedora and debian) and installation:<\/p>\n\n\n\n In the syslinux<\/em> menu configuration, ::<\/em> signifies the full path for tftp-root<\/em>. This is done because syslinux.efi<\/em> in my case runs from $tftp-root\/uefi<\/em> directory and hence pxelinux.cfg<\/em> is a symlink to another location. For the installation kernel and initrd, I used the following:<\/p>\n\n\n\n For example, I did this with Fedora :<\/p>\n\n\n\n With these in place, here are some of the results in terms of screenshots :<\/p>\n\n\n\n Fedora graphical installation :<\/p>\n\n\n\n Debian installation: (I didn’t get around to make gtk based installation work)<\/p>\n\n\n\n There are a number of useful pages that were very helpful for me.<\/p>\n\n\n\n They sometimes have conflicting information, but by and large (1) worked for me well.<\/p>\n\n\n\n Note:<\/strong> If there is any reference to arm64 in my setup, it is not complete at the time of writing this post. Only amd64 works with fedora and debian PXE network installation as of now.<\/p>\n","protected":false},"excerpt":{"rendered":" To get things going, I have implemented two Linux infrastructure services in the lab – serial console and PXE service. The serial console service allows me to connect to the serial port of any of the systems in the lab rack. PXE services provides PXE boot capabilities, for now only Fedora installation over network. Serial…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[6,8,3,5],"tags":[],"class_list":["post-260","post","type-post","status-publish","format-standard","hentry","category-homelab","category-kernel","category-linux","category-oss"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts\/260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/comments?post=260"}],"version-history":[{"count":47,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts\/260\/revisions"}],"predecessor-version":[{"id":641,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/posts\/260\/revisions\/641"}],"wp:attachment":[{"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/media?parent=260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/categories?post=260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hill9.org\/journal\/wp-json\/wp\/v2\/tags?post=260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}%YAML 1.1\n---\n# This is a ser2net configuration file, tailored to be rather\n# simple.\n#\n# Find detailed documentation in ser2net.yaml(5)\n# A fully featured configuration file is in\n# \/usr\/share\/doc\/ser2net\/examples\/ser2net.yaml.gz\n#\n# If you find your configuration more useful than this very simple\n# one, please submit it as a bugreport\n\ndefine: &banner \\r\\nser2net port \\p device \\d [\\B] (Debian GNU\/Linux)\\r\\n\\r\\n\n\nconnection: &con0096\n accepter: tcp,3000\n enable: on\n options:\n banner: *banner\n kickolduser: true\n telnet-brk-on-sync: true\n connector: serialdev,\n \/dev\/ttyUSB0,\n 115200n81,local\n\nconnection: &con1096\n accepter: tcp,3001\n enable: on\n options:\n banner: *banner\n kickolduser: true\n telnet-brk-on-sync: true\n connector: serialdev,\n \/dev\/ttyUSB1,\n 115200n81,local<\/pre>\n\n\n\n
[dipankar@conan ~]$ telnet pits1 3000\nTrying 192.168.10.101...\nConnected to pits1.\nEscape character is '^]'.\n\nser2net port tcp,3000 device serialdev, \/dev\/ttyUSB0, 115200n81,local [] (Debian GNU\/Linux)\n\n\ufffd\nPassword: \n\nLogin incorrect\naarch64-pi1 login: <\/pre>\n\n\n\n
Update on serial console setup [Dec, 2024]<\/h2>\n\n\n\n
[ 889.867709] pl2303 ttyUSB1: pl2303_get_line_request - failed: -32
[ 889.870804] pl2303 ttyUSB1: pl2303_get_line_request - failed: -32
[ 889.872675] pl2303 ttyUSB1: pl2303_get_line_request - failed: -32
[ 889.875285] pl2303 ttyUSB1: error sending break = -32
[ 904.942551] pl2303 ttyUSB1: pl2303_get_line_request - failed: -32
[ 905.229713] pl2303 ttyUSB1: error sending break = -32
[ 915.026993] pl2303 ttyUSB2: error sending break = -110
[ 952.579416] pl2303 1-1.3.2:1.0: pl2303_vendor_write - failed to write [0008]: -110
[ 952.683426] pl2303 1-1.3.2:1.0: pl2303_vendor_write - failed to write [0009]: -110
[ 952.787412] pl2303 ttyUSB1: pl2303_get_line_request - failed: -110<\/pre>\n\n\n\n![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/image-2-955x1024.png\")
![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/image-3-1024x1024.png\")
# The '-o' option value tells agetty to replace 'login' arguments with an
# option to preserve environment (-p), followed by '--' for safety, and then
# the entered username.
ExecStart=-\/sbin\/agetty -o '-p -- \\\\u' --keep-baud 115200,57600,38400,9600 %I $TERM<\/pre>\n\n\n\nDebian GNU\/Linux 11 aarch64-pi2 ttyS0
\ufffd\ufffd<<?<\ufffd\ufffdx`\ufffd\ufffd\ufffd\ufffd<\ufffd~<\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u6018\ufffd`0f<<\ufffd\ufffd<\ufffd\ufffd\ufffd`\u00d8f3\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdf?\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdf\ufffd\ufffd\ufffd\ufffd\ufffd\u03063\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdf\ufffd\ufffd\ufffd03`\ufffd\ufffd\ufffd\ufffd<<?<\ufffd\ufffdx`\ufffd\ufffd\ufffd\ufffd<\ufffd~<\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd<\/pre>\n\n\n\n[Service]
# The '-o' option value tells agetty to replace 'login' arguments with an
# option to preserve environment (-p), followed by '--' for safety, and then
# the entered username.
ExecStart=-\/sbin\/agetty -o '-p -- \\\\u' --keep-baud 115200 %I $TERM<\/pre>\n\n\n\nPXE Service<\/strong> for Linux Installation<\/h2>\n\n\n\n
port=0\nenable-tftp\ninterface=eth0\n# and don't bind to 0.0.0.0\nbind-interfaces\n# extra logging\nlog-queries\nlog-facility=\/var\/log\/dnsmasq.log\n# default location of tftp-server on Fedora\ntftp-root=\/home\/dipankar\/data\/netboot\/\n\n#PXE services for different architectures\npxe-service=X86-64_EFI,\"Network Boot from pits1 (amd64 UEFI)\",uefi\/syslinux.efi\npxe-service=22,\"Network Boot from pits1 (ARM64 uboot)\",rpi\/bootcode.bin\n\n# Use DNS proxy mode\ndhcp-range=192.168.10.101,proxy<\/pre>\n\n\n\n
IA64_EFI, Alpha, Arc_x86, Intel_Lean_Client, IA32_EFI, x86-64_EFI, Xscale_EFI, BC_EFI, ARM32_EFI <\/em>and ARM64_EFI<\/em>. The actual values for client architectures in real systems use values defined in IANA DHCPv6 standards<\/a> even though RFC 4578<\/a> has a slightly different set of values. The second parameter is a string to display and the third parameter is a file (program) to boot the PXE client system. For my x86_64 UEFI systems, I am using the syslinux package booting instead of the EFI bootloaders (e.g. bootnetx64.efi<\/em>, shimx64.efi<\/em>) provided by different distros. This allows me to have a distro agnostic PXE server setup. The boot program in my setup is syslinux.efi<\/em> which along with a set of other associated programs (ldlinux.e64, vesamenu.c32, libcom2.c32, libutil.c32<\/em> – ships with syslinux<\/em> package) need to be put in a TFTP accessible directory. In my case, the UEFI boot program directory is $tftp-root\/uefi<\/em>. Here is a picture of what my tftp-root<\/em> directory structure looks like:<\/p>\n\n\n\ndipankar@pits1:~\/data\/netboot$ tree\n.\n\u251c\u2500\u2500 debian\n\u2502 \u2514\u2500\u2500 12\n\u2502 \u251c\u2500\u2500 amd64\n\u2502 \u2502 \u2514\u2500\u2500 installer\n\u2502 \u2502 \u251c\u2500\u2500 initrd.gz\n\u2502 \u2502 \u2514\u2500\u2500 linux\n\u2502 \u2514\u2500\u2500 arm64\n\u251c\u2500\u2500 fedora\n\u2502 \u2514\u2500\u2500 38\n\u2502 \u2514\u2500\u2500 x86_64\n\u2502 \u251c\u2500\u2500 initrd.img\n\u2502 \u2514\u2500\u2500 vmlinuz\n\u251c\u2500\u2500 pxelinux.cfg\n\u2502 \u251c\u2500\u2500 default\n\u2502 \u251c\u2500\u2500 default-arm-bcm283x\n\u2502 \u2514\u2500\u2500 default.bak\n\u251c\u2500\u2500 rpi\n\u2502 \u251c\u2500\u2500 bootcode.bin\n\u2502 \u251c\u2500\u2500 start4.elf\n\u2502 \u2514\u2500\u2500 start.elf\n\u2514\u2500\u2500 uefi\n \u251c\u2500\u2500 ldlinux.e64\n \u251c\u2500\u2500 libcom32.c32\n \u251c\u2500\u2500 libutil.c32\n \u251c\u2500\u2500 pxelinux.cfg -> ..\/pxelinux.cfg\n \u251c\u2500\u2500 syslinux.efi\n \u2514\u2500\u2500 vesamenu.c32<\/pre>\n\n\n\n
dipankar@pits1:~$ sudo systemctl start dnsmasq\ndipankar@pits1:~$ sudo systemctl status dnsmasq\n\u25cf dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server\n Loaded: loaded (\/lib\/systemd\/system\/dnsmasq.service; enabled; vendor preset: enabled)\n Active: active (running) since Wed 2023-07-19 13:01:02 IST; 1min 34s ago\n Process: 3186 ExecStartPre=\/etc\/init.d\/dnsmasq checkconfig (code=exited, status=0\/SUCCESS)\n Process: 3194 ExecStart=\/etc\/init.d\/dnsmasq systemd-exec (code=exited, status=0\/SUCCESS)\n Process: 3203 ExecStartPost=\/etc\/init.d\/dnsmasq systemd-start-resolvconf (code=exited, status=>\n Main PID: 3202 (dnsmasq)\n Tasks: 1 (limit: 779)\n CPU: 168ms\n CGroup: \/system.slice\/dnsmasq.service\n \u2514\u25003202 \/usr\/sbin\/dnsmasq -x \/run\/dnsmasq\/dnsmasq.pid -u dnsmasq -r \/run\/dnsmasq\/resol>\n\nJul 19 13:01:01 pits1 systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...\nJul 19 13:01:02 pits1 dnsmasq[3202]: started, version 2.85 DNS disabled\nJul 19 13:01:02 pits1 dnsmasq[3202]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 >\nJul 19 13:01:02 pits1 dnsmasq-dhcp[3202]: DHCP, proxy on subnet 192.168.10.101\nJul 19 13:01:02 pits1 dnsmasq-dhcp[3202]: DHCP, sockets bound exclusively to interface eth0\nJul 19 13:01:02 pits1 dnsmasq-tftp[3202]: TFTP root is \/home\/dipankar\/data\/netboot\/\nJul 19 13:01:02 pits1 systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.<\/pre>\n\n\n\n
MENU TITLE PXE Boot Menu\nDEFAULT vesamenu.c32\n\n LABEL local\n MENU LABEL Boot from local drive\n LOCALBOOT 0xffff\n\n MENU BEGIN Fedora\n MENU TITLE Fedora\n\n MENU BEGIN 38\n MENU TITLE 38\n\n MENU BEGIN x86_64\n MENU TITLE x86_64\n\n LABEL installgui\n MENU LABEL ^Fedora Graphical install\n KERNEL ::fedora\/38\/x86_64\/vmlinuz\n APPEND vga=788 initrd=::fedora\/38\/x86_64\/initrd.img inst.repo=https:\/\/download.fedoraproject.org\/pub\/fedora\/linux\/releases\/38\/Server\/x86_64\/os\/ ip=dhcp --- quiet\n\n MENU END\n\n MENU END\n\n MENU END\n\n MENU BEGIN Debian\n MENU TITLE Debian\n\n MENU BEGIN 12\n MENU TITLE 12\n\n MENU BEGIN amd64\n MENU TITLE amd64\n\n LABEL install\n MENU LABEL ^Install\n KERNEL ::debian\/12\/amd64\/installer\/linux\n APPEND vga=788 initrd=::debian\/12\/amd64\/installer\/initrd.gz --- quiet\n\n MENU END\n\n MENU END\n\n MENU END<\/pre>\n\n\n\n
Fedora<\/strong>\n Copy initrd and kernel from images\/pxeboot in the Fedora netinst ISO\n image into the TFTP directory structure described above.\nDebian<\/strong>\n Copy initrd and kernel from debian netboot.tar.gz provided for each\n release. They are available under debian-installer directory.<\/pre>\n\n\n\n
dipankar@pits1:~\/data\/netboot$ sudo mount -o loop -t iso9660 ~\/images\/Fedora-Server-netinst-x86_64-38-1.6.iso ~\/data\/mnt\nmount: \/home\/dipankar\/data\/mnt: WARNING: source write-protected, mounted read-only.\ndipankar@pits1:~\/data\/mnt$ cp images\/pxeboot\/{initrd.img,vmlinuz} ..\/netboot\/fedora\/38\/x86_64\/<\/pre>\n\n\n\n![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/20230731_120949-1024x473.jpg\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/20230731_121020-1024x473.jpg\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/20230731_020205-1024x473.jpg\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/20230731_120817-1024x473.jpg\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/hill9.org\/journal\/wp-content\/uploads\/2023\/07\/20230731_002627-1024x473.jpg\")
<\/figure>\n\n\n\n\n