No interrupt for ttyS1?

[kpfleming]

Running Linux kernel 6.1.37 (plus ten64 patches for DPAA2), with a service reading/writing data continuously on /dev/ttyS1.

Looking in /proc/interrupts I see an IRQ assigned for ttyS0, but none assigned for ttyS1. Does that mean the ttyS1 UART is operated in polling mode?

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  9:          0          0          0          0          0          0          0          0     GICv3  25 Level     vgic
 11:   31031157   13300313   27776752   10660708   14428940   10451663    8915226   10484179     GICv3  30 Level     arch_timer
 12:          0          0          0          0          0          0          0          0     GICv3  27 Level     kvm guest vtimer
 14:          0          0          0          0          0          0          0          0     GICv3  68 Level     gpio-cascade, gpio-cascade
 15:     485035          0          0          0          0          0          0          0     GICv3  69 Level     gpio-cascade, gpio-cascade
 16:          0          0          0          0          0          0          0          0     GICv3  23 Level     arm-pmu
 17:   34425802          0          0          0          0          0          0          0     GICv3  64 Level     ttyS0
 18:          0          0          0          0          0          0          0          0     GICv3  45 Level     arm-smmu global fault
 19:          0          0          0          0          0          0          0          0     GICv3  46 Level     arm-smmu global fault
 20:          0          0          0          0          0          0          0          0     GICv3  47 Level     arm-smmu global fault
 21:          0          0          0          0          0          0          0          0     GICv3  48 Level     arm-smmu global fault
 22:          0          0          0          0          0          0          0          0     GICv3 243 Level     arm-smmu global fault
 23:          0          0          0          0          0          0          0          0     GICv3 244 Level     arm-smmu global fault
 24:          0          0          0          0          0          0          0          0     GICv3 245 Level     arm-smmu global fault
 25:          0          0          0          0          0          0          0          0     GICv3 246 Level     arm-smmu global fault
 26:          0          0          0          0          0          0          0          0     GICv3 247 Level     arm-smmu global fault
 27:          0          0          0          0          0          0          0          0     GICv3 248 Level     arm-smmu global fault
 28:          0          0          0          0          0          0          0          0     GICv3 249 Level     arm-smmu global fault
 29:          0          0          0          0          0          0          0          0     GICv3 250 Level     arm-smmu global fault
 30:          0          0          0          0          0          0          0          0     GICv3 178 Level     arm-smmu-context-fault
 31:          0          0          0          0          0          0          0          0     GICv3 179 Level     arm-smmu-context-fault
 32:          0          0          0          0          0          0          0          0     GICv3 180 Level     arm-smmu-context-fault
 33:          0          0          0          0          0          0          0          0     GICv3 181 Level     arm-smmu-context-fault
 94:          0          0          0          0          0          0          0          0  ITS-fMSI 230000 Edge      dprc.1
 95:        422          0          0          0          0          0     353677          0  ITS-fMSI 230001 Edge      dpio.7
 96:          0     653909          0          0          0          0          0          0  ITS-fMSI 230002 Edge      dpio.6
 97:          0          0     631164          0          0          0          0          0  ITS-fMSI 230003 Edge      dpio.5
 98:          0          0          0     554763          0          0          0          0  ITS-fMSI 230004 Edge      dpio.4
 99:          0          0          0          0     174133          0          0          0  ITS-fMSI 230005 Edge      dpio.3
100:     615408          0          0          0          0         42          0          0  ITS-fMSI 230006 Edge      dpio.2
101:          0          0          0          0          0          0    1686841          0  ITS-fMSI 230007 Edge      dpio.1
102:          0          0          0          0          0          0          0     373153  ITS-fMSI 230008 Edge      dpio.0
103:          0          0          0          0          0          0          0          0  ITS-fMSI 230009 Edge      dpni.9
104:          0          0          0          0          0          0          0          0  ITS-fMSI 230010 Edge      dpni.8
105:          0          0          0          0          0          0          0          0  ITS-fMSI 230011 Edge      dpni.7
106:          0          0          0          0          0          0          0          0  ITS-fMSI 230012 Edge      dpni.6
107:          0          0          0          0          0          0          0          0  ITS-fMSI 230013 Edge      dpni.5
108:          0          0          0          0          0          0          0          0  ITS-fMSI 230014 Edge      dpni.4
109:          0          0          0          0          0          0          0          0  ITS-fMSI 230015 Edge      dpni.3
110:          0          0          0          0          0          0          0          0  ITS-fMSI 230016 Edge      dpni.2
111:          0          0          0          0          0          0          0         15  ITS-fMSI 230017 Edge      dpni.1
112:          0          0          0          0          0          0          0          0  ITS-fMSI 230018 Edge      dpni.0
350:          0          0          0          0          0          0          0          0     GICv3 141 Level     PCIe PME, aerdrv
351:          0          0          0          0          0          0          0          0     GICv3 146 Level     PCIe PME, aerdrv
354:          0          0          0          0          0          0          0          0   ITS-MSI 135282689 Edge      pcie-dpc
356:          0          0          0          0          0          0          0          0   ITS-MSI 135299073 Edge      pcie-dpc
357:          0          0          0          0          0          0          0          0     GICv3 151 Level     PCIe PME, aerdrv
358:   55906574          0          0          0          0          0          0          0     GICv3  66 Level     2000000.i2c
359:          0          0          0          0          0          0          0          0  mpc8xxx-gpio  17 Edge      External Power Down
360:          0          0          0          0          0          0          0          0  mpc8xxx-gpio   8 Edge      ADMIN button
361:         52          0          0          0          0          0          0          0     GICv3  60 Level     mmc0
362:          0          0          0          0          0          0          0          0     GICv3 112 Level     xhci-hcd:usb1
363:    4971498          0          0          0          0          0          0          0     GICv3 113 Level     xhci-hcd:usb3
364:      11378          0          0          0          0          0          0          0     GICv3  67 Level     2020000.i2c, 2030000.i2c
365:          0          0          0          0          0        137         26          0   ITS-MSI 268959744 Edge      nvme0q0
366:     637934          0          0          0          0          0          0          0   ITS-MSI 268959745 Edge      nvme0q1
367:          0     645676          0          0          0          0          0          0   ITS-MSI 268959746 Edge      nvme0q2
368:          0          0     657979          0          0          0          0          0   ITS-MSI 268959747 Edge      nvme0q3
369:          0          0          0     675980          0          0          0          0   ITS-MSI 268959748 Edge      nvme0q4
370:          0          0          0          0     650014          0          0          0   ITS-MSI 268959749 Edge      nvme0q5
371:          0          0          0          0          0     662611          0          0   ITS-MSI 268959750 Edge      nvme0q6
372:          0          0          0          0          0          0     652423          0   ITS-MSI 268959751 Edge      nvme0q7
373:          0          0          0          0          0          0          0     664000   ITS-MSI 268959752 Edge      nvme0q8
374:     485035          0          0          0          0          0          0          0  mpc8xxx-gpio  28 Edge      PPS GPIO IRQ
376:       6806          0          0          0          0          0          0          0     GICv3 172 Level     8010000.jr
377:          5          0          0          0          0          0          0          0     GICv3 173 Level     8020000.jr
378:          0          0          0          0          0          0          0          0     GICv3 174 Level     8030000.jr
IPI0:    475794     322144     606658     332310     470891     320199     326372     323530       Rescheduling interrupts
IPI1:  14528576   22740993   38238206   13546204   18719847   13588524   10507625   14945448       Function call interrupts
IPI2:         0          0          0          0          0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0          0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0          0          0          0          0       Timer broadcast interrupts
IPI5:   1887582    2078279    2393177    1868795    1661697    2134155    1912661    2020001       IRQ work interrupts
IPI6:         0          0          0          0          0          0          0          0       CPU wake-up interrupts
Err:          0

[mcbridematt]

Good question!
I think the Linux device tree is set up incorrectly, it has both UARTS sharing the same interrupt:

torvalds/linux/blob/3f01e9fed8454dcd89727016c3e5b2fbb8f8e50c/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi#L359

The LS1088 reference manual shows a separate IRQ for duart1 (SPI 65 vs SPI64 for duart0 → GIC 32,33)

If you can generate your own DTB, try changing duart1 to:

interrupts = <0 33 IRQ_TYPE_LEVEL_HIGH>;

[kpfleming]

I’m not yet in a place to do that, and my systems seem to be stable without the IRQ separation (after I lowered the baud rate on ttyS1 to 38400 from 115200), so I’m happy to wait for a future firmware update. Thanks for the quick response!

[kpfleming]

Well, it turns out they are not as stable as I thought; in a few days, I’ll have one or two episodes of data being dropped, and then the application having to re-sync with the device. I’ll look into the custom DTB thing to see what I can do.

[kpfleming]

Oh joy… the firmware-builder repo has at least one submodule that requires Python 2.

[kpfleming]

Oops… I overlooked the code above, it’s in the kernel tree, right? That I can do much more easily, as I already build my own kernel packages.

[mcbridematt]

@mcbridematt

I think the Linux device tree is set up incorrectly, it has both UARTS sharing the same interrupt:

I’m wrong here, the device tree is right.
ttyS0 and ttyS1 share an interrupt because they are part of the DUART1 block.

(edit: if you try to change the ttyS1 IRQ, the kernel will oops because no one answers the IRQ, so it definitely has an interrupt)

@kpfleming

Well, it turns out they are not as stable as I thought; in a few days, I’ll have one or two episodes of data being dropped, and then the application having to re-sync with the device.

This is for GPS NMEA data, correct? I might be able to rig up my own experiment.
Is this with flow control (CTS/RTS) or just TX/RX ?

@kpfleming

Oops… I overlooked the code above, it’s in the kernel tree, right? That I can do much more easily, as I already build my own kernel packages.

Correct, all you need to do really is make ARCH=arm64 freescale/fsl-ls1088a-ten64.dtb
When the host is not arm64, you need to specify CROSS_COMPILE= to make or in an env var.

traversetech/ls1088firmware/firmware-builder/-/blob/master/build-dtb.sh

(Note: in that version, replace the codeaurora URL with https://github.com/nxp-qoriq/linux)

[kpfleming]

My attempt to change the IRQ failed; not sure if the patch didn’t get applied properly, or something else happened, but after a reboot the IRQ had not changed. I’ve reverted to the unpatched kernel now.

Yes, it’s GPS NMEA data, being fed into gpsd; no flow control, 38,400 baud. It was worse at 115,200 baud, so I’m nearly certain that data is occasionally being dropped.

[mcbridematt]

I did some more digging, this may interest you.
The Layerscape family DUART is capable of 64-byte FIFOs but the default device tree compatible only enables 16-byte

How to enable FIFO on ns16550A

Since kernel 5.19 you can use fsl,16550-FIFO64 to activate the 64 byte option.

serial: 8250: add compatible for fsl,16550-FIFO64

If you change the duart1 to this:

duart1: serial@21c0600 {
    compatible = "fsl,16550-FIFO64";

It will enumerate as this:

21c0600.serial: ttyS1 at MMIO 0x21c0600 (irq = 17, base_baud = 10937500) is a 16550A_FSL64

It passes a loopback test for me, will need to rig up a GPS receiver to do a NMEA test.

@kpfleming

no flow control,

Another option available is to configure the UARTs (via RCW) as 4 x 2-wire instead of 2 x 4-wire.
This would give you a RX and TX pair where the CTS and RTS pins are.
These would be on duart2 which has it’s own IRQ.

[kpfleming]

@mcbridematt

Another option available is to configure the UARTs (via RCW) as 4 x 2-wire instead of 2 x 4-wire.
This would give you a RX and TX pair where the CTS and RTS pins are.
These would be on duart2 which has it’s own IRQ.

Interesting!

I’ll give the 64-byte FIFO configuration a try today.

[kpfleming]

Well… I suppose this is a local issue, but even though I patched the DTB source and built a new kernel package (and I can see that the resulting DTB file is smaller than it was before, which makes sense as there were two ‘compatible’ entries previously), after a reboot that new DTB has not been used and the UARTs are still seen as regular 16550s.

I’m not really sure where the Debian kernel loads DTBs from, there are a lot of moving parts here (U-boot, systemd-boot, etc.). I’ll continue to investigate.

[mcbridematt]

@kpfleming

I’m not really sure where the Debian kernel loads DTBs from, there are a lot of moving parts here (U-boot, systemd-boot, etc.). I’ll continue to investigate.

Where are you putting the DTB? Somewhere under /boot?

The U-Boot in firmware v0.8.x is configured to only load the DTB from the devicetree partition on the flash.
The reason is to avoid issues that occur when a different DTB is substituted.
(There is the IOMMU fixup issue and also preventing older, incomplete versions of the DTB being loaded)

The new U-Boot in v0.9.x should support loading a DTB from the EFI/boot partition, but I need to check from where exactly.

So in summary:

• Use the recovery firmware to write a new DTB to flash (mtd erase devicetree && mtd write mydtb.dtb devicetree)
• Or change the U-Boot setup_distroboot_efi command to load the DTB from somewhere else

For your purposes, device tree overlay support will make this much easier, but there is just a bit more work to do before that can be enabled.

[kpfleming]

Thanks for the tips! I didn’t actually know how the DTB got loaded, so this is very helpful. I’ll write fsl-ls1088a-ten64.dtb to the devicetree partition.

[kpfleming]

That process worked, the system has booted with the revised DTB and the kernel messages show that the UARTs are in 64-byte FIFO mode. I’ll monitor the GPS receiver process over the next week or two to see if the situation has improved.

[kpfleming]

Unfortunately not good results; the GPS data on UART1 is now completely corrupted. I’ll revert back to the original DTB to confirm that there weren’t any other changes which caused it.

[mcbridematt]

Ah sorry, the U-Boot in v0.8 needs fsl,ns16650 in the compatible list, otherwise it will not set the base baud correctly.

So you will need:

duart1: serial@21c0600 {
compatible = "fsl,16550-FIFO64", "fsl,ns16550", "ns16550a";
}

I have a U-Blox GPS (LEA-M8S-0) feeding into ttyS1 now @ 115200, so I’ll keep an eye on it.

[kpfleming]

OK, now we’re cooking UARTs (both S0 and S1) are in 64-byte FIFO mode, and NMEA data is flowing on S1. Now we wait to see if it stays happy!

[kpfleming]

So far so good… I ran an 8-core kernel compile (keeps the box busy for hours) and there were no data drops during that time.

[kpfleming]

Well, no problems caused by this change (this is good).

Weirdly, the other box, which has not had the new DTB applied, has also not had any NMEA data disruption during the same time, so there may have been other changes (new kernel, etc.) which improved the situation.

However, ‘chronyc sourcestats’ shows that the GPS source is more stable (lower standard deviation of offset) on the one with the 64-byte FIFOs enabled, so I’ll apply that DTB to my second box too.

Is it worth sending a patch upstream to change the default in the kernel?

[kpfleming]

(continuing to talk to myself!)

The second box, without the updated DTB, had two NMEA data ‘blips’ overnight, and the one with the updated DTB did not. I’ll get the DTB applied to that box today and within a week or so I should be able to report whether the problem appears to have been resolved.