Update socket CAN to include CAN-FD

[kayoub5]

Context

The libpcap library used the socket CAN format for both CAN and CAN-FD, however the documentation did not reflect the new flags that were added when the CAN-FD format was used.

[mcr]

Thank you for updating the documentation.
You intro suggested that there were other things that needed to be done in libpcap.
Is there a matching PR for libpcap?

[guyharris]

The tcpdump documentation does not mention how the application can differentiate ...

Presumably you mean "the libpcap documentation", as tcpdump is an application.

[kayoub5]

The tcpdump documentation does not mention how the application can differentiate ...

Presumably you mean "the libpcap documentation", as tcpdump is an application.

I mean the tcpdump.org documentation for LINKTYPE: https://www.tcpdump.org/linktypes/LINKTYPE_CAN_SOCKETCAN.html

Libpcap documentation (Libpcap man pages) https://www.tcpdump.org/manpages/pcap.3pcap.html do not specify the format for any LINKTYPE

[kayoub5]

Thank you for updating the documentation.
You intro suggested that there were other things that needed to be done in libpcap.
Is there a matching PR for libpcap?

Once the concept is approved here, I will create a matching PR in Libpcap, If you think the concept should be discussed in https://github.com/the-tcpdump-group/libpcap instead then I will open an issue there.

[guyharris]

Note that libpcap itself no longer produces DLT_CAN_SOCKETCAN packets; support for them was introduced in

commit f3edbb599b8cbcc7e4560000dcba8e992dc11a31
Author: Akos Vandra [email protected]
Date: Fri Dec 23 03:29:52 2011 +0200

Added initial support for canusb devices.

CanUSB devices are not directly supported by the kernel.
LibUSB-1.0 is used to communicate them, with a special protocol.

Reading 2 bytes from interrupt endpoint 1 returns the number of packets in $
rx and the tx buffer,

Reading 20 bytes from bulk endpoint 5 returns a 4byte timestamp in 10us
resolution since powerup, and a can fram in socket_can frame format, so
DLT_SOCKETCAN may be used.

TODO: use async libusb communication

and was removed in

commit 93ca5ff7030aaf1219e1de05ec89a68384bfc50b
Author: Guy Harris [email protected]
Date: Thu Aug 25 13:26:43 2016 -0700

On Linux, handle all CAN captures with pcap-linux.c, in cooked mode.

There's no need to support capturing on PF_CAN sockets; CAN interfaces
show up as regular interfaces on which you can capture with PF_PACKET
sockets, so just let pcap-linux.c handle them, and get rid of
pcap-can-linux.c.

Capture on them in cooked mode, so we get the protocol field and can
distinguish between "classic" CAN and CAN FD.

The hardware for which pcap-canusb-linux.c was intended never reached
production:

    https://github.com/axos88/libpcap/commit/f3edbb599b8cbcc7e4560000dcba8e992dc11a31#commitcomment-18716617

so we don't need pcap-canusb-linux.c, either.

This all removes the need for the "host-endian" link-layer header type;
it never made it into a libpcap release, so we just remove it.  The
"big-endian" link-layer header type is kept for the benefit of packets
captured with the old pcap-can-linux.c code; we revert it to its old
name.

See this comment on that commit and the two following comments.

Is there any other packet source that produces LINKTYPE_CAN_SOCKETCAN packets?

[kayoub5]

Is there any other packet source that produces LINKTYPE_CAN_SOCKETCAN packets?

Yes, the following tools produce PCAPNG files with LINKTYPE_CAN_SOCKETCAN

Open Source Converters that convert files from automotive specific formats to PCAPNG

• https://github.com/Technica-Engineering/Technica.Traces.Tecmp.Converter
• https://github.com/Technica-Engineering/Technica.Traces.Blf.Converter
• https://github.com/Technica-Engineering/Technica.Traces.Asc.Converter

Commercial product that produce PCAPNG files with LINKTYPE_CAN_SOCKETCAN from Vector Informatik devices

• https://technica-engineering.de/en/produkt/andi-automotive-network-diagnoser-ethernet-rest-bus-simulation/

[guyharris]

So in what fashion is this format at all related to SocketCAN?

Is it a format that could be used to record CAN packets in a system that doesn't use the SocketCAN API (and might not even use Linux)?

If the answer is "yes", perhaps it should just be named LINKTYPE_CAN (I need to see if Gianluca Varenni still knows what LINKTYPE_CAN20B/DLT_CAN20B looks like).

[kayoub5]

So in what fashion is this format at all related to SocketCAN?

The format have the same layout as Linux Socket CAN, with only difference being the endianness of the ID

Is it a format that could be used to record CAN packets in a system that doesn't use the SocketCAN API (and might not even use Linux)?

Yes, LINKTYPE_CAN_SOCKETCAN is the only publicly documented LINKTYPE that offer native support for CAN, not all platforms offer support for Linux cooked packets, at least not Windows.

If the answer is "yes", perhaps it should just be named LINKTYPE_CAN (I need to see if Gianluca Varenni still knows what LINKTYPE_CAN20B/DLT_CAN20B looks like).

Naming it LINKTYPE_CAN sounds fine, LINKTYPE_CAN_SOCKETCAN as is support both CAN 2.0A and CAN 2.0B, and with updated docs in this PR (derived from Linux SocketCAN documentation) it would also support CAN-FD.
So having a LINKTYPE just for CAN 2.0B sounds redundant.

[kayoub5]

@guyharris I updated this MR to focus only on documenting the FD flags, could you review it?

[guyharris]

So has there ever been code that 1) put CAN FD packets into a pcap or pcapng file with LINKTYPE_CAN_SOCKETCAN and 2) did not set the CANFD_FDF flag?

[kayoub5]

So has there ever been code that 1) put CAN FD packets into a pcap or pcapng file with LINKTYPE_CAN_SOCKETCAN

Yes, I wrote such code.

and 2) did not set the CANFD_FDF flag?

Yes, since CANFD_FDF was introduced just last month.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=02546884221279da2725e87e35348290470363d7

So new writers should set it, and readers should check for both MTU and FDF flag.

[guyharris]

So when it speaks of the two values of the "frame size", when determining whether a frame without CANFD_FDF set is a CAN frame or a CAN-FD frame, is that:

• the value of the payload length field?
• the packet length from the pcap/pcapng block, minus 8 for the header?
• something else?

[kayoub5]

So when it speaks of the two values of the "frame size", when determining whether a frame without CANFD_FDF set is a CAN frame or a CAN-FD frame, is that:

• the value of the payload length field?

• the packet length from the pcap/pcapng block, minus 8 for the header?

• something else?

the full packet length from the pcap/pcapng block (including header, payload and unused padding bytes)

[guyharris]

So as I read Table 5 "Coding of the numbers of data bytes by the DLC" in ISO 11898-1:2015, "classic" frames can have up to 8 data bytes and FD frames can have up to 64 data bytes; an 8-byte SocketCAN header plus 8 data bytes is 16, and an 8-byte SocketCAN header plus 64 data bytes is 72, but does the Linux SocketCAN code always pad classic frames out to 16 bytes and FD frames out to 72 bytes, so the length and caplen are 16 for classic frames and 72 for FD frames (even if the FD frame has 8 or fewer data bytes)? (I don't have any of the other ISO 11898 standards specifying the layers below ISO 11898-1, so I don't know whether frames are padded on the wire.)

[kayoub5]

the Linux SocketCAN code always pad classic frames out to 16 bytes and FD frames out to 72 bytes, so the length and caplen are 16 for classic frames and 72 for FD frames (even if the FD frame has 8 or fewer data bytes)

Above statement is correct

I don't have any of the other ISO 11898 standards specifying the layers below ISO 11898-1, so I don't know whether frames are padded on the wire.

There is no padding on the wire

[guyharris]

While testing some changes to Wireshark to use the CANFD_FDF flag, I found several captures attached to Wireshark bugs that 1) had LINKTYPE_CAN_SOCKETCAN as the link-layer type and 2) had random garbage in the "reserved" fields of some packets - including the reserved-until-CAN-FD-support "FD flags" field. In some of those packets, the 0x04 bit was set in the "FD flags" field.

So I don't trust that field completely, at least for older packets. I don't know whether those captures came from libpcap on Linux or, if they did, whether it was from the old PF_CAN socket code or from the PF_PACKET socket code.

In kernels at least as far back as 3.0.4, there is a routine alloc_can_skb() that appears to allocate a new SKB, initialize it for classic CAN packets. and zero out the CAN frame header, and, in 3.6-rc4, which introduced CAN FD support, alloc_canfd_skb() was added to do the same for CAN FD packets.

In 2.6.31, the driver in drivers/net/can/sja1000/sja1000.c appears to zero out the frame header itself. If other drivers didn't do so, protocol analyzers - or capture file readers - will need to apply heuristics to determine whether to trust the "FD flags" field.

Given that all kernels with CAN FD support have alloc_canfd_skb(), a heuristic that assumes that if the CANFD_FDF flag is set, the only bits that should be set in the "FD flags" byte are 0x04, 0x02, and 0x01 (subject to change if Linux introduces new "FD flags" bits) and the two reserved bytes after the "FD flags" byte should be zero (subject to change if Linux starts using those fields in FD frames) and, if that's not the case for such a frame, that frame will be treated as a classic CAN frame.

[kayoub5]

@guyharris I understand the issue, what would you recommend future writers of LINKTYPE_CAN_SOCKETCAN do?

[guyharris]

what would you recommend future writers of LINKTYPE_CAN_SOCKETCAN do?

1. Make the "FD flags" field 0 for classic CAN frames.
2. Set the 0x04 bit of the "FD flags" field in CAN FD frames, and don't set any bits in that field other than the 0x02 and 0x01 bits unless and until Linux assigns them a meaning.
3. Make the first "Reserved/Padding" field 0 unless and until Linux assigns it a meaning.
4. Make the second "Reserved/Padding" field 0 unless and until Linux assigns it a meaning.

[guyharris]

So the first "Reserved/Padding" appears to have been assigned a meaning for classic CAN frames, len8_dlc.

I infer from the can/length.h header and section 8.4.2.4 "DLC field" of ISO 11898-1 that, if the len field is 8, len8_dlc contains the raw value of the DLC field, which can either be 8 or can be any value from 0x10 to 0x1F for a classic CAN frame with an 8-byte payload.

Is that the case? And is that only for sending classic CAN frames, so that the sender can specify what value to put into the DLC field, or is it also supplied when a frame is received? I see can_frame_set_cc_len(), which sets the len8_dlc field, called from mcp251xfd_hw_rx_obj_to_skb() in the mcp251xfd-core.c driver, and that appears to be a routine in the receive code path, so my guess is that it can be supplied when a frame is received.

That field isn't needed for CAN FD and isn't part of the , so I presume that for a CAN FD field it will be zero; it's a reserved field in the canfd_frame structure. This means that we don't need to do anything special to the checks to make sure we don't misidentify a field with uninitialized "FD flags", as a non-zero len8_dlc value will tell us not to trust the "FD flags" field.

[kayoub5]

@guyharris I have not used len8_dlc myself.
What I was able to find about its reason for existence in torvalds/linux@ea78005 maybe its author @hartkopp could explain it better.

[hartkopp]

So new writers should set it, and readers should check for both MTU and FDF flag.

Not really. The right (and safe) way is to check for the data structure length (aka MTU).
The FDF flag is intended to be used in CAN (userspace) applications where the developer uses the struct canfd_frame space for both types of CAN frames.

[hartkopp]

@guyharris I have not used len8_dlc myself. What I was able to find about its reason for existence in torvalds/linux@ea78005 maybe its author @hartkopp could explain it better.

Yes. The reason for can8_dlc is that CAN controllers send that 'raw' DLC value on the wire. The API extension is used for CAN security testing to check whether a receiving node grabs this 'raw' DLC value and uses values like 0xF in the former processing to trigger overflows or write behind buffer sizes. With a 'raw' DLC value of 0xF the Classical CAN controller still transfers 8 bytes of payload. The receivers should limit the length value to 8 bytes when reading the DLC value from the CAN controllers registers.

[guyharris]

The FDF flag is intended to be used in CAN (userspace) applications where the developer uses the struct canfd_frame space for both types of CAN frames.

The code in the main branch of libpcap does exactly that. If a packet in the PF_PACKET ring buffer has a hatype of ARPHRD_CAN, libpcap checks whether the protocol is LINUX_SLL_P_CANFD. If so, it sets the FDF flag, otherwise it clears the FDF flag in the header supplied in the packet (in case the kernel on which it's running or the driver doesn't bother initializing the entire header, and leaves random uninitialized junk there, which earlier kernels and drivers appeared to do).

[hartkopp]

The FDF flag is intended to be used in CAN (userspace) applications where the developer uses the struct canfd_frame space for both types of CAN frames.

The code in the main branch of libpcap does exactly that. If a packet in the PF_PACKET ring buffer has a hatype of ARPHRD_CAN, libpcap checks whether the protocol is LINUX_SLL_P_CANFD. If so, it sets the FDF flag, otherwise it clears the FDF flag in the header supplied in the packet (in case the kernel on which it's running or the driver doesn't bother initializing the entire header, and leaves random uninitialized junk there, which earlier kernels and drivers appeared to do).

Is this the right approach? From the kernel perspective the FDF flag is unreliable. Should therefore libpcap set this bit on its own or should it show the 'reality' which has been captured?

The reliable information whether we have a Classical CAN frame or a CAN FD frame is (still) only represented in the MTU, and LINUX_SLL_P_CANFD. So why fiddle with that FDF flag?

[kayoub5]

Is this the right approach? From the kernel perspective the FDF flag is unreliable. Should therefore libpcap set this bit on its own or should it show the 'reality' which has been captured?

Proposed as the-tcpdump-group/libpcap#1035

The reliable information whether we have a Classical CAN frame or a CAN FD frame is (still) only represented in the MTU, and LINUX_SLL_P_CANFD. So why fiddle with that FDF flag?

Because writing 64bytes of CAN-FD data when not all the bytes are used is a big waste of disk space.

Example:

Usual traffic in a CAN bus is 2 frames per millisecond.
Modern cars have more than 10 CAN busses
Below numbers for 1 hour

((72B*2)/1ms) * 1hour * 10 interfaces = 5GB
When stripping unused padding bytes
((16B*2)/1ms) * 1hour * 10 interfaces = 1GB

[guyharris]

From the kernel perspective the FDF flag is unreliable. Should therefore libpcap set this bit on its own or should it show the 'reality' which has been captured?

The "reality that was captured" is that a frame is either classic CAN or CAN FD.

One way to indicate the difference is to use a LINKTYPE_LINUX_SLL header and let the protocol field indicate which frame type it is, based on whether it's ETH_P_CAN or ETH_P_CANFD.

However, a user requested in the-tcpdump-group/libpcap#1052 that libpcap not add a LINKTYPE_LINUX_SLL header, so as to save space in the capture file.

Using the FDF flag is a way to provide the classic CAN vs. CAN FD information without having to add extra data to the frame.

The reliable information whether we have a Classical CAN frame or a CAN FD frame is (still) only represented in the MTU, and LINUX_SLL_P_CANFD. So why fiddle with that FDF flag?

To, as @kayoub5 noted, save space. Think of it as compressing the LINKTYPE_LINUX_SLL protocol field into oe bit that was already reserved by the Linux kernel, presumably for this very purpose. If the Linux kernel doesn't do anything with the FDF flag, and nobody's supposed to use that flag to determine whether a frame is classic CAN or CAN FD, why was that flag introduced in the first place? (He also notes that this also permits trimming padding from packets, saving more space.)

Note also that not all LINKTYPE_CAN_SOCKETCAN captures are generated from packets supplied by the Linux kernel, as @kayoub5 noted in an earlier comment. Perhaps, at least for Wireshark, the best answer is to have Wireshark capable of reading those files - and perhaps libpcap can also be enhanced to support reading those files, for the benefit of other programs - but LINKTYPE_CAN_SOCKETCAN is still not tied to Linux, and somebody might have equipment that doesn't use Linux's SocketCAN code but that produces CAN packet traces.

[hartkopp]

The reliable information whether we have a Classical CAN frame or a CAN FD frame is (still) only represented in the MTU, and LINUX_SLL_P_CANFD. So why fiddle with that FDF flag?

To, as @kayoub5 noted, save space. Think of it as compressing the LINKTYPE_LINUX_SLL protocol field into oe bit that was already reserved by the Linux kernel, presumably for this very purpose. If the Linux kernel doesn't do anything with the FDF flag, and nobody's supposed to use that flag to determine whether a frame is classic CAN or CAN FD, why was that flag introduced in the first place? (He also notes that this also permits trimming padding from packets, saving more space.)

Ok, got it (hopefully). It is not mainly about storing 1:1 content provided by the Linux kernel but about storing CAN traffic (in an efficient way).

Then the FDF flag is perfectly fine for this 'use-case in userspace'.

The https://github.com/linux-can/can-utils ASCII HEX logfile format does not store 'empty bytes' of CAN (FD) frames either:
https://github.com/linux-can/can-utils/blob/master/lib.h#L121

Note also that not all LINKTYPE_CAN_SOCKETCAN captures are generated from packets supplied by the Linux kernel, as @kayoub5 noted in an earlier comment. Perhaps, at least for Wireshark, the best answer is to have Wireshark capable of reading those files - and perhaps libpcap can also be enhanced to support reading those files, for the benefit of other programs - but LINKTYPE_CAN_SOCKETCAN is still not tied to Linux, and somebody might have equipment that doesn't use Linux's SocketCAN code but that produces CAN packet traces.

Right.

[guyharris]

Ok, got it (hopefully). It is not mainly about storing 1:1 content provided by the Linux kernel but about storing CAN traffic (in an efficient way).

Exactly. It can store CAN traffic from the Linux kernel, or from other arbitrary equipment that generates or captures CAN traffic, or converted from CAN traffic captures in other formats.