Ceph Kernel Module Missing.

[ajh34]

modprobe: FATAL: Module ceph not found in directory /lib/modules/4.14.98-v7+

Looks like the Ceph package does not install the required kernel module to mount cephfs filesystems. Same issue on stretch and previous release.

[JamesH65]

@ajh34 We do not enable CEPH in our kernel builds. It seems like a fairly niche system, and unless its impact on memory and CPU bandwidth is minimal (close to zero), we would be unlikely to incorporate it by default. If you can provide those figures, it would give us something to consider. It seems likely that as a module it impact would indeed be minimal, but we would need to be sure.

[imjustmatthew]

Unexpectedly ran into this same issue. I would never have expected a modern kernel didn't have Cpeh included. The performance impact of this as a module should be extremely small when it's not loaded and laoding of the module will only occur when a user actually uses it to mount a filesystem. There is a Ceph-FUSE option, but that is much lower performing that the kernel module.

[JamesH65]

@pelwell @popcornmix Any thoughts?

[pelwell]

As a filesystem I would expect no impact on performance if it isn't used, but there could be some overhead on the static (non-module) kernel size. The usual expectation on somebody asking for a feature to be added is for them to do a trial build and show the size of any additional modules, and the size of the kernel .img and free memory when it is used, along with the same information for a kernel that differs only in that the feature is not enabled.

[StrayLightning]

I built from fe915de, following the instructions here: https://www.raspberrypi.org/documentation/linux/kernel/building.md

The resultant image:
Linux cluster0 4.19.127-v7l-ceph+ #1 SMP Thu Jul 9 15:18:05 BST 2020 armv7l GNU/Linux

The kernel previously:
Linux cluster1 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux

(Different hostnames, but the nodes were updated at the same time.)

The resultant kernel images:
-rwxr-xr-x 1 root root 5801056 Jul 6 17:13 kernel7l.img
-rwxr-xr-x 1 root root 5777480 Jul 9 18:05 kernel-ceph.img

Of course, this is not an apples-to-apples comparison: If I get a chance I'll run a build from the same point that the released kernel was built from.

Changes to the kernel configuration:

21c21
< CONFIG_LOCALVERSION="-v7l"
---
> CONFIG_LOCALVERSION="-v7l-ceph"
1604c1604,1606
< # CONFIG_CEPH_LIB is not set
---
> CONFIG_CEPH_LIB=m
> # CONFIG_CEPH_LIB_PRETTYDEBUG is not set
> # CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set
6478c6480,6482
< # CONFIG_CEPH_FS is not set
---
> CONFIG_CEPH_FS=m
> # CONFIG_CEPH_FSCACHE is not set
> # CONFIG_CEPH_FS_POSIX_ACL is not set

[StrayLightning]

Using the modified config and building from the tag raspberrypi-kernel_1.20200601-1:

-rwxr-xr-x 1 root root 5776216 Jul 9 21:50 kernel-ceph.img

Linux cluster0 4.19.118-v7l-ceph+ #2 SMP Thu Jul 9 21:45:01 BST 2020 armv7l GNU/Linux

And this is the official build again for comparison:

-rwxr-xr-x 1 root root 5801056 Jul 6 17:13 kernel7l.img

For clarification, the configuration was modified by explicitly setting:

CONFIG_CEPH_LIB=m
CONFIG_CEPH_FS=m

and then choosing the default options for the other options that the build prompts for.

[StrayLightning]

Sorry, I forgot to include the sizes of the modules:

-rw-r--r-- 1 root root 299404 Jul 9 21:50 /lib/modules/4.19.118-v7l-ceph+/kernel/fs/ceph/ceph.ko
-rw-r--r-- 1 root root 302620 Jul 9 21:50 /lib/modules/4.19.118-v7l-ceph+/kernel/net/ceph/libceph.ko

[pelwell]

Please explain why we should tax all Pi users 4 * 600kB = 2.4MB for this feature.

[StrayLightning]

I'm not saying you should :-) I happened to be building the Ceph modules to try them out, and noticed this issue with the last comment being a request for the information.

I do notice, however, that the change in disk space for the kernel modules under /lib/modules is an increase of 0.14% between the supplied v7l directory and my build.

[StrayLightning]

Interestingly, I notice that there's just shy of 8MiB of kernel modules for OCFS2 (https://en.wikipedia.org/wiki/OCFS2) support in the current Raspberry Pi OS image. Would that be because of this:

https://blogs.oracle.com/developers/building-the-world%e2%80%99s-largest-raspberry-pi-cluster

[StrayLightning]

Add to that another 2MiB for GFS2 (https://en.wikipedia.org/wiki/GFS2) support and 2MiB plus change for DRBD (https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device) support.

At this point it is looking a bit strange that the Ceph modules are not included.

[cawolf]

I also ran into this issue trying to set up a local ceph cluster in k3s. As I do not know too much about kernel modules: is there a way that you compile the module, but do not load it by default until a user opts in to load it?