Cephalocon 2022

NVMe-over-Fabrics (NVMeoF) is an open, widely adopted, defacto standard in high performance remote block storage access. More and more storage vendors are introducing NVMeoF target support, with hardware offloads both for NVMeoF targets and initiators. Ceph does not support the NVMeoF protocol for block storage access; its clients use the Ceph RADOS protocol to access RBD images for good reason: RADOS is a distributed m-to-n protocol that provides reliable object access to sharded and replicated (or erasure coded) Ceph storage. However, there are good reasons to enable NVMeoF for Ceph: to enable its use in datacenters that are already utilizing storage hardware that offload NVMeoF capabilities, and to allow existing NVMeoF storage users to easily migrate to Ceph. In this talk we present our effort to integrate a native NVMeoF target for Ceph RBD. We discuss some of the challenges of implementing such a support for Ceph including subsystem/namespace discovery, multi-pathing for fault tolerance and performance, authentication and access control (e.g., namespace masking). Furthermore, we describe how the NVMeoF target design can be extended to allow reducing additional network hops by leveraging the Ceph CRUSH algorithm (ADNN).