Attach volumes locally on the storage node

By default every block consumer reaches a volume over its network protocol, NVMe-oF/TCP or iSCSI, including a pod scheduled on the storage node itself: that pod goes through a loopback connection to the kernel target on the same host. With localAttach enabled, a pod on the storage node uses the backend zvol or logical volume directly. Pods on every other node still use the network protocol, and a volume can move between the two paths each time it is published.

NFS is a file protocol and does not support localAttach, even when the pod runs on the storage node. NFS always uses its NFSv4.2 network mount so the dataset has one publication and recovery path. Provisioning an NFS volume with localAttach: true or the manual StorageClass parameter fails validation.

When a publish is local

ControllerPublishVolume picks the local path only when all of these hold:

• the volume was provisioned with localAttach enabled;
• the node being published to is the spec.nodeRef.name of the volume’s PillarAgent. An agent defined with spec.external never qualifies, because it runs outside the cluster;
• the access mode is a single-node mode: ReadWriteOnce, ReadWriteOncePod, or single-node read-only. Multi-node modes such as ReadOnlyMany always use the protocol.

NFS never qualifies for this path. Otherwise a block volume attaches over its network protocol exactly as it does without the flag. A local publish needs no NVMe host NQN or iSCSI initiator IQN on the CSINode and grants no initiator on the target.

Requirements

• The dm_mod kernel module on the storage node. The node plugin claims the backend device with a device-mapper target. The chart’s node init container loads dm_mod by default (node.initModprobe.modules); as with the other modules, load it on the host and list it in /etc/modules-load.d/ so a failed modprobe does not go unnoticed. See Prerequisites.
• dmsetup, which the node image ships.
• The node plugin must run on the storage node and see the target configfs there: /sys/kernel/config/nvmet for NVMe-oF/TCP, /sys/kernel/config/target for iSCSI. The chart’s node DaemonSet runs on every node unless you restrict node.nodeSelector, and its host /sys mount exposes both. If the node plugin cannot read the configfs, a local stage fails instead of guessing.

Enable it

On a PillarStorageClass, set spec.localAttach:

apiVersion: pillar-csi.bhyoo.com/v1alpha1
kind: PillarStorageClass
metadata:
  name: hot-local
spec:
  storeRef: storage-1-hot
  protocolRef: nvmeof-default
  localAttach: true
  storageClass:
    name: pillar-hot-local

The generated StorageClass does not carry the flag. The controller reads it from the live PillarStorageClass at CreateVolume, like the overrides.

On a StorageClass you write yourself, set the parameter pillar-csi.bhyoo.com/local-attach to "true" or "false". Any other value fails provisioning with InvalidArgument. Omitting it means false.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: manual-hot-local
provisioner: pillar-csi.bhyoo.com
parameters:
  pillar-csi.bhyoo.com/store-ref: storage-1-hot
  pillar-csi.bhyoo.com/protocol-ref: nvmeof-default
  pillar-csi.bhyoo.com/local-attach: "true"

There is no PVC annotation for it.

The setting is recorded per volume at provisioning, in spec.resolved.localAttach of the volume’s PillarVolumeState. Changing spec.localAttach later affects only volumes provisioned afterwards. Check an existing volume with:

kubectl get pvst <pv-name> -o jsonpath='{.spec.resolved.localAttach}'

What happens on a local attach

1. The controller records the storage node in the volume’s PillarVolumeState as status.localAttachNode, in the same update that reserves the publication.
2. The controller asks the agent to take the export away from remote initiators. For NVMe-oF/TCP the agent writes 0 to the namespace’s enable file in nvmet configfs and reads it back. The subsystem and the port stay configured; any remote session that is still connected can no longer do I/O. For iSCSI the agent disables the target’s portal group, which ends its sessions, and removes LUN 0 and its iblock backstore so the backend device is free. The target itself stays configured.
3. The agent returns the backend device path, and the controller hands it to the node plugin in the publish context.
4. NodeStageVolume on the storage node creates a device-mapper linear target named pillar-local-<16 hex> over the whole backend device. The name is pillar-local- followed by the first 16 hex characters of the SHA-256 of the volume ID. The target holds the backend device open exclusively.
5. With the claim in place, the node plugin reads the target state of the volume: for NVMe-oF/TCP the nvmet namespaces of its subsystem, for iSCSI whether LUN 0 of its LIO target still exists. If the export can still serve I/O, it backs off and fails the stage with FailedPrecondition.
6. The node plugin formats and mounts the filesystem on /dev/mapper/pillar-local-<16 hex>. A volumeMode: Block volume is bound from the same device.

If a stage fails at any step after the claim, the node plugin unmounts whatever it staged and then removes the target. If the unmount fails, it keeps the target and reports the unmount failure along with the original error, so no mount is left pointing at a removed device.

NodeUnstageVolume unmounts and then removes the device-mapper target. If a stage record is missing, for example after a failed stage, unstage still removes a pillar-local-* target left for that volume.

Moving a pod between the storage node and other nodes

Pods move freely; each publish picks the path again.

From another node to the storage node. Kubernetes unpublishes the old node first, which revokes its initiator on the target. The local publish then disables the export, so even a remote host that kept its NVMe-oF or iSCSI session, for example after a force-detach while its kubelet was down, cannot write to the volume.

From the storage node to another node. Kubernetes unstages the volume on the storage node, which removes the pillar-local-* target, and then unpublishes it. Unpublishing a local attach does not re-enable the export and leaves status.localAttachNode set. Every publish of the volume over the network re-enables it before granting the new node’s initiator:

1. The agent opens the backend device with O_EXCL. If something on the storage node still holds it, the agent refuses, the export stays disabled, and the publish fails with FailedPrecondition.
2. Otherwise the agent keeps the device open while it re-enables the export and reads it back, then closes it. For NVMe-oF/TCP it writes 1 to the namespace’s enable file. For iSCSI it recreates the iblock backstore and LUN 0 and enables the portal group again. While the agent holds the device, the node plugin cannot claim it, so the export and a local attach are never active together.
3. If status.localAttachNode was set, the controller clears it and asks the agent to re-enable the export once more under the fencing generation that the clear committed. Then it grants the new node’s initiator.

The re-enable call is a no-op when the export is already enabled. The controller makes it on every network publish of a localAttach volume, including when status.localAttachNode is already empty.

The external-attacher retries a failed publish, so the pod on the new node stays in ContainerCreating until the storage node really lets go of the device. This is what protects the volume during a force-detach: if the storage node’s kubelet is dead but the container still runs and writes through the local mount, the target stays in place and the export stays disabled. The publish proceeds only after the claim is gone, for example after the kubelet comes back and unstages the volume, or after the storage node reboots.

Do not remove a pillar-local-* target by hand to unblock a publish. While it exists, something on the storage node may still be writing to the volume.

Expansion

Expansion works the same way as for a protocol attach; see Expand a volume. While the NVMe namespace is disabled, the agent grows the zvol or logical volume but skips the NVMe revalidate_size step, which the kernel rejects on a disabled namespace. An iSCSI volume has no LUN while it is attached locally, so there is nothing on the target to resize. On the storage node, NodeExpandVolume reloads the device-mapper table to the new size of the backend device and then grows the filesystem. For a block volume it only reloads the table.

Observe it

Which node holds the volume locally:

kubectl get pvst <pv-name> -o jsonpath='{.status.localAttachNode}'
kubectl get pvst <pv-name> -o jsonpath='{.status.publishedNodes}'

A publication with local: true in status.publishedNodes is a local attach. While status.localAttachNode is set, the export serves no I/O to remote initiators, and export resync keeps it disabled (the NVMe namespace disabled, or the iSCSI target without LUN 0) after an agent restart or a storage-node reboot.

The device-mapper target on the storage node:

sudo dmsetup ls | grep pillar-local-
sudo dmsetup table pillar-local-<16 hex>

A publish to another node that is blocked by the local claim shows up on the VolumeAttachment and in the pod’s events:

kubectl get volumeattachment
kubectl describe volumeattachment <name>

The attach error contains FailedPrecondition and backend device ... is still held on the storage node (local attach in use). A local stage refused because the export was still enabled shows network export of volume ... is still serving remote initiators in the pod’s events and the node plugin log.

See Fencing and consistency for why each side checks the other.