scsi_logging_level

@amitmund September 11, 2026

Linux SCSI Logging & Diagnostics (`scsi_logging_level`) — Complete Learning Notes & Output Guide

In the Linux storage subsystem, traditional SAS/SATA hard drives, solid-state drives (/dev/sd*), USB mass storage, and Fibre Channel/iSCSI SAN LUNs are mediated through the SCSI (Small Computer System Interface) Mid-Layer. Linux provides dynamic kernel logging via scsi_logging_level and /proc/sys/dev/scsi/logging_level to trace SCSI commands, device resets, errors, timeouts, and hardware sense codes directly into the kernel ring buffer (dmesg).

1. What is SCSI Logging in Linux?

The Linux SCSI subsystem is architected in three distinct layers:

+-------------------------------------------------------------+
|                      SCSI Upper Layer                       |
|   Device Drivers: sd (disk), sr (CD/DVD), st (tape), sg     |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                      SCSI Mid-Layer                         |
|   - Routes commands, manages queues & error recovery (EH)   |
|   =======================================================   |
|   ===> [ scsi_logging_level logs events at this layer ] =   |
|   =======================================================   |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                      SCSI Lower Layer                       |
|   Host Bus Adapters (HBA) / Hardware Transport Drivers:     |
|   mpt3sas, megaraid_sas, lpfc (Fibre Channel), iscsi_tcp    |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                      HARDWARE LAYER                         |
|   SATA/SAS Disks, Fibre Channel SAN, iSCSI Arrays, RAID LUNs|
+-------------------------------------------------------------+

When a drive experiences bad sectors, path failovers, aborted commands, or cable disconnects, the SCSI mid-layer logs status codes. Dynamic SCSI logging answers critical hardware and driver diagnostic questions:

Why did an I/O operation fail with an I/O error (EIO) or hang indefinitely?
What exact SCSI Command Descriptor Block (CDB) was issued when the disk failed?
Did the physical disk return a CHECK CONDITION, and what do its Sense Key, ASC, and ASCQ mean?
Is a storage controller resetting the bus, or is a drive aborting commands due to hardware firmware timeouts?
Is a multi-path SAN failover event triggered by a link loss or target port error?

2. Enabling & Controlling SCSI Logging

SCSI logging is controlled dynamically via sysctl without restarting the server.

Method 1: Using the `scsi_logging_level` CLI Tool

Install the sg3_utils (or scsi-logging-level) package:

# Debian / Ubuntu
sudo apt install sg3-utils

# RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install sg3_utils

Check current logging levels:

scsi_logging_level --get

Enable logging for specific events (levels range from 0 [off] to 7 [verbose]):

# Enable logging for SCSI errors and timeouts (Level 3)
sudo scsi_logging_level --set --error=3 --timeout=3

# Enable all SCSI logging categories to level 2 (moderate tracing)
sudo scsi_logging_level --set --all=2

# Turn off all SCSI logging (CRITICAL when finished)
sudo scsi_logging_level --set --all=0

Method 2: Direct Kernel `/proc` Interface (Zero Dependencies)

If sg3_utils is not installed, you can write a raw 32-bit hex bitmask directly to /proc:

# Check current raw bitmask (0x0 means all SCSI logging disabled)
cat /proc/sys/dev/scsi/logging_level

# Enable Error (bits 0-2) and Timeout (bits 3-5) logging:
# Error=3 (0b011), Timeout=3 (0b011 << 3 = 0b011000 = 0x18) -> 0x1b
sudo sysctl dev.scsi.logging_level=0x1b
# or
echo 0x1b | sudo tee /proc/sys/dev/scsi/logging_level

# Reset / Disable all logging
sudo sysctl dev.scsi.logging_level=0

3. The SCSI Logging Categories (Bitmask Allocation)

The 32-bit integer in /proc/sys/dev/scsi/logging_level reserves 3 bits per category (values 0 to 7):

Bit Offset	Category Name	What It Traces
`0 - 2`	`error`	Command failures, check conditions, status completions.
`3 - 5`	`timeout`	Commands that exceeded timeout timers and aborted.
`6 - 8`	`scan`	Device discovery, LUN scanning, bus enumeration.
`9 - 11`	`mlqueue`	SCSI Mid-Layer queue management, command dispatching.
`12 - 14`	`mlcomplete`	Mid-Layer command completion handling.
`15 - 17`	`llqueue`	Lower-Layer (HBA driver) queue dispatch.
`18 - 20`	`llcomplete`	Lower-Layer command completions and interrupts.
`21 - 23`	`hlqueue`	High-Layer (`sd`) submission queue.
`24 - 26`	`hlcomplete`	High-Layer (`sd`) completion processing.
`27 - 29`	`ioctl`	User-space SCSI passthrough ioctl commands (`SG_IO`).

4. Anatomy of a SCSI Log Line in `dmesg`

When a storage event or error occurs, the kernel outputs multi-line entries into dmesg (or /var/log/messages / /var/log/syslog).

Raw Output Example: Medium Read Error (Bad Sector)

[ 1420.102345] sd 2:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_stat=0x02
[ 1420.102350] sd 2:0:0:0: [sda] tag#12 CDB: Read(10) 28 00 00 45 a1 00 00 00 08 00
[ 1420.102355] sd 2:0:0:0: [sda] tag#12 Sense Key : Medium Error [current] 
[ 1420.102360] sd 2:0:0:0: [sda] tag#12 Add. Sense: Unrecovered read error
[ 1420.102365] sd 2:0:0:0: [sda] tag#12 Sense Key: 0x3, ASC: 0x11, ASCQ: 0x0
[ 1420.102370] critical target error, dev sda, sector 4563200 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

5. Breakdown of Every Output Heading & Field

+----------------+-------+--------+----------------------------------------------------------+
| Device Handle  | Disk  | Tag    | Message Payload                                          |
+----------------+-------+--------+----------------------------------------------------------+
| sd 2:0:0:0:    | [sda] | tag#12 | FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK ...  |
+----------------+-------+--------+----------------------------------------------------------+

5.1 SCSI Device Address (`sd 2:0:0:0:`)

Format: H:C:T:L (Host:Channel:Target:LUN)
2 (Host): The HBA or storage controller bus number.
0 (Channel): The bus channel on that controller.
0 (Target): The specific physical drive or storage enclosure target ID.
0 (LUN): Logical Unit Number within that target.

5.2 Device Name (`[sda]`)

The traditional Linux block device naming handle associated with that SCSI target.

5.3 Queue Tag (`tag#12`)

The hardware queue command tag allocated in the controller's multiqueue (blk-mq) structure.

5.4 The 32-bit SCSI Result Word (`Result: hostbyte=... driverbyte=... cmd_stat=...`)

The Linux kernel splits the SCSI result status word into four distinct bytes:

Bits 31-24: Driver Byte | Bits 23-16: Host Byte | Bits 15-8: Msg Byte | Bits 7-0: Status Byte

Field Name	Example Value	Meaning
`hostbyte`	`DID_OK`	Status reported by the Host Bus Adapter (HBA) driver. `DID_OK` means the local HBA experienced no hardware or transport layer fault.
	`DID_NO_CONNECT`	HBA cannot reach target; cable unplugged or SAN zoning missing.
	`DID_BUS_BUSY`	Hardware bus was saturated; command could not be dispatched.
	`DID_TIME_OUT`	Hardware command timer expired before drive acknowledged completion.
	`DID_ABORT`	The command was explicitly aborted by the host.
`driverbyte`	`DRIVER_OK`	Internal Linux SCSI driver layer status.
	`DRIVER_SENSE`	Driver received and parsed valid SCSI Sense Data from the drive.
	`DRIVER_TIMEOUT`	Driver dropped the command after hitting execution threshold.
`cmd_stat`	`0x02` (CHECK CONDITION)	SCSI Status Byte returned by the storage drive firmware.

Common SCSI Status Byte Codes (`cmd_stat`)

0x00 (GOOD): Command completed successfully.
0x02 (CHECK CONDITION): Target encountered an error or state change; request Sense Data for details.
0x08 (BUSY): Target drive is busy and unable to accept new commands.
0x18 (RESERVATION CONFLICT): Another host holds a persistent SCSI reservation on this LUN (common in clustering/fencing setups).
0x28 (TASK SET FULL): Drive internal queue is saturated; cannot buffer more commands.

5.5 The SCSI Command Descriptor Block (`CDB`)

sd 2:0:0:0: [sda] tag#12 CDB: Read(10) 28 00 00 45 a1 00 00 00 08 00

Read(10): The human-readable SCSI opcode name (10-byte READ command).
28 00 00 45 a1 00 00 00 08 00: The exact raw hex bytes sent over the wire to the drive controller:
Byte 0 (28): Opcode for READ (10).
Bytes 2–5 (00 45 a1 00): Starting Logical Block Address (LBA). In hex: $0x45A100 = 4,563,200$.
Bytes 7–8 (00 08): Transfer Length (8 sectors = 4096 bytes / 4 KB).

5.6 SCSI Sense Data (`Sense Key`, `ASC`, `ASCQ`)

When a drive returns CHECK CONDITION (0x02), it provides structured diagnostic codes conforming to the ANSI SCSI primary commands (SPC) standard:

sd 2:0:0:0: [sda] tag#12 Sense Key : Medium Error [current] 
sd 2:0:0:0: [sda] tag#12 Add. Sense: Unrecovered read error
sd 2:0:0:0: [sda] tag#12 Sense Key: 0x3, ASC: 0x11, ASCQ: 0x0

1. `Sense Key` (Primary Category)

0x0 NO SENSE: No specific error.
0x1 RECOVERED ERROR: Device completed command with error recovery (e.g., sector ECC re-read).
0x2 NOT READY: Drive cannot be accessed (spinning up, offline, door open).
0x3 MEDIUM ERROR: Physical storage flaw (bad magnetic sector, corrupted NAND flash block).
0x4 HARDWARE ERROR: Internal controller or device component failure.
0x5 ILLEGAL REQUEST: Invalid parameter in CDB or unsupported command opcode.
0x6 UNIT ATTENTION: State changed (bus reset, media change, power-on cycle).
0x7 DATA PROTECT: Media write-protected or read-only mode engaged.
0xB ABORTED COMMAND: Drive aborted execution due to timeout or bus parity fault.

2. `ASC` (Additional Sense Code) & `ASCQ` (Additional Sense Code Qualifier)

The combination of ASC and ASCQ defines the exact hardware fault:

Sense Key	ASC	ASCQ	Description	Hardware Diagnostic Significance
`0x3`	`0x11`	`0x00`	Unrecovered read error	Bad physical block. Disk could not read data despite ECC retries.
`0x2`	`0x04`	`0x01`	Logical unit is in process of becoming ready	Disk is still spinning up or initializing firmware.
`0x2`	`0x04`	`0x02`	Logical unit not ready, initializing command required	Drive needs a `START STOP UNIT` command to spin up.
`0x6`	`0x29`	`0x00`	Power on, reset, or bus device reset occurred	Drive dropped power or received a hard SCSI bus reset.
`0x5`	`0x21`	`0x00`	Logical block address out of range	Requested LBA beyond the partition or disk capacity.
`0x4`	`0x44`	`0x00`	Internal target failure	Drive firmware or internal ASIC crashed. Replace disk.

6. Real-World Troubleshooting Scenarios

Scenario A: Identifying a Failing Hard Drive (Unrecovered Read Error)

dmesg outputs the following error during a database query:

sd 0:0:0:0: [sda] tag#4 Sense Key : Medium Error [current]
sd 0:0:0:0: [sda] tag#4 Add. Sense: Unrecovered read error (ASC: 0x11, ASCQ: 0x00)
critical medium error, dev sda, sector 8412032 op 0x0:(READ)

Diagnosis: Sector 8412032 on /dev/sda has sustained physical damage. The controller was unable to reconstruct data via ECC. Action: Replace the physical drive and restore affected database blocks from backup.

Scenario B: Diagnosing SAN Cable Disconnect or Port Flap

A cluster node loses access to its Fibre Channel LUN:

sd 3:0:1:5: [sdc] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
sd 3:0:1:5: [sdc] tag#0 CDB: Test Unit Ready 00 00 00 00 00 00

Diagnosis: hostbyte=DID_NO_CONNECT indicates the host HBA completely lost transport link connectivity to Target 1, LUN 5. This is not a drive failure; check Fibre Channel switches, SFP transceivers, or network patch cables.

Scenario C: Resolving a Reservation Conflict in Clustered Environments

During a Kubernetes or Pacemaker cluster failover:

sd 1:0:0:1: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_stat=0x18

Diagnosis: cmd_stat=0x18 represents RESERVATION CONFLICT. The previous cluster node still holds a SCSI-3 Persistent Reservation (PR) on /dev/sdb, blocking the new node from acquiring read/write locks.

7. Operational Best Practice & Warning

CRITICAL PRODUCTION WARNING: Never leave verbose SCSI logging (--all=7 or --mlqueue=7) enabled indefinitely on production servers processing high IOPS. Generating thousands of debug lines per second into dmesg can saturate syslog daemons, cause CPU lockups in the kernel printk buffer, and fill root filesystem partitions with log files. Always disable logging once troubleshooting is complete:
sudo scsi_logging_level --set --all=0

Linux SCSI Logging & Diagnostics (scsi_logging_level) — Complete Learning Notes & Output Guide

1. What is SCSI Logging in Linux?

2. Enabling & Controlling SCSI Logging

Method 1: Using the scsi_logging_level CLI Tool

Method 2: Direct Kernel /proc Interface (Zero Dependencies)

3. The SCSI Logging Categories (Bitmask Allocation)

4. Anatomy of a SCSI Log Line in dmesg

Raw Output Example: Medium Read Error (Bad Sector)

5. Breakdown of Every Output Heading & Field

5.1 SCSI Device Address (sd 2:0:0:0:)

5.2 Device Name ([sda])

5.3 Queue Tag (tag#12)

5.4 The 32-bit SCSI Result Word (Result: hostbyte=... driverbyte=... cmd_stat=...)

Common SCSI Status Byte Codes (cmd_stat)

5.5 The SCSI Command Descriptor Block (CDB)

5.6 SCSI Sense Data (Sense Key, ASC, ASCQ)

1. Sense Key (Primary Category)

2. ASC (Additional Sense Code) & ASCQ (Additional Sense Code Qualifier)