On Tue, Oct 16, 2018 at 12:32:29PM -0700, Aditya Kali wrote:

There is a deadlock on linux stable kernels (tested 4.9 and 4.14) when running an executable from an NFS mount when CONFIG_IMA is enabled (default IMA policy used).

Repro steps (tested on debian 9 with 4.9 kernel and with 4.14 kernel): $ sudo mkdir -p /mnt/nfs $ sudo mount -o ro,exec,nolock <ip>:/vol /mnt/nfs/ # "test.sh" is an executable on the nfs volume $ /mnt/nfs/test.sh <deadlock>

The following stack trace shows the deadlock.

[ 2479.260272] bash D 0 25187 25140 0x00000080 [ 2479.267680] ffff93f4d9eec000 0000000000000000 ffff93f4dc211040 ffff93f4ffd18980 [ 2479.278602] ffff93f4dd8370c0 ffffbc2ac386b8f0 ffffffffbde0fed9 0000000000000000 [ 2479.288154] 0000000000000046 ffff93f4ffd18980 ffffffffbd8a2cda ffff93f4dc211040 [ 2479.297712] Call Trace: [ 2479.300311] [<ffffffffbde0fed9>] ? __schedule+0x239/0x6f0 [ 2479.306005] [<ffffffffbd8a2cda>] ? check_preempt_curr+0x7a/0x90 [ 2479.313524] [<ffffffffbde103c2>] ? schedule+0x32/0x80 [ 2479.318801] [<ffffffffbde12d30>] ? rwsem_down_read_failed+0xf0/0x150 [ 2479.326855] [<ffffffffbdb3f054>] ? call_rwsem_down_read_failed+0x14/0x30 [ 2479.333802] [<ffffffffbde125ec>] ? down_read+0x1c/0x30 [ 2479.340542] [<ffffffffc050b6b9>] ? nfs_start_io_read+0x19/0x70 [nfs] [ 2479.348529] [<ffffffffc0502c6f>] ? nfs_file_read+0x2f/0xa0 [nfs] [ 2479.354779] [<ffffffffbda06fed>] ? new_sync_read+0xdd/0x130 [ 2479.361981] [<ffffffffbdae2a43>] ? integrity_kernel_read+0x43/0x60 [ 2479.368606] [<ffffffffbdae484c>] ? ima_calc_file_hash+0x36c/0x710 [ 2479.376312] [<ffffffffbdae453b>] ? ima_calc_file_hash+0x5b/0x710 [ 2479.382550] [<ffffffffbd98aa86>] ? __alloc_pages_nodemask+0xf6/0x260 [ 2479.389124] [<ffffffffbdae5485>] ? ima_collect_measurement+0x145/0x1b0 [ 2479.397258] [<ffffffffbda2e7fb>] ? vfs_getxattr_alloc+0x5b/0x110 [ 2479.403490] [<ffffffffbdae3c73>] ? process_measurement+0x393/0x550 [ 2479.409913] [<ffffffffc04a177f>] ? generic_hash_cred+0x2f/0x60 [sunrpc] [ 2479.418138] [<ffffffffc04a19c0>] ? rpc_lookup_cred_nonblock+0x20/0x20 [sunrpc] [ 2479.426972] [<ffffffffc04a058a>] ? rpcauth_lookup_credcache+0x9a/0x2c0 [sunrpc] [ 2479.434520] [<ffffffffc0465230>] ? nfs3_proc_commit_setup+0x10/0x10 [nfsv3] [ 2479.443082] [<ffffffffc050717f>] ? nfs_attribute_cache_expired+0x2f/0x80 [nfs] [ 2479.450672] [<ffffffffc050749a>] ? nfs_revalidate_inode_rcu+0x1a/0x40 [nfs] [ 2479.459215] [<ffffffffc05002de>] ? nfs_do_access+0x29e/0x350 [nfs] [ 2479.467010] [<ffffffffbda0e13a>] ? search_binary_handler+0x2a/0x1c0 [ 2479.473651] [<ffffffffbda0f8fa>] ? do_execveat_common.isra.37+0x5aa/0x790 [ 2479.482055] [<ffffffffbda0fd05>] ? SyS_execve+0x35/0x40 [ 2479.487508] [<ffffffffbd803b7d>] ? do_syscall_64+0x8d/0xf0 [ 2479.494586] [<ffffffffbde14c4e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6

Both process_measurement() and nfs_start_io_read() try to acquire the inode-lock:

process_measurement() { ... inode_lock(inode); { down_write(&inode->i_rwsem); } ... nfs_start_io_read { ... down_read(&inode->i_rwsem); ...

This deadlock is primarily fixed by following commit in linux master (since 4.15):

0d73a55208e94fc9fb6deaeea61438cd3280d4c0 ima: re-introduce own integrity cache lock

The complete list of commits required (with clean cherry-picks) to fix the issue in 4.14 stable kernel is below:

e2598077dc6a26c9644393e5c21f22a90dbdccdb ima: re-initialize iint->atomic_flags 0d73a55208e94fc9fb6deaeea61438cd3280d4c0 ima: re-introduce own integrity cache lock 50b977481fce90aa5fbda55e330b9d722733e358 EVM: Add support for portable signature format f3cc6b25dcc5616f0d5c720009b2ac66f97df2ff ima: always measure and audit files in policy

For 4.9 kernel, one additional commit is required and there is a minor conflict resolution needed:

19339c251607a3defc7f089511ce8561936fee45 Revert "evm: Translate user/group ids relative to s_user_ns when computing HMAC"

Since the kernel deadlock is trivial to trigger for common default configurations, I would like to request cherry-picking the above commits to linux-stable branches 4.14 and 4.9.

All now queued up, thanks.

greg k-h