[PATCH 4.16 107/196] xprtrdma: Fix corner cases when handling device removal