[PATCH 5.4 403/453] spi: atmel-quadspi: Fix use-after-free on unbind