[PATCH 5.4 34/84] spi: Cleanup on failure of initial setup