[added to the 4.1 stable tree] spi: sun4i: disable clocks in the remove function