[PATCH 5.4 163/367] serial: sprd: remove redundant sprd_port cleanup