[PATCH 5.4 164/367] serial: sprd: Assign sprd_port after initialized to avoid wrong access