[PATCH 6.4 517/737] serial: sprd: Assign sprd_port after initialized to avoid wrong access