[PATCH backport] ntp: Remove invalid cast in time offset math