What’s your NAT sort?
Solely circumstance I can think about is that your router is obstructing connections initialized externally to your own home community. Clearly your Xbox can initialize an exterior connection as a result of you might have add velocity.
I might test your router, make certain UPNP is enabled, be sure that the right port forwarding is enabled, see the hyperlink. https://help.xbox.com/en-GB/assist/hardware-network/connect-network/network-ports-used-xbox-live
I might additionally strongly advocate utilizing CAT6e over WiFi. WiFi is nice in the event you’re shifting round on a regular basis, however wired is extra steady, dependable, and has increased bandwidth.