Move all socket handling into RemotePeer