Folks do it all the time for the rarer cars, particularly ones that are still under warranty.
When I worked in auto sales we would sell used cars to folks all over the country - SRT8 Jeeps and 300s mostly, but also the ones with odd color/package combinations.
If it is at a dealership, read up on their reviews first. If it's private party, I'd be more wary.
If you're that certain it is the car you want, make it happen. Just make sure your deposit to hold it is fully refundable if the car isn't as advertised when you get there. And settle on the price before you head out there, as they'll know you are not at all likely to walk away at when you would rather drive away in your RS5...
23 Open Bank Cards / 7 Open Store Cards / TOO many inquires / Mid/High 600's