If I remember correctly it would have taken voyager 70 years to get home at fedration warp but they got some borg tech for getting there more quickly.

Ah, but they (SW folk) had to take thousands of years because somebody had to actually go in that direction, measure the gravity wells, and chart that system before continuing.

All the Empire need do is take one Federation ship, and download its star charts. The Federation has pretty detailed maps; sure, they're always exploring, but that's to go to each little system and make friends with everyone they meet. The Empire's first job in attacking would be to cripple any significant resistance, and make any secondary powers think twice before rising up.  They only have to destroy the fleets of the Federation and their Allies.

And, remember, the Federation is, predominantly in only one quadrant of the Milky Way galaxy.  And that's not a quarter of the galaxy, either.  The Alpha Quadrant is like a truncated wedge at most 60 degrees of the arc of the rim, going coreward about one-third the distance to center. Not much for the Empire to fight.

Now, the Federation would probably make deals with everyone they could, including the Borg and those in the Beta and Gamma Quadrants (Romulans, Cardassians, Dominion, uhm... Breen... whatever), and if by then they could communicate with the Delta Q., they'd try that, too.  But the core of any resistance to the Empire would be the Federation, keeping the groups working together (or far apart, for those who don't work well with others).  The Empire could figure that out pretty quickly, I'm sure.

Side note: Anybody ever notice that the few times we saw Breen in ST:DS9 they had helmets near-exactly like the one Leia wore when selling Chewbacca to/rescuing Han from Jabba? Including the visor with a sort of floating zero-point grid?

Edit: Sorry, on a whim I decided to find pics of each to add to the post, and found that there are plenty of sites, including wikipedia, which note the similarity between the helmets, and reasons why. Just search for Breen and Boushh.

The Borg would spend all their resources taking over a giant ship, then using that giant ship to take over other ships and giant ships and so on. It'd be like a cancer.

The Romulans would be the only one's that might join the Empire and they don't know a thing beyond an isolationist bubble of space. The federation ships would also not be easily boarded as they don't have any ports for that and the Empire doesn't have transport technology.

Besides, the rebel alliance would be on the milky way's side.