There may be many of us who wants to go skiing in the winter, but perhaps cannot (or do not want to) fit our skis in our little car. So we attach the roof rack when the first snow comes. And then keep it on until the skiing season is over. Well, honestly, who bothers with taking the rack off if you are going to use it the next weekend? So it stays on.
As long as you only drive around town or shorter trips to the store at lower speeds it may not have a big impact (as long as you do not have a cargo box attached!). But if you take longer trips and particularly at higher speeds it does affect your gas mileage quite substantially. Probably more than you think.
In an experiment by Darin at MetroMPG where he compared the gas mileage at 55 mph for a car with only a roof rack, with a bike attached on the roof or without anything mounted, the distance travelled with the empty rack attached decreased with 12,7% compared to a clean roof. When he then attached his bike on top of the car the mileage was reduced by a whopping 27,3%!
So make sure that you have the right tire-pressure, a light right foot and no unnecessary roof rack attached! That's a win for both the environment and your wallet.