Windhelm Guards attack on sight

Hi,

I happen to have the problem that the guards in Windhelm (starting with the ones directly infront of the gate) attack me on sight without doing anything.
As soon as I arrive by carriage or by foot or even at the bloody harbor entrance, they attack me. Is it because I joined the empire? (I even tried to get in wearing SC armor).

I also can't yield to them, it's either wipe out Windhelm or die trying :P

Cheers,
MacJoker

edit: I noticed that it's not every guard that is hostile towards me. Seems like only the "standard" ones.