Is it appropriate to wear a Bowler Hat in Melbourne?

People will look at you strangely as Bowler hats are not usually seen. Still does that matter? Just don't get yourself in a position of guy with bowler hat in a gang of bogans.