My cat's always getting into fights with other cats!?

We've got him castrated, however, he still seems to getting attacked! He doesn't start or provoke other cats! They just come by themselves!