Vulnerable and disadvantaged groups

How can a society which is supposed to be equal have several groups of people that are categorized as disadvantaged and vulnerable. I'm not only talking about the homeless but the elderly, the disabled, the mentally ill. We have a society still after centuries that some shun the people that I have mentioned. Sometimes it's not intentional. It seems to be an in built piece of mechanism we humans have. If I was an alien ( which some people think I am by the way) I would be confused. Although some of us are different in colour and looks. We are all still basically the same. We have red blood, eyes and ears, legs and all that stuff that makes us human but what should make us stand out as a compassionate world seems to be lacking in some of us. How come?
I would have thought the two world wars would have made us different but it really hasn't. We still fight wars still over pieces of land and religion. Instead of trying to live together as we should. We still rob and cheat our way to things. I'm not saying all society but if you look at the level of crime and stuff. It does seem that way.
When I see things like I did yesterday. It even makes me think more on actually who needs to learn how to be tolerant of other people. I sat in the my local Job center yesterday again ( actually got somewhere this time) but there was a girl who was partially deaf and when she spoke it was with sign language. These people came in sat down then after seeing this girl started to take the mickey out of her. They weren't young I would say in their late twenties. Anyway it got me to thinking that as a society we say lots of things but actually do the opposite like for instance when we go for jobs age isn't supposed to matter but all employers look at the likelihood of someone say 54 not being able to cope or someone with a stammer not being able to communicate. It doesn't matter that they can do the job. It's that thing of first appearances. The government say it's illegal to discriminate but society does. Is this one way on how we are perpetuating more disadvantaged and vulnerable people? It's just a thought.