Should I stop hanging out with these people?

Hey. Last year I was a pretty secluded guy. I used to just sit at home, hang with three people and just play games - nothing more, and I was happy with that lifestyle. This year I went to college and net around 5 people who I hung/hang around with regularly. Being with them made life seem more fun and enjoyable, however some of their traits have rubbed on me. For example, I would now look at women and discuss their looks with these friends, which was something I never did and thought I would never do. Some of these friends smoke weed and sometimes I think of doing it however, I know it's wrong but being around them makes it seem right even though I know it's not. I'm not sure if I should cut contact or what, because I feel I'm week and I might succumb to the peer pressure. I don't want to defriend them because I've never had fun like this in years but part of me says to let them go. What do you think is an appropriate choice? Has anyone ever been in said predicament?