Can I obtain a restraining order on behalf of my children against their mum's partner?

I am concerned for my children's well-being (aged 4 and 2). This week their mum's partner received a 'Street Caution' as he had 5 cannabis plants growing in the house that they live in. Incidentally it is also my house, my ex-partner gained the right to live there with the children about 12 months ago, which I had no problem with. Now her new partner lives there too, he's had violent episodes already and now I find out they've been growing cannabis in the basement! Do I have grounds to remove this awful influence from my children's lives? Also, I don't understand why but she didn't get in any trouble even though she's just as responsible, I guess that's British law for you. So would I have a reasonable case for full custody of the children and the rights to my house back? If I can be my children's main carer I can provide a much more stable, calm and above all, legal environment for them. Any advice would be appreciated, I plan on seeing a solicitor on Monday morning, but in the mean time I need to put together an action plan so I can act on it as quickly as possible.