"There's been a lot of speculation about every single girl I'm with and it actually does quite irritate me after a while, more so because it's a complete pain for the girls."