"The unions may continue to decline, but if they do, it'll be their fault."