"When you walk away from a really wonderful job like that, you start messing with everyone's priorities. It's like you're dissing them."