"I wanted all my visits to be official. When I sent the pass back with a note, I had no idea it would antagonize the president. I found out years later that it did."