"James Caan told me at the end of filming 'Elf' that he had been waiting through the whole film for me to be funny - and I never was."