Tard dans la nuit

Late at night I walked down an empty street alone and came to stop in front of a pretty building I had seen before. Up on the terrace sat a handsome man in a black velvet coat and he beckoned to me, "come up for a smoke." Unperturbed, yet alone, I refused and left never to see him again…
Last night and the night before I dreamt of him again. It's a recurring dream that has resulted in many restless nights. It disrupts my sleep with feelings of regret and longing.
Who was that man and why do I yearn to be with him again?