You know that one scene on the hospital roof, the 'I want to stay with you forever' scene? I had a thought, if you will. We know that Yugi is self-sacrificial, to his own destruction. Atem is too. Dude got himself locked up in a puzzle for thousands of years, lost his memory, lost his name, lost all his friends, basically lost his country, all so he could save the world.
During the fail-date with Anzu, Yami does express the desire to know his past. But he doesn't want to know it because knowing it would upset the norm. He knows, somehow, that learning his memories will take him away from Yugi. And he would rather sacrifice something he really wants, just to make sure Yugi is happy.
And on the flipside, Yugi has to keep pushing Atem towards his memories, because, he too, is a self-sacrificial little muffin. He knows that if he does not push, if he does not hurt his own heart, Yami, his other self, will never be more than that. Man. Neither of these guys know how to be selfish. Technically Anzu is the only reason that Yami decided to go to the museum. (Again, because she suggested it. Not because Yami wanted to go initially.)