In an interview to India TV's Chairman and Editor-in-Chief Rajat Sharma in front of nearly 2,500 people at Jawaharlal Nehru Stadium here, he said that he had made friendly gestures to both Prime Ministers Nawaz Sharif and Imran Khan but these were not reciprocated.
Describing his sudden visit to Pakistan in 2015 while returning from Afghanistan, he said Sharif called him over to Lahore to meet him.
Modi said that his visit was intended to send a message that India does "not bear any ill-will towards Pakistan".
"I discussed with Sushmaji (External Affairs Minister Sushma Swaraj), she said 'you decide'. I talked to NSA, SPG. Everybody was worried, since the officials had no visas nor were there any security arrangements in place, neither did anybody knew about the layout (of the place). We will have to land straight. I said 'Come on, let's go, we'll see."
He describe Sharif as a "genuine person".
"They were being fed lies about India. The message went to them that India desires the well-being of the people of Pakistan. We returned, and within a week, Pathankot (attack) happened," he said, referring to the attack at the IAF airbase in the Punjab border town.
Modi said that when Imran Khan became PM, they talked over phone. "I told him that both the countries have fought several wars, and every time Pakistan was defeated. Both of us as Prime Ministers should work towards eradication of poverty in the next five years," he said.
However, then came incidents like Pulwama.
"The biggest problem with Pakistan is that nobody knows who is running the country and whom we should talk to," he said, adding that his experience with Pakistan was not isolated but leaders from the US, China, Russia, the Gulf and Arab countries share the same views.
Modi said he was told by several world leaders that he would not come to know whom to talk in Pakistan. "Whom will you talk to... with the Army, with the ISI? Or, with an elected body? The leaders told me, 'We ourselves don't know who runs that country'."
"Let Pakistan resolve its problems first," he added.