Modi gave the invitation to the swearing-in ceremony in Mumbai before the polls