Modi’s own Union Minister spoke against him; said the reason for the failure