Doctors have called for Andrew Lansley to resign, saying they have "no confidence" in the Health Secretary.
Medics went head to head with Mr Lansley last week over the Government's controversial pension reforms.
They took industrial action for the first time in almost 40 years to voice their anger at the proposals.
The BMA accused Mr Lansley of "breaching doctors' trust" by tearing up the pension deal which was only agreed four years ago.
Delegates at the British Medical Association's (BMA) annual conference in Bournemouth agreed on a motion calling for Mr Lansley's resignation.
Delegates at the conference voted on the motion "This meeting has no confidence in Andrew Lansley, the Secretary of State for Health, and calls for him to resign".
The motion won by a slight majority, with 158 delegates voting in favour and 124 against.
Dr Gary Marlowe, who presented the motion, told the conference: "I do not trust this man."
He accused Mr Lansley misleading doctors and the public during the election over his plans for the Health and Social Care Act, which became law in March after a difficult passage through Parliament.
He said: "I understand that Mr Lansley is quite a nice man and can on occasion be quite charming.
"If only this was enough to be Health Secretary overseeing the biggest reorganisation of the NHS since its inception.
"It is clear that we have a Health Secretary who is ideologically-bound, thinks he knows best, is disingenuous and sticks his fingers in his ears 'la la la la'.
"I know there will be the argument that we should not call for someone to resign with whom we might have to negotiate tomorrow but he doesn't listen anyway so that is no argument.
"As every doctor knows, trust lies at the heart of everything we do. We expect a similar degree of trust with the Health Secretary. I do not trust this man."
Dr Peter Holden said: "I should declare an interest, I am a card-carrying Tory. Resignation is too good for Lansley - Cameron should sack him."
However, the BMA's outgoing chair of council Dr Hamish Meldrum urged doctors not to pass the motion, saying that it was about policies, not personality.
Also speaking against the motion, Dr Glynn Evans said: "Imagine our ongoing negotiations: 'Dear Secretary of State, officially I have no trust in you, officially I want you to resign, but can I put that aside and negotiate with you?' It's not going to work."