A packed meeting in parliament on Tuesday night, which included victims of the undercover officers, voted unanimously to send the high court judge packing.
Mr Mitting has indicated on the inquiry website that he may hear evidence in closed sessions despite victims campaigning for an open and transparent investigation.
Doreen and Neville Lawrence, whose son Stephen was murdered by racists in 1993, are among the victims of the undercover spying operation.
Mr Lawrence told the meeting he and many other victims of undercover police spying had “lost faith” in the inquiry given the new direction taken by Mitting.
Imran Khan, lawyer for Ms Lawrence and the Blacklist Support Group, told the meeting there had been a “paradigm shift” since Mr Mitting took over.
Victims say that in the three years since the inquiry was ordered, not a single witness has given evidence and not a single document disclosed to the lawyers of the victims. They accused the police of “deliberate obstruction.”
Stafford Scott, from Tottenham Rights, told the meeting that “families of murder victims are being denied access to files kept on them” due to ongoing institutional racism by the Met Police.
Dave Smith, a blacklisted trade union activist, said that victims had “always been sceptical whether the British state would truly expose the truth about the UK’s secret political police units.”
MP Naz Shah chaired the meeting, promising to raise the concerns on the home affairs select committee and with the Home Secretary.
The inquiry was set up by then home secretary Theresa May in 2014 after it was revealed that police officers who infiltrated political campaign groups had formed intimate relationships with campaigners.
They even stole the identities of dead children to cover their real identities.
The government chose Mr Mitting to chair the inquiry after the original chairman, Lord Justice Pitchford, was diagnosed with motor neurone disease.