It comes amid increasing pressure on the auditing giant after it cleared out its senior structures last Friday over its dealings with Gupta businesses.
KPMG has also admitted to shortcomings in its investigation into the so-called rogue unit at the South African Revenue Services (Sars) - saying the findings and recommendations should "no longer be relied upon".
There are also mounting calls on KPMG to be transparent and account for any wrongdoing.
Gigaba's predecessor, Pravin Gordhan, who faced legal action over the Sars report, met with an international delegation from the auditor on Thursday night.
"We insisted on full and proper disclosure of the role of various parties in the state capture project and the manner in which KPMG staff seem to have colluded in these processes," Gordhan says in a subsequent statement.
Meanwhile, Business Leadership South Africa has also suspended KPMG's membership pending the outcome of an independent investigation.
Gigaba has also joined this chorus, saying "bad elements" should be rooted out.
"It is therefore, warranted and critical that the relevant law enforcements and bodies such as the Independent Regulatory Board for Auditors (IRBA) look into this matter to identify and sanction those responsible for any wrong-doing," reads Gigaba's statement.
Gigaba adds the crisis speaks to risks of market dominance, highlighting the need for audit firms to rotate.
The minister further wants government to consider the possible "regulations for both the public and private sector in an effort to ensure and preserve the integrity and good governance in the audit fraternity".
KPMG has lost as many as three clients in the last week.
