Haven’t got time to read this story? Listen to it in audio format.

For cybercriminals, deepfakes, where synthetic video or voice is used to scam someone, is becoming big business. According to cybersecurity researchers Group-IB, the tools needed to create these fakes are selling on the dark web for less than US$30, with individually cloned faces and voices going for US$5 – and the payoffs are huge.

Group-IB found deepfake fraud led to verified losses of more than US$300 million in a single quarter during 2025 and the numbers are only going to keep going up.

“From the frontlines of cybercrime, we see AI giving criminals unprecedented reach,” Anton Ushakov, head of Group-IB’s Cybercrime Investigations Unit, told The Register. “Today, it helps scale scams with ease and hyper-personalisation at a level never seen before. Tomorrow, autonomous AI could carry out attacks that once required human expertise.”

However, there’s also a sense that, at its core, all fraud is essentially ‘same fish, different wrapper,’ says James Roberts, general manager of group fraud, Commonwealth Bank (CBA).

“With the introduction of the internet, there were new channels or ways of doing scams but the underlying identity or investment scams have been stable for years,” he says. “The way of conducting these scams has changed with technology but there’s a core that remains the same to them.”

The pretender

The technical tools to create audio and video deepfakes are easy to find and easy to use, says Kathy Reid, a PhD candidate specialising in AI voice at Australian National University’s (ANU) School of Cybernetics. Five years ago, cloning a voice required up to 1000 hours of high-quality studio recordings. Now, using tools like Microsoft’s VALL-E, the same job needs less than a three second sample.

With video deepfakes, up until recently models struggled with rendering fingers and joints, says Mike Seymour from the University of Sydney’s Motus Lab. “These problems have largely been solved,” he says.

Essentially, it is becoming very hard to simply look at something or listen to a voice and know it is fake.

Deepfakes and shallow fakes in financial fraud

Shallow fakes don’t require any specialist know-how to create and aren’t dependent on AI, says Jessica Tilbury, a partner at national law firm Holding Redlich. A shallow fake could be something like a scam email sent by a tax authority, asking an individual to update their personal details or make an urgent payment via a link. If someone opens the link, they’re exposing themselves to a high risk of financial fraud.

Deepfakes are more sophisticated but can also be created using commonly available tools online and via the dark web. According to Tilbury, a deepfake might be AI-generated audio of a company’s director asking someone to quickly transfer funds.

“In November 2024, a Northern Territory government agency received a shallow fake email which appeared to be from a construction contractor working with the agency,” Tilbury says.

The email included a vendor identification form with new bank account details and cc’d a variety of email addresses consistent with the actual construction company. 

“In reliance upon the email, the agency subsequently transferred A$3.5 million to the fraudulent account, however the bulk of funds were subsequently recovered by the Australian Federal Police,” she continues.

Deepfakes have also caught people out. In February 2024, the Hong Kong branch of British engineering firm Arup fell victim to a deepfake scam totalling US$25 million. An employee received an AI-generated video call purporting to be from a senior executive at Arup, requesting the transfer of the funds to five separate Hong Kong bank accounts.

“Believing the people present on the call to be authentic, the employee made the transfers under instruction from the deepfake scammers,” Tilbury says.

In New Zealand, the National Cyber Security Centre (NCSC) has reported an uptick in both deepfakes and shallow fakes aimed at defrauding individuals and businesses and acknowledges these fakes are hard to fight.

“The NCSC has had reports of deepfake videos of prominent New Zealand figures being used to endorse investment scams to make them seem more credible. These videos have been commonly observed on social media sites and therefore often only appear to certain individuals, making them challenging to combat,” a spokesperson says.

Fighting back against AI fraud

What techniques can CAs use to try and beat AI fraud?

The NCSC says the number one rule is to always verify the sources and check through independent channels who is making the request – especially in the event of a funds transfer.

According to the NCSC spokesperson, CAs should be on the lookout for behaviours including changes in normal processes, such as a request coming via a channel like Telegram. Scammers may also ask victims to bypass usual processes, including requests to skip the usual verification steps, or asking the victim not to involve certain individuals.

There will also often be a sense of urgency involved, the NCSC says. CBA’s Roberts concurs, observing scammers will attempt to pressure people into doing something fast and this pressure may lead the victim to bypass the usual steps in approving payments or making transfers.

PwC New Zealand’s director of GenAI, Kayur Patel FCA, says it was immense pressure which led to the success of a recent six-figure deepfake scam.

“In this instance, it was a live deepfake call taken by the CFO of an organisation from someone purporting to be the CEO, asking the CFO to make a payment,” Patel says. “In many ways it was the perfect storm because the deepfake was believable and the scammers used high-pressure techniques to get the victim to make the payment.”

The NCSC also says to watch for ‘out of the blue’ changes in contact or payment details, where the request does not come from the usual channels, as well as an avoidance of verification.

“For example, resisting any attempts to verify the person’s identity via mechanisms like a call-back or confirming the person’s identity with separate individuals,” the NCSC spokesperson explains. “This advice is very similar to the advice we would provide for scams not involving deep- and shallow fakes, as there is often overlap in the techniques used by malicious actors.

“You should also check if your insurance policy covers social engineering, deepfake or cyber-related fraud, which may cover financial loss, legal fees and forensic investigation costs.”

What to look out for

Professor Sanjay Jha from the University of New South Wales Institute for Cybersecurity says there are several things CAs should look out for when trying to spot deepfake audio or images. These include ‘unnatural audio’, where there is a lack of background noise, odd pauses or a lack of emotion in the voice.

“People can also look out for visual glitches, such as fuzzy face edges, poor lip-syncing and excessive or unnatural eye blinking,” Jha says.

“Ultimately,” he says, “these fakes have become so good you really need to use your ‘thinking hat’ – if the request or situation feels out of character or bizarre, it likely is.”

ANU’s Reid agrees, saying the fakes are now so good, the only place where they ever get tripped up is in what are called ‘named entities’ – pronouncing the names of places, people and so on.

Reid says to listen out for how names, particularly Indigenous place names, are pronounced. All Australians, for example, will pronounce Canberra as ‘Can-brah’ or Brisbane as ‘Brisbun’, but deepfake audio will often fall back to Americanisms, like ‘Can-berra’ or ‘Bris-bane’.

“Overall, however, these deepfake audios are incredibly hard to spot.”

The NCSC’s advice also follows these patterns, while admitting looking for unnatural details in audio or video deepfakes is very unreliable.

“Our advice is to always verify,” NCSC’s spokesperson says.

If you suspect you’ve been scammed

So, you acted on a suspicious call and you think you’ve been scammed. What next?

Even if you only have a niggling thought that the worst may have happened, act immediately, don’t pretend that nothing has happened, says Roberts. Report it to the higher-ups in your company and the authorities. Banks and financial institutions should also be contacted immediately, as sometimes they’re able to halt the transaction and recover the funds.

Patel also says it’s imperative to preserve the evidence. “Don’t delete the suspicious files or emails,” he says. “The IT team will need them to investigate.”

He also agrees with Roberts, saying scams must be reported to the appropriate channels, including the Australian Cyber Security Centre (ACSC) and call the hotline (24/7) on: 1300 292 371. In New Zealand, report to CERT NZ and call NCSC (Monday to Friday, 7am–7pm) on: 0800 114 115.

“Don’t be ashamed, either,” Patel adds. “Cybercriminals rely on the shame and embarrassment of the victim to stay hidden, and reporting is the only way to build societal safeguards.”

Holding Redlich’s Tilbury also observes accountants who are victims of financial fraud may be entitled to pursue civil remedies, including damages for monetary loss, consequential losses and expenses incurred as a result of the fraud.

“Unfortunately, the identity of a scammer is often unknown to the victims, making it difficult to apply for injunctive or other relief from the court,” she adds.

“You should also check if your insurance policy covers social engineering, deepfake or cyber-related fraud, which may cover financial loss, legal fees and forensic investigation costs.”

The truth of the matter is AI fraud is incredibly hard for the average person to spot and these fakes are only going to get better. The core advice for CAs is: don’t succumb to pressure to make payments, don’t go outside the normal processes for approval and, if it seems too good to be true, then it probably is.

Take a minute! Go with your gut. Trust your intuition and remember the mantra: ‘verify and verify again’.

---

Take away

For more advice on preventing AI-fueled cyberattacks, the CA Library offers Artificial Intelligence and Financial Security: Harnessing AI to Protect and Optimize Financial Systems.

---