The process of writing about the impact of LLM on audit felt like multiple reunions in the same venue - I started my career as an auditor at EY’s Hi-Tech practice, losing my hairs while doing one too many audits or FDDs. While I did manage to move the team from trial balances by fax to ascii files by email, the audit work and tools we used back then were quite primitive, but surprisingly (or not?) haven’t changed much across big firms in the 20 years that passed since, irrespective of the exponential increase in tech adoption and complexity of business model. Then, in different CFO roles over the years, it was interesting to see from the other side the same manual work, that often hardly scratched the surface of the areas that were supposed to be checked and assured. Years later at Lloyds, I met Validis, which provided an advanced algorithm to automatically assess the reliability of accounting data for lending and monitoring. Validis was also the matchmakers to Pollen Street. Both gave me the opportunity to then look much closer into audit transformation with tech as part of the Pollen Street Hub support to Validis.

While a proper comprehensive analysis of the transformational impact of LLM on Audit would require a book rather than a blogpost, I’ll try to cover the main aspects without oversimplifying it.

Audit 101

The audit process can be defined as a systematic and comprehensive examination of a company's financial statements and underlying accounting records to determine within a certain level of assurance, whether they are prepared, presented, and fairly stated in accordance with generally accepted accounting principles (GAAP). Financial statements (and audit process) are based on the following three tiers:

Tier 1: Core Business Operations

This tier involves the scrutiny of primary financial transactions as recorded in the accounting or ERP system by the core financial processes e.g.:

• Order-to-Cash cycle
• Purchase-to-Pay cycle
• Payroll processing
• Tax accruals and payments
• Debt Management
• Bank Reconciliation.

Tier 2: Accounting Adjustments

This tier addresses adjustments made to align Tier 1 records with established accounting principles. This includes for exmaple:

• Income and expense deferral
• Recognition of deferred items – accounting for previously postponed income and expenses.
• Depreciation calculations – assessing asset depreciation over time.

Tier 3: Accruals and Estimates

The final tier involves incorporating accruals and estimates into financial statements, e.g.:

• Project completion estimates – recognising income and expenses based on completion percentages
• Provisions for doubtful accounts – estimating potential losses from uncollectible accounts
• Legal claim outcomes – expected financial impact of pending legal matters
• Amortisation and write-offs – adjusting for asset value reductions.

The audit process for small and medium enterprises (SMEs) or private businesses typically includes verifying key aspects of financial items in the balance sheet and profit & loss statement. These include the existence, accuracy, completeness, and proper classification of financial items. Auditors employ various methods like external balance confirmations, flux analysis, and transaction sampling to ensure the financial statements accurately reflect the company's fiscal position according to Generally Accepted Accounting Principles (GAAP).

For larger or publicly listed companies, the audit extends to evaluating business processes, internal controls, and financial closing procedures. The effectiveness of an auditor is often measured by their ability to ascertain financial accuracy and compliance with minimal reliance on extensive sampling, thereby ensuring efficiency and reliability in the audit process.

The beginning of the end (or a new beginning)?

The launch of ChatGPT a year ago and the myriad of new LLM models that have launched since, marks the beginning of a new era in how we do audits.

At the most basic level, LLMs can run over general ledgers (GL) to perform most (if not all) of the basic checks including:

• accounts/ balances classification
• completeness checks
• third party (banks, customers, etc.) confirmation verification
• reconciliations
• search and summary of transactions – e.g., all related party transaction across multiple entities
• etc. etc.

Such checks can be operated manually through prompts, or automatically by agents based on an audit plan.

LLMs can also prepare an advance draft of the financials, including notes and other disclosures, based on the financial data and updates to accounting rules and policies. Again, manually prompted, or by agents.

The more interesting and advanced applications would include:

• automated flux analysis – analysing and explaining changes to items and ratios in the financials between different periods
• anomaly detection – for simple financial ratios, down to GL level transactions, post financials date transactions, etc.
• contract analysis and assurance – e.g. revenue recognition from a contract based on the relevant GAAP. As LLMs can automatically recognise the different commercial parameters in a contract and identify the footprint of which in the GL, they can verify the contract is correctly reflected in the data and presented in accordance with the applicable GAAP. In such a case, LLMs can audit 100% of the population! And at a fraction of the time a human auditor would take.

The following table summarises the level of automation LLMs can drive for the different audit procedures for each tier:

Such automation and the ability to compile and assess fully audited financials at the fraction of the time and cost, will clearly impact the wider group of stakeholders, including investors, lenders, investment banks, and CFOs.

AuditGPT

Any experienced auditor would rightly say that I’m missing a trick here, as at its core, audit is about risk management – i.e., properly identifying the audit risks of the business, and tailoring a bespoke audit plan to address those risks.

LLMs are great tools to help the auditor. They can and will improve the quality of audit while driving efficiencies to new levels. These models continue to get stronger and more widely used and the largest firms have embraced this technology already. For smaller/ straightforward audits, LLMs can do a good portion of the work today (as indicated in the above table). For larger more complex audits, it is a valuable tool to help in many areas to improve decision making.

The obvious question would then be – what will be the role of the future (human) auditor?

In the short-medium term LLMs will generate bespoke audit plans based on manual input from the audit manager about the audit risks and areas of focus. LLMs will then execute most (if not all) of the audit procedures, with a human in the loop to verify certain aspects and continue training the models through manual reinforcement (RL).

In the longer term, 5-7 years, LLMs should be able to identify the audit risks once these models will have the business context, wider datasets, human feedback (RL), and expectations from the different stakeholders. LLMs will only get better over time and a key element to help them improve is data. Validis provides this data – standardised across many industries, which improves the entire process and enables the adoption of LLMs in a much shorter cycle based on structured and unified data.

As with any other technology in other industries – LLM will push auditors further from the ‘How’, and focus much more on the ‘What’ and ‘Why’.

Will we still need humans in the loop and for what? The answer is yes, as it would likely take 5-10 years to train LLMs to address the more complex use cases above. And even then, auditors will be needed to continue developing and training new models.