Taking Analytics from Audits to Investigations

Beyond CAATs

Auditors embraced data analytics back in the 1980s to transform the way information was used to test and verify business processes. My experience in data analytics over the past 20 years has included both audit tests as well as investigative questions. I started out on a “CAATs Team” that served both groups, even though the acronym “CAATs” or “CAATTs” means Computer Assisted Audit Techniques or Computer Assisted Audit Tools and Techniques. The relationship between auditors and investigators when it came to using analytics to collaborate wasn’t strongly established.  Occasionally an audit would test some controls and find some weird results. Auditors might send these over to a criminal investigator colleague of theirs to take a look.

How did this Auditor-Investigator data analytics collaboration originally develop? It started with the auditors. Back in the 1980s, it was not common for auditors to use data analytics as part of their work. They might define the entire universe of what they want to audit, then select a sample, then review backup documents related to the sample. As an example, if they are reviewing the Purchasing process in their organization they would perform a manual process.

Audit Review of Purchasing

• Define the Universe – A standard audit could include the universe 5,000 Purchase Orders issued during the prior year.
• Select a Statistically Valid sample – Based on the errors discovered in their prior audit work, perhaps they knew there was an error rate of 5% and they wanted to operate under a confidence level of 95% margin of error. They would then select 357 of the Purchase Orders to manually review, possibly comparing the purchase order total dollar value with the total dollars invoiced and paid amounts of all bills received against that purchase order. 

• Review Backup Documents – The auditors would need to track down original documentation for the 357 Purchase Orders. This was quite a manual process even if the auditor was lucky to have a centralized storage system located locally in their primary work office. And how would they decide how to physically select these 357 purchase orders, pull every 14th document in the filing cabinet? Hope that all of the purchase orders were physically available and filed in order and choose the ones numbered with a multiple of 14, documenting if any were missing? Carolyn G, my favorite statistician, with whom I worked early in my career, told a story of auditors taking a ruler with them and pounding carpenter’s nails into it an inch and a half apart. They would go to the file drawer they needed to audit, slide the nails into the files, and pull the documents that were identified this way.  You could say they nailed the control anomaly test sampling process. After finding the 357 purchase orders, they would hope that a diligent clerk had properly filed all related vendor invoices and copies of remitted payments with each purchase order. Then they could manually total up the dollars and compare.

Transition to Electronic Records

Companies started to electronically store purchase orders, invoices, and payments. This changed the activity of auditing. No longer limited to a manual process, auditors were now able to incorporate computer techniques in their work. For me in the 1980s, I was just starting to learn how to code in BASIC on my Commodore64. However, other pioneers in the data analytics world were busy using personal computers to make the job of auditors both easier and more thorough. Two of these pioneers that I read the most when I was getting started in data analytics were David Coderre and Richard Lanza. Instead of physically/manually/awkwardly pulling a sample, these guys applied Computer Assisted Audit Tools and Techniques to perform a computerized process.

Computer Assisted Audit Techniques Review of Purchasing

• Define the Universe Electronically – Identify a spreadsheet or electronic data file that stores information about purchase orders. Find other files that stored the invoices and payment details. 
• Skip the Sample and Perform 100% Review – Run a straightforward SUMMARIZE command to total up all invoices and payments for totals per purchase order. JOIN the results of the Invoice Summary and Payments Summary to the list of Purchase Orders. Compare the TOTALS columns, perhaps adding a computed field that shows the difference between the authorized PO amount, the total invoiced, and the total paid. This results in a 100% review of the 5,000 Purchase Orders mentioned earlier. Additional analytics could be applied that identify UNMATCHED payments that don’t even apply to a Purchase Order but were paid anyway.

Computerized reviews became easier and more thorough as auditors applied these computer-assisted analytical techniques. Today, most audit shops use computerized techniques from MS Excel to specialized software such as SAS, ACL/Galvanize/Diligent, Arbutus, IDEA, or Alteryx. Some are even increasing their sophistication and are starting to use Python or R to apply advanced statistical methods and machine learning to their data and detect trends that a traditional sample might not reveal.

CAATs for Investigators?

Investigations teams were slower to adopt this “audit focused” computer assisted technology to their own fraud detection efforts. Some auditors make an effort to share weird anomalies with their investigator colleagues. Many times, however, by the time the audit takes place and finishes communicating the findings, the data tends to be stale. A reasonable “quick audit” might last eight months from beginning to end. An audit team could spend two months planning, four months performing fieldwork, and two months compiling the audit report. In an ideal world, the audit team may define their scope to include recent data, perhaps from the prior year before the audit starts. This means that for an audit that starts in January that focuses on the prior year’s data, the data is 8 – 20 months stale by the time the audit is complete. If any anomalies are identified during the audit tests and immediately referred to investigations, they are still stale as they come from the prior year as defined by the audit scope.

Different Goals

Investigators are the most impactful if they are able to find potential fraud as it happens. They cannot depend on stale referrals from completed audits. With an investigative dashboard that is connected to live data, they can see anomalies that exist as they occur, removing the latency for the potentially fraudulent activity.  Additionally, the queries don’t need to be based on audit techniques of testing internal controls but can be tailored to specific queries that seek out and identify suspicious individual behavior. Audits and Investigations have different goals:

Investigative dashboards are designed with the detection of fraudulent activity in mind. Instead of only identifying a list of potential duplicate invoices, the totals can be rolled up by invoicing clerk to draw out exceptions based on employees. An auditor may set a materiality figure of $10,000 for an invoice of concern, but an investigator might be interested in many small-dollar amounts that roll up to the vendor or employee level. Connecting to live data and refining analytic questions with an investigative perspective ensures that investigators can identify emerging threats quickly.

Computer Assisted Techniques for Investigators

Today, investigators are expanding the ways that they use data analytics to change how they identify high-risk activity. An excellent example of creating and using dashboards comes from the United States Postal Service – Office of Inspector General (USPS-OIG). They build dashboards that are “letting investigators drill down for details where suspicious trends are shown.” This drill-down customization of an inquiry can happen during the individual user session of the dashboard. USPS-OIG has been empowering its investigators with data since the early 2000s. But if an investigator doesn’t have a dashboard already, how can they get started without asking their auditors for the results of completed audits?

Where Can Investigators Start?

Investigators should start with the information they know and are most comfortable with.  If you are an investigator and are familiar with issuing subpoenas to retrieve phone records, start there. If you commonly pursue healthcare fraud cases, identify the source of your referrals and look at the universe. If you have been tasked with sorting through all of the Paycheck Protection Program (PPP) Loans in your geographical area and can’t imagine how to thoroughly sort through it, that is your starting point. Does your organization worry about how employees might be abusing their Purchasing Credit Cards? Get that data!

For smaller data sets, you can use the tools currently at your disposal such as MS Excel to profile the data. Does your organization have MS PowerBI as part of their Microsoft365 implementation? Could you try out Tableau Public? Try creating a simple visualization showing total costs per person. Identify your data, capture it, display it, and make a plan to investigate further. Keep identifying and investigating weird anomalies, adjusting to find true instances of fraud. Then, refer the common issues over to audit to test the controls and find the root cause of the problem, shutting down the fraud opportunity.

Keep Taking the Next Step

It has been a few decades since data analytics started being applied to audits and investigations. Techniques are changing and developing, much like my programming skills since the 1980s. It’s up to you to figure out where you are in the analytics process. Then take the next step to grow your understanding. Use the data you know to expand your insight beyond a manual review of records. Then do it again, take the next next step. Data Analytics and “Computer Assisted Assessment Techniques” are for both Auditors and Investigators!