How to get accurate AI answers from ERP data

AI tools give accurate answers from ERP data when they use shared definitions of industry terms like active customer and products margin, work from connected data, follow user permissions and show how the answer was reached. Generic AI tools connected straight to ERP tables often skip these steps, so wrong answers look like right ones.

A recent Phocas State of Sales in Distribution survey found that half of the distribution respondents are testing general-purpose AI tools like ChatGPT and Claude on their own, outside of any business system. It is important to know why a powerful generic AI tool can still be a wrong one and some of the important considerations of handing over your data to an AI engine.

Why generic AI tools get ERP numbers wrong

AI can connect to ERP tables or data cubes today and probably assumes that everyone in a business already agrees on data rules and how various transactions are handled. The ERP data is the authority at the transaction level because it's the most accurate record a distributor has. However, ERP providers and their BI development teams internally or externally know the data only becomes trustworthy for analysis once it's been layered and validated so it can be queried consistently across departments. This is an important step most generic AI tools skip or many IT managers learn the hard way. They can connect a system to an ERP but joining the data between systems is much harder and requires collaboration across the business.

When using generic AI for data analysis you need to tell it exactly what you want. You also need to know what a right answer looks like well enough to spot a wrong one and understand the business logic behind the question you're asking.

We know people share prompts and get AI guides to help them use generic AI tools but using Claude for data analysis in a distribution business is starting from scratch. The best prompts are not usually specific to distribution businesses and often don't know the intricacies of let’s say, an auto-parts margin definition. If ‘margin’ means something slightly different in your inventory system than it does in your finance reports, a skilled prompt engineer and an unskilled one will both get an answer, probably both wrong.

What accurate AI answers from ERP data depend on?

Shared definitions: Clear, agreed meanings for measures such as revenue, gross margin and available stock, so AI applies the right business rules.
Connected and validated data: Reliable ERP and other business data that is checked for errors, gaps and consistency.
Permissions and security: Access controls that ensure answers use only the data each person is authorised to see.
Links to transactional data: Clear explanations of the data, calculations and filters used and a confidence score for the result .
Human review: A person checking important answers against underlying records and business knowledge before acting.

How to test a conversational AI analytics tool for accuracy?

The best test of a conversational analytics tool is whether it gets your business numbers right. Start with 10 questions you already know the answers to from trusted ERP reports across sales, margin and inventory. Ask the AI tool each question and compare its answer with the ERP figure, using the same dates and filters.

Then ask the same question in two different ways. The wording can change, but the numbers should stay consistent. Test a definition of a metric specific to your business, such as margin after rebates, to check whether the tool applies your rules correctly.

Finally, log in as a sales rep whose access is restricted to their own accounts and check that the tool respects those limits. A correct answer must use the right calculation and only the data that person is authorised to see.

Where validation agents help and where they fall short

Some IT teams in distribution build a data validation agent to help spot problems. This is a second AI step that checks the first answer before it serves up the final figure. It's a sensible practice and can catch real errors like a wrong data join, a stale export or a percentage applied to the wrong number.

A validation agent checks whether a number is internally consistent but is not so good on your specific business rules. The AI doesn't know if active customer means what your sales team agreed the term to be twelve months ago. The person typing the original prompt often doesn’t know a validation step exists at all, so they don’t activate it. To make AI work when you build it inhouse comes back to getting the AI process or rules consistent across your business for all the workers as well as having shared and governed definitions for your AI tool to check.

Why connected data matters for distributors

Connected data is the difference between a distribution-specific AI tool and a generic AI tool because it gives AI the context to understand distribution. In a recent Distribution Strategy Group survey about the State of Technology in Distribution found AI implementation problems usually trace back to data and connectivity.

AI produces reliable business understanding when it's built on mapped definitions, governed data, historical context and industry logic rather than just a live connection to raw data tables.

Mapped definitions ensure that margin, active customer and sale value mean the same thing on every dashboard and in every AI answer, because they are defined for all use cases. Governed data ensures the same customer, product and account across your ERP, CRM and e-commerce platform is one record. Historical context means a slow-moving SKU or a growing account is judged against your business’s own trend or target. Industry logic gives the AI model distribution-specific data, so it understands your product mix or business model.

For example,  run another test to determine whether the AI gives you the same margin the finance team would sign off on.  At month-end, ask the distribution-specific conversational AI tool, “What was the gross margin for September?” The answer should use the company’s agreed treatment of returns, freight and supplier rebates, and match the ERP report for the same period.

If the ERP shows 24% and the AI returns 27%, the difference could be whether rebates are included. Checking the business definition matters as much as checking the maths.

With connected ERP data and business rules configured, you can then ask, “Does that include supplier rebates?” then, “Which branches contributed most to the decline?” The tool should retain context, show the calculations and underlying transactions, and respect each user’s permissions.

How Phocas approaches accuracy

Phocas approaches conversational AI accuracy by working from connected, validated data within each user’s permissions, while keeping customer data out of public model training.

“We've invested a lot of time so that our customers get the same results using conversational AI as they would with the classic query method,” says Blair Cassidy, Chief Product and Technology Officer.

Phocas’ ERP partners are adding strong built-in AI capabilities, and they would agree these advances don’t replace the need for analytics and connected data. The only instance would be for the ERP super user who knows the data and knows exactly what question to ask, maybe then this form of AI can be a substitute for an extra analytics layer.

Phocas AI-powered Analytics and Sales Insights help everyday users access the same trustworthy numbers, with validation and updates built in. Users can drill down into dashboards and get what they need without learning every rule, because Phocas handles that work for them. AI needs to sit on top of the same trusted data foundation your business runs on rather than becoming its own separate silo when using generic AI. Phocas continues its distribution-specific game plan by connecting to industry ERPs and other systems. We uses definitions, history and governance already built in, so the AI answer and the ERP source are connected and accurate.