Need to build an Expert System

Currently, we have a small team of accountants/bookkeepers whose occasional job it is to carefully review a merchant's credit card statement. The purpose is to find and add up all the charges on that month statement, as many are hidden and hard to decipher. Once the total is found, that amount is divided by total gross credit card sales to find the monthly aggregate transaction rate.

This number can be low, like 2.55%, but can be very high, particularly when sales are low. Monthly fixed charges can cause slow months to cost as mush as 10-12%!

Anyway, currently, experts do this analysis, with an eagle eye developed over decades of doing this. They support over 40 different report types, so they need to make judgement calls, based on their collective experiences. I'd like to harness these skills into a system which can provide instant turn-around for this entire effort.

At first, I would have those same experts be the sole users of this system until they find zero errors for a prolonged period of time and/or for particular report formats.

I am looking for a basic idea of the best technologies to use for this program. I am a C#.NET developer with javascript and some node experience. But I would switch to an entirely different platform, particularly if open sourced frameworks already existed or there were some other advantage of a specific platform.

A key element of this process is pattern matching. For example, in theory, each line item in a merchant's credit card monthly statement matches an item in the Interchange table. When there is an exact match, this would enable a validation that the rate charged for that line item matches the rate allowed by law.  This also enables the "expert system" to have a high confidence the match is correct.

Conversely, there are some companies that use "monkey business" when creating their reports, seemingly confusing their merchants intentionally. They change the names of these line items entirely, to obscure them, making it harder (or impossible) for a non-expert to audit that report.

There are fixed published lists which could be consumed every time a new version comes out...

Interchange Table
Interchange Table2
And the system would correlate items from the monthly statement to those line items...

Report Sample
So, I need to get a very high level explanation of the kinds of tools I could use for this expert system. There would be heavy use of pattern matching. Would this be simply regular expressions? Or would this require Python or some other language?

And for the training portion, I presume when a match is accepted by an expert, that association would be logged and potentially be a match just for a single credit card processor. It seems they each have their own "language" when they depart from the conventional names found in the Interchange Table.

It will be a web based technology, which does help guide us to the proper choice of this back-end logic needed for this.

Thanks for whatever help you can suggest.

And about about a Rules Engine? Isn't that something at the core of an expert system?

newbiewebSr. Software EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Seems like any scripting language will do.

I've written a similar system using PERL + tools to convert .pdf statements into text.

If you start your project by hiring a developer, they'll choose their language.

If you specify your required language up front, this will set the type of developer you hire for coding.

Tip: Avoid incompetent/hack coders who try to sell you on some sort of Web based system. This type of system is very difficult to debug.

When a client hire's me for this type of work, I use PERL + include writing a set of unit tests, so at any given moment the entire code base can be tested to ensure, no enhancement has broken the entire system.

When you're interviewing developers, ask the simple question, "How do you verify code quality?"

The correct answer is "I don't. That's what unit testing is for, to make sure I didn't destroy the entire code base by some enhancement update I made."

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
newbiewebSr. Software EngineerAuthor Commented:
I am the first coder on this project ;)

I am willing to switch languages, in order to find the best, most productive one for this job.

I am a C#.NET full stack developer, and COULD consider C# since Entity Framework may bring major value to the table. But I suspect PERL may be better.
 But, when building an Expert System, isn't a Rules Engine at the heart of that system? And if that's the case, aren't there existing framework for Rules Engines, so I do not need to reinvent the wheel? And what about "Decision Trees?"

Are there any existing frameworks that could help? How about Azure? Does this cloud database provide any configurable logic that might be usable?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.