We use cookies to understand how you use our site and improve your experience.
By Novalytics Team
AI tools can now read a document and answer questions about it in seconds. That speed makes it tempting to ask whether a general-purpose LLM is all your team needs.
In other words, can we just use ChatGPT, Claude, or Copilot to pull the data points we care about?
A general-purpose LLM is not a data platform. It answers questions. Your analysis needs a system that collects, stores, and defends every data point, across multiple sources.
We evaluate extraction performance with Novabench, our extraction evaluation dataset containing several thousand data points. In our own test analyzing a document with 200 data points, Novalytics answered about 90% of questions correctly. Copilot got only 50% of them right. Claude and ChatGPT answered about 75%.
On speed, the LLMs we tested needed about 10 to 20 minutes to work through one document. Novalytics pulled 200 data points from 30 different documents at once, within 5 minutes.
General-purpose LLMs are versatile assistants. They draft text, summarize documents and answer quick questions.
Custom data point analysis is a different job. It means pulling the exact figures and answers out of many sources. It means comparing them over time. And it means defending every number to the people who rely on it. A chat window of a general LLM alone can't do that.
A general LLM is a smart assistant. Novalytics is a full system built around AI: a database, quality checks, security and ongoing upkeep. It is designed for teams that turn data into decisions, including sustainable investment teams.
Here are six places where the gap shows.
General-purpose LLMs: They can state a wrong number with full confidence. They have no built-in way to check their own work. Your team has to catch the errors. In our own testing against a real document, Copilot got about 5 out 10 answers right while Claude and ChatGPT got 7 out of 10.
Novalytics: Data quality is tested on an ongoing basis to catch mistakes before they reach you. That includes made-up figures, known as AI "hallucinations." In the same testing, Novalytics got 9 out of 10 right.
General-purpose LLMs: There is no database behind them. Each chat starts fresh. You keep prompting to pull out data a little at a time. Your teammates may not get the same results you do. Chat tools often limit both the number of fix/les you can add at once and the size of each file, which constrain analysis.
Novalytics: It builds a database for your organization. Every document and data point is stored with its history. Through the Novalytics chat feature, you can query your database and build on your own analysis.
General-purpose LLMs: Without a structured workflow around the model, the same question can get a different answer each time, and a confident answer can still be wrong. There is no audit trail out of the box.
Novalytics: A repeatable, structured process gives you the same answer format on every run. The process is auditable, so you can see how an answer was produced.
General-purpose LLMs: Security depends on how you set it up. Your team decides who sees what, which tools connect and where files go. One wrong setting can expose sensitive data.
Novalytics: Your documents and data points are encrypted. Novalytics does not train its models on your data.
General-purpose LLMs: If you build your own agent, you own the testing, monitoring and upkeep. AI models and tools change often. An update can change how your agent behaves. Someone has to own it for the long term.
Novalytics: We handle updates, testing and support so your team can focus on the work that actually moves the needle instead of searching for data.
General-purpose LLMs: You pay by usage. Building an agent on the underlying model means per-token billing (you pay for every small chunk of text it reads and writes), on top of any per-user seat fee. Copilot Studio charges by the "credit" each time an agent answers or acts. The bill moves with how much you use it. Vendors can also change their prices over time, so the cost you plan for today may not be the cost you pay next year.
Novalytics: A flat annual contract includes unlimited usage and fixes your price for the term. There are no surprise charges from usage-based billing.
Decisions depend on data. If you can't trace a number to its source, or you get a different answer each time, you carry the risk.
Doing this work by hand takes hours. A general-purpose LLM cuts one document down to about 20 minutes, but it handles one document at a time. Incorrect answers and outputs that don't follow the required format still need manual review. A faster response doesn't remove that checking work; it can simply move it later in the process.
Every manual check, batch upload and surprise bill is a cost of using a tool that wasn't built for the job. Copilot, Claude and ChatGPT are strong assistants. Novalytics gives your team the data foundation.
Ready to see it in action? Book a demo with us!