Factors to consider when comparing AI knowledge management tools

By now, large language models have gotten really good at remembering our private conversations and preferences. But have you ever noticed how the same tools at your company tend to make up what they know? Let me explain why this happens, and the factors to consider when comparing AI knowledge management tools for your team.

Where does the gap come from?

Tomorrow someone turns on the requirement to enter a credit card for free trials. The person who clicked the switch in Stripe knows they did it, while everyone else on the team finds out only if that person writes the change down.

You would know it if you made the change, or if a close teammate told you, and a personal chatbot often knows it too, because you talked the decision through before you settled it. AI tools for knowledge management are good at finding that answer once it is in Slack or on a page, but if the fact never landed anywhere the team shares, the tool still answers, and it makes one up. That's why capturing information is the most important aspect of knowledge management with AI.

On your own, the answer is already somewhere: in your head, in a note, or in the chat you used to decide. A group does not remember on its own, so someone has to record the change, and that gets harder as more people are involved. On the Monday after the Stripe change, support searches the handbook and finds the old page, which still says the trial is free. The person who clicked the button had worked the decision out in a private chat that support never saw, so the search box writes an answer from those older pages, and the customer hears last month's rule in a confident voice.

That's why I always ask myself what the primary objective is with AI knowledge management tools. Is it to capture the knowledge, or to search through different systems to find the answer?

Three factors

Since the AI boom, I have become very cynical about AI knowledge management tools. Whenever someone suggested the rollout of a specific tool, I always looked at these three points to tell whether it was actually going to help us long term, or whether it was going to be another example of "Someone changed the settings in Stripe, but no one knows who or why."

1. Capture or search. Some tools record the new fact. Others can only search what is already in Slack or on a page. The second kind often lacks the actual knowledge that it needs to answer you accurately.

2. Who provides the fact. A model that drafts the page will fill a hole with a likely sentence, so a person has to state the fact, and the tool can help organize it.

3. The person who made the change will record it. If the primary objective is to record knowledge, not just to give it back, then this objective has to be so easy that people will actually do it.

Here is what I mean by each one.

1. Capture or search

I spent eight years as a software engineer, then worked in product leadership, and the wiki was already there in every company, whether that was Confluence, Notion, or a shared drive with three copies of the same handbook. Search has gotten very good, but the pages behind it have not improved.

Teams treat documentation the way they treat the office kitchen. A few groups clean up after themselves, but most do not, even if you put a sign on the fridge. Documentation is that same chore: most people agree that documentation is necessary, but if the person who made the change is already late for the next meeting, when is it going to get done? In my opinion, a wiki is still the best place to gather knowledge, if people maintain it. The failure is that one group never opens the editor, while the other keeps searching for answers.

Newer products try to get around that by connecting to many different sources within your company, such as Slack, Notion, Jira, your support tool, ticket queue, and other production systems. They try to solve the problem that people don't like to write by assuming that knowledge changes as people type, and that those messages are the new truth. The product recognizes questions and answers in Slack, and updates the documentation from them.

This means the knowledge is not really captured. It is interpreted from search. In my own categories, that counts as search rather than capture. You capture company knowledge when someone explicitly states how things work, and updates that statement as it changes.

2. Who provides the fact

If your primary objective with an AI knowledge system is to capture, the next question is who provides the knowledge.

I have watched teams keep docs in a markdown repo and use GitHub Copilot to summarize code changes into that repo. Generating from the code sounds fine until you realize why knowledge management is so important. The question is often not just what the state is, but why. Only the person who made the decision can answer that.

By now the support agent can see that the team requires a credit card for free trials. They still don't know why. That a card is required is obvious to everyone: open the app and you see it. What makes company knowledge valuable is the lesson you can share, so the team does not make the same mistake twice.

The reason the app now requires a credit card for the free trial is that they noticed fewer conversions, but a higher intent to upgrade, when trials are only offered to people who enter a card. That reason would not appear if an agent created the knowledge base. If a large language model summarizes the code changes, the documentation will say that a credit card is required to start a free trial. It will not say why. The why is what helps the team avoid the same mistake twice, and avoid spending hours deciding whether to revert.

One more thing about the GPT architecture these systems use when they generate knowledge on their own. That architecture predicts the next token from public text it has already read. Those predictions are statistics, and a knowledge system should not answer with the statistically likely line. It should answer with exactly your decision.

3. The person who made the change will record it

Once you have found a system where AI supports people in capturing what they know, one question remains: what else do you need for them to actually use it? You are back at square one, and the kitchen is still dirty. Writing the page does not take a few minutes. It takes tens of minutes, sometimes hours: finding the right place, thinking about the wording, inserting it, formatting it, grabbing a coffee in between, and publishing only when you are finally happy.

I believe recording knowledge should be simple and fast, because the objective is to capture knowledge, not to write a book or become an editor. When the loop is slower than mentioning the change, people mention it and move on. The page stays wrong, and a month later someone blames culture for a chore that lost to a ten-second conversation.

Examples of AI knowledge management tools

So you ask what actually works. Here are some examples of these three factors. Most of them are very good at one job, so keep your primary objective in mind.

A traditional wiki with AI features

Confluence, Notion, and a shared drive became extremely good editors and have very good AI agents that help you find information that already exists. Their biggest disadvantage, in my opinion, is that they expect you to be not only the information provider, but also the editor and the writer. That is why so many people never actually write the documentation. Not because traditional wikis aren't good enough, but because people do not want, or do not have time, to be a writer and an editor when they only wanted to share information.

Tools that read Slack, tickets, and the drive

Tools such as Slite and GitBook use connectors to pull code, messages, closed tickets, and files into one index. That is impressive when the fact was already typed, such as a Friday support reply the tool can surface on Monday. The index still holds only what people wrote in other apps, so a Stripe click never enters it, and the product answers anyway. Treat that answer as search over a wider archive. I wrote about search itself, including where Cortee is still weak, in What to Look for in IT Documentation Software.

A chatbot that remembers one person

ChatGPT with memory works so well because you captured the decision without noticing, talking it through and correcting it until the thread became the record. Your team does not share that thread, so a company full of private chats is a set of personal notebooks, none of which is the page that support opens. If a buying committee says the tool already knows your context, ask whose, because a month of one person's chats leaves everyone else with the same gap.

A short note on the page people already open

This is my personal recommendation. It is an emerging category where people lose the roles of writer and editor, and simply provide information to share. It does not rely on AI to create the knowledge. It uses AI to speed the process up.

That's what Cortee is built for. You type ordinary notes into a composer, which then updates and edits the documentation. You get the benefit of AI formatting and writing what you tell it, and you stay in control. Humans provide the knowledge. The AI processes it and makes it accessible to everyone.

How to close the gap

When you choose among AI knowledge management tools, start with the objective. Better answers from documents you already have is a search purchase. Knowing what changed this week is a capture purchase. I then decide who provides the information. There are cases where the model is good enough. I want people to provide what they know, and I want the AI to manage it.

Give the systems you are considering a week or two. Keep the one that turns the kitchen into a click rather than an hour, on a day when someone is already late to the next meeting. If that is the objective you have, try Cortee.