Introduction

The video above shows a Blazor application querying a locally hosted LLM running on decade-old consumer hardware. The server exposes endpoints via FastAPI and Uvicorn.

It's a small recruiter-style question-and-answer domain, running on a 13-year-old AMD A10-6700 quad-core machine with 12 GB of DDR3 RAM. Fortunately, an NVIDIA GTX 980 Ti with 6 GB of VRAM makes it possible to run a 3B model. Despite the modest hardware, the system is responsive enough for experimentation.

Kubuntu Linux is used for development, and Ubuntu 24.04 Server hosts the local AI.

Hallucination

Small 3B models are useful, but they are weak in certain domains, especially when asked to answer from ambiguous prose or make judgement calls.

Originally, I developed classifiers, routing logic, and engineered prompts to steer the RAG system toward better-grounded answers. Over time, that grew complex and brittle, and the weaker model began to hallucinate. And the Python scripts had started to balloon in size.

I then started exploring more explicit representations of knowledge and deterministic deduction alongside RAG. I structured the knowledge more formally, and explicitly prompted the model to use straightforward deduction rather than guided inference where possible:

  • no classifiers
  • minimal prompt engineering
  • no fuzzy interpretation of prose
  • deterministic, explainable answers

This seemed to work better; the next step was to introduce more content without any regressions.

I also tweaked several parameters to improve response times and moved up to the Q6_K model with all-mpnet-base-v2 for embeddings.

Adjacent to Intelligence

This experience has led me to the opinion that AI is not intelligent in principle.

It appears to be more like the cogitative power, adjacent to intelligence, concerned with pattern recognition among particulars, recombination, metadata and tagging. It's like a global cogitative power augmented by formal logic and mathematics.

Consequently, its capability is tied to scale: massive compute power over vast amounts of data. This makes it dependent upon cheap energy and resources.

It is incapable of simple apprehension.

Yet, given sufficient resources, its capabilities are staggering.

Other Approaches

I suspect that this kind of problem maps very naturally to logic systems like Prolog.

In fact, for pure factual questions — Did you use X in role Y? — a logic engine would likely be faster, more accurate, and completely deterministic, with the model using the results to present the answer in English.

Ironically, AI could generate that Prolog code faster than I could.

Another approach I'm considering is using the LLM for semantic mapping — reducing questions to a known vocabulary so my own intent-handling system can do the deterministic work.

I'm keen to get hold of better hardware, try out stronger models, and compare results.

The intersection between knowledge representation and AI agency is particularly interesting to me.

Closing Thoughts

I appreciate that AI provides an opportunity to work at a more abstract level, which I very much enjoy.

This was a fun experiment. The AI journey has begun, and it will be interesting to see where it leads.