Skip to content
The methodOpen data
Back to the collection

Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc

Doug Lenat, Gary Marcus

2023 · arXiv

Language models
Read the paper
HN score
0
Mentions
7
Readers
6
Months
4
What’s in a score?

From the abstract

Generative AI, the most popular current approach to AI, consists of large language models (LLMs) that are trained to produce outputs that are plausible, but not necessarily correct. Although their abilities are often uncanny, they are lacking in aspects of reasoning, leading LLMs to be less than completely trustworthy. Furthermore, their results tend to be both unpredictable and uninterpretable. We lay out 16 desiderata for future AI, and discuss an alternative approach to AI which could theoretically address many of the limitations associated with current approaches: AI educated with curated pieces of explicit knowledge and rules of thumb, enabling an inference engine to automatically deduce the logical entailments of all that knowledge. Even long arguments produced this way can be both trustworthy and interpretable, since the full step-by-step line of reasoning is always available, and for each step the provenance of the knowledge used can be documented and audited. There is however a catch: if the logical language is expressive enough to fully represent the meaning of anything we can say in English, then the inference engine runs much too slowly. That's why symbolic AI systems typically settle for some fast but much less expressive logic, such as knowledge graphs. We describe how one AI system, Cyc, has developed ways to overcome that tradeoff and is able to reason in higher order logic in real time. We suggest that any trustworthy general AI will need to hybridize the approaches, the LLM approach and more formal approach, and lay out a path to realizing that dream.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

musicale
Neutral mention · rule
See also (Lenat's final paper): D. Lenat, G. Marcus, "Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc", https://arxiv.org/abs/2308.04445
Original comment Obituary for Cyc
trott
Neutral mention · rule
> Marcus has always been a mouth just trying to take down neural networks. This isn't true. Marcus is against "pure NN" AI, especially in situations where reliability is desired, as would be the case with AGI/ASI. He advocates [1] neurosymbolic AI, i.e. hybridizing NNs with symbolic approaches, as a path to AGI. So he's in favor of NNs, but not "pure NNs". [1] https://arxiv.org/abs/2308.04445
Original comment LLMs have reached a point of diminishing returns
nickpsecurity
Neutral mention · rule
To parent: the spatial reasoning and blind person were great counterexamples. It still might be OK despite the blind exceptions if it showed general reasoning. To OP: I like your project goal. I think you should look at prior, reasoning engines that tried to build common sense. Cyc and OpenMind are examples. You also might find use for the list of AGI goals in Section 2 of this paper: https://arxiv.org/pdf/2308.04445 When studying intros of brain function, I also noted many regions tie into the hippocampus which might do both sense-neutral storage of concepts and make inner models (or approximations) of external world. The former helps tie concepts together through…
Read the full comment
various senses. The latter helps in planning when we are imagining possibilities to evaluate and iterate on them. Seems like AGI should have these hippocampus-like traits and those in the Cyc paper. One could test if an architecture could do such things in theory or on a small scale. It shouldn’t tie into just one type of sensory input either. At least two with the ability to act on what only exists in one or what is in both. Edit: Children also have an enormous amount of unsupervised training on visual and spatial data. They get reinforcement through play and supervised training by parents. A realistic benchmark might similarly require GB of prettaining.
Original comment ARC Prize – a $1M+ competition towards open AGI progress
Rochus
Neutral mention · rule
A very interesting review by John F. Sowa (https://en.wikipedia.org/wiki/John_F._Sowa) of the Cyc system as presented in Lenat's 1989 book (ISBN 0-201-51752-3) and his 1991 presentation at IBM Research in Yorktown. > In June, Doug sent me an article he was working on That's likely this one: https://arxiv.org/abs/2308.04445. See also https://garymarcus.substack.com/p/doug-lenat-1950-2023.
Original comment Lenat and Cyc
PaulHoule
Neutral mention · rule
One thing I thought was amusing was that there was a burst of articles about Cyc that got mentioned when Doug Lenat died including this arXiv paper https://arxiv.org/abs/2308.04445 and that one said that Cyc had over 1,100 special purpose reasoning engines. The general purpose resolution solver was nowhere near fast enough to be really useful. Early one there was https://en.wikipedia.org/wiki/General_Problem_Solver which would be capable in principle of finding a winning move in a chess position but because it worked by exhaustive search it would practically take too long. The thing is that a good chess playing program is not generally intelligent just as a chess…
Read the full comment
grandmaster isn't necessarily good at anything other than chess, it just has special purpose heuristics (as opposed to algorithms) that find good chess move. ChatGPT-like systems will be greatly improved by coupling them to other systems such as "write a Python/SQL script then run it", "run a query against bing and summarize the results", and "go find the chess engine and ask it what move to make", that is, like Cyc, it will get a swiss army knife of tools that help it do things it's not good at but it doesn't create general intelligence any more than Cyc did. Robert Penrose in the Emperor's New Mind suggests that there must be some quantum magic in the human mind because the human mind is able to solve any math problem whereas any machine is limited by Gödel's theorem. It's silly, however, because we don't humans are capable of proving any theorem: look at how we struggled with Fermat for nearly 360 years or how https://en.wikipedia.org/wiki/Collatz_conjecture seems not even tantalizingly out of reach. The difference might be that humans feel bad when they get the wrong answer whereas ChatGPT certainly doesn't. (as much as its empty apology can be satisifying to people) This isn't just an attribute of humans, working with other animals such as horses I'm convinced that they feel bad when they screw up too.
Original comment Recursively Summarizing Enables Long-Term Dialogue Memory in LLMs
PaulDavisThe1st
Neutral mention · rule
https://arxiv.org/pdf/2308.04445.pdf "Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc" Lenat's last paper (July 31st, with Gary Marcus) https://news.ycombinator.com/item?id=37354601 This may disabuse you of two ideas: 1. that NN models (LLMs) exhibit common sense reasoning today 2. that the approach to AI represented by Cyc and the one represented by LLMs are mutually exclusive
Original comment Doug Lenat has died

Keep following the thread.

More in Language models

More from the same topics.