World Models

of higher level understanding about how the parts of a human "fit together". what practical experience do these models have that could actually convey such an understanding? deriving a proper understanding of mechanics in 3D from one million independent 2D still frames of human hands performing various tasks seems like it should be extremely difficult at best > Could this be trained in? I expect so, but I think it would require multiple engines I think it requires a different sort of training algorithm entirely. work such as https://arxiv.org/abs/1803.10122 suggests to me that there might be little difference between the human ability to abstract and lossy compression. at the same time work such as https://arxiv.org/abs/2205.11502 makes it apparent that in many cases this sort of generalization simply does not happen the way we'd like > the neuron/synapse/neurotransmitter and brainstem/midbrain/cerebellum micro & macro-architectures are vastly different than the computer training models. So, I think we can be confident that something different is happening something being architected differently doesn't necessarily mean that the higher level functionality is any different moreover, in purely functional terms how do you propose to distinguish something that's different from something that's incomplete? ie a smaller piece of a larger whole? if someone constructs for example a passable digital model of the visual cortex of the mouse or human or other animal that's still only a single small piece of the whole so who is and how are we to say that we either have or haven't achieved a meaningful form of abstraction versus merely averaging bits of the training set together? at this point I'm not actually clear where the line between those two things even lies

("latent" version) -> intelligent actor -> muscles -> affects world. Now what everybody thinks of as a simulator is something that simulates the whole thing. But if you insert one more link (output of reasoning agent at time T -> simplified representation at time T+1) you can then run a "simulation": random simplified version ("what if your car became a tree and fell ?") -> intelligent actor -> next input for latent representation ("then what happens ?") For safety reasons, it is probably prudent to disconnect the muscles in this state. You know, so you don't knock out your mother when you dream about boxing. And as you say, this "predict the future" network is probably useful by itself in dealing with the world. So you can catch tennis balls and the like. https://arxiv.org/abs/1803.10122 Or, if you like: https://www.youtube.com/watch?v=dPsXxLyqpfs