The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities

is not correct. Yes, it is, absolutely. That's the entire point of science, to understand the principles by which the world works so we can explain observations. Whether this is easy or hard to do, is another matter, and, again, we're not talking about the natural world here but about computers, that we have made ourselves. Nobody expects an airplane to take off on itself and start chasing birds in the sky. What is the reason to assume that Large Language Models are doing something they weren't made to do? >> You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!". Yes, but we don't make human brains. We make computers though and we know how they're made, and how they work. Thanks for the paper. I read the abstract, but it seems to me it's also talking about unpredictable behaviour, rather than some mysterious forces hiding in the depths of a language model's weights. This paper lists many more instances of unpredictable behaviour of machine learning systems: The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities https://arxiv.org/abs/1803.03453 I recommend it, it's an entertaining read, if nothing else.

rely on noticing, then post-hoc disincentivizing, unwanted behaviors, because we don't know how to robustly specify objectives. When you train a system to, for example, "stop saying racist stuff", without actually understanding what you're doing, all you get is a system that "stops saying racist stuff" when measured by the specific standard you've deployed. Ask any security professional how seriously people take securing a system, let alone how ineffective they are at it. Now consider the same situation but worse because almost no one takes AI safety seriously. If you nod solemnly at the words "safety" and "reliability" but don't think anything "really bad" can happen, you will be satisfied with a solution that "works on your machine". If you aren't deeply motivated to build a safe system from the start because you can always correct things late, you are not building a safe system. It will be possible to produce economically viable autonomous agents without robustly specified objectives. But hey, surely a smart enough system won't even need a robustly specified objective because it knows what we mean and will always want it just as much. Surely dangerous behavior like "empowerment" isn't an instrumental goal that effective systems will simply rediscover. Surely the economic incentives of automation won't encourage just training bad behavior out of sight and out of mind. Surely in the face of overwhelming profit, corporations won't ignore warning signs of recurring dangerous behavior. Surely the only people capable of building an AGI will always be those guaranteed to prioritize alignment rather than the appearance of it. Surely you and every single person will be taking AI safety seriously, constantly watching out for strange behavior. Surely pulling the plug is easy, whether AI runs on millions of unsecured unmonitored devices or across hundreds of money printing server farms. Surely an AGI can only be dangerous if it explicitly decides to fool humans rather than earnestly pursuing underspecified objectives. Surely it's easy to build an epistemically humble AGI because epistemic humility is natural or easy enough to specify. Surely humanity can afford to delay figuring out how best to safely handle something so potentially impactful, because we always handle these things in time.

(see e.g. the article that [1] is a comment on). [1] https://www.lesswrong.com/posts/Ez4zZQKWgC6fE3h9G/plausibly-... [2] Lehman et al., 2018, The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities, https://arxiv.org/abs/1803.03453

grey skies, or letting heuristics like "humans pick up dogs and cats" override correct identifications. (It's a goat until you pick it up, then it's a dog!) - An agent playing Q*bert found a known bug for infinite lives, then escalated to an unknown bug which disabled the game while overflowing the score counter. - Agents in a physics sim tasked with jumping as high as possible instead learned to 'fly' by abusing collision detection bugs, hitting themselves in ways that created upward momentum. - Another "maximize jump height" task demonstrated that "highest" is an extremely fuzzy term. Initially measured by highest point, they became incredible tall. Measured by lowest point, they stayed tall and grew topheavy to 'kick' their base upwards. - Number-handling bugs of all kinds. In one case, small twitches led to floating-point errors that created energy. In another, a "minimize force" task got solved by maximizing force and triggering integer wraparound. My personal favorite is an adversarial bug. An agent playing tic-tac-toe on an infinite grid with a time limit submitted extremely remote moves which caused timeouts/crashes in any agent that tried to model the full board. [1] https://arxiv.org/pdf/1803.03453.pdf [2] https://aiweirdness.com/post/172894792687/when-algorithms-su...

unexpected adaptations, or exhibiting outcomes uncannily convergent with ones in nature. Such stories routinely reveal creativity by evolution in these digital worlds, but they rarely fit into the standard scientific narrative. Instead they are often treated as mere obstacles to be overcome, rather than results that warrant study in their own right. The stories themselves are traded among researchers through oral tradition, but that mode of information transmission is inefficient and prone to error and outright loss. Moreover, the fact that these stories tend to be shared only among practitioners means that many natural scientists do not realize how interesting and lifelike digital organisms are and how natural their evolution can be. To our knowledge, no collection of such anecdotes has been published before. This paper is the crowd-sourced product of researchers in the fields of artificial life and evolutionary computation who have provided first-hand accounts of such cases. It thus serves as a written, fact-checked collection of scientifically important and even entertaining stories. In doing so we also present here substantial evidence that the existence and importance of evolutionary surprises extends beyond the natural world, and may indeed be a universal property of all complex evolving systems.

duckduckgo for "dembski no free lunch" gets me web pages that provide a critique of Dembski's book (and not on information theoretic grounds, either), and his book on Amazon, but nothing I can print and read and digest. No PDFs. How does Dembski wave off actual examples of artificial evolution? Tierra (http://life.ou.edu/tierra/) seems to evolve little digital pseudo-life that does a specific task. Here's another example: http://www.genetic-programming.com/jkpdf/alife1990.pdf There are more "folksy" reports, too: https://www.allegro.cc/forums/thread/293256 . There's an entire field, genetic algorithms/genetic programming, that actually ends up at least some of the time, evolving a solution to the problem. Adrian Thompson's FPGA experiment (https://www.researchgate.net/profile/Adrian_Thompson5/public...) is pretty famous. Here's a distillation of the above paper with a few interesting editorial comments: https://static.aminer.org/pdf/PDF/000/308/779/an_evolved_cir... How does Dembski explain how every system that attempts to "evolve" something by successive generations that inherit and have minor changes (https://arxiv.org/abs/1803.03453, which is a summary of many experiments) end up doing something that's undeniably evolution?

out how to drain the lake”), and doomsdayish in another (“it drained the lake”). The authors concluded that there is “potential for perverse outcomes from optimizing reward functions that appear sensible.” That’s researcher for ¯\_(ツ)_/¯ ...as Tad Friend wrote in his excellent piece in the New Yorker on this topic[2]. In other words, humans can’t safeguard AI systems solely by defining what they believe to be sensible reward functions. Reward functions, regardless of whether they are sensible to humans or not, critically need to be mediated by additional regulatory mechanisms, like hard-set non-goals that aren’t just penalty terms relative to a specific reward function. The best non-goals are unequivocally defined and measurable against intermediates that are produced in the reward function optimization process. When done right, this sort of framework allows maladaptive processes to be detected reliably and effective interventions executed. Tad makes two other points that I think are worth noting in this discussion: #1. “It will be much easier and cheaper to build the first A.G.I, than to build the first safe A.G.I.” #2. “Lacking human intuition, A.G.I. can do us harm in the effort to oblige us” Given #1, when investing in AI companies, if you aim to be “on the more activist end of the spectrum” you’ll need to spend more money relative to market in order to support ethically responsible AI R&D programs, because they will necessarily be harder and more expensive than the irresponsible ones. Assuming you’re investing in AI companies for their products, and not as pure technology plays, this is simply a reality: the core functionalities needed for your portfolio company to sell product X will always be cheaper to develop than the core functionalities needed for your portfolio company to sell product X within a safe, secure framework. There will be no point for your firm to have codified principles without also having the fortitude to support your AI companies, financially and otherwise, with development processes that are harder and more expensive precisely because they’re more ethical. Many of these costs are absorbed in getting architectures and system designs right, which serve the product anyways, but big costs also come from running unique tests that would be erroneous if ethics weren’t in consideration. Before thinking about having investees agree to your ethical principles around AI, it may be good for your firm to think about whether you’re willing to pay more for those principles to be lived up to. If two identical companies pitch you with identical AI products, but one plans to take an extra 6 months and $10M to safeguard their technology before launching, while the other intends to capture 8% of the market in that time, who will you fund? Point #2 relates closely to “perverse outcomes.” In other words, human harm can be unintentional by AIs. Aside from weaponized-AI that does harm intentionally and raises its own separate ethical dilemmas, daily life AI can do damage in a great number of ways without intending to or even knowing it. The IEEE Standards Association together with the MIT Media Lab recently launched a global Council on Extended Intelligence[3] which addresses many of these issues. Joichi Ito, Director of the MIT Media Lab and a person on the more activist end of the spectrum himself, stresses that: “Instead of trying to control or design or even understand systems, it is more important to design systems that participate as responsible, aware and robust elements of even more complex systems.” Disclaimer: I’m a co-founder of Arctop. [1] https://arxiv.org/abs/1803.03453 [2] https://www.newyorker.com/magazine/2018/05/14/how-frightened... [3] https://globalcxi.org/#vision

directing their learning, they are prone to learn whatever is convenient, which is very often not what the human users want it to. For instance, see the following collection of anecdotes of evolutionary algorithms lerning whatever they please, rather than what the researchers were trying to teach them: https://arxiv.org/abs/1803.03453 Or the descriptions of the difficulties of training Deep Reinforcement Learning models in this article: https://www.alexirpan.com/2018/02/14/rl-hard.html Also, while I'm a staunch symbolicist myself, I'm not 100% convinced that symbolic reasoning is "natural". I think it rather took a lot of effort to develop such systems and most people still have a great deal of trouble using them with precision. If you meant to say that symbolic reasoning should arise spontaneously in a neural network, I see that as very unlikely.