Follow the ideas that keep turning up in Hacker News discussions.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.
Ashish Vaswani, Noam Shazeer, Niki Parmar +5
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration.
Leo Gao, Stella Biderman, Sid Black +9
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja +16
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B
Ziwei Xu, Sanjay Jain, Mohan Kankanhalli
Hallucination has been widely recognized to be a significant drawback for large language models (LLMs).
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai +3
Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools.
Luyu Gao, Xueguang Ma, Jimmy Lin +1
While dense retrieval has been shown effective and efficient across tasks and languages, it remains difficult to create effective fully zero-shot dense retrieval systems when no relevance label is available.
Ofir Press, Noah A. Smith, Mike Lewis
Since the introduction of the transformer model by Vaswani et al.
Kevin Meng, David Bau, Alex Andonian +1
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals +2
Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks.
Yutao Sun, Li Dong, Barun Patra +6
Position modeling plays a critical role in Transformers.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow +6
Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers.
Zhen Qin, Rolf Jagerman, Kai Hui +9
Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem.
Yiqun Duan, Jinzhao Zhou, Zhen Wang +2
The translation of brain dynamics into natural language is pivotal for brain-computer interfaces (BCIs).
Ryan Greenblatt, Carson Denison, Benjamin Wright +17
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training.
OpenAI, Josh Achiam, Steven Adler +278
We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs.
Jason Wei, Xuezhi Wang, Dale Schuurmans +6
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning.
Ronen Eldan, Yuanzhi Li
Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small.
Jason Wei, Yi Tay, Rishi Bommasani +13
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Nataliya Kosmyna, Eugene Hauptmann, Ye Tong Yuan +5
This study explores the neural and behavioral consequences of LLM-assisted essay writing.
Hattie Zhou, Azade Nova, Hugo Larochelle +3
Large language models (LLMs) have shown increasing in-context learning capabilities through scaling up model and data size.
Edward J. Hu, Yelong Shen, Phillip Wallis +5
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains.
John Kirchenbauer, Jonas Geiping, Yuxin Wen +3
Potential harms of large language models can be mitigated by watermarking model output, i.e., embedding signals into generated text that are invisible to humans but algorithmically detectable from a short span of tokens.
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya +8
Reliable generalization lies at the heart of safe ML and AI.
Mary Phuong, Marcus Hutter
This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results).
Lingjiao Chen, Matei Zaharia, James Zou
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services.
Shunyu Yao, Dian Yu, Jeffrey Zhao +4
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference.