A Watermark for Large Language Models

Watermarking. From an outsiders perspective, the issue appears to reaching consensus on how this can be implemented (but not in the technical sense). There's a game theoretic challenge in that if models define and publish detection mechanisms, this creates a motivation for people to use other systems that don't include this. On the technical front there's a good paper here: https://arxiv.org/pdf/2301.10226.pdf, and a nice very approachable video explaining it here: https://www.youtube.com/watch?v=XZJc1p6RE78.

A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation. After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts. The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).

>fails to understand the gumbel softmax technique I think in this case it doesn't help that there are multiple watermarking schemes, and the easiest for people to understand is the red/green scheme by Kirchenbauer et al. (https://arxiv.org/pdf/2301.10226), which does technically distort the logits (but I'd argue only in cases where you wouldn't notice it anyway). I wasn't aware of this gumbel softmax scheme, it seems you're referring to https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-o... ? That's really clever as it doesn't even distort the logits, basically cryptographically indistinguishable from a "real" random sample unless you have the key. The actual scheme Claude uses seems to be neither of those two though, they say it is SynthId-text which seems to be tournament sampling based.

> A proper watermarking system should be able to have an arbitrarily high accuracy - as many nines as you want. Probability of a false positive is 3 x 10^-5 in one example. * https://imgur.com/h5auUhG.jpg * https://arxiv.org/abs/2301.10226

There's a computerphile video (https://www.youtube.com/watch?v=XZJc1p6RE78) with Dr. Mark Pound explaining a paper by John Kirchenbauer, Jonas Geiping et al. (https://arxiv.org/abs/2301.10226) that described a method for watermarking LLM output like this. It's not directly stated anywhere in the Claude support article that this is what they're using, but the properties of the watermark described seem to point to this method.

Models can't reliably follow instructions involving their own logprobs unless they can take agentic control and use quite sophisticated dynamic grammars/structures/constraints to force this behavior in one shot (which can be slow and the dynamic grammar modification feature isn't supported in closed model APIs for safety reasons) or repeated attempts at rewriting which is expensive/slow. Yes they can do this, but it's more likely closer to the original "red token, green token" paper: https://arxiv.org/abs/2301.10226 i.e. take half of your LLMs vocabulary, and upweight its probabilities by ~55% to the other half's ~45%, and scan for overuse of this half of all tokens. You can even choose a different half/slice for every individual user, for every individual action. You can implement this under the hood cheaply with logit-biasing.

text watermarking works pretty well! https://arxiv.org/abs/2301.10226 When you generate a text with an LLM, you always have some choice. So you can sample in a way that is very likely under your watermark scheme, and unlikely otherwise

I wonder if they have a watermark built in that allows them to detect text generated by Llama easily. https://www.youtube.com/watch?v=XZJc1p6RE78 https://arxiv.org/pdf/2301.10226

https://arxiv.org/pdf/2301.10226.pdf Here's a decent paper on it. It covers private watermarking (you can't detect it exists without a key), resistance to modifications, etc. Essentially you wouldn't know it was there and you can't make simple modifications to fool it. OpenAI could already be doing this, and they could be watermarking with your account ID if they wanted to. The current best countermeasure is likely paraphrasing attacks https://arxiv.org/pdf/2303.11156.pdf

Yeah, if they put watermarks on generated images then everyone will just immediately stop using their product and switch to a competitor. There's some interesting work using steganography to mark generated text (https://arxiv.org/abs/2301.10226), but I'm not sure if that's what they're talking about here.

https://arxiv.org/abs/2301.10226 > We propose a watermarking framework for proprietary language models. The watermark can be embedded with negligible impact on text quality, and can be detected using an efficient open-source algorithm without access to the language model API or parameters. Alpaca uses this already.

I think watermarking is a better solution. There is a paper on it: Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023). A watermark for large language models. arXiv preprint arXiv:2301.10226. (https://arxiv.org/pdf/2301.10226.pdf) Also, Computerphile did a video on it: https://www.youtube.com/watch?v=XZJc1p6RE78 But of course, watermarking or your proposed solution (checksums) only work while the general public cannot run a tool like ChatGPT locally on personal computers. And it's only a matter of time before that happens. So in the long run, we need to either: 1. take away control from the users over their personal computers, in the form of what I would call 'AI DRM' (I strongly oppose this option), or 2. legislate: enforce a legal obligation to disclose that a text is not created by a human, or 3. accept that we cannot know whether or not a text is written by a human, and work with that reality

From the Alpaca report (https://crfm.stanford.edu/2023/03/13/alpaca.html): > we watermark all the model outputs using the method described in Kirchenbauer et al. 2023, so that others can detect (with some probability) whether an output comes from Alpaca 7B https://arxiv.org/abs/2301.10226 > We propose a watermarking framework for proprietary language models. The watermark can be embedded with negligible impact on text quality, and can be detected using an efficient open-source algorithm without access to the language model API or parameters. The watermark works by selecting a randomized set of "green" tokens before a word is generated, and then softly promoting use of green tokens during sampling. We propose a statistical test for detecting the watermark with interpretable p-values

A token is hashed and used to seed a random number generator, which produces the red list for the token after it. Paper: * https://arxiv.org/abs/2301.10226