Skip to content
The methodOpen data
Back to the collection

Virtual Machine Warmup Blows Hot and Cold

Edd Barrett, Carl Friedrich Bolz-Tereick, Rebecca Killick, Sarah Mount, Laurence Tratt

2016 · arXiv

Computer systemsStatistics & probabilityProgramming languages
Read the paper
HN score
0
Mentions
10
Readers
7
Months
10
What’s in a score?

From the abstract

Virtual Machines (VMs) with Just-In-Time (JIT) compilers are traditionally thought to execute programs in two phases: the initial warmup phase determines which parts of a program would most benefit from dynamic compilation, before JIT compiling those parts into machine code; subsequently the program is said to be at a steady state of peak performance. Measurement methodologies almost always discard data collected during the warmup phase such that reported measurements focus entirely on peak performance. We introduce a fully automated statistical approach, based on changepoint analysis, which allows us to determine if a program has reached a steady state and, if so, whether that represents peak performance or not. Using this, we show that even when run in the most controlled of circumstances, small, deterministic, widely studied microbenchmarks often fail to reach a steady state of peak performance on a variety of common VMs. Repeating our experiment on 3 different machines, we found that at most 43.5% of pairs consistently reach a steady state of peak performance.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

10 selected excerpts · 10 total mentions

CalChris
Neutral mention · rule
Laurence Tratt's paper Virtual machine warmup blows hot and cold [1] paper has been posted several times and never really discussed. It covers this problem for Java VMs and also presents a benchmarking methodology. [1] https://dl.acm.org/doi/10.1145/3133876
Original comment Parent story unavailable in snapshot
sa46
Neutral mention · rule
Reliably benchmarking optimizing VMs like Java and JavaScript is a research problem. See “Virtual machine warmup blows hot and cold”. The authors run a variety of benchmarks (with a pristine methodology) and find 43% of them provide “bad inconsistent” results. Paper summary: https://blog.acolyer.org/2017/11/07/virtual-machine-warmup-b... Paper: https://dl.acm.org/doi/pdf/10.1145/3133876
Original comment Parent story unavailable in snapshot
ltratt
Neutral mention · rule
I not only welcome reasonable scepticism, but I do my best to facilitate it. I have accrued sufficient evidence over time of my own fallibility, and idiocy, that I now try to give people the opportunity to spot mistakes so that I might correct them. As a happy bonus, this also gives people a way of verifying whether the work was done in the right spirit or not. To that end we work in the open, so all the evidence you need to back up your assertions, or assuage your doubts, has been available since the first day we started: * Here's the experiment, with its 1025 commits going back to 2015 https://github.com/softdevteam/warmup_experiment/ -- note that the benchmarks are…
Read the full comment
slurped in before we'd even got many of the VMs compiling. * You can also see from the first commit that we simply slurped in the CLBG benchmarks wholesale from a previous paper that was done some time before I had any inkling that there might be warmup problems https://github.com/ltratt/vms_experiment/ * Here's the repo for the paper itself, where you can see us getting to grips with what we were seeing over several years https://github.com/softdevteam/warmup_paper * The snapshots of the paper we released are at https://arxiv.org/abs/1602.00602v1 -- the first version ("V1") clearly shows problems but we had no statistical analysis (note that the first version has a different author list than the final version, and the author added later was a stats expert). * The raw data for the releases of the experiment are at https://archive.org/download/softdev_warmup_experiment_artef... so you can run your own statistical analysis on them. To be clear, our paper is (or, at least, I hope is) clear to scope its assertions. It doesn't say "VMs never warmup" or even "VMs only warmup X% of the time". It says "in this cross-language, cross-VM, benchmark suite of small benchmarks we observed warmup X% of the time, and that might suggest there are broader problems, but we can't say for sure". There are various possible hypotheses which could explain what we saw, including "only microbenchmarks, or this set of microbenchmarks, show this problem". Personally, that doesn't feel like the most likely explanation, but I have been wrong about bigger things before!
Original comment More evidence for problems in VM warmup
etiene
Neutral mention · rule
> Are they likely to be expert in all those programming languages? No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf > If Lua wasn't included would you complain ? Maybe I would, who knows. I did also mention in another comment it would be nice if Julia was there. It depends how relevant they are for the research being conducted. As a language used very often in microcontrollers, Lua is very relevant for energy efficiency research. I don't know anything about Forth, but I wouldn't just dismiss people saying it should be there with "then contribute forth algorithms to the benchmarks game yourself".
Original comment Energy Efficiency Across Programming Languages
yencabulator
Neutral mention · rule
Virtual Machine Warmup Blows Hot and Cold https://arxiv.org/abs/1602.00602 > [...] even when run in the most controlled of circumstances, small, deterministic, widely studied microbenchmarks often fail to reach a steady state of peak performance on a variety of common VMs.
Original comment Jitterdämmerung (2015, Updated 2021)
mafribe
Neutral mention · rule
I agree with you that idealised compilers have a beautiful pipeline structure. Finding this structure Is one of the great achievement of the early computing pioneers. turn it into a laughably simple chain. By this reasoning, any computable problem is trivially simple, because I can translate it into ARM/x86/MIPS/... machine code, and machine code is just a linear list of trivial commands, i.e. I can "turn it into a laughably simple chain". My point is: if you chain enough simple things, the result stops being simple. Look for example at modern JIT compilers, tracing or otherwise. They are not simple by any stretch of the imagination. To see that just look at a program, and predict how long the warm-up will take. Whoops ... unknown [1]. [1] E. Barrett et al., Virtual Machine Warmup Blows Hot and Cold. http://arxiv.org/abs/1602.00602
Original comment CS143: Compilers (2011)

Keep following the thread.

More in Computer systems

More from the same topics.