In-Datacenter Performance Analysis of a Tensor Processing Unit

The first TPU (Seastar) was designed, tested, and deployed in 15 months: https://arxiv.org/pdf/1704.04760 They started becoming available internally in mid 2015.

Might be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760). If so, does Groq actually provide hardware support for LLM inference?

One good approach could be to base the architecture on the TPU v1 from [1]. There are also open-source accelerators you could get inspiration from, for example [2][3]. If you want to do less work/not hand code the RTL yourself then you could look into methods for automatically mapping OpenCL to an FPGA accelerator architecture (or a service like [3] provides pre-designed architectures for multiple FPGAs). [1] https://arxiv.org/abs/1704.04760 [2] https://github.com/jofrfu/tinyTPU [3] https://github.com/tensil-ai/tensil

In a conv layer, weights are shared across many input features. E.g. assume a 1x1 conv layer with a 28x28x3 input. You only need to load 3 weights even though there are effectively 28x28=748 different dot products. In practice, the input and output activations can be stored on chip as well (except for the first layer) which means the ratio of operations to DRAM accesses can be incredibly high. For some real world examples, take a look at the classic Eyeriss paper[1] which finds a ratio of 345-285 for AlexNet and VGG-16 respectively. You can also check out the TPU paper[2] which places the ratio at >1000 for some unnamed CNNs. Compare that to your analysis which yields a ratio of 2. [1] https://people.csail.mit.edu/emer/papers/2017.01.jssc.eyeris... [2] https://arxiv.org/pdf/1704.04760.pdf

So far only very few details are disclosed. Here are two presentations: https://supercomputersfordl2017.github.io/Presentations/Imag... http://learningsys.org/nips17/assets/slides/dean-nips17.pdf For the last version of the TPU, Google provided more detail, e.g., in this paper: https://arxiv.org/pdf/1704.04760.pdf Hopefully, Google will publish something similar for TPUv2, but I have no knowledge whether or when that might happen.

The reserve TPU button has been available on the dashboard for the last few months. But I assume instances have been prioritized for large customers such as Two Sigma. From the paper: "Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU." In-Datacenter Performance Analysis of a Tensor Processing Unit https://arxiv.org/abs/1704.04760 Price is about 5x cloud nvidia gpu instance on an hourly basis.

They compared TPU v1 with server hardware available at the time (2015, i.e. K80 and Haswell). Are the cards you mention from 2015? Are they for gaming or servers? Do they use ECC? If not, that rules them out right away. You can find the paper with the methodology, theoretical peaks and latency management at https://arxiv.org/abs/1704.04760

One example is: https://arxiv.org/abs/1704.04760 There are many potential designs for these things, but the first gen TPU is one that works, is in production, and has been described in a paper. But you have to differentiate if you mean an inference engine, or something that can also do training. For HoloLens, it's probably going to be an inference unit, which means it'll possibly look something like a TPU, perhaps with more specific hardware support optimized for convolutions (which are very important for visual processing DNNs these days), as the NVidia tensor units are.