- TensorStore (github.com/google/tensorstore) has had a huge 2024 and continues to power checkpointing for much of the largest ML work at Google, including Gemini, and support major Google Cloud customers training models on GCP. Read on for some specific 2024 updates!
- The new optionally cooperative distributed b-tree (OCDBT) driver has greatly increased performance on big workloads, and in 2024 we added transactional support to further optimize performance on object storage systems like GCS: github.com/google/tenso...
Dec 18, 2024 16:54
- New batch read support has significantly improved efficiency when reading from sharded formats like zarr v3 and neuroglancer precomputed: github.com/search?q=rep... and many other general performance and efficiency improvements (e.g. github.com/google/tenso...)
- New C++ API documentation has greatly improved the legibility of the C++ API: google.github.io/tensorstore/...
- For more background on TensorStore check out the original blog post: research.google/blog/tensors... and reach out to me, jbms@google.com, or github.com/google/tenso... if you have any comments, questions, or suggestions.
