Stockholm, Oct. 11, 2023 (GLOBE NEWSWIRE) -- At the Feature Store Summit 2023, featurestore.org, a forum for the international community of users and developers of Feature Store platforms, announced new feature store benchmarks. The benchmarks were developed by Hopsworks in collaboration with Karolinska Institute and KTH University.
Feature stores for machine learning are a new class of data platform that support the development and operation of machine learning systems. Although new benchmarks for AI systems have recently appeared (such as TPCx-AI), these cover a very wide array of use cases, including video and images. In contrast, feature stores are designed primarily to manage structured data that comes from databases, data warehouses, and files.
In this context, the feature store community developed a first set of benchmarks for common usage patterns of feature stores. So far, three benchmarks have been published:
- Offline API Benchmark: Measures the throughput of a feature store for the creation of training data as Pandas DataFrames or files.
- Online API Benchmark: Measures the latency of online feature serving to AI-enabled applications.
- Feature Freshness Benchmark: Measure the time taken from when a feature is computed to when it becomes available in the online feature store for serving.
“We are happy to introduce these benchmarks to the feature store community so users can easily reproduce the performance claims of vendors. We need benchmarks as a feature store community so that we can measure progress in our field. The presented benchmarks follow the database benchmarking principles and overall promote reproducibility, fairness and realistic workloads. ” says Jim Dowling, CEO of Hopsworks.
The introduction of these benchmarks marks a significant step towards helping feature store users measure and understand the performance of feature stores in real-world applications. The current benchmarks are just a first step in a community effort towards building a suite of benchmarks for feature stores.
The benchmarks are available on a public Github repo.
Contribute to the benchmarks:
Feel free to create a PR on the Github repository to add a new feature store or benchmark. Be sure to include all the hardware setup and software version numbers, that should be as close as possible to existing benchmarks to ensure apple-to-apple comparisons.
