SAN MATEO, Calif., April 28, 2020 (GLOBE NEWSWIRE) -- Alluxio, the developer of open source cloud data orchestration software, today announced a go-to-market solution in collaboration with Intel to offer an in-memory acceleration layer with 2nd Gen Intel Xeon Scalable processors and Intel Optane persistent memory (PMem). The solution eliminates performance degradation of analytics clusters that are increasingly built on disaggregated compute and storage architecture.
“Today’s disaggregated cloud storage lacks efficient file system semantics support like ‘rename’. Additionally, disaggregated cloud storage typically can’t leverage compute side storage media such as DRAM and SSD for use as buffers and page caches,” said Haoyuan Li, founder and CEO, Alluxio. “Adding Alluxio Data Orchestration System and Intel’s Optane persistent memory solves both issues, enabling maximum benefit for cloud storage and achieving competitive and even better performance than traditional on-premises configurations. This is particularly helpful for hybrid cloud environments when data is remote.”
Benchmarking results show 2.13x faster completion compared to local HDFS and a 1.92x speedup over DRAM cache for 4TB decision support queries when adding Alluxio and Intel persistent memory1. Using Storage over App Direct, a feature of PMem App Direct mode, allows Alluxio to access high-performance block storage without the latency of moving data across the I/O bus to provide the data acceleration and reduction in query runtime. With Alluxio and Intel Xeon Scalable processors, an I/O intensive benchmark delivers a 3.4x speedup over disaggregated S3 object storage and a 1.3x speedup over a co-located compute and storage architecture2.
In addition, Alluxio has joined into a strategic collaboration with Intel aimed at improving their joint customers’ experience with managing and processing their data, such as optimizations for Intel Deep Learning Boost, the AI acceleration technology built into Intel Xeon Scalable processors. Together, Alluxio and Intel will bring solutions to market to help fuel next-generation data, analytics and AI applications and use cases.
“With this new collaboration we bring to market a solution that enables enterprise-grade shared storage and faster time to insights to solve for the challenges we see around bounded storage and compute resources on Hadoop,” said Rowan Scranage, Chief Business Officer, Alluxio. “Together with Intel, we plan to disrupt the advanced analytics and AI status quo with an in-memory data accelerator layer to accelerate intermediate data access and ease data bottlenecks that many of our customers are highlighting as key challenges with their increasing big data requirements.”
“We are thrilled to work with Alluxio and help bring this compelling solution to market,” said Arijit Bandyopadhyay, CTO of enterprise analytics & AI within Intel’s data platform group. “As we see more companies building advanced analytics and AI applications that are faced with performance challenges, the Alluxio-based in-memory data accelerator coupled with Intel’s Optane persistent memory will bring data locality, data accessibility, and data elasticity back into the environment. Based on the performance gains we are seeing with this joint solution, we are looking forward to bringing it to the market at scale.”
To learn more or get started with the Intel and Alluxio joint solution, please contact sales@alluxio.com.
Resources
- On-demand tech talk: Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
- Webinar: Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio
- Solution brief: Get insights faster with Alluxio and Intel
- Blog: Speeding Big Data Analytics on the Cloud with In-Memory Data Accelerator
Tweet this: .@Alluxio launches enhanced hybrid cloud solution based on #Intel Optane technology with major performance boost #DataOrchestration https://bit.ly/3bulmPB
1. Footnote: Tested by Intel as on 12/06/2019 with the following configuration. Detailed white paper will follow.
Optane PMem cache: 2 socket Intel® Xeon® Gold 6240 Processor, 18 cores HT On Turbo ON Total Memory 192 GB (12 slots/ 16GB/ 2666 MHz), DCPMM 1TB (8 slots/ 128GB/ 2666 MHz), DCPMM firmware version: 01.02.00.5410, BIOS: SE5C620.86B.0X.02.0094.102720191711 (ucode:0x500002c), BKC version: ww08.2019, Fedora 29 (Server Edition), 4.20.6-200.fc29.x86_64, Storage for application: 11x 1TB HDD (ST1000NX0313) for Ceph OSD, Hadoop version: Hadoop 3.1.2, Alluxio version: Alluxio 2.0.0, Spark version: Spark 2.3.0, Hive version: Hive 3.1.1, Ceph version: Ceph 12.2.12
DRAM cache: 2 socket Intel® Xeon® Gold 6240 Processor, 18 cores HT On Turbo ON Total Memory 768 GB (24 slots/ 32GB/ 2666 MHz), BIOS: SE5C620.86B.0X.02.0094.102720191711 (ucode:0x500002c), Fedora 29 (Server Edition), 4.20.6-200.fc29.x86_64, Storage for application: 11x 1TB HDD (ST1000NX0313) for Ceph OSD, Hadoop version: Hadoop 3.1.2, Alluxio version: Alluxio 2.0.0, Spark version: Spark 2.3.0, Hive version: Hive 3.1.1, Ceph version: Ceph 12.2.12
Without cache: 2 socket Intel® Xeon® Gold 6240 Processor, 18 cores HT On Turbo ON Total Memory 192 GB (12 slots/ 16GB/ 2666 MHz), BIOS: SE5C620.86B.0X.02.0094.102720191711 (ucode:0x500002c), Fedora 29 (Server Edition), 4.20.6-200.fc29.x86_64, Storage for application: 11x 1TB HDD (ST1000NX0313) for Ceph OSD, Hadoop version: Hadoop 3.1.2, Alluxio version: Alluxio 2.0.0, Spark version: Spark 2.3.0, Hive version: Hive 3.1.1, Ceph version: Ceph 12.2.12
2. See https://www.alluxio.io/blog/speeding-big-data-analytics-on-the-cloud-with-in-memory-data-accelerator/ for full details and configs.
About Alluxio
Proven at global web scale in production for modern data services, Alluxio is the developer of open source data orchestration software for the cloud. Alluxio moves data closer to big data and machine learning compute frameworks in any cloud across clusters, regions, clouds and countries, providing memory-speed data access to files and objects. Intelligent data tiering and data management deliver consistent high performance to customers in financial services, high tech, retail and telecommunications. Alluxio is in production use today at seven out of the top ten internet companies. Venture-backed by Andreessen Horowitz and Seven Seas Partners, Alluxio was founded at UC Berkeley’s AMPLab by the creators of the Tachyon open source project. For more information, contact info@alluxio.com or follow us on LinkedIn, or Twitter.
Intel, the Intel logo, Xeon and Optane are trademarks of Intel Corporation or its subsidiaries.
Media Contact:
Beth Winkowski
Winkowski Public Relations, LLC for Alluxio
978-649-7189
beth@alluxio.com