How to: Data Storage Optimisation
* [AIStore](https://github.com/NVIDIA/aistore) ![](https://img.shields.io/github/stars/NVIDIA/aistore.svg?cacheSeconds=86400) - AIStore is a lightweight object storage system with the capability to linearly scale out with each added storage node and a special focus on petascale deep learning.
* [Alluxio](https://github.com/Alluxio/alluxio) ![](https://img.shields.io/github/stars/Alluxio/alluxio.svg?cacheSeconds=86400) - A virtual distributed storage system that bridges the gab between computation frameworks and storage systems.
* [Apache Arrow](https://github.com/apache/arrow) ![](https://img.shields.io/github/stars/apache/arrow.svg?cacheSeconds=86400) - In-memory columnar representation of data compatible with Pandas, Hadoop-based systems, etc..
* [Apache Druid](https://github.com/apache/druid) ![](https://img.shields.io/github/stars/apache/druid.svg?cacheSeconds=86400) - A high performance real-time analytics database. Check this [article](https://towardsdatascience.com/introduction-to-druid-4bf285b92b5a) for introduction.
* [Apache Hudi](https://github.com/apache/hudi) ![](https://img.shields.io/github/stars/apache/hudi.svg?cacheSeconds=86400) - Hudi is a transactional data lake platform that brings core warehouse and database functionality directly to a data lake. Hudi is great for streaming workloads, and also allows creation of efficient incremental batch pipelines. Supports popular query engines including Spark, Flink, Presto, Trino, Hive, etc. More info [here](https://hudi.apache.org/).
* [Apache Iceberg](https://github.com/apache/iceberg) ![](https://img.shields.io/github/stars/apache/iceberg.svg?cacheSeconds=86400) - Iceberg is an ACID-compliant, high-performance format built for huge analytic tables (containing tens of petabytes of data), and it brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. More info [here](https://iceberg.apache.org/).
* [Apache Ignite](https://github.com/apache/ignite) ![](https://img.shields.io/github/stars/apache/ignite.svg?cacheSeconds=86400) - A memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale - [Demo](https://www.youtube.com/watch?v=Xt4PWQ__YPw).
* [Apache Parquet](https://github.com/apache/parquet-java) ![](https://img.shields.io/github/stars/apache/parquet-java.svg?cacheSeconds=86400) - On-disk columnar representation of data compatible with Pandas, Hadoop-based systems, etc..
* [Apache Pinot](https://github.com/apache/pinot) ![](https://img.shields.io/github/stars/apache/pinot.svg?cacheSeconds=86400) - A realtime distributed OLAP datastore. Comparison of the open source OLAP systems for big data: ClickHouse, Druid, and Pinot is found [here](https://medium.com/@leventov/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7).
* [Casibase](https://github.com/casibase/casibase) ![](https://img.shields.io/github/stars/casibase/casibase.svg?cacheSeconds=86400) - Casibase is a LangChain-like RAG (Retrieval-Augmented Generation) knowledge database with web UI and Enterprise SSO.
* [Chroma](https://github.com/chroma-core/chroma) ![](https://img.shields.io/github/stars/chroma-core/chroma.svg?cacheSeconds=86400) - Chroma is an open-source embedding database.
* [ClickHouse](https://github.com/ClickHouse/ClickHouse) ![](https://img.shields.io/github/stars/ClickHouse/ClickHouse.svg?cacheSeconds=86400) - ClickHouse is an open source column oriented database management system.
* [Delta Lake](https://github.com/delta-io/delta) ![](https://img.shields.io/github/stars/delta-io/delta.svg?cacheSeconds=86400) - Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines.
* [EdgeDB](https://github.com/geldata/gel) ![](https://img.shields.io/github/stars/geldata/gel.svg?cacheSeconds=86400) - Gel supercharges Postgres with a modern data model, graph queries, Auth & AI solutions, and much more.
* [GPTCache](https://github.com/zilliztech/GPTCache) ![](https://img.shields.io/github/stars/zilliztech/GPTCache.svg?cacheSeconds=86400) - GPTCache is a library for creating semantic cache for large language model queries.
* [InfluxDB](https://github.com/influxdata/influxdb) ![](https://img.shields.io/github/stars/influxdata/influxdb.svg?cacheSeconds=86400) Scalable datastore for metrics, events, and real-time analytics.
* [Milvus](https://github.com/milvus-io/milvus) ![](https://img.shields.io/github/stars/milvus-io/milvus.svg?cacheSeconds=86400) Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.
* [Marqo](https://github.com/marqo-ai/marqo) ![](https://img.shields.io/github/stars/marqo-ai/marqo.svg?cacheSeconds=86400) Marqo is an end-to-end