Overview

The course examines the fundamental problem of data storage in the context of modern technology for data-intensive applications. A particular focus is given to choosing the right database technology for a task at hand and tradeoffs between performance, ease of use, data integrity and other considerations. Hands-on lessons are done using PostgreSQL, MongoDB, Redis and Snowflake databases.

The course starts with a brief overview of a data storing task for an application. Data storing tasks in general, with all possible solutions, are like files, in-memory services or specialised applications (databases). We proceed with a list of requirements, which proved to be essential for a data storing tool: ACID, transactions, availability of data access languages (SQL, etc.). Afterwards, we will illustrate why given requirements determined the market dominance of classical relational databases (Oracle, MS SQL, PostgreSQL, MySQL, etc.) at the end of the 20th century. Later we describe why technological advances of the 21st century gave birth to a set of non-classical databases, such as in-memory storage, document storage, columnar storage, etc. The bulk of the remaining course focuses on the tradeoffs to be considered during technology selection and the database design. We discuss a Polyglot Persistence paradigm for combining multiple databases for different facets of an application, combining their strengths and mitigating their weaknesses. We discuss the balance between performance, complexity and permitted data delay for various databases and architectural approaches. We emphasise the difference between OLAP (analytical) and OLTP tasks and modern data warehouse designs (Data Vault, Anchor Modeling, etc.). Plenty of hands-on examples and homework is given to demonstrate ideas and compare and contrast various approaches and technologies. The course wraps up with a discussion of the modern state of the art databases, like serverless cloud databases (Snowflake) and global cloud tools, violating CAP-theorem (Google Cloud Spanner).