Challenges and Solutions in Database Scaling: Nextdoor’s Optimization Journey

In a detailed blog series, Nextdoor’s Core Services team provides valuable insights into their strategies for optimizing database and cache infrastructure. This series is aimed at development teams dealing with the scaling challenges of PostgreSQL and Redis.

The Initial Situation and Key Challenges Nextdoor Faced Two Major Issues:

Excessive load on the primary database, despite the use of read replicas. Issues with inconsistencies in the cache system. The backend architecture design, heavily reliant on the Django ORM, caused requests to often go to the primary database for security reasons. This triggered further issues, such as data inconsistencies due to concurrent writes and unsuccessful cache updates.

Innovative Solution Approach

1. Dynamic Routing for Read Replicas

The team developed an intelligent tracking of database changes to direct read requests accordingly to primary or replica databases. Changes in the data led to a preferred routing to the primary database, while stabilized datasets were sent to replicas.

2. Robust Cache Serialization

Switching from Python Pickle to MessagePack as the serialization format was a crucial step to avoid compatibility issues during schema changes. This switch also reduced the risks of a “Thundering Herd” problem during deployments.

3. Versioning of Cache Entries

By introducing automatically incremented version numbers (db_version) per dataset, updated by triggers in PostgreSQL, developers could prevent outdated data from landing in the cache. This was made possible through atomic operations in Redis using Lua scripts.

4. Gradual Consistency Assurance

A PostgreSQL Change Data Capture (CDC) based “Reconciler” ensured that missed cache updates could be corrected in real-time and with delay to maintain data integrity.

Additional Aspects:

In addition to the described solution approaches, the team also considers security aspects and continuous system monitoring. Monitoring tools are used to detect and resolve bottlenecks early. Another measure being considered is storing more complex data queries in the cache to enhance performance during demanding queries.

Conclusion:

Through this comprehensive optimization, Nextdoor was able to significantly reduce the load on the primary database, improve consistency in the cache infrastructure, and make the system more resilient overall. This innovative approach demonstrates that traditional relational databases can be highly efficiently scaled through extensive analysis and optimization strategies. The key lies in careful planning and implementation of tailored solutions.