How to: Reliability
**See also [System architecture](#system-architecture)**

Books:

- 📖 [Site Reliability Engineering](https://landing.google.com/sre/books/), Google
  - Written by members of Google's SRE team, with a comprehensive analysis of the entire software lifecycle - how to build, deploy, monitor, and maintain large scale systems.
- 📖 [Incident Metrics in SRE](https://static.googleusercontent.com/media/sre.google/en//static/pdf/IncidentMeticsInSre.pdf), Štěpán Davidovič, Google
  - Canonical definition of TTD (Time To Detect) TTM (Mitigate), TTR (Recover), TBF (Time Between Failure)

Quotes:

> Quality is a snapshot at the start of life and reliability is a motion picture of the day-by-day operation.
> – [NIST](https://www.itl.nist.gov/div898/handbook/apr/section1/apr111.htm)
> Reliability is the one feature every customer users. -- An auth0 SRE.

Articles:

- I already mentioned the book Release it! above. There's also a [presentation](http://www.slideshare.net/justindorfman/stability-patterns-presentation) from the author.
- [Service Recovery: Rolling Back vs. Forward Fixing](https://www.linkedin.com/pulse/service-recovery-rolling-back-vs-forward-fixing-mohamed-el-geish/)
- [How Complex Systems Fail](https://how.complexsystems.fail/)
  - Catastrophe requires multiple failures – single point failures are not enough.
  - Complex systems contain changing mixtures of failures latent within them.
  - Post-accident attribution to a ‘root cause’ is fundamentally wrong.
  - Hindsight biases post-accident assessments of human performance.
  - Safety is a characteristic of systems and not of their components
  - Failure free operations require experience with failure.
- [Systems that defy detailed understanding](https://blog.nelhage.com/post/systems-that-defy-understanding/)
  - Focus effort on systems-level failure, instead of the individual component failure.
  - Invest in sophisticated observability tools, aiming to increase the number of questions we can ask without deploying custom code
- [Operating a Large, Distributed System in a Reliable Way: Practices I Learned](https://blog.pragmaticengineer.com/operating-a-high-scale-distributed-system/), Gergely Orosz.
  - A good summary of processes to implement.
- [Production Oriented Development](https://paulosman.me/2019/12/30/production-oriented-development.html)
  - Code in production is the only code that matters
  - Engineers are the subject matter experts for the code they write and should be responsible for operating it in production.
  - Buy Almost Always Beats Build
  - Make Deploys Easy
  - Trust the People Closest to the Knives
  - QA Gates Make Quality Worse
  - Boring Technology is Great.
  - Non-Production Environments Have Diminishing Returns
  - Things Will Always Break
- 🏙 [High Reliability Infrastructure migrations](https://speakerdeck.com/jvns/high-reliability-infrastructure-migrations), Julia Evans.
- [Appendix F: Personal Observations on the Reliability of the Shuttle](https://www.refsmmat.com/files/reflections.pdf), Richard Feynman
- [Lessons learned from two decades of Site Reliability Engineering](https://sre.google/resources/practices-and-processes/twenty-years-of-sre-lessons-learned/)
- [Service Reliability Mathematics](https://addyosmani.com/blog/service-reliability/), Addy Osmani

Resources:

- 🧰 [dastergon/awesome-sre](https://github.com/dastergon/awesome-sre)
- 🧰 [upgundecha/howtheysre](https://github.com/upgundecha/howtheysre): a curated collection of publicly available resources on SRE at technology and tech-savvy organizations

#### Integration patterns (dependency management)

- [Circuit Breaker](https://martinfowler.com/bliki/CircuitBreaker.html) (mentioned in the Release it! book)
  - [Making the Netflix API More Resilient](https://netflixtechblog.com/making-the-netflix-api-more-resilient-a8ec62159c2d), Netflix Blog
  - 🏙 [Application Resilience Engineering & Operations at Netflix - Speaker Deck](https://speakerdeck.com/benjchristensen/application-resilience-engineering-and-operations-at-netflix)
  - [Design Pattern: Circuit Breaker](https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker), Microsoft Azure
- [Rate limiter algorithms](https://smudge.ai/blog/ratelimit-algorithms) (and their [implementation](https://github.com/upstash/ratelimit-js/blob/main/src/lua-scripts/single.ts))
- [Interactive Guide: Mastering Rate Limiting](https://blog.sagyamthapa.com.np/interactive-guide-to-rate-limiting)
- [Good Retry, Bad Retry: An Incident Story](https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6): insightful, well-written story about retries, circuit breakers, deadline, etc.

#### Resiliency

- 🏙 [The Walking Dead - A Survival Guide to Resilient Applications](https://speakerdeck.com/daschl/the-walking-dead-a-survival-guide-to-resilient-applications)
- 🏙 [Defensive Programming & Resilient systems in Real World (TM)](https://speakerdeck.com/tuenti/defensive-programming-and-resilient-systems-in-real-wor