How to: Releasing & deploying
- [How to deploy software](https://zachholman.com/posts/deploying-software), Zach Holman
- [BlueGreenDeployment](http://martinfowler.com/bliki/BlueGreenDeployment.html), Martin Fowler
- [Move fast and break nothing](https://zachholman.com/talk/move-fast-break-nothing/), Zach Holman
- 🏙 [Move fast and don't break things](https://docs.google.com/presentation/d/15gNk21rjer3xo-b1ZqyQVGebOp_aPvHU3YH7YnOMxtE/edit#slide=id.g437663ce1_53_591), Google
- [Shipping to Production](https://blog.pragmaticengineer.com/shipping-to-production/), The Pragmatic Programmer

#### Versioning

- [SemVer - Semantic Versioning](https://semver.org/)
- [CalVer - Calendar Versioning](https://calver.org/)
- [Semantic Versioning Will Not Save You](https://hynek.me/articles/semver-will-not-save-you/)
- [Version numbers: how to use them?](https://bernat.tech/posts/version-numbers/)

#### Checklists

- [Production Readiness Checklist](https://gruntwork.io/devops-checklist/), Gruntwork
- [Checklist: what had to be done before deploying microservices to production](https://habr.com/en/post/438186/)
- [Things end users care about but programmers don't](https://instadeq.com/blog/posts/things-end-users-care-about-but-programmers-dont/): includes colors, formatting, themes, integrations, UX, compatibility, operations.

#### Feature flags

- [Flipping out](http://code.flickr.net/2009/12/02/flipping-out/), Flickr. One of the first articles about feature flags.
- [Feature Flags, Toggles, Controls](https://featureflags.io/), a website documenting feature flags, from Launch Darkly.
- [Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html), Pete Hodgson, martinFowler.com. Comprehensive article on the topic.
  - Deliver new functionality to users rapidly but safely
  - Release Toggles allow incomplete and un-tested codepaths to be shipped to production as latent code which may never be turned on.
  - Experiment Toggles are used to perform multivariate or A/B testing.
  - Ops Toggles control operational aspects of our system's behavior.
  - Permissioning Toggles change the features or product experience that certain users receive.
  - Static vs dynamic toggles
  - Long-lived toggles vs transient toggles
  - Savvy teams view their Feature Toggles as inventory which comes with a carrying cost, and work to keep that inventory as low as possible.
- [Feature Flags Best Practices: Release Management](https://launchdarkly.com/blog/release-management-flags-best-practices/), LaunchDarkly
- [How we ship code faster and safer with feature flags](https://github.blog/2021-04-27-ship-code-faster-safer-feature-flags/), Github.
- [Flipr: Making Changes Quickly and Safely at Scale](https://eng.uber.com/flipr/), Uber
- [Feature flags are ruining your codebase](https://zaidesanton.substack.com/p/feature-flags-are-ruining-your-codebase)

#### Testing in production

- [Why We Leverage Multi-tenancy in Uber's Microservice Architecture](https://eng.uber.com/multitenancy-microservice-architecture/)
- [Developing in Production](https://tersesystems.com/blog/2020/01/22/developing-in-production/)
  - Complex systems have emergent behavior, producing epiphenomenon that only appears with sufficient scale.
  - Wood's theorem: As the complexity of a system increases, the accuracy of any single agent’s own model of that system decreases rapidly.
  - The more tools and code that you add to create elements in a system, the harder it is to replicate an environment encompassing those tools and code.
  - At the core of testing in production is the idea of splitting deployments (of artifacts) from releases (of features).
- [Testing in Production: the hard parts](https://medium.com/@copyconstruct/testing-in-production-the-hard-parts-3f06cefaf592), Cindy Sridharan
  - The whole point of [actual] distributed systems engineering is you assume you’re going to fail at some point in time and you design the system in such a way that the damage, at each point is minimized, that recovery is quick, and that the risk is acceptably balanced with cost.
  - How can you cut the blast radius for a similar event in half?
    - Differentiate between deployment (0 risk) and release
    - Build a deploy-observe-release pipeline
    - Make incremental rollouts the norm (canaries, %-based rollouts, etc.)
    - Test configuration changes just like you test code
    - Default to roll back, avoid fixing forward (slow!)
    - Eliminate gray failures - prefer crashing to degrading in certain cases
    - Prefer loosely coupled services at the expense of latency or correctness
    - Use poison tasters (isolate handling of client input)
    - Implement per-request-class backpressure
    - Have proper visibility from a client/end-user standpoint (client-side metrics)
- [Testing in Production, the safe way](https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1)
- [Multi-Tenancy in a Microservice Architecture](https://www.usenix.org/system/files/login/article