How to: Observability (monitoring, logging, exception handling)
_See also: [Site Reliability Engineering (SRE)](#site-reliability-engineering-sre)_

#### Logging

- [Do not log](https://sobolevn.me/2020/03/do-not-log) dwells on some logging antipatterns.
  - Logging does not make much sense in monitoring and error tracking. Use better tools instead: error and business monitorings with alerts, versioning, event sourcing.
  - Logging adds significant complexity to your architecture. And it requires more testing. Use architecture patterns that will make logging an explicit part of your contracts
  - Logging is a whole infrastructure subsystem on its own. And quite a complex one. You will have to maintain it or to outsource this job to existing logging services
- [Lies My Parents Told Me (About Logs)](https://www.honeycomb.io/blog/lies-my-parents-told-me-about-logs/)
  - Logs are cheap
  - I can run it better myself
  - Leveled logging is a great way to separate information
  - Logs are basically the same as events
  - A standard logging format is good enough
- [Logging - OWASP Cheat Sheet Series](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html)
- [The Audit Log Wall of Shame](https://audit-logs.tax/): list of vendors that don’t prioritize high-quality, widely-available audit logs for security and operations teams.
- [Guide on Structured Logs](https://signoz.io/blog/structured-logs/)
- [What an error log level should mean](https://utcc.utoronto.ca/~cks/space/blog/programming/ErrorsShouldRequireFixing)
- [Logging Sucks - Your Logs Are Lying To You](https://loggingsucks.com/)

#### Error/exception handling

- [Error handling antipatterns](./antipatterns/error-handling-antipatterns.md) in this repo.
- [Writing Helpful Error Messages](https://developers.google.com/tech-writing/error-messages), Google Developers' course on Technical Writing
  - Explain the problem
  - Explain the solution
  - Write clearly
- [Errors, Errors Everywhere: How We Centralized and Structured Error Handling](https://olivernguyen.io/w/namespace.error/) (for Go, but useful for any languages)
- For inspiration: [Handle Errors - Graph API](https://developers.facebook.com/docs/graph-api/guides/error-handling#receiving-errorcodes)

#### Metrics

- [Meaningful availability](https://blog.acolyer.org/2020/02/26/meaningful-availability/)
  - A good availability metric should be meaningful, proportional, and actionable. By "meaningful" we mean that it should capture what users experience. By "proportional" we mean that a change in the metric should be proportional to the change in user-perceived availability. By "actionable" we mean that the metric should give system owners insight into why availability for a period was low. This paper shows that none of the commonly used metrics satisfy these requirements…
- 📃 [Meaningful Availability](https://www.usenix.org/conference/nsdi20/presentation/hauer) paper.
  - This paper presents and evaluates a novel availability metric: windowed user-uptime

#### Monitoring

- Google, [Site Reliability Engineering, Monitoring Distributed Systems](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/)
  - [Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/)
- PagerDuty, [Monitoring Business Metrics and Refining Outage Response](https://www.pagerduty.com/blog/monitoring-business-metrics/)
- 🧰 [crazy-canux/awesome-monitoring](https://github.com/crazy-canux/awesome-monitoring): monitoring tools for operations.
- [Monitoring in the time of Cloud Native](https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e)
- [How to Monitor the SRE Golden Signals](https://medium.com/faun/how-to-monitor-the-sre-golden-signals-1391cadc7524)
  - From the Google SRE book: Latency, Traffic, Errors, and Saturation
  - USE Method (from Brendan Gregg): Utilization, Saturation, and Errors
  - RED Method (from Tom Wilkie): Rate, Errors, and Duration
- [Simple Anomaly Detection Using Plain SQL](https://hakibenita.com/sql-anomaly-detection)
- [How percentile approximation works (and why it's more useful than averages)](https://www.timescale.com/blog/how-percentile-approximation-works-and-why-its-more-useful-than-averages/)
- [Implementing health checks](https://aws.amazon.com/builders-library/implementing-health-checks/)
- [IETF RFC Health Check Response Format for HTTP APIs](https://datatracker.ietf.org/doc/html/draft-inadarei-api-health-check-06)