DevOps&SRE Library

Memory leaks are a common and frustrating problem in software development. These issues arise when a program fails to free up memory that is no longer being used, leading to a gradual loss of available memory over time.

This two-article series is about monitoring. Part One covers accumulating a multitude of different metrics in a single place, handling permissions for different aspects of those metrics, and storing large amounts of data. In Part Two, we then focus on choosing monitoring systems based on the brief example of a fictional company’s “journey” in struggling with continually expanding its monitoring system and growing its infrastructure.

Cluster.dev is an open-source tool designed to manage cloud native infrastructures with simple declarative manifests - infrastructure templates. The infrastructure templates could be based on Terraform modules, Kubernetes manifests, Shell scripts, Helm charts, Kustomize and ArgoCD/Flux applications, OPA policies etc. Cluster.dev sticks those components together so that you could deploy, test and distribute a whole set of components with pinned versions.

The new Terraform version v1.6.0 introduce a test framework, named “Terraform test”. Here’s how to use it.

Or, Eleven things we have learned as Site Reliability Engineers at Google

1. The riskiness of a mitigation should scale with the severity of the outage
2. Recovery mechanisms should be fully tested before an emergency
3. Canary all changes
4. Have a "Big Red Button"
5. Unit tests alone are not enough - integration testing is also needed
6. COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!!
7. Intentionally degrade performance modes
8. Test for Disaster resilience
9. Automate your mitigations
10. Reduce the time between rollouts, to decrease the likelihood of the rollout going wrong
11. A single global hardware version is a single point of failure