DevOps & SRE notes

Pipedrive Infra manages numerous Kubernetes clusters across different clouds, including AWS and on-premise OpenStack. They had been experiencing intermittent failing pod health checks, which became more frequent over time. After an extensive investigation, the team discovered that Kubelet was initiating TCP sessions to pods using random source ports within the same range reserved by Kubernetes nodeports. This caused the TCP SYN-ACK to be redirected to other pods, leading to failed health checks. The solution was to disallow the use of the nodeport range as the source port for TCP sessions with a single line of code, effectively resolving the issue. https://medium.com/pipedrive-engineering/solving-the-mystery-of-pods-health-checks-failures-in-kubernetes-55b375493d03

In this post, the author discusses potential PostgreSQL pitfalls that may not affect small databases, but can cause issues when databases grow. https://philbooth.me/blog/nine-ways-to-shoot-yourself-in-the-foot-with-postgresql

Streaming alert evaluation offers better scalability than traditional polling time-series databases, overcoming high dimensionality/cardinality limitations. This enables engineers to have more reliable and real-time alerting systems. The transition to the streaming path has opened doors for supporting more exciting use-cases and has allowed multiple platform teams at Netflix to generate and maintain alerts programmatically without affecting other users. The streaming paradigm may help tackle correlation problems in observability and offer new opportunities for metrics and events verticals, such as logs and traces. https://netflixtechblog.com/improved-alerting-with-atlas-streaming-eval-e691c60dc61e

A new terraform version has been released. Import already existed infrastructure to the terraform state become easier. https://www.hashicorp.com/blog/terraform-1-5-brings-config-driven-import-and-checks

Don't forget about security https://dzone.com/articles/container-security-top-5-best-practices-for-devops

In the second part of the DevOps project, the focus is on deploying monitoring tools like ArgoCD, Prometheus, and Grafana to a Kubernetes cluster. The blog post covers installing ArgoCD, deploying Prometheus using Helm charts, setting up monitoring for ArgoCD, visualizing ArgoCD metrics using Grafana dashboards, and continuous deployment of applications using ArgoCD. A useful tool, K8sgpt, is recommended to analyze the cluster for errors and potential issues. The next blog post will discuss configuring Alert Manager for notifications, setting up Slack alerts, and installing Loki for logs, enhancing the monitoring solution. https://blog.devgenius.io/optimizing-kubernetes-deployments-with-argocd-and-prometheus-aa86c11e2bba

🎨 Diagram as Code for prototyping cloud system architectures https://github.com/mingrammer/diagrams

🔥 Open source static (serverless) status page. Uses hyperfast Go & Hugo, minimal HTML/CSS/JS, customizable, outstanding browser support (IE8+), preloaded CMS, read-only API, badges & more. https://github.com/cstate/cstate

Nothin can be free forever or the story how Oracle take back a free cloud vms https://armin.su/oracle-cloud-and-loss-of-data-in-kubernetes-cluster-198d88181829

This post provides a guide to configuring and installing a multi-cluster observability solution for cloud computing environments like AWS, Azure, and Google Cloud. The solution includes Grafana, Prometheus, Thanos, and Loki for monitoring applications and microservices in multi-cluster environments. The guide assumes prior experience with AWS S3, Policy, IAM, EKS, and Kubernetes. It covers the creation of IAM policies and roles, the installation of Helm, Bitnami's Helm charts, and EKS, AWS CLI, eksctl, and kubectl tools. The guide details the process of setting up multi-cluster observability with metrics monitoring using kube-prometheus and Thanos and log monitoring using Grafana Loki and Promtail. https://medium.com/@bahungxt/multi-cluster-observability-solution-with-prometheus-thanos-loki-and-grafana-5d5be42635e8

K8sGPT gives Kubernetes Superpowers to everyone k8sgpt is a tool for scanning your kubernetes clusters, diagnosing and triaging issues in simple english. It has SRE experience codified into it’s analyzers and helps to pull out the most relevant information to enrich it with AI. https://k8sgpt.ai/

This article explores Kubernetes Resource Manager and the Google Config Connector, comparing them to Terraform, a popular infrastructure orchestration tool. Kubernetes, an open-source container orchestration tool, has gained market dominance with its Custom Resource Definitions (CRDs), which allows managing Google Cloud resources through Kubernetes using CRDs. Config Connector, an add-on to Kubernetes, can potentially replace Terraform in some workflows. However, the author's experiment shows that while Config Connector can be used to deploy a Google Cloud landing zone, it has limitations compared to Terraform, particularly in handling interdependencies based on values unknown until a resource is created. In conclusion, the author suggests a hybrid approach, with Terraform for platform-centric deployments and Config Connector for application-centric deployments. While Terraform's flexibility and provider support make it useful for organizations operating in multiple clouds, Config Connector has a compelling place in application-centric deployments where small amounts of infrastructure are deployed in support of Kubernetes-based services. https://medium.com/cts-technologies/are-terraforms-days-numbered-a9a15ec0435a

This blog post discusses the growing trend of Large Language Models (LLMs) and their impact on various use cases. One specific application discussed is K8sGPT, an AI-based Site Reliability Engineer (SRE) that runs inside Kubernetes clusters. It scans, diagnoses, and triages issues using SRE experience codified into its analyzers. LocalAI, another project, is a drop-in replacement API for local CPU inferencing. Combining K8sGPT and LocalAI enables powerful SRE capabilities without relying on expensive GPUs. https://itnext.io/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65