DevOps & SRE notes

Automatically cordon and drain Kubernetes nodes based on node conditions https://github.com/planetlabs/draino

Excellent article on how services work in k8s https://dev.to/danielepolencic/learn-why-you-cant-ping-a-kubernetes-service-3nlm

Steampipe is an open-source tool that lets you instantly query cloud services like AWS, Azure and GCP with SQL. With 100+ plugins and built-in support for creating dashboards, Steampipe makes it trivial to connect live cloud configuration data with internal or external data sets and create security or compliance dashboards. We've enjoyed working with Steampipe and created several such dashboards with AWS cloud configurations. https://github.com/turbot/steampipe

Gitleaks is an open-source SAST (static application security testing) command line tool for detecting and preventing hardcoded secrets like passwords, API keys and tokens in Git repositories. It can be used as a Git pre-commit hook or in the CI/CD pipeline. Our teams found Gitleaks to be more sensitive than some of the other secret-scanning tools. Gitleaks utilizes regular expressions and entropy string coding to detect secrets. In our experience, the flexibility to supply custom regex along with entropy coding allowed the teams to better categorize secrets based on their needs. For example, instead of categorizing all API keys as "generic-api-key," it allowed categorization as specific "cloud provider key." https://github.com/gitleaks/gitleaks

Enterprises now often use event streaming as the source of truth and as an information-sharing mechanism in microservices architectures. This creates the need to standardize event types and share those standards across the enterprise. Event schema registries are commonly deployed but the existing offerings tend to be specialized to a single broker such as Apache Kafka or Azure Event Hub. They also fall short of conveying rich documentation about event types that goes beyond simple schema definitions. EventCatalog is an open-source project that provides something we often see businesses building for themselves: a widely accessible repository of documentation for events and schemas. These describe the role the events play in the business, where they belong in a business domain model and which services subscribe and publish them. If you're looking for a way to publish event documentation to your organization, this tool might save you the trouble of building it yourself. https://github.com/boyney123/eventcatalog

Within any organization, API producers and consumers need to stay in sync about the schemas that will be used for communication among them. Especially as the number of APIs and related producers and consumers grow in the organization, what may start with simply passing around schemas among teams will start to hit scaling challenges An API/Schema registry - stores APIs and Schemas. https://github.com/apicurio/apicurio-registry

Good guide for begginers about IAM https://dev.to/aws-builders/a-beginners-guide-to-aws-identity-and-access-management-iam-4j5c

Interesting article describing how dns query works in k8s https://www.nslookup.io/learning/the-life-of-a-dns-query-in-kubernetes/

modern full-featured open source secure mail server for low-maintenance self-hosted email https://github.com/mjl-/mox

Kubernetes v1.25 has introduced the Container Checkpointing API as an alpha feature, allowing users to backup and restore containers without stopping them. This feature is primarily aimed at forensic analysis but can also be used for general backup and restore purposes. To set up the feature, a Kubernetes cluster (v1.25+) and container runtime supporting container checkpointing are required. Currently, only CRI-O supports checkpointing, with containerd support expected soon. The checkpointing API is exposed on the kubelet of each cluster node. To create a checkpoint, you need to have a running Pod and make a request to the kubelet directly. Once the checkpoint has been created, you can analyze the contents of the archive or restore the container from the archive by creating an image from the checkpoint and deploying a new Pod using that image. While the feature is usable, it lacks some essential functionality, such as native restore capabilities and support from all major container runtimes. Users are advised to be aware of its limitations before enabling it in production or development environments. https://martinheinz.dev/blog/85

An extremely fast Python linter, written in Rust. https://github.com/charliermarsh/ruff

Interesting article about experience with linkerd https://tech.loveholidays.com/linkerd-at-loveholidays-our-journey-to-a-production-service-mesh-9a6cd478d395

Exec into node via kubectl https://github.com/kvaps/kubectl-node-shell

In this blog post, Ahmet Alp Balkan explains the peculiar and undocumented behavior of file changes in Kubernetes Secret and ConfigMap volumes when using the inotify(7) syscall. He highlights that typical file watch events like IN_MODIFY or IN_CLOSE_WRITE don't occur for files in these volumes. Instead, only the IN_DELETE_SELF event is received, requiring code to handle re-establishing the monitor each time a file is updated. Balkan discusses the resilient file reloads from disk and the AtomicWriter algorithm used by kubelet for atomic and consistent updates to Secret/ConfigMap volumes. He explains the file structure in a mounted Secret/ConfigMap volume and the reason behind receiving only the IN_DELETE_SELF event. To handle this behavior, Balkan suggests mounting ConfigMaps/Secrets as directories, starting inotify watches on individual files, avoiding the use of IN_DONT_FOLLOW option, handling inotify deletion events, re-establishing inotify watches when receiving deletion events, and testing the file reloading logic on Kubernetes. He also mentions opening an issue to document this behavior in the official Kubernetes documentation. https://ahmet.im/blog/kubernetes-inotify/index.html

A web-based UI for deploying and managing applications in Kubernetes clusters https://github.com/vmware-tanzu/kubeapps

A new alternative to DockerDesktop was released by RedHat https://developers.redhat.com/articles/2023/05/23/podman-desktop-now-generally-available#local_kubernetes_with_kind

Spin up ready-to-code, disposable dev environments on your own servers. Self-hosted alternative to Gitpod and Github Codespaces. https://github.com/hocus-dev/hocus