Instrumentation and cAdvisor, with David Ashpole

Kubernetes Podcast from Google

Episode | Podcast

Date: Tue, 21 Jul 2020 15:10:55 +0000

<p>Released on the same day as Kubernetes, cadvisor is a container monitoring daemon that collects metrics and serves them to monitoring tools. It’s built into the Kubelet, and underpins many components in Kubernetes, such as eviction and autoscaling. <a href="https://twitter.com/k8s_dashpole">David Ashpole</a> of Google Cloud is TL of Kubernetes SIG Instrumentation, and the maintainer of cadvisor; he joins <a href="https://kubernetespodcast.com/about">Adam and Craig</a> this week to explain where instrumentation fits in the stack, and what you should do as a Kubernetes maintainer vs. a cluster administrator.</p> <p>Do you have something cool to share? Some questions? Let us know:</p> <ul> <li>web: <a href="https://kubernetespodcast.com">kubernetespodcast.com</a></li> <li>mail: <a href="mailto:kubernetespodcast@google.com">kubernetespodcast@google.com</a></li> <li>twitter: <a href="https://twitter.com/kubernetespod">@kubernetespod</a></li> </ul> <h3 id="chatter-of-the-week">Chatter of the week</h3> <ul> <li>In Craig’s neighbourhood: <ul> <li><a href="https://twitter.com/craigbox/status/1285555722546282496">Books</a></li> <li><a href="https://twitter.com/craigbox/status/1285556657569828865">More books</a></li> <li><a href="https://twitter.com/craigbox/status/1285556762901348352">Some less popular items</a></li> </ul> </li> <li><a href="https://twitter.com/craigbox/status/1285558219046346753">Masks</a> <ul> <li><a href="https://www.newsbreak.com/news/1597708175599/archie-the-mammoth-dons-mask-at-unl"> Archie the Mammoth</a></li> </ul> </li> <li><a href="https://nationaltoday.com/national-ice-cream-day/">National Ice Cream Day</a> <ul> <li><a href="https://upload.wikimedia.org/wikipedia/commons/7/71/Carmel_by_the_Sea_Coastline_%28Unsplash%29_%28cropped%29.jpg"> Carmel</a></li> <li><a href="https://images.unsplash.com/photo-1551024506-0bccd828d307?ixlib=rb-1.2.1&amp;auto=format&amp;fit=crop&amp;w=2800&amp;q=80"> Caramel</a></li> </ul> </li> </ul> <h3 id="news-of-the-week">News of the week</h3> <ul> <li>GKE Ingress features: <ul> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#associating_backendconfig_with_your_ingress"> BackendConfig CRD</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#cloud_cdn"> Cloud CDN</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#timeout"> Backend service timeout</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#draining_timeout"> Connection draining timeout</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#http_logging"> HTTP access logging</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#iap"> Identity-Aware Proxy (IAP)</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#session_affinity"> Session affinity</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#request_headers"> User-defined request headers</a></li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#cloud_armor"> Cloud Armor security policies</a> (Beta)</li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#associating_frontendconfig_with_your_ingress"> FrontendConfig CRD</a> (Beta)</li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#direct_health"> Custom GCLB health checks</a> (Beta)</li> <li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#ssl"> SSL policies</a> (Beta)</li> </ul> </li> <li><a href="https://cloud.google.com/blog/products/containers-kubernetes/exposing-services-on-gke"> Exposing services on GKE</a></li> <li><a href="https://www.openshift.com/blog/openshift-4.5-arrives-bringing-new-supported-installations"> OpenShift 4.5</a> <ul> <li><a href="https://www.openshift.com/blog/okd4-is-now-generally-available">OKD4</a></li> </ul> </li> <li><a href="https://tanzu.vmware.com/content/blog/announcing-spring-cloud-data-flow-for-kubernetes"> Spring Cloud Data Flow for Kubernetes</a> from VMware; part of the <a href="https://tanzu.vmware.com/spring-runtime">Spring Runtime</a> package</li> <li><a href="https://twitter.com/K8Spin/status/1284438986509950982">k8spin.cloud is closing</a> and <a href="https://www.k8spin.cloud/oss-projects">making their code open source</a> <ul> <li><a href="https://levelup.gitconnected.com/kubernetes-tips-all-i-need-is-a-namespace-c81dec026294"> Review of k8spin from launch</a></li> </ul> </li> <li><a href="https://github.com/jthomperoo/custom-pod-autoscaler">Custom Pod Autoscaler</a> (and <a href="https://custom-pod-autoscaler.readthedocs.io/en/stable/">docs</a>) by Jamie Thompson</li> <li><a href="https://www.tetrate.io/blog/envoy-proxy-1-15-release/">Envoy 1.15</a> round-up from Tetrate; <a href="https://www.envoyproxy.io/docs/envoy/v1.15.0/version_history/current"> release notes</a> from the team</li> <li><a href="https://www.cncf.io/blog/2020/07/14/fluent-bit-v1-5-lightweight-and-high-performance-log-processor/"> Fluent Bit 1.5</a> summary at the CNCF</li> <li><a href="https://github.com/rancher/k3d/releases/tag/v3.0.0">k3d v3.0</a> and <a href="https://k3d.io/">new web site</a></li> <li><a href="https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-creating-a-highly-available-gke-cluster/"> Best practices for creating a highly available GKE cluster</a></li> <li><a href="https://azure.microsoft.com/en-us/updates/ci-recommended-alerts/">Recommended alerts for AKS</a></li> <li><a href="https://aws.amazon.com/blogs/containers/introducing-ingress-support-in-aws-app-mesh/"> Ingress support added to AWS App Mesh</a></li> <li><a href="https://platform9.com/press/platform9-unveils-new-capabilities-for-its-freedom-growth-and-enterprise-managed-kubernetes-solutions"> Platform9 adds new apps to their Managed Kubernetes Service</a> <ul> <li><a href="https://kubernetespodcast.com/episode/088-vms-edge-and-platform9/"> Episode 88, with Madhura Maskasky</a></li> </ul> </li> <li><a href="https://github.com/kubernetes/kubernetes/issues/93032">CVE-2020-8557: Node disk DOS by writing to container /etc/hosts</a></li> <li><a href="https://github.com/kubernetes/kubernetes/issues/92914">CVE-2020-8559: Privilege escalation from compromised node to cluster</a> <ul> <li><a href="https://blog.alcide.io/new-kubernetes-api-server-vulnerability-enables-privileges-escalation-cve-2020-8559"> Alcide write-up</a></li> </ul> </li> <li><a href="https://blog.aquasec.com/malicious-container-image-docker-container-host"> Threat Alert: Attacker Building Malicious Images Directly on Your Host</a> from Aqua Security</li> <li><a href="https://www.cncf.io/blog/2020/07/15/certified-kubernetes-security-specialist-cks-coming-in-november/"> Certified Kubernetes Security Specialist (CKS) coming in November</a></li> <li><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/register/"> Sign up for a free pass to Virtual KubeCon EU keynotes</a></li> <li><a href="https://blog.christianposta.com/diving-into-istio-1-6-certificate-rotation/"> Diving Into Istio 1.6 Certificate Rotation</a> by Christian Posta</li> </ul> <h3 id="links-from-the-interview">Links from the interview</h3> <ul> <li><a href="https://github.com/kubernetes/community/tree/master/sig-instrumentation"> SIG Instrumentation</a></li> <li><a href="https://en.wikipedia.org/wiki/Inode">inodes</a> <ul> <li><a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#eviction-signals"> Eviction on inodes</a></li> </ul> </li> <li><a href="https://en.wikipedia.org/wiki/Cgroups">cgroups</a></li> <li><a href="https://github.com/google/cadvisor">cadvisor</a> <ul> <li><a href="https://cloudplatform.googleblog.com/2014/06/an-update-on-container-support-on-google-cloud-platform.html"> Launched on the same day as Kubernetes</a></li> </ul> </li> <li><a href="https://prometheus.io/docs/guides/cadvisor/">Monitoring metrics with Prometheus</a></li> <li><a href="https://twitter.com/v_marmol">Victor Marmol</a> and <a href="https://www.linkedin.com/in/vishnukanan/">Vish Kannan</a></li> <li><a href="https://kubernetespodcast.com/episode/022-sig-node/">Episode 22, with Dawn Chen</a></li> <li><a href="https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/"> CRI</a></li> <li><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/"> Resource metrics pipeline</a></li> <li><a href="https://github.com/kubernetes-retired/heapster">Heapster</a></li> <li><a href="https://github.com/kubernetes-sigs/metrics-server">Metrics Server</a></li> <li><a href="https://github.com/kubernetes/kube-state-metrics">kube-state-metrics</a> <ul> <li><a href="https://karlstoney.com/2018/07/07/managing-your-costs-on-kubernetes/"> Managing Your Costs on Kubernetes</a> by Karl Stoney from Autotrader</li> <li><a href="https://kubernetespodcast.com/episode/052-autotrader/">Episode 52, with Russell Warman and Karl Stoney</a></li> </ul> </li> <li><a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20191028-metrics-stability-to-beta.md"> Metrics Stability Framework</a></li> <li><a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging"> Structured logging</a></li> <li><a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/647-apiserver-tracing"> Distributed tracing in Kubernetes</a></li> <li><a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-oom-behavior"> Node out of memory eviction</a></li> <li><a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/"> Pod priority</a></li> <li><a href="https://twitter.com/k8s_dashpole">David Ashpole on Twitter</a></li> </ul>