Overview

The kubelet command-line has 177 parameters, making it the Kubernetes component with the most configuration options.

Kubelet’s options can be categorized into three types: kubeletFlags (those that cannot be changed after startup and can only be configured via the command line, such as IP addresses and paths), kubeletConfiguration (those that can be modified after startup and can be set via the command line, configuration files, or dynamic Kubelet config - configmap; configuring via the command line is deprecated, and some options can only be set through configuration files, e.g., NodeStatusReportFrequency), and globalFlags (those that can only be configured via the command line, such as log levels, log paths, and versions).

How does kubelet start, initialize, and recognize so many options? This article focuses on the command-line initialization of kubelet. The runtime process of various components will be analyzed in subsequent articles.

This article is based on Kubernetes version 1.18.6. Please visit the source code reading repository for more details.

1 Startup Process

The entry file for kubelet is cmd\kubelet\kubelet.go. The main process includes:

1. Initializing the random factor.
2. Creating a Cobra Command.
3. Initializing logging.
4. Executing the Cobra Command.

2 Creating a Cobra Command

During this stage, command-line options are defined, default values are set, various fields of the command are defined, command help functions are created, and command usage functions are defined.

The main process includes:

• Creating a flagset named “kubelet” to have a clean flagset (to avoid contamination by the CommandLine flagset) and setting the NormalizeFunc of the flagset to convert flag names containing underscores “_” to hyphens “-”. For example, –iptables-drop-bit and –iptables_drop_bit are considered the same.

• Creating a “kubeletFlags” structure that contains default values for some initial options. Specific field-to-command line option mappings can be found in the “kubeletFlags options” section below. The “NewKubeletFlags()” function is defined in cmd\kubelet\app\options\options.go.

  go

  // NewKubeletFlags will create a new KubeletFlags with default values
  func NewKubeletFlags() *KubeletFlags {
      remoteRuntimeEndpoint := ""
      if runtime.GOOS == "linux" {
          remoteRuntimeEndpoint = "unix:///var/run/dockershim.sock"
      } else if runtime.GOOS == "windows" {
          remoteRuntimeEndpoint = "npipe:////./pipe/dockershim"
      }
  
      return &KubeletFlags{
          EnableServer:                        true,
          ContainerRuntimeOptions:             *NewContainerRuntimeOptions(),
          CertDirectory:                       "/var/lib/kubelet/pki",
          RootDirectory:                       defaultRootDir, // defaultRootDir为/var/lib/kubelet
          MasterServiceNamespace:              metav1.NamespaceDefault, // NamespaceDefault为default
          MaxContainerCount:                   -1,
          MaxPerPodContainerCount:             1,
          MinimumGCAge:                        metav1.Duration{Duration: 0},
          NonMasqueradeCIDR:                   "10.0.0.0/8",
          RegisterSchedulable:                 true,
          ExperimentalKernelMemcgNotification: false,
          RemoteRuntimeEndpoint:               remoteRuntimeEndpoint,
          NodeLabels:                          make(map[string]string),
          VolumePluginDir:                     "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/",
          RegisterNode:                        true,
          SeccompProfileRoot:                  filepath.Join(defaultRootDir, "seccomp"),
          // prior to the introduction of this flag, there was a hardcoded cap of 50 images
          NodeStatusMaxImages:         50,
          EnableCAdvisorJSONEndpoints: false,
      }
  }

  NewContainerRuntimeOptions() is defined in cmd\kubelet\app\options\container_runtime.go

  go

  const (
      // When these values are updated, also update test/utils/image/manifest.go
      defaultPodSandboxImageName    = "k8s.gcr.io/pause"
      defaultPodSandboxImageVersion = "3.2"
  )
  
  var (
      defaultPodSandboxImage = defaultPodSandboxImageName +
          ":" + defaultPodSandboxImageVersion
  )
  
  // NewContainerRuntimeOptions will create a new ContainerRuntimeOptions with
  // default values.
  func NewContainerRuntimeOptions() *config.ContainerRuntimeOptions {
      dockerEndpoint := ""
      if runtime.GOOS != "windows" {
          dockerEndpoint = "unix:///var/run/docker.sock"
      }
  
      return &config.ContainerRuntimeOptions{
          ContainerRuntime:           kubetypes.DockerContainerRuntime, //DockerContainerRuntime为docker
          RedirectContainerStreaming: false,
          DockerEndpoint:             dockerEndpoint,
          DockershimRootDirectory:    "/var/lib/dockershim",
          PodSandboxImage:            defaultPodSandboxImage,
          ImagePullProgressDeadline:  metav1.Duration{Duration: 1 * time.Minute},
          ExperimentalDockershim:     false,
  
          //Alpha feature
          CNIBinDir:   "/opt/cni/bin",
          CNIConfDir:  "/etc/cni/net.d",
          CNICacheDir: "/var/lib/cni/cache",
      }
  }

• Create a kubeletConfiguration to set default values for flag options related to kubeletConfiguration. This is done by creating a v1beta1 KubeletConfiguration and generating the default configuration through scheme.Default(), which is then converted into the internal KubeletConfiguration.

  Specific command-line options corresponding to fields can be found in the “kubeletConfigurationFlag options” section below.

  go

   KubeletConfiguration{
  		SyncFrequency: metav1.Duration{Duration: 1 * time.Minute},
  		FileCheckFrequency: metav1.Duration{Duration: 20 * time.Second},
  		HTTPCheckFrequency: metav1.Duration{Duration: 20 * time.Second},
  		Address: "0.0.0.0",
  		Port: 10250,
  		Authentication.Anonymous.Enabled: true, // by applyLegacyDefaults, by default false,
  		Authentication.Webhook.Enabled: false, // by applyLegacyDefaults, by default true,
  		Authentication.Webhook.CacheTTL: metav1.Duration{Duration: 2 * time.Minute},
  		Authorization.Mode: "AlwaysAllow", // by applyLegacyDefaults, by default "webhook",
  		Authorization.Webhook.CacheAuthorizedTTL: metav1.Duration{Duration: 5 * time.Minute},
  		Authorization.Webhook.CacheUnauthorizedTTL: metav1.Duration{Duration: 30 * time.Second},
  		ReadOnlyPort: 10255, // by applyLegacyDefaults,
  		RegistryPullQPS: 5,
  		RegistryBurst: 10,
  		EventRecordQPS: 5,
  		EventBurst: 10,
  		EnableDebuggingHandlers: true,
  		HealthzPort: 10248,
  		HealthzBindAddress: "127.0.0.1",
  		OOMScoreAdj: -999,
  		StreamingConnectionIdleTimeout: metav1.Duration{Duration: 4 * time.Hour},
  		NodeStatusReportFrequency: metav1.Duration{Duration: 5 * time.Minute},
  		NodeStatusUpdateFrequency: metav1.Duration{Duration: 10 * time.Second},
  		NodeLeaseDurationSeconds: 40,
  		ImageMinimumGCAge: metav1.Duration{Duration: 2 * time.Minute},
  		ImageGCHighThresholdPercent: 85,
  		ImageGCLowThresholdPercent: 80,
  		VolumeStatsAggPeriod: metav1.Duration{Duration: time.Minute},
  		CgroupsPerQOS: true,
  		CgroupDriver: "cgroupfs",
  		CPUManagerPolicy: "none",
  		CPUManagerReconcilePeriod: metav1.Duration{Duration: 10 * time.Second},
  		TopologyManagerPolicy: "none",
  		RuntimeRequestTimeout: metav1.Duration{Duration: 2 * time.Minute},
  		HairpinMode: "promiscuous-bridge",
  		MaxPods: 110,
  		PodPidsLimit: -1,
  		ResolverConfig: "/etc/resolv.conf",
  		CPUCFSQuota: true,
  		CPUCFSQuotaPeriod: metav1.Duration{Duration: 100 * time.Millisecond},
  		MaxOpenFiles: 1000000,
  		ContentType: "application/vnd.kubernetes.protobuf",
  		KubeAPIQPS: 5,
  		KubeAPIBurst: 10,
  		SerializeImagePulls: true,
  		EvictionHard: map[string]string{
  			"memory.available":  "100Mi",
  			"nodefs.available":  "10%",
  			"nodefs.inodesFree": "5%",
  			"imagefs.available": "15%",
  		},
  		EvictionPressureTransitionPeriod: metav1.Duration{Duration: 5 * time.Minute},
  		EnableControllerAttachDetach: true,
  		MakeIPTablesUtilChains: true,
  		IPTablesMasqueradeBit: 14,
  		IPTablesDropBit: 15,
  		FailSwapOn: true,
  		ContainerLogMaxSize: "10Mi",
  		ContainerLogMaxFiles: 5,
  		ConfigMapAndSecretChangeDetectionStrategy: "Watch",
  		EnforceNodeAllocatable: []string{"pods"},
  	}

• Create a Cobra Command, defining fields such as Use, Long, DisableFlagParsing, and Run.

• Add kubeletFlags options to the flagset.

• Add kubeletConfigurationFlag options to the flagset.

• Add globalFlags to the flagset, which includes options related to klog, verflag, CAdvisor, and logflush.

• Set the help option.

• Customize the command’s helpfunc and usageFunc.

You may notice that kubelet does not use Cobra’s command parsing, definition of helpfunc and usageFunc. Why is that?

This is because the globalFlags, which define flag options, are registered under pflag.Command - automatically registered through imported modules. By default, Cobra’s helpfunc and usageFunc, as well as execute, execute mergePersistentFlags(), which imports flags from pflag.Command. Due to historical reasons, some unnecessary options are also registered under pflag.Command and would be included. Therefore, a new flagset is created to parse the command manually and keep it clean.

The code comments also mention this, although not in detail.

	// keep cleanFlagSet separate, so Cobra doesn't pollute it with the global flags
	kubeletFlags.AddFlags(cleanFlagSet)
	options.AddKubeletConfigFlags(cleanFlagSet, kubeletConfig)
	options.AddGlobalFlags(cleanFlagSet)
	cleanFlagSet.BoolP("help", "h", false, fmt.Sprintf("help for %s", cmd.Name()))

	// ugly, but necessary, because Cobra's default UsageFunc and HelpFunc pollute the flagset with global flags
	const usageFmt = "Usage:\n  %s\n\nFlags:\n%s"
	cmd.SetUsageFunc(func(cmd *cobra.Command) error {
		fmt.Fprintf(cmd.OutOrStderr(), usageFmt, cmd.UseLine(), cleanFlagSet.FlagUsagesWrapped(2))
		return nil
	})
	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
		fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine(), cleanFlagSet.FlagUsagesWrapped(2))
	})


// AddGlobalFlags explicitly registers flags that libraries (glog, verflag, etc.) register
// against the global flagsets from "flag" and "github.com/spf13/pflag".
// We do this in order to prevent unwanted flags from leaking into the Kubelet's flagset.
func AddGlobalFlags(fs *pflag.FlagSet) {

2.1 kubeletFlags options

The kubeletFlags options include options that cannot be changed while the kubelet is running, options that cannot be safely changed, and options that cannot be shared between different instances.

KubeletConfiguration defines options that can be shared between different instances.

// KubeletFlags contains configuration flags for the Kubelet.

// A configuration field should go in KubeletFlags instead of KubeletConfiguration if any of these are true:

// - its value will never, or cannot safely be changed during the lifetime of a node, or

// - its value cannot be safely shared between nodes at the same time (e.g. a hostname);

// KubeletConfiguration is intended to be shared between nodes.

2.2 kubeletConfigurationFlag options

kubeletConfigurationFlag options are divided into two categories. The first category, “deprecated command line,” can be configured in the command line (but is deprecated), in configuration files, and in dynamic kubelet configuration. The second category, “configuration,” cannot be configured in the command line and can only be set in configuration files and dynamic kubelet configuration.

2.3 GlobalFlags

globalFlags includes options related to klog, CAdvisor, CredentialProvider, and logflush.

The klog options are defined using klog.InitFlags.

// InitFlags is for explicitly initializing the flags.
func InitFlags(flagset *flag.FlagSet) {
	if flagset == nil {
		flagset = flag.CommandLine
	}

	flagset.StringVar(&logging.logDir, "log_dir", logging.logDir, "If non-empty, write log files in this directory")
	flagset.StringVar(&logging.logFile, "log_file", logging.logFile, "If non-empty, use this log file")
	flagset.Uint64Var(&logging.logFileMaxSizeMB, "log_file_max_size", logging.logFileMaxSizeMB,
		"Defines the maximum size a log file can grow to. Unit is megabytes. "+
			"If the value is 0, the maximum file size is unlimited.")
	flagset.BoolVar(&logging.toStderr, "logtostderr", logging.toStderr, "log to standard error instead of files")
	flagset.BoolVar(&logging.alsoToStderr, "alsologtostderr", logging.alsoToStderr, "log to standard error as well as files")
	flagset.Var(&logging.verbosity, "v", "number for the log level verbosity")
	flagset.BoolVar(&logging.skipHeaders, "add_dir_header", logging.addDirHeader, "If true, adds the file directory to the header") //这里应该是&logging.addDirHeader,在下一个klog版本v2.0.0修复了,所有的1.18版本都会有这个问题
	flagset.BoolVar(&logging.skipHeaders, "skip_headers", logging.skipHeaders, "If true, avoid header prefixes in the log messages")
	flagset.BoolVar(&logging.skipLogHeaders, "skip_log_headers", logging.skipLogHeaders, "If true, avoid headers when opening log files")
	flagset.Var(&logging.stderrThreshold, "stderrthreshold", "logs at or above this threshold go to stderr")
	flagset.Var(&logging.vmodule, "vmodule", "comma-separated list of pattern=N settings for file-filtered logging")
	flagset.Var(&logging.traceLocation, "log_backtrace_at", "when logging hits line file:N, emit a stack trace")
}

cadvisor define

The cadvisor flags are imported through anonymous imports and can be found in the code saved under vendor/github.com/google/cadvisor.

// ensure libs have a chance to globally register their flags
	_ "github.com/google/cadvisor/container/common"
	_ "github.com/google/cadvisor/container/containerd"
	_ "github.com/google/cadvisor/container/docker"
	_ "github.com/google/cadvisor/container/raw"
	_ "github.com/google/cadvisor/machine"
	_ "github.com/google/cadvisor/manager"
	_ "github.com/google/cadvisor/storage"

For example, taking github.com/google/cadvisor/container/docker as an example, it defines the following options.

var ArgDockerEndpoint = flag.String("docker", "unix:///var/run/docker.sock", "docker endpoint")
var ArgDockerTLS = flag.Bool("docker-tls", false, "use TLS to connect to docker")
var ArgDockerCert = flag.String("docker-tls-cert", "cert.pem", "path to client certificate")
var ArgDockerKey = flag.String("docker-tls-key", "key.pem", "path to private key")
var ArgDockerCA = flag.String("docker-tls-ca", "ca.pem", "path to trusted CA")

// The namespace under which Docker aliases are unique.
const DockerNamespace = "docker"

// Regexp that identifies docker cgroups, containers started with
// --cgroup-parent have another prefix than 'docker'
var dockerCgroupRegexp = regexp.MustCompile(`([a-z0-9]{64})`)

var dockerEnvWhitelist = flag.String("docker_env_metadata_whitelist", "", "a comma-separated list of environment variable keys that needs to be collected for docker containers")

var (
	// Basepath to all container specific information that libcontainer stores.
	dockerRootDir string

	dockerRootDirFlag = flag.String("docker_root", "/var/lib/docker", "DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker)")

	dockerRootDirOnce sync.Once

	// flag that controls globally disabling thin_ls pending future enhancements.
	// in production, it has been found that thin_ls makes excessive use of iops.
	// in an iops restricted environment, usage of thin_ls must be controlled via blkio.
	// pending that enhancement, disable its usage.
	disableThinLs = true
)

Except for --docker-root and housekeeping-interval, all other options in the cadvisor section are deprecated. Notably, the update_machine_info_interval flag is not registered with kubelet. Based on commit records, update_machine_info_interval was added when upgrading the cadvisor version to 0.33.0. It’s possible that these options are not registered with kubelet due to their deprecated status.

// These flags were also implicit from cadvisor, but are actually used by something in the core repo:
	// TODO(mtaufen): This one is stil used by our salt, but for heaven's sake it's even deprecated in cadvisor
	register(global, local, "docker_root")
	// e2e node tests rely on this
	register(global, local, "housekeeping_interval")

	// These flags were implicit from cadvisor, and are mistakes that should be registered deprecated:
	const deprecated = "This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed."

	registerDeprecated(global, local, "application_metrics_count_limit", deprecated)
	registerDeprecated(global, local, "boot_id_file", deprecated)
	registerDeprecated(global, local, "container_hints", deprecated)
	registerDeprecated(global, local, "containerd", deprecated)
	registerDeprecated(global, local, "docker", deprecated)
	registerDeprecated(global, local, "docker_env_metadata_whitelist", deprecated)
	registerDeprecated(global, local, "docker_only", deprecated)
	registerDeprecated(global, local, "docker-tls", deprecated)
	registerDeprecated(global, local, "docker-tls-ca", deprecated)
	registerDeprecated(global, local, "docker-tls-cert", deprecated)
	registerDeprecated(global, local, "docker-tls-key", deprecated)
	registerDeprecated(global, local, "enable_load_reader", deprecated)
	registerDeprecated(global, local, "event_storage_age_limit", deprecated)
	registerDeprecated(global, local, "event_storage_event_limit", deprecated)
	registerDeprecated(global, local, "global_housekeeping_interval", deprecated)
	registerDeprecated(global, local, "log_cadvisor_usage", deprecated)
	registerDeprecated(global, local, "machine_id_file", deprecated)
	registerDeprecated(global, local, "storage_driver_user", deprecated)
	registerDeprecated(global, local, "storage_driver_password", deprecated)
	registerDeprecated(global, local, "storage_driver_host", deprecated)
	registerDeprecated(global, local, "storage_driver_db", deprecated)
	registerDeprecated(global, local, "storage_driver_table", deprecated)
	registerDeprecated(global, local, "storage_driver_secure", deprecated)
	registerDeprecated(global, local, "storage_driver_buffer_duration", deprecated)

CredentialProviderFlags

In cmd\kubelet\app\options\globalflags.go, the CredentialProviderFlags are anonymously imported as follows:

	// ensure libs have a chance to globally register their flags
	_ "k8s.io/kubernetes/pkg/credentialprovider/azure"

In pkg\credentialprovider\azure\azure_credentials.go, the flagConfigFile is defined as:

var flagConfigFile = pflag.String("azure-container-registry-config", "",
	"Path to the file containing Azure container registry configuration information.")

version

The version option is defined in staging\src\k8s.io\component-base\version\verflag\verflag.go. It accepts three values: false (default), true (enabled without any parameters), and raw (displays detailed version information in JSON format). There will be a separate article dedicated to Kubernetes version-related release rules and various hacks related to versions.

func VersionVar(p *versionValue, name string, value versionValue, usage string) {
	*p = value
	flag.Var(p, name, usage)
	// "--version" will be treated as "--version=true"
	flag.Lookup(name).NoOptDefVal = "true"
}

func Version(name string, value versionValue, usage string) *versionValue {
	p := new(versionValue)
	VersionVar(p, name, value, usage)
	return p
}

const versionFlagName = "version"

var (
	versionFlag = Version(versionFlagName, VersionFalse, "Print version information and quit")
)

log options

in staging\src\k8s.io\component-base\logs\logs.go

const logFlushFreqFlagName = "log-flush-frequency"

var logFlushFreq = pflag.Duration(logFlushFreqFlagName, 5*time.Second, "Maximum number of seconds between log flushes")

2.4 help options

A special option is used to display the command’s help information, which can be triggered by -h or --help flags.

cleanFlagSet.BoolP("help", "h", false, fmt.Sprintf("help for %s", cmd.Name()))