By This Hour Development Desk

What you will learn

A Kubernetes controller cache is more than a performance detail: it defines which view of cluster state a reconciliation normally sees. Controller-runtime’s manager client reads structured objects through a shared local cache by default, while writes go directly to the API server. The client documentation describes this split and the cache-related client options in its client package source.

This tutorial gives you a repeatable workflow for making read-path choices without scattering ad hoc exceptions through a controller. You will identify the state a reconciler owns, decide whether eventual consistency is acceptable for each read, make direct reads deliberate, configure exceptions by object kind, and test both cache visibility and startup timing. The objective is not to make every read direct. It is to make the normal cached path reliable and make the exceptions explicit.

By the end, you should be able to answer three operational questions for every important read: Is this object part of the watched controller state? Can this decision safely use a cache that may not yet contain the latest API-server result? What test proves the intended path works before and after the cache is available?

Before you start

Begin with a small inventory of reads in one reconciliation path. Include Get calls for the reconciled object and related objects, List calls used to discover related state, and any read that follows a write or occurs during initialization. Record the object kind, why the result is needed, whether the controller watches that state, and what happens if the result is briefly older than the API server.

Separate the write path from the read path in your notes. A successful direct write does not mean that an immediate ordinary client read will observe that write, because the ordinary structured-object read uses the shared cache. This is the core distinction to design around, rather than a surprising test failure to patch later. Controller-runtime’s package documentation explains that its manager coordinates shared controller infrastructure; see the controller-runtime package documentation for that manager-and-controller context.

Use this decision order:

  1. Classify the read as normal reconciliation state, a freshness-sensitive decision, or a read required before the cache is running.
  2. For normal reconciliation state, prefer the cache when eventual consistency is acceptable and the relevant object belongs in watched state.
  3. For freshness-sensitive or pre-start reads, plan a direct API-server read through the manager’s API reader or another client created without Cache options.
  4. If an entire kind needs direct reads, configure that exception once and test it; do not rely on a developer remembering a special call at every site.
  5. Verify the cache’s scope, startup, synchronization, inclusion, and exclusion behavior in tests.

This inventory has a trade-off: it takes more thought than treating every client call identically. In return, a reviewer can evaluate consistency requirements at the call site and can see whether a direct read is justified rather than accidental.

Step 1: Map the default read path and mark consistency boundaries

First, make cached reading your stated baseline. For structured objects, a manager-created client normally uses the shared local cache for reads. Informers back that cache, so normal reconciliation can inspect watched state without making an API-server request for every read. Writes still go to the API server. This arrangement is appropriate when reconciliation can converge after the cache catches up.

Mark a boundary wherever the controller needs a result that must come from the API server now, or whenever the controller runs before cache startup. At those boundaries, use the manager’s API reader or a separate client that has no Cache options. Do not describe this as “better” or “safer” by default: direct reads serve a different consistency and lifecycle need. Their trade-off is that they do not use the ordinary shared cached read path.

Worked example: verifying a write that needs the latest API-server state

Scenario: A reconciler writes an object and then has a decision that cannot wait for the cached view to reflect the new API-server state.

Example: Keep ordinary reconciliation reads on the manager client, but make the freshness-sensitive verification an explicit API-reader operation.

Before: manager client reads structured object after write
client.Get(ctx, key, object)

After: direct API-server read for verification
apiReader.Get(ctx, key, object)

What this shows: The change is not a replacement of the controller’s normal read model. It isolates one read whose purpose requires a direct API-server query.

In your implementation review, name the condition that makes the second line necessary: “latest state required after write” is testable, whereas “cache seemed stale” is not a durable rule. Then check that the rest of the reconciler still uses cached state where convergence is acceptable. This avoids turning one exceptional requirement into a broad increase in direct reads.

Also inspect the lifecycle. If code can execute before cache startup, a cache-backed result is not a valid prerequisite for that code path. A direct reader is appropriate there for the same reason: the required cache state is not yet available. The controller-runtime FAQ discusses practical cache behavior and is useful background when reviewing these boundaries: controller-runtime FAQ.

Step 2: Choose the read mechanism from the controller’s state contract

For each inventory entry, write a short state contract. A cache-backed read is a good fit when the object is part of the controller’s watched state and the reconciliation can tolerate eventual consistency. That is the common case: a reconcile triggered by watched state works against the local informer-backed view and can run again as state changes.

A direct read is a good fit when the correctness of the immediate decision depends on querying the API server, or when the cache is not available yet. Keep the contract narrow. “This kind sometimes changes” is not enough, because all cached state can lag. Instead, specify the decision and its timing requirement. This makes it possible to check whether a future refactor has quietly moved a direct-read requirement into an ordinary cached helper.

Worked example: separating convergence from immediate verification

Scenario: One reconciliation reads a related object to compute desired state, then performs a separate check that requires the newest persisted state.

Example: Document the two reads as different contracts rather than assuming the same client should perform both.

Read A
Purpose: compute desired state
Contract: watched state; eventual consistency accepted
Path: shared cache

Read B
Purpose: immediate persisted-state check
Contract: newest API-server state required
Path: API reader

What this shows: Both reads may concern related controller work, yet their required consistency differs. The first supports normal convergence; the second has an explicit immediate-state requirement.

Check this choice by asking what occurs if Read A returns an older answer. If another reconcile can correct the result, the cached path fits. Then ask what occurs if Read B returns an older answer. If the immediate check would be invalid, retain the direct path and add a test that distinguishes the two behaviors.

Common mistakes to avoid

  • Assuming a direct write makes the next cached read current. Writes go to the API server, while normal structured reads use the cache. Treat those as separate paths.
  • Using direct reads as a blanket response to uncertainty. This removes the clear normal-path contract and makes it harder to tell which decisions truly require API-server freshness.
  • Forgetting cache scope. A filtered cache deliberately reduces watched state. A cache-backed Get or List can therefore return no object when that resource was excluded by the filter, as described in the cache selector design.
  • Testing only a successful cached lookup. A scoped cache also needs a negative test for excluded state; otherwise a future filter change can alter controller visibility without being detected.

Step 3: Configure exceptions and test cache lifecycle behavior

Configure exceptions at the client boundary when an object kind should consistently bypass the cache. Controller-runtime exposes client.CacheOptions.DisableFor for specific kinds. This is preferable to relying on many individual call sites to remember that a kind must be read directly. The trade-off is intentional: that kind no longer follows the shared cached structured-read path, so make the reason visible in configuration and in tests.

Handle unstructured objects separately. Unstructured reads are direct by default unless CacheOptions.Unstructured is enabled. Treat that default as part of your read inventory, not as an implementation detail. If you enable cached unstructured reads, add the same scope and synchronization checks you would require for structured cached reads.

Worked example: making a per-kind bypass reviewable

Scenario: A controller has determined that one object kind always needs direct reads, while its ordinary structured reconciliation state should remain cache-backed.

Example: Express the exception as a per-kind cache option and preserve the default behavior for other structured kinds.

Before
CacheOptions.DisableFor: none
Structured reads: shared cache

After
CacheOptions.DisableFor: [Kind requiring direct reads]
That kind: direct reads
Other structured kinds: shared cache

What this shows: The bypass is an intentional, centrally reviewable policy. It does not redefine how every kind is read.

Check the configuration with a focused test matrix. For a normally cached included object, confirm a cache-backed Get or List finds it after the cache has started and synchronized. For an object excluded by a cache filter, confirm those cache-backed calls return no object. Then test the bypass kind through the configured client and test an unstructured read according to whether cached unstructured objects were enabled. These paired assertions validate both the desired visibility and the intended absence of visibility.

Finally, test startup as a lifecycle condition. Controller-runtime starts the manager cache before controllers, and its own test coverage includes cache startup and missing-informer cases; the project’s discussion of this behavior is also visible in its cache-startup issue discussion. Your tests should not assume a cache result before startup or before synchronization. Arrange the test so startup and synchronization are complete before asserting normal cached visibility; separately exercise any code designed to read directly before that point.

Pre-publish checklist

  • Every important Get and List has a stated cached or direct-read reason.
  • Normal reconciliation state uses the cache only where eventual consistency is acceptable.
  • Every direct read identifies either an immediate API-server requirement or a pre-cache-start requirement.
  • Each DisableFor kind has a documented reason and a test.
  • Unstructured behavior is explicit: direct by default, or deliberately cached with CacheOptions.Unstructured.
  • Filtered-cache tests cover an included object and an excluded object.
  • Tests wait for cache startup and synchronization before expecting normal cached visibility.
  • Tests cover the direct-read path independently, rather than inferring it from a cached-read success.

Repeat this workflow whenever you add a watched kind, narrow cache filters, introduce unstructured access, or add a post-write verification. It keeps the Kubernetes controller cache a deliberate part of your controller’s consistency model rather than an invisible source of timing-dependent behavior.

Sources