feat: add Kubernetes leader election for multi-replica operator#133
Conversation
Implement leader election using Kubernetes Lease resources, aligned with client-go leaderelection semantics. The implementation is in a standalone crate (kube-leader-election) with Lock trait abstraction and structured concurrency. Key changes: - crates/leader-election/: new crate with elector, LeaseLock, callbacks - src/lib.rs: integrate LeaderElector with controller lifecycle - src/main.rs: add CLI flags (--leader-elect, --leader-elect-lease-name, etc.) - deploy/: add Lease RBAC and POD_NAME downward API env - crates/leader-election/README.md: crate-level documentation - 18 tests (10 unit + 8 integration), all passing
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9d969e0ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ATOR_NAMESPACE - Abort controller_handle when lease is lost to prevent multiple active controllers after transient API failure (P1 fix from PR review) - Move OPERATOR_NAMESPACE env var outside sts.enabled block so leader election uses correct namespace even when STS is disabled (P2 fix)
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c61aeef7d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| _ = &mut controller_handle => { | ||
| info!("controller finished"); | ||
| } |
There was a problem hiding this comment.
When leader election is enabled, if the controller task ever returns or panics (for example because the kube controller stream terminates), this branch lets on_started_leading return while the elector keeps renewing the Lease until renewal fails or the pod is killed. That leaves the elected pod holding leadership but no longer running reconciliation, so standby replicas cannot take over. Treat this branch as fatal/cancel leadership instead of just logging and returning.
Useful? React with 👍 / 👎.
1 participant
Type of Change
Related Issues
Summary of Changes
This PR now includes follow-up fixes to make leader-election behavior topology-safe and deployment-safe, and to keep CI pre-commit checks green.
1) Leader election race/stop improvements (already in codebase)
LeaseLock::update()now holds cache lock across read/build/update and clears cache on conflict.on_stopped_leading()is only emitted on final exit, not every transient renew-loss cycle.value_fromenv vars for parity-related env keys; it fails fast with explicit error.2) Helm/dev deployment leader election behavior
leader_elect=false, Helm and dev manifests now explicitly control leader election.--leader-electautomatically:operator.replicas > 1operator.replicas <= 1operator.leaderElectoperator.leaderElecttodeploy/rustfs-operator/values.yamland README config table.deploy/rustfs-operator/README.mdfor auto-enable and explicit override.deploy/k8s-dev/operator-deployment.yamlexplicitly sets--leader-elect=false.3) Fix
make pre-commite2e formatting blockere2e/Cargo.tomlis explicitly marked as an isolated workspace by adding[workspace].e2e/Cargo.lockaccordingly.Verification
make pre-commitAdditional Notes