Skip to content

Performance regression: v5 is 2–7× slower than v3.8.0 for local evaluation #198

@gagantrivedi

Description

@gagantrivedi

Benchmarked flagsmith 5.0.1 vs 3.8.0 against the same environment (edge API, enable_local_evaluation=True, 262 features, single-threaded, after 200-iter warmup).

p50 per call

Operation v3.8.0 v5.0.1 Regression
get_identity_flags 0.39 ms 0.82 ms 2.1×
get_environment_flags 0.11 ms 0.79 ms 7.4×
is_feature_enabled (cached Flags) 0.08 µs 0.08 µs

Throughput drops from ~2,570 → ~1,215 identity evals/sec per core. Latency variance also grew (stdev ~10 µs → ~220 µs).

Repro

from time import perf_counter
from flagsmith import Flagsmith

client = Flagsmith(
    environment_key="<server-side-key>",
    api_url="https://edge.api.flagsmith.com/api/v1/",
    enable_local_evaluation=True,
    environment_refresh_interval_seconds=3600,
)
client.update_environment()

# warmup
for _ in range(200):
    client.get_identity_flags(identifier="anonymous", traits={"venue_id": "12345"})

for n in (100, 1_000, 10_000):
    t0 = perf_counter()
    for _ in range(n):
        client.get_identity_flags(identifier="anonymous", traits={"venue_id": "12345"})
    elapsed = perf_counter() - t0
    print(f"{n:>6} iters: {elapsed*1000:8.1f} ms ({elapsed/n*1e6:7.2f} µs/call)")

Environment: flag-engine 10.0.1, Python 3.12, macOS.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions