ci: fix Skin Designer device DB scrape#4865
Merged
shai-almog merged 1 commit intomasterfrom May 5, 2026
Merged
Conversation
The scheduled trickle scrape has been hitting a 404 since #4758 landed (latest-mobiles.php3 doesn't exist on GSMArena), so every 6h run pulled zero phones and only opened PRs because the JSON envelope churned between runs (e.g. #4862). Replace the trickle path with a weekly run of the verified per-brand walk, and make the script skip writing the file when no device records actually change so envelope drift can't spam PRs again. - Workflow: weekly Mon 03:00 UTC, single brands scrape, 75m timeout (cold cache). Drop the mode dispatch input and trickle/full split. - Script: drop dead --mode latest path (walk_latest, RE_LATEST_LINK, --mode flag). Add _devices_changed() so the file is only rewritten when actual records added/removed/modified, ignoring envelope diffs. - README: document the new weekly cadence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
Contributor
Cloudflare Preview
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
latest-mobiles.php3doesn't exist on GSMArena), so every 6h run pulled zero phones — yet still opened PRs (e.g. Refresh Skin Designer device database #4862) because the JSON envelope churned between runs.devices.jsonwhen no device records actually changed, so envelope drift alone can't open PRs again.Changes
0 3 * * 1), single--delay 2.0 --max-pages 12brands scrape, timeout bumped 25 → 75 min for cold-cache first run. Dropped themodedispatch input and the trickle/full split.--mode latestpath (walk_latest,RE_LATEST_LINK,--modeflag). Added_devices_changed()so the file is only rewritten when records actually differ (id-keyed, order-insensitive). Removedfresh_countfrom the persisted envelope — it's per-run, doesn't belong in a committed artifact.Test plan
python3 -m py_compile build_devices_json.pypasses--helpshows the simplified flag set (no--mode)_devices_changedreturns true (file would be rewritten)🤖 Generated with Claude Code