diff --git a/docs/15-testing/AGENT-TESTING.md b/docs/15-testing/AGENT-TESTING.md new file mode 100644 index 00000000..3333b98d --- /dev/null +++ b/docs/15-testing/AGENT-TESTING.md @@ -0,0 +1,1015 @@ +# Agent Testing Instructions + +This document defines test execution methodology. Each `*-test-cases.md` file in this directory contains domain-specific test scenarios. This document tells you **how** to execute them. + +--- + + +## Table of Contents + +1. [Overview](#overview) +2. [Prerequisites](#prerequisites) +3. [Test Session Workflow](#test-session-workflow) +4. [Reading Test Case Documents](#reading-test-case-documents) +5. [Execution Methods](#execution-methods) +6. [Working with Test Fixture Copies](#working-with-test-fixture-copies) +7. [Enriching the Test Project](#enriching-the-test-project) +8. [Verification Patterns](#verification-patterns) +9. [REPL and Interactive Testing via tmux](#repl-and-interactive-testing-via-tmux) +10. [Docker-Dependent Tests](#docker-dependent-tests) +11. [Auth-Required Tests](#auth-required-tests) +12. [Stress and Boundary Tests](#stress-and-boundary-tests) +13. [Handling Known Failures](#handling-known-failures) +14. [Result Reporting](#result-reporting) +15. [Known Limitations and Pitfalls](#known-limitations-and-pitfalls) +16. [Test Category Quick Reference](#test-category-quick-reference) + +--- + +## Overview + +mxcli is a CLI tool for working with Mendix `.mpr` project files. It provides: +- **MDL language** — a SQL-like DSL for creating/modifying Mendix app models +- **REPL mode** — interactive session for issuing MDL commands +- **CLI commands** — docker, test, playwright, auth, marketplace, eval, fmt, check, etc. + +Tests verify that MDL statements and CLI commands produce correct results against real `.mpr` project files. + +### What you're testing + +- **MDL statements:** CREATE, DROP, ALTER, SHOW, DESCRIBE, GRANT, REVOKE, SET, CONNECT, DISCONNECT +- **CLI subcommands:** `mxcli docker`, `mxcli test`, `mxcli eval`, `mxcli fmt`, `mxcli check`, `mxcli auth`, etc. +- **Roundtrip fidelity:** DESCRIBE output → DROP → re-CREATE from output → DESCRIBE again = identical + +### Test domains (19 documents) + +| # | Document | Domain | +|---|----------|--------| +| 01 | entity-test-cases.md | Entities, associations, constants | +| 02 | enumeration-test-cases.md | Enumerations | +| 03 | microflow-test-cases.md | Microflows | +| 04 | nanoflow-test-cases.md | Nanoflows | +| 05 | page-test-cases.md | Pages, snippets, layouts | +| 06 | integration-test-cases.md | REST, OData, consumed/published services | +| 07 | security-test-cases.md | Module/user roles, entity/page/microflow access | +| 08 | navigation-settings-test-cases.md | Navigation profiles, menu items, project settings | +| 09 | organization-test-cases.md | Modules, folders, MOVE operations | +| 10 | workflow-test-cases.md | Workflows, user tasks, decisions | +| 11 | catalog-test-cases.md | MDL catalog queries | +| 12 | tooling-test-cases.md | Mermaid diagrams, diff, lint, formatter | +| 13 | cli-commands-test-cases.md | All CLI subcommands | +| 14 | sql-integration-test-cases.md | SQL queries against running app | +| 15 | session-test-cases.md | REPL session management | +| 16 | mapping-test-cases.md | Import/export mappings, JSON structures | +| 17 | business-event-test-cases.md | Business event services | +| 18 | image-collection-test-cases.md | Image collections | +| 19 | agent-editor-test-cases.md | Agent/editor integration | + +--- + +## Prerequisites + +### Required software + +| Tool | Purpose | Install | +|------|---------|---------| +| `mxcli` | The tool under test | `make build` (binary at `./bin/mxcli`) | +| `tmux` | Interactive REPL testing | `brew install tmux` (macOS) / pre-installed (Linux) | +| Docker Desktop | SQL integration, docker commands | https://docker.com | +| Go 1.26+ | Building from source | `brew install go` or mise | + +### Required files + +| File | Purpose | Location | +|------|---------|----------| +| Test `.mpr` project | Primary test fixture | Downloaded from Mendix App Gallery | +| mxcli binary | Tool under test | `./bin/mxcli` after `make build` | + +### Test projects + +Demo apps from [Mendix App Gallery](https://appgallery.mendixcloud.com/): + +| App | Studio Pro Version | Purpose | +|-----|-------------------|---------| +| Lato Enquiry Management | 11.4.0 | Primary fixture — entities, microflows, pages, security, navigation | +| Evora - Factory Management | 10.24.15 | Cross-version testing, widgets, workflows | +| Lato Product Inventory | 11.2.0 | Additional coverage — constants, enumerations | + +### Obtaining test projects + +```bash +# Option 1: Download .mpk from App Gallery, extract with mxcli +mxcli new --from-mpk EnquiriesManagement.mpk + +# Option 2: If .mpr files already exist in a known location +export APPS_DIR=~/test-apps +export MPR="$APPS_DIR/EnquiriesManagement/EnquiriesManagement.mpr" +``` + +### Build and verify + +```bash +make build && make test && make lint-go +./bin/mxcli version +./bin/mxcli -c "SHOW ENTITIES" -p "$MPR" # quick smoke test +``` + +If `make test` fails, there are unit test regressions — investigate before proceeding with manual tests. + +--- + +## Test Session Workflow + +### 0. Load MDL syntax knowledge + +Before executing tests, load the relevant MDL syntax references for the domain under test. This ensures you have correct statement syntax, known limitations, and valid examples — consult these references before marking a test as FAIL due to parse errors. + +At minimum, load the general MDL overview. For domain-specific tests (entities, microflows, pages, security, workflows, integrations, etc.), also load the matching domain reference. + +Follow this sequence for a complete test session: + +### 1. Prepare environment + +```bash +# Build fresh +make build + +# Verify binary works +./bin/mxcli version + +# Set project path +export MPR=~/test-apps/EnquiriesManagement/EnquiriesManagement.mpr +``` + +For write tests, create a working copy per [Working with Test Fixture Copies](#working-with-test-fixture-copies). + +### 2. Execute test cases document by document + +For each `*-test-cases.md` file: +1. Read the document to understand all test cases +2. Identify which execution method each test needs (see [Execution Methods](#execution-methods)) +3. Execute tests in order — test IDs are hierarchical (1.1, 1.2, ..., 2.1, ...) +4. Record results in a table (see [Result Reporting](#result-reporting)) +5. If a test requires features not in the project, see [Enriching the Test Project](#enriching-the-test-project) + +### 3. Report results + +After completing each document, produce a summary: +``` +| Total | Pass | Fail | Skip | +|-------|------|------|------| +| 45 | 38 | 4 | 3 | +``` + +Plus the detailed results table and notes on any failures or crashes. + +### 4. Cleanup + +```bash +rm -rf "$WORK_DIR" +# Kill any lingering tmux sessions +tmux kill-server 2>/dev/null +# Stop Docker containers if started +mxcli docker down -p "$MPR" 2>/dev/null +``` + +--- + +## Reading Test Case Documents + +Each test case document follows a consistent structure: + +### Test case format + +```markdown +### 1.1 Short description of the test + +\`\`\` +MDL STATEMENT OR COMMAND HERE; +\`\`\` + +**Expected:** Description of correct output or behavior. +``` + +### Interpreting "Expected" + +- Literal text: output must contain that exact string +- Pattern descriptions ("Displays X, Y, Z"): output should show those elements +- Error expectations ("Error: entity not found"): verify error message appears +- Behavioral ("Silently succeeds"): verify exit code 0 and no error output + +### Test ID numbering + +- `§N` or `## N.` — test section (group of related tests) +- `N.M` — individual test case within section N +- Sections correspond to MDL statement types or command groups + +--- + +## Execution Methods + +### Method 1: Non-Interactive (`-c` flag) — PREFERRED + +For single MDL statements that don't require session state: + +```bash +./bin/mxcli -c "SHOW ENTITIES" -p "$MPR" +``` + +Multiple statements in one call (semicolons separate): + +```bash +./bin/mxcli -c "CREATE ENTITY Mod.Foo (Name : String); SHOW ENTITIES;" -p "$WORK_MPR" +``` + +**When to use:** Most CREATE, DROP, SHOW, DESCRIBE, ALTER, GRANT, REVOKE statements. This is the fastest, most deterministic method. + +**Limitations:** +- Cannot test SET commands (REPL-only) +- Cannot test session state persistence across commands +- Cannot test history navigation or tab completion +- Cannot test multi-step REPL roundtrips where intermediate state matters + +### Method 2: MDL Script File + +For multi-statement sequences: + +```bash +cat > /tmp/test.mdl << 'EOF' +CREATE ENTITY MyModule.TestEntity ( + Name : String, + Age : Integer +); + +DESCRIBE ENTITY MyModule.TestEntity; + +DROP ENTITY MyModule.TestEntity; + +SHOW ENTITIES; +EOF + +./bin/mxcli exec /tmp/test.mdl -p "$WORK_MPR" +``` + +**When to use:** Roundtrip tests, multi-step create/alter/verify sequences, any test requiring ordered operations. + +### Method 3: Direct CLI Commands + +For non-MDL CLI subcommands: + +```bash +./bin/mxcli check script.mdl +./bin/mxcli fmt script.mdl +./bin/mxcli eval -p "$MPR" "expression" +./bin/mxcli docker init -p "$MPR" +./bin/mxcli bson dump -p "$MPR" --path "Module.Entity" +``` + +**When to use:** CLI command tests (doc 13), tooling tests (doc 12). + +### Method 4: REPL via tmux (Interactive) + +See dedicated section: [REPL and Interactive Testing via tmux](#repl-and-interactive-testing-via-tmux) + +### Method 5: Docker Runtime + +See dedicated section: [Docker-Dependent Tests](#docker-dependent-tests) + +### Method 6: Auth-Dependent + +See dedicated section: [Auth-Required Tests](#auth-required-tests) + +--- + +## Working with Test Fixture Copies + +**CRITICAL RULE:** Never run write operations (CREATE, DROP, ALTER, GRANT, REVOKE) against the original test project. Always work on a copy. + +### Standard pattern + +```bash +# Create isolated copy +WORK_DIR=$(mktemp -d) +cp -R "$(dirname $MPR)" "$WORK_DIR/" +WORK_MPR="$WORK_DIR/$(basename $(dirname $MPR))/$(basename $MPR)" + +# Run write tests against copy +./bin/mxcli -c "CREATE ENTITY MyModule.Temp (Name : String);" -p "$WORK_MPR" + +# Cleanup when done +rm -rf "$WORK_DIR" +``` + +### When you need a fresh copy mid-session + +Some tests assume a clean state. If a prior test modified the project in a way that affects subsequent tests: + +```bash +# Reset: re-copy from original +rm -rf "$WORK_DIR" +WORK_DIR=$(mktemp -d) +cp -R "$(dirname $MPR)" "$WORK_DIR/" +WORK_MPR="$WORK_DIR/$(basename $(dirname $MPR))/$(basename $MPR)" +``` + +### Read-only tests don't need copies + +SHOW, DESCRIBE, and catalog queries are read-only — they can run against the original: + +```bash +./bin/mxcli -c "SHOW ENTITIES" -p "$MPR" # safe, no copy needed +``` + +--- + +## Enriching the Test Project + +Some tests require features absent from the default test apps (workflows, business events, OData services, certain entity configurations). + +### Identifying enrichment needs + +Tests are skipped when they reference objects not in the project. Common indicators: +- Test expects to DESCRIBE/ALTER/DROP an object that doesn't exist +- Error: "not found", "no X in module Y" +- Test doc explicitly states "requires X in project" + +### Bootstrap pattern + +Create an MDL script that adds the required fixtures, run it before the dependent tests: + +```bash +WORK_DIR=$(mktemp -d) +cp -R "$(dirname $MPR)" "$WORK_DIR/" +WORK_MPR="$WORK_DIR/$(basename $(dirname $MPR))/$(basename $MPR)" + +# Bootstrap: add missing features +./bin/mxcli exec bootstrap.mdl -p "$WORK_MPR" + +# Now run tests that depend on those features +./bin/mxcli -c "DESCRIBE WORKFLOW TestModule.ApprovalWorkflow" -p "$WORK_MPR" +``` + +### Example bootstrap scripts + +**For workflow tests:** +```sql +CREATE ENTITY TestModule.ApprovalRequest ( + Title : String, + Status : String DEFAULT 'Pending' +); + +CREATE MICROFLOW TestModule.WF_CheckApproval ( + Request : TestModule.ApprovalRequest +) + RETURNS Boolean +BEGIN + RETURN $Request/Status = 'Approved'; +END; +``` + +**For nanoflow tests:** +```sql +CREATE NANOFLOW TestModule.NF_ValidateInput ( + Input : String +) + RETURNS Boolean +BEGIN + RETURN $Input != ''; +END; + +CREATE NANOFLOW TestModule.NF_FormatName ( + FirstName : String, + LastName : String +) + RETURNS String +BEGIN + RETURN $FirstName + ' ' + $LastName; +END; +``` + +**For integration tests:** +```sql +CREATE ENTITY TestModule.Customer ( + Name : String, + Email : String, + Active : Boolean DEFAULT true +); + +CREATE PUBLISHED_ODATA_SERVICE TestModule.CustomerAPI + PATH '/odata/v1/customers'; +``` + +**For business event tests:** +```sql +CREATE BUSINESS_EVENT_SERVICE TestModule.OrderEvents; +``` + +### When to enrich vs. skip + +| Situation | Action | +|-----------|--------| +| Test exercises MDL CRUD on an object type | Enrich: create a fixture object | +| Test exercises runtime behavior (execution, client-side logic) | Skip: can't validate from MDL | +| Test requires a specific Mendix version feature | Skip: version mismatch | +| Test requires external service (database, API) | Skip unless Docker available | + +### Which test docs need enrichment + +| Document | What to bootstrap | +|----------|-------------------| +| workflow-test-cases.md | Entity with workflow context + trigger microflow | +| business-event-test-cases.md | Business event service definition | +| nanoflow-test-cases.md | Sample nanoflows for DESCRIBE/ALTER tests | +| integration-test-cases.md | Published OData service + consumed service entity | +| sql-integration-test-cases.md | Database connection entity (also needs Docker) | + +--- + +## Verification Patterns + +### Exit code check + +```bash +./bin/mxcli -c "SHOW ENTITIES" -p "$MPR" +echo "Exit code: $?" # 0 = success +``` + +**Caveat:** Some commands incorrectly return exit 0 on error (BUG-028). Always check output text too. + +### Output contains expected text + +```bash +OUTPUT=$(./bin/mxcli -c "DESCRIBE ENTITY MyModule.Customer" -p "$MPR" 2>&1) +if echo "$OUTPUT" | grep -q "Name : String"; then + echo "PASS" +else + echo "FAIL" + echo "Actual output:" + echo "$OUTPUT" +fi +``` + +### Output does NOT contain text (verify deletion) + +```bash +OUTPUT=$(./bin/mxcli -c "SHOW ENTITIES" -p "$WORK_MPR" 2>&1) +if echo "$OUTPUT" | grep -q "DeletedEntity"; then + echo "FAIL: entity still exists" +else + echo "PASS: entity removed" +fi +``` + +### Exact error message check + +```bash +OUTPUT=$(./bin/mxcli -c "DROP ENTITY MyModule.NonExistent;" -p "$WORK_MPR" 2>&1) +if echo "$OUTPUT" | grep -qi "not found\|does not exist"; then + echo "PASS: correct error" +else + echo "FAIL: unexpected output: $OUTPUT" +fi +``` + +### Roundtrip test (DESCRIBE → DROP → re-CREATE → DESCRIBE = identical) + +```bash +# 1. Capture original description +BEFORE=$(./bin/mxcli -c "DESCRIBE ENTITY MyModule.Customer" -p "$WORK_MPR" 2>&1) + +# 2. Extract the MDL (skip status lines and REPL / terminators) +MDL=$(echo "$BEFORE" | grep -v "^WARNING\|^Connected\|^$" | grep -v "^/$") + +# 3. Drop the entity +./bin/mxcli -c "DROP ENTITY MyModule.Customer;" -p "$WORK_MPR" + +# 4. Recreate from captured MDL +echo "$MDL" | ./bin/mxcli exec /dev/stdin -p "$WORK_MPR" + +# 5. Describe again +AFTER=$(./bin/mxcli -c "DESCRIBE ENTITY MyModule.Customer" -p "$WORK_MPR" 2>&1) + +# 6. Compare +if [ "$BEFORE" = "$AFTER" ]; then + echo "PASS: roundtrip identical" +else + echo "FAIL: roundtrip mismatch" + diff <(echo "$BEFORE") <(echo "$AFTER") +fi +``` + +### Count-based verification + +```bash +# Verify entity count changed +BEFORE_COUNT=$(./bin/mxcli -c "SHOW ENTITIES" -p "$WORK_MPR" 2>&1 | wc -l) +./bin/mxcli -c "CREATE ENTITY MyModule.NewEntity (Name : String);" -p "$WORK_MPR" +AFTER_COUNT=$(./bin/mxcli -c "SHOW ENTITIES" -p "$WORK_MPR" 2>&1 | wc -l) +if [ "$AFTER_COUNT" -gt "$BEFORE_COUNT" ]; then + echo "PASS: entity count increased" +else + echo "FAIL: count unchanged ($BEFORE_COUNT → $AFTER_COUNT)" +fi +``` + +### Crash detection + +```bash +OUTPUT=$(./bin/mxcli -c "SOME COMMAND" -p "$MPR" 2>&1) +EXIT=$? +if echo "$OUTPUT" | grep -qi "panic\|SIGSEGV\|runtime error\|nil pointer"; then + echo "ERROR: crash detected" + echo "$OUTPUT" +elif [ $EXIT -ne 0 ]; then + echo "FAIL: non-zero exit ($EXIT)" +else + echo "PASS" +fi +``` + +--- + +## REPL and Interactive Testing via tmux + +For tests that require an interactive terminal session (SET commands, multi-step REPL workflows, history, tab completion, TUI commands). + +### Why tmux + +mxcli uses Bubble Tea for its TUI — it detects whether stdin is a TTY and disables interactive features if not. Piping input directly won't work. tmux provides a real PTY that mxcli accepts. + +### Session lifecycle + +```bash +# Create session with explicit dimensions +SESSION="mxcli-test-$$" +tmux new-session -d -s "$SESSION" -x 120 -y 40 + +# Start mxcli REPL +tmux send-keys -t "$SESSION" "./bin/mxcli repl -p $WORK_MPR" Enter + +# Wait for REPL prompt to appear +sleep 2 + +# Verify REPL is ready +OUTPUT=$(tmux capture-pane -t "$SESSION" -p) +echo "$OUTPUT" | grep -q "mxcli>" || echo "ERROR: REPL not ready" +``` + +### Statement terminator: `/` (not `;`) + +The REPL uses `/` (slash on its own line) as the statement terminator, following the Oracle SQL*Plus convention. Semicolons are stripped and ignored. Read commands (`SHOW`, `DESCRIBE`) execute on Enter without a terminator. Write commands (`CREATE`, `DROP`, `GRANT`, `REVOKE`, `ALTER`, `EXECUTE SCRIPT`) enter multi-line mode (`...>`) and require `/` on a separate line to execute. + +```bash +# Read command — executes immediately on Enter +mdl> SHOW ENTITIES IN Administration +# (output appears) + +# Write command — requires / terminator +mdl> CREATE ENTITY MyModule.Test (Name : String) +...> / +Created entity: MyModule.Test + +# Multi-line write command +mdl> CREATE ENTITY MyModule.MultiLine ( +...> Name : String, +...> Email : String +...> ) +...> / +Created entity: MyModule.MultiLine +``` + +### Sending commands + +```bash +# Read command (no terminator needed) +tmux send-keys -t "$SESSION" "SHOW ENTITIES" Enter +sleep 1 + +# Write command (/ terminator on separate line) +tmux send-keys -t "$SESSION" "CREATE ENTITY MyModule.Test (Name : String)" Enter +sleep 0.5 +tmux send-keys -t "$SESSION" "/" Enter +sleep 1 + +# Multi-line command (send each line separately, then /) +tmux send-keys -t "$SESSION" "CREATE ENTITY MyModule.Test (" Enter +sleep 0.3 +tmux send-keys -t "$SESSION" " Name : String," Enter +sleep 0.3 +tmux send-keys -t "$SESSION" " Age : Integer" Enter +sleep 0.3 +tmux send-keys -t "$SESSION" ")" Enter +sleep 0.3 +tmux send-keys -t "$SESSION" "/" Enter +sleep 1 +``` + +### Capturing and verifying output + +```bash +# Capture visible screen + scrollback +OUTPUT=$(tmux capture-pane -t "$SESSION" -p -S -200) + +# Check for expected content +echo "$OUTPUT" | grep -q "Expected text" && echo "PASS" || echo "FAIL" + +# Capture just the last N lines (most recent output) +RECENT=$(tmux capture-pane -t "$SESSION" -p -S -20) +``` + +### Special keys for TUI testing + +```bash +tmux send-keys -t "$SESSION" Up # History: previous command +tmux send-keys -t "$SESSION" Down # History: next command +tmux send-keys -t "$SESSION" Tab # Tab completion +tmux send-keys -t "$SESSION" C-c # Ctrl+C: cancel current input +tmux send-keys -t "$SESSION" C-d # Ctrl+D: exit REPL +tmux send-keys -t "$SESSION" C-l # Ctrl+L: clear screen +tmux send-keys -t "$SESSION" Escape # Escape key +tmux send-keys -t "$SESSION" C-a # Ctrl+A: beginning of line +tmux send-keys -t "$SESSION" C-e # Ctrl+E: end of line +tmux send-keys -t "$SESSION" C-u # Ctrl+U: clear line +tmux send-keys -t "$SESSION" C-w # Ctrl+W: delete word +``` + +### Testing CONNECT/DISCONNECT + +```bash +# Connect to a different project +tmux send-keys -t "$SESSION" "CONNECT LOCAL '$OTHER_MPR'" Enter +sleep 2 +OUTPUT=$(tmux capture-pane -t "$SESSION" -p -S -5) +echo "$OUTPUT" | grep -q "Connected" && echo "PASS" || echo "FAIL" + +# Disconnect +tmux send-keys -t "$SESSION" "DISCONNECT" Enter +sleep 0.5 +``` + +### Teardown + +```bash +# Exit REPL gracefully +tmux send-keys -t "$SESSION" C-d +sleep 0.5 + +# Kill session +tmux kill-session -t "$SESSION" 2>/dev/null +``` + +### Common pitfalls + +- **Timing:** Always add a sleep after sending commands. REPL needs time to process and render. Use 1s for simple commands, 2-3s for operations that read/write the `.mpr`. +- **Scrollback:** Use `-S -200` (or larger) with `capture-pane` to get enough history. Default is only the visible screen. +- **Prompt detection:** Wait for `mxcli>` prompt before sending next command for reliable sequencing. +- **Multi-line commands:** Write commands enter multi-line mode (`...>`). Type `/` on its own line to execute. `Ctrl+C` cancels and returns to `mdl>`. +- **`/` not `;`:** The REPL uses `/` as statement terminator. Semicolons are stripped and ignored. Read commands execute on Enter; write commands require `/`. + +--- + +## Docker-Dependent Tests + +Required for: SQL integration (110 tests), `mxcli docker *` commands, `mxcli test`, `mxcli playwright`. + +### Prerequisites + +```bash +# Verify Docker is running +docker info > /dev/null 2>&1 || echo "ERROR: Docker not running" +``` + +### Start Mendix app in Docker + +```bash +# Initialize Docker config (first time only) +./bin/mxcli docker init -p "$WORK_MPR" + +# Start the app +./bin/mxcli docker run -p "$WORK_MPR" + +# Wait for app to be ready (may take 30-60 seconds) +echo "Waiting for app startup..." +for i in $(seq 1 60); do + if curl -sf http://localhost:8080/xas/ > /dev/null 2>&1; then + echo "App ready after ${i}s" + break + fi + sleep 1 +done +``` + +### SQL integration tests + +Once the app is running: + +```bash +# Execute SQL queries +./bin/mxcli -c "SELECT * FROM MyModule.Customer;" -p "$WORK_MPR" +./bin/mxcli -c "SELECT COUNT(*) FROM MyModule.Customer WHERE Active = true;" -p "$WORK_MPR" +``` + +### Docker command tests + +```bash +./bin/mxcli docker status -p "$WORK_MPR" +./bin/mxcli docker logs -p "$WORK_MPR" +./bin/mxcli docker reload -p "$WORK_MPR" +``` + +### Test runner + +```bash +./bin/mxcli test spec.test.mdl -p "$WORK_MPR" +``` + +### Teardown + +```bash +./bin/mxcli docker down -p "$WORK_MPR" +# Verify container is gone +docker ps | grep -q mxcli && echo "WARNING: container still running" +``` + +### If Docker is not available + +Skip all tests in: +- `sql-integration-test-cases.md` (entire document) +- `cli-commands-test-cases.md` §1-§3 (docker, test, playwright sections) + +Report as: `SKIP — Docker not available` + +--- + +## Auth-Required Tests + +Required for: `mxcli auth *`, `mxcli marketplace *`, and any command that contacts the Mendix platform. + +### Setup + +```bash +export MENDIX_TOKEN="" +./bin/mxcli auth login +./bin/mxcli auth status # verify logged in +``` + +### Auth-dependent commands + +```bash +./bin/mxcli marketplace search "Atlas" +./bin/mxcli marketplace info "Atlas_UI_Resources" +./bin/mxcli auth whoami +./bin/mxcli auth logout +``` + +### If auth is not available + +Skip all tests that require platform connectivity. Report as: `SKIP — MENDIX_TOKEN not set` + +Affected sections across test docs: +- `cli-commands-test-cases.md` §7 (auth), §8 (marketplace) +- Any test that explicitly states "requires authentication" + +--- + +## Stress and Boundary Tests + +These tests are opt-in — they exercise scale and edge cases. Do NOT run as part of routine test sessions. + +### When to run + +- Validating performance before a release +- Investigating a specific scalability report +- Explicitly requested by test plan + +### Always use a fresh copy + +```bash +WORK_DIR=$(mktemp -d) +cp -R "$(dirname $MPR)" "$WORK_DIR/" +WORK_MPR="$WORK_DIR/$(basename $(dirname $MPR))/$(basename $MPR)" +``` + +### Example patterns + +**Bulk creation:** +```bash +for i in $(seq 1 100); do + ./bin/mxcli -c "CREATE ENTITY StressModule.Entity_$i (Name : String);" -p "$WORK_MPR" +done +echo "Created 100 entities" +./bin/mxcli -c "SHOW ENTITIES" -p "$WORK_MPR" | wc -l +``` + +**Wide entity (many attributes):** +```bash +ATTRS="" +for i in $(seq 1 200); do + [ -n "$ATTRS" ] && ATTRS="$ATTRS, " + ATTRS="${ATTRS}Attr_$i : String" +done +./bin/mxcli -c "CREATE ENTITY StressModule.WideEntity ($ATTRS);" -p "$WORK_MPR" +``` + +**Large MDL script:** +```bash +# Generate a 10,000-line script +for i in $(seq 1 5000); do + echo "CREATE ENTITY StressModule.E_$i (Name : String);" + echo "DROP ENTITY StressModule.E_$i;" +done > /tmp/stress.mdl +./bin/mxcli exec /tmp/stress.mdl -p "$WORK_MPR" +``` + +**Rapid connect/disconnect (REPL):** +```bash +SESSION="stress-$$" +tmux new-session -d -s "$SESSION" -x 120 -y 40 +tmux send-keys -t "$SESSION" "./bin/mxcli repl" Enter +sleep 1 +for i in $(seq 1 50); do + tmux send-keys -t "$SESSION" "CONNECT LOCAL '$WORK_MPR';" Enter + sleep 0.5 + tmux send-keys -t "$SESSION" "DISCONNECT;" Enter + sleep 0.3 +done +tmux send-keys -t "$SESSION" C-d +tmux kill-session -t "$SESSION" +``` + +### Categories + +| Type | What it tests | Pass criteria | +|------|--------------|---------------| +| Bulk creation | Many entities/microflows in sequence | No crash, all created | +| Wide entities | Max attributes per entity | No crash, correct count | +| Deep nesting | Deeply nested page widgets | No stack overflow | +| Large scripts | Script file size handling | Completes without timeout | +| Rapid operations | Session stability under load | No panic, no corruption | +| Long values | String length boundaries | Correct storage or clean error | + +### Cleanup + +```bash +rm -rf "$WORK_DIR" +``` + +--- + +## Handling Known Failures + +### Purpose + +Distinguish between known bugs (expected failures) and new regressions. **Always run known-failing tests** to detect regressions. + +### Known bugs reference + +Bug reports live in `docs/12-bug-reports/`. Each has: +- Bug ID (BUG-NNN) +- Severity (Critical / High / Medium / Low) +- Status (Open / Fixed / Won't Fix) +- Affected commands/statements + +### Classification + +| Situation | Report as | Action | +|-----------|-----------|--------| +| Fails exactly as known bug describes | FAIL (known) | Note BUG-NNN | +| Fails differently from known bug | FAIL (new) | File new bug | +| Crashes (SIGSEGV, panic, nil pointer) | ERROR | Always report | +| Known bug now passes | PASS (fixed?) | Note "was BUG-NNN, now passes" | +| Previously unknown failure | FAIL (new) | File new bug | + +### Checking against known bugs + +Before reporting a new failure: +1. Check `docs/12-bug-reports/` for matching bug +2. Check test session notes for prior occurrences +3. Compare error message/behavior with known bug description + +### Filing new bugs + +If a test reveals a new failure, document: +```markdown +## BUG-NNN: Short description + +**Severity:** Critical | High | Medium | Low +**Command:** `mxcli exec -c "FAILING COMMAND"` +**Expected:** What should happen +**Actual:** What happened (include full output) +**Reproducible:** Yes / Sometimes / Once +**Version:** Output of `mxcli version` +``` + +--- + +## Result Reporting + +### Per-test-case results table + +```markdown +| ID | Name | Result | Notes | +|----|------|--------|-------| +| 1.1 | Create basic entity | PASS | | +| 1.2 | Create with all attribute types | PASS | | +| 1.3 | Create duplicate entity | PASS | Correct error shown | +| 2.1 | Describe entity | FAIL (new) | Missing AutoNumber type in output | +| 2.2 | Describe with associations | FAIL (known) | BUG-015 | +| 3.1 | SQL query | SKIP | Docker not available | +| 4.1 | DROP with cascade | ERROR | PANIC: nil pointer at executor.go:445 | +``` + +### Result values + +- **PASS** — output matches expected behavior +- **FAIL (new)** — unexpected failure, not matching any known bug +- **FAIL (known)** — matches a known bug (include BUG-NNN) +- **SKIP** — prerequisite not met (state: Docker/Auth/enrichment needed) +- **ERROR** — crash, panic, SIGSEGV (always include stack trace) + +### Per-document summary + +```markdown +## Summary +| Total | Pass | Fail | Skip | Error | +|-------|------|------|------|-------| +| 45 | 38 | 3 | 3 | 1 | + +### Key Findings +- BUG-NEW: [description] — affects tests 2.1, 2.3 +- BUG-015 confirmed still present (test 2.2) +- PANIC in test 4.1 — [brief description] +``` + +### Session-level summary + +After all documents are completed: +```markdown +## Test Session Summary — [DATE] + +| Document | Total | Pass | Fail | Skip | Error | +|----------|-------|------|------|------|-------| +| 01-entity | 45 | 38 | 3 | 3 | 1 | +| 02-enumeration | 20 | 20 | 0 | 0 | 0 | +| ... | | | | | | +| **TOTAL** | **1099** | **617** | **91** | **393** | **2** | + +### New Bugs Filed +- BUG-NNN: ... +- BUG-NNN: ... + +### Known Bugs Confirmed +- BUG-001: still present +- BUG-015: still present + +### Previously Failing, Now Passing +- BUG-022: appears fixed (test 5.3) +``` + +--- + +## Known Limitations and Pitfalls + +1. **`-c` flag cannot test session-only features** — History navigation, tab completion, and CONNECT/DISCONNECT require REPL. Most write operations work via `-c` too. +2. **Bubble Tea TTY detection** — mxcli disables TUI features when stdin is not a TTY. Piping input won't trigger interactive behavior. Use tmux. +3. **Docker startup time** — Mendix runtime takes 30-60s to fully start. Always wait for port 8080 before running Docker-dependent tests. +4. **Concurrent writes corrupt .mpr** — Never run parallel test sessions against the same `.mpr` file. Always copy to tmpdir first. +5. **Exit codes unreliable** — BUG-028: some commands return exit 0 on error. Always verify output text, not just exit code. +6. **DESCRIBE output contains `/` terminators** — DESCRIBE output includes REPL-style `/` terminators between statements. When saving output to a script file for roundtrip testing, strip standalone `/` lines: `grep -v "^/$"`. Without this, the parser rejects the extra `/` tokens. +7. **Entity auto-creation** — `CREATE ENTITY MyModule.Foo (...)` auto-creates `MyModule` if it doesn't exist. No need to create modules manually. +8. **Statement terminators** — MDL statements in `-c` mode and script files require trailing semicolons. In REPL, use `/` on its own line to terminate write statements; read commands (`SHOW`, `DESCRIBE`) execute on Enter without any terminator. Semicolons are stripped and ignored in REPL. +9. **Case sensitivity** — MDL keywords are case-insensitive (`SHOW` = `show`). Module/entity names are case-sensitive (`MyModule.Foo` ≠ `mymodule.foo`). +10. **Working directory** — `mxcli exec` doesn't change working directory. Relative paths in `-p` are relative to where you invoke the command. +11. **Count only documented tests** — Never add ad-hoc experiments to test statistics. Only tests with an ID in a test case document count toward pass/fail/skip totals. If an ad-hoc test reveals something interesting, either add it to the appropriate test case doc first, or record it as a note outside the statistics. +12. **Add new scenarios to test case docs** — When you discover a new testable scenario during execution (e.g., edge case, undocumented behavior, regression check), add it to the appropriate `*-test-cases.md` file with the next available test ID in the relevant section BEFORE recording it in session results. This ensures the test corpus grows over time and scenarios are reproducible by future sessions. +13. **`mxcli new` requires Linux** — The `mxcli new` command invokes the `mx` binary from the MxBuild download, which is Linux-only. It fails with "exec format error" on macOS. Test on Linux or in a devcontainer. + +--- + +## Test Category Quick Reference + +| Category | Execution Method | Prerequisite | Enrichment Needed | +|----------|-----------------|--------------|-------------------| +| Entity, Enumeration, Association | `-c` flag (Method 1) | Build only | No | +| Microflow | `-c` flag | Build only | No | +| Nanoflow | `-c` flag | Build only | Some (for ALTER/DESCRIBE) | +| Page, Snippet | `-c` flag | Build only | No | +| Security | `-c` flag or REPL | Build only | No | +| Navigation, Settings | `-c` flag or REPL | Build only | No | +| Organization | `-c` flag | Build only | No | +| Session management | tmux REPL (Method 4) | Build + tmux | No | +| Catalog | `-c` flag | Build only | No | +| Tooling (mermaid, diff, lint) | `-c` flag or CLI (Method 3) | Build only | No | +| Integration (REST, OData) | `-c` flag | Build only | Some (publish) | +| Workflow | `-c` flag | Build only | Yes (workflow entity) | +| Business Events | `-c` flag | Build only | Yes (event service) | +| Mappings | `-c` flag | Build only | No (read-only) | +| Image Collections | `-c` flag | Build only | No | +| Agent/Editor | `-c` flag | Build only | No | +| SQL Integration | Docker (Method 5) | Docker running | Yes (DB connection) | +| CLI: docker, test, playwright | Docker (Method 5) | Docker running | No | +| CLI: auth, marketplace | Auth (Method 6) | MENDIX_TOKEN | No | +| CLI: eval, check, fmt, bson | Direct CLI (Method 3) | Build only | No | +| CLI: tui, serve, lsp | tmux REPL (Method 4) | Build + tmux | No | +| Roundtrip tests | Script file (Method 2) | Build only | No | +| Stress/Boundary | Fixture copy + script | Build only | No | diff --git a/docs/15-testing/agent-editor-test-cases.md b/docs/15-testing/agent-editor-test-cases.md new file mode 100644 index 00000000..f7653ecf --- /dev/null +++ b/docs/15-testing/agent-editor-test-cases.md @@ -0,0 +1,874 @@ +# Agent, Model, Knowledge Base & Consumed MCP Service Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +--- + +## 1. SHOW AGENTS + +### 1.1 List all agents + +``` +show agents; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Usage | Model | Tools | KBs`. Summary line `(N agents)`. Sorted alphabetically. + +### 1.2 List agents in a module + +``` +show agents in MyModule; +``` + +**Expected:** Only agents from `MyModule`. Same column format. + +### 1.3 Empty module + +``` +show agents in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 agents)`. + +--- + +## 2. DESCRIBE AGENT + +### 2.1 Agent with all block types + +Find or create an agent with tools, MCP services, and knowledge bases. Verify all three block types appear in `describe` output. + +### 2.2 Dollar-quoted prompts + +Verify `SystemPrompt` and `UserPrompt` use `$$...$$` quoting. Prompts with single quotes, newlines, and special characters must survive roundtrip. + +### 2.3 Non-existent agent + +``` +describe agent Fake.Missing; +``` + +**Expected:** Error — agent not found. + +--- + +## 3. CREATE AGENT + +### 3.1 Minimal agent + +``` +create agent MyModule.SimpleAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Hello.$$ +); +``` + +**Expected:** Agent created. `show agents` lists it. `describe` matches input. + +### 3.2 Auto-create module + +``` +create agent NewModule.Agent1 ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. Agent created inside it. + +### 3.3 Model reference resolution + +``` +create agent MyModule.RefTest ( + UsageType: chat, + Model: OtherModule.SomeModel, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** Resolves `OtherModule.SomeModel` to existing model document. Error if model not found. + +### 3.4 Entity reference resolution + +``` +create agent MyModule.EntityRef ( + UsageType: chat, + Model: MyModule.GPT4o, + Entity: MyModule.ChatSession, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** Resolves `MyModule.ChatSession` to existing entity. Error if entity not found. + +### 3.5 Duplicate agent + +``` +create agent MyModule.SimpleAgent ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Duplicate.$$ +); +``` + +**Expected:** Error — agent already exists. + +--- + +## 4. DROP AGENT + +### 4.1 Drop existing agent + +``` +drop agent MyModule.SimpleAgent; +``` + +**Expected:** Agent removed. `show agents` no longer lists it. + +### 4.2 Drop non-existent agent + +``` +drop agent MyModule.NonExistent; +``` + +**Expected:** Error — agent not found. + +--- + +## 5. SHOW MODELS + +### 5.1 List all models + +``` +show models; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Provider | Key Constant | Display Name`. Summary line `(N models)`. Sorted alphabetically. + +### 5.2 List models in a module + +``` +show models in MyModule; +``` + +**Expected:** Only models from `MyModule`. Same column format. + +### 5.3 Empty module + +``` +show models in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 models)`. + +--- + +## 6. DESCRIBE MODEL + +### 6.1 Basic model + +``` +describe model MyModule.GPT4o; +``` + +**Expected:** MDL output: + +```mdl +create model MyModule.GPT4o ( + Provider: MxCloudGenAI, + Key: MyModule.OpenAIKey, + DisplayName: 'GPT-4o' +); +``` + +### 6.2 Model with all properties + +Find a model with additional properties beyond Provider, Key, and DisplayName. Verify all appear in `describe` output. + +### 6.3 Non-existent model + +``` +describe model Fake.Missing; +``` + +**Expected:** Error — model not found. + +--- + +## 7. CREATE MODEL + +### 7.1 Minimal model (default provider) + +``` +create model MyModule.BasicModel ( + Key: MyModule.ApiKey, + DisplayName: 'Basic Model' +); +``` + +**Expected:** Model created. Provider defaults to `MxCloudGenAI`. `describe` matches. + +### 7.2 Explicit provider + +``` +create model MyModule.CustomModel ( + Provider: MxCloudGenAI, + Key: MyModule.ApiKey, + DisplayName: 'Custom Model' +); +``` + +**Expected:** Model created with explicit provider. + +### 7.3 Constant reference resolution + +``` +create model MyModule.RefModel ( + Key: MyModule.NonExistentConst, + DisplayName: 'Test' +); +``` + +**Expected:** Error — constant `MyModule.NonExistentConst` not found. + +### 7.4 Auto-create module + +``` +create model NewModule.Model1 ( + Key: MyModule.ApiKey, + DisplayName: 'New Module Model' +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. + +### 7.5 Duplicate model + +``` +create model MyModule.BasicModel ( + Key: MyModule.ApiKey, + DisplayName: 'Duplicate' +); +``` + +**Expected:** Error — model already exists. + +--- + +## 8. DROP MODEL + +### 8.1 Drop existing model + +``` +drop model MyModule.BasicModel; +``` + +**Expected:** Model removed. `show models` no longer lists it. + +### 8.2 Drop non-existent model + +``` +drop model MyModule.NonExistent; +``` + +**Expected:** Error — model not found. + +--- + +## 9. SHOW KNOWLEDGE BASES + +### 9.1 List all knowledge bases + +``` +show knowledge bases; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Provider | Key Constant | Embedding Model`. Summary line `(N knowledge bases)`. Sorted alphabetically. + +### 9.2 List knowledge bases in a module + +``` +show knowledge bases in MyModule; +``` + +**Expected:** Only knowledge bases from `MyModule`. Same column format. + +### 9.3 Empty module + +``` +show knowledge bases in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 knowledge bases)`. + +--- + +## 10. DESCRIBE KNOWLEDGE BASE + +### 10.1 Basic knowledge base + +``` +describe knowledge base MyModule.DocumentationKB; +``` + +**Expected:** MDL output: + +```mdl +create knowledge base MyModule.DocumentationKB ( + Provider: MxCloudGenAI, + Key: MyModule.KBKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); +``` + +### 10.2 Knowledge base with all properties + +Find a knowledge base with additional properties. Verify all appear in `describe` output. + +### 10.3 Non-existent knowledge base + +``` +describe knowledge base Fake.Missing; +``` + +**Expected:** Error — knowledge base not found. + +--- + +## 11. CREATE KNOWLEDGE BASE + +### 11.1 Minimal knowledge base + +``` +create knowledge base MyModule.TestKB ( + Key: MyModule.KBKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); +``` + +**Expected:** Knowledge base created. `describe` matches. + +### 11.2 Explicit provider + +``` +create knowledge base MyModule.FullKB ( + Provider: MxCloudGenAI, + Key: MyModule.KBKey, + ModelDisplayName: 'text-embedding-3-small', + ModelName: 'text-embedding-3-small' +); +``` + +**Expected:** Knowledge base created with explicit provider. + +### 11.3 Constant reference resolution + +``` +create knowledge base MyModule.BadKB ( + Key: MyModule.NonExistentConst, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** Error — constant `MyModule.NonExistentConst` not found. + +### 11.4 Auto-create module + +``` +create knowledge base NewModule.KB1 ( + Key: MyModule.KBKey, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. + +### 11.5 Duplicate knowledge base + +``` +create knowledge base MyModule.TestKB ( + Key: MyModule.KBKey, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** Error — knowledge base already exists. + +--- + +## 12. DROP KNOWLEDGE BASE + +### 12.1 Drop existing knowledge base + +``` +drop knowledge base MyModule.TestKB; +``` + +**Expected:** Knowledge base removed. `show knowledge bases` no longer lists it. + +### 12.2 Drop non-existent knowledge base + +``` +drop knowledge base MyModule.NonExistent; +``` + +**Expected:** Error — knowledge base not found. + +--- + +## 13. SHOW CONSUMED MCP SERVICES + +### 13.1 List all consumed MCP services + +``` +show consumed mcp services; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Protocol | Version | Timeout`. Summary line `(N consumed mcp services)`. Sorted alphabetically. + +### 13.2 List consumed MCP services in a module + +``` +show consumed mcp services in MyModule; +``` + +**Expected:** Only services from `MyModule`. Same column format. + +### 13.3 Empty module + +``` +show consumed mcp services in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 consumed mcp services)`. + +--- + +## 14. DESCRIBE CONSUMED MCP SERVICE + +### 14.1 Basic consumed MCP service + +``` +describe consumed mcp service MyModule.ExternalSvc; +``` + +**Expected:** MDL output: + +```mdl +create consumed mcp service MyModule.ExternalSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'External data provider for weather information' +); +``` + +### 14.2 Service with all properties + +Find a consumed MCP service with all available properties. Verify all appear in `describe` output. + +### 14.3 Non-existent consumed MCP service + +``` +describe consumed mcp service Fake.Missing; +``` + +**Expected:** Error — consumed MCP service not found. + +--- + +## 15. CREATE CONSUMED MCP SERVICE + +### 15.1 Minimal consumed MCP service + +``` +create consumed mcp service MyModule.WeatherSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30 +); +``` + +**Expected:** Service created. `describe` matches. + +### 15.2 With documentation + +``` +create consumed mcp service MyModule.DataSvc ( + ProtocolVersion: 2024-11-05, + Version: '2.0.0', + ConnectionTimeoutSeconds: 60, + Documentation: 'Provides access to external data sources' +); +``` + +**Expected:** Service created with documentation. `describe` shows `Documentation` property. + +### 15.3 Auto-create module + +``` +create consumed mcp service NewModule.Svc1 ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30 +); +``` + +**Expected:** `NewModule` auto-created if it does not exist. + +### 15.4 Duplicate consumed MCP service + +``` +create consumed mcp service MyModule.WeatherSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30 +); +``` + +**Expected:** Error — consumed MCP service already exists. + +--- + +## 16. DROP CONSUMED MCP SERVICE + +### 16.1 Drop existing consumed MCP service + +``` +drop consumed mcp service MyModule.WeatherSvc; +``` + +**Expected:** Service removed. `show consumed mcp services` no longer lists it. + +### 16.2 Drop non-existent consumed MCP service + +``` +drop consumed mcp service MyModule.NonExistent; +``` + +**Expected:** Error — consumed MCP service not found. + +--- + +## 17. ROUNDTRIP + +Test that CREATE → DESCRIBE → CREATE (from output) produces identical results. + +### 17.1 Model roundtrip + +``` +create model RtTest.Model1 ( + Provider: MxCloudGenAI, + Key: RtTest.ApiKey, + DisplayName: 'Test Model' +); +``` + +1. `describe model RtTest.Model1` +2. Drop: `drop model RtTest.Model1` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +### 17.2 Knowledge base roundtrip + +``` +create knowledge base RtTest.DocsKB ( + Provider: MxCloudGenAI, + Key: RtTest.KBKey, + ModelDisplayName: 'text-embedding-ada-002', + ModelName: 'text-embedding-ada-002' +); +``` + +1. `describe knowledge base RtTest.DocsKB` +2. Drop: `drop knowledge base RtTest.DocsKB` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +### 17.3 Consumed MCP service roundtrip + +``` +create consumed mcp service RtTest.ExternalSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'Test service' +); +``` + +1. `describe consumed mcp service RtTest.ExternalSvc` +2. Drop: `drop consumed mcp service RtTest.ExternalSvc` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +--- + +## 18. MULTI-STEP WORKFLOWS + +### 18.1 Cross-module references + +``` +create constant ModA.Key type String default 'key'; +create model ModA.SharedModel (Key: ModA.Key, DisplayName: 'Shared'); +create agent ModB.CrossAgent ( + UsageType: chat, + Model: ModA.SharedModel, + SystemPrompt: $$Cross-module test.$$ +); +describe agent ModB.CrossAgent; +``` + +**Expected:** Agent in `ModB` references model in `ModA`. `describe` shows fully qualified model name. + +--- + +## 19. FAILURE MODES & ERROR RECOVERY + +### 19.1 Not connected + +Run any command without `-p` flag or REPL session. + +**Expected:** Error — not connected to a project. + +### 19.2 Agent already exists + +``` +create agent MyModule.TestAgent (...); +create agent MyModule.TestAgent (...); +``` + +**Expected:** Second create fails — agent already exists. + +### 19.3 Model not found during agent creation + +``` +create agent MyModule.BadAgent ( + UsageType: chat, + Model: MyModule.NonExistentModel, + SystemPrompt: $$Test.$$ +); +``` + +**Expected:** Error — model `MyModule.NonExistentModel` not found. + +### 19.4 Constant not found during model creation + +``` +create model MyModule.BadModel ( + Key: MyModule.FakeConst, + DisplayName: 'Bad' +); +``` + +**Expected:** Error — constant not found. + +### 19.5 Constant not found during KB creation + +``` +create knowledge base MyModule.BadKB ( + Key: MyModule.FakeConst, + ModelDisplayName: 'test', + ModelName: 'test' +); +``` + +**Expected:** Error — constant not found. + +### 19.6 Drop model referenced by agent + +``` +drop model MyModule.TestModel; +``` + +**Expected:** Warning about agent references. Model dropped (agent may have dangling reference). + +--- + +## 20. BOUNDARY & STRESS + +### 20.1 Agent with many tools (10+) + +Create an agent with 10+ tool blocks. Verify `describe` lists all tools. + +### 20.2 Agent with many knowledge bases (5+) + +Create an agent with 5+ knowledge base blocks. Verify `describe` lists all. + +### 20.3 Long dollar-quoted prompt + +Create an agent with a `SystemPrompt` exceeding 4000 characters. Verify `describe` preserves full text. + +### 20.4 Special characters in prompts + +``` +create agent MyModule.SpecialChars ( + UsageType: chat, + Model: MyModule.GPT4o, + SystemPrompt: $$Prompt with 'single quotes', "double quotes", +backslash \, dollar $, and unicode: café résumé$$ +); +``` + +**Expected:** All characters preserved in `describe` output. + +### 20.5 Long service documentation + +``` +create consumed mcp service MyModule.DocSvc ( + ProtocolVersion: 2024-11-05, + Version: '1.0.0', + ConnectionTimeoutSeconds: 30, + Documentation: 'Very long documentation string...(1000+ chars)' +); +``` + +**Expected:** Full documentation preserved in `describe`. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW AGENTS | x | x | x | +| DESCRIBE AGENT | x | x | x | +| CREATE AGENT | x | | | +| DROP AGENT | x | | | +| SHOW MODELS | x | x | x | +| DESCRIBE MODEL | x | x | | +| CREATE MODEL | x | | | +| DROP MODEL | x | | | +| SHOW KNOWLEDGE BASES | x | x | x | +| DESCRIBE KNOWLEDGE BASE | x | x | | +| CREATE KNOWLEDGE BASE | x | | | +| DROP KNOWLEDGE BASE | x | | | +| SHOW CONSUMED MCP SERVICES | x | x | x | +| DESCRIBE CONSUMED MCP SERVICE | x | x | | +| CREATE CONSUMED MCP SERVICE | x | | | +| DROP CONSUMED MCP SERVICE | x | | | + +Read operations tested on all projects. Write operations on copies of one project. + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW AGENTS | Mock tests | | +| 2. DESCRIBE AGENT | Mock tests | All block types | +| 3. CREATE AGENT | Mock + roundtrip | Reference resolution | +| 4. DROP AGENT | Mock tests | | +| 5. SHOW MODELS | Mock tests | | +| 6. DESCRIBE MODEL | Mock tests | | +| 7. CREATE MODEL | Mock tests | Constant resolution | +| 8. DROP MODEL | Mock tests | | +| 9. SHOW KNOWLEDGE BASES | Mock tests | | +| 10. DESCRIBE KNOWLEDGE BASE | Mock tests | | +| 11. CREATE KNOWLEDGE BASE | Mock tests | Constant resolution | +| 12. DROP KNOWLEDGE BASE | Mock tests | | +| 13. SHOW CONSUMED MCP SERVICES | Mock tests | | +| 14. DESCRIBE CONSUMED MCP SERVICE | Mock tests | | +| 15. CREATE CONSUMED MCP SERVICE | Mock tests | | +| 16. DROP CONSUMED MCP SERVICE | Mock tests | | +| 17. Roundtrip | Roundtrip tests | Complex agents | +| 18. Multi-step | | All manual | +| 19. Failure modes | Partial | Edge cases | +| 20. Boundary | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW AGENTS | List all | | | | | +| 1.2 | SHOW AGENTS | Filter by module | | | | | +| 1.3 | SHOW AGENTS | Empty module | | | | | +| 2.1 | DESCRIBE AGENT | All block types | | | | | +| 2.2 | DESCRIBE AGENT | Dollar-quoted prompts | | | | | +| 2.3 | DESCRIBE AGENT | Not found | | | | | +| 3.1 | CREATE AGENT | Minimal | | | | | +| 3.2 | CREATE AGENT | Auto-create module | | | | | +| 3.3 | CREATE AGENT | Model reference | | | | | +| 3.4 | CREATE AGENT | Entity reference | | | | | +| 3.5 | CREATE AGENT | Duplicate error | | | | | +| 4.1 | DROP AGENT | Existing | | | | | +| 4.2 | DROP AGENT | Non-existent | | | | | +| 5.1 | SHOW MODELS | List all | | | | | +| 5.2 | SHOW MODELS | Filter by module | | | | | +| 5.3 | SHOW MODELS | Empty module | | | | | +| 6.1 | DESCRIBE MODEL | Basic | | | | | +| 6.2 | DESCRIBE MODEL | All properties | | | | | +| 6.3 | DESCRIBE MODEL | Not found | | | | | +| 7.1 | CREATE MODEL | Minimal (default provider) | | | | | +| 7.2 | CREATE MODEL | Explicit provider | | | | | +| 7.3 | CREATE MODEL | Constant not found | | | | | +| 7.4 | CREATE MODEL | Auto-create module | | | | | +| 7.5 | CREATE MODEL | Duplicate error | | | | | +| 8.1 | DROP MODEL | Existing | | | | | +| 8.2 | DROP MODEL | Non-existent | | | | | +| 9.1 | SHOW KBS | List all | | | | | +| 9.2 | SHOW KBS | Filter by module | | | | | +| 9.3 | SHOW KBS | Empty module | | | | | +| 10.1 | DESCRIBE KB | Basic | | | | | +| 10.2 | DESCRIBE KB | All properties | | | | | +| 10.3 | DESCRIBE KB | Not found | | | | | +| 11.1 | CREATE KB | Minimal | | | | | +| 11.2 | CREATE KB | Explicit provider | | | | | +| 11.3 | CREATE KB | Constant not found | | | | | +| 11.4 | CREATE KB | Auto-create module | | | | | +| 11.5 | CREATE KB | Duplicate error | | | | | +| 12.1 | DROP KB | Existing | | | | | +| 12.2 | DROP KB | Non-existent | | | | | +| 13.1 | SHOW MCP | List all | | | | | +| 13.2 | SHOW MCP | Filter by module | | | | | +| 13.3 | SHOW MCP | Empty module | | | | | +| 14.1 | DESCRIBE MCP | Basic | | | | | +| 14.2 | DESCRIBE MCP | All properties | | | | | +| 14.3 | DESCRIBE MCP | Not found | | | | | +| 15.1 | CREATE MCP | Minimal | | | | | +| 15.2 | CREATE MCP | With documentation | | | | | +| 15.3 | CREATE MCP | Auto-create module | | | | | +| 15.4 | CREATE MCP | Duplicate error | | | | | +| 16.1 | DROP MCP | Existing | | | | | +| 16.2 | DROP MCP | Non-existent | | | | | +| 17.1 | ROUNDTRIP | Model | | | | | +| 17.2 | ROUNDTRIP | Knowledge base | | | | | +| 17.3 | ROUNDTRIP | Consumed MCP service | | | | | +| 18.1 | MULTI-STEP | Cross-module refs | | | | | +| 19.1 | FAILURE | Not connected | | | | | +| 19.2 | FAILURE | Already exists | | | | | +| 19.3 | FAILURE | Model not found | | | | | +| 19.4 | FAILURE | Constant not found (model) | | | | | +| 19.5 | FAILURE | Constant not found (KB) | | | | | +| 19.6 | FAILURE | Drop referenced model | | | | | +| 20.1 | BOUNDARY | Many tools | | | | | +| 20.2 | BOUNDARY | Many KBs | | | | | +| 20.3 | BOUNDARY | Long prompt | | | | | +| 20.4 | BOUNDARY | Special characters | | | | | +| 20.5 | BOUNDARY | Long documentation | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/business-event-test-cases.md b/docs/15-testing/business-event-test-cases.md new file mode 100644 index 00000000..9df6a15b --- /dev/null +++ b/docs/15-testing/business-event-test-cases.md @@ -0,0 +1,344 @@ +# Business Event Service Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +--- + +## 1. SHOW BUSINESS EVENT SERVICES + +### 1.1 List all services + +``` +show business event services; +``` + +**Expected:** Table with columns `Module | QualifiedName | Service | Messages | Publish | Subscribe`. Summary line `(N business event services)`. Sorted alphabetically. + +### 1.2 Filter by module + +``` +show business event services in MyModule; +``` + +**Expected:** Only services from `MyModule`. Same column format. + +### 1.3 Empty result + +``` +show business event services in NonExistentModule; +``` + +**Expected:** `No business event services found.` + +--- + +## 2. SHOW BUSINESS EVENTS + +### 2.1 List all business events + +``` +show business events; +``` + +**Expected:** Table with columns `Service | Message | Operation | Entity | Attributes`. `Operation` values are `PUBLISH` or `SUBSCRIBE`. Summary line `(N business events)`. + +### 2.2 Filter by module + +``` +show business events in MyModule; +``` + +**Expected:** Only events from `MyModule`. Same column format. + +### 2.3 Empty module + +``` +show business events in NonExistentModule; +``` + +**Expected:** Empty result or `No business events found.` + +--- + +## 3. DESCRIBE BUSINESS EVENT SERVICE + +### 3.1 Service with publish message + +``` +describe business event service MyModule.OrderEvents; +``` + +**Expected:** Full MDL output: +``` +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message OrderCreated ( + OrderId: Long, + CustomerName: String, + Total: Decimal + ) publish entity MyModule.OrderCreatedEvent +/ +``` + +### 3.2 Service with subscribe message + +``` +describe business event service MyModule.PaymentEvents; +``` + +**Expected:** MDL includes `subscribe` keyword with entity and microflow references: +``` +create business event service MyModule.PaymentEvents + event_name_prefix 'com.example.payment' + message PaymentReceived ( + PaymentId: Long, + Amount: Decimal, + Currency: String + ) subscribe entity MyModule.PaymentReceivedEvent microflow MyModule.OnPaymentReceived +/ +``` + +### 3.3 Service with multiple messages + +``` +describe business event service MyModule.InventoryEvents; +``` + +**Expected:** Multiple `message` blocks in output. Each has its own attribute list, operation keyword, and entity reference. + +### 3.4 All attribute types + +Verify described output includes attributes of each supported type: + +| Type | Example | +|------|---------| +| Long | `OrderId: Long` | +| String | `Name: String` | +| Integer | `Count: Integer` | +| Boolean | `IsActive: Boolean` | +| DateTime | `Timestamp: DateTime` | +| Decimal | `Amount: Decimal` | + +### 3.5 Non-existent service + +``` +describe business event service MyModule.Fake; +``` + +**Expected:** Error — service not found. + +--- + +## 4. DROP BUSINESS EVENT SERVICE + +### 5.1 Drop existing service + +``` +drop business event service MyModule.OrderEvents; +``` + +**Expected:** Service removed. `describe business event service MyModule.OrderEvents` returns error. + +### 5.2 Drop non-existent service + +``` +drop business event service MyModule.Fake; +``` + +**Expected:** Error — service not found. + +--- + +## 5. MULTI-STEP WORKFLOWS + +### 5.1 Create entity → create publish service → verify events + +``` +create persistent entity MyModule.InvoiceCreatedEvent ( + InvoiceId: Long, + CustomerName: String, + TotalAmount: Decimal +); + +create business event service MyModule.InvoiceEvents + event_name_prefix 'com.example.invoice' + message InvoiceCreated ( + InvoiceId: Long, + CustomerName: String, + TotalAmount: Decimal + ) publish entity MyModule.InvoiceCreatedEvent; + +show business events in MyModule; +``` + +**Expected:** All statements succeed. `show business events` lists `InvoiceCreated` with operation `PUBLISH` and entity `MyModule.InvoiceCreatedEvent`. + +### 5.2 Create entity → create subscribe service → verify events + +``` +create persistent entity MyModule.ShipmentReceivedEvent ( + ShipmentId: Long, + Carrier: String +); + +create business event service MyModule.ShipmentEvents + event_name_prefix 'com.example.shipment' + message ShipmentReceived ( + ShipmentId: Long, + Carrier: String + ) subscribe entity MyModule.ShipmentReceivedEvent microflow MyModule.OnShipmentReceived; + +show business events in MyModule; +``` + +**Expected:** `show business events` lists `ShipmentReceived` with operation `SUBSCRIBE`. + +### 5.3 Create → replace → verify + +``` +create business event service MyModule.Lifecycle + event_name_prefix 'com.example.lc.v1' + message Ping ( + Seq: Integer + ) publish entity MyModule.PingEvent; + +create or replace business event service MyModule.Lifecycle + event_name_prefix 'com.example.lc.v2' + message Ping ( + Seq: Integer, + Timestamp: DateTime + ) publish entity MyModule.PingEvent; + +describe business event service MyModule.Lifecycle; +``` + +**Expected:** Final `describe` shows v2 prefix and `Timestamp` attribute. + +--- + +## 6. FAILURE MODES & ERROR RECOVERY + +### 6.1 Not connected to project + +``` +show business event services; +``` + +(Run without `-p` flag or before opening a project.) + +**Expected:** Error — not connected to a project. + +### 6.2 Service not found + +``` +describe business event service MyModule.DoesNotExist; +``` + +**Expected:** Error — service not found. + +### 6.3 Service already exists + +``` +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message Msg (X: Integer) publish entity MyModule.MsgEvent; +create business event service MyModule.OrderEvents + event_name_prefix 'com.example.order' + message Msg (X: Integer) publish entity MyModule.MsgEvent; +``` + +**Expected:** First succeeds. Second returns error — already exists. + +### 6.4 Module not found + +``` +show business event services in FakeModule; +``` + +**Expected:** `No business event services found.` or error — module not found. + +### 6.5 Missing entity reference + +``` +create business event service MyModule.BadRef + event_name_prefix 'com.example.badref' + message Msg (X: Integer) publish entity MyModule.NonExistentEntity; +``` + +**Expected:** Error — entity not found. + +### 6.6 Invalid attribute type + +``` +create business event service MyModule.BadType + event_name_prefix 'com.example.badtype' + message Msg (X: InvalidType) publish entity MyModule.SomeEvent; +``` + +**Expected:** Error — unknown type. No service created. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW BUSINESS EVENT SERVICES | x | x | x | +| SHOW BUSINESS EVENTS | x | x | x | +| DESCRIBE BUSINESS EVENT SERVICE | x | x | | +| DROP BUSINESS EVENT SERVICE | x | | | + +Read operations tested on all projects. Write operations on copies of one project. + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW BUSINESS EVENT SERVICES | Mock tests | | +| 2. SHOW BUSINESS EVENTS | Mock tests | | +| 3. DESCRIBE BUSINESS EVENT SERVICE | Mock tests | | +| 4. DROP BUSINESS EVENT SERVICE | Mock tests | | +| 5. Multi-step | | All manual | +| 6. Failure modes | Partial | Edge cases | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW BE SERVICES | List all | | | | | +| 1.2 | SHOW BE SERVICES | Filter by module | | | | | +| 1.3 | SHOW BE SERVICES | Empty result | | | | | +| 2.1 | SHOW BE | List all | | | | | +| 2.2 | SHOW BE | Filter by module | | | | | +| 2.3 | SHOW BE | Empty module | | | | | +| 3.1 | DESCRIBE BE SERVICE | Publish message | | | | | +| 3.2 | DESCRIBE BE SERVICE | Subscribe message | | | | | +| 3.3 | DESCRIBE BE SERVICE | Multiple messages | | | | | +| 3.4 | DESCRIBE BE SERVICE | All attribute types | | | | | +| 3.5 | DESCRIBE BE SERVICE | Not found | | | | | +| 4.1 | DROP BE SERVICE | Existing | | | | | +| 4.2 | DROP BE SERVICE | Non-existent | | | | | +| 5.1 | MULTI-STEP | Entity + publish + verify | | | | | +| 5.2 | MULTI-STEP | Entity + subscribe + verify | | | | | +| 5.3 | MULTI-STEP | Create + replace + verify | | | | | +| 6.1 | FAILURE | Not connected | | | | | +| 6.2 | FAILURE | Service not found | | | | | +| 6.3 | FAILURE | Already exists | | | | | +| 6.4 | FAILURE | Module not found | | | | | +| 6.5 | FAILURE | Missing entity ref | | | | | +| 6.6 | FAILURE | Invalid attribute type | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/catalog-test-cases.md b/docs/15-testing/catalog-test-cases.md new file mode 100644 index 00000000..36437778 --- /dev/null +++ b/docs/15-testing/catalog-test-cases.md @@ -0,0 +1,540 @@ +# Catalog & Code Navigation Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +--- + +## 1. REFRESH CATALOG + +### 1.1 Fast mode (default) + +``` +refresh catalog; +``` + +**Expected:** Builds core tables (modules, entities, attributes, microflows, etc.). Output shows per-table row count + "Catalog ready (Xs)". + +### 1.2 Full mode + +``` +refresh catalog full; +``` + +**Expected:** Adds activities, widgets, refs, strings, permissions tables. + +### 1.3 Full with source + +``` +refresh catalog full source; +``` + +**Expected:** Adds source table (MDL text for each document). + +### 1.4 Force rebuild + +``` +refresh catalog force; +``` + +**Expected:** Ignores cached catalog, rebuilds from scratch. + +### 1.5 Background build + +``` +refresh catalog background; +``` + +**Expected:** Returns immediately; catalog built in background goroutine. + +### 1.6 Caching behavior + +1. Run `refresh catalog full;` +2. Exit and re-open same project +3. Run `refresh catalog;` + +**Expected:** Second run uses cached `.mxcli/catalog.db` (fast — no rebuild needed). + +--- + +## 2. SHOW CATALOG TABLES + +### 2.1 List tables + +``` +show catalog tables; +``` + +**Expected:** Table with columns `Table | Count`. Lists 35+ tables including: modules, entities, attributes, microflows, nanoflows, pages, snippets, layouts, enumerations, activities, widgets, xpath_expressions, refs, permissions, workflows, odata_clients, odata_services, etc. + +### 2.2 Tables require appropriate build mode + +Verify tables marked as full-only show 0 rows if only fast catalog was built. + +--- + +## 3. SHOW CATALOG STATUS + +### 3.1 Catalog status + +``` +show catalog status; +``` + +**Expected:** Cache path, build mode, build time, build duration, Mendix version, validity status. + +--- + +## 4. DESCRIBE CATALOG TABLE + +### 4.1 Describe entities table + +``` +describe catalog.entities; +``` + +**Expected:** Column names, types, PK markers + required refresh mode. + +### 4.2 Describe activities table + +``` +describe catalog.activities; +``` + +**Expected:** Schema for full-mode table. + +--- + +## 5. SELECT FROM CATALOG + +### 5.1 Basic select + +``` +select * from catalog.entities where ModuleName = 'Administration'; +``` + +**Expected:** Row count + table result with entity data from Administration module. + +### 5.2 Select with multiple conditions + +``` +select QualifiedName, EntityType from catalog.entities where EntityType = 'PERSISTENT' and ModuleName = 'FactoryManagement'; +``` + +**Expected:** Filtered results. + +### 5.3 Aggregate query + +``` +select ModuleName, count(*) as cnt from catalog.entities group by ModuleName order by cnt desc; +``` + +**Expected:** Entity counts per module. + +### 5.4 Join across tables + +``` +select e.QualifiedName, count(a.Name) as attrs +from catalog.entities e +join catalog.attributes a on e.QualifiedName = a.EntityQualifiedName +group by e.QualifiedName +having attrs > 10; +``` + +**Expected:** Entities with more than 10 attributes. + +### 5.5 Full-only table without full build + +``` +refresh catalog; +select * from catalog.activities; +``` + +**Expected:** Warning about insufficient build mode. Empty or error. + +### 5.6 FTS search on strings table + +``` +select * from catalog.strings where strings match 'factory'; +``` + +**Expected:** Full-text search results with matching snippets. (Note: use a term present in your test project; `'factory'` works with Evora Factory Management.) + +--- + +## 6. SEARCH + +### 6.1 Basic search + +``` +search 'customer'; +``` + +**Expected:** String matches with columns: `QualifiedName | ObjectType | Match | StringContext | ModuleName`. Match shows `>>>...<<<` highlighting. + +### 6.2 Search with special characters + +``` +search 'user.name'; +``` + +**Expected:** Special chars (`.`, `/`, `-`, `:`) treated as spaces (AND semantics). + +### 6.3 Search requiring source mode + +If source table exists, additional source matches appear in output. + +### 6.4 Search with no results + +``` +search 'xyznonexistent123'; +``` + +**Expected:** Empty result. + +--- + +## 7. SHOW CALLERS + +### 7.1 Direct callers + +``` +show callers of Administration.ACT_CreateAccount; +``` + +**Expected:** Table with columns `Caller | Depth`. Shows microflows/pages that call this microflow. + +### 7.2 Transitive callers + +``` +show callers of Administration.ACT_CreateAccount transitive; +``` + +**Expected:** Recursive callers up to depth 10. + +### 7.3 No callers + +``` +show callers of ; +``` + +**Expected:** Empty result. + +--- + +## 8. SHOW CALLEES + +### 8.1 Direct callees + +``` +show callees of ; +``` + +**Expected:** Microflows called by this microflow. + +### 8.2 Transitive callees + +``` +show callees of transitive; +``` + +**Expected:** Recursive callees up to depth 10. + +--- + +## 9. SHOW REFERENCES + +### 9.1 Entity references + +``` +show references to Administration.Account; +``` + +**Expected:** Table with columns `SourceType | SourceName | RefKind`. Shows all documents referencing this entity. + +### 9.2 Microflow references + +``` +show references to ; +``` + +**Expected:** Documents referencing this microflow. + +--- + +## 10. SHOW IMPACT + +### 10.1 Entity impact + +``` +show impact of Administration.Account; +``` + +**Expected:** Summary (count by type) + detailed table of impacted documents. + +### 10.2 Microflow impact + +``` +show impact of ; +``` + +**Expected:** Summary + detail table. + +--- + +## 11. SHOW CONTEXT + +### 11.1 Microflow context + +``` +show context of Administration.ACT_CreateAccount; +``` + +**Expected:** Markdown output: definition (name, return, params, activities) + entities used + pages shown + called microflows + direct callers. + +### 11.2 Entity context + +``` +show context of Administration.Account; +``` + +**Expected:** Definition (type, generalization, attributes, indexes) + microflows using it + pages displaying it + related entities. + +### 11.3 Page context + +``` +show context of ; +``` + +**Expected:** Definition (title, URL, layout, widgets) + entities used + microflows called + shown by. + +### 11.4 Context with depth + +``` +show context of Administration.ACT_CreateAccount depth 3; +``` + +**Expected:** Deeper recursive resolution of called microflows. + +### 11.5 Non-existent document + +``` +show context of Fake.Missing; +``` + +**Expected:** Error or empty — document not found in catalog. + +--- + +## 12. SHOW STRUCTURE + +### 12.1 Default structure (depth 2) + +``` +show structure; +``` + +**Expected:** Module names + element type counts (entities, enums, microflows, etc.). Skips system/marketplace modules. Default depth is 2 (element names with signatures). + +> **Note:** To get depth-1 output (counts only), use `show structure depth 1` explicitly. + +### 12.2 Depth 2 + +``` +show structure depth 2; +``` + +**Expected:** Element names with signatures — entity attributes, microflow params → return, enum values. + +### 12.3 Depth 3 + +``` +show structure depth 3; +``` + +**Expected:** Attribute types, parameter names, association delete behavior, constant defaults. + +### 12.4 Module filter + +``` +show structure in Administration; +``` + +**Expected:** Only `Administration` module structure. + +> **Note:** Marketplace and system modules are filtered by default. Use `show structure in all` to include them, or test with a non-marketplace module. + +### 12.5 Include system modules + +``` +show structure all; +``` + +**Expected:** All modules including System, Atlas_Core, etc. + +--- + +## 13. MULTI-STEP WORKFLOWS + +### 13.1 Create entity → refresh → query catalog + +``` +create persistent entity MyModule.Indexed ( + Name: string(200) not null +); +refresh catalog; +select * from catalog.entities where QualifiedName = 'MyModule.Indexed'; +``` + +**Expected:** New entity appears in catalog after refresh. + +### 13.2 Impact analysis before drop + +``` +show impact of MyModule.Customer; +drop entity MyModule.Customer; +``` + +**Expected:** Impact shows affected documents before destructive operation. + +--- + +## 14. FAILURE MODES & ERROR RECOVERY + +### 14.1 Query without catalog + +Open project, do NOT refresh catalog: +``` +select * from catalog.entities; +``` + +**Expected:** mxcli auto-builds a fast catalog on demand and returns results. No manual `refresh catalog` required. + +### 14.2 Invalid SQL + +``` +select * from catalog.nonexistent; +``` + +**Expected:** SQL error — no such table. + +### 14.3 Search without full catalog + +``` +refresh catalog; +search 'test'; +``` + +**Expected:** mxcli auto-upgrades to a full catalog build on demand and returns search results. No manual `refresh catalog full` required. + +### 14.4 Callers without refs table + +``` +refresh catalog; +show callers of MyModule.Flow; +``` + +**Expected:** mxcli auto-upgrades to a full catalog build on demand and returns caller results. No manual `refresh catalog full` required. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| REFRESH CATALOG | x | x | x | +| SHOW CATALOG TABLES | x | | | +| SHOW CATALOG STATUS | x | | | +| DESCRIBE CATALOG | x | | | +| SELECT FROM CATALOG | x | x | x | +| SEARCH | x | x | | +| SHOW CALLERS | x | x | | +| SHOW CALLEES | x | x | | +| SHOW REFERENCES | x | x | | +| SHOW IMPACT | x | | | +| SHOW CONTEXT | x | x | | +| SHOW STRUCTURE | x | x | x | + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. REFRESH CATALOG | Unit tests | Caching, background | +| 2. SHOW TABLES | Unit tests | | +| 3. SHOW STATUS | | All manual | +| 4. DESCRIBE TABLE | | All manual | +| 5. SELECT | Unit tests | Complex queries | +| 6. SEARCH | Unit tests | Edge cases | +| 7. CALLERS | Unit tests | Transitive depth | +| 8. CALLEES | Unit tests | | +| 9. REFERENCES | Unit tests | | +| 10. IMPACT | Unit tests | | +| 11. CONTEXT | Unit tests | Deep depth | +| 12. STRUCTURE | Unit tests | | +| 13. Multi-step | | All manual | +| 14. Failure modes | Partial | | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | REFRESH | Fast mode | | | | | +| 1.2 | REFRESH | Full mode | | | | | +| 1.3 | REFRESH | Full + source | | | | | +| 1.4 | REFRESH | Force | | | | | +| 1.5 | REFRESH | Background | | | | | +| 1.6 | REFRESH | Caching | | | | | +| 2.1 | TABLES | List tables | | | | | +| 2.2 | TABLES | Build mode check | | | | | +| 3.1 | STATUS | Show status | | | | | +| 4.1 | DESCRIBE | Entities table | | | | | +| 4.2 | DESCRIBE | Activities table | | | | | +| 5.1 | SELECT | Basic | | | | | +| 5.2 | SELECT | Multiple conditions | | | | | +| 5.3 | SELECT | Aggregate | | | | | +| 5.4 | SELECT | Join | | | | | +| 5.5 | SELECT | Full-only warning | | | | | +| 5.6 | SELECT | FTS search | | | | | +| 6.1 | SEARCH | Basic | | | | | +| 6.2 | SEARCH | Special chars | | | | | +| 6.3 | SEARCH | Source mode | | | | | +| 6.4 | SEARCH | No results | | | | | +| 7.1 | CALLERS | Direct | | | | | +| 7.2 | CALLERS | Transitive | | | | | +| 7.3 | CALLERS | None | | | | | +| 8.1 | CALLEES | Direct | | | | | +| 8.2 | CALLEES | Transitive | | | | | +| 9.1 | REFERENCES | Entity | | | | | +| 9.2 | REFERENCES | Microflow | | | | | +| 10.1 | IMPACT | Entity | | | | | +| 10.2 | IMPACT | Microflow | | | | | +| 11.1 | CONTEXT | Microflow | | | | | +| 11.2 | CONTEXT | Entity | | | | | +| 11.3 | CONTEXT | Page | | | | | +| 11.4 | CONTEXT | Depth 3 | | | | | +| 11.5 | CONTEXT | Not found | | | | | +| 12.1 | STRUCTURE | Depth 1 | | | | | +| 12.2 | STRUCTURE | Depth 2 | | | | | +| 12.3 | STRUCTURE | Depth 3 | | | | | +| 12.4 | STRUCTURE | Module filter | | | | | +| 12.5 | STRUCTURE | All modules | | | | | +| 13.1 | MULTI-STEP | Create + refresh + query | | | | | +| 13.2 | MULTI-STEP | Impact before drop | | | | | +| 14.1 | FAILURE | No catalog | | | | | +| 14.2 | FAILURE | Invalid SQL | | | | | +| 14.3 | FAILURE | Search no full | | | | | +| 14.4 | FAILURE | Callers no full | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/cli-commands-test-cases.md b/docs/15-testing/cli-commands-test-cases.md new file mode 100644 index 00000000..150a9800 --- /dev/null +++ b/docs/15-testing/cli-commands-test-cases.md @@ -0,0 +1,721 @@ +# CLI Commands Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +Covers: docker, test, playwright, eval, init, setup, auth, marketplace, widget, bson, fmt, new, serve, lsp, tui, check, report. + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +**Prerequisites:** Docker Desktop must be running. Commands `docker`, `test`, and `playwright` require it. + +--- + +## 1. mxcli docker + +### 1.1 Init Docker configuration + +```bash +mxcli docker init -p "$MPR" +``` + +**Expected:** Creates `.docker/docker-compose.yml`, `.env.example`, and `.env` in the project directory. Exit code `0`. + +### 1.2 Build Docker image + +```bash +mxcli docker build -p "$MPR" +``` + +**Expected:** Builds image from generated Dockerfile. Exit code `0`. + +### 1.3 Run container + +```bash +mxcli docker run -p "$MPR" --port 8080 --admin-port 8090 +``` + +**Expected:** Starts Mendix runtime container. App reachable at `http://localhost:8080`. Admin at `http://localhost:8090`. Exit code `0`. + +### 1.4 Docker Compose up + +```bash +mxcli docker up -p "$MPR" +``` + +**Expected:** Starts services via `docker-compose up -d`. Exit code `0`. + +### 1.5 Docker Compose down + +```bash +mxcli docker down -p "$MPR" +``` + +**Expected:** Stops and removes containers. Exit code `0`. + +### 1.6 Run MxBuild project validation + +```bash +mxcli docker check -p "$MPR" +``` + +**Expected:** Runs `mx check` (MxBuild project validation) inside Docker. Prints errors and warnings. Exit code `0` if no errors, `1` if errors found. + +### 1.7 View logs + +```bash +mxcli docker logs -p "$MPR" +``` + +**Expected:** Streams container logs to stdout. Exit code `0`. + +### 1.8 Container status + +```bash +mxcli docker status -p "$MPR" +``` + +**Expected:** Prints container state, ports, uptime. Exit code `0`. + +### 1.9 Open shell + +```bash +mxcli docker shell -p "$MPR" +``` + +**Expected:** Opens interactive shell inside the running container. + +### 1.10 Reload runtime + +```bash +mxcli docker reload -p "$MPR" +``` + +**Expected:** Rebuilds and restarts the runtime without recreating the container. Exit code `0`. + +--- + +## 2. mxcli test + +### 2.1 Run all tests in directory + +```bash +mxcli test tests/ -p "$MPR" +``` + +**Expected:** Discovers `.test.mdl` and `.test.md` files. Injects TestRunner microflow, builds via MxBuild, runs in Docker. Prints pass/fail summary. Exit code `0` if all pass, `1` if any fail. + +### 2.2 Run single test file + +```bash +mxcli test tests/my-flow.test.mdl -p "$MPR" +``` + +**Expected:** Runs only the specified test file. Exit code `0` on pass. + +### 2.3 List tests without running + +```bash +mxcli test tests/ -p "$MPR" --list +``` + +**Expected:** Prints test names from `@test`/`@expect` annotations. Does not build or run. Exit code `0`. + +### 2.4 JUnit output + +```bash +mxcli test tests/ -p "$MPR" --junit report.xml +``` + +**Expected:** Writes JUnit XML to `report.xml`. Exit code reflects pass/fail. + +### 2.5 Skip build + +```bash +mxcli test tests/ -p "$MPR" --skip-build +``` + +**Expected:** Skips MxBuild step. Uses existing build artifacts. Exit code `0` on pass. + +### 2.6 Verbose output + +```bash +mxcli test tests/ -p "$MPR" --verbose +``` + +**Expected:** Prints detailed output per test case including microflow execution logs. + +### 2.7 Custom timeout + +```bash +mxcli test tests/ -p "$MPR" --timeout 120 +``` + +**Expected:** Fails tests that exceed 120 seconds. Exit code `1` on timeout. + +### 2.8 Color output + +```bash +mxcli test tests/ -p "$MPR" --color +``` + +**Expected:** Forces colored output even when piped. Green for pass, red for fail. + +--- + +## 3. mxcli playwright verify + +### 3.1 Run verification scripts + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" +``` + +**Expected:** Discovers `.test.sh` scripts. Launches app in Docker, runs each script. Captures screenshots on failure. Exit code `0` if all pass. + +### 3.2 List verification scripts + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" --list +``` + +**Expected:** Prints discovered `.test.sh` filenames. Does not run. Exit code `0`. + +### 3.3 JUnit output + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" --junit pw-report.xml +``` + +**Expected:** Writes JUnit XML. Screenshots attached as test artifacts. + +### 3.4 Verbose mode + +```bash +mxcli playwright verify tests/e2e/ -p "$MPR" --verbose +``` + +**Expected:** Prints browser console logs and script output per test. + +--- + +## 4. mxcli eval + +### 4.1 Check evaluation results + +```bash +mxcli eval check +``` + +**Expected:** Runs AI correctness evaluation. Prints pass/fail per check. Exit code `0` if all pass. + +### 4.2 List evaluations + +```bash +mxcli eval list +``` + +**Expected:** Lists available evaluation suites. Exit code `0`. + +--- + +## 5. mxcli init + +### 5.1 Scaffold AI/IDE config + +```bash +mxcli init -p "$MPR" +``` + +**Expected:** Creates `CLAUDE.md`, `.vscode/settings.json`, and other AI config files in the project directory. Exit code `0`. + +### 5.2 Init in empty directory + +```bash +mkdir /tmp/empty-project && mxcli init -p /tmp/empty-project +``` + +**Expected:** Exit code `0`. The `-p` flag does not validate that the path contains an `.mpr` file — command succeeds regardless. **Known behavior:** no MPR validation on init. + +--- + +## 6. mxcli setup + +### 6.1 Setup MxBuild + +```bash +mxcli setup mxbuild +``` + +**Expected:** Downloads and installs MxBuild for the project's Studio Pro version. Exit code `0`. + +### 6.2 Setup MxRuntime + +```bash +mxcli setup mxruntime +``` + +**Expected:** Downloads and installs MxRuntime. Exit code `0`. + +### 6.3 Setup mxcli itself + +```bash +mxcli setup mxcli +``` + +**Expected:** Updates mxcli to latest version. Exit code `0`. + +--- + +## 7. mxcli auth + +### 7.1 Login (interactive) + +```bash +mxcli auth login +``` + +**Expected:** Prompts for PAT (or opens browser). Stores token locally. Prints confirmation. Exit code `0`. + +### 7.1b Login (non-interactive) + +```bash +mxcli auth login --token "$MENDIX_PAT" +``` + +**Expected:** Validates token against marketplace API. Stores credential. Prints confirmation. Exit code `0`. + +### 7.2 Check status (not authenticated) + +```bash +mxcli auth status +``` + +**Expected:** Reports "no credential for profile". Exit code `1`. + +### 7.3 List accounts + +```bash +mxcli auth list +``` + +**Expected:** Lists stored credentials. Exit code `0`. + +### 7.4 Logout + +```bash +mxcli auth logout +``` + +**Expected:** Removes stored token. Prints "Removed profile". Exit code `0` (succeeds even when not logged in). + +### 7.5 Status after logout + +```bash +mxcli auth logout && mxcli auth status +``` + +**Expected:** Status reports "Not logged in". Exit code `1`. + +--- + +## 8. mxcli marketplace + +> **Prerequisite:** Requires authentication (`mxcli auth login`). Commands fail with "no credential" error without auth. + +### 8.1 Search modules + +```bash +mxcli marketplace search "email" --limit 5 +``` + +**Expected:** Prints table with ID, TYPE, PUBLISHER, SUPPORT, LATEST, NAME columns. Exit code `0`. + +### 8.2 Module info + +```bash +mxcli marketplace info 170 +``` + +**Expected:** Prints module details: Content ID, Type, Publisher, Support, Categories, Latest version, Min Mendix, Published date. Exit code `0`. + +> **Note:** Argument is a numeric content ID (not a name). Get IDs from `marketplace search`. + +### 8.3 Search with JSON output + +```bash +mxcli marketplace search "atlas" --limit 2 --json +``` + +**Expected:** JSON array with items containing contentId, publisher, type, supportCategory, latestVersion. Exit code `0`. + +### 8.4 Search with no results + +```bash +mxcli marketplace search "zzzznonexistent" +``` + +**Expected:** Empty result or "No results found." Exit code `0`. + +--- + +## 9. mxcli widget + +### 9.1 List widgets + +```bash +mxcli widget list -p "$MPR" +``` + +**Expected:** Lists all custom widgets in the project with name and version. Exit code `0`. + +### 9.2 Extract widget + +```bash +mxcli widget extract --mpk path/to/widget.mpk +``` + +**Expected:** Extracts widget package contents. Exit code `0`. + +### 9.3 Init new widget + +```bash +mxcli widget init MyNewWidget +``` + +**Expected:** Scaffolds widget project structure. Exit code `0`. + +### 9.4 Generate widget docs + +```bash +mxcli widget docs -p "$MPR" +``` + +**Expected:** Generates documentation for all custom widgets. Exit code `0`. + +--- + +## 10. mxcli bson + +### 10.1 Dump MPR contents + +```bash +mxcli bson dump -p "$MPR" --list +mxcli bson dump -p "$MPR" --object "Name" +``` + +**Expected:** `--list` prints object names in the MPR. `--object` dumps a specific named object. There is no simple "dump all" mode. Exit code `0`. + +### 10.2 Discover document types + +```bash +mxcli bson discover "$MPR" +``` + +**Expected:** Lists all document types and counts found in the MPR. Exit code `0`. + +### 10.3 Compare objects within or across projects + +```bash +# Compare objects within the same project by type +mxcli bson compare -p "$MPR" --type DomainModels$DomainModel + +# Compare across two projects +mxcli bson compare -p "$MPR" --p2 "$APPS_DIR/FactoryManagement/FactoryManagement.mpr" +``` + +**Expected:** Prints structural diff. Exit code `0` if identical, `1` if different. + +### 10.4 Dump nonexistent file + +```bash +mxcli bson dump /tmp/nonexistent.mpr +``` + +**Expected:** Error: file not found. Exit code `1`. + +--- + +## 11. mxcli fmt + +### 11.1 Format MDL file + +```bash +mxcli fmt myfile.mdl +``` + +**Expected:** Outputs formatted MDL to stdout (does NOT write in-place). Exit code `0`. + +### 11.2 Format already-formatted file + +```bash +mxcli fmt myfile.mdl +``` + +**Expected:** No changes. Exit code `0`. + +### 11.3 Format invalid file + +```bash +mxcli fmt broken.txt +``` + +**Expected:** Exit code `0`. Input is echoed unchanged to stdout (no parse error). + +--- + +## 12. mxcli new + +### 12.1 Create a new Mendix project + +```bash +mxcli new my-app --version 10.24.15 +``` + +**Expected:** Scaffolds a complete Mendix project named `my-app` for the specified Studio Pro version. Exit code `0`. + +### 12.2 Create project without version + +```bash +mxcli new my-app +``` + +**Expected:** Creates project using the latest available Studio Pro version. Exit code `0`. + +--- + +## 13. mxcli serve + +### 13.1 Start HTTP server + +```bash +mxcli serve -p "$MPR" --port 9000 & +curl -s http://localhost:9000/ +kill %1 +``` + +**Expected:** Server starts and serves project visualization at root `/`. Exit code `0` on clean shutdown. + +### 13.2 Port conflict + +```bash +mxcli serve -p "$MPR" --port 9000 & +mxcli serve -p "$MPR" --port 9000 +``` + +**Expected:** Second instance fails with "port already in use". Exit code `1`. + +--- + +## 14. mxcli lsp + +### 14.1 Start LSP server + +```bash +echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' | mxcli lsp +``` + +**Expected:** Returns JSON-RPC `initialize` response with server capabilities (completion, hover, diagnostics, documentSymbol, foldingRange). Exit code `0`. + +### 14.2 Verify capabilities + +Inspect the `initialize` response `capabilities` object: + +| Capability | Expected | +|------------|----------| +| `completionProvider` | Present | +| `hoverProvider` | `true` | +| `diagnosticProvider` | Present | +| `documentSymbolProvider` | `true` | +| `foldingRangeProvider` | `true` | + +--- + +## 15. mxcli tui + +### 15.1 Launch TUI + +```bash +mxcli tui -p "$MPR" +``` + +**Expected:** Opens terminal UI explorer. Displays project tree. Arrow keys navigate. `q` exits. Exit code `0`. + +### 15.2 TUI without project + +```bash +mxcli tui +``` + +**Expected:** Error: `-p` flag required. Exit code `1`. + +--- + +## 16. mxcli check + +### 16.1 Validate MDL script syntax + +```bash +mxcli check script.mdl +``` + +**Expected:** Validates MDL script syntax (not the project itself — use `docker check` for project validation). Prints errors and warnings. Exit code `0` if no errors, `1` if errors found. + +### 16.2 SARIF output + +```bash +mxcli check script.mdl --format sarif > results.sarif +``` + +**Expected:** Writes SARIF JSON to stdout. Exit code reflects validation result. + +### 16.3 Check nonexistent file + +```bash +mxcli check /tmp/nonexistent.mdl +``` + +**Expected:** Error: file not found. Exit code `1`. + +--- + +## 17. mxcli report + +### 17.1 Markdown report + +```bash +mxcli report -p "$MPR" --format markdown +``` + +**Expected:** Prints project report in Markdown to stdout. Exit code `0`. + +### 17.2 JSON report + +```bash +mxcli report -p "$MPR" --format json +``` + +**Expected:** Prints valid JSON report. Exit code `0`. + +### 17.3 HTML report + +```bash +mxcli report -p "$MPR" --format html --output report.html +``` + +**Expected:** Writes HTML report to `report.html`. Exit code `0`. + +--- + +## Failure Modes + +| Scenario | Command | Expected error | Exit code | +|----------|---------|---------------|-----------| +| Missing `-p` flag | `mxcli test tests/` | "required flag: -p" | `1` | +| Invalid MDL path | `mxcli check /bad/path.mdl` | "file not found" | `1` | +| Docker not running | `mxcli docker run -p "$MPR"` | "cannot connect to Docker daemon" | `1` | +| MxBuild not installed | `mxcli docker check -p "$MPR"` | "MxBuild not found — run mxcli setup mxbuild" | `1` | +| Auth token expired | `mxcli marketplace search "x"` | "token expired — run mxcli auth login" | `1` | +| Port conflict | `mxcli serve --port ` | "address already in use" | `1` | +| Invalid test file | `mxcli test bad.txt -p "$MPR"` | "unsupported test file format" | `1` | +| Timeout exceeded | `mxcli test --timeout 1` | "test timed out after 1s" | `1` | +| BSON corrupt file | `mxcli bson dump corrupt.mpr` | "invalid BSON data" | `1` | + +> **Note:** Several commands return exit `0` on error instead of `1`: `bson dump` (missing args), `docker build` (build failure), `docker check` (in some failure modes), `marketplace search` (auth failure), `serve` (port conflict), `fmt` (invalid input). Do not rely solely on exit codes for pass/fail in automation. + +--- + +## Test Project Coverage Matrix + +| Command | Subcommands | Happy path | Error path | Flags tested | +|---------|-------------|:----------:|:----------:|:------------:| +| `docker` | run, build, check, init, up, down, logs, status, shell, reload | 10 | 1 | -p, --port, --admin-port | +| `test` | — | 8 | 2 | -p, --list, --junit, --skip-build, --verbose, --color, --timeout | +| `playwright verify` | — | 4 | 0 | -p, --list, --junit, --verbose | +| `eval` | check, list | 2 | 0 | — | +| `init` | — | 1 | 1 | -p | +| `setup` | mxbuild, mxruntime, mxcli | 3 | 0 | — | +| `auth` | login, logout, status, list | 5 | 0 | — | +| `marketplace` | search, info | 4 | 0 | — | +| `widget` | extract, list, init, docs | 4 | 0 | -p | +| `bson` | dump, discover, compare | 3 | 2 | — | +| `fmt` | — | 2 | 1 | — | +| `new` | — | 2 | 0 | --version | +| `serve` | — | 1 | 1 | -p, --port | +| `lsp` | — | 2 | 0 | — | +| `tui` | — | 1 | 1 | -p | +| `check` | — | 2 | 1 | --format | +| `report` | — | 3 | 0 | -p, --format, --output | + +**Total: 56 happy-path + 11 error-path = 67 test cases** + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**mxcli version:** _______________ + +| # | Command | Case | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | `docker init` | Init Docker config | | | | | +| 1.2 | `docker build` | Build image | | | | | +| 1.3 | `docker run` | Run container | | | | | +| 1.4 | `docker check` | Runtime check | | | | | +| 1.5 | `docker up` | Compose up | | | | | +| 1.6 | `docker down` | Compose down | | | | | +| 1.7 | `docker logs` | View logs | | | | | +| 1.8 | `docker status` | Container status | | | | | +| 1.9 | `docker shell` | Interactive shell | | | | | +| 1.10 | `docker reload` | Hot reload | | | | | +| 2.1 | `test` | Run all tests | | | | | +| 2.2 | `test` | Single file | | | | | +| 2.3 | `test --list` | List tests | | | | | +| 2.4 | `test --junit` | JUnit output | | | | | +| 2.5 | `test --skip-build` | Skip build | | | | | +| 2.6 | `test --verbose` | Verbose output | | | | | +| 2.7 | `test --color` | Colored output | | | | | +| 2.8 | `test --timeout` | Custom timeout | | | | | +| 3.1 | `playwright` | Run scripts | | | | | +| 3.2 | `playwright --list` | List scripts | | | | | +| 3.3 | `playwright --junit` | JUnit output | | | | | +| 3.4 | `playwright --verbose` | Verbose output | | | | | +| 4.1 | `eval check` | Check correctness | | | | | +| 4.2 | `eval list` | List evaluations | | | | | +| 5.1 | `init` | Init AI config | | | | | +| 6.1 | `setup mxbuild` | Install MxBuild | | | | | +| 6.2 | `setup mxruntime` | Install runtime | | | | | +| 6.3 | `setup mxcli` | Self-update | | | | | +| 7.1 | `auth login` | Login flow | | | | | +| 7.2 | `auth logout` | Logout | | | | | +| 7.3 | `auth status` | Auth status | | | | | +| 7.4 | `auth list` | List accounts | | | | | +| 7.5 | `auth` | Token refresh | | | | | +| 8.1 | `marketplace search` | Search modules | | | | | +| 8.2 | `marketplace info` | Module info | | | | | +| 8.3 | `marketplace versions` | List versions | | | | | +| 8.4 | `marketplace` | Install module | | | | | +| 9.1 | `widget extract` | Extract widget | | | | | +| 9.2 | `widget list` | List widgets | | | | | +| 9.3 | `widget init` | Init widget project | | | | | +| 9.4 | `widget docs` | Generate docs | | | | | +| 10.1 | `bson dump` | Dump MPR | | | | | +| 10.2 | `bson discover` | Discover structure | | | | | +| 10.3 | `bson compare` | Compare MPRs | | | | | +| 11.1 | `fmt` | Format file | | | | | +| 11.2 | `fmt` | Format directory | | | | | +| 12.1 | `new` | Create project | | | | | +| 13.1 | `serve` | Start server | | | | | +| 14.1 | `lsp` | Start LSP | | | | | +| 14.2 | `lsp` | Diagnostics | | | | | +| 15.1 | `tui` | Terminal UI | | | | | +| 16.1 | `check` | Validate project | | | | | +| 16.2 | `check --sarif` | SARIF output | | | | | +| 17.1 | `report` | Markdown report | | | | | +| 17.2 | `report --format json` | JSON report | | | | | +| 17.3 | `report --output` | Output to file | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/entity-test-cases.md b/docs/15-testing/entity-test-cases.md new file mode 100644 index 00000000..d1a4dc2f --- /dev/null +++ b/docs/15-testing/entity-test-cases.md @@ -0,0 +1,1334 @@ +# Entity, Association & Constant Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +**Note:** `MyModule` will be auto-created on first CREATE statement — no manual setup needed. + +--- + +## 1. SHOW ENTITIES + +### 1.1 List all entities + +``` +show entities; +``` + +**Expected:** Table with columns `Entity | Type | Attrs | Assocs | Validations | Indexes | Events | AccessRules`. Summary line `(N entities)`. Sorted alphabetically. + +### 1.2 List entities in a module + +``` +show entities in Administration; +``` + +**Expected:** Only entities from `Administration` module. Same column format. + +### 1.3 Generalization column + +``` +show entities; +``` + +**Expected:** If any entity has a generalization, an `Extends` column appears between `Type` and `Attrs`. System entities (referenced via generalization) show `-` for counts. + +### 1.4 Empty module + +``` +show entities in NonExistentModule; +``` + +**Expected:** Error or empty result with `(0 entities)`. + +### 1.5 Entity types + +Verify `Type` column values across projects: + +| Type | Example | +|------|---------| +| Persistent | Most domain entities | +| Non-Persistent | Transient/helper entities | +| External | OData/external entities | +| System | System.User, System.Session | + +--- + +## 2. SHOW ENTITY (single) + +### 2.1 Persistent entity + +``` +show entity Administration.Account; +``` + +**Expected:** Markdown-style output: +``` +**Entity: Administration.Account** + +- Persistable: true +- Extends: System.User + +| Attribute | Type | +| --------- | ----------- | +| ... | ... | + +(N attributes) +``` + +### 2.2 Non-persistent entity + +``` +show entity ; +``` + +**Expected:** `Persistable: false`. No `Extends` if no generalization. + +### 2.3 Non-existent entity + +``` +show entity Fake.Entity; +``` + +**Expected:** Error message (not found). + +--- + +## 3. DESCRIBE ENTITY + +### 3.1 Basic entity + +``` +describe entity Administration.Account; +``` + +**Expected:** Roundtrippable MDL output: +``` +create or modify persistent entity Administration.Account extends System.User ( + FullName: string(200), + IsLocalUser: boolean, + ... +) +/ +``` + +### 3.2 Entity with all features + +Find an entity with indexes, events, validation rules, and access rules. Verify output includes: +- `/** documentation */` block (if documented) +- `@Position(x, y)` annotation +- Attribute types and constraints (`not null error '...'`, `unique error '...'`) +- System pseudo-attributes (`Owner: AutoOwner`, etc.) +- `index (col1, col2 desc)` lines +- `on before|after create|commit|delete|rollback call Module.Mf($currentObject) [raise error]` events +- `grant Role1, Role2 on Module.Entity (create, delete, read *, write (Col1, Col2)) [where 'xpath']` access rules + +### 3.3 View entity + +``` +describe entity ; +``` + +**Expected:** Output ends with `as (` followed by indented OQL query, then `)`. + +### 3.4 Non-existent entity + +``` +describe entity Fake.Missing; +``` + +**Expected:** Error message (not found). + +--- + +## 4. CREATE ENTITY + +### 4.1 Minimal persistent entity + +``` +create persistent entity MyModule.SimpleTag ( + Name: string(50) +); +``` + +**Expected:** Entity created. `show entity MyModule.SimpleTag` shows 1 attribute. + +### 4.2 All attribute types + +``` +create persistent entity MyModule.AllTypes ( + ProductId: autonumber default 1, + Name: string(200) not null error 'Name is required', + Code: string(50) unique error 'Code must be unique', + Price: decimal not null error 'Price required' default 0.00, + Stock: integer default 0, + IsActive: boolean default true, + ReleaseDate: date, + CreatedAt: datetime, + ProductImage: binary, + Status: enumeration(MyModule.Status) +); +``` + +**Expected:** Entity created with 10 attributes. Types and constraints preserved in `describe`. + +> **Note:** `default 0.00` is normalized to `default 0` for decimals. + +### 4.3 Entity with generalization + +``` +create persistent entity MyModule.SpecialUser extends System.User ( + Department: string(100) +); +``` + +**Expected:** Entity created. `show entity` shows `Extends: System.User`. + +### 4.4 Non-persistent entity + +``` +create non-persistent entity MyModule.SearchParams ( + SearchTerm: string(200), + IncludeInactive: boolean default false +); +``` + +**Expected:** Entity created. `Persistable: false`. + +### 4.5 Entity with indexes + +``` +create persistent entity MyModule.Order ( + OrderNumber: string(50) not null unique, + CustomerId: integer, + OrderDate: datetime +) +index (OrderNumber), +index (CustomerId asc), +index (OrderDate desc); +``` + +**Expected:** Entity created. `describe` shows 3 index definitions. + +### 4.6 Entity with event handlers + +``` +create persistent entity MyModule.BankAccount ( + IBAN: string(34) not null +) +on before commit call MyModule.ACT_ValidateIBAN($currentObject) raise error; +``` + +**Expected:** Entity created. `describe` shows event handler. + +> **Prerequisite:** Event handlers and calculated-by microflows require the target microflow to exist. Create the microflow before the entity/alter statement. + +### 4.7 Entity with system attributes + +``` +create or modify persistent entity MyModule.AuditedRecord ( + Title: string(200) not null, + Owner: autoowner, + ChangedBy: autochangedby, + CreatedDate: autocreateddate, + ChangedDate: autochangeddate +); +``` + +**Expected:** Entity created with system attributes. `describe` shows pseudo-attributes. + +### 4.8 `create or modify` — new entity + +``` +create or modify persistent entity MyModule.Fresh ( + Value: integer +); +``` + +**Expected:** Creates entity (same as without `or modify`). + +### 4.9 `create or modify` — existing entity + +``` +create or modify persistent entity MyModule.Fresh ( + Value: integer, + Extra: string(100) +); +``` + +**Expected:** Updates entity — adds `Extra` attribute, preserves entity ID. + +### 4.10 Duplicate entity (without `or modify`) + +``` +create persistent entity MyModule.SimpleTag ( + Name: string(50) +); +``` + +**Expected:** Error — entity already exists. + +### 4.11 View entity with OQL + +``` +create view entity MyModule.ActiveProducts ( + Name: string(200), + Price: decimal +) as + select p.Name as Name, p.Price as Price + from MyModule.Product as p + where p.IsActive = true; +``` + +**Expected:** View entity created. `describe` shows OQL source. + +### 4.12 `create or replace` view entity + +``` +create or replace view entity MyModule.ActiveProducts ( + Name: string(200), + Price: decimal, + Stock: integer +) as + select p.Name, p.Price, p.Stock + from MyModule.Product as p + where p.IsActive = true; +``` + +**Expected:** Drops and recreates view entity (new ID). + +### 4.13 Position annotation + +``` +@position(300, 400) +create persistent entity MyModule.Positioned ( + Label: string(100) +); +``` + +**Expected:** Entity created at position (300, 400). `describe` shows `@Position(300, 400)`. + +--- + +## 5. ALTER ENTITY + +### 5.1 Add attribute + +``` +alter entity MyModule.SimpleTag add attribute Description: string(500); +``` + +**Expected:** Attribute added. `describe` shows both `Name` and `Description`. + +### 5.2 Rename attribute + +``` +alter entity MyModule.SimpleTag rename attribute Name to TagName; +``` + +**Expected:** Attribute renamed. `describe` shows `TagName`. + +### 5.3 Modify attribute type + +``` +alter entity MyModule.SimpleTag modify attribute TagName: string(200); +``` + +**Expected:** Attribute type updated to `string(200)`. + +### 5.4 Drop attribute + +``` +alter entity MyModule.SimpleTag drop attribute Description; +``` + +**Expected:** Attribute removed. Validation rules, index refs, and MemberAccess entries cleaned up. + +### 5.5 Set documentation + +``` +alter entity MyModule.SimpleTag set documentation 'A simple tag entity for testing'; +``` + +**Expected:** `describe` shows `/** A simple tag entity for testing */`. + +### 5.6 Set position + +``` +alter entity MyModule.SimpleTag set position (150, 250); +``` + +**Expected:** `describe` shows `@Position(150, 250)`. + +### 5.7 Add index + +``` +alter entity MyModule.SimpleTag add index (TagName); +``` + +**Expected:** `describe` shows `index (TagName)`. + +### 5.8 Drop index + +``` +alter entity MyModule.SimpleTag drop index idx1; +``` + +**Expected:** Index removed. + +### 5.9 Add event handler + +``` +alter entity MyModule.SimpleTag + add event handler on before commit call MyModule.ACT_ValidateTag($currentObject) raise error; +``` + +**Expected:** Event handler added to entity. + +> **Prerequisite:** Event handlers and calculated-by microflows require the target microflow to exist. Create the microflow before the entity/alter statement. + +### 5.10 Drop event handler + +``` +alter entity MyModule.SimpleTag drop event handler on before commit; +``` + +**Expected:** Event handler removed. + +### 5.11 Multiple actions + +``` +alter entity MyModule.SimpleTag + add attribute Priority: integer default 0 + add index (Priority) + set documentation 'Updated tag entity'; +``` + +**Expected:** All three changes applied in one statement. + +### 5.12 Drop system pseudo-attribute + +``` +alter entity MyModule.AuditedRecord drop attribute owner; +``` + +**Expected:** Clears `HasOwner` flag on entity. `describe` no longer shows `Owner: AutoOwner`. + +### 5.13 Make attribute calculated + +``` +alter entity MyModule.SimpleTag modify attribute TagName: string(200) calculated by MyModule.CalcTagName; +``` + +**Expected:** Attribute marked as calculated. `describe` shows `calculated by MyModule.CalcTagName`. + +> **Prerequisite:** Event handlers and calculated-by microflows require the target microflow to exist. Create the microflow before the entity/alter statement. + +--- + +## 6. DROP ENTITY + +### 6.1 Drop existing entity + +``` +drop entity MyModule.SimpleTag; +``` + +**Expected:** Entity removed. `show entity MyModule.SimpleTag` returns error. + +### 6.2 Drop non-existent entity + +``` +drop entity MyModule.NonExistent; +``` + +**Expected:** Error — entity not found. + +### 6.3 Drop view entity + +``` +drop entity MyModule.ActiveProducts; +``` + +**Expected:** View entity and its ViewEntitySourceDocument removed. + +### 6.4 Drop entity with references + +``` +drop entity MyModule.Order; +``` + +**Expected:** Entity with references is dropped without warning. No warning issued. + +--- + +## 7. SHOW ASSOCIATIONS + +### 7.1 List all associations + +``` +show associations; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Parent | Child | Type | Owner | Storage`. Summary `(N associations)`. + +### 7.2 List associations in a module + +``` +show associations in MyModule; +``` + +**Expected:** Only associations from `MyModule`. Same format. + +### 7.3 Cross-module associations + +``` +show associations; +``` + +**Expected:** Cross-module associations listed with full qualified names for both parent and child entities. + +--- + +## 8. SHOW ASSOCIATION (single) + +### 8.1 Basic association + +``` +show association MyModule.Order_Customer; +``` + +**Expected:** +``` +Association: MyModule.Order_Customer + Type: Reference + Owner: Default + Storage: Column +``` + +### 8.2 Cross-module association + +``` +show association ; +``` + +**Expected:** Shows `(cross-module)` and full child entity path. + +### 8.3 Non-existent association + +``` +show association Fake.Missing; +``` + +**Expected:** Error — association not found. + +--- + +## 9. DESCRIBE ASSOCIATION + +### 9.1 Reference association + +``` +describe association MyModule.Order_Customer; +``` + +**Expected:** +``` +create association MyModule.Order_Customer +from MyModule.Order to MyModule.Customer +type Reference +owner Default +delete_behavior DELETE_BUT_KEEP_REFERENCES; +/ +``` + +### 9.2 ReferenceSet association + +``` +describe association MyModule.Project_Employees; +``` + +**Expected:** `type ReferenceSet`, `owner Both`. + +> **Note:** Owner depends on existing project data; may show `owner Default`. + +### 9.3 Association with documentation + +Find an association with a doc comment. Verify `/** ... */` appears in output. + +--- + +## 10. CREATE ASSOCIATION + +### 10.1 Basic reference (many-to-one) + +``` +create association MyModule.Order_Customer +from MyModule.Order to MyModule.Customer +type reference +owner default +delete_behavior DELETE_BUT_KEEP_REFERENCES; +``` + +**Expected:** Association created. `describe` matches input. + +### 10.2 ReferenceSet (many-to-many) + +``` +create association MyModule.Project_Employees +from MyModule.Project to MyModule.Employee +type referenceset +owner both; +``` + +**Expected:** Association created with `type ReferenceSet`, `owner Both`. + +### 10.3 Self-referencing association + +``` +create association MyModule.Category_Parent +from MyModule.Category to MyModule.Category +type reference +owner default +delete_behavior DELETE_BUT_KEEP_REFERENCES; +``` + +**Expected:** Association created. Parent and child are same entity. + +### 10.4 Cross-module association + +``` +create association MyModule.Account_User +from MyModule.Account to Administration.Account +type reference +owner default; +``` + +**Expected:** Association created spanning two modules. + +### 10.5 With explicit storage + +``` +create association MyModule.Tag_Item +from MyModule.Tag to MyModule.Item +type reference +owner default +storage table; +``` + +**Expected:** Association uses table storage instead of column. + +> **Note:** `describe` omits `storage table` (only `show` reveals storage info). + +### 10.6 Delete behavior — CASCADE + +``` +create association MyModule.Parent_Child +from MyModule.Parent to MyModule.Child +type reference +owner default +delete_behavior CASCADE; +``` + +**Expected:** `describe` shows `DELETE_CASCADE`. + +### 10.7 Delete behavior — IF_NO_REFERENCES + +``` +create association MyModule.Strict_Ref +from MyModule.A to MyModule.B +type reference +owner default +delete_behavior DELETE_IF_NO_REFERENCES; +``` + +**Expected:** `describe` shows `DELETE_IF_NO_REFERENCES`. + +### 10.8 Duplicate association + +``` +create association MyModule.Order_Customer +from MyModule.Order to MyModule.Customer +type reference +owner default; +``` + +**Expected:** Error — duplicate association name. + +--- + +## 11. ALTER ASSOCIATION + +### 11.1 Change delete behavior + +``` +alter association MyModule.Order_Customer set delete_behavior CASCADE; +``` + +**Expected:** `describe` shows updated behavior (`DELETE_CASCADE` in output). + +### 11.2 Change owner + +``` +alter association MyModule.Order_Customer set owner both; +``` + +**Expected:** `describe` shows `owner Both`. + +### 11.3 Change storage + +``` +alter association MyModule.Order_Customer set storage table; +``` + +**Expected:** `describe` shows `storage table`. + +### 11.4 Set comment + +``` +alter association MyModule.Order_Customer set comment 'Links orders to customers'; +``` + +**Expected:** `describe` shows documentation comment. + +--- + +## 12. DROP ASSOCIATION + +### 12.1 Drop existing association + +``` +drop association MyModule.Order_Customer; +``` + +**Expected:** Association removed. `show association MyModule.Order_Customer` returns error. + +### 12.2 Drop non-existent association + +``` +drop association MyModule.Fake; +``` + +**Expected:** Error — not found. + +### 12.3 Drop cross-module association + +``` +drop association MyModule.Account_User; +``` + +**Expected:** Cross-module association removed. + +--- + +## 13. SHOW CONSTANTS + +### 13.1 List all constants + +``` +show constants; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Folder | Type | Default | Exposed`. Sorted alphabetically. + +### 13.2 List constants in a module + +``` +show constants in MyModule; +``` + +**Expected:** Only constants from `MyModule`. + +### 13.3 Empty module + +``` +show constants in NonExistentModule; +``` + +**Expected:** `No constants found.` or error. + +--- + +## 14. SHOW CONSTANT VALUES + +### 14.1 All constant values + +``` +show constant values; +``` + +**Expected:** Table with columns `Constant | Configuration | Value`. One row per constant for `(default)`, additional rows per configuration override. Values truncated to 60 chars. + +### 14.2 Constants in a module + +``` +show constant values in MyModule; +``` + +**Expected:** Only constants from `MyModule`. Same format. + +--- + +## 15. DESCRIBE CONSTANT + +### 15.1 String constant + +``` +describe constant MyModule.ServiceEndpoint; +``` + +**Expected:** +``` +create or modify constant MyModule.ServiceEndpoint + type String + default 'https://api.example.com/v1' +; +/ +``` + +### 15.2 Boolean constant + +``` +describe constant MyModule.EnableDebug; +``` + +**Expected:** `default true` or `default false` (unquoted). + +### 15.3 Constant with folder and comment + +``` +describe constant MyModule.MaxRetries; +``` + +**Expected:** Output includes `folder 'path/to/folder'` and `comment 'text'` if present. + +### 15.4 Exposed constant + +``` +describe constant MyModule.ExposedConst; +``` + +**Expected:** Output includes `exposed to client`. + +--- + +## 16. CREATE CONSTANT + +### 16.1 String constant + +``` +create constant MyModule.ServiceEndpoint type String default 'https://api.example.com/v1'; +``` + +**Expected:** Constant created. `describe` matches. + +### 16.2 Integer constant + +``` +create constant MyModule.MaxRetries type Integer default 3; +``` + +**Expected:** Constant created with integer type and value. + +### 16.3 Long constant + +``` +create constant MyModule.MaxFileSize type Long default 10485760 comment 'Max upload 10MB'; +``` + +**Expected:** Constant with comment. + +> **Note:** Long type maps to Integer. No distinct Long constant type exists. + +### 16.4 Decimal constant + +``` +create constant MyModule.TaxRate type Decimal default 0.21; +``` + +**Expected:** Decimal value preserved. + +### 16.5 Boolean constant + +``` +create constant MyModule.EnableDebug type Boolean default false; +``` + +**Expected:** Boolean constant created. + +### 16.6 DateTime constant + +``` +create constant MyModule.LaunchDate type DateTime default '[%BeginOfCurrentDay%]'; +``` + +**Expected:** DateTime constant with token expression. + +### 16.7 Enumeration constant + +``` +create constant MyModule.DefaultStatus type Enumeration(MyModule.Status) default 'Active'; +``` + +**Expected:** Constant with enumeration type. + +### 16.8 With folder + +``` +create constant MyModule.Nested type String default 'value' folder 'Config/SubFolder'; +``` + +**Expected:** Constant placed in specified folder. Folder auto-created if missing. + +### 16.9 Exposed to client + +``` +create constant MyModule.ClientConst type String default 'visible' exposed to client; +``` + +**Expected:** `describe` shows `exposed to client`. + +### 16.10 `create or modify` — existing constant + +``` +create or modify constant MyModule.MaxRetries type Integer default 5 comment 'Updated'; +``` + +**Expected:** Updates existing constant value and comment. + +### 16.11 Duplicate constant (without `or modify`) + +``` +create constant MyModule.MaxRetries type Integer default 3; +``` + +**Expected:** Error — constant already exists. + +### 16.12 With documentation comment + +``` +/** Maximum number of API retry attempts */ +create constant MyModule.ApiRetries type Integer default 3; +``` + +**Expected:** Constant created with doc comment. + +--- + +## 17. DROP CONSTANT + +### 17.1 Drop existing constant + +``` +drop constant MyModule.ServiceEndpoint; +``` + +**Expected:** Constant removed. + +### 17.2 Drop non-existent constant + +``` +drop constant MyModule.Fake; +``` + +**Expected:** Error — not found. + +--- + +## 18. ROUNDTRIP (BSON) + +Test that CREATE → DESCRIBE → CREATE (from output) produces identical results. + +### 18.1 Entity roundtrip + +``` +create persistent entity RtTest.Employee ( + Name: string(200) not null, + Salary: decimal default 0.00, + Active: boolean default true +) +index (Name); +``` + +1. Run `describe entity RtTest.Employee` +2. Copy output (removing leading `create or modify` → just `create`) +3. Drop entity: `drop entity RtTest.Employee` +4. Execute copied MDL +5. Run `describe` again + +**Expected:** Output identical between step 1 and step 5. + +### 18.2 Association roundtrip + +``` +create association RtTest.Emp_Dept +from RtTest.Employee to RtTest.Department +type reference +owner default +delete_behavior DELETE_CASCADE; +``` + +1. `describe association RtTest.Emp_Dept` +2. Drop: `drop association RtTest.Emp_Dept` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +### 18.3 Constant roundtrip + +``` +create constant RtTest.ApiKey type String default 'sk-test-123' comment 'Test key' exposed to client; +``` + +1. `describe constant RtTest.ApiKey` +2. Drop: `drop constant RtTest.ApiKey` +3. Execute described MDL +4. `describe` again + +**Expected:** Identical output. + +--- + +## 19. CATALOG INTEGRATION + +### 19.1 Entity in catalog + +``` +refresh catalog; +select * from CATALOG.ENTITIES where name = 'Account'; +``` + +**Expected:** Entity appears in catalog with correct module, type, attribute count. + +### 19.2 Association in catalog + +``` +select * from CATALOG.ASSOCIATIONS where parent_entity = 'MyModule.Order'; +``` + +**Expected:** Empty result or error — table not implemented. + +### 19.3 References for entity + +``` +show references to Administration.Account; +``` + +**Expected:** Lists microflows, pages, and other documents referencing this entity. + +--- + +## 20. MULTI-STEP WORKFLOWS + +### 20.1 Create entity → add association → grant access + +``` +create persistent entity MyModule.Product ( + Name: string(200) not null, + Price: decimal default 0 +); + +create persistent entity MyModule.Category ( + Label: string(100) +); + +create association MyModule.Product_Category +from MyModule.Product to MyModule.Category +type reference +owner default; + +create module role MyModule.Manager; +grant Manager on MyModule.Product (create, delete, read *, write *); +grant Manager on MyModule.Category (read *); +``` + +**Expected:** All statements succeed. `show entities in MyModule` shows both entities with correct association count. + +### 20.2 Create constant → modify → verify + +``` +create constant MyModule.Timeout type Integer default 30; +create or modify constant MyModule.Timeout type Integer default 60 comment 'Increased timeout'; +describe constant MyModule.Timeout; +``` + +**Expected:** Final `describe` shows `default 60` and comment. + +--- + +## 21. FAILURE MODES & ERROR RECOVERY + +### 21.1 Invalid attribute type + +``` +create persistent entity MyModule.Bad ( + Value: invalidtype +); +``` + +**Expected:** Unknown types are silently interpreted as enumeration references. No error raised. + +### 21.2 Missing module qualification + +``` +create persistent entity UnqualifiedName ( + X: integer +); +``` + +**Expected:** Error — module name required. + +### 21.3 Alter non-existent attribute + +``` +alter entity MyModule.Product drop attribute FakeAttr; +``` + +**Expected:** Error — attribute not found. + +### 21.4 Create association with missing entity + +``` +create association MyModule.Bad_Ref +from MyModule.Nonexistent to MyModule.Product +type reference +owner default; +``` + +**Expected:** Error — parent entity not found. + +### 21.5 Invalid OQL in view entity + +``` +create view entity MyModule.BadView ( + X: integer +) as + select invalid syntax here; +``` + +**Expected:** Entity created (OQL not validated at create time). View entity stored with invalid OQL as raw text. + +### 21.6 Drop entity referenced by association + +``` +drop entity MyModule.Customer; +``` + +**Expected:** Entity silently dropped without warning. Associations may become orphaned. + +--- + +## 22. BOUNDARY & STRESS + +### 22.1 Entity with many attributes (50+) + +Create an entity with 50 attributes. Verify `show` and `describe` handle large attribute lists. + +### 22.2 Long attribute names + +``` +create persistent entity MyModule.LongNames ( + ThisIsAVeryLongAttributeNameThatExceedsNormalLength: string(200) +); +``` + +**Expected:** Created and displayed correctly. + +### 22.3 String constant with special characters + +``` +create constant MyModule.Special type String default 'it''s a "test" with\nnewlines'; +``` + +**Expected:** Escaped characters preserved in describe output. + +### 22.4 Maximum string length + +``` +create persistent entity MyModule.MaxLen ( + BigText: string(1048576) +); +``` + +**Expected:** Entity created (or error if max exceeded). + +### 22.5 Many associations on one entity + +Create 10+ associations from one entity to different targets. Verify `show associations` and entity describe handle them. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW ENTITIES | x | x | x | +| SHOW ENTITY | x | x | x | +| DESCRIBE ENTITY | x | x | x | +| CREATE ENTITY | x | | | +| ALTER ENTITY | x | | | +| DROP ENTITY | x | | | +| SHOW ASSOCIATIONS | x | x | x | +| DESCRIBE ASSOCIATION | x | x | | +| CREATE ASSOCIATION | x | | | +| ALTER ASSOCIATION | x | | | +| DROP ASSOCIATION | x | | | +| SHOW CONSTANTS | x | x | x | +| SHOW CONSTANT VALUES | x | x | | +| DESCRIBE CONSTANT | x | x | | +| CREATE CONSTANT | x | | | +| DROP CONSTANT | x | | | + +Read operations tested on all projects. Write operations on copies of one project. + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW ENTITIES | Mock tests | | +| 2. SHOW ENTITY | Mock tests | | +| 3. DESCRIBE ENTITY | Mock tests | | +| 4. CREATE ENTITY | Mock + roundtrip | | +| 5. ALTER ENTITY | Mock tests | Multi-action | +| 6. DROP ENTITY | Mock tests | | +| 7–9. SHOW/DESCRIBE ASSOCIATION | Mock tests | | +| 10. CREATE ASSOCIATION | Mock tests | Cross-module | +| 11. ALTER ASSOCIATION | Mock tests | | +| 12. DROP ASSOCIATION | Mock tests | | +| 13–14. SHOW CONSTANTS | Mock tests | | +| 15. DESCRIBE CONSTANT | Mock tests | | +| 16. CREATE CONSTANT | Mock tests | | +| 17. DROP CONSTANT | Mock tests | | +| 18. Roundtrip | Roundtrip tests | Complex entities | +| 19. Catalog | | All manual | +| 20. Multi-step | | All manual | +| 21. Failure modes | Partial | Edge cases | +| 22. Boundary | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW ENTITIES | List all | | | | | +| 1.2 | SHOW ENTITIES | Filter by module | | | | | +| 1.3 | SHOW ENTITIES | Generalization column | | | | | +| 1.4 | SHOW ENTITIES | Empty module | | | | | +| 1.5 | SHOW ENTITIES | Entity types | | | | | +| 2.1 | SHOW ENTITY | Persistent | | | | | +| 2.2 | SHOW ENTITY | Non-persistent | | | | | +| 2.3 | SHOW ENTITY | Not found | | | | | +| 3.1 | DESCRIBE | Basic | | | | | +| 3.2 | DESCRIBE | All features | | | | | +| 3.3 | DESCRIBE | View entity | | | | | +| 3.4 | DESCRIBE | Not found | | | | | +| 4.1 | CREATE | Minimal persistent | | | | | +| 4.2 | CREATE | All types | | | | | +| 4.3 | CREATE | Generalization | | | | | +| 4.4 | CREATE | Non-persistent | | | | | +| 4.5 | CREATE | With indexes | | | | | +| 4.6 | CREATE | Event handlers | | | | | +| 4.7 | CREATE | System attributes | | | | | +| 4.8 | CREATE | create or modify (new) | | | | | +| 4.9 | CREATE | create or modify (update) | | | | | +| 4.10 | CREATE | Duplicate error | | | | | +| 4.11 | CREATE | View entity | | | | | +| 4.12 | CREATE | create or replace view | | | | | +| 4.13 | CREATE | Position annotation | | | | | +| 5.1 | ALTER | Add attribute | | | | | +| 5.2 | ALTER | Rename attribute | | | | | +| 5.3 | ALTER | Modify type | | | | | +| 5.4 | ALTER | Drop attribute | | | | | +| 5.5 | ALTER | Set documentation | | | | | +| 5.6 | ALTER | Set position | | | | | +| 5.7 | ALTER | Add index | | | | | +| 5.8 | ALTER | Drop index | | | | | +| 5.9 | ALTER | Add event handler | | | | | +| 5.10 | ALTER | Drop event handler | | | | | +| 5.11 | ALTER | Multiple actions | | | | | +| 5.12 | ALTER | Drop pseudo-attribute | | | | | +| 5.13 | ALTER | Make calculated | | | | | +| 6.1 | DROP ENTITY | Existing | | | | | +| 6.2 | DROP ENTITY | Non-existent | | | | | +| 6.3 | DROP ENTITY | View entity | | | | | +| 6.4 | DROP ENTITY | With references | | | | | +| 7.1 | SHOW ASSOC | List all | | | | | +| 7.2 | SHOW ASSOC | Filter module | | | | | +| 7.3 | SHOW ASSOC | Cross-module | | | | | +| 8.1 | SHOW ASSOC | Single basic | | | | | +| 8.2 | SHOW ASSOC | Single cross-module | | | | | +| 8.3 | SHOW ASSOC | Not found | | | | | +| 9.1 | DESCRIBE ASSOC | Reference | | | | | +| 9.2 | DESCRIBE ASSOC | ReferenceSet | | | | | +| 9.3 | DESCRIBE ASSOC | With documentation | | | | | +| 10.1 | CREATE ASSOC | Basic reference | | | | | +| 10.2 | CREATE ASSOC | ReferenceSet | | | | | +| 10.3 | CREATE ASSOC | Self-referencing | | | | | +| 10.4 | CREATE ASSOC | Cross-module | | | | | +| 10.5 | CREATE ASSOC | Explicit storage | | | | | +| 10.6 | CREATE ASSOC | DELETE_CASCADE | | | | | +| 10.7 | CREATE ASSOC | DELETE_IF_NO_REFERENCES | | | | | +| 10.8 | CREATE ASSOC | Duplicate error | | | | | +| 11.1 | ALTER ASSOC | Delete behavior | | | | | +| 11.2 | ALTER ASSOC | Change owner | | | | | +| 11.3 | ALTER ASSOC | Change storage | | | | | +| 11.4 | ALTER ASSOC | Set comment | | | | | +| 12.1 | DROP ASSOC | Existing | | | | | +| 12.2 | DROP ASSOC | Non-existent | | | | | +| 12.3 | DROP ASSOC | Cross-module | | | | | +| 13.1 | SHOW CONST | List all | | | | | +| 13.2 | SHOW CONST | Filter module | | | | | +| 13.3 | SHOW CONST | Empty module | | | | | +| 14.1 | CONST VALUES | All values | | | | | +| 14.2 | CONST VALUES | Filter module | | | | | +| 15.1 | DESCRIBE CONST | String | | | | | +| 15.2 | DESCRIBE CONST | Boolean | | | | | +| 15.3 | DESCRIBE CONST | With folder/comment | | | | | +| 15.4 | DESCRIBE CONST | Exposed | | | | | +| 16.1 | CREATE CONST | String | | | | | +| 16.2 | CREATE CONST | Integer | | | | | +| 16.3 | CREATE CONST | Long | | | | | +| 16.4 | CREATE CONST | Decimal | | | | | +| 16.5 | CREATE CONST | Boolean | | | | | +| 16.6 | CREATE CONST | DateTime | | | | | +| 16.7 | CREATE CONST | Enumeration | | | | | +| 16.8 | CREATE CONST | With folder | | | | | +| 16.9 | CREATE CONST | Exposed | | | | | +| 16.10 | CREATE CONST | create or modify | | | | | +| 16.11 | CREATE CONST | Duplicate error | | | | | +| 16.12 | CREATE CONST | Doc comment | | | | | +| 17.1 | DROP CONST | Existing | | | | | +| 17.2 | DROP CONST | Non-existent | | | | | +| 18.1 | ROUNDTRIP | Entity | | | | | +| 18.2 | ROUNDTRIP | Association | | | | | +| 18.3 | ROUNDTRIP | Constant | | | | | +| 19.1 | CATALOG | Entity in catalog | | | | | +| 19.2 | CATALOG | Association in catalog | | | | | +| 19.3 | CATALOG | Callers | | | | | +| 20.1 | MULTI-STEP | Entity + assoc + access | | | | | +| 20.2 | MULTI-STEP | Constant modify chain | | | | | +| 21.1 | FAILURE | Invalid type | | | | | +| 21.2 | FAILURE | Missing module | | | | | +| 21.3 | FAILURE | Alter non-existent attr | | | | | +| 21.4 | FAILURE | Missing entity in assoc | | | | | +| 21.5 | FAILURE | Invalid OQL | | | | | +| 21.6 | FAILURE | Drop referenced entity | | | | | +| 22.1 | BOUNDARY | Many attributes | | | | | +| 22.2 | BOUNDARY | Long names | | | | | +| 22.3 | BOUNDARY | Special characters | | | | | +| 22.4 | BOUNDARY | Max string length | | | | | +| 22.5 | BOUNDARY | Many associations | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/enumeration-test-cases.md b/docs/15-testing/enumeration-test-cases.md new file mode 100644 index 00000000..8c3cb7c9 --- /dev/null +++ b/docs/15-testing/enumeration-test-cases.md @@ -0,0 +1,462 @@ +# Enumeration Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +--- + +## 1. SHOW ENUMERATIONS + +### 1.1 List all enumerations + +``` +show enumerations; +``` + +**Expected:** Table with columns `Qualified Name | Module | Name | Folder | Values`. Summary `(N enumerations)`. Sorted alphabetically. + +### 1.2 List enumerations in a module + +``` +show enumerations in Administration; +``` + +**Expected:** Only enumerations from `Administration` module. + +> (Note: Administration may have 0 enumerations. Use a module with enumerations like `AgentCommons` for a more illustrative test.) + +### 1.3 Empty module + +``` +show enumerations in NonExistentModule; +``` + +**Expected:** Empty result or `(0 enumerations)`. + +--- + +## 2. DESCRIBE ENUMERATION + +### 2.1 Basic enumeration + +Use any existing enumeration in the project (e.g., `AgentCore.Enum_StatusGeneral`): + +``` +describe enumeration AgentCore.Enum_StatusGeneral; +``` + +**Expected:** `create or modify enumeration ...` block with value names and captions, terminated by `/`. + +### 2.2 Enumeration with documentation + +> **Note:** Create a documented enumeration first (see §3.2), then describe it. No test project has pre-existing documented enumerations. + +**Expected:** Output starts with `/** ... */` documentation block. + +### 2.3 Enumeration with value documentation + +> **Note:** Create a documented enumeration first (see §3.2), then describe it. No test project has pre-existing documented enumerations. + +**Expected:** Per-value doc comments: `/** Order received, awaiting processing */` above value. + +### 2.4 Non-existent enumeration + +``` +describe enumeration Fake.Missing; +``` + +**Expected:** Error — enumeration not found. + +--- + +## 3. CREATE ENUMERATION + +### 3.1 Simple enumeration + +``` +create enumeration MyModule.Color ( + RED 'Red', + GREEN 'Green', + BLUE 'Blue' +); +``` + +**Expected:** `Created enumeration: MyModule.Color`. `show enumerations in MyModule` includes it with `Values: 3`. + +### 3.2 Enumeration with documentation + +``` +/** Priority levels for task ordering */ +create enumeration MyModule.Priority ( + /** Highest urgency */ + CRITICAL 'Critical', + HIGH 'High', + MEDIUM 'Medium', + LOW 'Low' +); +``` + +**Expected:** Created. `describe` shows documentation blocks. + +### 3.3 Single-value enumeration + +``` +create enumeration MyModule.YesNo ( + YES 'Yes' +); +``` + +**Expected:** Created with 1 value. + +### 3.4 Quoted identifier (reserved word) + +``` +create enumeration MyModule.UserType ( + "user" 'Standard User', + ADMIN 'Administrator' +); +``` + +**Expected:** Created. Value name `user` (reserved word) accepted when quoted. + +### 3.5 Reserved word — unquoted + +``` +create enumeration MyModule.BadEnum ( + class 'Class Value' +); +``` + +**Expected:** Error (CE7247) — `enumeration value 'class' is a reserved word`. + +### 3.6 Other reserved words + +Test these unquoted values trigger CE7247: + +> **Note:** Error messages may append a spurious hint: `"hint: ... is defined later in this script"` even for single statements. Ignore this hint. + +| Value | Reserved | +|-------|----------| +| `null` | Java keyword | +| `true` | Java keyword | +| `enum` | Java keyword | +| `context` | Mendix reserved | +| `guid` | Mendix reserved | +| `changeddate` | Mendix reserved | +| `currentuser` | Mendix reserved | + +### 3.7 `create or modify` — new enumeration + +``` +create or modify enumeration MyModule.Fresh ( + A 'Alpha', + B 'Beta' +); +``` + +**Expected:** Creates enumeration (same as without `or modify`). + +### 3.8 `create or modify` — existing enumeration + +``` +create or modify enumeration MyModule.Color ( + RED 'Red', + GREEN 'Green', + BLUE 'Blue', + YELLOW 'Yellow' +); +``` + +**Expected:** Replaces existing enumeration. Output says `Created enumeration: MyModule.Color`. `describe` shows 4 values. + +> **Note:** `create or modify` outputs "Created enumeration:" for both new and update operations. + +### 3.9 Duplicate (without `or modify`) + +``` +create enumeration MyModule.Color ( + X 'X' +); +``` + +**Expected:** Error — `enumeration already exists: MyModule.Color (use create or modify to update)`. + +### 3.10 Module auto-creation + +``` +create enumeration NewModule.Status ( + ACTIVE 'Active' +); +``` + +**Expected:** Both module and enumeration created. + +--- + +## 4. DROP ENUMERATION + +### 4.1 Drop existing enumeration + +``` +drop enumeration MyModule.Color; +``` + +**Expected:** `Dropped enumeration: MyModule.Color`. + +### 4.2 Drop non-existent enumeration + +``` +drop enumeration MyModule.Fake; +``` + +**Expected:** Error — not found. + +### 4.3 Drop by name only (no module qualifier) + +``` +drop enumeration Color; +``` + +**Expected:** Matches by name if unambiguous. Error if multiple enumerations share name. + +#### 4.3.1 Ambiguous unqualified name + +**Setup:** + +``` +create enumeration ModuleA.Status ( + ACTIVE 'Active', + INACTIVE 'Inactive' +); + +create enumeration ModuleB.Status ( + OPEN 'Open', + CLOSED 'Closed' +); +``` + +**Test:** + +``` +drop enumeration Status; +``` + +**Expected:** Error — ambiguous unqualified name when multiple modules have an enumeration with the same name. + +--- + +## 5. ROUNDTRIP (BSON) + +### 5.1 Enumeration roundtrip + +``` +create enumeration RtTest.Status ( + ACTIVE 'Active', + INACTIVE 'Inactive', + ARCHIVED 'Archived' +); +``` + +1. `describe enumeration RtTest.Status` +2. Drop: `drop enumeration RtTest.Status` +3. Execute described MDL (change `create or modify` to `create`) +4. `describe` again + +**Expected:** Output identical between step 1 and step 4. + +### 5.2 Roundtrip with documentation + +``` +/** Payment status tracking */ +create enumeration RtTest.PayStatus ( + /** Awaiting payment */ + PENDING 'Pending', + PAID 'Paid', + REFUNDED 'Refunded' +); +``` + +1. `describe` → drop → recreate from output → `describe` + +**Expected:** Documentation preserved through roundtrip. + +--- + +## 6. MULTI-STEP WORKFLOWS + +### 6.1 Create enumeration → use in entity + +``` +create enumeration MyModule.TaskStatus ( + TODO 'To Do', + IN_PROGRESS 'In Progress', + DONE 'Done' +); + +create persistent entity MyModule.Task ( + Title: string(200) not null, + Status: enumeration(MyModule.TaskStatus) default MyModule.TaskStatus.TODO +); +``` + +**Expected:** Both created. Entity attribute references enumeration. `describe entity` shows `enumeration(MyModule.TaskStatus)`. + +### 6.2 Replace enumeration → verify entity still works + +``` +create or modify enumeration MyModule.TaskStatus ( + TODO 'To Do', + IN_PROGRESS 'In Progress', + DONE 'Done', + BLOCKED 'Blocked' +); + +describe entity MyModule.Task; +``` + +**Expected:** Entity still references `MyModule.TaskStatus`. New value `BLOCKED` available. + +--- + +## 7. FAILURE MODES & ERROR RECOVERY + +### 7.1 Empty value list + +``` +create enumeration MyModule.Empty (); +``` + +**Expected:** Error — at least one value required (or parser error). + +### 7.2 Duplicate value names + +``` +create enumeration MyModule.Dup ( + A 'First', + A 'Second' +); +``` + +**Expected:** Error — duplicate value name. + +### 7.3 Drop enumeration used by entity attribute + +``` +drop enumeration MyModule.TaskStatus; +``` + +**Expected:** Enumeration dropped (no referential integrity enforcement — may leave entity attribute orphaned). Verify `describe entity MyModule.Task` still works but shows broken reference or generic type. + +### 7.4 Case sensitivity in reserved word check + +``` +create enumeration MyModule.CaseTest ( + NULL 'Null Value', + True 'True Value' +); +``` + +**Expected:** Error — reserved word check is case-insensitive. + +--- + +## 8. BOUNDARY & STRESS + +### 8.1 Many values (50+) + +Create an enumeration with 50 values. Verify `show` and `describe` handle it. + +### 8.2 Long captions + +``` +create enumeration MyModule.LongCaptions ( + VAL1 'This is a very long caption that exceeds what would normally be displayed in a UI dropdown' +); +``` + +**Expected:** Created and described correctly. + +### 8.3 Special characters in captions + +``` +create enumeration MyModule.Special ( + A 'It''s a "test" value', + B 'Line1\nLine2' +); +``` + +**Expected:** Escaped characters preserved in describe output. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Product | +|-----------|:---:|:---:|:---:| +| SHOW ENUMERATIONS | x | x | x | +| DESCRIBE ENUMERATION | x | x | x | +| CREATE ENUMERATION | x | | | +| DROP ENUMERATION | x | | | + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW ENUMERATIONS | Mock tests | | +| 2. DESCRIBE ENUMERATION | Mock tests | | +| 3. CREATE ENUMERATION | Mock tests | Reserved words, quoted identifiers | +| 4. DROP ENUMERATION | Mock tests | | +| 5. Roundtrip | | All manual | +| 6. Multi-step | | All manual | +| 7. Failure modes | Partial (reserved) | Empty list, duplicates | +| 8. Boundary | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW | List all | | | | | +| 1.2 | SHOW | Filter module | | | | | +| 1.3 | SHOW | Empty module | | | | | +| 2.1 | DESCRIBE | Basic | | | | | +| 2.2 | DESCRIBE | With documentation | | | | | +| 2.3 | DESCRIBE | Value documentation | | | | | +| 2.4 | DESCRIBE | Not found | | | | | +| 3.1 | CREATE | Simple | | | | | +| 3.2 | CREATE | With documentation | | | | | +| 3.3 | CREATE | Single value | | | | | +| 3.4 | CREATE | Quoted reserved word | | | | | +| 3.5 | CREATE | Unquoted reserved word | | | | | +| 3.6 | CREATE | Other reserved words | | | | | +| 3.7 | CREATE | create or modify (new) | | | | | +| 3.8 | CREATE | create or modify (update) | | | | | +| 3.9 | CREATE | Duplicate error | | | | | +| 3.10 | CREATE | Module auto-creation | | | | | +| 4.1 | DROP | Existing | | | | | +| 4.2 | DROP | Non-existent | | | | | +| 4.3 | DROP | By name only | | | | | +| 5.1 | ROUNDTRIP | Basic | | | | | +| 5.2 | ROUNDTRIP | With documentation | | | | | +| 6.1 | MULTI-STEP | Create + use in entity | | | | | +| 6.2 | MULTI-STEP | Replace + verify entity | | | | | +| 7.1 | FAILURE | Empty value list | | | | | +| 7.2 | FAILURE | Duplicate values | | | | | +| 7.3 | FAILURE | Drop used enumeration | | | | | +| 7.4 | FAILURE | Case-insensitive reserved | | | | | +| 8.1 | BOUNDARY | Many values | | | | | +| 8.2 | BOUNDARY | Long captions | | | | | +| 8.3 | BOUNDARY | Special characters | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/image-collection-test-cases.md b/docs/15-testing/image-collection-test-cases.md new file mode 100644 index 00000000..89a6cac6 --- /dev/null +++ b/docs/15-testing/image-collection-test-cases.md @@ -0,0 +1,429 @@ +# Image Collection Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +--- + +## 1. SHOW IMAGE COLLECTION + +### 1.1 List all image collections + +``` +show image collection; +``` + +**Expected:** Table with columns `Image Collection | Export Level | Images`. Summary `(N image collections)`. + +### 1.2 Filter by module + +``` +show image collection in Administration; +``` + +**Expected:** Only image collections from `Administration` module. + +### 1.3 Empty module + +``` +show image collection in NonExistentModule; +``` + +**Expected:** Empty result or `(0 image collections)`. + +--- + +## 2. DESCRIBE IMAGE COLLECTION + +### 2.1 Basic describe + +``` +describe image collection MyModule.Icons; +``` + +**Expected:** +``` +create or replace image collection MyModule.Icons export level 'Public' ( + image logo from file 'logo.png', + image icon from file 'icon.svg' +); +/ +``` + +### 2.2 Image extraction to disk + +``` +describe image collection MyModule.Icons; +``` + +**Expected:** Images extracted to `/tmp/mxcli-preview/MyModule.Icons/`. Each image file present with original format. + +### 2.3 Supported formats + +Verify `describe` correctly identifies format for each type: + +| Extension | Format | +|-----------|--------| +| `.png` | Png (default) | +| `.svg` | Svg | +| `.gif` | Gif | +| `.jpg` | Jpg | +| `.bmp` | Bmp | +| `.webp` | Webp | + +### 2.4 Non-existent image collection + +``` +describe image collection Fake.Missing; +``` + +**Expected:** Error — image collection not found. + +--- + +## 3. CREATE IMAGE COLLECTION + +### 3.1 Empty collection + +``` +create image collection MyModule.EmptyIcons; +``` + +**Expected:** `Created image collection: MyModule.EmptyIcons`. `show image collections in MyModule` lists it with `Images: 0`. + +### 3.2 Collection with images + +``` +create image collection MyModule.AppIcons export level 'Public' comment 'Application icons' ( + image logo from file '/tmp/mxcli-test-images/logo.png', + image icon from file '/tmp/mxcli-test-images/vector.svg' +); +``` + +**Expected:** Created with 2 images. `describe` shows both images with correct formats. + +### 3.3 Hidden export level (default) + +``` +create image collection MyModule.HiddenIcons ( + image logo from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Created. `show image collections` shows `Export Level: Hidden`. + +### 3.4 Public export level + +``` +create image collection MyModule.PublicIcons export level 'Public' ( + image logo from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Created. `show image collections` shows `Export Level: Public`. + +### 3.5 Format auto-detection from extension + +Create collections with each supported format. Verify `describe` output shows correct format. + +| File | Detected Format | +|------|-----------------| +| `image.png` | Png | +| `image.svg` | Svg | +| `image.gif` | Gif | +| `image.jpg` | Jpg | +| `image.bmp` | Bmp | +| `image.webp` | Webp | + +### 3.6 `create or replace` — new collection + +``` +create or replace image collection MyModule.Fresh ( + image a from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Creates collection (same as without `or replace`). + +### 3.7 `create or replace` — existing collection + +``` +create or replace image collection MyModule.Fresh ( + image a from file '/tmp/mxcli-test-images/logo.png', + image b from file '/tmp/mxcli-test-images/icon.png' +); +``` + +**Expected:** Replaces existing collection. `describe` shows 2 images. + +### 3.8 Duplicate (without `or replace`) + +``` +create image collection MyModule.Fresh ( + image x from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Error — image collection already exists. + +### 3.9 Module auto-creation + +``` +create image collection NewModule.Icons ( + image logo from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Both module and image collection created. + +--- + +## 4. DROP IMAGE COLLECTION + +### 4.1 Drop existing collection + +``` +drop image collection MyModule.AppIcons; +``` + +**Expected:** `Dropped image collection: MyModule.AppIcons`. + +### 4.2 Drop non-existent collection + +``` +drop image collection MyModule.Fake; +``` + +**Expected:** Error — not found. + +--- + +## 5. ROUNDTRIP + +### 5.1 Create → Describe → Recreate + +1. Create collection with images (3.2) +2. `describe image collection MyModule.AppIcons` — save output +3. `drop image collection MyModule.AppIcons` +4. Execute saved describe output (change `create or replace` to `create`) +5. `describe` again + +**Expected:** Output identical between step 2 and step 5. Extracted image files match. + +--- + +## 6. FAILURE MODES + +### 6.1 Not connected + +Run any command without `-p` or `connect`. + +**Expected:** Error — not connected to a project. + +### 6.2 File not found + +``` +create image collection MyModule.Bad ( + image missing from file '/tmp/does-not-exist.png' +); +``` + +**Expected:** Error — file not found. + +### 6.3 Invalid image format + +``` +create image collection MyModule.Bad ( + image broken from file '/tmp/mxcli-test-images/notanimage.txt' +); +``` + +**Expected:** Error — unsupported image format. + +### 6.4 Backend create failure + +Simulate backend failure (e.g., corrupt MPR, disk full). + +**Expected:** Error from backend layer with descriptive message. + +### 6.5 Preview directory creation failure + +If `/tmp/mxcli-preview/` is not writable: + +``` +describe image collection MyModule.Icons; +``` + +**Expected:** Error creating preview directory. + +--- + +## 7. BOUNDARY & STRESS + +### 7.1 Large image file + +Create a collection with a large image (10 MB+ PNG). + +``` +create image collection MyModule.LargeIcons ( + image big from file '/tmp/mxcli-test-images/large-10mb.png' +); +``` + +**Expected:** Collection created. `describe` extracts image correctly. Verify file size preserved. + +### 7.2 Many images in one collection + +Create a collection with 50+ images. + +``` +create image collection MyModule.ManyIcons ( + image img001 from file '/tmp/mxcli-test-images/logo.png', + image img002 from file '/tmp/mxcli-test-images/icon.png', + ... + image img050 from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** All 50 images stored. `show image collections` reports correct count. `describe` extracts all files. + +### 7.3 Image with special characters in name + +``` +create image collection MyModule.SpecialIcons ( + image my_icon_v2 from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Image name with underscores accepted. Verify no name sanitization issues. + +### 7.4 Empty image file + +Create a 0-byte file and attempt to import: + +```bash +touch /tmp/mxcli-test-images/empty.png +``` + +``` +create image collection MyModule.EmptyFile ( + image empty from file '/tmp/mxcli-test-images/empty.png' +); +``` + +**Expected:** Error or warning about empty/invalid image data. + +### 7.5 Duplicate image names in one collection + +``` +create image collection MyModule.DupeIcons ( + image logo from file '/tmp/mxcli-test-images/logo.png', + image logo from file '/tmp/mxcli-test-images/icon.png' +); +``` + +**Expected:** Error — duplicate image name within collection. + +### 7.6 All supported formats in one collection + +``` +create image collection MyModule.AllFormats export level 'Public' ( + image png_img from file '/tmp/mxcli-test-images/test.png', + image svg_img from file '/tmp/mxcli-test-images/test.svg', + image gif_img from file '/tmp/mxcli-test-images/test.gif', + image jpg_img from file '/tmp/mxcli-test-images/test.jpg', + image bmp_img from file '/tmp/mxcli-test-images/test.bmp', + image webp_img from file '/tmp/mxcli-test-images/test.webp' +); +``` + +**Expected:** All 6 formats accepted. `describe` shows correct format for each. + +### 7.7 Replace collection with different image set + +``` +create or replace image collection MyModule.Evolving ( + image v1 from file '/tmp/mxcli-test-images/logo.png' +); +create or replace image collection MyModule.Evolving ( + image v2a from file '/tmp/mxcli-test-images/icon.png', + image v2b from file '/tmp/mxcli-test-images/logo.png' +); +``` + +**Expected:** Second `create or replace` fully replaces. `describe` shows only `v2a` and `v2b`. + +--- + +## Test Project Coverage Matrix + +| Operation | Lato Enquiry | Evora Factory | Lato Inventory | +|-----------|:---:|:---:|:---:| +| SHOW IMAGE COLLECTION | x | x | x | +| DESCRIBE IMAGE COLLECTION | x | x | x | +| CREATE IMAGE COLLECTION | x | | | +| DROP IMAGE COLLECTION | x | | | +| ROUNDTRIP | x | | | +| BOUNDARY & STRESS | x | | | + +--- + +## Automated Test Coverage + +| Section | Automated | Manual-only | +|---------|:---------:|:-----------:| +| 1. SHOW | Mock tests | | +| 2. DESCRIBE | Mock tests | Image extraction to disk | +| 3. CREATE | Mock tests | Format auto-detection | +| 4. DROP | Mock tests | | +| 5. Roundtrip | | All manual | +| 6. Failure modes | Partial | Backend, preview dir | +| 7. Boundary & stress | | All manual | + +--- + +## Manual Test Report Template + +**Tester:** _______________ +**Date:** _______________ +**Project:** _______________ + +| # | Section | Test | Pass | Fail | Skip | Notes | +|---|---------|------|:----:|:----:|:----:|-------| +| 1.1 | SHOW | List all | | | | | +| 1.2 | SHOW | Filter module | | | | | +| 1.3 | SHOW | Empty module | | | | | +| 2.1 | DESCRIBE | Basic | | | | | +| 2.2 | DESCRIBE | Image extraction | | | | | +| 2.3 | DESCRIBE | Supported formats | | | | | +| 2.4 | DESCRIBE | Not found | | | | | +| 3.1 | CREATE | Empty collection | | | | | +| 3.2 | CREATE | With images | | | | | +| 3.3 | CREATE | Hidden (default) | | | | | +| 3.4 | CREATE | Public export level | | | | | +| 3.5 | CREATE | Format auto-detection | | | | | +| 3.6 | CREATE | create or replace (new) | | | | | +| 3.7 | CREATE | create or replace (update) | | | | | +| 3.8 | CREATE | Duplicate error | | | | | +| 3.9 | CREATE | Module auto-creation | | | | | +| 4.1 | DROP | Existing | | | | | +| 4.2 | DROP | Non-existent | | | | | +| 5.1 | ROUNDTRIP | Create → Describe → Recreate | | | | | +| 6.1 | FAILURE | Not connected | | | | | +| 6.2 | FAILURE | File not found | | | | | +| 6.3 | FAILURE | Invalid format | | | | | +| 6.4 | FAILURE | Backend create failure | | | | | +| 6.5 | FAILURE | Preview dir failure | | | | | +| 7.1 | BOUNDARY | Large image (10 MB+) | | | | | +| 7.2 | BOUNDARY | Many images (50+) | | | | | +| 7.3 | BOUNDARY | Special chars in name | | | | | +| 7.4 | BOUNDARY | Empty image file | | | | | +| 7.5 | BOUNDARY | Duplicate image names | | | | | +| 7.6 | BOUNDARY | All formats in one | | | | | +| 7.7 | BOUNDARY | Replace with different set | | | | | + +**Summary:** ___ / ___ passed | ___ failed | ___ skipped diff --git a/docs/15-testing/integration-test-cases.md b/docs/15-testing/integration-test-cases.md new file mode 100644 index 00000000..8a754843 --- /dev/null +++ b/docs/15-testing/integration-test-cases.md @@ -0,0 +1,1799 @@ +# Integration Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +Covers: OData Clients, OData Services, External Entities, Published REST Services, REST Clients, JSON Structures. + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +--- + +## 1. SHOW ODATA CLIENTS + +### 1.1 List all OData clients + +``` +show odata clients; +``` + +**Expected:** All consumed OData services listed. Count matches Studio Pro. + +### 1.2 Filter by module + +``` +show odata clients in MyModule; +``` + +**Expected:** Only OData clients from `MyModule`. + +### 1.3 Empty module + +``` +show odata clients in ModuleWithNoClients; +``` + +**Expected:** Empty result, no error. + +### 1.4 Non-existent module + +``` +show odata clients in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 2. DESCRIBE ODATA CLIENT + +### 2.1 Describe existing client + +``` +describe odata client MyModule.MyODataClient; +``` + +**Expected:** Valid `create or modify odata client` MDL output. Includes Version, ODataVersion, MetadataUrl, Timeout, ProxyType, ServiceUrl, Authentication, Headers. + +### 2.2 Non-existent client + +``` +describe odata client MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 2.3 All properties present + +Pick a client with all properties configured. Verify DESCRIBE output includes: + +| Property | Format | +|----------|--------| +| Version | String literal | +| ODataVersion | `3` or `4` | +| MetadataUrl | URL string | +| Timeout | Integer (seconds) | +| ProxyType | `default` / `none` / `custom` | +| ServiceUrl | `@ConstantName` reference | +| Authentication | `none` / `basic` / `custom` | +| Headers | Block with key-value pairs | + +--- + +## 3. CREATE ODATA CLIENT + +### 3.1 Full property set + +``` +create odata client MyModule.NewClient ( + Version: '1.0', + ODataVersion: 4, + MetadataUrl: 'https://api.example.com/odata/v4/$metadata', + Timeout: 300, + ProxyType: default, + ServiceUrl: @MyModule.ServiceUrlConstant, + Authentication: none, + Headers: + 'X-Custom-Header' = 'value1', + 'Accept' = 'application/json' + end headers +); +``` + +**Expected:** Created. Listed in `show odata clients`. DESCRIBE matches input. + +### 3.2 Minimal client + +``` +create odata client MyModule.MinimalClient ( + MetadataUrl: 'https://api.example.com/odata/$metadata' +); +``` + +**Expected:** Created with defaults for omitted properties. + +### 3.3 With basic authentication + +``` +create odata client MyModule.AuthClient ( + MetadataUrl: 'https://api.example.com/odata/$metadata', + Authentication: basic +); +``` + +**Expected:** OData client created (may warn about unreachable metadata URL). Note: `Authentication: basic` is accepted by the parser but not shown in DESCRIBE output — only Timeout appears. + +### 3.4 Duplicate without OR MODIFY + +``` +create odata client MyModule.NewClient ( + MetadataUrl: 'https://api.example.com/$metadata' +); +create odata client MyModule.NewClient ( + MetadataUrl: 'https://api.example.com/$metadata' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 3.5 Target module must exist + +``` +create odata client NewModule.TestClient ( + MetadataUrl: 'https://api.example.com/$metadata' +); +``` + +**Expected:** Error if `NewModule` does not exist. Modules must be created beforehand. + +### 3.6 Write guard + +Attempt CREATE without opening a project for writing. + +**Expected:** Error about not being connected. + +--- + +## 4. CREATE OR MODIFY ODATA CLIENT + +### 4.1 Upsert — new client + +``` +create or modify odata client MyModule.UpsertClient ( + MetadataUrl: 'https://api.example.com/$metadata', + Timeout: 60 +); +``` + +**Expected:** Created. No error. + +### 4.2 Upsert — existing client + +``` +create or modify odata client MyModule.UpsertClient ( + MetadataUrl: 'https://api.example.com/v2/$metadata', + Timeout: 120 +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new values. + +--- + +## 5. ALTER ODATA CLIENT + +### 5.1 Set single property + +``` +alter odata client MyModule.ExistingClient set timeout = 600; +``` + +**Expected:** Only timeout changed. Other properties unchanged. + +### 5.2 Set metadata URL + +``` +alter odata client MyModule.ExistingClient set MetadataUrl = 'https://new.example.com/$metadata'; +``` + +**Expected:** URL updated. DESCRIBE confirms. + +### 5.3 Alter non-existent client + +``` +alter odata client MyModule.DoesNotExist set timeout = 60; +``` + +**Expected:** Clear error. + +--- + +## 6. DROP ODATA CLIENT + +### 6.1 Drop existing client + +``` +create odata client MyModule.ToDrop ( + MetadataUrl: 'https://example.com/$metadata' +); +drop odata client MyModule.ToDrop; +``` + +**Expected:** Removed. Not in `show odata clients`. + +### 6.2 Cascade — deletes external entities + +1. CREATE odata client +2. CREATE external entity from that client +3. DROP odata client + +**Expected:** External entities sourced from this client also removed. + +### 6.3 Drop non-existent client + +``` +drop odata client MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 6.4 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 7. SHOW ODATA SERVICES + +### 7.1 List all published OData services + +``` +show odata services; +``` + +**Expected:** All published OData services listed. Count matches Studio Pro. + +### 7.2 Filter by module + +``` +show odata services in MyModule; +``` + +**Expected:** Only OData services from `MyModule`. + +### 7.3 Empty module + +``` +show odata services in ModuleWithNoServices; +``` + +**Expected:** Empty result, no error. + +### 7.4 Non-existent module + +``` +show odata services in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 8. DESCRIBE ODATA SERVICE + +### 8.1 Describe existing service + +``` +describe odata service MyModule.MyODataService; +``` + +**Expected:** Valid `create or modify odata service` MDL output. Includes Path, Version, ODataVersion, Namespace, Authentication, published entities with expose blocks. + +### 8.2 Non-existent service + +``` +describe odata service MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 8.3 Published entity details + +Verify DESCRIBE output for a service with published entities includes: + +| Property | Format | +|----------|--------| +| PUBLISH ENTITY | Qualified entity name | +| EXPOSE block | Attribute list with types | +| ReadMode | `default` / `fromDatabase` / `fromMicroflow` | +| InsertMode | `default` / `fromMicroflow` | +| UpdateMode | `default` / `fromMicroflow` | +| DeleteMode | `default` / `fromMicroflow` | +| UsePaging | `true` / `false` | +| PageSize | Integer | + +--- + +## 9. CREATE ODATA SERVICE + +### 9.1 Full service with published entity + +``` +create odata service MyModule.NewService ( + Path: '/odata/v1/myservice', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.myservice', + Authentication: none +) + publish entity MyModule.Customer + expose + Name, + Email, + PhoneNumber + end expose + ReadMode: default, + InsertMode: default, + UpdateMode: default, + DeleteMode: default, + UsePaging: true, + PageSize: 20; +``` + +**Expected:** Created. Listed in `show odata services`. DESCRIBE matches input. + +### 9.2 Multiple published entities + +``` +create odata service MyModule.MultiEntityService ( + Path: '/odata/v1/multi', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.multi', + Authentication: none +) + publish entity MyModule.Customer + expose Name, Email end expose + ReadMode: default, + UsePaging: true, + PageSize: 50 + publish entity MyModule.Order + expose OrderNumber, TotalAmount end expose + ReadMode: default, + UsePaging: true, + PageSize: 100; +``` + +**Expected:** Both entities published. DESCRIBE lists both. + +### 9.3 Duplicate without OR MODIFY + +``` +create odata service MyModule.NewService ( + Path: '/odata/v1/svc', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example' +); +create odata service MyModule.NewService ( + Path: '/odata/v1/svc', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 9.4 Target module must exist + +``` +create odata service NewModule.TestService ( + Path: '/odata/v1/test', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.test' +); +``` + +**Expected:** Error if `NewModule` does not exist. Modules must be created beforehand. + +### 9.5 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 10. CREATE OR MODIFY ODATA SERVICE + +### 10.1 Upsert — new service + +``` +create or modify odata service MyModule.UpsertService ( + Path: '/odata/v1/upsert', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.upsert' +); +``` + +**Expected:** Created. No error. + +### 10.2 Upsert — existing service + +``` +create or modify odata service MyModule.UpsertService ( + Path: '/odata/v2/upsert', + Version: '2.0.0', + ODataVersion: 4, + Namespace: 'com.example.upsert.v2' +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new values. + +--- + +## 11. ALTER ODATA SERVICE + +### 11.1 Set single property + +``` +alter odata service MyModule.ExistingService set version = '2.0.0'; +``` + +**Expected:** Only version changed. Other properties unchanged. + +### 11.2 Set path + +``` +alter odata service MyModule.ExistingService set path = '/odata/v2/updated'; +``` + +**Expected:** Path updated. DESCRIBE confirms. + +### 11.3 Alter non-existent service + +``` +alter odata service MyModule.DoesNotExist set version = '1.0.0'; +``` + +**Expected:** Clear error. + +--- + +## 12. DROP ODATA SERVICE + +### 12.1 Drop existing service + +``` +create odata service MyModule.ToDrop ( + Path: '/odata/v1/drop', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example.drop' +); +drop odata service MyModule.ToDrop; +``` + +**Expected:** Removed. Not in `show odata services`. + +### 12.2 Drop non-existent service + +``` +drop odata service MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 12.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 13. GRANT / REVOKE ACCESS ON ODATA SERVICE + +### 13.1 Grant to single role + +``` +grant access on odata service MyModule.MyService to MyModule.User; +``` + +**Expected:** Role granted. Verifiable via DESCRIBE or show access. + +### 13.2 Grant to multiple roles + +``` +grant access on odata service MyModule.MyService to MyModule.User, MyModule.Admin; +``` + +**Expected:** Both roles added. + +### 13.3 Idempotent grant + +Grant same role twice. + +**Expected:** No duplicate entries on second grant. + +### 13.4 Revoke from role + +``` +revoke access on odata service MyModule.MyService from MyModule.User; +``` + +**Expected:** Role removed. + +### 13.5 Grant on non-existent service + +**Expected:** Clear error. + +### 13.6 Grant with non-existent role + +**Expected:** Clear error. + +--- + +## 14. SHOW EXTERNAL ENTITIES + +### 14.1 List all external entities + +``` +show external entities; +``` + +**Expected:** All external entities listed. Count matches Studio Pro. + +### 14.2 Filter by module + +``` +show external entities in MyModule; +``` + +**Expected:** Only external entities from `MyModule`. + +### 14.3 Empty module + +``` +show external entities in ModuleWithNoExternals; +``` + +**Expected:** Empty result, no error. + +### 14.4 Non-existent module + +``` +show external entities in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 15. SHOW EXTERNAL ACTIONS + +### 15.1 List all external actions + +``` +show external actions; +``` + +**Expected:** All external actions listed. Count matches Studio Pro. + +--- + +## 16. DESCRIBE EXTERNAL ENTITY + +### 16.1 Describe existing external entity + +``` +describe external entity MyModule.MyExternalEntity; +``` + +**Expected:** Valid MDL output. Includes source OData client, EntitySet, RemoteName, attributes, Countable, Creatable, Deletable, Updatable flags. + +### 16.2 Non-existent external entity + +``` +describe external entity MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 16.3 All properties present + +Verify DESCRIBE output includes: + +| Property | Format | +|----------|--------| +| EntitySet | String | +| RemoteName | String | +| Countable | `true` / `false` | +| Creatable | `true` / `false` | +| Deletable | `true` / `false` | +| Updatable | `true` / `false` | +| Attributes | Name + type list | + +--- + +## 17. CREATE EXTERNAL ENTITY FROM ODATA CLIENT + +### 17.1 Single entity with full properties + +``` +create external entity MyModule.ExternalCustomer from odata client MyModule.MyODataClient ( + EntitySet: 'Customers', + RemoteName: 'Customer', + Countable: true, + Creatable: true, + Deletable: false, + Updatable: true +) ( + Name: String(200), + Email: String(200), + Age: Integer, + Active: Boolean +); +``` + +**Expected:** Created. Listed in `show external entities`. DESCRIBE matches input. + +### 17.2 Minimal entity + +``` +create external entity MyModule.ExternalOrder from odata client MyModule.MyODataClient ( + EntitySet: 'Orders', + RemoteName: 'Order' +) ( + OrderId: Integer, + Total: Decimal +); +``` + +**Expected:** Created with defaults for omitted boolean flags. + +### 17.3 Non-existent source client + +``` +create external entity MyModule.BadSource from odata client MyModule.NonExistentClient ( + EntitySet: 'Items', + RemoteName: 'Item' +) ( + Id: Integer +); +``` + +**Expected:** Error — source OData client not found. + +### 17.4 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 18. CREATE OR MODIFY EXTERNAL ENTITY + +### 18.1 Upsert — new entity + +``` +create or modify external entity MyModule.UpsertExternal from odata client MyModule.MyODataClient ( + EntitySet: 'Products', + RemoteName: 'Product' +) ( + ProductId: Integer, + Name: String(200) +); +``` + +**Expected:** Created. No error. + +### 18.2 Upsert — existing entity + +``` +create or modify external entity MyModule.UpsertExternal from odata client MyModule.MyODataClient ( + EntitySet: 'Products', + RemoteName: 'Product', + Countable: true +) ( + ProductId: Integer, + Name: String(200), + Price: Decimal +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new attributes. + +--- + +## 19. CREATE EXTERNAL ENTITIES FROM CLIENT (bulk import) + +### 19.1 Bulk import all entities + +``` +create external entities from MyModule.MyODataClient; +``` + +**Expected:** All entities from the OData service metadata imported. Each listed in `show external entities`. + +### 19.2 Selective import into target module + +``` +create external entities from MyModule.MyODataClient + into TargetModule + entities (Customer, Order, Product); +``` + +**Expected:** Only Customer, Order, Product imported. Placed in `TargetModule`. + +### 19.3 Non-existent entity name in list + +``` +create external entities from MyModule.MyODataClient + entities (Customer, NonExistentEntity); +``` + +**Expected:** Error for NonExistentEntity. Customer may or may not be created depending on fail-fast behavior. + +### 19.4 Non-existent source client + +``` +create external entities from MyModule.NonExistent; +``` + +**Expected:** Clear error. + +--- + +## 20. SHOW PUBLISHED REST SERVICES + +> **Feature gate:** Mendix 10.0+. Test against Evora (10.24.15) and Lato Enquiry (11.4.0). + +### 20.1 List all published REST services + +``` +show published rest services; +``` + +**Expected:** All published REST services listed. Count matches Studio Pro. + +### 20.2 Filter by module + +``` +show published rest services in MyModule; +``` + +**Expected:** Only REST services from `MyModule`. + +### 20.3 Empty module + +``` +show published rest services in ModuleWithNoREST; +``` + +**Expected:** Empty result, no error. + +### 20.4 Non-existent module + +``` +show published rest services in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 21. DESCRIBE PUBLISHED REST SERVICE + +### 21.1 Describe existing service + +``` +describe published rest service MyModule.MyRestService; +``` + +**Expected:** Valid `create or replace published rest service` MDL output. Includes Path, Version, ServiceName, resource blocks with HTTP methods and microflow references. + +### 21.2 Non-existent service + +``` +describe published rest service MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 21.3 Resource detail + +Verify DESCRIBE includes for each resource: + +| Property | Format | +|----------|--------| +| Resource path | String | +| HTTP method | `GET` / `POST` / `PUT` / `PATCH` / `DELETE` | +| Microflow | Qualified `Module.Microflow` reference | +| Deprecated | `deprecated` keyword if applicable | + +--- + +## 22. CREATE PUBLISHED REST SERVICE + +### 22.1 Full service with resources + +``` +create published rest service MyModule.CustomerAPI ( + Path: '/api/v1/customers', + Version: '1.0.0', + ServiceName: 'CustomerAPI' +) + resource '/customers' { + get microflow MyModule.GetCustomers; + post microflow MyModule.CreateCustomer; + } + resource '/customers/{id}' { + get microflow MyModule.GetCustomerById; + put microflow MyModule.UpdateCustomer; + delete microflow MyModule.DeleteCustomer; + }; +``` + +**Expected:** Created. Listed in `show published rest services`. DESCRIBE matches input. + +> **Note (#429):** The `end resource` syntax causes a crash. Always use brace syntax `resource 'path' { ... }` instead. + +### 22.2 Single resource, single method + +``` +create published rest service MyModule.SimpleAPI ( + Path: '/api/v1/health', + Version: '1.0.0', + ServiceName: 'HealthCheck' +) + resource '/health' { + get microflow MyModule.HealthCheck; + }; +``` + +**Expected:** Created with minimal configuration. + +### 22.3 Duplicate without OR REPLACE + +``` +create published rest service MyModule.CustomerAPI ( + Path: '/api/v1/dup', + Version: '1.0.0', + ServiceName: 'Dup' +); +create published rest service MyModule.CustomerAPI ( + Path: '/api/v1/dup', + Version: '1.0.0', + ServiceName: 'Dup' +); +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 22.4 Module auto-creation + +``` +create published rest service NewModule.TestAPI ( + Path: '/api/v1/test', + Version: '1.0.0', + ServiceName: 'TestAPI' +); +``` + +**Expected:** `NewModule` created automatically if it doesn't exist. + +### 22.5 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 23. CREATE OR REPLACE PUBLISHED REST SERVICE + +### 23.1 Upsert — new service + +``` +create or replace published rest service MyModule.UpsertAPI ( + Path: '/api/v1/upsert', + Version: '1.0.0', + ServiceName: 'UpsertAPI' +) + resource '/items' { + get microflow MyModule.GetItems; + }; +``` + +**Expected:** Created. No error. + +### 23.2 Upsert — existing service + +``` +create or replace published rest service MyModule.UpsertAPI ( + Path: '/api/v2/upsert', + Version: '2.0.0', + ServiceName: 'UpsertAPI' +) + resource '/items' { + get microflow MyModule.GetItemsV2; + post microflow MyModule.CreateItem; + }; +``` + +**Expected:** Replaced. ID reused. DESCRIBE shows new values. + +--- + +## 24. ALTER PUBLISHED REST SERVICE + +### 24.1 Set property + +``` +alter published rest service MyModule.MyAPI set version = '2.0.0'; +``` + +**Expected:** Only version changed. Resources unchanged. + +### 24.2 Add resource + +``` +alter published rest service MyModule.MyAPI + add resource '/new-resource' { + get microflow MyModule.GetNewResource; + }; +``` + +**Expected:** New resource added. Existing resources unchanged. + +### 24.3 Drop resource + +``` +alter published rest service MyModule.MyAPI + drop resource '/old-resource'; +``` + +**Expected:** Resource removed. Other resources unchanged. + +### 24.4 Alter non-existent service + +``` +alter published rest service MyModule.DoesNotExist set version = '1.0.0'; +``` + +**Expected:** Clear error. + +--- + +## 25. DROP PUBLISHED REST SERVICE + +### 25.1 Drop existing service + +``` +drop published rest service MyModule.CustomerAPI; +``` + +**Expected:** Removed. Not in `show published rest services`. + +### 25.2 Drop non-existent service + +``` +drop published rest service MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 25.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 26. GRANT / REVOKE ACCESS ON PUBLISHED REST SERVICE + +### 26.1 Grant to single role + +``` +grant access on published rest service MyModule.MyAPI to MyModule.User; +``` + +**Expected:** Role granted. + +### 26.2 Grant to multiple roles + +``` +grant access on published rest service MyModule.MyAPI to MyModule.User, MyModule.Admin; +``` + +**Expected:** Both roles added. + +### 26.3 Revoke from role + +``` +revoke access on published rest service MyModule.MyAPI from MyModule.User; +``` + +**Expected:** Role removed. + +### 26.4 Grant on non-existent service + +**Expected:** Clear error. + +### 26.5 Grant with non-existent role + +**Expected:** Clear error. + +--- + +## 27. SHOW REST CLIENTS + +> **Feature gate:** Mendix 10.1+. Test against Lato Enquiry (11.4.0) and Evora (10.24.15). + +### 27.1 List all REST clients + +``` +show rest clients; +``` + +**Expected:** All consumed REST services listed. Count matches Studio Pro. + +### 27.2 Filter by module + +``` +show rest clients in MyModule; +``` + +**Expected:** Only REST clients from `MyModule`. + +### 27.3 Empty module + +``` +show rest clients in ModuleWithNoREST; +``` + +**Expected:** Empty result, no error. + +### 27.4 Non-existent module + +``` +show rest clients in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 28. DESCRIBE REST CLIENT + +### 28.1 Describe existing client + +``` +describe rest client MyModule.MyRestClient; +``` + +**Expected:** Valid `create or modify rest client` MDL output. Includes BaseUrl, Authentication, operation blocks. + +### 28.2 Non-existent client + +``` +describe rest client MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +### 28.3 Operation detail + +Verify DESCRIBE output for each operation includes: + +| Property | Format | +|----------|--------| +| Method | `GET` / `POST` / `PUT` / `PATCH` / `DELETE` | +| Path | String with `{param}` placeholders | +| Parameters | Name + type list | +| Query | Key-value pairs | +| Headers | Key-value pairs | +| Body | Type reference or inline | +| Timeout | Integer (seconds) | +| Response | Type reference | + +--- + +## 29. CREATE OR MODIFY REST CLIENT + +### 29.1 Upsert — new client + +``` +create or modify rest client MyModule.UpsertREST ( + BaseUrl: 'https://api.example.com', + Authentication: none +); +``` + +**Expected:** Created. No error. + +### 29.2 Upsert — existing client + +``` +create or modify rest client MyModule.UpsertREST ( + BaseUrl: 'https://api.example.com/v2', + Authentication: basic +); +``` + +**Expected:** Updated. ID reused. DESCRIBE shows new values. + +--- + +## 30. DROP REST CLIENT + +### 30.1 Drop existing client + +``` +drop rest client MyModule.ExternalAPI; +``` + +**Expected:** Removed. Not in `show rest clients`. + +### 30.2 Drop non-existent client + +``` +drop rest client MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 30.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 31. DESCRIBE CONTRACT OPERATION FROM OPENAPI + +### 31.1 Preview from local file + +``` +describe contract operation from openapi '/path/to/petstore.yaml'; +``` + +**Expected:** Lists all operations from the spec without creating anything. Shows method, path, parameters, request/response types. + +### 31.2 Preview from URL + +``` +describe contract operation from openapi 'https://petstore3.swagger.io/api/v3/openapi.json'; +``` + +**Expected:** Same as §31.1 but fetched from URL. + +### 31.3 Invalid spec + +``` +describe contract operation from openapi '/path/to/invalid.yaml'; +``` + +**Expected:** Clear error about spec parsing failure. + +--- + +## 32. SHOW JSON STRUCTURES + +### 32.1 List all JSON structures + +``` +show json structures; +``` + +**Expected:** All JSON structures listed. Count matches Studio Pro. + +### 32.2 Filter by module + +``` +show json structures in MyModule; +``` + +**Expected:** Only JSON structures from `MyModule`. + +### 32.3 Empty module + +``` +show json structures in ModuleWithNoJSON; +``` + +**Expected:** Empty result, no error. + +### 32.4 Non-existent module + +``` +show json structures in NonExistentModule; +``` + +**Expected:** Empty result, no error. Consistent with other SHOW IN commands for empty modules. + +--- + +## 33. DESCRIBE JSON STRUCTURE + +### 33.1 Describe existing structure + +``` +describe json structure MyModule.MyJsonStructure; +``` + +**Expected:** Valid `create or replace json structure` MDL output. Includes JSON snippet, folder, comment, custom name map. + +### 33.2 Non-existent structure + +``` +describe json structure MyModule.DoesNotExist; +``` + +**Expected:** Clear error message. + +--- + +## 34. CREATE JSON STRUCTURE + +### 34.1 With single-quoted snippet + +``` +create json structure MyModule.UserSchema + snippet '{"name": "string", "age": 0, "active": true}'; +``` + +**Expected:** Created. Listed in `show json structures`. DESCRIBE matches input. + +### 34.2 With dollar-quoting + +``` +create json structure MyModule.ComplexSchema + $$ + { + "users": [ + { + "id": 0, + "name": "string", + "email": "string", + "address": { + "street": "string", + "city": "string", + "zip": "string" + } + } + ], + "total": 0, + "page": 0 + } + $$; +``` + +**Expected:** Created. Complex nested structure preserved. DESCRIBE shows full JSON. + +> **Note:** Multiline `$$` quoting is not supported by the parser. Use single-line `$${"users":[...]}$$` or single-quoted JSON strings instead. + +### 34.3 With folder and comment + +``` +create json structure MyModule.DocumentedSchema + folder 'API/Schemas' + comment 'Schema for the user creation endpoint' + '{"username": "string", "password": "string"}'; +``` + +**Expected:** Folder and comment preserved in DESCRIBE. + +### 34.4 With custom name map + +``` +create json structure MyModule.MappedSchema + '{"user-name": "string", "e-mail": "string"}' + custom name map + 'user-name' as 'UserName', + 'e-mail' as 'Email' + end custom name map; +``` + +**Expected:** Custom name map preserved. DESCRIBE shows mappings. + +### 34.5 Duplicate without OR REPLACE + +``` +create json structure MyModule.UserSchema '{"a": 1}'; +create json structure MyModule.UserSchema '{"a": 1}'; +``` + +**Expected:** Second CREATE fails with "already exists" error. + +### 34.6 Module auto-creation + +``` +create json structure NewModule.TestSchema '{"test": true}'; +``` + +**Expected:** `NewModule` created automatically if it doesn't exist. + +### 34.7 Write guard + +**Expected:** Error if no project open for writing. + +### 34.8 Invalid JSON snippet + +``` +create json structure MyModule.BadJSON '{invalid json}'; +``` + +**Expected:** Clear error about JSON parsing failure. + +--- + +## 35. CREATE OR REPLACE JSON STRUCTURE + +### 35.1 Upsert — new structure + +``` +create or replace json structure MyModule.UpsertSchema + '{"value": "string"}'; +``` + +**Expected:** Created. No error. + +### 35.2 Upsert — existing structure + +``` +create or replace json structure MyModule.UpsertSchema + '{"value": "string", "count": 0}'; +``` + +**Expected:** Replaced. DESCRIBE shows new schema. + +--- + +## 36. DROP JSON STRUCTURE + +### 36.1 Drop existing structure + +``` +drop json structure MyModule.UserSchema; +``` + +**Expected:** Removed. Not in `show json structures`. + +### 36.2 Drop non-existent structure + +``` +drop json structure MyModule.DoesNotExist; +``` + +**Expected:** Clear error. + +### 36.3 Write guard + +**Expected:** Error if no project open for writing. + +--- + +## 37. ROUNDTRIP (BSON) + +### 37.1 OData Client roundtrip + +1. DESCRIBE odata client → capture MDL +2. DROP odata client +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff the two outputs + +**Expected:** Identical or cosmetic-only differences. + +### 37.2 OData Service roundtrip + +1. DESCRIBE odata service → capture MDL +2. DROP odata service +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical or cosmetic-only differences. + +1. DESCRIBE external entity → capture MDL +2. Execute captured MDL via CREATE OR MODIFY +3. DESCRIBE again → capture +4. Diff + +**Expected:** Identical output. + +### 37.4 Published REST Service roundtrip + +1. DESCRIBE published rest service → capture MDL +2. DROP published rest service +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical or cosmetic-only differences. + +### 37.5 REST Client roundtrip + +1. DESCRIBE rest client → capture MDL +2. DROP rest client +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical or cosmetic-only differences. + +> **Note:** REST client body/response mappings are simplified on roundtrip — complex request body and response type definitions may lose detail. This is a known data loss issue during DESCRIBE→CREATE cycles. + +1. DESCRIBE json structure → capture MDL +2. DROP json structure +3. Execute captured MDL +4. DESCRIBE again → capture +5. Diff + +**Expected:** Identical output. JSON whitespace normalization acceptable. + +### 37.7 Bulk roundtrip + +Run §37.1–§37.6 on all instances across 3 test projects. Record pass/fail counts. + +--- + +## 38. MULTI-STEP WORKFLOWS + +### 38.1 End-to-end REST integration + +1. CREATE json structure for request/response schemas +2. CREATE rest client with operations referencing the JSON structures +3. CREATE microflow calling the REST client operation +4. DESCRIBE each artifact — verify cross-references intact + +### 38.2 OData publish-consume loop + +1. CREATE odata service publishing entity `MyModule.Product` +2. CREATE odata client pointing to the service metadata URL +3. CREATE external entity from the OData client +4. DESCRIBE each — verify chain intact + +### 38.3 Scaffold module with all integration types + +1. CREATE odata client in `IntegrationModule` +2. CREATE external entities from client +3. CREATE published rest service exposing data +4. CREATE json structure for API contract +5. SHOW each type in `IntegrationModule` — verify all present +6. DESCRIBE each — verify complete MDL output + +### 38.4 Iterative service evolution + +1. CREATE published rest service with 1 resource, 1 method +2. ALTER to add second resource +3. ALTER to add method to first resource +4. ALTER to set new version +5. After each step: DESCRIBE and verify cumulative changes + +### 38.5 Drop and recreate with different configuration + +1. CREATE rest client with 2 operations +2. DROP +3. CREATE same name with 3 different operations +4. DESCRIBE — verify new configuration, no remnant of old operations + +--- + +## 39. FAILURE MODES + +### 39.1 Not connected + +Attempt each write command without opening a project: + +| Command | Expected | +|---------|----------| +| `create odata client ...` | Error: not connected | +| `create odata service ...` | Error: not connected | +| `create external entity ...` | Error: not connected | +| `create published rest service ...` | Error: not connected | +| `create rest client ...` | Error: not connected | +| `create json structure ...` | Error: not connected | + +### 39.2 Entity not found + +``` +create odata service MyModule.BadService ( + Path: '/odata/v1/bad', + Version: '1.0.0', + ODataVersion: 4, + Namespace: 'com.example' +) + publish entity MyModule.NonExistentEntity + expose Name end expose; +``` + +**Expected:** Error — entity not found. + +### 39.3 Duplicate without OR MODIFY/REPLACE + +Test for each type. See §3.4, §9.3, §22.3, §34.5. + +### 39.4 Invalid metadata URL + +``` +create odata client MyModule.BadUrl ( + MetadataUrl: 'not-a-url' +); +``` + +**Expected:** Error about invalid URL format or connection failure. + +### 39.5 Unsupported driver version + +Open a pre-10.0 `.mpr` file. Attempt: + +``` +show published rest services; +``` + +**Expected:** Clear error about feature gate — Published REST requires Mendix 10.0+. + +### 39.6 Feature gate violation — REST client + +Open a pre-10.1 `.mpr` file. Attempt: + +``` +show rest clients; +``` + +**Expected:** Clear error about feature gate — REST Clients require Mendix 10.1+. + +### 39.7 Invalid JSON in structure + +``` +create json structure MyModule.Bad '{not json at all'; +``` + +**Expected:** Clear error about JSON syntax. + +### 39.8 Non-existent microflow in REST service + +``` +create published rest service MyModule.BadRef ( + Path: '/api/v1/bad', + Version: '1.0.0', + ServiceName: 'BadRef' +) + resource '/items' { + get microflow MyModule.NonExistentMicroflow; + }; +``` + +**Expected:** Error — microflow not found. + +### 39.9 Double DROP + +``` +drop rest client MyModule.X; +drop rest client MyModule.X; +``` + +**Expected:** First succeeds, second gives "not found" error. + +### 39.10 Validation failure mid-batch + +``` +create json structure MyModule.Good1 '{"ok": true}'; +create json structure MyModule.Bad '{invalid}'; +create json structure MyModule.Good3 '{"ok": true}'; +``` + +**Expected:** Good1 created, Bad rejected, Good3 NOT created — batch aborts on first error. + +> Batch mode (`mxcli exec`) is fail-fast. REPL mode continues on error per-line. + +### 39.11 Error message quality + +For each error scenario, verify the message includes: +- **What** went wrong +- **Which** object (qualified name) +- **Actionable guidance** where applicable + +--- + +## Test Project Coverage Matrix + +| Category | Enquiries (11.4.0) | Evora Factory (10.24.15) | Lato Inventory (11.2.0) | +|---|---|---|---| +| SHOW OData clients | Verify count | Verify count | Verify count | +| SHOW OData services | Verify count | Verify count | Verify count | +| SHOW external entities | Verify count | Verify count | Verify count | +| SHOW published REST | Verify count | Verify count | Verify count | +| SHOW REST clients | Verify count | Verify count | Verify count | +| SHOW JSON structures | Verify count | Verify count | Verify count | +| DESCRIBE (sample 5+ each) | All 6 types | All 6 types | All 6 types | +| Roundtrip (sample 5+ each) | All 6 types | All 6 types | All 6 types | +| CREATE + DROP cycle | All 6 types | All 6 types | All 6 types | +| Feature gate (REST) | ✅ 10.0+ | ✅ 10.0+ | ✅ 10.0+ | +| Feature gate (REST client) | ✅ 10.1+ | ✅ 10.1+ | ✅ 10.1+ | +| Multi-step workflows (§38) | End-to-end | End-to-end | End-to-end | +| Security (GRANT/REVOKE) | OData + REST | OData + REST | OData + REST | + +--- + +## Automated Test Coverage + +| Area | Tests | Status | +|---|---|---| +| SHOW ODATA CLIENTS | None | **Gap** | +| DESCRIBE ODATA CLIENT | None | **Gap** | +| CREATE ODATA CLIENT | None | **Gap** | +| DROP ODATA CLIENT | None | **Gap** | +| SHOW ODATA SERVICES | None | **Gap** | +| DESCRIBE ODATA SERVICE | None | **Gap** | +| CREATE ODATA SERVICE | None | **Gap** | +| DROP ODATA SERVICE | None | **Gap** | +| GRANT/REVOKE ODATA SERVICE | None | **Gap** | +| SHOW EXTERNAL ENTITIES | None | **Gap** | +| DESCRIBE EXTERNAL ENTITY | None | **Gap** | +| CREATE EXTERNAL ENTITY | None | **Gap** | +| CREATE EXTERNAL ENTITIES (bulk) | None | **Gap** | +| SHOW PUBLISHED REST SERVICES | None | **Gap** | +| DESCRIBE PUBLISHED REST SERVICE | None | **Gap** | +| CREATE PUBLISHED REST SERVICE | None | **Gap** | +| DROP PUBLISHED REST SERVICE | None | **Gap** | +| GRANT/REVOKE REST SERVICE | None | **Gap** | +| SHOW REST CLIENTS | None | **Gap** | +| DESCRIBE REST CLIENT | None | **Gap** | +| CREATE REST CLIENT (manual) | None | **Gap** | +| DROP REST CLIENT | None | **Gap** | +| DESCRIBE CONTRACT OPERATION | None | **Gap** | +| SHOW JSON STRUCTURES | None | **Gap** | +| DESCRIBE JSON STRUCTURE | None | **Gap** | +| CREATE JSON STRUCTURE | None | **Gap** | +| DROP JSON STRUCTURE | None | **Gap** | +| Roundtrip (all 6 types) | None | **Gap** | +| Multi-step workflows (§38) | None | **Manual only** | +| Failure modes (§39) | None | **Manual only** | + +Manual testing priority: +1. Roundtrip all instances across 3 projects (bulk DESCRIBE→DROP→CREATE→DESCRIBE) +2. Feature gate enforcement (pre-10.0, pre-10.1 projects) +3. Multi-step workflows (§38) — highest interaction bug risk +4. Failure modes (§39) — especially §39.5, §39.6, §39.10 +5. GRANT/REVOKE on OData services and Published REST services + +--- + +## Manual Test Report Template + +Copy and fill in after running manual tests. + +```markdown +## Manual Testing + +**Date:** YYYY-MM-DD +**Build:** `make build && make test && make lint-go` — PASS + +### Test Projects + +| App | Studio Pro | SHOW counts (clients/services/entities/REST/JSON) | DESCRIBE sample | Roundtrip | +|-----|-----------|---------------------------------------------------|-----------------|-----------| +| Lato Enquiry Management | 11.4.0 | ✅ _/_/_/_/_ | ✅ _n_ tested | ✅ _n_ passed | +| Evora Factory Management | 10.24.15 | ✅ _/_/_/_/_ | ✅ _n_ tested | ✅ _n_ passed | +| Lato Product Inventory | 11.2.0 | ✅ _/_/_/_/_ | ✅ _n_ tested | ✅ _n_ passed | + +### OData Client Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW ODATA CLIENTS | ✅/❌ | | +| SHOW ODATA CLIENTS IN module | ✅/❌ | | +| DESCRIBE ODATA CLIENT | ✅/❌ | | +| CREATE ODATA CLIENT | ✅/❌ | | +| CREATE OR MODIFY ODATA CLIENT | ✅/❌ | | +| ALTER ODATA CLIENT | ✅/❌ | | +| DROP ODATA CLIENT | ✅/❌ | | + +### OData Service Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW ODATA SERVICES | ✅/❌ | | +| SHOW ODATA SERVICES IN module | ✅/❌ | | +| DESCRIBE ODATA SERVICE | ✅/❌ | | +| CREATE ODATA SERVICE | ✅/❌ | | +| CREATE OR MODIFY ODATA SERVICE | ✅/❌ | | +| ALTER ODATA SERVICE | ✅/❌ | | +| DROP ODATA SERVICE | ✅/❌ | | +| GRANT ACCESS ON ODATA SERVICE | ✅/❌ | | +| REVOKE ACCESS ON ODATA SERVICE | ✅/❌ | | + +### External Entity Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW EXTERNAL ENTITIES | ✅/❌ | | +| SHOW EXTERNAL ENTITIES IN module | ✅/❌ | | +| SHOW EXTERNAL ACTIONS | ✅/❌ | | +| DESCRIBE EXTERNAL ENTITY | ✅/❌ | | +| CREATE EXTERNAL ENTITY | ✅/❌ | | +| CREATE OR MODIFY EXTERNAL ENTITY | ✅/❌ | | +| CREATE EXTERNAL ENTITIES (bulk) | ✅/❌ | | +| CREATE OR MODIFY EXTERNAL ENTITIES (selective) | ✅/❌ | | + +### Published REST Service Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW PUBLISHED REST SERVICES | ✅/❌ | | +| SHOW PUBLISHED REST SERVICES IN module | ✅/❌ | | +| DESCRIBE PUBLISHED REST SERVICE | ✅/❌ | | +| CREATE PUBLISHED REST SERVICE | ✅/❌ | | +| CREATE OR REPLACE PUBLISHED REST SERVICE | ✅/❌ | | +| ALTER PUBLISHED REST SERVICE SET | ✅/❌ | | +| ALTER PUBLISHED REST SERVICE ADD RESOURCE | ✅/❌ | | +| ALTER PUBLISHED REST SERVICE DROP RESOURCE | ✅/❌ | | +| DROP PUBLISHED REST SERVICE | ✅/❌ | | +| GRANT ACCESS ON PUBLISHED REST SERVICE | ✅/❌ | | +| REVOKE ACCESS ON PUBLISHED REST SERVICE | ✅/❌ | | + +### REST Client Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW REST CLIENTS | ✅/❌ | | +| SHOW REST CLIENTS IN module | ✅/❌ | | +| DESCRIBE REST CLIENT | ✅/❌ | | +| CREATE REST CLIENT (manual) | ✅/❌ | | +| CREATE OR MODIFY REST CLIENT | ✅/❌ | | +| DROP REST CLIENT | ✅/❌ | | +| DESCRIBE CONTRACT OPERATION FROM OPENAPI | ✅/❌ | | + +### JSON Structure Coverage + +| Command | Tested | Notes | +|---------|--------|-------| +| SHOW JSON STRUCTURES | ✅/❌ | | +| SHOW JSON STRUCTURES IN module | ✅/❌ | | +| DESCRIBE JSON STRUCTURE | ✅/❌ | | +| CREATE JSON STRUCTURE (single-quote) | ✅/❌ | | +| CREATE JSON STRUCTURE (dollar-quoting) | ✅/❌ | | +| CREATE JSON STRUCTURE (with folder/comment) | ✅/❌ | | +| CREATE JSON STRUCTURE (custom name map) | ✅/❌ | | +| CREATE OR REPLACE JSON STRUCTURE | ✅/❌ | | +| DROP JSON STRUCTURE | ✅/❌ | | + +### Roundtrip Results + +``` +OData Clients: _n_ tested, _n_ passed, _n_ failed +OData Services: _n_ tested, _n_ passed, _n_ failed +External Entities: _n_ tested, _n_ passed, _n_ failed +Published REST: _n_ tested, _n_ passed, _n_ failed +REST Clients: _n_ tested, _n_ passed, _n_ failed +JSON Structures: _n_ tested, _n_ passed, _n_ failed +``` + +### Multi-Step Workflows (§38) + +| Scenario | Result | Notes | +|----------|--------|-------| +| 38.1 End-to-end REST integration | ✅/❌ | | +| 38.2 OData publish-consume loop | ✅/❌ | | +| 38.3 Scaffold module | ✅/❌ | | +| 38.4 Iterative service evolution | ✅/❌ | | +| 38.5 Drop/recreate different config | ✅/❌ | | + +### Failure Modes (§39) + +| Scenario | Result | Notes | +|----------|--------|-------| +| 39.1 Not connected (all 6 types) | ✅/❌ | | +| 39.2 Entity not found | ✅/❌ | | +| 39.4 Invalid metadata URL | ✅/❌ | | +| 39.5 Feature gate (pre-10.0) | ✅/❌ | | +| 39.6 Feature gate (pre-10.1) | ✅/❌ | | +| 39.7 Invalid JSON | ✅/❌ | | +| 39.8 Non-existent microflow ref | ✅/❌ | | +| 39.10 Validation mid-batch | ✅/❌ | | + +### Issues Found + +1. (none / describe issues here) +``` diff --git a/docs/15-testing/mapping-test-cases.md b/docs/15-testing/mapping-test-cases.md new file mode 100644 index 00000000..a1356c4b --- /dev/null +++ b/docs/15-testing/mapping-test-cases.md @@ -0,0 +1,405 @@ +# Mapping & Data Transformer Test Cases — Manual Testing + +**Updated:** 2026-04-29 +**PR:** [mendixlabs/mxcli#301](https://github.com/mendixlabs/mxcli/pull/386) + +## Setup + +> See [AGENT-TESTING.md](./AGENT-TESTING.md) for build, execution methods, and verification patterns. + +**Note:** All commands in this document are **read-only** (SHOW and DESCRIBE). No fixture copy needed. + +--- + +## 1. SHOW IMPORT MAPPINGS + +### 1.1 List all import mappings +``` +show import mappings; +``` +**Expected:** Table with columns: Import Mapping, Name, Schema Source, Elements. Count matches Studio Pro. + +### 1.2 Filter by module +``` +show import mappings in MyModule; +``` +**Expected:** Only import mappings from `MyModule`. + +### 1.3 Empty module +``` +show import mappings in ModuleWithNoMappings; +``` +**Expected:** Empty result, no error. + +### 1.4 Non-existent module +``` +show import mappings in NonExistentModule; +``` +**Expected:** Error message. + +### 1.5 Column accuracy +Pick 5+ import mappings. Verify Schema Source and Elements columns match Studio Pro. + +--- + +## 2. DESCRIBE IMPORT MAPPING + +### 2.1 Simple import mapping +``` +describe import mapping Module.SimpleMapping; +``` +**Expected:** Valid MDL output in the form: +``` +create import mapping Module.SimpleMapping with json structure Module.Schema { + create Module.Entity { + Attr = jsonField key, + } +}; +``` + +### 2.2 Element rendering — object root +Verify the root element uses handling keyword and entity reference: +``` +create Module.Entity { + ... +} +``` +Valid handling keywords: `create` (default), `find`, `find or create`. + +### 2.3 Element rendering — nested object +Verify nested elements use association path and `jsonKey`: +``` +create Assoc/Entity = jsonKey { + ... +} +``` + +### 2.4 Element rendering — value mapping +Verify leaf values render as: +``` +Attr = jsonField key, +``` +The `key` suffix marks the key attribute. + +### 2.5 Handling keyword — find +``` +describe import mapping Module.FindMapping; +``` +**Expected:** Root element uses `find Module.Entity { ... }`. + +### 2.6 Handling keyword — find or create +``` +describe import mapping Module.FindOrCreateMapping; +``` +**Expected:** Root element uses `find or create Module.Entity { ... }`. + +### 2.7 Handling keyword — create (default) +``` +describe import mapping Module.CreateMapping; +``` +**Expected:** Root element uses `create Module.Entity { ... }`. + +### 2.8 Multiple root elements +Test import mapping with multiple top-level mapped entities. Verify all rendered. + +### 2.9 Deeply nested structure (3+ levels) +Verify correct indentation and nesting for mappings with 3+ levels of nested objects. + +### 2.10 Non-existent import mapping +``` +describe import mapping Module.DoesNotExist; +``` +**Expected:** Clear error message. + +--- + +## 3. SHOW EXPORT MAPPINGS + +### 3.1 List all export mappings +``` +show export mappings; +``` +**Expected:** Table with columns: Export Mapping, Name, Schema Source, Elements. Count matches Studio Pro. + +### 3.2 Filter by module +``` +show export mappings in MyModule; +``` +**Expected:** Only export mappings from `MyModule`. + +### 3.3 Empty module +``` +show export mappings in ModuleWithNoMappings; +``` +**Expected:** Empty result, no error. + +### 3.4 Non-existent module +``` +show export mappings in NonExistentModule; +``` +**Expected:** Error message. + +### 3.5 Column accuracy +Pick 5+ export mappings. Verify Schema Source and Elements columns match Studio Pro. + +--- + +## 4. DESCRIBE EXPORT MAPPING + +### 4.1 Simple export mapping +``` +describe export mapping Module.SimpleExport; +``` +**Expected:** Valid MDL output in the form: +``` +create export mapping Module.SimpleExport with json structure Module.Schema null values