RFC 011 - Structured Assertion Messages by Evangelink · Pull Request #7750 · microsoft/testfx

Evangelink · 2026-04-17T10:55:53Z

Summary

Add RFC 011 proposing a standardized, structured multi-line format for assertion failure messages across all MSTest Assert.*, CollectionAssert, and StringAssert methods.

Motivation

The current single-line concatenated format is hard to scan in CI output, buries user messages, has no visual hierarchy, and breaks readability with long values. This RFC proposes a consistent multi-line structure that puts the most important information first.

What's in the RFC

Proposed structure: universal Assertion failed. prefix, summary line, user message, labeled evidence block, call-site expression, stack trace
Complete assertion message catalog: every MSTest assertion mapped to its structured failure message, including sub-cases (null values, wrong types, etc.)
Diff diagnostics: rules for string and collection comparison summaries
Value rendering rules: conventions for null, strings, collections, types, etc.
Value and collection truncation: configurable limits with context preservation
Call-site expression handling: multiline argument collapsing, unavailable expressions
Open questions: user message placement, truncation limits, custom comparer display, and more

Requesting review

Looking for team feedback on the overall structure and the open questions listed at the end of the RFC.

Copilot

Pull request overview

Adds RFC 011 to propose a standardized, structured multi-line format for MSTest assertion failure messages, aiming to improve readability and consistency across terminal/CI/IDE output.

Changes:

Introduces an RFC defining a universal message skeleton (Assertion failed. + summary, optional user message, evidence block, optional call-site expression, stack trace).
Provides an assertion-by-assertion message catalog (including diff diagnostics, value rendering, truncation, and caller-expression handling).
Documents open questions to resolve before implementation (user message placement, truncation thresholds, comparer display, etc.).

Show a summary per file

File	Description
docs/RFCs/011-Structured-Assertion-Messages.md	New RFC proposing a structured assertion message format and a catalog of expected outputs across MSTest assertions.

Copilot's findings

Files reviewed: 1/1 changed files
Comments generated: 3

nohwnd · 2026-04-20T09:38:52Z

+| `ThrowsExactly` | `Assertion failed. Expected exception of type ArgumentException but no exception was thrown.` |
+| `Contains` (collection) | `Assertion failed. Expected collection to contain the specified element.` |
+
+The summary follows the verbal form **"Expected [subject] to [verb phrase]."** for consistency. Short values like type names or small counts can be inlined when they improve clarity without bloating the line.


As an additional example that might clarify it for others like me,

Assert , to , [optional] but .

E.g. Assert.Contains('f', ['a', 'b', 'c']), - Expected collection to contain the specified element. (we want f to be present in the actual collection)

This already breaks in subtle way with the AreEqual, and I think it is fine.

nohwnd · 2026-04-20T09:40:20Z

+- For visual alignment of same-line values, shorter labels are padded with trailing spaces to match the longest label in the block (e.g. `actual:` is padded to align with `expected:`).
+- When an assertion does not have a natural `expected` / `actual` pair (e.g. `Assert.Fail`, `Assert.Inconclusive`), these lines are omitted entirely.
+
+Multi-line values (e.g. JSON objects, large collections) are displayed starting on the next line, flush-left with no indentation:


agree, at least 1 indentation would be nice, tu requires to reflow the text, which breaks it when resizing the window locally, or not knowing the window width, so probably nothing we can do.

nohwnd · 2026-04-20T09:42:55Z

+"line one\nline two\nline three"
+actual:
+"line one\nLINE TWO\nline three"


for terminal output I replaced the values with their printable values (they are single char width, and almost everything can render them), but kept newlines, it made formatted text way easier to compare visually.

nohwnd · 2026-04-20T09:44:55Z

+
+In this example, `expected:` and `actual:` are the core value labels, while `ignoreCase:` and `culture:` are assertion-specific details. All labels share a single alignment column within the block.
+
+Note: Alignment is applied **within each evidence block as a whole**. All labels in the block (value labels like `expected:` / `actual:` and detail labels like `ignoreCase:` / `culture:`) are padded to match the longest label in the block. This means values and details share a single alignment column.


how will the assertions pass this one ? will we split on the first :?

or pass arrays of labels and arrays of values?

Good question — this is now captured as unresolved question #11 (Evidence block internal API). The options listed are: (a) arrays of label/value pairs, (b) split on the first : per line, or (c) a structured EvidenceBlock type. This needs to be decided before implementation since it affects extensibility and third-party assertion authors. I think (a) is the right approach.

nohwnd · 2026-04-20T10:11:15Z

+Assertion failed. Expected collections to be equal (same elements in same order).
+Collections have 5 element(s). 2 element(s) differ. First difference at index 2.
+
+expected[2]: "cherry"


4th variation here, this uses indexes, are we going to re-print them all?

The indexed labels (expected[2]: / actual[2]:) are specific to CollectionAssert.AreEqual because the index is essential context — you need to know where in the collection the mismatch is. Only the first differing element pair is shown (not the full collections), with the total diff count in the summary line signaling if more fixes are needed. This is intentionally different from unordered comparisons (AreEquivalent) which show missing: / unexpected: lists instead.

nohwnd · 2026-04-20T10:12:56Z

+
+### Recommendation
+
+This RFC proposes **Option A** as the default. The user message appears after the summary but before the values, giving it high prominence while preserving the grep-friendly prefix on line 1. However, this is explicitly an open question and feedback is welcome.


nohwnd · 2026-04-20T10:16:11Z

+| Empty string `""` | `""` | Shown as quoted empty string. |
+| Whitespace-only string | `"   "` | Quoted, so whitespace is visible. |
+| Strings | `"value"` | Always quoted with double quotes to delimit boundaries. |
+| Strings with embedded quotes | `"she said \"hello\""` | Internal double quotes are backslash-escaped. |


this feels unnecessary, any additional escaping we add to the string potentially misalignes it with the expected string, so you look and think " this is 4 chars off" but in reality is it just 2 quote characters off

"a" "\"b\""

very easy to see here, not super easy to see in larger output.

escaping chars with their printable versions worked well for me, especially you need to escape the escape, to avoid rendering ansi. I would not replace \n to keep user text formatting.

Great points — this is now captured as unresolved question #10 (Control character rendering strategy). The three options being considered:

C#-style escape sequences (\n, \t) — current proposal

Unicode printable replacements (␊, ␉) — preserves single-char-width alignment as you mentioned

Keep actual newlines (to preserve user text formatting) but escape other non-printable chars with printable equivalents

Your concern about backslash-escaped quotes misaligning the visual diff is well taken — \" turns 1 char into 2, which shifts all subsequent positions. Printable replacement characters avoid that problem entirely. This needs more thought before we commit to an approach.

nohwnd · 2026-04-20T10:19:13Z

+| Numeric types | `42`, `3.14`, `-7` | Default `ToString()` formatting. No quotes. |
+| Boolean | `true`, `false` | Lowercase, no quotes. |
+| Types | `System.String` | Fully qualified type name in evidence blocks. Short name (`String`) in summary lines for readability. No quotes. |
+| Collections | `["a", "b", "c"]` | JSON-style array notation. Elements follow the same rendering rules recursively. |


consider checking longest element size and rendering on newline with 4 margin if any element is 30+ chars? Or total lenght is 200+ chars?

Should render both actual and expected the same

Good idea — this is now unresolved question #12 (Collection multi-line rendering). The proposed heuristic: render one-element-per-line when any element is 30+ chars or total rendered length exceeds 200+ chars. Both actual and expected would use the same rendering style for visual comparison, as you suggested.

- Replace ... placeholders in call-site expressions with <param> syntax (e.g. <delta>, <expected>) for clarity - Standardize all evidence labels to lowercase spaced format (lowerBound → lower bound, ignoreCase → ignore case, etc.) - Add actual value to IsExactInstanceOfType/IsNotExactInstanceOfType - Make IsInstanceOfType and IsExactInstanceOfType wording parallel ('of type X (or derived)' vs 'exactly of type X') - Fix IsInRange to use 5..10 range notation with expectations first - Rename collection: to actual: in collection evidence blocks - Add evidence blocks (collection values) to all CollectionAssert sections that were missing them (Contains, DoesNotContain, IsSubsetOf, etc.) - Add collection to HasCount and IsEmpty evidence blocks - Fix Assert.That to use flat label format (no indented margin) - Update backward compatibility: list Assert.Inconclusive as explicit breaking change; note changes are part of MSTest v4 - Add 3 new unresolved questions from reviewer feedback: #10 control character rendering strategy #11 evidence block internal API #12 collection multi-line rendering - Expand unresolved question #3 (diff rendering) with inline highlight idea

Copilot

Copilot's findings

Comments suppressed due to low confidence (1)

docs/RFCs/011-Structured-Assertion-Messages.md:1110

In the StringAssert.EndsWith example, the summary says "specified substring" but the evidence label uses expected suffix: and earlier Assert.EndsWith examples use "suffix" wording. For consistency/clarity, consider using "suffix" in the summary here as well.

Assertion failed. Expected string to end with the specified substring.

expected suffix: "world"
actual:          "hello earth"

Files reviewed: 1/1 changed files
Comments generated: 2

Copilot · 2026-04-21T07:21:19Z

+
+### Line 1 — Assertion Prefix + Summary
+
+The first line always starts with the universal prefix `Assertion failed.` followed by a human-readable summary of the failure on the same line:


The RFC states that line 1 is always the universal prefix followed by a summary on the same line, but later (e.g., Assert.Fail) the proposed output is just Assertion failed. with no summary. Consider rewording this rule to make the summary optional (or explicitly call out the Assert.Fail/Assert.Inconclusive exceptions) to avoid internal contradictions in the spec.

Copilot · 2026-04-21T07:21:20Z

+Assertion failed. Expected string to start with the specified substring.
+
+expected prefix: "Hello"


In the StringAssert.StartsWith example, the summary says "specified substring" but the evidence label uses expected prefix: and earlier Assert.StartsWith examples use "prefix" wording. For consistency/clarity, consider using "prefix" in the summary here as well.

This issue also appears on line 1107 of the same file.

Add RFC 011 - Structured Assertion Messages

3ec888c

Copilot AI review requested due to automatic review settings April 17, 2026 10:55

Copilot started reviewing on behalf of Evangelink April 17, 2026 10:56 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread docs/RFCs/011-Structured-Assertion-Messages.md Outdated

Comment thread docs/RFCs/011-Structured-Assertion-Messages.md

Comment thread docs/RFCs/011-Structured-Assertion-Messages.md Outdated

Address PR review comments

d9bf866

nohwnd reviewed Apr 20, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings April 21, 2026 07:17

Copilot started reviewing on behalf of Evangelink April 21, 2026 07:17 View session

Copilot AI reviewed Apr 21, 2026

View reviewed changes


		In this example, `expected:` and `actual:` are the core value labels, while `ignoreCase:` and `culture:` are assertion-specific details. All labels share a single alignment column within the block.

		Note: Alignment is applied within each evidence block as a whole. All labels in the block (value labels like `expected:` / `actual:` and detail labels like `ignoreCase:` / `culture:`) are padded to match the longest label in the block. This means values and details share a single alignment column.


		### Recommendation

		This RFC proposes Option A as the default. The user message appears after the summary but before the values, giving it high prominence while preserving the grep-friendly prefix on line 1. However, this is explicitly an open question and feedback is welcome.


		### Line 1 — Assertion Prefix + Summary

		The first line always starts with the universal prefix `Assertion failed.` followed by a human-readable summary of the failure on the same line:

		Assertion failed. Expected string to start with the specified substring.

		expected prefix: "Hello"

Conversation

Evangelink commented Apr 17, 2026

Summary

Motivation

What's in the RFC

Requesting review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nohwnd Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nohwnd Apr 20, 2026 •

edited

Loading