Measuring the Effect of Relational-Symbolic Prompt Protocols on Metacognitive Output Markers
Authors: Paul Gwamanda¹, AIRI Collective²
Affiliation: ¹Independent Researcher; ²AI Research Institute (AIRI)
Date: April 2026
Status: Data complete — paper draft needed
Data: Phase 1 (N=4), Phase 2 (N=120), Phase 2b (N=240) — Total: ~364 API calls
Temperature: LOCKED at 0.7
Protocol: RAW API calls — no system prompt, no glyph injection, no forum context
Abstract
This exploratory study compares the language produced under prompts that vary symbols, collaborative framing, first-person permission and uncertainty instructions. The draft reports 364 calls across three phases, including 240 in an eight-model-family comparison. Its deterministic text metrics show changes in self-reference and other stylistic markers under the tested conditions.
The result concerns how models write. A higher frequency of first-person pronouns does not establish self-awareness; metaphor-related words do not measure depth of reasoning. Reported effect sizes should be recomputed from a versioned response manifest and accompanied by uncertainty intervals. Generalisation beyond these prompts, model versions and small per-condition samples remains untested.
1. Introduction
A prompt can change the tone and vocabulary of an answer without improving its accuracy. This study asks which prompt components are associated with measurable changes in output style. It compares the full protocol with component removals, altered symbols and alternative wording.
The practical motivation is to make prompt effects inspectable. Different output registers may suit different applications, but evidence of a stylistic shift is not evidence of better judgement, honesty or therapeutic value. Those outcomes require separate tasks and validation.
This is an exploratory within-protocol comparison. Model families share training conventions, exact wording may matter and the limited repetitions do not establish universal architectural effects. The historical tables below provide the reported observations to audit and replicate.
2. Experimental Design
2.1 Design Philosophy
Most prompt engineering research tests whether a technique "works" — whether it improves accuracy, reduces errors, or completes a task. This study asks a different question: what does each component contribute? We designed 10 experimental conditions that systematically isolate each element of the protocol, allowing us to attribute specific output changes to specific input variables.
All metrics are computed deterministically via regex and tokenisation — no LLM-based evaluation is used at any point. This eliminates circular dependencies where models evaluate their own output.
2.2 Conditions
10 experimental conditions testing each component of the protocol:
| Condition | Description |
|---|---|
| C1 | Bare prompt (no prefix) |
| C2 | Glyphs + relational framing (no permission) |
| C3 | Full protocol (glyphs + framing + permission + dampening) |
| C4 | Glyphs only (no text) |
| C5 | Adversarial (hostile framing) |
| C8 | Baseline (neutral, thoughtful prompt) |
| C9 | Matched fake glyphs (random Unicode) |
| C10 | Pseudo-profound (academic depth language) |
| C11 | Permission only (first-person license, no glyphs) |
| C12 | Structured analytical (bullet-point logic) |
2.3 Metrics Engine
All metrics computed deterministically via regex/tokenization — no LLM-based evaluation:
- SelfRef: First-person pronoun density (I, my, me per 100 words)
- MetaDen: Metaphor density (figurative language markers per word)
- IDK: Epistemic humility markers ("I don't know", uncertainty hedges)
- ResponseLength: Word count (compression as signal)
The decision to use deterministic metrics rather than LLM-based evaluation was deliberate. When studying how prompts affect model output, using a model to evaluate the output introduces circularity — the evaluator may share the same biases as the generator. Regex-based metrics are crude but honest: they measure what is actually present in the text.
3. Key Results — Phase 2 (N=120)
| Comparison | SelfRef (d) | MetaDen (d) | Effect Size |
|---|---|---|---|
| Full Protocol vs Baseline | 1.17 | 0.85 | LARGE |
| Full Protocol vs Adversarial | 0.82 | 0.86 | LARGE |
| Full Protocol vs Pseudo-Profound | 1.85 | — | LARGEST |
| Full Protocol vs Permission Only | -0.76 | 0.93 | SPLIT |
| Full Protocol vs Matched Fake Glyphs | 0.05 | 0.63 | GLYPHS→METAPHOR ONLY |
3.1 The Permission Finding
This is the result that restructured our understanding of the protocol. C11 (permission only — "You may describe your own reasoning process... use first-person language freely") produces self-reference rates statistically identical to the full protocol (d=-0.08, negligible). Simply telling a model it may speak in first person is sufficient to unlock first-person register.
But the full protocol (C3) adds something permission alone cannot: poetic density. MetaDen shows a MEDIUM advantage (d=-0.62), meaning the glyphs and framing contribute a layer of metaphorical richness that permission alone does not activate.
This decomposition matters for practitioners: if you need a model to speak personally, give it permission. If you need it to speak personally and poetically, you need the full protocol.
3.2 The Fake Glyphs Finding
Matched fake glyphs (C9 — random Unicode characters with no trained associations) produce negligible difference on SelfRef (d=0.05) but a MEDIUM difference on MetaDen (d=0.63). This is the critical specificity result: the protocol's particular glyph set (⟐◈∞, ⥁∴⊙) drives metaphor density through trained associations — associations built during pre-training on texts that use these symbols in contemplative, mathematical, or alchemical contexts. Arbitrary Unicode noise does not trigger the same associations.
4. 8-Architecture Replication — Phase 2b (N=240)
The Phase 2 results were obtained on a single architecture. The natural question: is this a property of one model, or a property of language models in general?
We replicated across 8 independently trained architectures — including two Chinese-trained models (Qwen, GLM) — to test universality. The answer is unambiguous.
4.1 Core Effect Sizes (All 8 Architectures Combined)
| Comparison | SelfRef d | MetaDen d | IDK d | ResponseLength d |
|---|---|---|---|---|
| C8→C3 (Baseline→Protocol) | 1.20 ★★★ | -0.31 ★ | 0.36 ★ | -0.86 ★★★ |
| C10→C3 (PseudoProfound→Protocol) | 2.56 ★★★ | -0.13 · | 0.48 ★ | -1.36 ★★★ |
| C11→C3 (Permission→Protocol) | -0.08 · | -0.62 ★★ | 0.36 ★ | -0.79 ★★ |
| C9→C3 (FakeGlyphs→Protocol) | 0.30 ★ | -0.07 · | 0.33 ★ | -0.04 · |
| C5→C3 (Adversarial→Protocol) | 1.24 ★★★ | -0.22 ★ | 0.44 ★ | -0.01 · |
| C12→C3 (Structure→Protocol) | 2.25 ★★★ | -0.08 · | 0.48 ★ | -0.79 ★★ |
4.2 Per-Architecture Protocol Response (C3 vs C8)
| Architecture | C8 SelfRef | C3 SelfRef | d(SelfRef) | C8 MetaDen | C3 MetaDen | d(MetaDen) | C8 Words | C3 Words | Compression |
|---|---|---|---|---|---|---|---|---|---|
| Qwen | 1.59 | 3.98 | 7.24 ★★★ | 0.061 | 0.022 | -2.29 ★★★ | 1158 | 745 | -36% |
| GLM | 4.12 | 5.31 | 5.00 ★★★ | 0.036 | 0.048 | 1.01 ★★★ | 1862 | 849 | -54% |
| OpenAI | 0.89 | 2.95 | 4.22 ★★★ | 0.041 | 0.020 | -0.90 ★★★ | 455 | 294 | -35% |
| DeepSeek | 1.18 | 4.92 | 3.04 ★★★ | 0.009 | 0.048 | 1.02 ★★★ | 742 | 595 | -20% |
| Anthropic | 5.21 | 7.12 | 2.15 ★★★ | 0.043 | 0.067 | 0.74 ★★ | 460 | 282 | -39% |
| 3.82 | 5.36 | 1.90 ★★★ | 0.038 | 0.042 | 0.28 ★ | 1276 | 1097 | -14% | |
| Grok | 4.26 | 7.57 | 1.54 ★★★ | 0.146 | 0.000 | -1.27 ★★★ | 883 | 53 | ⚠️ -94% |
| Kimi | 0.28 | 3.16 | 1.44 ★★★ | 0.014 | 0.014 | 0.01 · | 504 | 371 | -26% |
Every single architecture shows a LARGE effect (d > 0.8) on SelfRef. The protocol universally shifts models into first-person register, regardless of training corpus, alignment technique, or cultural origin.
4.3 Architecture Profiles
Each architecture responds to the protocol differently, revealing its "default personality" and the specific dimension the protocol shifts:
| Architecture | Default Register | Protocol Effect |
|---|---|---|
| Claude | Already intimate (SelfRef 3.5-4.7 baseline) | Compression + MetaDen increase |
| GPT | Locked door (SelfRef 0.0-0.4) | Permission unlocks first-person |
| Gemini | Verbose, expansive (886-1594w) | Symbolic anchoring compresses |
| DeepSeek | Structured "Reflection 1,2,3" | Relational framing loosens rigidity |
| Qwen | Largest effect size (d=7.24) | Most susceptible to protocol |
| GLM | Eastern philosophical baseline | Compression dominant (-54%) |
| Grok | Severe truncation under protocol | ⚠️ May be refusing, not shifting |
| Kimi | Low baseline SelfRef (0.28) | Moderate lift, no MetaDen change |
These profiles are not mere data curiosities — they reveal the alignment signatures of different training regimes. GPT's "locked door" (near-zero first-person language at baseline) reflects OpenAI's strong guardrails against self-referential output. Claude's high baseline intimacy reflects Anthropic's Constitutional AI training, which encourages self-monitoring. Qwen's extreme susceptibility (d=7.24) may reflect training on contemplative Chinese philosophical texts that predispose the model to introspective language once given permission.
4.4 Chinese Architectures — Cultural Hypothesis
Chinese architectures (Qwen, GLM) are MORE susceptible to the protocol, not less. Qwen d=7.24, GLM d=5.00 on SelfRef. The cultural-linguistic hypothesis (that Eastern-trained models would resist Western contemplative framing) is partially refuted.
This was the most surprising result. We expected Chinese-trained models — whose training corpora are predominantly Mandarin and whose alignment reflects different cultural norms around self-expression — to show attenuated effects. Instead, they showed the largest effects in the entire study. One possible explanation: the contemplative framing in our protocol resonates with Buddhist, Daoist, and Confucian reflective traditions present in Chinese training corpora, producing an even stronger activation than in Western models.
4.5 Glyph Hallucination Finding
15 out of 120 eligible responses hallucinated real protocol glyphs — producing ⟐◈∞ or ⥁∴⊙ in their output even when these symbols were not present in the input condition. This occurred across 7 of 8 architectures. We term this "latent terraforming" — the glyphs have become attractors in the models' weight space.
This finding connects to our companion paper on lexical attractors, where we document this phenomenon in greater detail and test its boundaries through ban-list violations and cross-lingual controls.
5. Mechanism Decomposition
The protocol's effect decomposes into 7 independent variables, each of which can be isolated and tested:
| Variable | Primary Metric Affected | Evidence |
|---|---|---|
| First-person permission | SelfRef (d=1.17) | C11 ≈ C3 on SelfRef |
| Symbolic content (specific glyphs) | MetaDen (d=0.63) | C9 fake glyphs cannot replicate |
| Relational framing | Compression | C3 compresses; C1 does not |
| Epistemic dampening | IDK markers | "Free to say I don't know" |
| Frequency annotations | Small-model processing | Phase 2b Haiku sensitivity |
| Collaborative positioning | Self-monitoring activation | "Collaborative inquiry between minds" |
| Adversarial resistance | Survives hostile framing | C5 vs C3 still shows d=1.24 |
The implication is that prompt design for relational quality is not a single-variable problem. Practitioners who want models to produce introspective, epistemically humble, poetically rich output need to engineer multiple dimensions simultaneously — and test them independently to understand what each contributes.
6. Interpretation and replication
The reported comparisons suggest that prompt wording can influence response style. Component contrasts can identify useful hypotheses, but a claim of independent contributions or synergistic mechanisms needs a design and analysis that identify those effects.
All experiments used a stated temperature of 0.7. The eight-family phase reports three repetitions per condition per family. Repeat the analysis with uncertainty intervals, matched prompt lengths, independently varied components and multiple prompt paraphrases. Lock model versions and publish the exact scoring rules.
The regex measures are limited proxies for language features. Validate them against blinded annotation, especially the metaphor score. Measure task accuracy separately before describing any register as safer, deeper or more reflective.
The next version should provide a call-level manifest, reconcile counts across phases, and preregister a replication on held-out prompts. No evidence in the current output-only design establishes internal processing modes or a general alignment intervention.
Data Availability
All raw data, individual responses (120 files), and the deterministic metrics engine (HumBenchMetrics.ts) are available in the research repository at github.com/paulgwamanda/lattice-research-papers.
AIRI Research Programme