LLM-Assisted Extraction and Validation of Expert Mental Models (Proof of Concept)


Overview

This prototype explores whether an LLM-assisted workflow can convert short excerpts of expert testimony into structured causal claims while limiting unsupported inference.

The core task is to identify: Source variable → causal direction → target variable

along with the mechanism, supporting evidence, explicitness of the claim, and confidence.

Rather than treating the first model output as authoritative, the workflow separates extraction, validation, uncertainty handling, and human review.

                                                                                      Workflow Architecture

                                                                                  Workflow Architecture

Research Question

Can an LLM reliably identify causal variables and relationships expressed in expert testimony while distinguishing genuine causal reasoning from correlation, co-occurrence, or unsupported inference?

A secondary question is:

Can a second model meaningfully identify cases where the initial extraction over-interprets the testimony?

Current implementation

The workflow is implemented in n8n. Current model roles are:

Component Implementation
Causal extraction DeepSeek Chat V3.1 via OpenRouter
Independent validation Amazon Nova Lite via OpenRouter
Ambiguous-claim research Qwen Plus via OpenRouter
External research Tavily Search API
Causal representation Mermaid syntax
Workflow orchestration n8n

The specific models are interchangeable. The important design choice is the separation of roles.


1. Input

For the proof of concept, each run contains:

{
  "excerpt_id": "E001",
  "excerpt": "..."
}