Overview

The Thread Encoder is CLM's compression engine for conversation-based content — support calls, chat transcripts, email threads, and any structured two-sided exchange. It acts as the umbrella component for all conversation encoding modes in the SDK.

Overview

Thread Encoder compresses conversation threads into compact semantic token sequences that LLMs can parse just as well as — often better than — the original text, at a fraction of the token cost.

Typical compression: 75–80% for customer service transcripts

Key capabilities:

• Conversation-level semantic extraction (intent, domain, actions, resolution)
• Ordered system/agent action chains
• Sentiment trajectory tracking
• PII-safe context representation (fact-of-information, not the data itself)
• Commitment and artifact extraction
• Redaction pattern support
• Four languages: EN, PT, ES, FR

Architecture

CLMEncoder.encode(input_=transcript, metadata={...})
        │
        ▼
  DataClassifier  ──→  DataTypes.TRANSCRIPT
        │
        ▼
  ThreadEncoder.encode(transcript=..., metadata=...)
        │
        ├──▶  TranscriptAnalyzer.analyze(...)
        │          │
        │          ├── Turn segmentation + spaCy NLP
        │          ├── Domain / service classification
        │          ├── Intent detection (customer utterances)
        │          ├── Agent action extraction
        │          ├── Sentiment trajectory
        │          ├── Promise / commitment detection
        │          ├── Artifact extraction (IDs, amounts, refs)
        │          └── Redacted field detection
        │
        └──▶  Token assembly → ThreadOutput

Core files:

Quick Start

from clm_core import CLMConfig, CLMEncoder

transcript = """
Agent: Thank you for calling support. This is Raj. How can I help you?
Customer: Hi, I was charged twice for my subscription this month.
Agent: I'm sorry to hear that. Let me look into your account...
Agent: I can see the duplicate charge. I'm processing a refund now.
         Your reference number is RFD-908712. You'll see it in 3-5 business days.
Customer: Thank you so much!
"""

cfg = CLMConfig(lang="en")
encoder = CLMEncoder(cfg=cfg)

result = encoder.encode(
    input_=transcript,
    metadata={
        "call_id": "CX-0001",
        "channel": "voice",
        "duration": "6m"
    }
)

print(result.compressed)
# [INTERACTION:SUPPORT:CHANNEL=VOICE] [DURATION=3m] [LANG=EN]
# [DOMAIN:BILLING] [SERVICE:SUBSCRIPTION]
# [CUSTOMER_INTENT:REPORT_DUPLICATE_CHARGE]
# [AGENT_ACTIONS:ACCOUNT_VERIFIED→REFUND_INITIATED]
# [RESOLUTION:REFUND_INITIATED] [STATE:PENDING_CUSTOMER]
# [COMMITMENT:REFUND_3-5_BUSINESS_DAYS]
# [ARTIFACT:REFUND_REF=RFD-908712]
# [SENTIMENT:NEUTRAL→GRATEFUL]

Encoding Modes

Thread Encoder supports two encoding modes, selected automatically via format detection or set explicitly with the transcript_format parameter:

The encoder detects the format automatically: if at least 50% of non-empty lines carry a recognized speaker prefix the input is treated as "turns", otherwise as "free_form". You can also override detection explicitly:

result = encoder.encode(input_=text, metadata={...}, transcript_format="free_form")

• See Transcript Encoder for the labeled-turn format reference.
• See Free-Form Encoder for emails, threads, and unstructured text.

ThreadOutput

ThreadEncoder.encode() returns a ThreadOutput object (a subclass of CLMOutput).

Compressed String

result.compressed   # "[INTERACTION:SUPPORT:CHANNEL=VOICE] [DOMAIN:BILLING] ..."
result.n_tokens     # Estimated input token count
result.c_tokens     # Estimated compressed token count
result.compression_ratio  # e.g. 88.3 (percent reduction)

Structured Dict — to_dict()

result.to_dict() parses the compressed token string into a typed Python dictionary. All recognised fields are present; tokens absent from the output have None as their value.

result_dict = result.to_dict()

Dict structure:

{
    # Identifiers (from metadata)
    "id": "CX-0001",
    "createdAt": None,

    # Call metadata
    "channel": "VOICE",         # VOICE | CHAT | EMAIL | SLACK
    "lang": "EN",
    "durationSeconds": 360,     # Converted from [DURATION=6m]

    # Classification
    "domain": "BILLING",
    "service": "SUBSCRIPTION",

    # Intent
    "customerIntent": "REPORT_DUPLICATE_CHARGE",
    "secondaryIntent": None,    # Set when [CUSTOMER_INTENTS:PRIMARY=...;SECONDARY=...]

    # What triggered the contact
    "supportTrigger": None,     # e.g. "FIELD_LOCKED", "MISSING_DELIVERY"

    # Context provided (PII-safe)
    "context": ["EMAIL_PROVIDED"],

    # Actions
    "agentActions": ["ACCOUNT_VERIFIED", "REFUND_INITIATED"],
    "systemActions": None,      # e.g. ["PAYMENT_RETRY_DETECTED"]

    # Outcome
    "resolution": "REFUND_INITIATED",
    "state": "PENDING_CUSTOMER",

    # Commitments with ETA days
    "commitments": [
        {"type": "REFUND", "etaDays": 4}
    ],

    # Artifacts (identifiers, amounts)
    "artifacts": [
        {"key": "REFUND_REF", "value": "RFD-908712"}
    ],

    # Sentiment trajectory
    "sentiment": ["NEUTRAL", "GRATEFUL"]
}

ETA day mapping for commitments:

Configuration

Thread Encoder behaviour is controlled via ThreadConfig, passed as thread_config inside CLMConfig.

from clm_core import CLMConfig
from clm_core.types import ThreadConfig

# Minimal — uses all defaults
cfg = CLMConfig(lang="en")

# With custom ThreadConfig
cfg = CLMConfig(
    lang="en",
    thread_config=ThreadConfig(
        detect_lang=True,
        include_ctx_values=True,
        estimate_thread_duration=True,
        include_summary=True,
        custom_summary_template=None,  # Uses built-in template when None
        redaction_pattern=r"\[.*?REDACTED.*?\]",
    )
)

ThreadConfig parameters

CLMConfig.lang controls the spaCy model and language dictionary used by the encoder. All other analysis decisions (what to extract, what to drop) are governed by the language dictionary and the internal NLP pipeline.

Redaction Support

When a transcript contains redacted PII (e.g. [*REDACTED*], ***, [PHONE_NUMBER]), Thread Encoder detects the surrounding context and emits CONTEXT:FIELD_REDACTED tokens instead of silently dropping the information.

from clm_core.types import ThreadConfig

# Default pattern covers common redaction styles
cfg = CLMConfig(lang="en")
# Matches: [*REDACTED*], [REDACTED], ***, <redacted>, XXX, [PII]

# Custom pattern for your own redaction format
cfg = CLMConfig(
    lang="en",
    thread_config=ThreadConfig(redaction_pattern=r"\[.*?REDACTED.*?\]")
)

Output example:

[CONTEXT:PHONE_REDACTED]
[CONTEXT:EMAIL_REDACTED]

Summary Templates

When include_summary=True, CLM generates a natural-language summary from the compressed token output without an additional LLM call. A custom Jinja2 template can be provided via custom_summary_template; when omitted, the built-in template is used.

Available template variables:

Example with a custom template:

from clm_core.types import ThreadConfig

template = """
Support {{ CHANNEL | lower }} – {{ DOMAIN | lower }}: {{ CUSTOMER_INTENT | lower }}.
Outcome: {{ RESOLUTION | lower }}.
{% if COMMITMENT %}Next step: {{ COMMITMENT | lower }}.{% endif %}
""".strip()

cfg = CLMConfig(
    lang="en",
    thread_config=ThreadConfig(
        include_summary=True,
        custom_summary_template=template,
    )
)

Language Support

Each language ships its own dictionary (clm_core/dictionary/{lang}/) with:

• Speaker label detection
• Action vocabulary
• Intent patterns
• Commitment and timeline patterns
• Sentiment keywords

Data Models

All models are exported from clm_core.components.thread_encoder:

from clm_core.components.thread_encoder import (
    TranscriptAnalysis,  # Complete analysis result
    ThreadOutput,  # Compressed output (returned by encode)
    CallInfo,  # Call/session metadata
    Turn,  # Single conversation turn
    Issue,  # Customer issue
    Action,  # Agent action
    Resolution,  # How the conversation resolved
    SentimentTrajectory,  # Sentiment across the call
    ResolutionState,  # Granular resolution state
    RefundReference,  # Billing/refund case details
    PromiseCommitment,  # Agent commitments
    MonetaryAmount,  # Extracted monetary values
    ConversationTimeline,  # Timeline of key events
    TimelineEvent,  # Single timeline event
    TemporalPattern,  # Extracted temporal expressions
)

Key model: TranscriptAnalysis

Holds the full structured result of the NLP pipeline, accessible via encoder._ts_encoder.analysis after a call to encode():

result = encoder.encode(input_=transcript, metadata=metadata)

# Access the raw analysis
analysis = encoder._ts_encoder.analysis

print(analysis.domain)           # "BILLING"
print(analysis.service)          # "SUBSCRIPTION"
print(analysis.customer_intent)  # "REPORT_DUPLICATE_CHARGE"
print(analysis.secondary_intent) # None

for action in analysis.actions:
    print(f"{action.type}: {action.result}")

for promise in analysis.promises:
    print(f"{promise.type} → {promise.timeline} (conf: {promise.confidence})")

if analysis.refund_reference:
    print(analysis.refund_reference.reference_number)
    print(analysis.refund_reference.amount)

print(analysis.resolution_state.type)
print(analysis.resolution_state.customer_satisfaction)

Next Steps

• Transcript Encoder — Complete reference: token schema, examples, use cases, best practices
• Free-Form Encoder — Encoding emails, Slack threads, and unstructured prose
• Advanced: Token Hierarchy — Token structure deep-dive
• Advanced: CLM Dictionary — Language-specific vocabularies
• CLM Output — CLMOutput / ThreadOutput reference