WHERE CLINICAL KNOWLEDGE LIVES IN EHR FOUNDATION MODELS

Electronic health records capture longitudinal patient histories — diagnoses (READ / ICD), prescriptions (ATC), lab results and clinical encounters — the richest real-world-evidence base in medicine. EHR foundation models pretrained on sequences of these clinical events learn dense vector representations of patients and clinical codes without any labelled data, mirroring the success of large language models in natural language processing.

These models produce two distinct levels of representation. Static token embeddings are fixed lookup vectors — one per clinical code — trained as model inputs via masked language modelling; they capture population-level co-occurrence statistics across millions of timelines. Contextual embeddings are transformer outputs that encode how each token is used within a specific patient context, refined through multi-head self-attention that composes clinical relationships dynamically.

Frozen foundation-model representations transfer effectively to downstream tasks: a lightweight MLP probe trained on fixed patient embeddings reaches strong performance with far fewer labels than supervised training from scratch. This work demonstrates label-free pretraining followed by linear evaluation across eight clinical onset and initiation tasks on 1M THIN UK primary-care timelines.

The model is pretrained on 1M patient timelines sampled from the THIN UK healthcare database (22.6M patients; ~2.5B longitudinal events in total). The vocabulary spans 58,800 clinical tokens across five code types, all sharing a single 256-d embedding space; tokenisation follows the CEHR-BERT scheme with Artificial Time Tokens.

• READ diagnoses — 51K codes, 20 chapters
• ATC drugs — 3.5K codes
• LAB results · HX history · MED molecules

An MLP probe (two hidden layers) is trained on top of the frozen BERT-MLM encoder to predict eight clinical outcomes: CKD onset, COPD onset, low eGFR, elevated HbA1c, heart failure, insulin initiation, statin initiation and T2D onset. Patient timelines are strictly truncated at the prediction time point before encoding, eliminating any risk of leakage from future events.

AUROC ranges from 0.877 to 0.973 (mean 0.947), showing that the self-supervised representations carry strong discriminative power for risk stratification. AUPRC values are intentionally modest (0.028–0.091) because all eight tasks involve rare events in real-world primary care. Critically, no clinical labels were used during pretraining — all predictive signal emerges from 1M timelines of routine EHR data.

To isolate the contribution of contextual processing, both embedding types are extracted for the same 20 drugs and evaluated independently on the drug–indication retrieval task. This separates the signal in the vocabulary's co-occurrence structure (static) from the compositional knowledge built by attention layers (contextual), enabling a direct quantitative comparison of what each representation level encodes about pharmacology.

For 20 drugs spanning antidiabetic, cardiovascular, respiratory and neurological areas, the top-5 nearest READ diagnosis codes are retrieved by cosine similarity and validated against Open Targets ground truth (Precision@5). The random baseline is P@5 = 1.7% — chance retrieval from the full 58,800-token vocabulary.

Static embeddings achieve P@5 = 30.0% (18× random): co-occurrence structure alone spontaneously organises the clinical vocabulary by pharmacological function, without any drug–indication labels. Performance varies — common drugs with strong signals (glyceryl trinitrate, levothyroxine) exceed 60%, while sparse or ambiguous ones score lower. Contextual processing raises the mean to P@5 = 39.0% (23× random) — a 30% relative gain — with the largest improvements where attention can compose multi-hop paths: drug → co-prescribed counterparts → shared diagnoses.

Comparing the single nearest READ diagnosis in the static vs contextual space reveals where attention adds pharmacological information beyond raw co-occurrence. Context corrects systematic errors: Metformin's static top-1 is upper respiratory infection — a co-prescription artefact of broad antibiotic use — while contextual processing correctly retrieves Type 2 Diabetes; Furosemide shifts from generic fluid retention to the more specific swollen legs.

Context also sharpens already-correct signals: GTN (angina), Omeprazole (oesophagitis) and Ramipril (hypertension) were ranked correctly statically and become stronger contextually. The mechanism: attention layers traverse drug → co-prescribed drugs → shared diagnoses, amplifying true indication signal while suppressing spurious co-occurrence.

Across the 20-drug set, contextual embeddings improve every retrieval metric over static, and both sit far above the random baseline — confirming at aggregate level that self-supervised co-occurrence learning, then attention-based composition, organises the clinical vocabulary by pharmacological function entirely without drug–indication labels.

A t-SNE projection of 58,810 contextual embeddings, coloured by READ chapter and vocabulary type, shows that drugs (ATC), diagnoses and labs from the same specialty form mixed-vocabulary neighbourhoods — without any cross-vocabulary supervision. This emergent organisation is exactly the geometry that makes drug–indication retrieval possible.