Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

TL;DR. A public method called Model DNA — with a live tool — lets outsiders estimate whether an LLM was trained from scratch or derived from an open-weight base, using nothing but public artifacts: config.json, tokenizer.json, and embedding weights. This is a technical deep-dive that cites and reproduces that method in PyTorch, then analyzes where it holds up and where it breaks. Three public signals — architecture config, tokenizer overlap, and embedding-space similarity via Linear CKA — combine to place a model on a lineage spectrum. One matching field is noise; five at once is a fingerprint. Provenance is a preponderance-of-evidence judgment, not a single test. The method's real strengths are reproducibility and rotation-invariant similarity; its real limits are the continued-pretraining gray zone, threshold sensitivity, and an embedding-only view. Fingerprinting reveals lineage, not intent. Building on an open-weight base is a legitimate, industry-standard practice; the output is a label, not an accusation. Every few weeks a lab announces a "from-scratch, self-developed" foundation model. In mid-2026 those claims stopped being taken on faith. A Zhihu roundtable on the summer model wave and a thread with millions of views became the venue where "self-developed" claims were publicly stress-tested — and several were found more derivative than advertised (coverage). Model DNA matters because it moved the argument from vibes to a reproducible procedure, and it has been run across major Korean foundation-model builders — among them LG, NAVER, Kakao, SKT, KT, NCSOFT, Upstage, and Motif. (This piece stays at the method level and assigns no verdict to any named company; per-model labels belong to the tool, not to a blog post.) What follows reproduces the procedure in PyTorch and evaluates it as a method, within the scope the source already made public. The premise: no leaked internals required. Everything is read from a model's public files. config.json Compare the structural fields a lab chooses at design time: model_type, vocab_size, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads. Independently designed models rarely align on all of them at once. import json ARCH_FIELDS = [ "model_type", "vocab_size", "hidden_size", "intermediate_size", "num_hidden_layers", "num_attention_heads", "num_key_value_heads", "max_position_embeddings", "rope_theta", ] def arch_match_count(cfg_a: dict, cfg_b: dict, fields=ARCH_FIELDS) -> int: """Number of structural fields that match simultaneously.""" return sum(1 for k in fields if cfg_a.get(k) is not None and cfg_a.get(k) == cfg_b.get(k)) Reading rule (as the source puts it): a single coincidental field means nothing; five simultaneously is a fingerprint. How many to treat as a threshold depends on the diversity of your candidate base pool. tokenizer.json Two models trained truly independently almost never converge on the same vocabulary. Normalize shared tokens against the smaller vocabulary. def tokenizer_overlap(vocab_a: dict, vocab_b: dict) -> float: sa, sb = set(vocab_a), set(vocab_b) return len(sa & sb) / min(len(sa), len(sb)) A supporting signal only — see Trap 2. The most robust signal compares representation geometry. Naive cosine comparison is fooled by rotation (Trap 1), so the method uses Linear CKA (Centered Kernel Alignment) — from Kornblith et al. (2019), Similarity of Neural Network Representations Revisited (ICML) — which is invariant to rotation, orthogonal transforms, and isotropic scaling. For row-centered matrices X ∈ ℝ^{n×d1} and Y ∈ ℝ^{n×d2}: CKA(X, Y) = ||Yᵀ X||²_F / ( ||Xᵀ X||_F · ||Yᵀ Y||_F ) Crucially, it is defined even when d1 ≠ d2, so models with different hidden sizes compare directly. import torch @torch.no_grad() def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float: # X:(n,d1), Y:(n,d2) — embeddings over the SAME token set (rows aligned) X = X - X.mean(0, keepdim=True) Y = Y - Y.mean(0, keepdim=True) num = ((Y.t() @ X) ** 2).sum() den = torch.sqrt(((X.t() @ X) ** 2).sum() * ((Y.t() @ Y) ** 2).sum()) return (num / den).clamp(0, 1).item() Alignment is the catch. The two embedding matrices must index the same tokens. In practice you take the shared-token subset of the two tokenizers and gather those rows: def aligned_embeddings(emb_a, vocab_a, emb_b, vocab_b): shared = sorted(set(vocab_a) & set(vocab_b)) idx_a = torch.tensor([vocab_a[t] for t in shared]) idx_b = torch.tensor([vocab_b[t] for t in shared]) return emb_a[idx_a], emb_b[idx_b] The tool collapses the three signals into four labels — a clean way to read any result (Model Genome Korea): Genotype Meaning 🟢 Native Self-designed architecture and from-scratch weights 🔵 Adapted Mostly original, one borrowed axis 🟡 Mixed Partial inheritance on both axes 🔴 Ported Exact foreign architecture match and inherited weights Trap 1 — row-wise cosine similarity looks rigorous but isn't. It is fooled by rotation invariance: a genuinely derived model can be rotated to look "different," and a naive check clears it. That is exactly why CKA and config signals carry the weight. Trap 2 — a shared tokenizer proves nothing alone. Tokenizer reuse is often a licensing or convenience decision. Treat overlap as supporting evidence, never a conclusion. Strengths Reproducibility. All three signals compute from public artifacts in a few dozen lines. Claim and verification live on the same plane. Right invariance. Choosing Linear CKA is correct — it neutralizes the most common disguise (orthogonal transforms) that defeats cosine comparisons. Evidence fusion. Judging on the simultaneous agreement of three axes suppresses both false positives and false negatives. Limits (must be acknowledged) Continued-pretraining gray zone. Embedding CKA identifies from-scratch training well but does not cleanly separate derivatives that keep a base's weights and train heavily on top. Here the verdict is probabilistic and config/tokenizer evidence dominates. Threshold sensitivity. "How many fields," "what CKA cutoff" depend on the candidate pool. Hard-coding constants makes conclusions wobble when the pool changes — which is why this write-up prescribes none. Embedding bias. Looking only at the embedding layer is cheap, but a model's "identity" also lives in mid and upper layers. A layer-wise CKA profile improves resolution in the gray zone. Alignment dependence. Few shared tokens (language- or domain-specific tokenizers) shrink the CKA sample and inflate variance. Report shared-token count alongside CKA. Improvements worth adopting Extend single-layer embedding CKA to a layer-wise CKA curve (input → mid → output). Score against the entire candidate base pool and judge by relative rank, not an absolute cutoff. Report shared-token counts and bootstrap confidence intervals for statistical significance. In short, Model DNA fuses the right signals under the right invariance — a solid starting point. It only avoids misjudgment when read as a spectrum with uncertainty, not a from-scratch/not binary. Lineage, not intent. It can show B shares structure with A; it cannot say whether that was disclosed, licensed, or hidden — ethics and paperwork, not linear algebra. "From scratch" is a spectrum, not a boolean. Data, init, architecture, and post-training each sit on a continuum of originality. Building on open weights is legitimate. The goal is transparency and accurate labeling, not accusation. Can you tell if an LLM was really trained from scratch? What is Model DNA / model provenance? Is building on Llama, Qwen, or DeepSeek legitimate? How do you tell a fine-tuned model from a from-scratch one? Why Linear CKA instead of cosine? Can models with different hidden sizes be compared? d1 ≠ d2 is fine — as long as embedding rows are aligned to the same tokens. Method under analysis: Model DNA — architecture & weight lineage Live tool (space): Model Genome Korea Announcement post: Architecture lineage of Korea's sovereign-AI models Academic basis: Kornblith, Norouzi, Lee, Hinton (2019), Similarity of Neural Network Representations Revisited, ICML. The debate: Zhihu roundtable "大模型卷一夏" · Zhihu thread · coverage Provenance is becoming a norm, not a gotcha. The healthiest version is one where "we trained it from scratch" arrives with — or at least survives — the fingerprint. If you build models, publish the check yourself.