Preprint adapts language models to read graph structure directly

Researchers report that their Graph Transformer Language Model lets a pretrained language model process relationships between graph nodes directly, rather than compressing each node's text before passing it to a separate graph model. The preprint adds graph-aware attention biases with structure-related parameters equal to 0.015% of the base model. In a test requiring retrieval of information within a graph, accuracy stayed flat from 1k to 64k tokens and 4x beyond the training length, while an identically trained flat-text baseline collapsed. Training updates the structural parameters and a small adapter; the model reduces exactly to the pretrained model when no graph is present.