Recently, the team led by Associate Researcher Zuchao Li of the National Archives Administration Key Laboratory of Intelligent Development and Service of Archives and the School of Computer Science, Wuhan University, published its latest research in the field of document understanding, titled "Enhancing Visually-Rich Document Understanding via Layout Structure Modeling," at the ACM International Conference on Multimedia (ACM MM). ACM MM is the premier international event for academic and industrial exchange in the multimedia field, and the only Class A international academic conference in multimedia recommended by the China Computer Federation.

In recent years, Transformers with multimodal pre-training have made significant progress in visually-rich document understanding. However, existing models focus mainly on features such as text and vision, overlooking the importance of layout relationships between text nodes. In this paper, we propose GraphLayoutLM, a novel document understanding model that leverages layout structure graph modeling to inject document layout knowledge into the model. GraphLayoutLM uses a graph reordering algorithm to adjust text sequences based on the graph structure. In addition, our model employs a layout-aware multi-head self-attention layer to learn document layout knowledge. The model can understand the spatial arrangement of text elements, improving document understanding.

Figure 1. GraphLayoutLM Architecture

We evaluated our model on multiple benchmark datasets, including FUNSD, XFUND, and CORD, and achieved state-of-the-art results on these datasets. Experimental results show that our proposed method significantly improves upon existing methods and demonstrates the importance of incorporating layout information into document understanding models. We also conducted ablation experiments to study the contribution of each component of our model. The results show that the graph reordering algorithm and the layout-aware multi-head self-attention layer play a crucial role in achieving optimal performance.

Figure 2. Results on the FUNSD and CORD Test Sets

This research was supported by the National Natural Science Foundation of China, the Natural Science Foundation of Hubei Province, the Fundamental Research Funds for the Central Universities and other projects.