| 面向预训练语言模型的动态Schema 表示压缩方法 |
| 点此下载全文 |
| 引用本文：李云飞,谭金菁,谭文安.面向预训练语言模型的动态Schema 表示压缩方法[J].上海第二工业大学(中文版),2026,43(1):64-72 |
| 摘要点击次数: 267 |
| 全文下载次数: 11 |
|
| 基金项目:国家自然科学基金项目 (61672022, U1904186), 上海市教委青年教师培养项目 (A60GY24C002) 资助 |
|
| 中文摘要:在结构化自然语言任务中, 输入中的 Schema 信息至关重要。现有方法多采用静态拼接或固定嵌入方式表示 Schema, 易造成信息冗余和计算开销过大, 难以适应多任务或跨域场景。为此, 本文提出一种动态 Schema表示压缩模块 (dynamic Schema representation compression module, DSRCM)。该模块基于上下文信息实现 Schema的动态选择与压缩, 通过上下文感知打分、Top-K 稀疏选择与多头融合注意力机制, 在保持语义完整性的同时有效降低表示维度, 从而实现性能与效率的双重优化。DSRCM 具有轻量化与可插拔特性, 可无缝集成于BERT、RoBERTa、TinyBERT 等多种预训练语言模型中。实验证明, 在文本分类与关系抽取任务中, DSRCM 平均实现约 3.5 倍 Schema 压缩、推理速度提升 20%, 且无性能损失, 展现出良好的跨域通用性与应用潜力。 |
| 中文关键词:预训练语言模型 Schema 表示 模型压缩 动态表示 信息抽取 |
| |
| Dynamic Schema Representation Compression Method for Pre-Trained Language Models |
|
|
| Abstract:In structured natural language tasks, Schema information in the input is crucial. Existing methods often use static concatenation or fixed embedding to represent Schema, which can lead to information redundancy and excessive computational overhead, making them difficult to adapt to multi-task or cross-domain scenarios. To address this, the paper proposes a dynamic Schema representation compression module (DSRCM). This module dynamically selects and compresses Schemas based on contextual information. Through context-aware scoring, Top-K sparse selection, and a multi-head fusion attention mechanism, it effectively reduces the representation dimensionality while maintaining semantic integrity, thereby achieving both performance and efficiency optimization. The DSRCM is
lightweight and pluggable, allowing for flexible integration into various pre-trained language models, such as BERT, RoBERTa, and TinyBERT. Experiments demonstrate that in text classification and relation extraction tasks, the DSRCM achieves an average Schema compression of approximately 3.5× and a 20% improvement in inference speed, without any performance loss, demonstrating excellent cross-domain versatility and application potential. |
| keywords:pre-trained language model Schema representation model compression dynamic representation information extraction |
| 查看全文 查看/发表评论 下载PDF阅读器 |
|
|
|