Background
Resource-limited environments require fewer trainable parameters while preserving adaptability to complex tasks. LoRA-MoE divides work among low-rank experts and uses routing to select experts for each input.
Model Design
The study compares LoRA-MoE with QLoRA-MoE and single-expert with four-expert configurations. Alongside mathematical and commonsense reasoning, it integrates AutoTimes to evaluate long-term forecasting on ETTh1, ECL, Traffic and Solar.
Parameter-efficient
Uses LoRA to reduce the number of model parameters requiring updates.
Expert Routing
Low-rank experts specialize according to input features.
Cross-domain Evaluation
Evaluates reasoning tasks and long-term time-series forecasting.
Results
Four-expert LoRA-MoE records improvements of 18.11% on AQuA and 13.97% on MathQA. The study also observes that quantization errors may destabilize MoE routing, informing subsequent model design.
research architecture
