Publications
Transformers as Provable Approximators of Sparse Principal Component Analysis
Qinyan Liu, Yan Chen, and Canhong Wen
STAI-X 2026 Strong Accept (with top review scores)
Paper (pdf) | Code | OpenReview | Poster
- Proved that multi-head Transformers can implement Sparse Principal Component Analysis (SPCA) under noise, extending existing theoretical work on Transformer approximation to the** fully unsupervised** setting.
- Designed parameter constructions aligned with iterative Truncated Power updates, and derived guarantees by decomposing recovery error into algorithmic approximation, statistical estimation, and generalization components.
- Conducted experiments on synthetic and high-dimensional gene-expression data, achieving competitiveperformance against classical baselines, with extensive ablations across model configurations and noise levels.
- Demonstrated efficient batched inference after training, highlighting amortized Transformer-based SPCA inference.
Projects
Leveraging Zero-Inflated Data via Double Machine Learning
Advisor: Prof. Kan Xu, W. P. Carey School of Business, Arizona State University
- Studied treatment effect estimation in real-world online marketing data with zero-inflated and censored demand.
- Implemented preliminary Double Machine Learning and doubly robust estimators to study sparse click outcomes.
- Investigated zero-inflated models and semiparametric methods for treatment effect estimation.