About
I am an AI researcher working on efficient neural network architectures and training methods. I am currently a Member of Technical Staff at Abridge. Previously, I was a Research Scientist at Meta, where I worked on developing on-device LLMs and was technical lead on pretraining text encoder foundation models for recommendation and retrieval systems.
My research has explored ways to leverage structure and invariances in data to build more efficient and effective machine learning models: for instance, by exploiting the structure of language with Tree-LSTMs, the various symmetries in images with Equivariant Transformers, and the underlying physical constraints of the Earth for ML-driven earthquake detection methods.
I hold a Ph.D. in Computer Science from Stanford University, where I was advised by Peter Bailis and Gregory Valiant. While at Stanford, I was a member of the Future Data Systems group and the DAWN project. During my time as an MS student, I was also affiliated with the Stanford NLP Group.
News
- Sep 2026 Joined Abridge as a Member of Technical Staff.
- Sep 2026 The Path Not Taken: RLVR Provably Learns Off the Principals was accepted to NeurIPS 2026.
Selected Publications
Full list on Google Scholar.
-
The Path Not Taken: RLVR Provably Learns Off the Principals
NeurIPS 2026
Why reinforcement learning on reasoning tasks changes so few of a model’s weights and how its updates differ from supervised fine-tuning.
Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR. Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.
@inproceedings{zhu2026path, title = {The Path Not Taken: {RLVR} Provably Learns Off the Principals}, author = {Zhu, Hanqing and Zhang, Zhenyu and Huang, Hanxian and Su, DiJia and Liu, Zechun and Zhao, Jiawei and Fedorov, Igor and Pirsiavash, Hamed and Sha, Zhizhou and Lee, Jinwon and Pan, David Z. and Wang, Zhangyang and Tian, Yuandong and Tai, Kai Sheng}, booktitle = {Advances in Neural Information Processing Systems}, year = {2026} } -
MobileLLM-Pro Technical Report
arXiv 2025
A 1B-parameter language model built to run on phones and wearables with a 128K-token context window and 4-bit quantization.
Efficient on-device language models around 1 billion parameters are essential for powering low-latency AI applications on mobile and wearable devices. However, achieving strong performance in this model class, while supporting long context windows and practical deployment remains a significant challenge. We introduce MobileLLM-Pro, a 1-billion-parameter language model optimized for on-device deployment. MobileLLM-Pro achieves state-of-the-art results across 11 standard benchmarks, significantly outperforming both Gemma 3-1B and Llama 3.2-1B, while supporting context windows of up to 128,000 tokens and showing only minor performance regressions at 4-bit quantization. These improvements are enabled by four core innovations: (1) implicit positional distillation, a novel technique that effectively instills long-context capabilities through knowledge distillation; (2) a specialist model merging framework that fuses multiple domain experts into a compact model without parameter growth; (3) simulation-driven data mixing using utility estimation; and (4) 4-bit quantization-aware training with self-distillation. We release our model weights and code to support future research in efficient on-device language models.
@article{huber2025mobilellmpro, title = {{MobileLLM-Pro} Technical Report}, author = {Huber, Patrick and Chang, Ernie and Wen, Wei and Fedorov, Igor and Elgamal, Tarek and Huang, Hanxian and Suda, Naveen and Sankar, Chinnadhurai and Vogeti, Vish and Wang, Yanghan and Gladkov, Alex and Tai, Kai Sheng and Elogeel, Abdelrahman and Hefny, Tarek and Chandra, Vikas and Aly, Ahmed and Kumar, Anuj and Krishnamoorthi, Raghuraman and Sagar, Adithya}, journal = {arXiv preprint arXiv:2511.06719}, year = {2025} } -
uCAP: An Unsupervised Prompting Method for Vision-Language Models
ECCV 2024Oral
Learns the prompt for zero-shot image classification with vision-language models from unlabeled images instead of tuning it by hand.
This paper addresses a significant limitation that prevents Contrastive Language-Image Pretrained Models (CLIP) from achieving optimal performance on downstream image classification tasks. The key problem with CLIP-style zero-shot classification is that it requires domain-specific context in the form of prompts to better align the class descriptions to the downstream data distribution. In particular, prompts for vision-language models are domain-level texts (e.g., “a centered satellite image of ...”) which, together with the class names, are fed into the text encoder to provide more context for the downstream dataset. These prompts are typically manually tuned, which is time consuming and often sub-optimal. To overcome this bottleneck, this paper proposes uCAP, a method to automatically learn domain-specific prompts/contexts using only unlabeled in-domain images. We achieve this by modeling the generation of images given the class names and a domain-specific prompt with an unsupervised likelihood distribution, and then performing inference of the prompts. We validate the proposed method across various models and datasets, showing that uCAP consistently outperforms manually tuned prompts and related baselines on the evaluated datasets: ImageNet, CIFAR-10, CIFAR-100, OxfordPets (up to 2%), SUN397 (up to 5%), and Caltech101 (up to 3%).
@inproceedings{nguyen2024ucap, title = {{uCAP}: An Unsupervised Prompting Method for Vision-Language Models}, author = {Nguyen, A. Tuan and Tai, Kai Sheng and Chen, Bor-Chun and Shukla, Satya Narayan and Yu, Hanchao and Torr, Philip and Tian, Tai-Peng and Lim, Ser-Nam}, booktitle = {European Conference on Computer Vision (ECCV)}, year = {2024} } -
Spartan: Differentiable Sparsity via Regularized Transportation
NeurIPS 2022
Trains neural networks to a target sparsity by gradually sharpening a soft, differentiable choice of which weights to keep.
We present Spartan, a method for training sparse neural network models with a predetermined level of sparsity. Spartan is based on a combination of two techniques: (1) soft top-k masking of low-magnitude parameters via a regularized optimal transportation problem and (2) dual averaging-based parameter updates with hard sparsification in the forward pass. This scheme realizes an exploration-exploitation tradeoff: early in training, the learner is able to explore various sparsity patterns, and as the soft top-k approximation is gradually sharpened over the course of training, the balance shifts towards parameter optimization with respect to a fixed sparsity mask. Spartan is sufficiently flexible to accommodate a variety of sparsity allocation policies, including both unstructured and block structured sparsity, as well as general cost-sensitive sparsity allocation mediated by linear models of per-parameter costs. On ImageNet-1K classification, Spartan yields 95% sparse ResNet-50 models and 90% block sparse ViT-B/16 models while incurring absolute top-1 accuracy losses of less than 1% compared to fully dense training.
@inproceedings{tai2022spartan, title = {Spartan: Differentiable Sparsity via Regularized Transportation}, author = {Tai, Kai Sheng and Tian, Taipeng and Lim, Ser-Nam}, booktitle = {Advances in Neural Information Processing Systems}, year = {2022} } -
An End-to-End Earthquake Detection Method for Joint Phase Picking and Association using Deep Learning
* Equal contribution
Journal of Geophysical Research: Solid Earth, 2022
A single neural network that detects earthquakes and picks seismic phases jointly across an entire network of stations.
Earthquake monitoring by seismic networks typically involves a workflow consisting of phase detection/picking, association, and location tasks. In recent years, the accuracy of these individual stages has been improved through the use of machine learning techniques. In this study, we introduce a new, end-to-end approach that improves overall earthquake detection accuracy by jointly optimizing each stage of the detection pipeline. We propose a neural network architecture for the task of multi-station processing of seismic waveforms recorded over a seismic network. This end-to-end architecture consists of three sub-networks: a backbone network that extracts features from raw waveforms, a phase picking sub-network that picks P- and S-wave arrivals based on these features, and an event detection sub-network that aggregates the features from multiple stations and detects earthquakes. We use these sub-networks in conjunction with a shift-and-stack module based on back-projection that introduces kinematic constraints on arrival times, allowing the model to generalize to different velocity models and to variable station geometry in seismic networks. We evaluate our proposed method on the STanford EArthquake Dataset (STEAD) and on the 2019 Ridgecrest, CA earthquake sequence. The results demonstrate that our end-to-end approach can effectively pick P- and S-wave arrivals and achieve earthquake detection accuracy rivaling that of other state-of-the-art approaches.
@article{zhu2022endtoend, title = {An End-to-End Earthquake Detection Method for Joint Phase Picking and Association Using Deep Learning}, author = {Zhu, Weiqiang and Tai, Kai Sheng and Mousavi, S. Mostafa and Bailis, Peter and Beroza, Gregory C.}, journal = {Journal of Geophysical Research: Solid Earth}, volume = {127}, number = {3}, pages = {e2021JB023283}, year = {2022}, doi = {10.1029/2021JB023283} } -
Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training
ICML 2021
Semi-supervised learning that treats assigning pseudo-labels to unlabeled examples as an optimal transport problem.
Self-training is a standard approach to semi-supervised learning where the learner's own predictions on unlabeled data are used as supervision during training. In this paper, we reinterpret this label assignment process as an optimal transportation problem between examples and classes, wherein the cost of assigning an example to a class is mediated by the current predictions of the classifier. This formulation facilitates a practical annealing strategy for label assignment and allows for the inclusion of prior knowledge on class proportions via flexible upper bound constraints. The solutions to these assignment problems can be efficiently approximated using Sinkhorn iteration, thus enabling their use in the inner loop of standard stochastic optimization algorithms. We demonstrate the effectiveness of our algorithm on the CIFAR-10, CIFAR-100, and SVHN datasets in comparison with FixMatch, a state-of-the-art self-training algorithm. Our code is available at https://github.com/stanford-futuredata/sinkhorn-label-allocation.
@inproceedings{tai2021sinkhorn, title = {Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training}, author = {Tai, Kai Sheng and Bailis, Peter and Valiant, Gregory}, booktitle = {International Conference on Machine Learning (ICML)}, year = {2021} } -
Equivariant Transformer Networks
ICML 2019
A drop-in layer that undoes rotation, scaling, shear and perspective changes in an image before it is classified.
How can prior knowledge on the transformation invariances of a domain be incorporated into the architecture of a neural network? We propose Equivariant Transformers (ETs), a family of differentiable image-to-image mappings that improve the robustness of models towards pre-defined continuous transformation groups. Through the use of specially-derived canonical coordinate systems, ETs incorporate functions that are equivariant by construction with respect to these transformations. We show empirically that ETs can be flexibly composed to improve model robustness towards more complicated transformation groups in several parameters. On a real-world image classification task, ETs improve the sample efficiency of ResNet classifiers, achieving relative improvements in error rate of up to 15% in the limited data regime while increasing model parameter count by less than 1%.
@inproceedings{tai2019equivariant, title = {Equivariant Transformer Networks}, author = {Tai, Kai Sheng and Bailis, Peter and Valiant, Gregory}, booktitle = {International Conference on Machine Learning (ICML)}, year = {2019} } -
Compressed Factorization: Fast and Accurate Low-Rank Factorization of Compressively-Sensed Data
* Equal contribution
ICML 2019
Shows that low-rank factorizations like NMF can be computed directly on randomly compressed data and still recover the original factors.
What learning algorithms can be run directly on compressively-sensed data? In this work, we consider the question of accurately and efficiently computing low-rank matrix or tensor factorizations given data compressed via random projections. We examine the approach of first performing factorization in the compressed domain, and then reconstructing the original high-dimensional factors from the recovered (compressed) factors. In both the matrix and tensor settings, we establish conditions under which this natural approach will provably recover the original factors. While it is well-known that random projections preserve a number of geometric properties of a dataset, our work can be viewed as showing that they can also preserve certain solutions of non-convex, NP-Hard problems like non-negative matrix factorization. We support these theoretical results with experiments on synthetic data and demonstrate the practical applicability of compressed factorization on real-world gene expression and EEG time series datasets.
@inproceedings{sharan2019compressed, title = {Compressed Factorization: Fast and Accurate Low-Rank Factorization of Compressively-Sensed Data}, author = {Sharan, Vatsal and Tai, Kai Sheng and Bailis, Peter and Valiant, Gregory}, booktitle = {International Conference on Machine Learning (ICML)}, year = {2019} } -
Moment-Based Quantile Sketches for Efficient High Cardinality Aggregation Queries
VLDB 2018
A 200-byte quantile summary that merges in nanoseconds to speed up percentile queries across many slices of a dataset.
Interactive analytics increasingly involves querying for quantiles over sub-populations of high cardinality datasets. Data processing engines such as Druid and Spark use mergeable summaries to estimate quantiles, but summary merge times can be a bottleneck during aggregation. We show how a compact and efficiently mergeable quantile sketch can support aggregation workloads. This data structure, which we refer to as the moments sketch, operates with a small memory footprint (200 bytes) and computationally efficient (50ns) merges by tracking only a set of summary statistics, notably the sample moments. We demonstrate how we can efficiently and practically estimate quantiles using the method of moments and the maximum entropy principle, and show how the use of a cascade further improves query time for threshold predicates. Empirical evaluation on real-world datasets shows that the moments sketch can achieve less than 1 percent error with 15 times less merge overhead than comparable summaries, improving end query time in the MacroBase engine by up to 7 times and the Druid engine by up to 60 times.
@article{gan2018moment, title = {Moment-Based Quantile Sketches for Efficient High Cardinality Aggregation Queries}, author = {Gan, Edward and Ding, Jialin and Tai, Kai Sheng and Sharan, Vatsal and Bailis, Peter}, journal = {Proceedings of the VLDB Endowment}, volume = {11}, number = {11}, pages = {1647--1660}, year = {2018} } -
Sketching Linear Classifiers over Data Streams
SIGMOD 2018
A compact sketch that learns a linear classifier over a data stream while keeping track of its most important features.
code extended abstract long version slides
We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model. This enables memory-limited execution of several statistical analyses over streams, including online feature selection, streaming data explanation, relative deltoid detection, and streaming estimation of pointwise mutual information. Unlike related sketches that capture the most frequently-occurring features (or items) in a data stream, the Weight-Median Sketch captures the features that are most discriminative of one stream (or class) compared to another. The Weight-Median Sketch adopts the core data structure used in the Count-Sketch, but, instead of sketching counts, it captures sketched gradient updates to the model parameters. We provide a theoretical analysis that establishes recovery guarantees for batch and online learning, and demonstrate empirical improvements in memory-accuracy trade-offs over alternative memory-budgeted methods, including count-based sketches and feature hashing.
@inproceedings{tai2018sketching, title = {Sketching Linear Classifiers over Data Streams}, author = {Tai, Kai Sheng and Sharan, Vatsal and Bailis, Peter and Valiant, Gregory}, booktitle = {Proceedings of the 2018 International Conference on Management of Data (SIGMOD)}, pages = {757--772}, year = {2018}, doi = {10.1145/3183713.3196930} } -
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks
ACL 2015
Extends LSTMs from word sequences to parse trees for better sentence similarity and sentiment classification.
Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).
@inproceedings{tai2015improved, title = {Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks}, author = {Tai, Kai Sheng and Socher, Richard and Manning, Christopher D.}, booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, pages = {1556--1566}, year = {2015}, doi = {10.3115/v1/P15-1150} }
Open Source Projects
- MobileLLM-Pro (2025): A 1B parameter LLM developed at Meta for efficient on-device inference.
- index-baselines (2017): A library for comparing learned index structures to classical data structures like cuckoo hash tables.
- neuralart (2016): An early reimplementation of the paper ‘A Neural Algorithm of Artistic Style’ by Gatys et al.
- torch-ntm (2015): A Neural Turing Machine implementation using LuaTorch.