Hi, I'm Tyler Romero.

Tyler Romero I am a Member of Technical Staff on the MAI Superintelligence Team, where I work on post-training. Previously, I was a Research Engineer at the Allen Institute (Ai2), where I worked on pretraining research for open language models.

Before that, I led ML at Groundlight and built large-scale recommender systems at Twitter, some of which is now open source. I studied CS and ML at Stanford and computer engineering at Texas A&M.

Posts

Publications & Preprints

  • Oct 2026
    Frontier Pre-training at Microsoft AI — The Microsoft AI Team.
    Microsoft AI technical report.

    We describe pre-training MAI-Base-2, an MoE foundation model with 80B active and 2.8T total parameters, trained on 16k GB200 GPUs. MAI-Base-2 achieves lower held-out bits-per-byte than every open-weight model we tested and is 4–12× more pre-training FLOPs-efficient than DeepSeek-V4-Pro-Base (depending on eval domain). Key ingredients include scaling ladders with source scaling for repeated data, which accurately forecast the final model's performance, along with model-infrastructure co-design and expanded data pipelines.

  • Oct 2026
    Supercharging Olmo-core for Efficient and Scalable MoE Training MoE transformer block: a router dispatches each token to 2 of 4 routed experts plus an always-active shared expert, whose outputs are combined — Tianhua Tao*, Akshita Bhagia, Dirk Groeneveld, Pete Walsh, Tyler Romero, Yashas Samaga, Jacob Morrison, Iz Beltagy, Taira Anderson, Noah A. Smith, Hannaneh Hajishirzi.
    Allen Institute for AI technical report.

    MoE training costs scale with total parameter capacity, not just active compute, so infrastructure built for dense models loses throughput as experts are added. We redesign the Olmo-core training stack around DDP with expert and pipeline parallelism, adding NVSHMEM-based rowwise expert parallelism, synchronization-free routing with device-scheduled grouped GEMMs, and integrated MXFP8. On B300 nodes, the stack trains models from 12.9B to 1.2T total parameters on up to 512 GPUs, reaching 858 TFLOP/s/GPU at 1.2T. We also identify Token Gerrymandering, where the MoE router learns to hack the load-balancing loss.

  • Jun 2026
    MAI-Thinking-1 Watercolor circles arranged as a thought bubble for MAI-Thinking-1 — The Microsoft AI Team.
    Microsoft AI technical report.

    MAI-Thinking-1 is a 35B active / 1T total parameter MoE reasoning model trained from scratch (without distillation from third-party models). We present a scaling-focused pre-training framework paired with a robust RL recipe and infrastructure that sustains long, log-linear improvement. It is competitive with frontier models of similar size on STEM reasoning and coding.

  • Mar 2026
    Olmo Hybrid Olmo Hybrid expressivity Venn diagram — William Merrill*, Yanhong Li*, Tyler Romero*, Anej Svete*, Caia Costello*, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, Chuan Li, Kyle Lo, Saumya Malik, DJ Matusz, Benjamin Minixhofer, Jacob Morrison, Luca Soldaini, Finbarr Timbers, Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi, Ashish Sabharwal*.
    Conference on Language Modeling (COLM) 2026.

    We provide evidence that hybrid models, which mix attention and linear RNN layers, improve on pure transformers on several fronts. Theoretically, hybrids can express tasks beyond both transformers and linear RNNs individually; empirically, they scale more efficiently. We train Olmo Hybrid, a 7B model that replaces sliding-window layers with Gated DeltaNet layers, and show it outperforms Olmo 3 on standard pretraining and long-context evaluations.

  • Nov 2025
    Olmo 3 Olmo 3 32B training loss curve — Team Olmo (incl. Tyler Romero, core contributor).
    arXiv preprint arXiv:2512.13961.

    We introduce Olmo 3, a family of state-of-the-art, fully-open language models at the 7B and 32B parameter scales. Olmo 3 model construction targets long-context reasoning, function calling, coding, instruction following, general chat, and knowledge recall. This release includes the entire model flow, i.e., the full lifecycle of the family of models, including every stage, checkpoint, data point, and dependency used to build it. Our flagship model, Olmo 3 32B Think, is the strongest fully-open thinking model released to-date.

Projects & Open Source

  • greyhound

    I've been using greyhound as a place to build and experiment with CuTe DSL kernels for PyTorch, with an eye toward making training and inference workloads faster and easier to hack on. So far it includes kernels for cross-entropy and chunked linear losses, selective log-softmax, and causal Conv1D; see the greyhound documentation and benchmarks.

  • olmo-core

    Building blocks for OLMo modeling and distributed training. I did substantial work in OLMo-core on pretraining and model development, including MFU improvements, scaling ladders, and the training infrastructure used for OLMo 3 and OLMo Hybrid.

  • Liger-Kernel

    I contribute to Liger-Kernel, a collection of custom Triton kernels for efficient LLM training. I've found these kernels very useful for training LLMs/VLMs on my RTX 4090. My contributions, as well as those of other top collaborators, were featured in LinkedIn Engineering's overview of the Liger-Kernel ecosystem.

  • microR1

    A micro-scale DeepSeek-R1 reproduction in the style of Karpathy's nanoGPT. Intended to be easy to understand and to hack on top of.

Favorite Reads

Fun Stuff