People are constantly pushing what AI can do, and you know what’s really exciting? Being able to train reasoning models yourself! And speaking of pushing boundaries, the team over at Unsloth has just dropped some seriously good news for anyone interested in training reasoning models. Remember all the hype around their GRPO release a couple of weeks back? Well, they’ve outdone themselves again!

They’ve managed to slash the VRAM needed to train your own reasoning models down to a ridiculously low 5GB! Yep, five gigabytes. This is thanks to some seriously smart engineering, specifically for Qwen2.5 (1.5B). To give you some perspective, that’s a drop from 7GB in their previous release. Talk about making reasoning model training more accessible!

This update centers around GRPO, or Guided Reasoning Policy Optimization. Turns out, it’s the same algorithm that powered DeepSeek-R1, a seriously capable reasoning model. And the cool thing about GRPO, according to Unsloth, is how flexible it is. It can take pretty much any open source Large Language Model think Llama, Mistral, Phi, all those guys and turn it into a reasoning machine, ready for chain-of-thought processes. So, basically, you can train reasoning models from a whole range of base models now.

Why is 5GB VRAM Such a Game Changer for Training Reasoning Models?

Now, if you’re not knee-deep in the tech details, you might be wondering, “Okay, 5GB VRAM, big deal?”. But seriously, it is a big deal, especially if you’re looking to train reasoning models. Training AI models, especially the brainy, reasoning types, usually gobbles up processing power and memory, especially VRAM (Video RAM). VRAM is that super-speedy memory on your graphics card that’s essential for heavy-lifting tasks like AI training.

Traditionally, you needed super expensive, powerful hardware with tons of VRAM to even think about training reasoning models. This was a real roadblock for a lot of folks, researchers on a budget, students, hobbyists, even smaller companies. It made advanced AI development feel like it was only for the big players with deep pockets.

But cutting the VRAM requirement down to just 5GB changes the whole landscape of reasoning model training. Suddenly, training sophisticated reasoning models becomes within reach for way more people. You might even be able to do it on your everyday gaming PC! Think about it – the power to train your own reasoning AI, right on your desk. That’s a serious shift.

And it’s not just about being more accessible to train reasoning models. Unsloth points out that training smaller models with GRPO can actually be faster than training larger models, while getting you similar results. So, you’re not just saving on VRAM, you’re potentially saving time too, when you train reasoning models this way. Plus, they mention you can even let these training runs run in the background while you do other things. Multitasking and training reasoning models? Sounds like a win-win.

The Secret Sauce: Efficient GRPO Algorithm

So, how did Unsloth pull off this VRAM magic trick, making it so easy to train reasoning models now? It’s all thanks to their newly developed “Efficient GRPO algorithm.” Apparently, this smart piece of tech does something pretty amazing: 10 times longer context lengths while using 90% less VRAM compared to other GRPO setups. And they’re not just talking about basic GRPO. They’re comparing against setups using things like LoRA, QLoRA, and even Flash Attention 2 (FA2), which are all about being memory efficient. Despite the huge VRAM savings, Unsloth says there’s zero loss in accuracy when you train reasoning models this way. That’s a bold claim, but if it’s true, it’s a huge win.

To really hit home how much VRAM is saved when you train reasoning models using their method, they give a striking example. Training Llama 3.1 (8B) at a 20K context length using a standard GRPO setup with TRL + FA2 would usually need a massive 510.8GB of VRAM. Yep, over half a terabyte! But Unsloth’s 90% VRAM reduction brings that down to a much more manageable 54.3GB for the same setup, making it far easier to train reasoning models even with long contexts. That’s almost 460GB of VRAM saved! Suddenly, training reasoning models at long context lengths, which is key for reasoning tasks, becomes doable without needing a supercomputer in your basement.

How They Make Training Reasoning Models So Memory Efficient

Okay, so 90% less VRAM sounds unreal, but how exactly are they making training reasoning models so memory-light? Unsloth breaks it down into a few key tricks they’re using under the hood.

First off, they’re using their own gradient checkpointing algorithm. It’s not brand new; they released it before. But it seems to be a big part of this. Think of gradient checkpointing like this: when you train reasoning models, the model makes a lot of in-between data (activations). Normally, all this data sits in VRAM, and it eats up memory fast. Unsloth’s algorithm cleverly moves some of these activations to system RAM (your regular computer memory) asynchronously. This means it happens in the background, without really slowing things down – they say it’s only about 1% slower. By doing this smart offloading, they say they save a whopping 372GB of VRAM in their example, just because GRPO training involves multiple “generations” (num_generations = 8 in their example). That’s a massive amount of memory freed up, making training reasoning models on lower VRAM machines a reality! And they mention they can push memory use even lower with something called “intermediate gradient accumulation.” Sounds like they’re really squeezing every bit of efficiency out of the process to make reasoning model training accessible.

But the VRAM savings don’t stop there for training reasoning models. Unsloth also mentions that their system uses the same GPU memory space as the underlying inference engine, vLLM (Very Large Language Model). This is different from some other setups where the training might make separate memory spaces, which is inefficient. By sharing memory space with vLLM, Unsloth says they save another 16GB of VRAM when you train reasoning models. It’s like tidying up memory use to avoid extra stuff.

Comparison Table

To really show the difference, they give a cool table comparing their approach to a standard GRPO setup using TRL + FA2 for training reasoning models:

Looking at those numbers, it’s pretty clear how much less VRAM is needed when you train reasoning models with Unsloth. It’s not just a little saving; it’s a complete overhaul of memory efficiency.

Under the Hood: A Peek at the Efficient GRPO Algorithm

While they don’t get super technical in the announcement, Unsloth does drop a few hints about what makes their “Efficient GRPO algorithm” so special for training reasoning models. They mention inspiration from “Horace He’s linear cross entropy implementation.” If you know about that, it suggests they’re using some clever math tricks to make the GRPO calculations faster, especially when dealing with long context lengths, which is key for effective reasoning model training.

They also found some interesting quirks and details in the standard GRPO setup. For example, they point out that the usual GRPO setup uses “reverse KL divergence,” not the more common “forward KL divergence.” They also found that just using linear cross entropy with mixed precision (float16 or float8) and automatic mixed precision scaling can cause problems if you’re not careful. It sounds like they really dug into the math and code of GRPO to make it super efficient for memory use and smooth reasoning model training.

They even briefly touch on the math behind GRPO and some possible issues they found in other setups, specifically about the formula for reverse KL divergence. They did tests to look into these details, comparing different setups and versions. One interesting find was the need for a line of code that looks like it shouldn’t do anything: torch.exp(q – q.detach()) * advantages.unsqueeze(1). At first, it looks like it should just become 1 and not matter. But Unsloth found it’s actually really important, maybe because of how the autograd engine (the part of PyTorch that handles gradient calculations) works. These kinds of details come from careful testing and really knowing the tech, and it’s this level of detail that probably leads to their awesome results in making reasoning model training more accessible.

Getting Started with Reasoning Models Training is Easier Than Ever

Besides all the tech magic, Unsloth has also made sure this new thing is easy to use. They say you don’t need to manually “patch” GRPO into your code anymore. Apparently, that’s all automatic now, simplifying the process of training reasoning models. One less step to worry about when you’re trying to get started.

And for those who want to jump right in and try it out, they’ve made a free GRPO notebook that you can use with Google’s Colab, even using their free GPUs! The notebook is set up for Llama 3.1 (8B) with 10x longer context. This is an awesome way for anyone to try out GRPO training with almost no setup and no cost for hardware. It’s perfect for getting your hands dirty with reasoning model training.

For anyone wanting to go deeper, they really recommend checking out their full Guide that covers everything GRPO, including reward functions and verifiers. It sounds like they’ve made some great resources to help people understand and use GRPO effectively for reasoning model training.

And if you’re interested in using vLLM’s inference stuff, they’ve got you covered there too. The update includes support for using FP8 KV caches in vLLM. This can cut KV cache space use in half on newer GPUs (RTX 3090, A100, and newer). They even give code examples showing how to turn on float8 KV cache and how to pass vLLM’s sampling settings like min_p. It’s clear they’re thinking about the whole process, from training reasoning models to using them.

Dynamic 4-bit Quantization with vLLM

As if the GRPO improvements weren’t enough to boost your reasoning model training, Unsloth also snuck in another update. You can now run their “Dynamic 4-bit” quantization directly with vLLM for inference. This is because of something they added to the vLLM project itself. Dynamic 4-bit quantization is another way to make models more efficient, and Unsloth claims their dynamic quantization works better accuracy-wise than standard 4-bit quantization. They even point to examples and tests to prove it. It’s like getting a bonus feature on top of an already awesome update for reasoning model training.

Wrapping it Up

Unsloth’s latest news is a real step forward in making advanced AI easier for everyone to access. Cutting the VRAM needed for training reasoning models down to just 5GB is a huge deal. Combined with their Efficient GRPO algorithm, the promise of longer context lengths and faster training, while keeping accuracy, is super exciting for anyone looking to train reasoning models.

If you’re keen to explore training reasoning models, or if you’ve been held back by VRAM limits, this update from Unsloth is definitely worth checking out. The free notebooks and detailed guide make it easy to start, and the chance to train powerful AI models on more common hardware is truly amazing. It feels like they’re opening doors for more people to innovate and play around with smart AI through accessible reasoning model training.

| Latest From Us

• 
  Forget Towers: Verizon and AST SpaceMobile Are Launching Cellular Service From Space
• 
  This $1,600 Graphics Card Can Now Run $30,000 AI Models, Thanks to Huawei
• 
  The Global AI Safety Train Leaves the Station: Is the U.S. Already Too Late?
• 
  The AI Breakthrough That Solves Sparse Data: Meet the Interpolating Neural Network
• 
  The AI Advantage: Why Defenders Must Adopt Claude to Secure Digital Infrastructure