DeepSeek R-1 Model Overview and how it Ranks Versus OpenAI’s O1

DeepSeek is a Chinese AI company “committed to making AGI a truth” and open-sourcing all its models. They began in 2023, but have actually been making waves over the previous month or two, and specifically this past week with the release of their two most current reasoning models: DeepSeek-R1-Zero and the advanced DeepSeek-R1, also called DeepSeek Reasoner.

They have actually not only the designs however also the code and evaluation triggers for public usage, in addition to an in-depth paper describing their method.

Aside from producing 2 highly performant designs that are on par with OpenAI’s o1 design, the paper has a lot of important information around reinforcement learning, chain of thought thinking, prompt engineering with thinking models, and more.

We’ll begin by focusing on the training process of DeepSeek-R1-Zero, which distinctively relied solely on reinforcement learning, instead of traditional supervised knowing. We’ll then move on to DeepSeek-R1, how it’s reasoning works, and some prompt engineering best practices for thinking designs.

Hey everyone, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s newest model release and comparing it with OpenAI’s thinking models, specifically the A1 and A1 Mini designs. We’ll explore their training process, thinking capabilities, and some crucial insights into timely engineering for reasoning designs.

DeepSeek is a Chinese-based AI business devoted to open-source advancement. Their current release, the R1 reasoning design, is groundbreaking due to its open-source nature and innovative training methods. This includes open access to the models, triggers, and research study documents.

Released on January 20th, DeepSeek’s R1 achieved remarkable efficiency on different standards, measuring up to OpenAI’s A1 models. Notably, they also launched a precursor design, R10, which works as the structure for R1.

Training Process: R10 to R1

R10: This model was trained solely utilizing support knowing without supervised fine-tuning, making it the very first open-source model to achieve high performance through this approach. Training included:

– Rewarding proper answers in deterministic jobs (e.g., math issues).
– Encouraging structured reasoning outputs using templates with “” and “” tags

Through thousands of models, R10 developed longer reasoning chains, self-verification, and even reflective behaviors. For example, during training, the model demonstrated “aha” moments and self-correction habits, which are uncommon in conventional LLMs.

R1: Building on R10, R1 included numerous improvements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human choice positioning for sleek actions.
– Distillation into smaller designs (LLaMA 3.1 and 3.3 at numerous sizes).

Performance Benchmarks

DeepSeek’s R1 design carries out on par with OpenAI’s A1 models throughout numerous thinking standards:

Reasoning and Math Tasks: R1 competitors or exceeds A1 designs in precision and depth of reasoning.
Coding Tasks: A1 designs generally carry out much better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 typically exceeds A1 in structured QA jobs (e.g., 47% accuracy vs. 30%).

One noteworthy finding is that longer thinking chains typically improve efficiency. This aligns with insights from Microsoft’s Med-Prompt structure and OpenAI’s observations on test-time calculate and thinking depth.

Challenges and Observations

Despite its strengths, R1 has some restrictions:

– Mixing English and Chinese reactions due to an absence of supervised fine-tuning.
– Less refined responses compared to talk models like OpenAI’s GPT.

These issues were addressed during R1’s improvement process, including monitored fine-tuning and human feedback.

Prompt Engineering Insights

A fascinating takeaway from DeepSeek’s research study is how few-shot triggering degraded R1’s performance compared to zero-shot or concise tailored triggers. This aligns with findings from the Med-Prompt paper and OpenAI’s suggestions to limit context in thinking models. Overcomplicating the input can overwhelm the model and minimize accuracy.

DeepSeek’s R1 is a considerable advance for open-source reasoning designs, showing capabilities that match OpenAI’s A1. It’s an interesting time to experiment with these models and their chat interface, which is complimentary to use.

If you have questions or wish to find out more, have a look at the resources connected below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only technique

DeepSeek-R1-Zero sticks out from the majority of other state-of-the-art designs because it was trained utilizing only support knowing (RL), no monitored fine-tuning (SFT). This challenges the current standard method and opens brand-new chances to train thinking models with less human intervention and effort.

DeepSeek-R1-Zero is the very first open-source model to validate that sophisticated thinking abilities can be established simply through RL.

Without pre-labeled datasets, the model discovers through trial and error, refining its behavior, criteria, and weights based solely on feedback from the options it produces.

DeepSeek-R1-Zero is the base design for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training procedure for DeepSeek-R1-Zero included providing the model with different thinking tasks, ranging from mathematics issues to abstract logic obstacles. The model created outputs and was examined based on its efficiency.

DeepSeek-R1-Zero received feedback through a reward system that helped guide its knowing procedure:

Accuracy benefits: Evaluates whether the output is appropriate. Used for when there are deterministic results (mathematics problems).

Format benefits: Encouraged the design to structure its thinking within and tags.

Training prompt template

To train DeepSeek-R1-Zero to generate structured chain of idea series, the scientists utilized the following timely training template, replacing prompt with the reasoning concern. You can access it in PromptHub here.

This design template prompted the model to clearly detail its idea process within tags before providing the last response in tags.

The power of RL in reasoning

With this training process DeepSeek-R1-Zero started to produce advanced thinking chains.

Through countless training actions, DeepSeek-R1-Zero progressed to solve significantly complex issues. It discovered to:

– Generate long reasoning chains that enabled deeper and more structured problem-solving

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own mistakes, showcasing emerging self-reflective behaviors.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is mostly a precursor to DeepSeek-R1, it still attained high efficiency on numerous standards. Let’s dive into some of the experiments ran.

Accuracy enhancements throughout training

– Pass@1 accuracy began at 15.6% and by the end of the training it improved to 71.0%, comparable to OpenAI’s o1-0912 model.

– The red solid line represents efficiency with majority ballot (similar to ensembling and self-consistency techniques), which increased precision further to 86.7%, exceeding o1-0912.

Next we’ll take a look at a table comparing DeepSeek-R1-Zero’s performance across numerous thinking datasets against OpenAI’s reasoning models.

AIME 2024: 71.0% Pass@1, slightly below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a rating of 73.3%.

– Performed much even worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll look at how the action length increased throughout the RL training procedure.

This graph reveals the length of responses from the design as the training procedure advances. Each “action” represents one cycle of the design’s learning procedure, where feedback is offered based upon the output’s efficiency, evaluated utilizing the prompt design template talked about previously.

For each concern (representing one action), 16 reactions were sampled, and the typical accuracy was determined to ensure steady evaluation.

As training advances, the model produces longer thinking chains, permitting it to fix progressively intricate thinking tasks by leveraging more test-time calculate.

While longer chains don’t constantly guarantee better outcomes, they typically correlate with enhanced performance-a trend likewise observed in the MEDPROMPT paper (read more about it here) and in the initial o1 paper from OpenAI.

Aha minute and self-verification

One of the coolest elements of DeepSeek-R1-Zero’s advancement (which also applies to the flagship R-1 model) is just how great the model ended up being at thinking. There were advanced thinking habits that were not explicitly programmed however arose through its reinforcement discovering procedure.

Over thousands of training steps, the model began to self-correct, reevaluate problematic reasoning, and verify its own solutions-all within its chain of thought

An example of this kept in mind in the paper, referred to as a the “Aha minute” is below in red text.

In this circumstances, the model actually stated, “That’s an aha minute.” Through DeepSeek’s chat function (their variation of ChatGPT) this kind of thinking generally emerges with phrases like “Wait a minute” or “Wait, but … ,”

Limitations and challenges in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to perform at a high level, there were some drawbacks with the model.

Language mixing and coherence problems: The design periodically produced responses that mixed languages (Chinese and English).

Reinforcement learning trade-offs: The lack of monitored fine-tuning (SFT) indicated that the model lacked the improvement needed for fully polished, human-aligned outputs.

DeepSeek-R1 was established to deal with these problems!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning model from the Chinese AI laboratory DeepSeek. It develops on DeepSeek-R1-Zero, which was trained entirely with support learning. Unlike its predecessor, DeepSeek-R1 integrates monitored fine-tuning, making it more refined. Notably, it exceeds OpenAI’s o1 design on a number of benchmarks-more on that later on.

What are the primary differences between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 constructs on the foundation of DeepSeek-R1-Zero, which works as the base design. The two differ in their training methods and overall performance.

1. Training approach

DeepSeek-R1-Zero: Trained entirely with support learning (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes supervised fine-tuning (SFT) initially, followed by the same support discovering procedure that DeepSeek-R1-Zero damp through. SFT assists enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Had problem with language blending (English and Chinese) and readability issues. Its reasoning was strong, but its outputs were less polished.

DeepSeek-R1: Addressed these issues with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong reasoning design, sometimes beating OpenAI’s o1, however fell the language blending concerns minimized use greatly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on most reasoning criteria, and the actions are much more polished.

Simply put, DeepSeek-R1-Zero was a proof of idea, while DeepSeek-R1 is the completely optimized version.

How DeepSeek-R1 was trained

To take on the readability and coherence issues of R1-Zero, the scientists included a cold-start fine-tuning phase and a multi-stage training pipeline when building DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a premium dataset of long chains of idea examples for preliminary supervised fine-tuning (SFT). This information was collected using:- Few-shot triggering with comprehensive CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, refined by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the same RL procedure as DeepSeek-R1-Zero to improve its reasoning capabilities further.

Human Preference Alignment:

– A secondary RL phase improved the design’s helpfulness and harmlessness, making sure much better positioning with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller, effective models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark efficiency

The researchers tested DeepSeek R-1 throughout a range of benchmarks and against leading models: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The standards were broken down into a number of classifications, revealed below in the table: English, Code, Math, and Chinese.

Setup

The following specifications were used across all designs:

Maximum generation length: 32,768 tokens.

Sampling configuration:- Temperature: 0.6.

– Top-p worth: 0.95.

– DeepSeek R1 exceeded o1, Claude 3.5 Sonnet and other models in the bulk of reasoning benchmarks.

o1 was the best-performing design in four out of the 5 coding-related criteria.

– DeepSeek performed well on creative and long-context job task, like AlpacaEval 2.0 and ArenaHard, surpassing all other designs.

Prompt Engineering with reasoning designs

My favorite part of the post was the researchers’ observation about DeepSeek-R1’s level of sensitivity to prompts:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which recommendations Microsoft’s research on their MedPrompt framework. In their research study with OpenAI’s o1-preview design, they discovered that overwhelming thinking models with few-shot context broken down performance-a sharp contrast to non-reasoning models.

The crucial takeaway? Zero-shot prompting with clear and succinct directions seem to be best when utilizing thinking models.