Anthropic announced the release of Claude 2, its new model, which offers improved performance, longer responses, and is accessible via API and a new public-facing beta website, claude.ai. The company stated that Claude 2 provides a 100K token context window, allowing it to process hundreds of pages of technical documentation or an entire book. It can also generate longer documents, such as memos, letters, or stories, up to a few thousand tokens in a single output.
Key Points
- Claude 2 scored 76.5% on the multiple-choice section of the Bar exam, an increase from 73.0% for Claude 1.3.
- The model achieved 71.2% on the Codex HumanEval Python coding test, up from 56.0%.
- On the GSM8k grade-school math problems, Claude 2 scored 88.0%, an improvement from 85.2%.
- Claude 2 demonstrated 2x better harmlessness in internal red-teaming evaluations compared to Claude 1.3.
- The Claude 2 API is offered at the same price as Claude 1.3.
- The beta chat experience for Claude 2 is available in the US and UK.
Context
According to Anthropic, Claude 2 shows improved reasoning, coding, and mathematical abilities. For instance, it scored above the 90th percentile on the GRE reading and writing exams, aligning with the median applicant on quantitative reasoning when compared to college students applying to graduate school. The company also highlighted that Claude 2 was trained on more recent data, providing knowledge of newer frameworks and libraries. Anthropic previously introduced Claude as a next-generation AI assistant available via chat interface and API, with early partners including Notion, Quora, and DuckDuckGo. At that time, Anthropic offered two versions: Claude, a high-performance model, and Claude Instant, a lighter, less expensive, and faster option.
Why It Matters
Builders and researchers can note the specific performance benchmarks and the expanded context window of Claude 2, which may enable new applications requiring extensive document processing or complex reasoning. The stated improvements in harmlessness also indicate Anthropic's continued focus on safety, providing a reference point for evaluating model robustness against harmful outputs.
What To Do
- Compare the performance metrics of Claude 2 against previous models or other state-of-the-art systems for specific use cases like coding or long-form content generation.
- Test the 100K token context window with large documents or complex prompts to assess its practical utility for summarization, analysis, or content creation.
- Review the safety improvements and red-teaming evaluation results to understand Anthropic's approach to reducing harmful outputs.
- If in the US or UK, try the claude.ai beta chat experience to interact with Claude 2 directly.
Keep Exploring
/atlas/claude-family /techniques/system-user-separation /techniques/ptcf /studio?pack=foundation