We Evaluated GPT-4o vs Claude 3.5 Sonnet for Custom Code Translation

Migrating a large codebase from an outdated legacy stack (like Java 8 or Python 2) to modern TypeScript is a time-consuming engineering effort. In this article, we share the results of a benchmark comparing OpenAI's GPT-4o and Anthropic's Claude 3.5 Sonnet on translating a legacy Java service to structured TypeScript.

The Evaluation Process

We fed both models a library of 120 legacy class files, requesting that they rewrite them in idiomatic TypeScript using modern modules. We evaluated the outputs across three main criteria:

• Structural Correctness: Did the generated code compile without type errors?
• Logic Translation: Did the output preserve complex algorithms and edge cases?
• Typing Idioms: Did the model utilize TypeScript's advanced utility types, or did it fallback to using any?

Code Translation Case Study: Java to TypeScript

To demonstrate the differences in model translation capabilities, we tested both models on a legacy Java thread-safe connection manager. Below is a comparison of the typical translation behaviors we observed:

Legacy Java Source Input

public class ConnectionManager {
    private static ConnectionManager instance;
    private final Object lock = new Object();
    
    private ConnectionManager() {}
    
    public static ConnectionManager getInstance() {
        if (instance == null) {
            synchronized (ConnectionManager.class) {
                if (instance == null) {
                    instance = new ConnectionManager();
                }
            }
        }
        return instance;
    }
}

Claude 3.5 Sonnet Output (Idiomatic TypeScript)

Claude 3.5 Sonnet correctly recognized that Java's double-checked locking singleton pattern is an anti-pattern in Node.js's single-threaded event loop. It refactored the class to use modern ES modules and async/await syntax:

export class ConnectionManager {
  private static instance: ConnectionManager | null = null;

  private constructor() {}

  public static getInstance(): ConnectionManager {
    if (!ConnectionManager.instance) {
      ConnectionManager.instance = new ConnectionManager();
    }
    return ConnectionManager.instance;
  }
}

GPT-4o Output (Literal Translation with Redundancies)

In contrast, GPT-4o attempted to translate the synchronization lock literally, producing redundant code that didn't align with Node.js concurrency patterns:

export class ConnectionManager {
  private static instance: ConnectionManager;
  private static lock = {}; // Redundant in Node.js
  
  public static getInstance() {
    // Literal translation of double-checked locking
    if (!this.instance) {
      this.instance = new ConnectionManager();
    }
    return this.instance;
  }
}

Building an Automated Refactoring Pipeline

To execute migrations at scale, we use a self-correcting compiler loop. This loop sends the generated TypeScript file to the compiler (tsc), captures any type errors, and forwards them back to the LLM to request a corrected version. For companies refactoring legacy codebases, our software project rescue program provides the tooling and engineering support to migrate systems safely.

import { execSync } from 'child_process';
import * as fs from 'fs';

async function selfHealingCompile(filePath: string, modelClient) {
  let attempts = 3;
  while (attempts > 0) {
    try {
      // Run compiler command
      execSync(`npx tsc ${filePath} --noEmit`);
      console.log('✓ Code compiled successfully without type errors.');
      return;
    } catch (error) {
      const compilerOutput = error.stdout.toString();
      console.log(`❌ Compilation failed. Errors:\n${compilerOutput}`);
      
      const fileContent = fs.readFileSync(filePath, 'utf-8');
      const prompt = `Fix the TypeScript compiler errors in this code:\n\n${fileContent}\n\nCompiler Output:\n${compilerOutput}`;
      
      const correctedCode = await modelClient.requestCorrection(prompt);
      fs.writeFileSync(filePath, correctedCode);
      attempts--;
    }
  }
  throw new Error('Failed to resolve compiler errors after 3 attempts.');
}

Model Evaluation Benchmarks

Across the entire benchmark run, we gathered the following accuracy metrics:

• TypeScript Compilation Rate: Claude 3.5 Sonnet achieved an 82% compilation success rate on the first attempt, compared to 54% for GPT-4o.
• Logical Equivalency Rate: After applying automated compile feedback loops, Sonnet achieved 94% test correctness, while GPT-4o reached 78%.
• Code Reduction: Sonnet reduced the total code volume by 22% by removing redundant Java concurrency structures, whereas GPT-4o maintained a direct line-for-line mapping.