Tips for getting GPT to return JSON

Here at Prompt Wrangler we help you turn prompts into structured APIs. Often you will want to those APIs to return JSON instead of just text. This guide walks you through some of the tips and tricks that we use to get GPT to return structured valid JSON in the right format.

💡 While these tips are focused on GPT-3, GPT3.5 and GPT-4, they should work well for most LLMs.

1. Tell GPT what you want

The most important thing (and probably the most important thing for any prompt) is to tell GPT what you want. In your prompt include text that says something like respond in a structured JSON format. You might even want to repeat this a few times. If you are using a chat model then include this in the system message as well.

2. Specify a specific JSON schema

Usually there is a specific JSON structure that you want and you should include your schema in the prompt. The exact format of your schema specification doesn’t matter that much. You can use JSON Schema, a Zod Schema, a markdown table or a JSON example with comments. Most structures work pretty well. The important thing is to include this schema and tell GPT to follow it. Here is an example using JSON pseudocode with comments:

Return the data as JSON in the following format:
{
   "name": // string
	  "address":  {
     "line1": // string
     "city": // string
     "state": // string (two letter abbreviation)
     "zip": // string
  },
 "phone": // phone number in format 203-273-1232
}

3. Ask GPT to explain it’s reasoning

Often in your GPT schema you will have have a enum or classification as part of the JSON object. In order for GPT to work well for these cases you will want to ask it to explain it’s reasoning. You can either include a “thought” property in the JSON structure that is returned (very common in Agent workflows) or just tell GPT to explain it’s reasoning.

4. Start the response for the assistant

When you chat models include an assistant message after the user message that says something like Sure! Here's your JSON. This further enforces that the assistant is going to respond with JSON in the response. In the API this would look like this:

# Note: you need to be using OpenAI Python v0.27.0 for the code below to work
import openai

openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Parse out the phone number, name and address from the following text. \n\nText:\nHello. It's Michael scott. Can you give me a call back at 2832340371. My address is 1 Madison ave, new york 10010\n\nReturn the data as JSON in the following format:\n{\n   \"name\": // string\n  \"address\":  {\n     \"line1\": // string\n     \"city\": // string\n     \"state\": // string\n     \"zip\": // string\n  },\n \"phone\": // phone number in format 203-273-1232\n"},
				# Key line right here
        {"role": "assistant", "content": "Sure! Here's your json..."},
    ]
)

5. Parse JSON from text

Instead of trying to get GPT to only return the JSON, just parse the JSON from the response. This will allow you to ask GPT to explain it’s reasoning and not have to worry about the extra text. To parse the JSON the simplest method is to just look for the opening { and closing }. In typescript it could look something like this.

// parse json 
const parseJson = (json: string) => {
	const jsonStart = json.indexOf("{");
	const jsonEnd = json.lastIndexOf("}");
	
	if (jsonStart === -1 || jsonEnd === -1) {
	      return undefined;
	}
	
	const jsonString = json.slice(jsonStart, jsonEnd + 1);

  // Might throw and error - see below
	const json = JSON.parse(jsonString)
	return json
}

6. Use jsonrepair to…repair json

Sometimes the JSON that is returned isn’t valid JSON. This happens less with GPT 4 but still occurs occasionally. This can lead to runtime error. Instead of trying a bunch of stuff in the original prompt to try and fix this occasional error, it’s easier to just fix it after the fact. Instead of just blindly parsing the JSON that was parsed, use a library like jsonrepair to fix any json error. The code above can be updated to repair the json before parsing it. This will fix most of the issues that arise.

import { jsonrepair } from "jsonrepair";

// parse and repair json
const parseAndRepairJson = (json: string) => {
	const jsonStart = json.indexOf("{");
	const jsonEnd = json.lastIndexOf("}");
	
	if (jsonStart === -1 || jsonEnd === -1) {
	      return undefined;
	}
	
	const jsonString = json.slice(jsonStart, jsonEnd + 1);

  // Repair json before parsing it
	const repairedJson = jsonrepair(jsonString);
  const jsonResponse = JSON.parse(repairedJson
}

7. Have GPT fix json

If jsonrepair isn’t enough to fix json, one last method you can take is to have GPT fix the JSON. To do this take the string that is return from GPT and feed it back into the model with a new prompt like so:

Fix the following JSON so that it is valid json that parses correctly
{{ INSERT_JSON_HERE }}

While this usually works, this does add cost and latency. It’s recommended to only do this if you are having problems with jsonrepair and only run this approach after jsonrepair fails.

Wrapping Up

With these tips you should consistently get good JSON responses with any of the latest GPT models. If you are trying to turn prompts into structured APIs then check out Prompt Wrangler where we handle all these steps for you.