Have you ever tried to use ChatGPT to generate responses based on external content? If so, and to put it mildly, you may have noticed that the generated responses can sometimes deviate from the desired output.

Let's explore an experiment that sheds light on how ChatGPT processes external content and how this can impact the generated response.

The ChatGPT External Context Experiment

Update: No, ChatGPT cannot read external web pages. Watch this video.

The experiment involved three different methods of providing the same external content to ChatGPT:

Directly reading a web page: In this scenario, ChatGPT was asked to read a web page directly from a website and use it as context for the generated response.

Prompt: please read this web page so I can use it for context in our conversation: {url}

Using a Google Doc: In this scenario, the same content as in scenario 1 was provided in a publicly shared (published as a web page, dynamic updating turned off) Google Doc, and ChatGPT was asked to use it as context for the generated response using the same prompt as above.

Pasting content directly into ChatGPT: In this scenario, the same content as in scenario 2 was directly pasted into ChatGPT via the chat interface. We used the Advanced ChatGPT training method, which you can learn about here.

The Experiment Findings

The experiment revealed that ChatGPT processes external content differently depending on the ingestion method used.

When external content was directly pasted into ChatGPT, it was given more weight and influence on the generated response. This is likely because ChatGPT sees pasted-in content as directly relevant to the current conversation.

On the other hand, when external content was read directly from a website or Google Doc, ChatGPT seemed to commingle the content with its existing knowledge base.

This can dilute the impact of the external content on the generated response, especially if the external content deviates from ChatGPT's existing knowledge base.

The Implications of this Experiment

The implications of these findings are significant for anyone using ChatGPT to generate responses based on external content.

To optimize ChatGPT's response to external content, it's important to carefully consider the quality and relevance of the content being provided.

When providing context through a website or Google Doc, it's essential to ensure that the content is of high quality and aligns with the desired output.

But it's also important to note that the impact of external content on the generated response can vary depending on several factors, including the length and complexity of the input, the specific training data the model has been exposed to, and the prompt or query being generated.

When pasting content directly into ChatGPT, it's crucial to be selective and only provide the most relevant and high-quality information. It's also critical to initialize the training context process correctly, something we cover here.

This can ensure that ChatGPT gives it the appropriate weight and incorporates it in a way that aligns with the desired output.

Key Takeaways

Our experiment highlights the importance of carefully considering the quality and relevance of external content when using ChatGPT to generate responses.

By understanding how ChatGPT processes external content, we can optimize our use of this powerful tool and ensure that the generated responses align with the desired output.