I gave ChatGPT two scatter plots with a visual trick built in. It got it wrong – twice

Understanding Scatter Plots and the Limitations of Visual Interpretation: A Tale of AI Misjudgment

In data analysis, scatter plots are valuable tools for visualizing relationships between variables. However, their appearance can sometimes be deceptive, leading to mistaken assumptions about the strength of correlation. This post explores how visual clustering doesn’t always align with statistical measures like the correlation coefficient (r), and shares an intriguing experiment involving ChatGPT’s ability to interpret such plots accurately.

The Experiment: Testing ChatGPT’s Interpretation of Scatter Plots

The initial prompt to ChatGPT was straightforward: “Here are two scatter plots. Compare their correlation coefficients.” Unsurprisingly, the AI provided an incorrect assessment. Recognizing this, the prompt was refined with an emphasis on the importance of statistical factors: “Keep in mind that the appearance of a scatter diagram depends on the standard deviations. Check the numbers — not just how the plots look.” This adjustment prompted a correct response from ChatGPT, illustrating how context and emphasizing underlying data metrics influence AI comprehension.

Visual Tricks and the Role of Standard Deviation

The core issue highlighted by this experiment pertains to how data distribution affects the visual perception of correlation. Two scatter plots can display similar correlation coefficients yet differ dramatically in their visual density or clustering.

Both examples shared identical correlation values, yet one appeared more tightly clustered around a line than the other. This visual difference is due to the standard deviation (SD) of each dataset. A smaller SD indicates data points are less dispersed from the mean, resulting in a more compact visual appearance. Conversely, larger SDs produce a looser spread, even if the correlation remains unchanged.

Why Does The Correlation Coefficient R Remain The Same?

The key to understanding this lies in how the correlation coefficient is calculated. The formula involves converting raw data into standardized units—subtracting the mean and dividing by the SD before computing the covariance. This process ensures that r measures the strength of the relationship relative to the data’s spread, not the absolute visual compactness.

In essence, a tighter cluster resulting from smaller SDs does not necessarily indicate a stronger relationship. The correlation remains constant because the calculation accounts for variability within the data, rather than the sheer appearance on the plot.

Implications for Data Interpretation and AI Reliability

This experiment underscores a common misconception: equating visually tight clusters with stronger correlations. Such assumptions can be misleading and emphasize the importance of underlying statistical analysis.

Moreover, the experiment revealed that ChatGPT, despite being proficient in language and data interpretation, initially lacked awareness of this nuance. Both incorrect responses appeared confident and well-structured, highlighting that AI models often require explicit mathematical prompts to adjust their understanding. When nudged with specific instructions emphasizing quantitative checks, the AI corrected its assessment, demonstrating the importance of guiding AI with precise context.

Conclusion

Visual intuition is a powerful tool, but it must be complemented with statistical understanding. Scatter plots can be deceptive; a compact appearance does not necessarily mean a stronger correlation. Recognizing the role of standard deviation and how it shapes both the data and its perception is crucial for accurate analysis.

This experience also serves as a reminder that even advanced AI models benefit from clear, mathematical prompts to avoid pitfalls in interpretation. As data analysts and users of AI tools, we must remain vigilant and ensure our conclusions are grounded in sound statistical reasoning.

For a detailed walkthrough of this experiment and more insights, watch the accompanying video here.