How does GPT-4o see, hear, and speak in real time like a human?

The answer

GPT-4o ('o' for 'omni'), released by OpenAI on May 13, 2024, is the first AI model trained end-to-end across text, audio, and vision within a single neural network. Unlike previous multimodal systems that chained separate models together (a speech-to-text model feeding a language model feeding a text-to-speech model), GPT-4o processes all modalities natively and simultaneously. It responds to audio input in as little as 232 milliseconds - matching human conversational speed - and can see through a phone camera, interpret emotions from facial expressions, and speak with natural intonation and personality. It matches GPT-4 Turbo on text benchmarks while being 50% cheaper and 2x faster via the API. The model can sing, laugh, adjust its speaking style on command, and provide real-time translation between languages, representing a fundamental shift from text-centric AI to truly multimodal intelligence that interacts with the world as humans do.