OpenAI’s Sora video-generating model can also render video games

OpenAI’s new video-generating model, Sora, is capable of impressive cinematographic feats. A technical paper titled “Video generation models as world simulators” reveals that Sora can generate videos of up to 1080p resolution, perform various editing tasks, and simulate digital worlds, as well as control player characters in games such as Minecraft.

In an experiment, OpenAI fed Sora prompts containing the word “Minecraft” and had it render a convincingly Minecraft-like HUD and game dynamics while controlling the player character.

The model’s capabilities have been described as more of a data-driven physics engine than a creative tool, as it determines the physics of each object in an environment and renders a photo, video, or interactive 3D world based on these calculations.

According to senior Nvidia researcher Jim Fan, Sora could pave the way for highly-capable simulators of the physical and digital world, though it has limitations in accurately approximating certain physics and interactions.

Despite its limitations, the paper suggests that Sora could lead to the development of more realistic procedurally generated games purely from text descriptions. However, due to potential deepfake implications, OpenAI has opted to limit access to Sora.