Researchers from MIT recently demonstrated a computer algorithm that can scan a silent video and determine what sounds are supposed to be present. This isn’t a ploy to replace Foley artists with artificial intelligence, but an attempt at giving robots another way to learn about the physical world.
Says PhD student Andrew Owens: “Think of it as a toy version of learning about the world the way that babies do, by banging, stomping, and playing with things.” The theory is that robots will eventually be able to determine the composition and properties of objects by the sounds they make. For instance, if a robotic appendage tapped a pane of glass, it might understand that the surface is more fragile than it if had tapped concrete.
The project is coming along — and it’s even starting to look promising — but the technology has a long way to go. The AI can only match audio to the series of video clips that it has been analyzing — there are over a thousand of them, but they all involve the researchers hitting things with a drumstick. The algorithm can’t generate sounds yet, either, rather it will browse its database of tens of thousands of sound clips and attempt to choose the sound that best fits the action that it’s “seeing.” If it determines that the drumstick is hitting a bush, it will usually choose a sound that fits at least somewhat closely . . . although it will sometimes choose the sound of a rustling plastic bag.
I invite you to watch this video clip released by the MIT Computer Science and Artificial Intelligence Laboratory. After watching it myself, I think I’ve figured out what will ultimately drive computers to become self-aware and wipe out the human race: being forced to listen to a drumstick hitting a sidewalk for countless thousands of hours.
All opinions are that of the author and do not necessarily reflect those of Deck Ape...or anyone else. Arrr!