Visual Microphones: Next-Generation Spy Technology
There are definitely limitations to the technology, Davis said, and it may not make for better sound reconstruction than other methods already in use. “Big brother probably won't be able to hear anything that anyone ever says all of a sudden,” Davis said. “But it is possible that you could use this to discover sound in situations where you couldn’t before. It’s just adding one more tool for those forensic applications.”
Imagine someone listening in to your private conversation by filming the bag of chips sitting on the other side of the room.
This particular study grew out of an earlier experiment at MIT, led by Michael Rubinstein, now a postdoctoral researcher at Microsoft Research New England. In 2012, Rubinstein amplified tiny variations in video to detect things like the skin color change caused by the pumping of blood. Studying the vibrations caused by sound was a logical next step. But getting intelligible speech out of the analysis was surprising, Davis said.
The results are certainly impressive (and a little scary). In one example shown in a compilation video, a bag of chips is filmed from 15 feet away, through sound-proof glass. The reconstructed audio of someone reciting “Mary Had a Little Lamb” in the same room as the chips isn’t crystal clear. But the words being said are possible to decipher.
In most cases, a high-speed camera is necessary to accomplish the feat. Still, at 2,000 to 6,000 frames per second, the camera used by the researchers is nothing compared to the best available on the market, which can surpass 100,000 frames per second. And the researchers found that even cheaper cameras could be used.
“It’s surprisingly possible to take advantage of a bug called rolling shutter,” Davis said. “Usually, it creates these artifacts in the image that people don’t like.” When cameras use rolling shutter to capture an image, they don’t capture one single point in time. Instead, the camera scans across the frame in one direction, picking up each row at a slightly different moment.
By doing so, the camera happens to encode information at a much higher rate than its actual frame rate. For the researchers, that meant being able to analyze vibrations that should have happened too quickly for capture on film. “It kind of turns a two-dimensional low-speed camera into a one-dimensional high-speed camera,” Davis explained.
Davis and his colleagues care more about applications in scientific research. “This is a new dimension to how you can image objects,” he said. “It tells you something about how they respond physically to pressure, but instead of poking and prodding at them, all you need is to play sound at them.”
Melissa Block talks to Abe Davis, a graduate student at the Massachusetts Institute of Technology. Davis helped author a paper on a visual system to detect sound, which can recover intelligible speech from the vibrations of a potato chip bag photographed through soundproof glass.
In other experiments, they extracted useful audio signals from videos of aluminum foil, the surface of a glass of water, and even the leaves of a potted plant. The researchers will present their findings in a paper at this year’s Siggraph, the premier computer graphics conference.
“When sound hits an object, it causes the object to vibrate,” says Abe Davis, a graduate student in electrical engineering and computer science at MIT and first author on the new paper. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”
Joining Davis on the Siggraph paper are Frédo Durand and Bill Freeman, both MIT professors of computer science and engineering; Neal Wadhwa, a graduate student in Freeman’s group; Michael Rubinstein of Microsoft Research, who did his PhD with Freeman; and Gautham Mysore of Adobe Research.
Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.
Commodity Hardware
In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as that with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
The researchers’ technique has obvious applications in law enforcement and forensics, but Davis is more enthusiastic about the possibility of what he describes as a “new kind of imaging.”
“We’re recovering sounds from objects,” he says. “That gives us a lot of information about the sound that’s going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways.” In ongoing work, the researchers have begun trying to determine material and structural properties of objects from their visible response to short bursts of sound.
Watch how MIT researchers extract audio from the vibrations of a plant, potato-chip bag, and other objects.
In the experiments reported in the Siggraph paper, the researchers also measured the mechanical properties of the objects they were filming and determined that the motions they were measuring were about a tenth of micrometer. That corresponds to five thousandths of a pixel in a close-up image, but from the change of a single pixel’s color value over time, it’s possible to infer motions smaller than a pixel.
Suppose, for instance, that an image has a clear boundary between two regions: Everything on one side of the boundary is blue; everything on the other is red. But at the boundary itself, the camera’s sensor receives both red and blue light, so it averages them out to produce purple. If, over successive frames of video, the blue region encroaches into the red region — even less than the width of a pixel — the purple will grow slightly bluer. That color shift contains information about the degree of encroachment.
Putting It Together
Some boundaries in an image are fuzzier than a single pixel in width, however. So the researchers borrowed a technique from earlier work on algorithms that amplify minuscule variations in video, making visible previously undetectable motions: the breathing of an infant in the neonatal ward of a hospital, or the pulse in a subject’s wrist.
That technique passes successive frames of video through a battery of image filters, which are used to measure fluctuations, such as the changing color values at boundaries, at several different orientations — say, horizontal, vertical, and diagonal — and several different scales.
The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it’s struck by sound waves. Different edges of the object may be moving in different directions, so the algorithm first aligns all the measurements so that they won’t cancel each other out. And it gives greater weight to measurements made at very distinct edges — clear boundaries between different color values.
The researchers also produced a variation on the algorithm for analyzing conventional video. The sensor of a digital camera consists of an array of photodetectors — millions of them, even in commodity devices. As it turns out, it’s less expensive to design the sensor hardware so that it reads off the measurements of one row of photodetectors at a time. Ordinarily, that’s not a problem, but with fast-moving objects, it can lead to odd visual artifacts. An object — say, the rotor of a helicopter — may actually move detectably between the reading of one row and the reading of the next.
For Davis and his colleagues, this bug is a feature. Slight distortions of the edges of objects in conventional video, though invisible to the naked eye, contain information about the objects’ high-frequency vibration. And that information is enough to yield a murky but potentially useful audio signal.
“This is new and refreshing. It’s the kind of stuff that no other group would do right now,” says Alexei Efros, an associate professor of electrical engineering and computer science at the University of California at Berkeley. “We’re scientists, and sometimes we watch these movies, like James Bond, and we think, ‘This is Hollywood theatrics. It’s not possible to do that. This is ridiculous.’ And suddenly, there you have it. This is totally out of some Hollywood thriller. You know that the killer has admitted his guilt because there’s surveillance footage of his potato chip bag vibrating.”
Efros agrees that the characterization of material properties could be a fruitful application of the technology. But, he adds, “I’m sure there will be applications that nobody will expect. I think the hallmark of good science is when you do something just because it’s cool and then somebody turns around and uses it for something you never imagined. It’s really nice to have this type of creative stuff.”
ABE DAVIS: Hi, thanks for having me.
BLOCK: And let me see if I've gotten this right. The idea is that sound waves will cause an object to vibrate. You are capturing that vibration on video and then - this is the tricky part - you're turning that vibration back into sound. Am I close?
DAVIS: Yeah, that's pretty accurate.
BLOCK: OK, well let's take a listen to what this sounds like. You played a song to a house plant, and the song sounded like this.
(SOUNDBITE OF SONG, "MARY HAD A LITTLE LAMB")
BLOCK: And let me see if I've gotten this right. The idea is that sound waves will cause an object to vibrate. You are capturing that vibration on video and then - this is the tricky part - you're turning that vibration back into sound. Am I close?
DAVIS: Yeah, that's pretty accurate.
BLOCK: OK, well let's take a listen to what this sounds like. You played a song to a house plant, and the song sounded like this.
(SOUNDBITE OF SONG, "MARY HAD A LITTLE LAMB")
BLOCK: And Abe, what are we hearing there?
DAVIS: Well, that's the sound that we played out of the speaker, and, you know, it created these fluctuations in air pressure. And when those fluctuations in air pressure hit the object they move the object a very, very, very small amount. And usually we can't see that, but it turns out that it does create this very, very, very miniscule changes in video. And if you look at the video locally - if you just look at some - one part of the plant in the image that you see, then you can't really get the sound from that one part. But if you start to combine all these tiny noisy signals from all across the surface of an object, then you can start to filter out some of that noise and you can actually recover the sound that produced that motion.
DAVIS: Well, that's the sound that we played out of the speaker, and, you know, it created these fluctuations in air pressure. And when those fluctuations in air pressure hit the object they move the object a very, very, very small amount. And usually we can't see that, but it turns out that it does create this very, very, very miniscule changes in video. And if you look at the video locally - if you just look at some - one part of the plant in the image that you see, then you can't really get the sound from that one part. But if you start to combine all these tiny noisy signals from all across the surface of an object, then you can start to filter out some of that noise and you can actually recover the sound that produced that motion.
(Buy/Rent/Layaway)
(SOUNDBITE OF SONG, "MARY HAD A LITTLE LAMB")
BLOCK: So deep in their, Abe, we're hearing "Mary Had A Little Lamb." I'm not sure how it happened, but we are.
DAVIS: Yeah. That's kind of what the plant heard - or really, more accurately, what the plant felt. All sound creates these vibrations when it comes into contact with an object.
BLOCK: Well, Abe let's listen to another example. You tried out human speech on a bag of potato chips. So here's what went in, in terms of the sound.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED MAN: Mary had a little lamb whose fleece was white as snow.
BLOCK: And here is what you extracted.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED MAN: Mary had a little lamb whose fleece was white as snow.
BLOCK: And here is what you extracted.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED MAN: Mary had a little lamb whose fleece was white as snow.
BLOCK: Were you surprised when that was the result, Abe, when you first heard that?
DAVIS: Yes. Well, sort of - I mean, that wasn't the first experiment that we did where we recovered human speech. I do remember that the first time that we recovered really clear speech, I had to keep double checking to make sure that I hadn't, you know, mixed up my signals or something.
(Buy/Rent/Layaway)
BLOCK: Abe, what are you thinking about when you think about practical uses for what you've done here? What do you - where does take you?
DAVIS: Most people - when they hear about this work, their mind sort of immediately goes to espionage and spying.
DAVIS: Most people - when they hear about this work, their mind sort of immediately goes to espionage and spying.
(Buy/Rent/Layaway)
BLOCK: The idea there would be you could take video that has no sound and somehow extract what the conversation was in that video?
DAVIS: Yeah. I mean, in some situations I think that you could do that. When people hear about what we do here, it's easy to imagine that it would just kind of work in any arbitrary situation, and that's not exactly the case. I mean, what we do is limited. I think there are a lot of things that we could potentially do with it, but a lot of that is stuff that's going to hopefully be fleshed out in future work.
(Buy/Rent/Layaway)
BLOCK: Well, Abe Davis, thanks so much for talking to us about your visual microphone.
DAVIS: Thank you.
BLOCK: Abe Davis is one of the MIT scientists who, along with researchers at Microsoft and Adobe, figured out how to recover audio from silent video.
Listen to the Story
DAVIS: Thank you.
BLOCK: Abe Davis is one of the MIT scientists who, along with researchers at Microsoft and Adobe, figured out how to recover audio from silent video.
Listen to the Story
Monty Henry, Owner
Additional Resources:
* The Creature From Jekyll Island: This Blog And Video Playlist Explains Why The U.S. Financial System is Corrupt and How It Came To Be That Way
* Dropping Off The Grid: A Growing Movement In America: Part I
Next-Generation Bug / Microwave / ELF / Spy Phone / GSM And Camera Detectors (Buy, Rent, Layaway) tinyurl.com/2eo8mlz Open...
— Spy Store Rentals (@MontyHenry1)
Nanny IP (Internet) Cameras, GPS Trackers, Bug Detectors and Listening Devices, etc, (Buy / Rent / Layaway): tinyurl.com/396jlw6...
— Spy Store Rentals (@MontyHenry1)
NOW, look in on your home, second home, lake house or office anytime, anywhere from any internet connected PC/Lap-top or Internet active cell phone, including iphone or PDA.
Watch your child's caregiver while sitting at a traffic light or lunch meeting, or check on your business security from the other side of the world. Our built-in hidden video features all digital transmissions providing a crystal clear image with zero interference. With the IP receiver stream your video over the internet through your router, and view on either a PC or smart phone. Designed exclusively for DPL-Surveillance-Equipment, these IP hidden wireless cameras come with multiple features to make the user's experience hassle-free.
NOW, look in on your home, second home, lake house or office anytime, anywhere from any internet connected PC/Lap-top or Internet active cell phone, including iphone or PDA: http://www.dpl-surveillance-equipment.com/wireless_hidden_cameras.html
Watch your child's caregiver while sitting at a traffic light or lunch meeting, or check on your business security from the other side of the world. Our built-in hidden video features all digital transmissions providing a crystal clear image with zero interference. With the IP receiver stream your video over the internet through your router, and view on either a PC or smart phone. Designed exclusively for DPL-Surveillance-Equipment, these IP hidden wireless cameras come with multiple features to make the user's experience hassle-free.
• Remote Video Access
• Video is Recorded Locally To An Installed SD Card (2GB SD Card included)
• Email Notifications (Motion Alerts, Camera Failure, IP Address Change, SD Card Full)
• Live Monitoring, Recording And Event Playback Via Internet
• Back-up SD Storage Up To 32GB (SD Not Included)
• Digital Wireless Transmission (No Camera Interference)
• View LIVE On Your SmartPhone!
Includes:
* Nanny Cameras w/ Remote View
* Wireless IP Receiver
* Remote Control
* A/C Adaptor
* 2GB SD Card
* USB Receiver
FACT SHEET: HIDDEN NANNY-SPY (VIEW VIA THE INTERNET) CAMERAS
Specifications:
Receiver Specs:
* Transmission Range of 500 ft Line Of Sight
* Uses 53 Channels Resulting In No Interference
* 12V Power Consumption
* RCA Output
* Supports up to 32gig SD
Camera Specs:
* 640x480 / 320x240 up to 30fps
* Image Sensor: 1/4" Micron Sensor
* Resolution: 720x480 Pixels
* S/N Ratio: 45 db
* Sensitivity: 11.5V/lux-s @ 550nm
* Video System: NTSC
* White Balance: Auto Tracking
• Video is Recorded Locally To An Installed SD Card (2GB SD Card included)
• Email Notifications (Motion Alerts, Camera Failure, IP Address Change, SD Card Full)
• Live Monitoring, Recording And Event Playback Via Internet
• Back-up SD Storage Up To 32GB (SD Not Included)
• Digital Wireless Transmission (No Camera Interference)
• View LIVE On Your SmartPhone!
Includes:
* Nanny Cameras w/ Remote View
* Wireless IP Receiver
* Remote Control
* A/C Adaptor
* 2GB SD Card
* USB Receiver
FACT SHEET: HIDDEN NANNY-SPY (VIEW VIA THE INTERNET) CAMERAS
Specifications:
Receiver Specs:
* Transmission Range of 500 ft Line Of Sight
* Uses 53 Channels Resulting In No Interference
* 12V Power Consumption
* RCA Output
* Supports up to 32gig SD
Camera Specs:
* 640x480 / 320x240 up to 30fps
* Image Sensor: 1/4" Micron Sensor
* Resolution: 720x480 Pixels
* S/N Ratio: 45 db
* Sensitivity: 11.5V/lux-s @ 550nm
* Video System: NTSC
* White Balance: Auto Tracking
Make Your Own Nanny Cameras: Make Tons Of Money In A Booming, Nearly Recession-Proof Industry!
Your Primary Customers Include But Are Not Limited To Anyone In The Private Investigator, Government, Law Enforcement And/Or Intelligence Agencies Fields!
* You Buy Our DVR Boards And We'll Build Your Products! (Optional)
* You Buy Our DVR Boards And We'll Build Your Products! (Optional)
Our New Layaway Plan Adds Convenience For Online Shoppers
DPL-Surveillance-Equipment's
layaway plan makes it easy for you to buy the products and services
that you want by paying for them through manageable monthly payments
that you set. Our intuitive calculator allows you to break down your
order's purchase price into smaller payment amounts. Payments can be
automatically deducted from your bank account or made in cash using
MoneyGram® ExpressPayment® Services and you will receive your order once
it's paid in full. Use it to plan and budget for holiday purchases,
anniversaries, birthdays, vacations and more!
DPL-Surveillance-Equipment's
Customers can now use the convenience of layaway online to help them
get through these tough economic times.
We all shop now
and then just to face a hard reality -- big credit card bills. However,
our latest financing innovation can help you avoid that. Find out why
more and more shoppers are checking out DPL-Surveillance-Equipment's
e-layaway plan.
If you're drooling over a new nanny
camera, longing for a GPS tracker, or wishing for that spy watch, but
you're strapped for cash and can't afford to do credit, do what Jennie
Kheen did. She bought her iPod docking station (hidden camera
w/motion-activated DVR) online using our convenient lay-away plan.
Our
online layaway plan works like the old-fashioned service stores used to
offer. But, in Kheen's case, she went to
DPL-Surveillance-Equipment.com, found the iPod docking station (hidden
camera w/motion-activated DVR), then set up a payment plan.
"It's automatically drawn from my account," she said. "I have a budget, $208.00 a month.
In
three months, Kheen had paid off the $650.00 iPod docking station. She
paid another 3.9 percent service fee, which amounted to about $25.35
(plus $12.00 for shipping) for a total of $687.35.
"You
pay a little bit each month," Kheen said. "It's paid off when you get
it and you don't have it lingering over your head. It's great."
Flexible
payment terms and automated payments make our layaway plan an
affordable and fiscally responsible alternative to credit cards.
1. Register:
It's quick, easy and FREE! No credit check required!
2. Shop:
Select
the items or service you want and choose "e-layaway" as your payment
option. Our payment calculator makes it easy for you to set up your
payment terms.
3. Make Payments:
Payments are made on the schedule YOU set. Check your order status or adjust your payments online in a secure environment.
4. Receive Products:
Receive the product shortly after your last payment. The best part, it's paid in full... NO DEBT.
More Buying Power:
*
Our lay-away plan offers a safe and affordable payment alternative
without tying up your credit or subjecting the purchase to high-interest
credit card fees.
No Credit Checks or Special Qualifications:
* Anyone 18 years old or older can join. All you need is an active bank account.
Freedom From Credit Cards:
*
If you are near or beyond your credit limit or simply want to avoid
high interest credit card fees, our e-layaway is the smart choice for
you.
Flexible Payment Schedules:
*
Similar to traditional layaway, e-layaway lets you make regular payments
towards merchandise, with delivery upon payment in full. Payments are
automatically deducted from your bank account or made in cash using
MoneyGram® ExpressPayment®
A Tool for Planning Ahead:
*
Our e-layaway makes it easy for smart shoppers like you to plan ahead
and buy items such as bug detectors, nanny cameras, audio bugs, gps
trackers, and more!
No Hidden Charges or Mounting Interest:
Our
e-layaway makes shopping painless by eliminating hidden charges and
monthly interest fees. Our customers pay a flat transaction fee on the
initial purchase price.
NO RISK:
* You have the right to cancel any purchase and will receive a refund less a cancellation fee. See website for details.
Security and Identity Protection:
DPL-Surveillance-Equipment
has partnered with trusted experts like McAfee and IDology to ensure
the security and integrity of every transaction. Identity verification
measures are integrated into our e-layaway system to prevent fraudulent
purchases.
Note: Simply Choose e-Lay-Away as a "Payment Option" in The Shopping Cart
DPL-Surveillance-Equipment.com
is a world leader in providing surveillance and security products and
services to Government, Law Enforcement, Private Investigators, small
and large companies worldwide. We have one of the largest varieties of
state-of-the-art surveillance and counter-surveillance equipment
including Personal Protection
and Bug Detection Products.
Buy, rent or lease the same
state-of-the-art surveillance and security equipment Detectives, PI's,
the CIA and FBI use. Take back control!
DPL-Surveillance-Equipment.com
Phone: (1888) 344-3742 Toll Free USA
Local: (818) 344-3742
Fax (775) 249-9320
Monty@DPL-Surveillance-Equipment.com
Google+ and Gmail
DPLSURVE
Twitter
DPLSURVE
MSN
Monty@DPL-Surveillance-Equipment.com
AOL Instant Messenger
DPLSURVE32
Skype
Montyl32
Yahoo Instant Messenger
Montyi32
Alternate Email Address
montyi32@yahoo.com
Join my Yahoo Group!
My RSS Feed
Phone: (1888) 344-3742 Toll Free USA
Local: (818) 344-3742
Fax (775) 249-9320
Monty@DPL-Surveillance-Equipment.com
Google+ and Gmail
DPLSURVE
DPLSURVE
MSN
Monty@DPL-Surveillance-Equipment.com
AOL Instant Messenger
DPLSURVE32
Skype
Montyl32
Yahoo Instant Messenger
Montyi32
Alternate Email Address
montyi32@yahoo.com
Join my Yahoo Group!
My RSS Feed
0 Comments:
Post a Comment
Note: Only a member of this blog may post a comment.
<< Home