Video latency investigation

Following my first FPV flight, I wanted to get a better idea of the latency of the video feed. How does the Oculus Rift compare to a regular RC FPV system? What's the latency of different cameras and displays?

In this experiment, I only consider the image capture and display stages, not the wireless transmission stage involved in an FPV system. Why not? Because I fried my 5.4Ghz transmitter so I'm waiting for a new one!!!

OK, so the experiment is pretty straight forward:

• A display (TV or monitor) shows "in real time" what a camera sees
• I drop a ping-pong ball in front of the camera
• At the same time, I film the actual physical event AND what the display shows with another camera
• Then by watching the recorded video, I can spot the exact time when the ball bounces against the floor in real-life. Then I can do the same thing a few milliseconds later when the display shows the event. The difference in these two times gives me the latency of the entire image-capture-and-display system

The camera I use for filming the whole scene is a Go Pro Hero 3 set to its minimum resolution: WVGA 848x480 and maximum frame rate: 240 frames per second. This rate is huge, each frame of the video represents only about 4 milliseconds! This sounds great in theory in terms of precision but in practice things are not that clear when it comes to measuring the exact time of the "second" bounce on the TV/monitor. Indeed, due to the nature of image sensors and LCD screens, images of a rapid event look blurry. This can be seen on the following sample footage (resampled at 25 fps so it appears slow-mowy):

Here are the parts I've combined in the tests:

Sony CCD. A traditional analog camera used in FPV systems
GoPro Hero 3. Also used in FPV systems (analog live output)
EasyCap USB adapter. Analog video input for PC
Sony PS3 Eye. Often used in augmented reality or vision applications
Raspberry PI camera
Funai LCD TV. Old-school 4/3 TV
LG PC Monitor
Oculus Rift

And here are the combinations I've tried and the corresponding results. Once again, these are pretty "crude" latencies, but they allow comparison of various pieces of equipment.

Setup	Average latency in ms
Sony CCD + LCD TV	37
GoPro Hero 3 + LCD TV	54
Sony CCD + EasyCap + PC + Monitor	120
GoPro Hero 3 + EasyCap + PC + Monitor	143
Sony CCD + EasyCap + PC + Oculus Rift	123
GoPro Hero 3 + EasyCap + PC + Oculus Rift	131
Sony PS3 Eye (640x480 75Hz) + PC + Monitor	160
Sony PS3 Eye (320x240 187Hz) + PC + Monitor	58
Pi Camera (default HD) + Raspberry Pi + LCD TV	98
Pi Camera (160x120) + Raspberry Pi + LCD TV	90

In conclusion:

• You can't beat a  traditional "analog" RC FPV system as it's latency is around 40 ms. Note again that we're not counting the wireless transmission here, just the image capture and display.

• Adding a PC between the camera and the display (USB capture + rendering) adds 80 to 100 ms... In total we get about 150 ms. That's 3 to 4 times the latency of an RC system!

• The webcams I've tried (i.e cameras that provide the computer with digital images directly) don't do very well either. Decreasing the resolution can help though. That's the case for the PS3 Eye at 320x240.

Once I receive my 5.8Ghz video transmitter, I'll be able to do a couple more tests with it (inserting it between the camera and the display). We'll see how much it adds in terms of latency (I suspect not much).

My first Oculus-based FPV!

At last, I was finally able to do my first FPV flight ever! Not the usual FPV system here as I'm using an Oculus Rift for display. The camera on the tricopter is a GoPro Hero 3. Both of these have a huge field of view so the feeling of immersion is just great!

On the down side though, this requires quite a bit of hardware (a laptop, an USB video capture device, the Rift itself, batteries...) and software (home made in my case). So this looks quite messy and not (yet!) as practical as a commercial out-of-the box FPV solution, but I'm working on it. There's also a bit of lag in the video feed most probably due to the video capture stage. It's still flyable though.

The reception/display set up

Here are some of the videos recorded by the GoPro. Note that these are not what I'm seeing live when flying. The transmitted video is noisy and smaller in resolution.

Testing the GoPro Hero 3 on a tricopter

Here are my first videos taken from a GoPro Hero 3 (black edition) mounted on a tricopter (RC-explorer 2.5 style). These are still visual flights here. It was getting dark, so the quality is not great.

I managed to lose one of the propellers at the end... (ahem!).

Getting closer to FPV!

The standard waterproof GoPro case is very nice but it's quite heavy (about 105 grammes with the mount at the base) and you can't use the live video output cable at the same time. Nonetheless, I tried to attached it to the tricopter just to see how it'd fit...

In this configuration it would work fine for taking videos but it wouldn't be possible to fly FPV. So I designed an alternative casing out of foam (the kind used to protect things in transport boxes). Pretty crude but it does the job...

I'll try to come up with a safer way of fixing the camera to the tricopter, something less rubber-bandy! I couldn't resist doing a quick flight test (in my living room. Don't try this at home!). It's looking quite promising: there doesn't seem to be much vibrations.