Johannes Kopf, a researcher at Microsoft, has recently published a stunning video about a software project he seems to have participated in. The video is about the automatic creation of hyperlapse videos:
We present a method for converting first-person videos, for example, captured with a helmet camera during activities such as rock climbing or bicycling, into hyperlapse videos: time-lapse videos with a smoothly moving camera.
So we are now only speaking about first-person videos. As videos created by a helmet camera might be very long and (as he accurately described it) "dead boring", you want to speed that up. A time-lapse would be a subsampling to every n-th frame. Those might be very shaky and hard to watch. That means you would at least want some image stabilization.
Microsoft Hyperlapse
But the software they created does more. Much more.
Reconstruction
First, it reconstructs the geometry of the captured environment. That alone is absolutely awesome. They apply a group of techniques called structure from motion.
Path planning
Then they plan the path of the camera. This is a 6-dimensional problem for every point in time where you want to get an image. The 6 dimensions are:
- (x, y, z): Position of the camera
- roll: horizontal, vertical angle
- pitch: up, down view angle
- yaw: left, right view angle
In case you have trouble imagining roll, pitch and yaw, you should take a look at the following image:
By ZeroOne
The chosen path should meet several criteria:
- It should be smooth in space
- Every path position should be close to input positions
- The path should be short
- Rotation should be smooth (that is what makes videos "shaky")
- The rendering quality should be as high as possible
The first three criteria were achieved by spline fitting.
Rendering
That step combines several input frames to render the desired camera image.