Anima

Here is an example of the best 360 video quality we can pull off at the moment:
https://youtu.be/WrMnOPckXng
Want details on what we’re doing here?

The raw materials:
We set up a Rylo 360 camera on a tripod and shot a full take where I walked to both spots and talked. Then we put a Panasonic Lumix DC-GH5 camera on the same tripod and shot 2D footage of me at the 2 spots. Then we shot 2 other inserts (me on hill, me in window area, and a bottle on the ground).
(also, you can see Rick holding the camera in the 360 footage, but that was just to get the lapel mic audio recorded)

The editing details:
I exported the rylo footage to an ipad, and exported that as 360 video to google drive (3840×2160 91,436 kbps, 23.98 fps. 1 GB). Rick converted the 2D camera footage from .raw to .mov files (3840×2160 434,655 kbps, 23.98 fps. 10GB total for the 4 files).

In Adobe Premiere Pro CC 2019 i started with the rylo’s equirectangular footage, and put the 2D shots in layers above. For each of these, i used the “VR plane to sphere” effect and used these steps to dial it in:

1. First move the anchor point, to get the 2D footage into roughly the right area (if you don’t move anchor first, other distortions will get weird in a moment).
2. Adjust the effect’s “Scale (Degrees)” to get roughly the right distance away from camera (this bends the perspective lines in it’s frame)
3. Use the “Rotate Projection” controls to finish placement of vid. (Don’t use “Rotate Source” controls because it swivels the frame around the anchor point, but doesn’t apply the “scale (degrees)” fisheye distortion from your central perspective).
4. Switch to VR view with “Toggle VR video display” (click “+” in the monitor window and drag it to the bar if you don’t have it already). Drag around to center your perspective on the 2D footage (this minimizes distortion distractions/fake-outs)
5. Toggle the timeline layer on and off, or lower transparency, to try and get good distortion fit.
6. hot tip: the feather is effective! I cranked it to the max!

Finally, separately, I added “Video Effects > Adjust > Levels” to try and match original footage saturation and contrast. (first Gamma, then RGB Black Input or RGB white Output)
… (i tried adding “Color Correction > Color Balance (HLS)” effect, but it glitched everything out. erg. Some of the tutorials i found suggest against any normal effects, because they may mess with the seam or the whole VR distortion)

The compression details:
The first export (with no “injection”. You used to have to download a free tool from Youtube to “inject” meta data so their machines would know to interpret it as 360. I guess check marking “VR” in the Adobe export is enough now)
https://youtu.be/1rWYdP6lY0I
(h.264, 12 MBPS, 162mb, 23.976fps – 3840×2160)

Then I exported the exact same file, but boosted the MB per second from the default 10 to the max 30. (I think this is the key to why our initial rylo and ricoh 360 videos looked so aweful).
https://youtu.be/x0TIo261v4c
(h.264, 30mbps, 399mb, 23.976fps – 3840×2160)

Then I switched to h.265 compression (because some website recommended it), but it limited the max mbps to 25mbps. I started this export around 6pm, and it completed around 2am. 8 hours. yikes!
https://youtu.be/ZwstVajtg0s
(h.265 (HEVC): 25,530 kbps, 399mb, 23.98 fps – 7168×4032)

Finally i imported the .mp4 from the last step and composited the Tvori footage over it.(also took 8 hours).
https://youtu.be/WrMnOPckXng
(h.265 (HEVC): 30,999 kbps, 406mb, 23.98fps – 7168×4032)

A little extra about VR animation piece(s):
There is this great program “Tvori” which lets you animate 3D objects extremely quickly in virtual reality. I think of it as the ideal way to make “animatics”.

For the “fire animation” piece, i used the maximum Tvori settings for exporting a 360 video (3840×1920, 873 kbps, 24fps. 1.15 MB), and Color Key’d the grey background out in Premiere (plus a little “Ultra Key” to get a band of grey gradient) (plus a “VR Rotate Sphere” effect to move it around into place). It looks like I missed a small white dot (hot spot) up near the top. oops.

For the “buildings animation” piece, I had a ton of problems keying. I made an inverted normals sphere with green texture in Maya, then brought it into Tvori to try and hide the gradient of the sky. But the sphere caught the light/shadows (and wasn’t smooth. my bad?). So I gave up on the whole 360 video export (because I’d already animated each thing. I couldn’t turn off “the floor” in tvori, so it was making the bottom half of the frame a much lighter green. I would need to delete all the animation to move everything up away from the floor, and reanimate. pssh.). Instead I exported the highest res 2D video I could from Tvori (3840×2160, 1225 kbps, 24fps. 4.39 MB). Took that into After Effects to key out the faded floor (using “Color Range” and eye dropping until i got it all. plus a Solid layer below it with the same color as my green sphere sky). Exported that (3840×2160, 3858 kbps, 24fps. 14.9 MB) and took it back into Premiere to comp like the other 2D footage.

Final thoughts:
There are lot of rough edges here, but I think it’s a huge success. We can get pretty decent 360 video quality out of what we have, and putting 3D objects into it possible. woo hoo!

Sorry for the quirks in there (audio overlaps. when i looped the 360 background video to fill time while the 2D footage played out). The second “talking spot” really doesn’t line up well with the building awning. hmmf. Need to be careful about what size frame we shoot to insert, and what items are at it’s edges (like, i think the dead grass and grean leaves edges look fine. but the hard lines of the building really stick out as bad).
I need to find some way to crop the 2D footage to make all this easier. But taking it over to AE to mask things out sounds like a pain. Like the “window” footage. if that was matted to just be the square window area, i think it’d be much easier to fit into place. trying to match the distortion one the lift side, roof slats, pillars on each side, and the window middle bar on the right: was impossible.

+ I”m not clear why the Rylo cam (the 360 background footage) appears to wiggle slightly several times. maybe tripod needed more weight on it? other ideas?
(if you just look at the seam for one of the HD 2d video clip overlays the whole time, you can really see it).

+ Also, Might be good to try and match the lower contrast grey of the rylo? I think i had to dim the HD footage because it was right on the edge of blowing-out white. Which is maybe how we tend to shoot HD video. Not sure how to dial this in. (fstops will mess it all up yeah? and might have polarizing filters to consider, depending on light? ugh)

+ might be good to aim the left edge of the rylo at minimal-detail part of scene (as it gets blurry from there forward). And/or stage things so nobody ever walks past left side of it.

+ not sure how to best shoot replacements for the above below points. (like, cover the tripod). We could just tilt the camera back 90 degrees to get sky, but moving clouds will be a nightmare to comp in. Maybe we could build a giant tripod to get the camera in roughly the same spot aiming straight down (and hopefully rotate it to avoid shadows). or a gib arm? i dunno. hmmf.

+ maybe we should settle on ideal height to center of 360 lens and do our best to measure and match that for everything else.

anywho. Just wanted to record some details before I forget. If you have any questions, please feel free to ask.