Greeting Wizards!
Well super close and exciting super bowl game for the ad ratings.
Today I started to work on the WebRTC work to pipe the pose into the Ship of the Dead demo. This requires the camera feed and the accelerometer feed from the device (if the device has one, otherwise, it’s pretty safe to assume it’s not being held in your hand I suppose). The immediate problem I ran into is that since I changed the way video feeds via WebRTC to support… Safari… which is awesome, because it’s actually better now, I now have no way to send extra data in the frame since I am using normal WebRTC video stream.
We want to sync that accelerometer data into that frame so we can make best use of our machine learning data. We can send that data on the data channel but it’s kind of like sending one package via UPS and one package via Fed-Ex, it’s not really going to arrive at the same time. Generally around the same time, but enough time difference and enough randomness that you can’t really trust that this data matches this frame. We use that data to further match the overlay or pose of 3D into the video stream return pipeline.
Do people complain about this online? Sure. There’s a big discussion about it. And there’s a draft specification to fix it! Awesome! It was published, as a draft for comment, on February 4th 2021. The Unreal Engine is using a WebRTC library from 2018. So…. That’s yers away.
I’m actually not going to worry about it for now. Because the other thing I need to do is stop using Windows as a target. It turns out that Windows servers cost a lot more than Linux servers to run 24/7 on AWS or even Microsoft Azure. When we move to Linux (once we have some funding presumably and I have some time) we will need to make a few changes anyway since Unreal’s Pixel Streaming has some huge Windows dependencies on Windows Media Foundation. These are generally replicated in Nividia’s SDK on Linux. But code. Is the price difference worth it? Yes. The Ship of the Dead instance yesterday running all day (I hope you all tried it by the way!) cost $7.17 to run, and a Linux server would cost a little over $5.00 to run for the day.
The pipeline change from the Metropolis demo to the Ship of the Dead demo also requires the camera code on mobile to change from selfie mode to environment mode. Which is HTML code I LOVE WRITING.
Graeme.