• Graeme Devine

WebRTC

So.


It’s been a productive week. It was really the week between Christmas and New Years when the meetings dropped off, but more than that, I had I think been learning enough about how to actually do what I said I could do that it became practical.

We use a method called webRTC to communicate between device (client, phone, you) and the server (cloud, AWS, my house). We use that because basically there’s no easy way to send data back and forth between the various mobile providers (AT&T, Mint, are there any more?) and servers because HTTP and HTTPS is basically the only trusted protocol that’s not totally blocked. Stadia, for example, uses webRTC and Google has their own implementation of it.


These servers can use intermediary servers to act as Switzerland as you establish a peer to peer connection (peer to peer here means direct connection). You can either host these on your own or reflect off several I’m sure highly over used public ones. This is basically how FaceTime works when you call from one device to another inside a Starbucks. I should know that, I was at Apple when we invented this stuff (but it was a lot buggier then, and boy, Starbucks wifi, let me tell you some stories about that Cupertino Starbucks and testing..).

For a long time I was stuck, it was easy to get the connection downstream trusted but the uplink wouldn’t connect. Then my dog Bella had an idea to use a socket connector on the Unreal server and that fixed almost everything. ALMOST.



At first it was awesome. I had two way communication all trusted and I was jubilant. I even tweeted so.

Then I worked out that my uplink was trying to send about 400mb/sec and that MBs a second and not KBs a second. You see, HTTP, you’ve got to encapsulate the message, you can’t just send a binary message, it’s TEXT. At first I was sending a JSON array (as in { 1: “244”, 2: “120”, 3: “0”, 4: “255” ….} except for zillions of pixels because when you are debugging stuff you absolutely want to use JSON because it’s easy to debug and readable right? And I need to sync the accelerometer data to the frame as well and send it as data in a dictionary format like this. Well, that’s good theory..

But two things, I was being an idiot ( that’s a given), it was huge, I was sending even the alpha channel, and it was sending the indexes, spaces even, and even though I naively thought to myself when I wrote the code “it’ll be fine, I’ll optimize that later because it’s so readable right now” it’s surprising just how many home networks get clogged when you try to stuff 400mb/sec over them.

I fixed that. At first I took out the alpha channel and just sent the pixels as one big hex string. At most that’s 2x as bad as just pure binary frames. But I kept on thinking, I’ve got a video feed here, I can surely send the video frame right? That’s when I delved into how to pipe media streams into base64 and boy, yes, you can do that, and it works, and now I’m not clogging the network.


The end result? I have a streaming Unreal video coming into my phone with the Main Street from Metropolis being rendered in Unreal while you can see my lovely camera upload on a cube floating in the street the same way a huge test cube in Unreal floats. It’s the smoke test. It’s like Apollo 10.


One funny thing. It did take me a week to notice that the red and blue channels were crossed over in the video streams. It wasn’t until I had a Starbucks holiday cup (because my coffee machine broke January 1st!!!!!) that I noticed it was blue on the stream and not red that I had that swapped…


So let me just say. It’s been productive. I’m really close to showing you all something awesome. This is the most complicated thing I’ve ever coded. I’m used to a video game having a lot of ground to cover before seeing a single pixel jump around but I’m not used to having to go build a moon before we get to send astronauts to it.

2020 was shit. But we launched QXR in October and that was the beginning of the turnaround. It took me a long time, longer than I thought, to work through my Mom dying suddenly last year, I kept on thinking what if this is it, I can’t write or code anymore? Living in the USA in this pandemic has been a descent into the trench of despairs I thought I had shaken many years ago, but you’ve been family to me throughout it all and I’m optimistic for 2021 (don’t jinx it Graeme, don’t jinx it.…).

Anyway. You are all awesome. Especially you dear reader who reached the bottom of this email and actually read the whole thing.

Graeme.