Lockdown Dad Project: An AR Donut Hunt

Inspiration

On the first warm sunny weekend of 2021 my wife and I were looking for yet another local place to entertain our three boys (9/7/4). We stumbled upon a post promoting an augmented reality dinosaur trail.

"How hard is that to make?" asked my wife.

I had no idea. Over a cup of Earl Grey I started Googling, and piecing together all the different components that would be needed to build a similar experience. They all seemed to involve reasonably accessible technologies and needed relatively little "glue" to connect them.

Best of all it sounded like a fun project to do with the kids. It's nice to do something novel and educational with them, and I'm always keen to pull back the tech "magic curtain" and show them how the IT they use every day actually works.

I should note at this point that this technology stack involved here only works for Apple AR-enabled devices. I have an iPhone 11 Pro, and basically wanted to use that as the AR platform. I'm sure you can do the same with other tech, I'm just working with what I've got to hand.

Here's a preview of how it turned out!

The big idea

The experience I was aiming for in our AR project was as follows:

1. Visit a website to read the locations of several hidden items
2. Physically travel to each location and find a geocache containing a QR code
3. Scan the QR code and click through to a secret URL
4. Load a web-embedded augmented reality model
5. Take a photo with the model (optionally sharing it on social media!)

All we need to accomplish this is a website, a QR code generator, some geocache containers, a printer, and a few 3D models.

Yummy 3D models

To create the 3D models we downloaded the (free) Open Source modelling tool Blender.

But we also needed something to model, and dinosaurs would be too hard (plus... they've been done already).

YouTube to the rescue! We found an excellent series on Blender basics by Andrew Price (a.k.a Blender Guru) which explains how to create... donuts.

An augmented reality Donut Hunt? Sure, why not.

The YouTube tutorials are super easy to follow. I watched ahead of the kids one evening and picked out the key elements needed to create a couple of reasonably simple donuts without the modelling getting too overwhelming.

While the resulting models aren't photorealistic, I'm not sure how much of the additional detail would transfer through to the AR model anyway.

Small batch testing

Apple provide tools to convert 3D models into the ARKit format via Xcode and USDZ Tools, both of which are available on the AR Creation Tools portal. Once installed, running USD.cmd spawns a handy terminal with all the necessary paths configured to run the conversion tool.

I was keen to get an end-to-end small batch delivered to confirm that the idea was viable. Nobody wants to spend hours crafting donuts only to find that they can't be exported to reality, so it was important to prove out the toolchain.

We apparently needed to end up with a USDZ file, so I initially opted to export a USDA file from Blender. I assumed these would be simple and interoperable because... they only have one letter different.

Apparently we need to do this:

usdzconvert choc.usda choc.usdz -v

However, as we happened to be on an M1 Mac Mini (Apple Silicon) it took an additional prefix to get the converter working:

arch -x86_64 usdzconvert choc.usda choc.usdz -v

Success!

I dumped them onto a quick Azure public Blob store and hit the URL from my phone.

It soon became apparent that we had two problems:

1. The model didn't have its associated textures or materials.
2. Our donuts were hilariously small -- as in they would fit inside the hole of a Cheerio small.

The first issue was quickly fixed by scaling the model up. I'd followed Blender Guru's tutorial and sized the donuts using realistic measurements in Blender, but that information seemed to be lost in the AR conversion.

Addressing the texture issue took a little more work. Exploring the usdzconvert -h output showed that it was possible to set mesh materials via command line parameters, but these options didn't apply to some input file formats that apparently already include material data.

After a little (brute force) experimentation with some of those export formats the winner was GLB. This is a binary version of the Graphics Language Transmission Format, which helpfully wraps all object resources in a single file.

No command line material overrides are needed:

arch -x86_64 usdzconvert choc.glb choc.usdz -v

...and we have a working model.

Coding a website

Next we needed a website.

I wanted to use this as a learning experience for the older two boys, so we started off hand-coding the site in HTML using VS Code (now working on Apple Silicon!) With a few basics out of the way, we copied @mdo's Bootstrap example Cover and tweaked the content to our needs.

A lot of copy-paste later we had a series of pages with unguessable URLs.

Embedding the AR Quick Look preview into a web page is really easy. All you actually need is the rel="ar" property on your hyperlink and it seems to work just fine.

<a rel="ar" href="/assets/models/my-model.usdz">
  <img src="/assets/images/thumbnail.jpg">
</a>

Later I rewrote the site using Jekyll, largely for my own sanity keeping all the HTML pages consistent.

There were still a few gaps to address though, namely:

• Previewing the AR model only works over HTTPS
• You should serve the model with a MIME type of model/vnd.usdz+zip

Hosting everything in Azure

My main criteria for the website was that it should be cheap, secure, and easy to run. I have a favourite pattern for this: Azure Blob Storage and a CDN.

This costs virtually nothing to run, comes with free SSL, support for custom domains, and plenty of advanced control options. We also bought a shiny new domain: www.cheswickdonuts.com.

For the CDN we choose the Microsoft flavour because it allows a Custom Rule Engine. A custom rule lets us redirect all HTTP traffic to HTTPS, add some security headers, and potentially do a little HTTP rewriting/SEO too.

The last task was to set the blob Content Type for the USDZ files to ensure they were served properly. I should probably have done this using Azure CLI or PowerShell, but I was lazy and did it via 'Properties' of Azure Storage Explorer instead.

Geocache

With the tech ready we were finally to hide some geocaches!

We first created QR codes with links to our HTML donut pages, adding them to a Word document alongside a few printed instructions.

I ordered a couple of transparent film canisters from Amazon and we fit the instructions in each one, sealing the lids. Then, we hid them around the village noting their locations, and updated the website with details when we arrived home.

Finally, we posted to a few friends and local school groups to get the word out.

Try it

If you'd like to join in but can't quite make it to Cheswick Green (please don't break COVID-19 Lockdown), you can access all the AR models by visiting:

https://www.cheswickdonuts.com/cheat

The boys and I would love to see where our donuts have ended up - if you share an AR photo of a donut on Facebook, Instagram, Twitter, or LinkedIn and tag it with #cheswickdonuts, I'll be keeping an eye out for them.

The web code is available on GitHub if anyone wants to tinker or explain it to kids, though it's more complicated since I rolled Jekyll into the mix:

https://github.com/nathankitchen/cheswickdonuts

What's next?

I've got a couple of ideas for where to take this project, though it'll largely be led by how interested the kids (and their friends) are. It's probably been a one-time thing.

Ideas list for if anyone else wants to contribute:

• Get it working on Android
• See if we can put the donuts on people's faces
• Animate the donuts: make them spin or something
• Host user-submitted AR donuts?!
• Ramp up the donut photorealism

For now though, I hope you found this vaguely interesting and are inspired to have a play with some Augmented Reality yourself. You'll need a little tech know-how, but it's a shallow learning curve and it's quite rewarding seeing your virtual creations in a real life photo!