In my role at WP Engine, I have the opportunity to travel the world serving as an ambassador for new and interesting ways of using WordPress. I was recently in Raleigh, North Carolina, at All Things Open talking about the possibilities of a CMS in the spatial computing future. This was a lofty topic for 45-minute talk, so I thought I’d share a more in-depth look at a new way to build Extended Reality (XR) experiences with WordPress using a headless approach.
The Headless Craze
I think many of us see headless primarily as a way to make our sites faster and decouple them from the CMS that controls our content. While true in some cases, this is probably not the most compelling lens to view this concept in when we consider what this means for native apps and Augmented, Virtual and Mixed Reality, more broadly defined as Extended Reality.
In order to understand how WordPress can power XR experiences we must take a step back and understand what key features developers and designers desire in interactive applications.
What is the core appeal of a CMS?
Zooming out and looking at what a CMS is at its core highlights a few key features that make my future-looking imagination race. A CMS can be broadly defined as:
- post/page data
- meta associated with data like users and posts
- a secure user system
- a rich content editor
- extendability through plugins/themes/etc.
WordPress shines brightly across this entire spectrum and, pairing that with open-source flexibility, you can build whatever you can dream. User systems and content are key requirements in interactive and media-rich applications such as XR. An open-source solution like WordPress gives developers greater freedom and more control over the source code and the software meaning they can tweak it to perform how they want, unrestricted by licensing. Another level of freedom was achieved when the new Gutenberg editor was merged into WordPress 5.0, further democratizing the power of the world’s most popular CMS.
Possibilities with the New WordPress Editor
We’ve spent a lot of time thinking through the front-end benefits in a unified way to control markup using blocks. I think one aspect of the new WordPress editor that has largely been overlooked in front-end development is the data structure of the blocks themselves.
Pulling data via the REST API typically returns the final markup of a post. This makes it hard for non-html based applications to parse or use the data. You can certainly sanitize the final output, but it doesn’t provide confidence going forward and often removes important context.
Before a post renders markup in the front end, there are valuable attributes used to determine that final markup. These options are often just as useful to a non-browser application as they would be for a browser-based experience; it’s just a matter of making sure the content is sanitized for your purposes.
Let’s imagine we have an audio block with the attributes of src, loop, autoplay, etc. Those attributes are just as important to a non-browser application that expects the same decisions to be made at runtime. By querying a Gutenberg post, one can pull the audio file and also the attributes of how it should behave, leading to the application being unified with browser behavior.
In the not too distant future, the physical and digital worlds will continue to blend together. As younger generations increasingly interact in ways other than through screens, keyboards, and mice, we should probably begin to create content with that future in mind. Do we really want all that markup saved? Is it even important? If it is, maybe that attribute should be saved, keyed, and exposed via JSON. Markup is really only useful to the front-end rendering.
Sound-Only AR Blog Reader Concept
Bose is pioneering an interesting approach to augmented reality. They have quickly been updating their product offerings to include 6 Degrees of Freedom (6DoF) sensors inside their headphones and sunglasses to power sound-only AR experiences. This gives us three important mechanics for XR; 6DoF head tracking, private sound experiences, and gesture recognition. I decided to use this technology to illustrate the potential in the data structure of Gutenberg blocks.
Using the Gutenberg Object Plugin open-sourced and built by Roy Sivan and contributors, I was able to expose the attribute data in a REST friendly way. I was then able to shift my focus on building mechanics and an MVP application to read a blog without sight. The application puts buttons in 3D space around the visitor. To the right, a “Next Post” button. Directly in front of them, a post body button that plays back a reading of the post data. The voice-to-text technology was made possible by using the Amazon AI Plugin for WordPress.
When double-tapping on the frame of the glasses while facing the post body, a voice prompts the visitor to use a nod gesture to playback the post audio. Within 24 hours of sketching the idea, I had a working proof of concept and I attribute that largely to not having to figure out how to prepare my data. Blocks and JSON did it all for me!
The following image is a sketch of the concept along with the final product.
Blocks in Traditional Augmented Reality
Spatializing blocks was interesting to me, so I built another proof of concept that demonstrates how 3D prefab objects in a game engine can be associated with 2D blocks on a webpage. Below is a screenshot of a Gutenberg post rendered around my hotel room.
While a literal 1:1 representation of a webpage in AR illustrates that we can couple our design decisions in and out of a browser, I think we can go a step further. A block can represent many things; so why not use custom blocks to abstract 3D ideas?
Non-Standard File Types and Abstracting 3D Objects
HTML has a finite list of file types allowed out of the box. That is not to say we are limited to only those options. WordPress is flexible enough to extend to others by simply adding allowed types.
I’ve been building a music game called Broken Place that allows players to interact with real-time music in AR. The sound engine is running an open-source programming language called Pure Data (PD) that has a very specific file type that only it can read. In WordPress, I am managing post content in a custom post type called “songs” where players can upload their PD patch. Players simply log into the website as a contributor user role and upload their patch. The song post is queried from the native app and the patch is loaded.
The content submission options open the potential of allowing users to design their own layouts for music interfaces. Imagine in the graphic below, that each of the 3D knobs pictured were associated with a Gutenberg block called “Knob” and each of those knobs have an attribute associated with the sound message being sent to the audio engine. Content creators could determine how the sound messages are mapped in a grid using the column block. When you nest blocks within a column block, the REST response also nests the objects as children of the block and logic can be applied using the JSON structure.
Narrative Design Potential
In post data, we are telling stories; some informational, some personal, but they serve as thought capsules. I dream of a future where posts are much more than just text and images on a screen. Spatializing content provides amazing opportunities for narrative design. I’ve seen this potential represented in the AMP Stories and hope that structure could help inspire stories told within 3D physical spaces.
Imagine the same post structure of AMP stories, but for 3D spaces that you can drop in your home or office. Mozilla Hubs is leading the charge in collaborative XR spaces open to all devices and I see interesting potential in combining a WordPress narrative design for Mozilla Hubs to allow shared storytelling experiences.
I’m excited to continue exploring new ways of using headless WordPress to power XR experiences and help usher in an entirely new era in how we interact with and create digital content.
