This project is firstly inspired by the differences between the physical CDs and the digital albums.
Before, we got a sense of ceremony by purchasing the physical CDs, and had more physical interactions and feedbacks when playing the music.
While now, with the digital albums and streaming music, we listen to songs in a much more convenient way, but this sense of ceremony is missing. Why is that?
Perhaps we may look into the devices we use for playing and listening to the music.
We used CD plyers and speakers for the physical CDs, and radios for the music broadcast.
Then, we had the MP3, the iPod, or the laptop for listening to the digital music or digital albums.
And today, we are more likely to put on our headphones and listen to the digital albums or streaming music on our smartphones.
There’re some supporting data about this trend. According to the RIAA, the proportion of streaming music revenues in the U.S increases year by year, while the physical one declines to only 7% in the first half year of 2020.
And from Bankmycell, the total number of smartphones users is huge in the US, and the smartphone to population penetration is also as high as 79.1%. And even for the 18-29-year-olds, 94% of them have a smartphone.
The online music platforms and smartphones dominates most people’s ways of listening to music nowadays.
So let’s get back to the device, the smartphones we use for enjoying music.
Does that really mean that the experience of listening to digital music from smartphones should be really poor in comparison with the physical old ways?
Here are some questions.
Q1: How we use the smartphones to enjoy music?
A1: We just simply open the app, tap the music, put on headphones and listen. Some people may also see the lyrics or watch the music video.
Q2: What smartphones do when playing music?
A2: It just simply act as a media player with a display showing some basic information.
Q3: But what smartphones can do today? It has the camera systems for seeing the world. The powerful chips for comprehending the world. And it offers more interesting way of storytelling with a display.
Therefore, I come up with the idea of AR that makes full use of the smartphones to enhance the online music listening experience.
Here is some further research on the streaming music and current AR situation.
About the streaming music, one drawback I found from the paper “Music Everywhere: Setting a Digital Music Trap” is that, streaming music makes more people less contemplative in music, where they just hear a song distractedly and roughly.
So, one goal of my design should be to provide a more immersive experience and let audience be more aware of the stories behind the song.
As for the current AR situation, a paper points out that the disappointing AR experience lay on the unsuccessfully integration of digital online and offline customer, and the lack of thoughts about actual customer needs.
So the seamless integration of the virtual and real world is another key point.
Here are some precedents that adopted AR to the music or album.
A Chinese singer, Zhangjie, has a album called 未Live that offers the AR content. It is based on the book image recognition and can play the AR MV. It does offer additional stories and gives both physical and digital feedbacks. Yet, it is not immersive at all, and the integration of real and virtual world is not smooth.
Another precedents is Spark AR, which applies a face filter that changes with the rhythm or demonstrates simple fixed location animation. It is visually appeal at the first glance. But, it doesn’t offer additional information and doesn’t match the music content.
So, here comes to my design.
It offers a smartphone AR experience provided by the streaming music platforms.
It has two parts, one is the album unboxing that provides additional information in an interesting way, the other is enhancing listening experience that tells the story behind the song.
Details are shown in the user journey map below.
Then, I try to make a prototype. I choose Unity and iPhone with ARKit, where Xcode could help to build an iOS App.
Apart from learning the development setups and the basic knowledge about Unity and C#. I build those small prototypes for testing the techniques, including:
~ using ARKit to detect the surface and place an object;
~ using Universal Rendering Pipeline for post-processing and control it from the script;
~ using particle system and gravity to generate different effects like waterfall;
~ building animation and control it by scripts;
~ building the camera pointing triggered animation.
About the prototype, I choose Troye Sivan’s newly released album “In a Dream” and my favorite song “Easy” in this album.
The song is in an 80s disco style, but the emotion and story is forlorn and heartbroken. Two elements, water and fire, go through the whole song, and they could be used in the AR prototype.
Details are shown below with a demo video.
Finally, about future works.
Of course the prototype needs a lot of visual improvements and so it is with the AR listening experience.
More fun and magical experience and means of storytelling could be introduced, like the varying and more interactive experience based on different spaces and time.
Further, it may even work with the IoT smart devices and be designed in a universal modular way so that it works with other albums and songs by simply changing the elements and their animation.
