Blog

My interest in animation and music visualization goes way back. In the early 90’s, I worked on shows for the Bays Mountain Planetarium, producing soundtracks, introducing their artists to Autodesk Animator, and even creating music-synced laser visuals on the dome for their popular Star Fest show. I worked as an animator at ZFx, Inc until the Internet as we know it came along, and I was saddled with the responsibility of being their offshoot Tri-Cities Connection’s Webmaster. I also created a music visualizer for the Sonique Music Player.

Now, I’m happy to be revisiting that interest with a business partner and long time friend, Adam Houghton. We noticed that independent artists were uploading their music to YouTube in droves. That may seem odd, since YouTube is a video site, and usually, the videos have just a static image that shows while the music plays. Adam, a member of the electronic duo Splatinum, saw an opportunity here. What if artists could cheaply and easily add interesting visual content for their music before uploading? He put out a call to his sizable community of artists and friends to see what they thought. The response was an overwhelming YES!

He began to put together a proof of concept built on Three.js, and soon put out another call, this time for artists to play with this toy and give constructive feedback. I was a member of that group, and like the other participants, I was impressed and intrigued at the possibilities. But Adam needed some help building this application out. He works full time as a multimedia content strategist for Amazon, but has always had a strong entrepreneurial spirit. In the past we’d worked together on 3D graphics and artwork, so when he asked who’d like to help, I was the first to raise my hand.

Taking the original demo and refactoring it into an application with an architecture that can be built out is my specialty, and soon, I had it under source control and looking like a modular enterprise app on the inside. Sinewav3 uses Three.js / WebGL, PureMVC, Socket.io, React, Node.js, Firebase, FFMpeg and a number of other supporting libraries to allow artists to create unique visuals for their music, render HD videos, and upload them to YouTube. Currently, it is a work in progress, but you can twiddle various options like adding different objects to the scene, tweaking their appearance, cloning them, tying their position to different bands of the audio spectrum or an LFO.

Building in the Open

The app is still very much an early-stage proof-of-concept. And we’ve decided to develop it ‘out in the open’ rather than build it in ‘skunk works’ mode, so that we can rapidly iterate and get feedback early on. To that end, I’ve been chronicling the technical challenges we’re facing and overcoming on a semi-regular basis.

Adding users will not be a major issue, as we’re building on Firebase, and I’ve already created a User Profile system, which allows user/password as well as social sign-in. And I’ve added a feature to the application that allows you to snapshot all your settings to the browser’s local storage and recall them by refreshing the page, so storing projects in the database and returning to them later will be easy to implement.

However, before taking on a user base, we needed to do a functionality spike all the way through creating an HD video that could be uploaded to YouTube. That led to the following series of articles on DZone, where you can read more about our journey at the code level…

• Should I Render Three.js to Video on the Client or Server?
• Creating Video on the Server in Node.js
• Persistent Connections With Node.js and Socket.io
• Attempt to Render Thirty Frames per Second and Send Them to the Se
• Break Free of the Realtime Jail: Creating Audio-modulated HD Video With Three.js

I’ll keep the above article list up to date as we continue to build out the application.

The Future

Eventually, you’ll be able to create an account, upload some music (and assets like logos, your own meshes, album cover art, etc.), put together a fun and unique video, and upload it to YouTube all in one go.

But beyond that, we plan to add an API, creating a marketplace that allows developers to add visualization plugins to the system, and get paid each time someone uses them in a video. Likewise, we’ll also allow 3D modelers to add custom meshes and textures that expand the universe of options available to the user. If you’re an artist who’d like to work with us to add features or test the system, please do get in touch!