Blog

Mind the Gap

Since mid-2016, in between client projects, we’ve been laboring away on Sinewav3. Things have changed dramatically since our last report.

In case you missed it, Sinewav3 is a music visualizer for artists that runs in the browser and is powered by WebGL and Web Audio API. It will allow them to create 3D worlds that respond to their music and generate HD videos for upload to YouTube and other video services.

The demo we had up and running as of that last post, although able to create a spiffy music-responsive 3D environment with multiple things going on, had some severe limitations which have since been addressed, including:

1. It couldn’t generate a video

Ultimately, not a big issue; we figured out how to do that. In short, we found that rendering Web Audio modulated WebGL on the server was not a viable option. Nor was realtime rendering and simultaneous transmission of frames to the server for assembly.

The result was that we must offer realtime visualization for preview during project development, and non-realtime rendering and frame transmission for video production. The upside is that we can easily do 60fps HD videos with the non-realtime approach.

Rendering is disabled during the current plugin developer preview period however, since with realtime visualization it isn’t necessary, and could be costly in terms of server time and bandwidth. Also, that’s an area where a lot of work is ongoing.

2. A paucity of features

The limited number of things you could do in your 3D world meant most videos would end up looking the same. This problem had the biggest impact on the direction of the whole endeavor. We were building something that could keep us on a hamster wheel of feature development indefinitely. As soon as we reached a threshold of feature-richness that truly unique videos could be produced, users would inevitably exhaust the space and things would start to look the same again.

Committing to an ongoing investment in feature development was scary. We’ve sidestepped that issue entirely, by adopting a plugin model with revenue sharing for plugin developers. More on this in a moment.

3. One scene per video

Although you could tweak the visuals live, there was no way to entirely change them between parts of the song. If you have an eight-minute dance track, that’s going to get pretty boring by the end.

We solved this problem by giving a project multiple “worlds”, each of which has its own camera, lighting, and objects. The user adds any number of worlds to their project and any number of plugins to each world. As the video is rendering, we can switch from one world’s camera to another at a frame’s notice. The active world’s set of plugins can create and/or manipulate objects, lights, and the camera within the world’s 3D scene.

We’re in the process of creating a switcher that will allow you to record the timing of the scene changes as the music plays. In the meantime, you can open the settings menu during the visualization and switch between worlds, tweaking their plugins’ settings live.

4. Limited customization

There was no way for users to supply their own models or textures. Obviously, we could never provide enough stock models and textures to keep things perpetually interesting. But if we weren’t careful, the cost of transferring and storing a user’s custom assets could well outweigh any profit we might make from the videos they generate with them.

As a result we implemented an asset management system that tracks audio, images, models, cube maps, and fonts. We know exactly how much space is taken up by a user’s assets and display it prominently on the Assets page. At launch, the amount you can store will be governed by your plan. There’ll be a reasonably generous free plan that allows users to evaluate the system and create eye-catching watermarked videos to lead viewers back to the site where they can create their own videos.

5. Memory management

The underlying WebGL support was (and continues to be) provided by the popular Three.js library. It also used the dat.gui menu system which appears in most of the Three.js examples.

In a one-page example on the web, you only have to set things up and allow people to tweak some exposed parameters. Memory management is rarely an issue.

But in a system where a user might dive into one project, visualize and manipulate it, and then navigate to another project to work on a different set of parameters and 3D scenes, you must make certain all allocated memory is freed. This is where we began to have problems.

It turns out that you simply cannot simply remove a dat.gui instance and create another. Apparently it’s only intended for one-page demos. And in your Three.js scene, you must go to great lengths to free up memory associated with scene objects.

Consequently we ditched dat.gui while building a proper application shell with Angular and Material Design. The visualization and audio analysis engine is still built with PureMVC, but now uses the State Machine utility, which allows us to maintain rigid control over the memory state as the user enters and exits the visualization or rendering states.

Also, with our Plugin API, rather than have the developer add things to the 3D scene willy-nilly, possibly not cleaning up after themselves, we have the API keep track of things so it can clean up after you. While the developer should write cleanup code if they knowingly allocate memory that the API isn’t aware of, usually it isn’t necessary at all.

Now, the memory footprint is fully under control.

6. No way to work with 3D text

No longer a problem! If you want to put some 3D lettering into the video, you can easily create your own 3D font and upload it as a Font asset. Use the stock Flying Logo plugin to place it in the scene, texture it and set it spinning. We encourage plugin developers to do fun things with text as well.

Call for Plugin Developers

At this point, what we’d like to do is invite developers to create an account, play around with the examples, and have a crack at building a plugin. Tell us what bothers you about the current interface, what you’d like to see, and any feedback about the system in general.

What’s in it for you? I’m glad you asked.

During this preview period, build one viable commercial plugin or two useful example plugins (other developers can see the code) and get free rendering of your own video projects for three years, in addition to revenue sharing whenever someone uses your commercial plugins in a paid video production.

In case you’re wondering what’s involved, we have the simplest plugin system ever devised ™. Writing a Sinewav3 plugin requires no special tools or build process. A plugin is merely few functions (setup, render, and optionally destroy), entered through the browser. Also, you can define settings that allow the user to customize your plugin’s behavior and appearance at runtime.

Below is a video from about six months ago, which outlines the basic areas of the system and how to create a plugin. A few things have changed in the interface since then, most notably for developers – use of the beautiful Ace code editor, which understands Javascript intimately, and not only colorizes the code, but gives you margin hints about your errors. Also, two quick intros have been added to the landing page that explain how to create a visualization project and how to create a plugin.

When you’re ready to give it a whirl, just head over to the Sinewav3 app, create an account and see what you come up with. And you can join our Slack Team to participate in the discussion about moving the platform forward.