Pens and stylus’ have been employed as computer interaction devices for quite some time now. Most commonly they were used along with peripheral graphics tablets to give a more natural flow to the artist or designer than a mouse could muster. With the release of the Surface Pro hybrid laptop by Microsoft in 2012 they brought a digital pen along to party that could work directly on the screen. It was intended to bridge the gap between the demands of desktop software and the tablet touch screen form factor. In a mouse and track-pad free computing environment how better to access the finer details that your thick fingertips couldn’t manage.
The advantages for the artist become quickly apparent. As the Surface Pro has evolved the graphical power has gotten to the point where it’s a completely competent sketching, drawing and design platform. But there’s another group of artists for whom the digital pen has an awful lot of potential, and that’s the musician.
This is probably most joyously demonstrated by the Windows 10 app Staffpad. Staffpad takes the idea of writing music completely literally and presents you with a blank sheet of manuscript paper and asks you to start writing. Combining the digital pen with hand writing recognition Staffpad is able to interpret your hand written notes into digital MIDI information directly onto a score. It can then be played back through a virtual orchestra. It’s a stunning piece of work and remarkably fluid and creative to use.
Most of us approach music creation in a more sequenced format. The pen has a lot to offer here as well. Entering notes into a piano roll immediately comes to mind, as does the editing of notes, the trimming of clips or moving blocks in an arrangement. Consider drawing in track automation, with a pen rather than a mouse. How much more fluid and natural could that be?
In many ways the pen feels like it’s simply replacing the actions of a mouse – but it doesn’t quite work like that. The Surface Pen works through a combination of technology in the pen and a layer of corresponding technology on the screen. It’s not just touch-screen technology, you can’t take the Surface Pen and use it on another brand of screen, it will only work on Surface products. While that affords the technology a great deal of power it can also trip up software that isn’t able to interpret the technology properly. In many cases the pen works just like a mouse replacement, but in others it can cause weird or no behaviour at all.
When PreSonus first released their new touch-enabled version 3 of Studio One the reaction to the Surface Pen when running on the Surface Pro 3 was to get quickly confused and then lock up. In Cakewalk Sonar, again touch-enabled, there were areas in the software that completely refused to acknowledge the presence of a pen on the screen. Both of those DAWs have far better support for it now. Ableton Live appeared to work with both touch and the pen without any trouble except that when grabbing a fader or knob control the value would leap between the maximum and minimum making it impossible to set it accurately. Adding support for “AbsoluteMouseMode” in a preferences file cured that particular oddity.
Where it’s been most unflinchingly successful is within Steinberg’s Cubase and Avid’s Pro Tools neither of which has expressed any interest in touch or pen interaction – but it simply works anyway. From entering and editing notes to drawing in long wiggly lines of modulation and automation the pen becomes a very expressive tool.
However, for the full immersion that the pen can offer, this tends to mean eschewing the keyboard. When you are leaned in, as I mentioned earlier, having to then pull back to use a keyboard shortcut can be rather jarring and interrupting to your workflow. There’s a certain amount you can do with the on-screen virtual keyboard but it can completely cover what it is you’re trying to edit, so it’s not ideal. This highlights what I see as being the current flaw in the Surface Pen workflow – the lack of a relevant, customisable toolbar.
When editing notes or an arrangement with the pen the ability to do simple tasks such as copy and paste become cumbersome. You can evoke a right-click with the squeeze of a button and then select these task from the list, or you can glide through the menu system but neither of these options are as elegant as a simple Ctrl-C and Ctrl-V. You can quickly extend that to other actions – opening the editor, or the mixer, duplicating, setting loop points there’s a whole raft of commands that are hidden away behind menus or keyboard shortcuts that are annoying to reach with just the pen for input. Adding a simple macro toolbar with user definable keyboard shortcuts would greatly enhance the pen’s workflow. It’s possible to do this with third party applications but it really needs support at the OS level.
This is something Dell have considered with their Canvas touch-screen and digital pen system. They have incorporated floating “palettes” that are little toolbars to access useful keyboard shortcuts. Some DAWs, such as Bitwig Studio and PreSonus Studio One, have fingerable toolbars that can perform a similar function – but something more global would be helpful.
With the release of the Surface Pro (2017) Microsoft have introduced an improved Surface Pen with 4 times the resolution of the previous version. Although more relevant to the artist who draws, it’s interesting to see pen support improving in many DAWs. It’s usefulness is becoming more apparent and if you consider the Dell Canvas and the iPad Pro Pencil, along with the development of the Surface into the larger Surface Studio and laptop form factors, it’s also becoming more widespread.
At the time of writing only one DAW manufacturer has stepped up to push the digital pen into more than just emulating mouse tasks. Bitwig Studio has some special MPE (Multidimensional Polyphony Expression) functionality that allows you to map the pen pressure to parameters on MPE compatible virtual instruments. More on that in another article, but hopefully more creative uses will emerge as this gains popularity.
The digital pen offers many creative opportunities. It unhinges you from the mouse/keyboard paradigm and pushes you into a more natural and fluid way of working. It lacks support in some software and there’s some work to be done on optimising the workflow by combining it with a toolbar, but it offers a different and creative approach to musical computer interaction.
Here’s a video of me reviewing the Microsoft Surface Book for music production which has a lot of pen use and examples in it. There’s plenty more on the YouTube channel: