Talking Sheet Music - app to describe sheet music to blind musicians - with speech recognition



If the music can talk to me - then I should be able to talk to the music! Yes - it makes an impressive video - but I believe it has genuine practical use too - such as helping blind musicians interact with sheet music.

I was born quite early so my eyes hadn't really formed properly. This has meant I need music to be enlarged in order to see it easily. To solve this problem - I wrote some software to enlarge music / lyrics and scroll with a pedal.

As a visually impaired musician with a Computing Science degree and lots of experience writing MIDI software - it rather caught my interest to read read that the RNIB was working on ways to automate describing sheet music.

My Initial Thoughts

  • Interactive - both the music and description should be adaptable to the user's requirements as they work through the music. This could include selecting a phrase (or repeating it), changing the speed, adding a metronome, arpeggiating chords and separating the rhythm, the pitch separating eg eg top note to learn the tune.
  • Hands free - it can be detrimental for a musician to move their hands off an instrument to start a phrase again. Hands free use can be achieved with pedals (short / long / double presses etc) and speech recognition - with a limited set of commands but being aware of the context.
  • Describe and play simultaneously - either so the user can play along - or as a way of helping the user understand a phrase.
  • Respond to user input - either to give feedback on what the user is playing right or wrong. Or to prompt the user what is coming next whilst they play.
  • Be selective - give the user only the information they want - this will aid navigation, comprehension and musical learning.
  • Patterns - the user may wish to know eg - 'this bar is the same as bar X but with a B in the left hand' or 'the rhythm is the repeated' or 'the same chord shape but start on a D'.

Talking Music Android App

The above video is an early version - but it demonstrates some of my initial ideas. The app was supplied a standard midi file (track 1 = right hand, track 2 = left hand) - the metronome, speed, apreggiation and description is dynamically generated based on spoken input.

I welcome comments and feedback on my suggestions for Talking Music, the current app and how the principles behind it could be applied to solve other problems.