Talking Sheet Music - app to describe sheet music to blind musicians - with speech recognition

[video description] 1st few seconds - text reads Talking Music, Video 2, 16th November 2017. All music and description is dynamically generated from speech input - using a single midi file (track 1 right hand, track 2 left hand). After the text, an android app shows 4 buttons labelled 'pedal 1', 'pedal 1 long', 'pedal 1 double' and 'recognise speech'. When Google speech recognition is being used, an image of a microphone is displayed along with the words 'Speak now'. The final few seconds shows a list of other projects on [end video description]

If the music can talk to me - then I should be able to talk to the music! Yes - it makes an impressive video - but I believe it has genuine practical use too - such as helping blind musicians interact with sheet music.

I was born quite early so my eyes hadn't really formed properly. This has meant I need music to be enlarged in order to see it easily. To solve this problem - I wrote some software to enlarge music / lyrics and scroll with a pedal.

As a visually impaired musician with a Computing Science degree and lots of experience writing MIDI software - it rather caught my interest to read that the RNIB was working on ways to automate describing sheet music.

My Initial Thoughts

  • Interactive - both the music and description should be adaptable to the user's requirements as they work through the music. This could include selecting a phrase (or repeating it), changing the speed, adding a metronome, arpeggiating chords and separating the rhythm, the pitch separating eg eg top note to learn the tune.
  • Hands free - it can be detrimental for a musician to move their hands off an instrument to start a phrase again. Hands free use can be achieved with pedals (short / long / double presses etc) and speech recognition - with a limited set of commands but being aware of the context.
  • Describe and play simultaneously - either so the user can play along - or as a way of helping the user understand a phrase.
  • Respond to user input - either to give feedback on what the user is playing right or wrong. Or to prompt the user what is coming next whilst they play.
  • Be selective - give the user only the information they want - this will aid navigation, comprehension and musical learning.
  • Patterns - the user may wish to know eg - 'this bar is the same as bar X but with a B in the left hand' or 'the rhythm is the repeated' or 'the same chord shape but start on a D'.

Talking Music Android App

The above video is an early version - but it demonstrates some of my initial ideas. The app was supplied a standard midi file (track 1 = right hand, track 2 = left hand) - the metronome, speed, apreggiation and description is dynamically generated based on spoken input.

If you would like to try this app then you can download the Alpha Version (very unfinished) from Google Play. Please send me your feedback - perhaps on the general concept as opposed to the current state! This prototype was developed for Android devices due to personal ease of development and reuse of code. If the idea is successful then it could be developed for other platforms to eg iPhone / Windows / Mac.

Currently speech recognition may well be quite inaccurate - especially for pronunciations other than my own - but a pdf of all the commands it understands (dated 14th January 2018) can be found here.

I welcome comments and feedback on my suggestions for Talking Music, the current app and how the principles behind it could be applied to solve other problems.