Fill in the blanks

Given an ABC score and a non-exhaustive set of fingerings, we should be able to determine if the fully annotated score can be inferred, based on the fingerings provided and some set of formalized editorial conventions.

The only conventions that seem apparent to me at present are the "next note, next finger" (NNNF) convention and the "only one way" (OOW) convention. There may also be a simile convention that repeats fingerings for repeated sections of (similarly beamed?) notes in cases when NNNF does not strictly apply.

NNNF seems to have a straightforward linear-time algorithm (for melodies at least), where we move left-to-right through a list of notes n_1, n_2, n_3, .... The first notes will infer their fingerings from the first downstream annotation encountered. Subsequent notes will infer fingerings from themselves (trivially) and from the most recent prior annotation encountered. We keep track of n_i.finger (the controlling annotation) and n_i.offset (the signed difference--as a count of lines and spaces--between n_i and the note with the controlling annotation.) And we simply assert that 1 ≤ n_i.finger + n_i.offset ≤ 5. There may be some subtlety when accidentals are involved.

Dealing with polyphony is another kettle of fish. OOW seems to be in play for chords in The Little Pianist. There may also be standard fingerings (SF) that are assumed for chords.

Chunking into phrases seems very material here. We should look at Radicioni's work in this area. I know he had some things to say about phrases, and this is its own research topic, I believe.

If we can implement something reasonable in this area, we should also be able to auto-fill fingerings in MDC4 (MDC5, anyone?). We could pad with xes in the input field and (optionally) render the complete annotations on the score. Or maybe we need separate input fields for each note or a pop-up for each note as it is clicked for more surgical entry in this mode of input. And do we want to revisit the utility of voice fingering (as opposed to the left-to-right, low-to-high approach currently employed in MDC4) in this new context?

Another application would be to identify "unnecessary" annotations in a score and to learn (infer rules for) when such unnecessary annotations are appropriate (i.e., appreciated by performers). We could try to come up with some sort of model of good editorial practice.

2015-10-28 status

Done

Administrivia

• Accepted invitation to talk to the Pardo Lab at NU about piano fingering on December 8.

Data Collection

• Sent reminder email to the 203-strong Midwest piano faculty list.
• Distributed initial survey recruitment email to newly assembled list of 168 music departments across the U.S. and Canada.
• Distributed initial survey recruitment email to 51 additional piano faculty, mostly in Canada.
• Distributed initial survey recruitment email to list of 280 piano teachers, assembled from music teacher association sites in Milwaukee, Madison, Kansas City, and Minnesota.
• Received 62 responses so far (with 53 completing both parts).
• Exchanged several emails with people who had problems with the survey tool or questions about the study.
• Outlined simple algorithm for classifying scores as fully (unambiguously) specified, identifying unnecessary annotations, and generating fully annotated scores from fully specified scores. I currently only contemplate the "next note next finger" convention, but even if this is as sophisticated as it gets, this should be a nice little sub-project that could yield a quick workshop paper.

BowTIE Revival

• Explored Yeoman/Angular/Ionic approach to PhoneGap development, completing this tutorial.
• Dr. Forbes and his advisee Alex Pieczynski think Alex should be able to get CS 398 credit for BowTIE work next semester. Sounds good to me.

Doing

1. Defining SQLite schemata for (anonymized) survey data.
2. Writing script to load said schemata.
3. Adding file save and print features to MDC4.
4. Adding support for non-exhaustive input in MDC4.
5. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
6. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
7. Standardizing file format for fingering.

Struggling

• Response rate stinks (but with a big enough denominator, all things are possible). Currently, its upper bound is 62/868 = 7.14%.
• Do I need to worry about selection bias? (Are we only going to get outlier subjects with unusually diverse notions of fingering?)

BowTIE intrinsic evaluation

Here are a few thoughts on how we might intrinsically evaluate BowTIE's effects, which I pitched to Ashkan Rezaei, a fellow music informatics enthusiast in the CS Department at UIC.

Dear Ashkan:
I don't know if this will interest you, but I just restarted a project ​involving an long-idle open-source project of mine called BowTIE. 

The basic idea behind this tool is that a trumpeter's "mental practice" time away from the horn can be enhanced with audio feedback. The system was originally designed with the idea of enhancing transposing skills, but it may have broader application to practice in general. 

The basic plan is to port this old project to mobile touch-screen platforms (probably for tablet form factors) and then do controlled experiments with a few trumpeters to see if we can measure any benefit. 

Measuring the benefit is where I think you might come in. In my mind, evaluation must be done by listening to subjects' actually playing the pieces being studied. To make this as precise as possible, translating their monophonic audio performance to MIDI would be very useful. I could use your help in constructing (or selecting) an appropriate system for doing this and in defining a specific evaluation methodology. This might be a useful baby step toward tangling with the much thornier polyphonic transcription problem, which I think is a significant part of what you want to investigate. 

Here are a few theses that seem to declare the monophonic transcription problem solved:

• http://www.utdallas.edu/~hynek/citing_papers/Vass_thesis.pdf
• http://www.cs.tut.fi/sgn/arg/music/klapuri2.pdf

Please let me know if you want to pursue this idea. More hands make light work. 

Thanks,
Dave

Vision pops back into view

Okay, we are now turning back to the ADC rigamarole. Here is what I just sent to my new mentee Pablo on the subject:

Dear Pablo: 

Here are some links:

• Open Frameworks

• Processing

• OpenCV

• C-MIDI

• GitHub

• Mendeley

​And I attach a few other relevant papers I have collected on your new favorite topic. I recommend that you read these papers and try to do some sort of "Hello CV" program in Processing, just to get your feet wet. 

The last link is to Mendeley, which I use to manage my references and make bibliographies easy. We can also set up a common folder to share references for this project. I think it is handy.

Also, if you haven't already done this, you should sign up for a GitHub account. We will use it as a central repository for our code, and rumor has it that many developers in Silicon Valley don't have resumes. They just have GitHub profiles. Anyway, send me your user ID when you have one. I am dvdrndlph. (This will probably live as a directory under https://github.com/dvdrndlph/didactyl.) 

I think that is enough mentoring for now. Have fun.

Cheers,
Dave

2015-10-21 staus

Done

Administrivia

• Hosted (with a lot of help from colleagues at UIC and NU) Chris Raphael's distinguished lecture, "The Informatics Philharmonic." It was a real career highlight for me.
• Interviewed two undergraduate mentee (yes, that is a word) candidates for WiCS DREAM program. They were both interested in the BowTIE project. One (trumpeter Jackson Price) will start this semester. One (guitarist Alex Pieczynski) will join next semester.
• Reached out to CR for help identifying a more research oriented to piano pedagogue to collaborate with.
• Sent a status update to Dr. Forbes per his request at the Raphael dinner. It would be great to engage him more.
• Received a nice email from Eugene Graub, a piano professor at Grinnell, asking to be kept apprised of the Didactyl project.

Data Collection

• Distributed initial survey recruitment email to list of 203 piano faculty in Midwest universities. Received 20 (with 15 completing both parts) responses so far.
• Sent reminder email to CAMTA mailing list.
• Started compiling music department contact list, as third recruitment wave will apparently be needed to get to 50 respondents, which is still my minimal goal.
• Met with SJ to demo MDC4 and discuss my planned data collection approach.
• Exchanged email with Dr. Cruz, whose brother is a classical pianist. As tradition mandates, I tried to rope him in as a subject or collaborator. It was nice of her to think of my work.

BowTIE Revival

• Researched cross-platform mobile development.
• Created new GitHub project for this development work.
• Created "Hello World" PhoneGap application. PhoneGap seems a promising way to get a research system up fast.
• Jackson has trumpet and horn performance undergraduate friends (at Northern Illinois and UIUC, as I recall) who (along with me and him) might form be a subject pool to evaluate this system. I am very excited to have Jackson on board.

Doing

1. Defining SQLite schemata for (anonymized) survey data.
2. Writing script to load said schemata.
3. Adding file save and print features to MDC4.
4. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
5. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
6. Standardizing file format for fingering.

Struggling

• Response rate still stinks.
• Do I need to worry about selection bias? (Are we only going to get outlier subjects with unusually diverse notions of fingering?)

Annotation conventions and ambiguity

I received a nice email from Eugene Gaub, a music professor at Grinnell. In my response to him, I outlined my current thinking on the minimal annotation in edited scores. Here it is so I don't lose track of it. . . .

Actually, we just started seriously looking at how completely "determined" or "specified" a given passage is, given the apparently sparse annotations included in the score. I wonder if there are standard conventions for how to decide when to include an annotation and how much variability there is among editors and composers. Would you be able to point me to any good pedagogical sources that might talk about this? 

I am actually only a beginning pianist (and to say even this is an insult to beginning pianists everywhere), and I have been struck by how few numbers appear on an exercise, yet the fingerings are in fact unambiguously specified--given an implied "next-note-next-finger" convention. But as you describe, I have often experienced the desire to have more numbers on the page, to be sure.

We plan to explore the notion of optimal verbosity of annotations, perhaps for varying skill levels or familiarity with a given piece. Given a "fully determined" set of fingerings, I envision a sort of verbosity knob where we display just the right number of numbers. Measuring intrinsic "optimality" in this context might pose a special challenge. But we could do some usability testing with actual pianists if push came to shove. The more I think about this, the more I like it as a related sub-project. 

Status and near-term plans

Here is some information I sent to AF to summarize where I am at and where I think I am going. . . .

The happy news is that we do have survey data starting to trickle in now. If you want to see the survey, it lives here.

This survey includes a little JavaScript program to facilitate fingering data entry. This code has been largely rewritten recently to support the more complex music (from Book 1 of Bach's Well Tempered Clavier) planned for a second phase of "manual" data collection. So the dominoes are mostly in place now for data collection through the end of the year (during which we will happily pay $700 of my $2000 Provost's Award to pianists).

However, by November 15, I should have sufficient data to say something new. (Indeed, at Dr. Jordan's suggestion, I am starting to analyze data now. I have 19 responses so far.) So I plan to be shopping my first conference paper around by the first of the year. I really want to get into ISMIR 2016, as it is in New York this year in August. But something smaller and sooner will also be targeted.

In parallel, work will resume in earnest soon on our computer vision system to collect fingering data from actual performances. (There was a lot of rumination on this topic in the early going.) This part should be a lot of fun, but it is unexplored terrain for me. I am therefore hoping to rope in an undergraduate student (under the auspices of the WiCS DREAM mentoring program) to help. We have $500 budgeted to spend on equipment for this and then our last $800 to compensate pianists to evaluate and leverage it.

Conferences

The ISMIR conference will be in New York in 8-12 August 2016! So I am setting my sights squarely on that. I will probably go even if I can't get a paper accepted there. The conference was in October last year. Not sure why they moved it up (and I wish they hadn't).

MCM 2016 does not appear to be scheduled yet. While it was in London in June this year, it has been in the US, Canada, and Mexico in the recent past.

Sound and Music Computing also does not seem to have their 2016 act together yet. It was in Ireland in 2015 at the end of July, but it seems always to be in Europe.

The International Computer Music Conference was in Texas this year at the end of September. It will be in the Netherlands 2016 on September 12-16.

Another, lower key, event to keep in mind is the one that Bryan Pardo is trying to get established: The Midwest Music Information Retrieval Gathering. He and CR were actually talking about revitalizing this event at CR's distinguished lecture at UIC. It would be good if UIC had a presence there.

It sucks that all these things are right on top of each other on the calendar.

2015-10-14 status

Done

Administrivia

• Obtained $200 to pay lottery winners for first two surveys.

Data Collection

• Invited several more acquaintances to participate in survey and did some ineffectual nagging.
• Distributed initial survey recruitment email to list of 165 Chicago area piano teachers. Received 6 responses so far.
• Added character buffer to make MDC4 fingering entry more fluid. Performance seems good now.
• Added ability to specify hand in use for fingerings to allow either hand to be used on either staff.
• Implemented output and validation functions in MDC4 to enable interoperation with Qaultrics.
• Fixed problem with whitespace entry in MDC4.
• Changed MDC4 rendering of left and right hand fingerings to be in different fonts, so they can be easily differentiated.
• Implemented method in MDC4 for annotating synchronous pitches from multiple voices.
• Supported "redundant" identical synchronous pitches from multiple voices in MDC4.
• Took a stab at defining BNF grammar for fingering entry strings, using ANTLR. Regular expressions currently in use are buggy and restrict expressiveness. But they are probably sufficient for the current task.

Doing

1. Auditing Music 170 (Keyboard Skills I), TuTh 3:30-4:45.
2. Adding file save and print features to MDC4.
3. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
4. Standardizing file format for fingering.

Struggling

• Response rate stinks.

2015-10-02 status

Done

Administrivia

• Resubmitted program advance for initial surveys per Sherice.
• Obtained IRB approval for data collection activities.
• Worked out some scheduling details for CR's Distinguished Lecture at UIC.
• Reached out to the local academic friends of the project about the lecture.
• Reached out to Peter Snyder about grad student panel to talk to CR when he is here. 
• Made pitch for lecture and panel at Friendly Friday.
• Posted two flyers for the lecture in the Music Department (with their office's approval).
• Got positive response from Bryan Pardo at NU. It seems likely there will be a contingent from his lab.

Data Collection

• Fixed two typos in the survey, which I caught during another "final" review.
• Sent out initial survey to my short list.
• Improved latency between fingering entry and rendered annotation by more than 50%. But it still feels sluggish.
• Pinged Jef Moine (abc2svg developer and general mensch) and learned how to make the fingering symbols bigger.
• Tested MDC4 with 5-voice piece (P&F 1 from WTC). Fixed several issues.

Doing

1. Auditing Music 170 (Keyboard Skills I), TuTh 3:30-4:45.
2. Adding character buffering to MDC4 to make input smoother.
3. Implementing output hooks in MDC4 to play nice with Qualtrics.
4. Standardizing file format for fingering.

Struggling

• Pinged M. Moine to complain about bad associations of fingerings to notes that start at the same time in different voices. This really seems like a bug. Waiting for response. If no fix is forthcoming, I will need to code around this.