2015-12-17 status

Done

Data Collection

• Closed out initial program advance in TEM.
• Created a number of database views to make survey data easier to process.
• Created 50 bar charts using R and LaTeX to describe distribution of answers to 50 survey questions.
• Calculated "course-grained" mean agreement of Parncutt sequences in both Parncutt et al. and in our survey. (Think Fleiss's Kappa with each fingering representing a category, 191 annotators, one example to classify, and no need to worry about chance agreement. Anyway, it made sense to me while I was doing it.)
• Split 191 complete responses into two datasets ("exploratory" and "validation") for out-of-sample testing in an effort to mitigate effects of data dredging if/when our quest for correlation runs amok.

 Doing

1. Arranging with BDE for Alex to receive CS 398 credit next term for BowTIE work.
2. Performing Chi Square analysis of exploratory dataset to correlate abbreviated Parncutt fingerings with gender, reach, age, Hanon usage, technical practice, preparation actions, injury, etc.
3. Looking at how well selecting fingering a in Exercise A predicts selecting fingering b in Exercise B. That is, do people have common patterns of fingering preference?
4. Analyzing "consensus" of finger choice that follows abbreviated Parncutt fingering. How arbitrary were these sequences? Can we identify more suitable sequences in our data?
5. Doing more basic descriptive statistical analysis of survey data.
6. Evaluating Tableau for easier (and richer) data visualization. (SQLite support missing for OS X. 

Struggling

• Can we legitimately treat fingerings as categories? 
• How can we conflate the unpopular fingerings meaningfully? 
• Not sure how out-of-sample testing will complicate contemplated ad hoc category definition.

Data dredge

As I am about to embark on my quest to find correlation in the data, I am chastened by fears of data dredging. So I propose we do randomized out-of-sample testing. Toward this end, we will split the data into two subsets of approximately equal size.

The following query gives us the identifiers for subjects who completed both parts of the survey and for whom at least some fingering data were recorded:

select response_id
from well_known_subject s
inner join
(select distinct subject
from finger where fingers != '') f
on s.response_id = f.subject

We save this query as the "complete_response_id" view. There are 191 such response_ids.
So we load the "exploratory_response_id" table like so:

insert into exploratory_response_id
select response_id
from complete_response_id
order by random()
limit 96

The 95 response_ids  not included in this table are stored in the "validation_response_id" view:

select c.response_id
from complete_response_id c
where not exists (select response_
from exploratory_response_id e
where e.response_id = c.response_id)

The actual (scrubbed) profile data will remain in the "subject" table. We will create views to provide access to the appropriate data ("exploratory_subject" and "validation_subject"), which will leverage the subject_latexable view of the subject data to use camel-case column names. This makes it unnecessary to remap the column names in R.

2015-12-02 status

Done

Data Collection

• Paid lottery winner for Survey II and notified UPAY1099 through PEAR of two payments to winners. Received confirmation of receipt from Amazon. 
• Loaded data from Survey II to SQLite. 
• Developed workflow to create charts in R and render them in LaTeX. 
• Did some manual cleanup for "well-known subjects" (people who completed both parts of the survey), on whom we will focus our analysis. (In most conservative interpretation of the protocol, subjects who walk away at any point are discarded, except for two subjects from whom I obtained permission to retain their nearly complete submissions. I manually set the "finished_2" field in the database, so they would be included.) There are 199 such well-known subjects, of whom we have actual fingering data for 191. 

BowTIE

• Met with Jackson and helped get his Android "Hello World" working. 
• Arranged for Alex to come to my meeting with BDE this week to discuss CS 398 credit next term. 

Doing

• Relearning R and LaTeX. 
• Doing basic descriptive statistical analysis of survey data.

Struggling

• Upgrade to El Capitan at your peril. Root is no longer root in Mac Land. Broke Perl and LaTeX environment. 
• Working with Sherice to close out $200 cash advance.

2015-11-19 status

Done

Data Collection

• Paid lottery winner for Survey I.
• Invited two people who asked for a second chance at Survey I (fingering exercises).
• Followed up again on Survey II, providing a 21 November deadline. Received 14 responses. So we have 198 full responses to date.
• Found apparent problem with fingering collection script, so we are missing fingering data from 10 subjects. I suspect a browser incompatibility with the JavaScript code or something wonky in Qualtrics. 
• Wrote and tested script to load data from Survey I to SQLite.

BowTIE

• Met with Jackson. He is going to work on "native" Java code for the performance component, as his programming experience is limited to Java.
• Played around with more tutorials.

Doing

1. Analyzing survey data.
2. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
3. Adding file save and print features to MDC4.
4. Adding support for non-exhaustive input in MDC4.
5. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
6. Standardizing file format for fingering.

Struggling

• What to do with partial results from people who completed Part I, said they had time for Part II, but never finished? Protocol says we delete data from people who walk away.
• Should we contact these people to try to rectify missing fingering data in Survey I? Ivana is one of them.
• Should we offer small direct payment for Survey II in addition to lottery?
• Should we offer larger payment to fewer people?
• Should we give subjects in Phase II the option of photocopying their annotated scores and mailing them to us?

2015-11-11 status

Done

Data Collection

• Closed out Survey I (fingering exercises) after collecting 203 responses. Response per email rate is 3.8%. Completion rate for all people who started the survey is 203/352 = 57.7%.
• Followed up with 20 subjects who did not complete profile part of Survey I, asking them to complete Survey II. Received 9 responses. So we have 193 full responses to date.
• Exchanged more emails with people who had problems with the survey tool or questions about the study.
• C-MIDI source code provided is incomplete and cannot be built. We will inspect the code and port any bits that look promising.
• Defined SQLite schema for (anonymized) survey data.

Doing

1. Writing script to load SQLite.
2. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
3. Rethinking incentive scheme going forward.
4. Adding file save and print features to MDC4.
5. Adding support for non-exhaustive input in MDC4.
6. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
7. Standardizing file format for fingering.

Struggling

• What to do with partial results from people who completed Part I, said they had time for Part II, but never finished?
• Should we give subjects in Phase II the option of photocopying their annotated scores and mailing them to us?

2015-11-04 status

Done

Data Collection

• Distributed initial survey recruitment to new email lists of 4,424 piano teachers, assembled from music teacher association sites in Ohio, Pennsylvania, California, Massachusetts, Florida, New Jersey, and Georgia.
• Received 154 responses so far (with 139 completing both parts).
• Exchanged more emails with people who had problems with the survey tool or questions about the study.
• Obtained C-MIDI source code from Dr. Dimitry Gorodnichy of the University of Ottawa. This may give us a head start on our contemplated automatic data collector (ADC).
• Created new repo for ADC at https://github.com/dvdrndlph/dactylize. It is supposed to be a pun, as in dactyl (finger) + ize (eyes). But I don't think anyone is going to get it.
• Created virtual Windows host and opened the C-MIDI code in Visual Studio. Not too surprisingly, the projects won't build.
• Gnashed teeth when survey participant pointed out mistake in Exercise 3.

BowTIE

• Reached out to Ashkan Rezaei, a first year CS PhD student at UIC and jazz aficionado about maybe helping us evaluate BowTIE. He has an interest in automatic transcription, which would be very useful for our evaluation. He is receptive to the idea.

Doing

1. Defining SQLite schemata for (anonymized) survey data.
2. Writing script to load said schemata.
3. Adding file save and print features to MDC4.
4. Adding support for non-exhaustive input in MDC4.
5. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
6. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
7. Standardizing file format for fingering.

Struggling

• Response rate stinks. Currently, its upper bound is 154/5292 = 2.9%.
• Do I need to worry about selection bias? (Are we only going to get outlier subjects with unusually diverse notions of fingering?)

Visual Studio, ho!

Here are the steps I took to get Visual Studio working on my Mac (a puny MacBook Air to be specific):

1. Download the ISO image for Visual Studio license through UIC's WebStore.
2. Copy the Visual Studio license in a safe place.
3. Download the ISO for Windows Server 2012 R2.
4. Save its keys to a safe place.
5. Download and install VirtualBox for OS X hosts.
6. Launch Virtual Box.
7. Click the New button to create a new virtual host.
8. Enter a name and select "Windows 2012" as the version for the host. Configure at least 2GB of RAM.
9. Click Create and select your Windows Server ISO file as the File location.
10. Click Create.
11. Run the new virtual host. You will be asked to install Windows.
12. Click Devices --> Insert Guest Additions CD image.
13. Run VBoxWindowsAdditions.exe from the CD image.
14. Click Devices --> Shared Folders --> Shared Folder Settings and share the folders you stored the Visual Studio ISO and our Git repo.
15. Reboot your virtual host.
16. On the virtual host, install MagicISO.
17. Use MagicISO to open the VisualStudio ISO and install VisualStudio from it.
18. Open the .dsp files from our repo in VisualStudio.
19. Figure out why none of them will build.

C-MIDI, ho!

In a rather exciting development, I just received the source code from Dimitry Gorodnichy for the C-MIDI project. This might give us a head start on our automatic data collection system.

Here is the conference paper that describes the marker-less C-MIDI system:

D. Gorodnichy and A. Yogeswaran, “Detection and tracking of pianist hands and fingers,” in Proceedings of the Third Canadian Conference on Computer and Robot Vision, 2006.

The paper is long on technical details, but short on performance data. This is all they have to say on the subject of system performance:

The professional pianists, as they play and see at the same time on a computer screen the visual annotation of their playing, are content acknowledging the correct finger annotation in about a half of cases. Of the other half, the fingers are either left unmarked or can provide a set of possibilities to choose from.​

​Still, it is worth a long hard look.

I have checked the C-MIDI source code into our new GitHub repo. (I think it is good to keep this project separate from the other Didactyl code in case this one gets a little bloated, which seems likely.)

Unfortunately, the code provided is a Microsoft Visual Studio (Windows) project, which I have little experience with. But we do have a free Visual Studio license through UIC, so we should be able to compile the code easily on a Windows box.

Pablo and I are looking into running Windows on our Macs using VirtualBox. I have had luck doing this in the past, and we have free Windows Server 2012 R2 licenses through UIC, so this can be done at no cost.

Another option, and a better one if we actually want to test this system, is to run Windows under Boot Camp.

Fill in the blanks

Given an ABC score and a non-exhaustive set of fingerings, we should be able to determine if the fully annotated score can be inferred, based on the fingerings provided and some set of formalized editorial conventions.

The only conventions that seem apparent to me at present are the "next note, next finger" (NNNF) convention and the "only one way" (OOW) convention. There may also be a simile convention that repeats fingerings for repeated sections of (similarly beamed?) notes in cases when NNNF does not strictly apply.

NNNF seems to have a straightforward linear-time algorithm (for melodies at least), where we move left-to-right through a list of notes n_1, n_2, n_3, .... The first notes will infer their fingerings from the first downstream annotation encountered. Subsequent notes will infer fingerings from themselves (trivially) and from the most recent prior annotation encountered. We keep track of n_i.finger (the controlling annotation) and n_i.offset (the signed difference--as a count of lines and spaces--between n_i and the note with the controlling annotation.) And we simply assert that 1 ≤ n_i.finger + n_i.offset ≤ 5. There may be some subtlety when accidentals are involved.

Dealing with polyphony is another kettle of fish. OOW seems to be in play for chords in The Little Pianist. There may also be standard fingerings (SF) that are assumed for chords.

Chunking into phrases seems very material here. We should look at Radicioni's work in this area. I know he had some things to say about phrases, and this is its own research topic, I believe.

If we can implement something reasonable in this area, we should also be able to auto-fill fingerings in MDC4 (MDC5, anyone?). We could pad with xes in the input field and (optionally) render the complete annotations on the score. Or maybe we need separate input fields for each note or a pop-up for each note as it is clicked for more surgical entry in this mode of input. And do we want to revisit the utility of voice fingering (as opposed to the left-to-right, low-to-high approach currently employed in MDC4) in this new context?

Another application would be to identify "unnecessary" annotations in a score and to learn (infer rules for) when such unnecessary annotations are appropriate (i.e., appreciated by performers). We could try to come up with some sort of model of good editorial practice.

2015-10-28 status

Done

Administrivia

• Accepted invitation to talk to the Pardo Lab at NU about piano fingering on December 8.

Data Collection

• Sent reminder email to the 203-strong Midwest piano faculty list.
• Distributed initial survey recruitment email to newly assembled list of 168 music departments across the U.S. and Canada.
• Distributed initial survey recruitment email to 51 additional piano faculty, mostly in Canada.
• Distributed initial survey recruitment email to list of 280 piano teachers, assembled from music teacher association sites in Milwaukee, Madison, Kansas City, and Minnesota.
• Received 62 responses so far (with 53 completing both parts).
• Exchanged several emails with people who had problems with the survey tool or questions about the study.
• Outlined simple algorithm for classifying scores as fully (unambiguously) specified, identifying unnecessary annotations, and generating fully annotated scores from fully specified scores. I currently only contemplate the "next note next finger" convention, but even if this is as sophisticated as it gets, this should be a nice little sub-project that could yield a quick workshop paper.

BowTIE Revival

• Explored Yeoman/Angular/Ionic approach to PhoneGap development, completing this tutorial.
• Dr. Forbes and his advisee Alex Pieczynski think Alex should be able to get CS 398 credit for BowTIE work next semester. Sounds good to me.

Doing

1. Defining SQLite schemata for (anonymized) survey data.
2. Writing script to load said schemata.
3. Adding file save and print features to MDC4.
4. Adding support for non-exhaustive input in MDC4.
5. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
6. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
7. Standardizing file format for fingering.

Struggling

• Response rate stinks (but with a big enough denominator, all things are possible). Currently, its upper bound is 62/868 = 7.14%.
• Do I need to worry about selection bias? (Are we only going to get outlier subjects with unusually diverse notions of fingering?)

BowTIE intrinsic evaluation

Here are a few thoughts on how we might intrinsically evaluate BowTIE's effects, which I pitched to Ashkan Rezaei, a fellow music informatics enthusiast in the CS Department at UIC.

Dear Ashkan:
I don't know if this will interest you, but I just restarted a project ​involving an long-idle open-source project of mine called BowTIE. 

The basic idea behind this tool is that a trumpeter's "mental practice" time away from the horn can be enhanced with audio feedback. The system was originally designed with the idea of enhancing transposing skills, but it may have broader application to practice in general. 

The basic plan is to port this old project to mobile touch-screen platforms (probably for tablet form factors) and then do controlled experiments with a few trumpeters to see if we can measure any benefit. 

Measuring the benefit is where I think you might come in. In my mind, evaluation must be done by listening to subjects' actually playing the pieces being studied. To make this as precise as possible, translating their monophonic audio performance to MIDI would be very useful. I could use your help in constructing (or selecting) an appropriate system for doing this and in defining a specific evaluation methodology. This might be a useful baby step toward tangling with the much thornier polyphonic transcription problem, which I think is a significant part of what you want to investigate. 

Here are a few theses that seem to declare the monophonic transcription problem solved:

• http://www.utdallas.edu/~hynek/citing_papers/Vass_thesis.pdf
• http://www.cs.tut.fi/sgn/arg/music/klapuri2.pdf

Please let me know if you want to pursue this idea. More hands make light work. 

Thanks,
Dave

Vision pops back into view

Okay, we are now turning back to the ADC rigamarole. Here is what I just sent to my new mentee Pablo on the subject:

Dear Pablo: 

Here are some links:

• Open Frameworks

• Processing

• OpenCV

• C-MIDI

• GitHub

• Mendeley

​And I attach a few other relevant papers I have collected on your new favorite topic. I recommend that you read these papers and try to do some sort of "Hello CV" program in Processing, just to get your feet wet. 

The last link is to Mendeley, which I use to manage my references and make bibliographies easy. We can also set up a common folder to share references for this project. I think it is handy.

Also, if you haven't already done this, you should sign up for a GitHub account. We will use it as a central repository for our code, and rumor has it that many developers in Silicon Valley don't have resumes. They just have GitHub profiles. Anyway, send me your user ID when you have one. I am dvdrndlph. (This will probably live as a directory under https://github.com/dvdrndlph/didactyl.) 

I think that is enough mentoring for now. Have fun.

Cheers,
Dave

2015-10-21 staus

Done

Administrivia

• Hosted (with a lot of help from colleagues at UIC and NU) Chris Raphael's distinguished lecture, "The Informatics Philharmonic." It was a real career highlight for me.
• Interviewed two undergraduate mentee (yes, that is a word) candidates for WiCS DREAM program. They were both interested in the BowTIE project. One (trumpeter Jackson Price) will start this semester. One (guitarist Alex Pieczynski) will join next semester.
• Reached out to CR for help identifying a more research oriented to piano pedagogue to collaborate with.
• Sent a status update to Dr. Forbes per his request at the Raphael dinner. It would be great to engage him more.
• Received a nice email from Eugene Graub, a piano professor at Grinnell, asking to be kept apprised of the Didactyl project.

Data Collection

• Distributed initial survey recruitment email to list of 203 piano faculty in Midwest universities. Received 20 (with 15 completing both parts) responses so far.
• Sent reminder email to CAMTA mailing list.
• Started compiling music department contact list, as third recruitment wave will apparently be needed to get to 50 respondents, which is still my minimal goal.
• Met with SJ to demo MDC4 and discuss my planned data collection approach.
• Exchanged email with Dr. Cruz, whose brother is a classical pianist. As tradition mandates, I tried to rope him in as a subject or collaborator. It was nice of her to think of my work.

BowTIE Revival

• Researched cross-platform mobile development.
• Created new GitHub project for this development work.
• Created "Hello World" PhoneGap application. PhoneGap seems a promising way to get a research system up fast.
• Jackson has trumpet and horn performance undergraduate friends (at Northern Illinois and UIUC, as I recall) who (along with me and him) might form be a subject pool to evaluate this system. I am very excited to have Jackson on board.

Doing

1. Defining SQLite schemata for (anonymized) survey data.
2. Writing script to load said schemata.
3. Adding file save and print features to MDC4.
4. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
5. Assembling prerequisites (JavaScript libraries, data layer) for BowTIE.
6. Standardizing file format for fingering.

Struggling

• Response rate still stinks.
• Do I need to worry about selection bias? (Are we only going to get outlier subjects with unusually diverse notions of fingering?)

Annotation conventions and ambiguity

I received a nice email from Eugene Gaub, a music professor at Grinnell. In my response to him, I outlined my current thinking on the minimal annotation in edited scores. Here it is so I don't lose track of it. . . .

Actually, we just started seriously looking at how completely "determined" or "specified" a given passage is, given the apparently sparse annotations included in the score. I wonder if there are standard conventions for how to decide when to include an annotation and how much variability there is among editors and composers. Would you be able to point me to any good pedagogical sources that might talk about this? 

I am actually only a beginning pianist (and to say even this is an insult to beginning pianists everywhere), and I have been struck by how few numbers appear on an exercise, yet the fingerings are in fact unambiguously specified--given an implied "next-note-next-finger" convention. But as you describe, I have often experienced the desire to have more numbers on the page, to be sure.

We plan to explore the notion of optimal verbosity of annotations, perhaps for varying skill levels or familiarity with a given piece. Given a "fully determined" set of fingerings, I envision a sort of verbosity knob where we display just the right number of numbers. Measuring intrinsic "optimality" in this context might pose a special challenge. But we could do some usability testing with actual pianists if push came to shove. The more I think about this, the more I like it as a related sub-project. 

Status and near-term plans

Here is some information I sent to AF to summarize where I am at and where I think I am going. . . .

The happy news is that we do have survey data starting to trickle in now. If you want to see the survey, it lives here.

This survey includes a little JavaScript program to facilitate fingering data entry. This code has been largely rewritten recently to support the more complex music (from Book 1 of Bach's Well Tempered Clavier) planned for a second phase of "manual" data collection. So the dominoes are mostly in place now for data collection through the end of the year (during which we will happily pay $700 of my $2000 Provost's Award to pianists).

However, by November 15, I should have sufficient data to say something new. (Indeed, at Dr. Jordan's suggestion, I am starting to analyze data now. I have 19 responses so far.) So I plan to be shopping my first conference paper around by the first of the year. I really want to get into ISMIR 2016, as it is in New York this year in August. But something smaller and sooner will also be targeted.

In parallel, work will resume in earnest soon on our computer vision system to collect fingering data from actual performances. (There was a lot of rumination on this topic in the early going.) This part should be a lot of fun, but it is unexplored terrain for me. I am therefore hoping to rope in an undergraduate student (under the auspices of the WiCS DREAM mentoring program) to help. We have $500 budgeted to spend on equipment for this and then our last $800 to compensate pianists to evaluate and leverage it.

Conferences

The ISMIR conference will be in New York in 8-12 August 2016! So I am setting my sights squarely on that. I will probably go even if I can't get a paper accepted there. The conference was in October last year. Not sure why they moved it up (and I wish they hadn't).

MCM 2016 does not appear to be scheduled yet. While it was in London in June this year, it has been in the US, Canada, and Mexico in the recent past.

Sound and Music Computing also does not seem to have their 2016 act together yet. It was in Ireland in 2015 at the end of July, but it seems always to be in Europe.

The International Computer Music Conference was in Texas this year at the end of September. It will be in the Netherlands 2016 on September 12-16.

Another, lower key, event to keep in mind is the one that Bryan Pardo is trying to get established: The Midwest Music Information Retrieval Gathering. He and CR were actually talking about revitalizing this event at CR's distinguished lecture at UIC. It would be good if UIC had a presence there.

It sucks that all these things are right on top of each other on the calendar.

2015-10-14 status

Done

Administrivia

• Obtained $200 to pay lottery winners for first two surveys.

Data Collection

• Invited several more acquaintances to participate in survey and did some ineffectual nagging.
• Distributed initial survey recruitment email to list of 165 Chicago area piano teachers. Received 6 responses so far.
• Added character buffer to make MDC4 fingering entry more fluid. Performance seems good now.
• Added ability to specify hand in use for fingerings to allow either hand to be used on either staff.
• Implemented output and validation functions in MDC4 to enable interoperation with Qaultrics.
• Fixed problem with whitespace entry in MDC4.
• Changed MDC4 rendering of left and right hand fingerings to be in different fonts, so they can be easily differentiated.
• Implemented method in MDC4 for annotating synchronous pitches from multiple voices.
• Supported "redundant" identical synchronous pitches from multiple voices in MDC4.
• Took a stab at defining BNF grammar for fingering entry strings, using ANTLR. Regular expressions currently in use are buggy and restrict expressiveness. But they are probably sufficient for the current task.

Doing

1. Auditing Music 170 (Keyboard Skills I), TuTh 3:30-4:45.
2. Adding file save and print features to MDC4.
3. Integrating MDC4 into draft Qualtrics survey for WTC, Book 1, Prelude 1.
4. Standardizing file format for fingering.

Struggling

• Response rate stinks.

2015-10-02 status

Done

Administrivia

• Resubmitted program advance for initial surveys per Sherice.
• Obtained IRB approval for data collection activities.
• Worked out some scheduling details for CR's Distinguished Lecture at UIC.
• Reached out to the local academic friends of the project about the lecture.
• Reached out to Peter Snyder about grad student panel to talk to CR when he is here. 
• Made pitch for lecture and panel at Friendly Friday.
• Posted two flyers for the lecture in the Music Department (with their office's approval).
• Got positive response from Bryan Pardo at NU. It seems likely there will be a contingent from his lab.

Data Collection

• Fixed two typos in the survey, which I caught during another "final" review.
• Sent out initial survey to my short list.
• Improved latency between fingering entry and rendered annotation by more than 50%. But it still feels sluggish.
• Pinged Jef Moine (abc2svg developer and general mensch) and learned how to make the fingering symbols bigger.
• Tested MDC4 with 5-voice piece (P&F 1 from WTC). Fixed several issues.

Doing

1. Auditing Music 170 (Keyboard Skills I), TuTh 3:30-4:45.
2. Adding character buffering to MDC4 to make input smoother.
3. Implementing output hooks in MDC4 to play nice with Qualtrics.
4. Standardizing file format for fingering.

Struggling

• Pinged M. Moine to complain about bad associations of fingerings to notes that start at the same time in different voices. This really seems like a bug. Waiting for response. If no fix is forthcoming, I will need to code around this.

2015-09-25 status

Done

Administrivia

• Created $200 program advance through Travel and Expense Management (TEM) system for initial surveys.
• Submitted response materials to IRB East Campus office.
• Made Sherice my proxy in the TEM system per her request.
• Responded to request from WiCS to mentor undergraduate students in a research project. I thought one of them might be interested in working on computer vision and automating data collection with me.

Data Collection

• Added note highlighting on annotation entry to MDC4.
• Implemented caret repositioning for note clicks in MDC4.
• Condensed vertical spacing of elements in MDC4.

Doing

1. Auditing Music 170 (Keyboarding I), TuTh 3:30-4:45.
2. Testing support for more than two voices in MDC4.
3. Implementing output hooks in MDC4 to play nice with Qualtrics.
4. Standardizing file format for fingering.

Struggling

• No receipt email for IRB submission. Need to follow up.

2015-09-18 status

Done

Administrivia

• Talked to Ivy and Tricia to get the ball rolling for me to use the TEM system, so I can set up a program advance and make everybody happy. It will take 2-3 business days for me to have access.
• Completed and printed out the materials requested by the IRB. These are ready for BDE's signature and submission.
• Overhauled my resume until it almost looks like a CV.
• Completed and submitted application for Chancellor's Graduate Research Award. All letters of support from faculty (BDE, SJ, IB) are in Santhi's capable hands.
• Applied for a $3000 Qualtrics Behavioral Research Award.
• Submitted letter of interest for a $10000 Grammy Foundation Grant.

Data Collection

• Inspired by the Grammy Foundation's concern for the "health and occupational well being of musicians" and the generally deep pockets for biomedical research, developed questions on posture and pain/discomfort and added them to part 2 of the survey.

Doing

1. Auditing Music 170 (Keyboarding I), TuTh 3:30-4:45.
2. Adding note highlighting on annotation entry to MDC4.
3. Supporting caret repositioning for note clicks in MDC4.
4. Testing support for more than two voices in MDC4.
5. Condensing vertical spacing of elements in MDC4.
6. Standardizing file format for fingering.

Struggling

• Tripping on OBFS hoops, but the path seems clear now.

You can't win if you don't play

I submitted a "letter of interest" to the Grammy Foundation Grant Program (https://www.grammy.org/grammy-foundation/grants), asking for $10K over two years. Their stated focus on the "medical and occupational well-being of music professionals" led me to talk this angle up a bit, like so:

Another potential benefit of an accurate fingering model is being able to emphasize ergonomic principles when appropriate. Allsop and Ackland recently reported that 71.9% of professional pianists have experienced "playing-related muscular disorders." Among all pianists, more than half of such complaints were in areas distal from the upper arm. It therefore seems likely that fingering choices contribute significantly to occupational injuries among pianists.

And this led me to add a few relevant questions about (hand and wrist) postures and injuries to (part 2 of) the survey. Since we are held up by the IRB, I figured I might as well make virtue out of necessity. I hope to run these new questions past IB today (and BDE tomorrow).

And last night, on another whim, I applied for a Qualtrics Behavioral Research Grant (http://www.qualtrics.com/innovation-exchange/research-grants/). Or at least I think I did. I didn't receive any email confirmation. Their max is $3K.

You can't win if you don't play.

2015-09-11 status

Done

Administrivia

• Implemented changes to address requirements from IRB amendment review, based on my understanding of Office of Business and Financial Services (OBFS) policies. This is ready to be submitted, but I think we need to talk to the department's financial people (Ivy, Sherice) first to get our story absolutely straight.
• Assembled application materials for Chancellor's Graduate Research Award. Due to CS Department by 18 September at noon. Waiting on letters of support from faculty (BDE, SJ, IB).

Data Collection

• Implemented basic caret management in MDC4.

Doing

1. Auditing Music 170 (Keyboarding I), TuTh 3:30-4:45.
2. Adding note highlighting on annotation entry to MDC4.
3. Supporting caret repositioning for note clicks in MDC4.
4. Testing support for more than two voices in MDC4.
5. Condensing vertical spacing of elements in MDC4.
6. Standardizing file format for fingering.

Struggling

• Waiting for response from Scott R. Zalatoris, Policy Specialist with the Office of Business and Financial Services, to clarify generic questions on how to pay subjects.

2015-09-04 status

Done

Administrivia

• Developed plan to respond to requirements from IRB amendment review, based on my understanding of Office of Business and Financial Services (OBFS) policies.

Data Collection

• Implemented basic rendering, fingering annotation, and multiple voice support in MDC4, leveraging the abc2svg JavaScript library and its ABC parse.
• Developed "Winner's Survey" to collect information from lottery winners needed to get them paid within University regulations.

Doing

1. Implementing protocol changes to address requirements from IRB amendment review.
2. Reworking application materials for Chancellor's Graduate Research Award (née Fellowship). Statement of intent no longer required. DGS must submit by 23 September (4 p.m. CDT).
3. Leveraging abc2svg for proper caret maintenance in MDC4.
4. Standardizing file format for fingering.

Struggling

• Working with Scott R. Zalatoris, Policy Specialist with the Office of Business and Financial Services, to clarify questions on how to pay subjects.

IRB amendment bounce

Dear Barbara:

The notion of a "confidential" study seem to be a bit of a red herring. The only hard requirement from the IRS is that the University must report payments that are $600 or more in a calendar year, regardless of how the study is classified. See https://www.obfs.uillinois.edu/bfpp/section-18-taxes/section-18-10.

I think the problem is actually with inaccuracies in the OBFS policy document https://www.obfs.uillinois.edu/bfpp/section-8-payments-reimbursements/open-program-advance-pay-human-subjects. I just submitted the following comment on its content:

The text seems to imply a few things that I don't think are true. First, it says that non-confidential studies may not pay more than $100 to an individual in a tax year. I think it is trying to say that we can't pay $100 *or more* without reporting the subjects name and address to UPAY1099, as stated in https://www.obfs.uillinois.edu/bfpp/section-8-payments-reimbursements/tax-implications-for-payments . The issue here is about taxation, not about establishing an upper limit on incentives that can be paid to human subjects. Is this not correct?

Indeed, https://www.obfs.uillinois.edu/bfpp/section-18-taxes/section-18-10 says that we are not *required* to report anything if the subject receives less than $600 in a calendar year.

The policies documented in the three links provided above are not entirely consistent.

Another problem is with the blanket statement that *all* payments to human subjects must be made through a program advance. Is it not allowable to derive funds from those that have been forwarded to my department's FOAP? Again, I think recording 1099 information when payments are $600 or more is the only hard requirement. Note that https://www.obfs.uillinois.edu/bfpp/section-18-taxes/section-18-10 does not mention program advances.

Please advise.

Many thanks,
Dave

​I am going to add a "Winner's Survey" in Qualtrics to collect the information we need from the lottery winners in a secure way:

1. Are you an employee of the University of Illinois?
2. Are you required to file a federal income tax return in the United States?
3. For tax purposes, are you a non-resident alien of the United States?​
4. For income reporting purposes, please provide your mailing address:
5. Have you received a total of $500 or more in the current calendar year for participating in research studies hosted by the University of Illinois?
6. [If yes to #5] As it is required for income reporting purposes, please provide your Social Security number:

​This way, we leave the current survey(s) untouched.

​I plan simply to go through normal IRB channels to get this reviewed again. I think I understand the issues involved now, and if I was in the reviewer's shoes, I would be annoyed by email, especially from some grad student.​ 

Survey split

We want to split the survey but give people the option of continuing directly to Part II at the completion of Part I. I don't see a way to do this with separate surveys in Qualtrics, but we can just add the Part II questions in the Part I survey after a question about whether the person would like to continue:

Congratulations, you have completed the fingering exercise survey. Thank you for sharing your expertise.

The next step would be for you to answer some questions about your background and attitudes about fingering. Completing this "player profile" will earn you a chance to win another $100 Amazon gift card and will make the fingering data you have just provided even more valuable to our study. Would you like to do this now?

• Yes, I have time.

• No, maybe later.

If the subject says yes or no, we add her name, email, and response ID to the "Survey I Complete" panel (via a panel trigger) and send her a thank you email that includes the consent agreement.

If the subject says yes, we press on. When she complete the survey, we add the subject's name, email, and response ID to the "Survey II Complete" panel. We also send a thank you and copy of the consent agreement via an email trigger.

(Another important change: Add a panel trigger to add the user to the "Lottery I" panel if she says yes to participating in the Part I lottery. Do the same for the Part II lottery. This will make it easier to process the survey awards. Unfortunately, or perhaps fortunately, this mechanism cannot be used to capture contact information from people who do not complete the survey within two weeks and effectively withdraw from the study. This is because the panel updates are only triggered on survey completion. Moreover, I can't delete all information about a person while still maintaining her contact information. So I am changing the consent form to promise compensation only for in-person sessions that are terminated early. Whew.)

If the subject says no, we add her information to the "Survey II Separate" panel, so we can follow up with her later. The profile part will be advertised to members of this panel as a separate, abbreviated survey.

Lottery II contact information for this follow-up survey will be written to the "Lottery II" panel and derived from the "recipient information," which should be available because we will be using unique survey links (from the "Separate" panel) for all surveys but the first one.

Should we include a second question about participating in the lottery for the integrated Part II? What about the standalone Part II? I say yes to both.

The progress bar is really discouraging, since it includes a bunch of future questions that a lot of people won't be answering and also because it does not have any notion of weighting the effort required for a given question. The progress bar creates a distinct impression that very little progress is being made. So I am not going to show it in Survey I and am instead including a heading (e.g. "Exercise 1 of 7") at the top of each fingering exercise. This should give the subject a better sense of progress. I just wish I could start the progress bar for the embedded Part II, but this does not seem possible. So they will have to do without.

Per BDE's request, I am also changing required to initial when talking about the survey activities in the consent agreement.

These changes will be made to version 5 of the survey (now surveys).

I must add our single v4 respondent to the "Complete" and "Lottery" panels manually.

Also, the follow up survey must write the original "${e://Field/ResponseID}​" value (extracted from the panel data field "TriggerResponseID") to an embedded data field called "TriggerResponseID." Then we should be all set. This should add only minor complexity to the previously planned data plumbing. We just have another opportunity to use "TriggerResponseID," and whatever process loads the data to MySQL will have to be a teeny bit smarter. While I still need validate all this end to end, I am quite certain it will work.

2015-08-07 status

Done

Administrivia

• Started reworking application materials for Chancellor's Graduate Research Award (née Fellowship). Statement of intent no longer required. DGS must submit by 23 September (4 p.m. CDT).

Data Collection

• Added email trigger in Qualtrics to send thank you message and copy of consent document on survey completion, as promised in survey text.
• Announced "final" survey to three people I know using generic recruitment email.
• Received one response from someone I don't know.
• Acting on advice from BDE, who was rightfully worried about length of monolithic server, split survey into two parts (fingering and player profile), to be sent out separately with separate $100 lottery incentives. AFL thought this was a good idea.
• Decided fingering part should go out first.
• After a budgetary epiphany, decided that all online activities will be incentivized with separate $100 lotteries.
• Updated Qualtrics and affected IRB documents to reflect these changes.
• Redrafted recruitment emails to reflect these changes.
• Refinalized surveys, leaving question content unchanged.

Doing

1. Leveraging abc2svg for MDC4.
2. Standardizing file format for fingering.

Struggling

• Contemplating a major rewrite for MDC4, where I get out of the business of parsing abc myself. This should simplify the code greatly, but the way is not clear.
• Prelude and Fugue 18 appears to be corrupted by my hacking efforts to migrate to abc.

The curse of the monolithic consent agreement

On Thu, Aug 6, 2015 at 12:04 PM, David Randolph wrote:

Dear Barbara:

I think 45 minutes is a pretty fair estimate [for the full-blown v4 survey]. Yes, they can do it in more than one session if they use the same browser. (I should highlight this to the recruitment email.) Ivana did it with a big extra fingering exercise in about an hour total, across two sessions. She would probably do what remains in the current survey in 40 minutes.

If this is a major concern, we could split this into two parts--the player profile (10-15 minutes) and the fingering exercises (25-30 minutes) and give away a gift card for each part. But this adds complexity to keeping the pieces together, and I really need both parts.

Will we lose more people to attrition between part 1 and part 2 than we scare off with the 45 minute estimate? That is the big question. There is some value to fingering data without the player profile, I suppose.

And would this imply another run through the IRB gauntlet?

The more I think about it now, the more I think it might be worth it.

What do you think?

Thanks,

Dave

On 8/6/15 1:25 PM, David Randolph wrote:

Dear Barbara:

Another problem has occurred to me. Doling out $50 for each set of two scores is going to deplete our funds pretty rapidly. Ideally (?), we would have each of say, ten pieces, fingered by five different pianists (preferably with a range of hand sizes). This would cost 5 x 5 x $50 = $1250. After the $200 we now plan to spend on the initial surveys, we are $50 over budget, and we have nothing to pay anyone for on-campus visits and whiz-bang computer-vision experiments, which I am still pretty keen to do.

So the other change I would make is for all online activities (not just the required ones) to use lotteries for incentives. If we simply treat each of the five (most well-known to our survey respondents) pieces as separate $100 lotteries, we are covered for $500 + $200 = $700. And we still have $700 to play with. This sounds like a better approach than what I have in the consent form right now.

I am going to lock the current survey. I only have one response so far (from someone I don't know actually--I think my sister-in-law forwarded the link). We should meet soon to strategize.

Thanks,

Dave

And finally. . . .

Dear Barbara:

Yes, please hold off until we get our story straight.

I have closed the survey I sent out. I can reopen it if we decide to press on. But I think changes are in order.

I am creating alternate surveys in Qualtrics now. The protocol changes I have in mind should be pretty minor, but should improve our response rate and bang for buck. I think it is worth the extra hassle.

The idea now is to start with the (25-30 minute) fingering exercises and then do the (10-15 minute) profile questions. It might help response rates if we did it in the other order (and ease them into our clutches). But you had recommended doing the data collection first to eliminate any prejudice the questions might exert on the fingerings. So there is a trade-off to consider. Separating the questioning from the annotation in time probably makes me lean more toward starting with the shorter questioning part.

I am still worried about attrition between the two surveys, to be sure. But this seems like our best guess at an optimal approach.

Thanks,

Dave

Onion of assumption

Assumptionion?

There are a lot of assumptions we are making in our planned data collection and how it relates to the the fingerings that are actually used in performance.

1. (MDC) Fingerings for an imaginary (unmotivated, unvalidated) performance are the same as those generated for an actual performance.
2. (MDC4) Fingerings made for a performance in the past remain relevant for a new performance.
3. Entering fingerings into a web application produces the same results as those collected on paper. (Using an electronic device might disrupt usual procedures for determining fingerings.)
4. Full annotations, and not the presumably more typical minimal or targeted annotations, are accurate.
5. Annotations reflect the fingerings used in actual performance.
6. Actual performances are consistent over time.
7. Read performances are the same as memorized performances.

This all again makes me want to study how people (composers, editors, and performers) decide which notes to annotate and which to ignore. The notion of fingering ambiguity should be explored. When does it exist? How can it be measured? Are annotations primarily to flag difficult passages--warning signs in a sea of automation? Or markers of orthodoxy in a sea of freedom/chaos? Or do they, with the help of some conventions, unambiguously define the entire performance? Where are these conventions documented? Ivana and Anne should be able to help with these questions.

Rage about order

Barbara advises I think a bit more about the randomization of the fingering exercises in the first survey. Here goes. . . .

There may be a natural order for the exercises (A-G)--assuming Czerny was presenting these in an order that makes pedagogical sense. The motivation of randomizing is to disperse any influence that fingering one piece would exert on the fingering of another. But why is that important to do? Why not just let this influence exert itself. If such an effect is significant, is it not better to have it be applied in a uniform way? And if it is subtle, wouldn't testing it repeatedly help us detect it. Also, we are not randomizing the order in which A and E are presented. I think showing them in a fixed order might be the way to go. And we just stick A and E (with their coin toss on recommended fingerings) back into the original order.

The other important matter is minimizing the differences between what we did and the stated procedure of Parncutt et al.:

Each pianist received the complete score of the seven pieces. The scores included tempo and dynamic markings, but not fingerings, which had been carefully erased (by whiting out and recopying). The 12 professional pianists and 3 students received the scores after partici- pating in a companion study that involved sight-reading each piece twice. They took the scores home before writing down their preferred fingerings. The other participants received the scores by mail. None of the participants were directly paid for their services; however, the pianists who had participated in the sight-reading study were reimbursed (£25 for pro- fessionals and £5 per hour for students). 

Pianists were informed that the pieces were by Czerny, but were not told the title or opus number of the collection. They were asked to write on the score their “preferred fingerings” for the right hand of the first two measures of each piece. In order to avoid ambiguity, they were asked to write a finger number against each and every note. A preferred fingering was defined as “the fingering that you would probably use in performance.” They were asked to disregard any fingerings that they thought or knew that the composer intended for the pieces; retrospectively, we found no reason to believe that any of them were influenced by such knowledge or belief. They were also asked to write brief explanations for their choice of fingerings directly on the scores; the aim of asking for this commentary was primarily to ensure that the pianists had given the matter some thought and were sure of their fingering solutions.

Note they say nothing about the order in which the pieces were presented, and note that there were different procedures followed followed for two groups of participants. I think it safe to assume that the players in the sight-reading experiment were exposed to the pieces in a fixed order, and that all of the participants were delivered a stack of papers in a particular order. At this point, they could solve the problems in any order they chose. But it seems likely that expert pianists, unless they dropped the papers on the floor, would simply work through the pieces in order.

Our electronic survey interface allows subjects to move between the pieces as well by way of NEXT and BACK buttons, but I doubt much use of this will be made. Also, we are not asking for piece-by-piece written rationale, as I am not sure how doing so "ensures" anything about how diligent the subjects were. They say nothing about how these "commentaries" were evaluated to vet subjects--not too surprising for researchers starved for data.

Also, we are not going to tell the students who wrote the pieces, so we don't later have to tell them to ignore this fact. Also, we are simply asking them to "Enter the fingerings you would use to perform this music." We do not say anything about probability. To align with Parncutt (and likely with the variability of such things in reality), we should change this to "Enter the fingerings you would probably use to perform this music." This change can be made easily to the MDC JavaScript.

Data plumbing

Okay, the prime directive is this: No PII gets out of Qualtrics. The following should work:

1. Deliver "Survey I" (which leverages MDC) via anonymous links.
2. Use the survey settings to suppress the IP and geolocation data and leverage the Question selection when we download the CSV to skip My contact information. . . .
3. Load the data from the CSV to a MySQL (or SQLite) database, using the "ResponseID" as the primary key (our "arbitrary label") for user records.
4. Assemble a Qualtrics panel for the "Survey II"/MDC4 activities, adding "ResponseID" as a field. (The automatically generated "Survey I Complete" and "Survey II Complete" panels will include this setting as "TriggerResponseID.")
5. Deliver "Survey II" (which will leverage MDC4) via individualized links.
6. Store the original "ResponseID" (pulled from the panel record field "TriggerResponseID") as embedded data in "Survey II." You will need to use the name "TriggerResponseID," as there is no apparent way to rename this field.
7. Use the survey settings to suppress the IP and geolocation data when downloading the CSV.
8. Load to MySQL using the anonymized "TriggerResponseID" as primary key.
9. Use the same original "ResponseID" to identify subjects in the in-person SADC and FADC data collection activities.

2015-07-07 status

Done

Administrivia

• Assembled amendment materials for IRB.

Data Collection

• Integrated new Manual Data Collector (MDC) into Qualtrics survey, with low-to-high input ordering for chords (per Ivana's feedback).
• Completed pilot with Ivana.
• Drafted recruitment emails for faculty and for department staff.
• Established project web site at http://drando2.gsites.uic.edu (https://sites.google.com/a/uic.edu/didactyl).
• Compiled mailing list (Qualtrics panel) of nearly 200 piano faculty in Illinois, Iowa, Wisconsin, Indiana, Missouri, Michigan, and Ohio. The hope is that a significant subset of these people and their students will a) respond and b) hail from Chicago.
• Settled on "Open Well-Tempered Clavier" (OpenWTC) and "Open Goldberg Variations" (OpenGoldberg) for our initial corpus, as these were well-publicized (Kickstarted) efforts by MuseScore to produce high-quality editions.
• Translated OpenWTC to abc format.
• Added a little experiment to the survey for two of the fingering exercises. Half of the subjects will see the suggested Czerney fingerings.
• Finalized survey.

Doing

1. Leveraging abc2svg for MDC4.
2. Standardizing file format for fingering.

Struggling

• Contemplating a major rewrite for MDC4, where I get out of the business of parsing abc myself. This should simplify the code greatly, but the way is not clear.
• Prelude and Fugue 18 appears to be corrupted by my hacking efforts to migrate to abc.

Surveying the survey

From: David Randolph
Date: Fri, Jun 19, 2015 at 3:48 PM
Subject: Re: Help needed for piano fingering study
To: Ivana Bukvich


​I like the term *holistic*. You sound like my kind of teacher.

On trumpet, Hanon goes by the name of Arban. I hate that guy--or at least the first half of his "conservatory method." But I have always assumed that my playing stalled because I couldn't bring myself to master all of his exercises.​ Every time I pick up the horn again, there he is.

Cheers,
Dave

On Fri, Jun 19, 2015 at 2:44 PM, Ivana Bukvich wrote:


Between 8-11. Teachers use them with all age groups though. My personal opinion is that they are boring and most of the students agree. However, they can very useful. In terms of sight reading they reinforce interval if reading so that is why I like them. Also, for the same reason they are easy to transpose and useful that way. You are correct about the repetitive nature that allows one "not to think" so the motor memory is primarily used. I think that can be useful as well with an average student , but I like more "holistic" approach to practicing .


On Jun 19, 2015, at 2:18 PM, David Randolph wrote:


Dear Ivana:

I am going to leave the Bach out. We will have activities down the road for larger scale manual annotation. These activities should have their own incentives, I think, and deserve an interface with no warts on it. Moreover, we don't want to scare people off with too much work up front.

Can you tell me how old you were when you played Hanon, or, more generally, at what ages it tends to be used? I may add a question specifically about if, when, and for how long Hanon was used by the subject.

And when you say you "use" Hanon, do you mean play it yourself or use it with students? I assume the latter. I am surprised by your using it as "sight reading exercises," as my assumption is that these exercises would be the least surprising (and most internalized) pieces in the canon. They would pose technical challenges more than cognitive challenges, no? I would like to discuss these matters further. I wonder what we could do in the way of simulating a practice regimen or capturing the essence of a player's total experience with their repertoire. If we actually get to the point where we can record people's fingerings as fast as they can play, a lot of options will open up. We are long way from there, but it is fun to think about.

Are there any alternatives to Hanon? Or does it have a corner on the market of such drudgery?

Thanks,
Dave


On Fri, Jun 19, 2015 at 12:58 PM, Ivana Bukvich wrote:


Well, since I only worked on fragments it took about 15 min, it was clearly an optional activity. Hanon is probably useful as data, but , really doesn't tell you anything about fingering choices as there is only one way to play them. On the other hand,it tells something about how to finger patterns. I find them terribly boring and only use them as remedial sight reading exercises, of course , had to practice them as a child.. I agree on things you can leave out.


On Jun 19, 2015, at 11:49 AM, David Randolph wrote:


​Dear Ivana:

Thanks for the great feedback.

How long did the "Invention 1" annotation take? I expect this to add something like 30-40% to the ​time it takes to complete the survey. Was it clear that this was an optional activity? If we are keeping this in, I will add language to make it clear that this may be time consuming and is purely optional. Or do you think we should just leave it out?

We could also remove one or more of the other fingering exercises. Any thoughts on this?

Also, do you have anything similar to Hanon that you use with your students or use/used in your own studies? The thing is, we have fingering data (in machine readable form) for Hanon that we can just use. I have (vague) plans to incorporate these data as foundational fingering patterns in our models. Are you familiar with Hanon at all? Are you "anti-Hanon" for some reason? What else is out there that is similar? I am just trying to find out if there are any canonical rudiments that might influence fingering choices significantly.

Here are the changes I plan to make. . . .

I will implement the left-to-right, low-to-high fingering input you suggested. This is a good change.

I will investigate using anonymous survey links.

Having to select each of the 24 inventions had to be tedious. I will look for a way to select all or revise how we get at this information.

I will remove the questions about problematic reach between finger pairs. They show little promise of providing us anything useful.

I will add questions about any Hanon alternatives that you think are worthwhile.

Many many thanks,
Dave


On Fri, Jun 19, 2015 at 9:07 AM, Ivana Bukvich wrote:

I see, it looked like short fragments, but perhaps it was like that because of the fact that I did it on the phone, I have i-phone 5c. The whole survey took about 45 min to an one hour, it is a bit long.


On Jun 19, 2015, at 8:22 AM, David Randolph wrote:


Dear Ivana:

You should have seen the whole "Invention" score with separate input fields for each of the lines. The only problem should have been with missing continuation ties across lines. I will take another look to see if anything is missing. If you say this looks like fragments, then we may just drop this question. I have a plan to update the interface to make this look better, but I don't want to hold up the survey waiting for it.

Did you really do this whole thing on an iPhone? That is really good news. What kind of iPhone is it? I couldn't do it on my dinky little Android phone.

How long did it take you to do this in total? Do you think it is too long?

Thanks,
Dave



On Thu, Jun 18, 2015 at 10:59 PM, Ivana Bukvich wrote:


I just completed it, Bach Invention w/o score . Did you mean to have only fragments , that's what I got on I phone. I


On Thu, Jun 18, 2015 at 4:47 PM, Ivana wrote:
Hi Dave, I've just completed most of it ,and had to stop as I was on I phone and it ran out of batteries.I will continue tonight afterI get home. It looks good, small suggestion:for the left hand, when having double notes(intervals) it would be better to enter bottom finger first, then the top as in reading chords one always looks at the root of the chord first then the rest of the notes,same with fingering. I
Also, I will forward the survey to my UIC  students. Grant looks good but I have no idea how to do it really.
Sent from my iPad

Grant me a grant

• http://www.grammy.org/grammy-foundation/grants (Deadline: Oct 1?)
• Chancellor's Graduate Research Award (née Fellowship) (Submission from department September 23, 4:00 p.m. CDT)
• http://www.qualtrics.com/innovation-exchange/research-grants/ (Rolling deadline)
• http://www.ams-net.org/fellowships/ams50.php (Deadline: Dec 15)

A fool's errand in four phases

After the survey, I am leaning toward an initial follow-up study that involves manual annotation (similar to what is requested for "Invention No. 1" in the survey, but a bit more complex) of one or more selections from Bach's Well Tempered Clavier (WTC).

This is because I have discovered high-quality machine-readable (MuseScore) editions of WTC and the Goldberg Variations are available. Also, the pianists can do this work anywhere and therefore need not be local.

Oh, and building the input interface for this is a much lower risk proposition than building the sensor/computer-vision system, and the manual system is seen as a necessary component of the "semi-automated" system we also described in the Provost's Award proposal.

So the plan is 1) survey, 2) MDC4, 3) SADC, 4) FADC.

The fully automated system is still planned, but actual work on it probably won't start in earnest until school ramps up in the fall. The next development milestone will be a robust manual data collector that copes with up to four voices. (The current system only deals with two voices and has at least one problem with pieces that contain multiple lines. It is only just barely able to support the music in the survey.)

The survey has been updated to randomize the order of presenting exercises B, C, D, F, and G. Then, based on whether the subject's age is odd or even, it either presents Czerny's fingered context or a completely unfingered context for exercise A. Then it does the same for E.

I chose A and E for the experiment because they had the highest variability reported by Parncutt et al. However, I probably should have normalized by the length of fingered fragment. A and E are the longest fragments at 8 notes each.

• A: 10 fingerings / 8 notes = 1.25 fingering/note 
• B: 5 fingerings / 4 notes = 1.25 
• C: 9 / 5 = 1.8 
• D: 8 / 7 = 1.143 
• E: 18 / 8 = 2.25 
• F: 5 / 6 = 0.833 
• G: 9 / 7 = 1.286 

But this doesn't really measure the consensus I see, as A and E clearly have the widest disagreement. I need to get my head around this.

Agreement, expressed as mode count over total fingering count.

• A: 8/28 = 0.2857 
• B: 17/28  = 0.6071
• C: 10/28 = 0.3571
• D: 15/28  = 0.5375
• E: 4/28 = 0.1429 
• F: 23/28 = 0.8214
• G: 14/28  = 0.5000

This seems like one avenue to a Kappa score. Or maybe a weighted Kappa score, as the fingers do indeed have a natural order. Or should we think of every note as being "categorized" with a particular (weighted) finger? This probably amounts to the same thing.

I think the average edit distance from the mode (most popular fingering) gets us a pretty good measure of overall similarity for a bunch of fingerings. The greater the consensus on a single fingering, the more edit distances of zero we have. We can normalize the edit distance by dividing by the note count. This should give us a number between 0 (perfect agreement) and 1 (no agreement).

But what about the case, like piece A, where there are two popular--and quite dissimilar--fingerings? Is this showing agreement or disagreement? Using our normalized edit distance approach, this is going to register as disagreement. Is this fair? Should we be calculating all of the distances between all of the fingerings instead? Yes, I think so.

This probably just boils down to Fleiss's Kappa. But can this be weighted? It seems as though this should be possible.

I have also added a mechanism in the survey to measure how much time the user spends on each fingering exercise, though it is still unclear what happens to this data the user backtracks and visits the exercise more than once. We apparently need to keep the BACK button active to cope with the inconsistent way that different browsers cope with our error message for incomplete fingerings. (Safari takes you to the next screen. Chrome leaves you on the current screen after sometimes printing a confusing message about the BACK button.

Zombie survey

The survey just won't die.

I am thinking that we need to do something more experimental with the fingering exercises. In including the full exercises, it occurs to me that we could include a few exercises with Czerny's fingerings included. This plus a measure of variability would give us a way to measure the influence and see how far it reaches across hand sizes. I really think this is worth doing, but how do we pick the exercises that tip their hands? The pieces with the highest and lowest variability? Randomly?

Having timing information would also be useful--if we randomly show fingerings or not (and make it so the surveys are not uniform). I know Barbara will worry about this, as we will need more subjects to say anything with statistical significance. Can we mitigate this somehow?

We have 7 exercises. In half the surveys (the experimental group), we show 3 fingered exercises and 4 unfingered.  In the other half (the control group), all pieces are unfingered. The pieces are displayed in random order.

Throw out the piece with three voices?

I also want to ask about specific pieces that we actually have transcriptions for (or are willing to commit to transcribe). The idea is to take the intersection of a university or publisher "standard baroque/classical repertoire" and the contents available at MuseScore. We could even provide links to the current transcriptions, since I am not sure how memorable the titles are going to be, even to people who know the pieces well.

Is there a "standard" Czerny method that competes with Hanon? Are there any other comparable methods with significant mindshare?

2015-06-02 status

Done

Administrivia

• Received notification that CS Department had provided feedback on annual review. Asked Santhi how to access.

Data Collection

• Integrated new Manual Data Collector (MDC) into Qualtrics survey.
• Fixed Safari MDC rendering problems.
• Added cursor management on failed input validation.
• Improved note highlighting.
• Fixed problem annotating notes with accidentals.
• Revised survey recruitment email and integrated it into Qualtrics.
• Validated end-to-end survey flow in Qualtrics.
• Discovered rendering problem with multiline tunes.
• Drafted alternative "MDC2" interface for longer tunes with scrolling windows for both input and rendered output.
• Identified performance problem with abcjs library for longer tunes.
• Opened several issues against abcjs.
• Discovered and evaluated abcjs alternative abc2svg and found it much more performant.
• Removed several questions from survey and debugged several problems.
• Decided to remove "bonus" annotation question from survey and to plan for a manual data collection follow-up survey with a separate $100 gift card lottery.
• Decided to use email to send gift cards to winners.

Doing

1. Updating IRB documents to reflect changing plans.
2. Piloting abbreviated survey with my friend Alex.
3. Defining survey distribution lists.
4. Migrating MDC2 to abc2svg.

Struggling

• The UI for MDC2 needs to keep everything visible and rendering quickly.
• I think we should have three consent forms for the three planned data collection stages. Subjects don't need to learn about things they aren't going to do.

2015-05-18 status

Done

Administrivia

• $2000 from Provost's Award apparently received by CS Department.
• Had phone interview with Boaz Super. Job search losing momentum.

Data Collection

• Integrated new Manual Data Collector (MDC) into Qualtrics survey.
• Added data validation/retrieval method to MDC.
• Added support for remembering data input after BACK button clicked in survey.
• Added support to validate and remember data after NEXT click.
• Fixed problem where trailing non-notes go missing in MDC.
• Fixed problem where forte dynamic marks are confused with f notes.
• Added feature to highlight note in score corresponding to fingering being entered (caret location).
• Added bonus optional annotation item.
• Added nag screen for all incomplete questions.
• Adopted new official release patch of ABCJS to address annotations overwriting each other.
• Decided on $100 gift card lottery and extra chance for annotating Bach's "Invention No. 1."
• Tabling efforts to support more than two "voices" in MDC. Affects only one note in survey.
• Eita Nakamura needs more time to share data. Pointed him to our little corpus.

Doing

1. Defining survey distribution lists.
2. Fixing Safari MDC rendering problems.
3. Piloting survey with my wife.
4. Pinging IRB to ask about lottery protocol.
5. Updating IRB documents to reflect changing plans.

Struggling

• MDC and survey look good in Chrome. Multiline scores not drawing correctly in Safari.

2015-05-01 status

Done

Administrivia

• Spoke with Daniel Murphy. He agreed to wait list me for the group piano class next semester, and he can waive he corequisites.
• Pinged Michael Anderson again about using the piano lab. No reply.
• Confirmed $2000 from Provost's Award. Funds available for work 1 Jun 2015 through 30 Jun 2016.

Data Collection

• Various bug fixes in MDC.
• Added help page to MDC.
• Obtained patch of ABCJS to address annotations overwriting each other.
• Was promised patch of ABCJS to fix annotations overwriting notes. Developer Paul Rosen is on the job, which is great news.
• Added graphics to scale questions in survey.
• Pinged Michael Anderson again. No reply. I will try calling next.
• Saw Ivana in the piano lab. They have two Canon VIXIA HFR500 cameras we might be able to use. The camera set up is over an acoustic piano, which we can't move. But there are shelf mounts we could use to rig something for an electronic keyboard. This is still the preferred venue if we can get permission to use it.

Doing

1. Integrating new Manual Data Collector (MDC) into Qualtrics survey. Gnashing teeth.
2. Defining survey distribution lists.
3. Weighing lottery options for survey. This is definitely the direction we will take.
4. Trying to quantify variability in the Parncutt sight-reading corpus with a mind toward defining statistical approaches, as requested in protocol template.
5. Trying to run finger6 over input file.
6. Playing with Synthesia AFP (one .exe, one .dll, and that's it).
7. Collecting all the Synthesia data there is from their forum.
8. Implementing basic color tracking synchronized with MIDI events.

Struggling

• Interviewing with SAP (hybris) and Braintree.
• More than two "voices" choke MDC.

Canon VIXIA HFR500

There are two of these bad boys in the Piano Lab at UIC. How can we use them?

2015-04-24 status

Done

Administrivia

• Defying all odds, won me one of those Provost's Awards. No word from Graduate College. I wonder about the dollar amount. I think it has to be at least $1K. Amazing.
• Completed end-of-year progress review.
• Moved Didactyl repo to https://github.com/dvdrndlph/didactyl, which will be its permanent home.

Data Collection

• Debugged problem with ornamented notes in MDC.
• Played with new release of ABCJS. The annotations are bigger, don't seem to scale with the rest of the score, and still overwrite note stems and beams. Not much of an upgrade for my purposes.
• Opened issue against ABCJS regarding annotation positioning.
• Emailed various paper authors about sharing training data. One response from Eita Nakamura. Promises to send data in two weeks.
• Per CR's suggestion, contacted Eric Nichols (via email) and Alia Al Kasimi (through Facebook) about the data and heuristic questions used in their paper. Would like to see these questions with a mind toward adapting them for the survey.
• Cloned an orphaned MusicXML JavaScript project known as score-library from the now-defunct Google Code to my GitHub account. If ABC turns out to be too limiting, I might try to resuscitate this project.

Doing

1. Integrating new Manual Data Collector (MDC) into Qualtrics survey.
2. Trying to quantify variability in the Parncutt sight-reading corpus with a mind toward defining statistical approaches, as requested in protocol template.
3. Trying to run finger6 over input file.
4. Playing with Synthesia AFP (one .exe, one .dll, and that's it).
5. Collecting all the Synthesia data there is from their forum.
6. Implementing basic color tracking synchronized with MIDI events.

Struggling

• Motorola is really starting to cramp my style, as cost reduction is on everybody's mind.
• Working on my resume, even applying for a few jobs. Got a bong email from Narrative Science, a startup that grew out of NLP research at Northwestern. My new resume did nothing for them.

2015-04-10 status

Done

Administrivia

• Pinged Daniel Murphy about taking Music 170 (group piano lessons). No reply.
• Formally requested support from Michael J. Anderson, head of the Department of Music. No reply.

Data Collection

• Drafted more survey questions about attitudes toward fingering.
• Improved MDC JavaScript tool to support grand staffs, chords, grace notes, and ornamented notes.
• Made annotation positionings configurable in MDC.

Doing

1. JavaScripting the Manual Data Collector (MDC).
2. Finalizing survey instrument by April 20 (when decision on Provost's Award is due).
3. Contacting prior system developers to request data.
4. Trying to quantify variability in the Parncutt sight-reading corpus with a mind toward defining statistical approaches, as requested in protocol template.
5. Trying to run finger6 over input file.
6. Playing with Synthesia AFP (one .exe, one .dll, and that's it).
7. Collecting all the Synthesia data there is from their forum.
8. Implementing basic color tracking synchronized with MIDI events.

Struggling

• New Scrum process being implemented at work.

2015-03-20 survey

Done

Administrivia

• Received exemption from IRB. I wasn't expecting that.
• Exchanged email with Richard Parncutt regarding committee membership. He is open to the idea, but seemed to have a hard time with the concept of a committee member being both examiner and consultant on the research. He thinks he should be consulting with faculty on the matter, not me. 

Data Collection

• Defined more survey questions about hand size (reach) in Qualtrics.
• Improved MDC JavaScript tool to support slurs, ties, and annotations in ABC.
• Deployed new MDC to Qualtrics.
• Met with IB to refine survey. It did not appear to be too long at all, even with the fingering exercises, some of which she could do almost at a glance. We started talking about content for the in-person data collection, but this is still an open question. The piano lab now has one piano outfitted with an overhead camera and a secondary camera on a tripod. We may be able to leverage some of this equipment for data collection. 

Doing

1. JavaScripting the Manual Data Collector (MDC).
2. Requesting letter of support from head of the Department of Music.
3. Trying to quantify variability in the Parncutt sight-reading corpus with a mind toward defining statistical approaches, as requested in protocol template.
4. Trying to run finger6 over input file.
5. Playing with Synthesia AFP (one .exe, one .dll, and that's it).
6. Collecting all the Synthesia data there is from their forum.
7. Implementing basic color tracking synchronized with MIDI events.

Struggling

• A lot of work at work.

2015-03-13 status

Done

Administrivia

• Submitted IRB application for expedited review.
• Submitted application for Provost's Award.

Data Collection

• Refined survey questions in Qualtrics.
• Proved concept of embedding MDC JavaScript tool in Qualtrics.
• Stored required JavaScript libraries on my UIC Google Drive.
• Explored VexTab as alternative to abcjs for MDC. The +1+ fingering annotations are not well rendered in abcjs, but text annotations work well. So I decided to stick with abcjs.

Doing

1. JavaScripting the Manual Data Collector (MDC).
2. Meeting with IB to refine survey questions.
3. Trying to quantify variability in the Parncutt sight-reading corpus with a mind toward defining statistical approaches, as requested in protocol template.
4. Trying to run finger6 over input file.
5. Playing with Synthesia AFP (one .exe, one .dll, and that's it).
6. Collecting all the Synthesia data there is from their forum.
7. Implementing basic color tracking synchronized with MIDI events.

Struggling

• We need to define the set of content we want to focus on. The last survey question will ask subjects to identify/select pieces that raise fingering difficulties. We need to narrow down the list of usual suspects in a way that will make the whole process tractable and scalable. I don't want to have to typeset of bunch of music.

Second sight

Could sight reading performance be improved with full (automatically generated and customized) fingerings? Or is this too much to process? Would full annotation affect look-ahead span a la Sloboda? How much annotation is ideal in a sight reading situation? Does this vary by individual? There is a literature on the subject of sight reading. Maybe I should (sight) read it.

What is the optimal amount of fingering annotation? How do editors currently decide what notes to annotate? Instead of aligning to the full annotations of performers, we could try to predict the partial annotations of editors. Ultimately we want to generate the most useful annotations. This implies determining both the correct fingerings and the ideal set of notes to be annotated. We could have a verbosity knob on the application, or the application could learn the player's preferred level of verbosity. Annotations could disappear from the displayed score, as the program determines they are note needed. They could reappear if the player falters. Would this drive the player nuts? Would they demand a static representation? Would they do this because that is what they are used to (because that is how paper works) or because this has inherent merit. Does this inherent merit, if it exists, apply only to performance situations, but not to a practice situations?

The idealized performance is memorized. As progress is made toward this ideal, should not fingering annotations gradually vanish? What about other annotations? What should disappear last--the notes, dynamic contours? Could practice for (memorized) performance benefit from this sort of reverse scaffolding?

Questions about the adequacy of editorial annotations should be included in our survey. What do the people think? Are the annotations useful even if they are ignored or crossed out? Pianists say they are at least some of the time (interpretation uber alles), but can this utility be measured?

Content manifesto

KernScores' ABC links are all broken.

KernScores seems to be the best source of symbolic music, not IMSLP, which really just gives us PDF versions of public-domain music. But it is not clear if KernScores is still being curated.

We need ABC or a way to translate data to ABC if our MDC solution based on ABC is to have legs. Guido and MuseData (md2) formats are present, but their renderings are rather unsightly. These formats do not have the JAvaScript support that make ABC and Vex attractive. Need to try hum2abc, but I am dubious. Another thing to try, since MusicXML format is available, is xml2abc. 

Will we live to regret the design decision to standardize on ABC?! I think it is sufficiently expressive, but is there something newer, cheaper, faster, stronger that we are overlooking?

ABC gives us the following awesome things:

• Readability
• Rich toolset for high quality rendering
• A JavaScript library (abcjs) for easy implementation of MDC
• Interoperability with music21.
• MIDI note numbers
• In the Dactyler soup already

All we need, as I see things now, is a clear (read: automated?) path from KernScores to ABC.

The last survey question will ask subjects to pick piece that raise fingering difficulties. I really hope this is a question they can answer, or we can do something to tease this information out.

To do  this, we need to narrow down the list of usual suspects in a way that will make the whole process tractable and scalable. I don't want to have to typeset of bunch of music. 

A modest proposal: We limit the size of our repertoire to the 108,703 scores available through KernScores (and the few other scraps we are able to create or scrounge from Synthesia). We start with some subset of Bach. Then move through Mozart and Beethoven, or stay in the baroque period. Perhaps the latter approach is preferred, as I think we want to avoid polyphony as much as possible. Two hands playing intertwining monophonic (rapid or finger legato) lines are the focus. One-voice, then two-voice, then three-voice inventions? Maybe we just stick with Bach?