Survey split

We want to split the survey but give people the option of continuing directly to Part II at the completion of Part I. I don't see a way to do this with separate surveys in Qualtrics, but we can just add the Part II questions in the Part I survey after a question about whether the person would like to continue:

Congratulations, you have completed the fingering exercise survey. Thank you for sharing your expertise.

The next step would be for you to answer some questions about your background and attitudes about fingering. Completing this "player profile" will earn you a chance to win another $100 Amazon gift card and will make the fingering data you have just provided even more valuable to our study. Would you like to do this now?

• Yes, I have time.

• No, maybe later.

If the subject says yes or no, we add her name, email, and response ID to the "Survey I Complete" panel (via a panel trigger) and send her a thank you email that includes the consent agreement.

If the subject says yes, we press on. When she complete the survey, we add the subject's name, email, and response ID to the "Survey II Complete" panel. We also send a thank you and copy of the consent agreement via an email trigger.

(Another important change: Add a panel trigger to add the user to the "Lottery I" panel if she says yes to participating in the Part I lottery. Do the same for the Part II lottery. This will make it easier to process the survey awards. Unfortunately, or perhaps fortunately, this mechanism cannot be used to capture contact information from people who do not complete the survey within two weeks and effectively withdraw from the study. This is because the panel updates are only triggered on survey completion. Moreover, I can't delete all information about a person while still maintaining her contact information. So I am changing the consent form to promise compensation only for in-person sessions that are terminated early. Whew.)

If the subject says no, we add her information to the "Survey II Separate" panel, so we can follow up with her later. The profile part will be advertised to members of this panel as a separate, abbreviated survey.

Lottery II contact information for this follow-up survey will be written to the "Lottery II" panel and derived from the "recipient information," which should be available because we will be using unique survey links (from the "Separate" panel) for all surveys but the first one.

Should we include a second question about participating in the lottery for the integrated Part II? What about the standalone Part II? I say yes to both.

The progress bar is really discouraging, since it includes a bunch of future questions that a lot of people won't be answering and also because it does not have any notion of weighting the effort required for a given question. The progress bar creates a distinct impression that very little progress is being made. So I am not going to show it in Survey I and am instead including a heading (e.g. "Exercise 1 of 7") at the top of each fingering exercise. This should give the subject a better sense of progress. I just wish I could start the progress bar for the embedded Part II, but this does not seem possible. So they will have to do without.

Per BDE's request, I am also changing required to initial when talking about the survey activities in the consent agreement.

These changes will be made to version 5 of the survey (now surveys).

I must add our single v4 respondent to the "Complete" and "Lottery" panels manually.

Also, the follow up survey must write the original "${e://Field/ResponseID}​" value (extracted from the panel data field "TriggerResponseID") to an embedded data field called "TriggerResponseID." Then we should be all set. This should add only minor complexity to the previously planned data plumbing. We just have another opportunity to use "TriggerResponseID," and whatever process loads the data to MySQL will have to be a teeny bit smarter. While I still need validate all this end to end, I am quite certain it will work.

2015-08-07 status

Done

Administrivia

• Started reworking application materials for Chancellor's Graduate Research Award (née Fellowship). Statement of intent no longer required. DGS must submit by 23 September (4 p.m. CDT).

Data Collection

• Added email trigger in Qualtrics to send thank you message and copy of consent document on survey completion, as promised in survey text.
• Announced "final" survey to three people I know using generic recruitment email.
• Received one response from someone I don't know.
• Acting on advice from BDE, who was rightfully worried about length of monolithic server, split survey into two parts (fingering and player profile), to be sent out separately with separate $100 lottery incentives. AFL thought this was a good idea.
• Decided fingering part should go out first.
• After a budgetary epiphany, decided that all online activities will be incentivized with separate $100 lotteries.
• Updated Qualtrics and affected IRB documents to reflect these changes.
• Redrafted recruitment emails to reflect these changes.
• Refinalized surveys, leaving question content unchanged.

Doing

1. Leveraging abc2svg for MDC4.
2. Standardizing file format for fingering.

Struggling

• Contemplating a major rewrite for MDC4, where I get out of the business of parsing abc myself. This should simplify the code greatly, but the way is not clear.
• Prelude and Fugue 18 appears to be corrupted by my hacking efforts to migrate to abc.

The curse of the monolithic consent agreement

On Thu, Aug 6, 2015 at 12:04 PM, David Randolph wrote:

Dear Barbara:

I think 45 minutes is a pretty fair estimate [for the full-blown v4 survey]. Yes, they can do it in more than one session if they use the same browser. (I should highlight this to the recruitment email.) Ivana did it with a big extra fingering exercise in about an hour total, across two sessions. She would probably do what remains in the current survey in 40 minutes.

If this is a major concern, we could split this into two parts--the player profile (10-15 minutes) and the fingering exercises (25-30 minutes) and give away a gift card for each part. But this adds complexity to keeping the pieces together, and I really need both parts.

Will we lose more people to attrition between part 1 and part 2 than we scare off with the 45 minute estimate? That is the big question. There is some value to fingering data without the player profile, I suppose.

And would this imply another run through the IRB gauntlet?

The more I think about it now, the more I think it might be worth it.

What do you think?

Thanks,

Dave

On 8/6/15 1:25 PM, David Randolph wrote:

Dear Barbara:

Another problem has occurred to me. Doling out $50 for each set of two scores is going to deplete our funds pretty rapidly. Ideally (?), we would have each of say, ten pieces, fingered by five different pianists (preferably with a range of hand sizes). This would cost 5 x 5 x $50 = $1250. After the $200 we now plan to spend on the initial surveys, we are $50 over budget, and we have nothing to pay anyone for on-campus visits and whiz-bang computer-vision experiments, which I am still pretty keen to do.

So the other change I would make is for all online activities (not just the required ones) to use lotteries for incentives. If we simply treat each of the five (most well-known to our survey respondents) pieces as separate $100 lotteries, we are covered for $500 + $200 = $700. And we still have $700 to play with. This sounds like a better approach than what I have in the consent form right now.

I am going to lock the current survey. I only have one response so far (from someone I don't know actually--I think my sister-in-law forwarded the link). We should meet soon to strategize.

Thanks,

Dave

And finally. . . .

Dear Barbara:

Yes, please hold off until we get our story straight.

I have closed the survey I sent out. I can reopen it if we decide to press on. But I think changes are in order.

I am creating alternate surveys in Qualtrics now. The protocol changes I have in mind should be pretty minor, but should improve our response rate and bang for buck. I think it is worth the extra hassle.

The idea now is to start with the (25-30 minute) fingering exercises and then do the (10-15 minute) profile questions. It might help response rates if we did it in the other order (and ease them into our clutches). But you had recommended doing the data collection first to eliminate any prejudice the questions might exert on the fingerings. So there is a trade-off to consider. Separating the questioning from the annotation in time probably makes me lean more toward starting with the shorter questioning part.

I am still worried about attrition between the two surveys, to be sure. But this seems like our best guess at an optimal approach.

Thanks,

Dave

Onion of assumption

Assumptionion?

There are a lot of assumptions we are making in our planned data collection and how it relates to the the fingerings that are actually used in performance.

1. (MDC) Fingerings for an imaginary (unmotivated, unvalidated) performance are the same as those generated for an actual performance.
2. (MDC4) Fingerings made for a performance in the past remain relevant for a new performance.
3. Entering fingerings into a web application produces the same results as those collected on paper. (Using an electronic device might disrupt usual procedures for determining fingerings.)
4. Full annotations, and not the presumably more typical minimal or targeted annotations, are accurate.
5. Annotations reflect the fingerings used in actual performance.
6. Actual performances are consistent over time.
7. Read performances are the same as memorized performances.

This all again makes me want to study how people (composers, editors, and performers) decide which notes to annotate and which to ignore. The notion of fingering ambiguity should be explored. When does it exist? How can it be measured? Are annotations primarily to flag difficult passages--warning signs in a sea of automation? Or markers of orthodoxy in a sea of freedom/chaos? Or do they, with the help of some conventions, unambiguously define the entire performance? Where are these conventions documented? Ivana and Anne should be able to help with these questions.

Rage about order

Barbara advises I think a bit more about the randomization of the fingering exercises in the first survey. Here goes. . . .

There may be a natural order for the exercises (A-G)--assuming Czerny was presenting these in an order that makes pedagogical sense. The motivation of randomizing is to disperse any influence that fingering one piece would exert on the fingering of another. But why is that important to do? Why not just let this influence exert itself. If such an effect is significant, is it not better to have it be applied in a uniform way? And if it is subtle, wouldn't testing it repeatedly help us detect it. Also, we are not randomizing the order in which A and E are presented. I think showing them in a fixed order might be the way to go. And we just stick A and E (with their coin toss on recommended fingerings) back into the original order.

The other important matter is minimizing the differences between what we did and the stated procedure of Parncutt et al.:

Each pianist received the complete score of the seven pieces. The scores included tempo and dynamic markings, but not fingerings, which had been carefully erased (by whiting out and recopying). The 12 professional pianists and 3 students received the scores after partici- pating in a companion study that involved sight-reading each piece twice. They took the scores home before writing down their preferred fingerings. The other participants received the scores by mail. None of the participants were directly paid for their services; however, the pianists who had participated in the sight-reading study were reimbursed (£25 for pro- fessionals and £5 per hour for students). 

Pianists were informed that the pieces were by Czerny, but were not told the title or opus number of the collection. They were asked to write on the score their “preferred fingerings” for the right hand of the first two measures of each piece. In order to avoid ambiguity, they were asked to write a finger number against each and every note. A preferred fingering was defined as “the fingering that you would probably use in performance.” They were asked to disregard any fingerings that they thought or knew that the composer intended for the pieces; retrospectively, we found no reason to believe that any of them were influenced by such knowledge or belief. They were also asked to write brief explanations for their choice of fingerings directly on the scores; the aim of asking for this commentary was primarily to ensure that the pianists had given the matter some thought and were sure of their fingering solutions.

Note they say nothing about the order in which the pieces were presented, and note that there were different procedures followed followed for two groups of participants. I think it safe to assume that the players in the sight-reading experiment were exposed to the pieces in a fixed order, and that all of the participants were delivered a stack of papers in a particular order. At this point, they could solve the problems in any order they chose. But it seems likely that expert pianists, unless they dropped the papers on the floor, would simply work through the pieces in order.

Our electronic survey interface allows subjects to move between the pieces as well by way of NEXT and BACK buttons, but I doubt much use of this will be made. Also, we are not asking for piece-by-piece written rationale, as I am not sure how doing so "ensures" anything about how diligent the subjects were. They say nothing about how these "commentaries" were evaluated to vet subjects--not too surprising for researchers starved for data.

Also, we are not going to tell the students who wrote the pieces, so we don't later have to tell them to ignore this fact. Also, we are simply asking them to "Enter the fingerings you would use to perform this music." We do not say anything about probability. To align with Parncutt (and likely with the variability of such things in reality), we should change this to "Enter the fingerings you would probably use to perform this music." This change can be made easily to the MDC JavaScript.

Data plumbing

Okay, the prime directive is this: No PII gets out of Qualtrics. The following should work:

1. Deliver "Survey I" (which leverages MDC) via anonymous links.
2. Use the survey settings to suppress the IP and geolocation data and leverage the Question selection when we download the CSV to skip My contact information. . . .
3. Load the data from the CSV to a MySQL (or SQLite) database, using the "ResponseID" as the primary key (our "arbitrary label") for user records.
4. Assemble a Qualtrics panel for the "Survey II"/MDC4 activities, adding "ResponseID" as a field. (The automatically generated "Survey I Complete" and "Survey II Complete" panels will include this setting as "TriggerResponseID.")
5. Deliver "Survey II" (which will leverage MDC4) via individualized links.
6. Store the original "ResponseID" (pulled from the panel record field "TriggerResponseID") as embedded data in "Survey II." You will need to use the name "TriggerResponseID," as there is no apparent way to rename this field.
7. Use the survey settings to suppress the IP and geolocation data when downloading the CSV.
8. Load to MySQL using the anonymized "TriggerResponseID" as primary key.
9. Use the same original "ResponseID" to identify subjects in the in-person SADC and FADC data collection activities.

2015-07-07 status

Done

Administrivia

• Assembled amendment materials for IRB.

Data Collection

• Integrated new Manual Data Collector (MDC) into Qualtrics survey, with low-to-high input ordering for chords (per Ivana's feedback).
• Completed pilot with Ivana.
• Drafted recruitment emails for faculty and for department staff.
• Established project web site at http://drando2.gsites.uic.edu (https://sites.google.com/a/uic.edu/didactyl).
• Compiled mailing list (Qualtrics panel) of nearly 200 piano faculty in Illinois, Iowa, Wisconsin, Indiana, Missouri, Michigan, and Ohio. The hope is that a significant subset of these people and their students will a) respond and b) hail from Chicago.
• Settled on "Open Well-Tempered Clavier" (OpenWTC) and "Open Goldberg Variations" (OpenGoldberg) for our initial corpus, as these were well-publicized (Kickstarted) efforts by MuseScore to produce high-quality editions.
• Translated OpenWTC to abc format.
• Added a little experiment to the survey for two of the fingering exercises. Half of the subjects will see the suggested Czerney fingerings.
• Finalized survey.

Doing

1. Leveraging abc2svg for MDC4.
2. Standardizing file format for fingering.

Struggling

• Contemplating a major rewrite for MDC4, where I get out of the business of parsing abc myself. This should simplify the code greatly, but the way is not clear.
• Prelude and Fugue 18 appears to be corrupted by my hacking efforts to migrate to abc.