Talked to AFL on about all manners pianistic. Discussed a few new ideas for data collection: 1) comparison of different editorial editions of the same score, 2) interview an editor (who Anne thinks tend to be academics) about his or her process with respect to fingering annotations, 3) include fill-in-the-blank exercises to try to see where the non-controversial and the controversial meet, 4) give one group partially annotated and another group fully annotated, 5) do a case study of a pianist's personal library--transcribe a set of scores, but how to select? What gets added? What gets changed? I may need to bring a scanner when I go to Albuquerque at the end of March.
Scheduled meeting Monday at 11 with Dr. Zefran to discuss sensors, cybergloves, and all matters electronic.
Doing
Drafting consent request for IRB.
Drafting recruitment email for IRB.
Trying to run finger6 over input file.
Playing with AFP (one .exe, one .dll, and that's it).
Trying to quantify variability in the Parncutt sight-reading corpus.
Collecting all the Synthesia data there is from their forum.
Implementing basic color tracking synchronized with MIDI events.
Trying to get data from IMU sensor through Arduino and into Minion (my cleverly named Mac Mini).
The survey will be authored in Qualtrics. It even supports JavaScript, so I am hoping to integrate the MDC right into the survey tool. Here are some ideas for questions to include. I will update this post as new ideas occur to me.
What is your gender?
Male
Female
What is your age? _____
I am left/right handed.
I use conventional fingerings for scales.
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
I use conventional fingerings for broken chord exercises.
I use conventional fingerings for arpeggio exercises.
I adhere to fingerings suggested by Hanon.
I adhere to fingerings suggested by Czerny.
What is your glove size? Use this chart if you are not sure. (Make sure to print it out at 100% size. Your application may try to scale it. The unscaled image should fit on an 8.5x11 inch piece of paper.)
< 6
6
7
8
9
10
11
12
12
14
>14
I wish my hands were larger/smaller/neither.
If you have ready access to a tape measure, measure distance around your palm. _____ (inches/mm)
Do you own a MIDI-enabled keyboard? If so, how often do you play it?
What piano method(s) did you use regularly when you were developing your skills as a pianist?
Hanon
Specific Czerny?
How does the action of your favorite MIDI keyboard (not including a MIDI-enabled acoustic piano) compare to your favorite acoustic piano?
How much do you change your fingerings when playing your favorite MIDI keyboard, compared to playing your favorite acoustic piano?
How much do your fingering choices change from performance to performance of the same piece?
Describe your approach to sight-reading.
What percentage of the time that you spend playing the piano is devoted to each of the following activities?
___ Private practice
___ Solo performance
___ Performance with instrumental ensemble
___ Accompaniment
How much of your practice and performance time is spent sight reading? ____%
How much of your practice and performance time is spent playing prepared pieces? ____%
In the past month, on how many days have you played the piano?
In the past month, on how many days have you performed the piano for a public audience?
How many years have you studied and/or performed the piano on a regular basis?
Overall, I am happy with my finger strength.
In general, I am happy with my reach.
Playing the piano is slightly/somewhat/very painful for me.
Very frequently
Frequently
Sometimes
Rarely
Never
The reach between the following finger pairs are problematic for me. Check all that apply.
1-5
1-4
1-3
1-2
2-3
2-4
2-5
3-4
3-5
4-5
Which of the following best describes you?
Professional pianist
Professional piano teacher
Professional pianist and piano teacher
Expert pianist
Semi-professional pianist
Serious amateur pianist
Other: _________________________________
Compared to other pianists, how would you describe the size of your hands?
Very small
Somewhat small
Average size
Somewhat large
Very large
How technically challenging is the music you play most frequently?
How much of your time spent in preparing a piece for performance is devoted to determining the fingering sequence?
Less than 5%
5-10%
11-20%
21-30%
More than 30%
How do you feel about the following statement: "Learn the printed fingerings before considering alternatives?"
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
How do you feel about this statement: "Determining effective fingerings is time consuming for me when preparing a technically challenging piece."
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
As I become more familiar with a technically challenging piece, my fingerings evolve. For some pieces, I cannot settle on the best fingerings. Cite examples of pieces in your repertoire for which it was particularly difficult to determine fingerings. Cite at least one piece, if you can, for which fingering decisions are still in flux for you. When did you first start practicing this piece?
How often do you seek guidance from others on how best to finger a piece?
How often have you crossed out a fingering suggested by an editor?
How often have you crossed out a fingering suggested by a composer?
Who (or what) do you consult when challenged by the fingering of a piece? (Check all that apply.)
Pedagogical texts
Other editions of the same score
Other personally annotated scores from your repertoire
Teachers/mentors
Peers (other pianists)
Online forums
How much do you typically write on a printed score for a technically challenging piece?
A lot
A fair amount
Some
Hardly any
None at all
How often do you deviate from editorial fingering annotations printed in a score?
Never
Rarely
Sometimes
Frequently
Very frequently
Annotation Corpus
Hand legato or fast tempi
Two-hand interplay
Established Repertoire
Bach Inventions and Sinfonias
Well-Tempered Clavier
Hanon's Virtuoso Pianist in 60 Exercises
What does everyone know? What methods do everyone use (daily)?
AF = Angus Forbes
AFL = Anne Ferguson Laird
BDE = Barbara Di Eugenio
CR = Christopher Raphael
DM = David Meyer
DS = David Schrader
IB = Ivana Bukvich
MZ = Milos Zefran
PG = Piotr Gmytrasiewicz
PL = Paul Landes
RBS = R. Benjamin Sutherland
RP = Richard Parncutt
SJ = Steven Jordan
TG = Tim Gutowski
On BDE's advice, I am going to try for the Provost's Award at UIC. This is a grant-like application process, which awards up to $2K sans matching funds or $3K if you can get your department or advisor to kick in $500. They expect, but do not require, PhD students to have advanced past the qualifier, so this will hurt my chances until that hurdle is passed. Still, I think it is worth a shot. The more times I try, the more chances of success.
This is offered twice a year, with deadlines in mid March and mid October.
Let there be light (emitting diodes). I feel as though I am approaching some sort of nerdvana with the acquisition of some actual electronics. I was a bit surprised at how insecure the jumper connections were. I really do need to find that soldering iron and learn how to use it.
Initial Automatic Data Collector (ADC) sensor components.
Arduino software now needs to be programmed, at which point my neck will cease to have a profile and the Singularity will approach.
Where the magic may or may not happen
Also on display are some pretty serious carpentry skills. I screwed a wall mount into a 2x4 so I could aim the Kinect down at the pianist's unsuspecting fingers. I used a cordless drill that had been untouched since I got it I don't know how many Christmases ago.
Piotr Gmytrasiewicz has agreed serve on prelim committee. So we have five members now.
I floated the idea to Richard Parncutt for him to be on the thesis committee. I don't think we need him for the prelim, but I think he would be invaluable for the long game. I received an auto-response that he is on holiday. It seems there is no school in February in Austria. Who knew?
Data Collection
Received Arduino board, wires, and one sensor.
Discovered application of vibration sensors to piano fingering detection. This may be the best way to augment CV.
Got color tracking code to work with my Kinect. It looks quite promising.
Discovered Synthesia project as potential source for fingering data.
Met with Steven Jordan at Loyola. Trying to define procedures for collecting anthropometric (hand) data.
WebService
Stumbled on a significant effort to develop a plugin to Synthesia known as Automatic Fingering Prediction (AFP), expressly building on the Parncutt, Jacobs, and Kasimi papers. This forum thread is an interesting picture of the road we are on. It would be great if we could track down the developer--user name "Frost"--and pick his or her brain. But Frost vanished abruptly from the forum in 2010.
Doing
Trying to run finger6 over input file.
Playing with AFP (one .exe, one .dll, and that's it).
Trying to quantify variability in the Parncutt sight-reading corpus.
Trying to interpret the Synthesia data I have collected and translate to my format--or adopt theirs. It is more terse and presumably more comprehensive. Moreover, they seemed to be flirting with a good way to crowdsource fingering data collection. They claim a user base in the millions.
Implementing basic color tracking synchronized with MIDI events.
Trying to get data from IMU sensor through Arduino and into Minion (my cleverly named Mac Mini).
Fighting with edge detection example with PS3Eye camera. App keeps freezing. I may need to explore pricier camera options.
Need to talk to EE/robotics/CV expert sooner rather than later. You mentioned you had some influence with Milos Zefran? Maybe he or one of his students would like to collaborate?
I am not sure how this will interface with the whichever electronic survey system we adopt. (UIC has one I think we can use for free.) But I think step one is to come up with some way for pianists to tell us their preferred fingerings for predefined repertoire. I am thinking we can do something in Java or JavaScript and HTML5 that can render ABC or LilyPond on the fly. We will also leverage this system as an essential component of the "Semi-Automated" system below, to allow automatic correction of detected fingerings. In my mind, this should all be very straightforward. I am quite sure there is a JavaScript library for ABC that should make this pretty easy to implement. (I was planning to use this library for BowTIE.)
Sketch of Manual Data Collector (MDC)
The display is needed for the Manual Data Collector (MDC) to allow pianists to correct system errors. We could also track these corrections to help us analyze automation errors.
Sketch of Semi-Automated Data Collector (SADC)
We may want to dispense with the pre-defined content and let the pianists just play. Once the automation is sufficiently robust, it really should not matter what they play (only how they play it). Why do I suspect these words will come back to haunt me? Anyway, this will be the most painless way to collect data. Even if we always want to have the screen and people playing from a score we provide and know a priori, eventually we will want to stop asking them to correct things.
Sketch of Fully Automated Data Collector (FADC)
Deployment
MDC as part of needs analysis survey.
Prototype SADC in CS lab environment for testing by Didactyl team. Ivana, I am looking at you. Please don't avert your eyes. Maybe Paul Landes from the NLP Lab could help too.
I just stumbled on a commercial system called Synthesia, which seems to offer an avenue to some annotated corpora. They have MIDIfied a big chunk of Hanon and some Czerny beginner book. But I don't quite understand their file format yet. It does appear to be human-readable, but the fingers are supposed to be numbered 1-5 (for the left hand) and 6-0 (for the right), to align with the digits on a QWERTY keyboard. But I can't seem to line this up with the sample of Czerny files I have downloaded. Still, just having the MIDI will save me some time. It should be pretty quick just to read strings of digits out of Hanon. MIDI + fingers = corpus. Less than ideal, but it is something. I was going to nose around and see if I could find anything else.
There was even an effort started in 2009 by someone named "Frost" to automate the generation of fingering hints for Synthesia. This project, as described in the Synthesia forum (where a binary .NET distribution is made available), combines the approaches described in the Parncutt, Jacobs, and Kasimi papers. Active development seems essentially to have stalled a few years ago, but there are many lessons to glean from this forum thread.
Hsiao, C., Li, R., Yan, X., & Do, E. Y. (2015). Tactile Teacher: Sensing finger Tapping in Piano Playing. In Proceedings of the 9th Conference on Tangible, Embedded and Embodied Interaction (pp. 257–260). Stanford, California, USA.
In the paper, they report 100% accuracy for the thumb in melodic playing with the vibration sensors attached at the wrist. This means no glove would be necessary. This may be the way to go to augment the vision system, for melodic playing at least.
Another idea. What if we positioned a tiny camera on the outside of the heel of each thumb, pointing inward--directly at the other thumb. These two cameras should be able to monitor the thumb activity very closely, and it seems this would be less intrusive than the accelerometer.
So we would have two cameras looking downward at the keys and fingers and two cameras looking inward at the thumbs and globs of (UV?) paint for color markers.
Just a thought. I really need some advice on cameras to prove this concept.
Compare this with the IMU-based picture:
All of which makes me start thinking of scalability and money. Even assuming we can build this with cheap components (PS3Eye cameras or similar), how many can we afford to build?
How much CPU do we need? Can we share CPUs in a piano lab environment? Is that the best environment to collect data?
Could we set up in a practice room or two to which only project participants have access? Do I need to be there whenever data are collected? What about a take-home system for a smaller, more expert set of pianists (with larger repertoires)?
Do we need to control the repertoire and display it on a screen? This is not really necessary to the collection system. We don't really care what the user plays. We can characterize it later from the MIDI stream. We might suggest they play predominantly finger legato single-note lines using both hands. Initially at least, I am thinking of ignoring chords and focusing more on the interplay of melodic interplay between the two hands. This was suggested by someone at DHCS.
Another thought is a three-camera, one-hand tracking system with one camera staring down the length of the keyboard in lieu of the hand-mounted cameras. Come to think of it, having the cameras in motion may be a significant problem.
One last interesting approach is worth mentioning. Check out this video:
And this one:
On the surface, this looks insanely promising for our purposes. (I especially like how he complains about the thumb occluding the other fingers.) I am thinking about contacting David Kim, who is now a post-doc with Microsoft Research in Cambridge, UK.
More details in the following article:
Kim, D., Hilliges, O., Izadi, S., Butler, A., Chen, J., Oikonomidis, I., & Olivier, P. (2012). Digits: Freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In UIST ’12: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (pp. 167–176). Honolulu, Hawaii, USA: Association for Computing Machinery.
Okay, I decided to spend something like 50 bucks and get components that should allow me to build a crude initial prototype. I bought two of the cheapest inertial measurement units (IMUs) I could find, a feast of wires, and what should be an authentic Arduino Uno R3.
IMUs have both gyros and accelerometers onboard. Since I couldn't manufacture any strong reason to prefer one over the other, I figured I might as well cover both bases. It would be nice if we only needed one or the other, so the breakout boards could be smaller.
And why two IMUs? Well, I thought this would double my chances of getting a chip that actually worked. (Did I mention these were cheap?) Also, I think we might want two sensors on each hand--one on the thumb and one on the back of the hand. During periods of loud playing, the whole hand may be accelerating. We will probably want to discount acceleration detected by the thumb when the whole hand is accelerating. We will want to pay more attention to the CV input, especially since--I think--occlusion of the thumb will be less likely in these cases.
The least ambiguous signal to be sure. But pricey and intrusive.
Would the Robotics Lab or EVL have something I could play around with? According to Wikipedia, the EVL developed the very first wired glove back in 1977.
Or we could try to roll our own for similar functionality like this:
Schaechter, J. D., Stokes, C., Connell, B. D., Perdue, K., & Bonmassar, G. (2006). Finger motion sensors for fMRI motor studies. NeuroImage, 31, 1549–1559. doi:10.1016/j.neuroimage.2006.02.029
Maybe just the thumb and no wrist band? Maybe on the thumb nail?
1.0 CV system to capture fingerings from performances on MIDI enabled keyboards (with or without accelerometer on thumb).
2.0 CV application to automate collection of hand measurements. Ideally, pianists could measure themselves in the privacy of their own homes.
2.1 We could also ask them to estimate their hand size with respect to other pianists they know. We could get a good start with a well-constructed survey and a good pool of pianists. Offer chance to win $100 gift card?
3.0 OMR system that extracts fingerings from scanned scores (plus MIDI performance data).
4.0 Web service including our spins of published methods.
5.0 $$$ to pay professional pianists. Should we look for NSF funding opportunity and try some more grant writing? Chris Raphael recently got some EAGER NSF money.
6.0 Utility to take fingering data + abc and produced abc with complete fingering annotations (and PDF output).
7.0 A forum where pianists could be solicited to submit data or a mailing list of good leads.
8.0 IRB approvals:
Web service
Survey a la Parncutt for professionals, students
Data collection in group piano class at UIC
Data collection from Ivana Bukvich's students?
Other students?
List of professionals, collected from Schrader, Parncutt, Bukvich, Laird, etc.
Emailed Steven Jordan and confirmed his membership on committee. Sent him WCP report and link to DHCS abstract. He is available 11:30-2:15 MWF at Loyola Chicago. Plan to meet next Friday.
Emailed Angus Forbes to ask about committee membership. Sent same documents as above.
Data Collection
Rigged mount on 2x4 to point Kinect down at keyboard.
Thought about automated ways to collect hand dimensions a la Wagner. I am thinking CV in lieu of calipers.
Corpora
After consulting with Barbara, I determined it was not necessary to obtain IRB clearance for the Sloboda sight-reading data set.
Emailed Chris Raphael about MIDI input to OMR and extracting fingering data from scanned scores with associated MIDI files. He said it was an interesting idea, but no code to share and seemed to think it would be a tough slog. I asked about Audiveris, and he said I should check it out and report back.
Investigated Audiveris as potential basis for MIDI-infused OMR. Seems like a fork is in order.
WebService
Started looking through the Parncutt source. There seems to be some code relating to physical distances a la Jacobs.
Doing
Comparing Parncutt source to my Python code.
Implementing basic color tracking synchronized with MIDI events.
Trying to obtain evaluation corpus used in (Parncutt et al., 1997) with a mind toward quantifying IAA.
Thinking about ways to quantify similarity between fingerings.
Thinking about ways not to quantify any such thing: maybe we measure the quality of a model by its ability to effect change in a player's preconceived notion of how a piece should be fingered.
Red Flags
Over-extending Brian Ziebart vs. under-representing ML on my committee.
Don't we need a psychologist on the committee? Parncutt? Stellan Ohlsson?
