June 28th - More ESL HIT

I posted the ESL HIT on MTurk. It is not integrated with our full MTurk pipeline, but I decided that that is not relevant for this stage, and rather than wait to hear back from Dmitry, I figured I could post it as a stand-alone-not-as-pretty-and-streamlined HIT. Basically, I am too impatient and want to start having some data to play with... :-) 

I took a brief foray into some cool mechanical turk tools while trying to set this up as an External Question. I thought doing so would save a lot of trouble (like having to copy and paste code into MTurk's childish little text editor) and would be closer to what Dmitry has in store for it in the end, but MTurk didn't like me hosting it out of my Hopkins Ugrad account (which I don't even know if I am still technically supposed to have) since it wasn't secure enough. So alas, having no other ideas for where to host it securely, I defeatedly copied-and-pasted. But it is posted now, so I won't dwell on my inefficient path of getting here (Hess's law? Sunk cost? I'm sure it is justified by one of those little lessons from one of my past courses). 

So you can try it out on the sandbox site (it shows up when you search 'ESL'), or just look at it if you don't feel like signing into MTurk. The sentence data I used for this was not pre-processed at all, so there are a few that contain quotes or odd symbols and cause the entire HIT to appear blank. These should be ignored for now while I process the data to fix this. Others of the sentences are completely fubar, and should also be skipped. I believe that once it is integrated with the full pipeline, these sentences will be weeded out through other HITs, but perhaps I will add a "WTF this isn't even in English" button to the interface just in case.

June 26th - ESL Hit

I am almost done with the ESL HIT and am putting together finishing touches. This is what it looks like currently, still with a few bugs (mostly new ones that were introduced with the new drop-down menu). I'm still adding in the descriptions of the new error annotation types, and need to add some more help icons and explanations. 

My plan is (fingers crossed) to post it for real on the sandbox site this week- I'm ready to start collecting data, and to begin ending this little affair with Javascript development. (It has been very dysfunctional...its a passive aggressive language and I can never tell what it wants from me. Its really just jQuery that has held the relationship together for this long. But I hear Javascript is one of those languages that will inevitably be on-again-off-again for the rest of my programming career. Sigh.)

June 25th - ESL Errors

Per Chris's request, I recolored the error-type hierarchy to reflect the mode of correction that should be used for each error (changing a word, deleting a word, moving a word etc...) The new version is below, and available as a pdf here.

I found a great tutorial for multi-level menus with CSS and jQuery, so I made this cute little javascript drop-down that will allow turkers to follow through the hierarchy in order to choose the right error type for their annotation. Hopefully this will make it more intuitive to use, and it will make it easier for us to measure the deepest level of detail at which we can still get cross-annotator agreement. 

June 16th - ESL Error Types

Now that Dmitry has passed the MTurk baton back to me, I am beginning to work on finalizing the design of our HIT so that we can start collecting data soon. The first step is to incorporate the error types that I found during the lit review so that turkers are able to tell us which kinds of errors they are correcting. I have tried to impose some order onto the laundry list of error types that I had posted before, so below is a hierarchy of error types, which I think describes the errors we are most likely to see. The green ones are the ones I plan to put into the HIT as options for annotating; in most cases, they are the most granular error types, except that I have grouped all article errors together and all preposition errors together since differentiating based on inserts/substitutions/deletions seems unnecessary. A pdf of the image is available here. 

June 13th - General Updates

I'm back in the real world again. Mancester was...an adventure, to say the least. I'm spending the next two weeks at the CLSP summer school, so I'll have lectures and workshops during the day and will be continuing to work on Joshua and MTurk HITs in the evenings. 

The learning curve for Joshua is continuing to grow, and will hopefully be complete in the next few days. (Before I left for Mancester, I forgot to start my script running to finish the rest of the trials. My heart sank when I got back and didn't have piles of data waiting for me...like finding out Christmas was coming late...)

I am fiddling with more video archiving. I archived and uploaded two videos (one on Minimum Bayes Risk Decoding and one on Speaker Recognition) to Vimeo. Now I'm running into problems generating the slides portion of the video. I had had problems with this at the start of the summer- I'm doing a conversion from jpeg to avi in an Ubuntu VM and had run into a race condition that was scrambling the slide order before creating the video. I had thought I'd fixed it my adding a line to the python script to sort the slides, but it appears to be happening again, even with the sort, so I will have to do more trouble shooting. 

Dmitry made some adjustments on the ESL HIT, so I have begun to revisit that project. I have just started looking through the code to reorient myself and get familiar with the differences. I'll hopefully get to make good progress this weekend after figuring out exactly what the next steps are for getting the HIT on MTurk and starting to test it out.

The summer school has been great. We had two fantastic ML lectures this morning, and its making me very impatient for fall and the start of the semester. (Also, Charles Commons is an infinitely better dorm than anywhere I lived while I was actually a student here...figures...)

June 3rd - Leave of Absence

I'll be gone, without computer, from tomorrow (the 4th) until the 11th. Where could I be going where there is no internet? Mancester, Tennessee, of course. Where better to vacation then a temporary shanty town surrounded by unshowered white kids and Walmarts? I'll be back and very sunburned on the 11th with more exciting posts about grammatical errors and BLEU scores. :-)

June 3rd - Lit Review

I compiled a full list of all the types of ESL errors that have come up in my research, including NLP Papers and Linguistics/Education papers. My full write up can be found here, but the list of error types is below for convenience.

• Article Omission: I go to store.    
• Wrong Article:  I want to be the doctor when I grow up.  
• Extra Article:  I like to drink the tea.  
• Prep. Omission:  I go store.  
• Wrong Prep.:  I feel very worried for it. 
• Extra Prep.:  Let's meet at tomorrow.  
• Subject-Verb Agreement: I goes home after school.  
• Verb Tense: Yesterday, I go home after school.  
• Verb Form (verb is not conjugated when it should be, or is conjugated when it should not be): My mom likes when I cleaning my room.  
• Word Form (word is the incorrect part of speech or the incorrect form): I gave the book to he.  
• Style/Wordiness (sentence is clumsy, awkward, does not sound fluent):  Suzie believed, but could not confirm, that Bill had feelings of affection for her.  
• Infinitive Use (an infinitive is used in place of a conjugated verb or a gerund):  I to go running every morning.   
• Gerund Use (a gerund is used instead of a conjugated verb or an infinitive):  Everyone was to dance together.  
• Part of Speech (word is in the incorrect part of speech):  I was very delightful about the good news.  
• Verb-Verb Agreement:  Yesterday I woke up and going downstairs for breakfast.  
• Mass Count: I hope you have many success in life.
• Passive Voice (passive voice is used unnaturally, or with a non-transitive verb):  I was slept for 10 hours last night.  
• Run-on Sentence: I like school and especially I like math class but I do not like when we have a lot of homework.  
• Sentence Fragment:  And then going outside.   
• Punctuation: I like, chocolate 
• Adjective Order: I am next to the brick tall building.  
• Word Order:  Marie returned the bike to the store that she borrowed.
• Wrong Word: Global warming is one of the affects of burning so much gasoline.  
• Unnecessary Transitional Phrase:  I fell off my bike and therefore I scrapped my knee. 
• Incorrect Pluralization:  It was in the Stevensons's yard.  
• Split Infinitive: It is best to quickly run. 
• Repeated Subject/ Redundant Pronoun (pronoun appears unnecessarily when the subject is already clear): My sister she likes to sing in the shower. 
• Incorrect Negation: I didn't swore in from of my mother.  
• Insert Word: I by the car.  
• Delete Word: I wasn't not able to go with them. 

This list of error types presents the following complications:

• Of the error types presented in the list, many cover the same domain of errors. For example, any Verb-Verb agreement error or Subject-Verb agreement error could also be tagged and corrected as a Verb Tense error. 
• Many error types are context-dependent and difficult to tag and correct in isolation without further information about what the speaker intended to say. For example, the sentence "My mom likes when I cleaning my room" could be tagged as Verb Form and changed to "My mom likes when I clean my room," or more generally as Missing Word and corrected to say "My mom likes when I am cleaning my room."
• Several error types- for example, Sentence Fragments and Wordy Sentences- may require significant rewriting rather than single word insertions, deletions, or substitutions. This could create complicated morphologies and a very low probability or redundancies. Allowing more open-ended types of edits on MTurk might get overused, since it is usually easier to write a new sentence from scratch than to decompose the desired changes into multiple discrete edits.

June 3rd - Joshua

With Matt's help, I have finally been having success with Joshua. I am currently running the pipeline on the Urdu language data in order to generate a learning curve. I now have BLEU scores for Joshua trained on 5%, 10%, 15%, and 20% of the data, with more to come. Here is the lovely Google Plot of the progress thus far.