User Evaluation of Technology

What is the course about?

This course presents a wide variety of methodologies for evaluating technological interfaces with users. It is aimed at providing engineering students with the necessary knowledge on working with users, statistical techniques for analyzing the information gathered from the users, and how to interpret the results from the user studies.

Logistics

When and where?

J Baskin Engr 169, Tuesdays and Thursdays 10:00-11:45AM

Who to contact?

Sri Kurniawan: Instructor

Assessments

  • Take home exam: 30%.

This exam requires you to perform data snooping on the quantitative and qualitative data from a survey that we conducted. The survey and the csv file are in the class material page, password protected. Just like in real life, the data is messy, some people probably filled in the survey more than once (there was a draw for two $100 Amazon gift cards for those who completed the survey), and we probably did not phrase the questions appropriately. The deliverable is a paper (no page limit, please use the ACM template) that has the following sections:

  1. Introduction: why you chose to investigate this particular case (e.g., why are you comparing data from older vs. younger people, or men vs. women, etc?).
  2. Data analysis: quantitative and qualitative. As discussed, let's assume any numerical (or those that can be converted to numerical) are scale and only need parametric statistics. I also need you to do content analysis (you can choose either conceptual analysis or relational analysis) on a subset of the data (coding all 300 will take a while). For the quantitative analysis, I also need to know which data you discarded and why (not row by row, explain at a high level, e.g., you threw away all participants above 65 years old of age because there were only 2 of them). For the qualitative analysis, explain the sampling strategy, e.g., every 5 rows. 
  3. Discussion and conclusion. For discussion, I expect to see an analysis of your findings, relating qualitative and quantitative data (e.g., women on prepaid scheme spent significantly less than men on similar scheme because they fear for brain cancer (70% women in this sample stated this fear compared to 40% men).
  • One homework: 10%. Please note that in general I do not accept late homework. Any late submission will require evidence of a mitigating circumstance. The deliverables are:
    1. IRB protocol form (the full review, not the exemption)
    2. Test plan (see lecture notes from the first day of class)
    3. Recruitment material
    4. Survey, interview or observation guide
    5. Consent form. 
  • Student-led class presentation: 10%. In relevant weeks, students will be asked to read some chapters from the textbook and when appropriate some additional papers. The students may be asked to synthesize the chapters and papers; when appropriate collect/analyze some data as a proof of concept of the topic; and present the synthesis/data analysis in class. This presentation should be presented in lecture format, around 30 minutes long, followed by 15 minutes of class discussion. Please send me the ppt or pdf file the day before the presentation so that I can upload it to the class website.
  • Class discussion participation: 10%. In almost every week, I will assign paper readings that are relevant to the topics presented that week, and will randomly ask a student to lead the discussion. The lead will summarize the paper and critique it, and the rest of the class then can start the discussion/debate - you have 20 minutes for this exercise, it is up to you how you are going to use this time slot. The lead will not be the student presenting the chapters of that week. Of course, most likely you will need to read the relevant chapters to understand the papers. In addition, in every class meeting, we will discuss one evaluation method, again I will point to the readings relevant to that method, and another student must present the summary of the method and the discussion related to this method.
  • Individual project: 40%.
    For this project I need you to recruit an appropriate number of users to evaluate a technological interface of your own choosing. You need to collect both qualitative and quantitative data, analyze the data, interpret them and provide conclusions in terms of improvements to the technological interface to make it more user-friendly. 10% of this grade is for the presentations during the final exam slot, and 30% is for the report. 
    As a reference point, this is the sort of data gathering and analysis I am expecting you to do in your project: A paper on Older people and mobile phones: A multi-method investigation. The deliverables for the final exam are:
    1. A report in the format resembling the example paper above.
    2. 10 minutes presentation covering: motivation (why did you choose that particular interface), methodology (participants, procedures, measures), results, discussion and conclusion.
    3. 5 minutes Q&A.

Policies and accommodation

  • All students enrolling in this class are advised that Academic Integrity will be strictly enforced. Academic misconduct includes but is not limited to cheating, fabrication, plagiarism, or facilitating academic dishonesty or as further specified in campus regulations.
  • If you qualify for classroom accommodations because of a disability, please submit your Accommodation Authorization from the Disability Resource Center (DRC) to me in a timely manner, preferably within the first two weeks of the quarter. Contact DRC at 459-2089 V, 459-4806 TTY.

 

Readings/resources

  1. Textbook: Graziano, A. M., & Raulin, M. L. (2006). Research methods: A process of inquiry (you can get this used for $1-4 on Amazon, get the various editions, they do not differ much).
  2. Textbook: Lazar, J., Feng, J. & Hochheiser, H. (2010). Research Methods in HCI (around $30 on Amazon used).
  3. Cairns, P. & Cox, A.L. (2008). Research Methods for Human-Computer Interaction
  4. Usability Evaluation Methods
  5. Emerging research methods for understanding mobile technology use. In Proceedings of OZCHI, pp. 1-10.
  6. Various papers assigned in class.
  7. IRB.

Instructors and Assistants