I gave a talk at the Graduate Student Symposium at Odum School of Ecology last week. Each talk was judged by several faculties and students. Each judge has a judging form, consisting of 13 scoring items. Each item has a score from 1 to 5. The maximum score is 65. The 13 scoring items are grouped into three categories: content, presentation and delivery. Each judge will identify himself/herself as faculty, post doc or student. I got 9 judging forms back, 4 from faculty, 1 from post doc and 4 from student. I noticed some differences between how faculties and students judge the talk. Here are some more exploration.
What are the differences and similarities?
These feedbacks promote me to think about what makes a good presentation. More specifically, are elements of a good talk more diverse and specific to one’s own preference, or more objective and universal? There are certainly quite a few similarities in the weakness and strength identified by all judges. For example, many pointed out that my bullet points are a little wordy, texts in a few figures are too small to see, and I was talking to the slides too much. Most judges agreed that I introduced the background well, was enthusiastic, and handled the questions well. But it appears that faculties and students have different views on a few scoring items. That promotes me to analyze the data in a little more detail and here are what I find.
The total score differs significantly between student to faculty judges (p=0.044 from t-test assuming equal variance). Scores given by faculties are significantly higher than that given by fellow graduate students. Since I only have one post doc judge, I group her to the faculty category in data analysis. I used a t-test assuming equal variance because the difference in variance is insignificant based on a F-test for equal variance (F4,2=0.67, p=0.66).
As mentioned above, the judging form consists of three categories: content (25 points), presentation (20 points) and delivery (20 points). Content include whether speaker provides a solid conceptual foundation, a clearly stated objective/hypothesis, proper experimental design, logical interpretation and question handling. Presentation is mostly about the design of slides, such as whether the figures and text are legible, contributes visually, appropriate number of slides and whether speaker elaborate the visuals well. The delivery part is mostly about whether the speaker present in a logical fashion, show enthusiasm and engage with audience.
Faculty and students score the content part similarly. The differences in total score are mostly attributable to the differences in how faculties and students view presentation (p=0.0043) and delivery (p=0.032) differently. What is also evident is that faculty has more consistent score shown as smaller variance. In all categories, variation in scores among student judges are larger than that from the faculties.
If we examine the differences between faculty and student score for each item, three items show up as significant: 1) Visual contribute rather than distract (p=0.018); 2) Number of slides is appropriate (p=0.00074); 3) Speaker elaborates on information provide in visuals (p=0.0006).
What are the possible interpretations?
Clearly, this analysis is based on a very limited sample size. Thus, take a grain of salt when looking at the result. It is not likely to be that generalizable. But we may still gain some insights into where and why similarity and differences occur.
Faculty and students judges share similar opinions on the content. Given that items in content mostly cover the proper structure of a talk, it suggests that people mostly agree on what a good talk should contain, i.e. there should be clear background, motivation, goal/hypothesis, results and interpretation. But faculty and students differ quite a bit in what they view as effective visual and delivery. Some of the differences are quite unexpected. For example, students and faculty judges differs dramatically in whether they think the number of slides is appropriate. They also differs in whether they think I elaborated the figures. I would not have predicted such difference, as I thought these are the more “objective” things.
Surprising results often warrant some extra thoughts. So I will share a few speculations on why this difference occurs.
Perhaps there is a difference in the level of knowledge/experience between faculties and students. Faculties may have more experiences and more exposure to researches, and can follow things more quickly. Thus they may think that the number of slides and degree of elaboration are appropriate while students get lost. It would be interesting to have many judges with different amount of experiences. Having a gradient of experience and see how that influence the judging result may help us test this “experience hypothesis”.
Or perhaps faculties are a group of people different than students. What I mean here is that faculties are a particular group of people who lead labs, write grants and focus on research. They were trained in a more or less similar way. What they look for in a presentation may be different than students, who may have a variety of career choice and a variety of focus in course training. This is consistent with the fact that scores from faculties are much less variable. I personally wants to pursue a academic career in the futures. Perhaps the way I present shares more similarities with the view of faculties. It would be interesting to analyze the score from fellow graduate students who pursue other careers. The comparison may give us a little more insights on whether this “similarity hypothesis” is valid or not.
Or perhaps we graduate students judge peers more strictly than our professors. This is not something I would expect. But maybe we as students are more focused on our own research. When we are so into one thing, we may value it more than other things. We spent a lot of time on our own project and probably love our own research area a lot more than others. If we have a lot of judges, some share similar research interest as the speaker and some don’t, we may evaluate this “specialization hypothesis”
What lessons do I learn from this?
I don’t have an answer to this question as I don’t really know why the difference between faculties and students occurs. One general and obvious lesson I learnt is to tailor the talk, in both contents and fashion of delivery, to the audience. But what is not obvious to me is how we tailor the talk to a particular audience. That requires us to know what the audience is looking for. Understanding why there are differences in how people view a talk is quite valuable. I am puzzled and I would like to hear what you think.