Nicole Vogt » Instructor Interface, Data Analysis for an Online Tutor
I am the developer of an adaptive online tutor for general astronomy, one designed to help students master both astronomy concepts and related mathematical skills. My initial motivation for constructing this tool was to increase access to physical science laboratory coursework for students in the Southwest Border Region who could not attend on-campus courses during business hours. In conjunction with a suite of other GEAS educational resources, the tutor is now also used to extend access to astronomy courses across North America by supporting instructors working at a variety of high school- and college-level institutions. I provide individual study accounts for students and cohorts of students working with instructors, to give them access to the types of educational resources that modern astronomy can offer.
Instructors can guide cohorts of students through a semester's worth of material, while having access to detailed information about their progress both as a group and as individuals. A web-based interface provides summary information on the cohort and each individual student in table- and figure-based formats, including recreations of every quiz taken. Instructors can track the rate and accuracy of homework (practice questions) as well as more formal weekly quizzes, and easily access the most-missed types of questions for a student or for the entire cohort.
 |
| The number of self-review quizzes taken per student by non-science undergraduates in a general astronomy course over the course of a semester, separated by gender. We observe a sharp upturn in self-review work before the midterm and final exams, strongest for women. Students were fully engaged in the library by the time of the final exam, having perceived its utility in the aftermath of the midterm exam. |
As shown above, students made extensive use of the library, particularly as a study resource for the midterm and final exams, with their usage increasing with time. Female students were early and strong adapters, which is unusual and encouraging for a new-technology resource.
The next figure shows the average time spent per week studying with the self-review library versus graded quiz scores, for students who did well (left), passably (middle) and poorly (right panel) on weekly graded quizzes. Some students spent more than ten hours reviewing during certain weeks, and described how welcome it was to be able to do so anonymously (free from exposure to potentially negative reactions to their study habits from peers). To encourage weaker students to complete more review problems each week, homework grades were linked to the quality and the amount of self-review conducted, and the average amount of time spent studying by the two weaker group increased to match or exceed that of the strongest cohort.
 |
| Time spent studying with the self-review library per week, for students doing well (left), passably (middle) and poorly (right panel) on weekly graded quizzes. In previous semesters the weakest students spent significantly less time than their peers in study. By linking the quality and the amount of self-review conducted to homework grades, this group was encouraged to increase their studies. The number of students in the weakest group also decreased as students who studied more then performed better on their weekly graded quizzes. |
The following figure shows the distribution of self-review library usage throughout the week for men and women. Individual students seemed to select certain times of day to study more than certain days. They speak of fitting in quizzes during breaks between classes, when their children are sleeping, during free hours, and on the weekends. System updates had to be done between 3 am and 6 am because these were the only times the system was not in use!
 | |
| The pattern of self-review library usage throughout the week for a 60-student course over a semester. The red line shows the distribution of 13,632 quizzes completed by 30 female students, and the blue line shows that distribution of 6,881 quizzes completed by 30 male students. The cohort completed 102,565 self-review questions in total, for an average of 1,709 questions per student. Weekly quizzes were due each week by Saturday night, so many students studied quite a bit on Saturdays (and Sundays were typically a day of rest). The self-review library offers students immediate, detailed feedback outside of business hours and the flexibility of being able to study at all hours of the day or night. | |
The following figure contrasts the weekly studying pattern for three representative distance education students over a semester, averaging 13 weeks of work together. All students improved their performance over time; the level of improvement correlated well with self-assessed study habits and overall course grades.
 | |
|
The green line for Student A shows the success rate of the top student in the cohort. She routinely completed 100+ questions per week (students were incentivized to complete at least 100 questions each week), beginning with scores above 80% and ascending to well above 90%. (The line tails are dashed where fewer than five weeks of work were averaged.) The tan line (Student B) tracks a student who characterized herself as "hard-working but afraid of math" for whom English was a second language. When she began a week with low scores, she completed more questions (and took more time per question) than average. She consistently raised her average score the most, to well above 80% (the target for success). The red line (Student C) shows the work of a student with a full-time job and a two hour-long daily commute who critiqued herself for "not spending enough time studying, and usually not being well-rested when studying". | |
The final figure shows the percentage difference in scores between male and female students on exam problems sorted according to the level of visualization required. Problems ranged from those with no visualization aspect at all to predictions of the location and appearance of the Moon in the sky at various times (level 4) and phase and transit observations of neighboring planets (level 5). A strong gendered effect is observed for problems which require significant amounts of three-dimensional visualization. However, focused intervention in the form of detailed animations showcasing multiple points of view, tools for visualization (simple sextants), and guided tutorials reduced this effect considerably. All students improved their performance when given access to such resources, and the larger improvement shown by female students greatly reduced the significance of the gendered effect.
 | |
| The relationship between gender and the level of visualization skills for exam problems. We plot the level of visualization on the x-axis, where a value of zero means that no visualization is used to solve problems and values of four and five means that problems require very significant amounts of visualization, and against the difference in percentage scores between male and female students. Points below the dotted line represent problems where female students outscored males, and points above the line those where male students performed more strongly. The black points show a clear trend with gender, with male students (in agreement with the literature) performing more strongly on problems which require the use of three-dimensional visualization skills. The three colored points with visualization levels of four represent three different cohorts of students who were exposed to no, moderate, and major levels of intervention to aid them in visualizing the movements of the Moon around the Earth. (The black point is simply the average between the red and green points.) Intervention on a semester-long timescale proved useful in dramatically decreasing the gender disparity in success with this material. | |
This material is based upon work supported by the National Science Foundation (NSF) under Grant No. AST-0349155 and the National Aeronautics and Space Administration (NASA) under Grant No. NNX09AV36G. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF or NASA.