Posted: Wednesday 7th June 2017
Author: Claudia Cox
That's it! The main examinations period is over, and we have successfully conducted 18 digital examinations. Our students are no doubt breathing a sigh of relief, and the examiners are finishing marking 1519 submissions - made somewhat easier by WISEflow's integrated assessor tools and the fact that all of the "papers" are stored online. Now that we're no longer focusing all of our attention on running the exams, it would be an excellent time to talk in depth about how things went:
The Blackboard Learn Resource
A Blackboard organisation called "Brunel Digital Exams" was released to all Computer Science students and staff on 10th April 2017, just before the Easter break. The organisation features:
- A downloadable .pdf student user guide for WISEflow
- A "Laptop Healthcheck" section with basic advice on how to care for and maintain a device, as well as contact information for the University's Tech Zone which offers further technical support
- A forum for students to post any questions or discussions in
- An interactive resource with advice on what to do before, during, and after the exams - including a short quiz
- An FAQ section for quick reading
Between the day of launch and the last access on 6th June 2017, the introductory page had 1267 hits and the other pages of the organisation had 46 hits on average. A big factor in this was likely the timing. The Easter break began on 12th April 2017, just 2 days after the organisation was made live. Students and staff were probably making sure any final errands were complete before the University closure, and could have briefly checked Blackboard, seen it wasn't an urgent matter, and enjoyed the break - forgetting all about it.
Another potential issue is that due to the time constraints we were under. Research suggests that a significant portion of people prefer to receive information through audio/ visual means as opposed to reading text, so tutorial videos on how to use the exam software were in the process of being scripted but couldn't be finalised in time to meet the release deadline. Now that the exams are finished we'll resume work on these.
We are anticipating that there might be a higher rate of engagement once the resources are rolled out to more of our students, and once we've finished adding all of the features we'd like to see in there.
The Practice Sessions
As mentioned in our post on pre-exam preparations, a week of drop-in practice sessions were organised for students to help familiarise them with taking the exams on their laptops. Each practice session had two opportunities to try out exams which ran from 10:00 - 10:30 and 14:30 - 15:00. Each session was aimed at a specific level from first year Undergraduates through to Postgraduates, though students were encouraged to attend any session that they could. All students taking digital exams (approximately 620) were invited, and about 38.8% of those attended a session and took part in a practice exam. The content of the practice exam was not related to their course in any way, and instead focused on encouraging them to explore the exam software's interface:
This is an example of the case study or assignment you’ll be provided with which forms the basis for your long essay exam questions, along with any instructions. The real thing will probably not be this short.
NOTE: If your laptop does not have a built-in webcam, you do not need to answer question 3.
For this practice session:
1) Explain whether you believe that the digitisation of exams is overall beneficial or not.
2) Use the drawing tool in the FLOWlock exam interface to draw something that you associate with Computer Science and add this to your appendix material.
3) Make another drawing on the paper provided and use the webcam function to take a photo of this and add it to your appendix material.
Once you have completed the tasks above, you may hand in your submission.
After analysing the 214 submissions, we found that 45.7% of students indicated a clear overall positive outlook and agreed with the belief that digital exams are beneficial:
11.2% responded that they didn't think digitisation was a good thing, while 8.4% provided a balanced viewpoint which sensibly weighed up the pros and cons of digital exams but ultimately did not express a definitive good or bad opinion. The remaining 34.5% that didn't answer the question largely made use of the time to type up revision notes in WISEflow's exam environment and test out the text editor. Regardless of the responses, the students that attended seemed to have a lot of fun getting a bit of practice in and making use of the drawing tool - with some rather creative results:
As you can see from the submissions for task 2 above, "Computer Science" can mean a lot of different things for different people. One of the really interesting points that emerged from the practice session data was that each group of students seemed to have quite different priorities or expectations for their exams. We'll sum up some of the most frequent comments below:
First year Undergraduates
Approx. 220 unique students
49 total submissions
Morning session: 18 submissions
Afternoon session: 31 submissions
Total engagement rate: 22.2%
7 students clearly indicated that they did not think digital exams were beneficial. The main concern was that technical issues such as battery life or machine failure may cause them to lose their exam script or be disadvantaged in some way.
14 students clearly indicated a positive or overall positive outlook and agreed that there were more benefits to digital exams. The most common positive comment was that many students believed digitising exams would optimise the marking process, potentially to the point of automated marking or near-instant grades.
Other criticisms/ concerns included:
- Security (i.e. the exam software may be able to be hacked or bypassed)
- Having to adapt to a digital format when individuals are used to handwriting
- Questions involving drawing/ diagrams may be harder to answer
Other noted benefits included:
- The ability to type an answer much faster than they could handwrite it
- BYOD - the psychological reassurance of being able to take an exam on a familiar device
- The ability to edit work neatly
Second Year Undergraduates
Approx. 208 unique students
95 total submissions
Morning session: 40 submissions
Afternoon session: 55 submissions
Total engagement rate: 45.6%
10 students clearly indicated that they did not think digital exams were beneficial. The main concerns were that digital exams inhibit the ability to think creatively and that technical issues with software/ hardware may have a negative impact on the exam.
39 students clearly indicated a positive or overall positive outlook and agreed that there were more benefits to digital exams. The most common positive comment was the belief that a digital exam would allow students to provide a clear answer to examiners, meaning that they would not be disadvantaged by poor handwriting or illegible editing.
Other criticisms/ concerns included:
- More disruption (loud typing, mechanical keyboards, loud fans, moving to plug into power)
- Implementation being so close to exams (i.e. wanting more time to practice using the software)
- Health concerns (i.e. eye strain and headaches from continued screen time)
Other noted benefits included:
- The ability to type an answer much faster than they could handwrite it
- The prospect of the ability to take exams at home in the future
- Reduction in cost/ logistical efforts for students and examiners
Final year Undergraduates
Approx. 148 unique students
76 total submissions
Morning session: 22 submissions
Afternoon session: 54 submissions
Total engagement rate: 51.3%
5 students clearly indicated that they did not think digital exams were beneficial. The main concern was that digital exams are a drastic change from pen and paper, and this was an additional thing to worry about during the final high stakes exams.
37 students clearly indicated a positive or overall positive outlook and agreed that there were more benefits to digital exams. The most common positive comment was that many students believed a digital exam would allow them to write their answers much faster.
Other criticisms/ concerns included:
- Finding the drawing tool slow/ time consuming to use
- Needing a grammar checker along will the spell check function
- The belief that the individual is a more efficient hand writer than typist
Other noted benefits included:
- BYOD - the appeal of using the individual’s own device to sit exams
- The prospect of the ability to take exams at home in the future
- Digital exams complement existing methods of study such as writing coursework, revision, etc.
Postgraduates
Approx. 43 unique students
21 total submissions
Morning session: 8 submissions
Afternoon session: 13 submissions
Total engagement rate: 48.8%*
2 students clearly indicated that they did not think digital exams were beneficial. The main concern was that individuals were more accustomed to pen and paper and have difficulty with typing quickly and the exam software's interface.
8 students clearly indicated a positive or overall positive outlook and agreed that there were more benefits to digital exams. The most common positive comment was that digitisation would mean ease of marking for the examiners.
It should be noted that one of the students that initially indicated a negative perception of digital exams remained for the afternoon practice exam and uploaded a second submission, expressing that it was much easier after some practice.
Other criticisms/ concerns included:
- Battery life of an individual’s device
- Implementation being so close to exams (i.e. wanting more time to practice using the software)
- Formatting answers is time consuming
Other noted benefits included:
- The ability to type an answer much faster than they could handwrite it
- Preventing cramps associated with handwriting
- The ability to edit work neatly
*the attendance rate for this session is a bit skewed because all students were contacted and reminded that this was the final practice session available before exams started. This session had the highest attendance of students from other levels of study (i.e. Undergraduates), and the number of Postgraduates in attendance is likely somewhat lower.
Bearing in mind that the sessions were optional we've been pleasantly surprised by the turnout for them.
Students Without Laptops
Students who didn't have a suitable laptop were advised to email and notify us so that we could make alternative arrangements for them. These arrangements included seating the students in an alternative exam venue with pre-prepared desktop PCs, or allocating them a device from a small reserve of emergency laptops available at the main exam venue. In total, 98 Computer Science students were not able to use their own laptops during the examination period, which translates to about 15.4% of the entire cohort.
A more detailed analysis of why these students did not have suitable devices of their own will be available in our third and final post-mortem instalment.
We're very aware that this entry is fairly enormous now, so in part 2, we'll be discussing our findings about performance during the exam!