As engineering academics work through this challenging year resulting from the continuing worldwide novel coronavirus pandemic, the Engineers Ireland Academic Society hosted its second webinar on October 28, 2020 on the topic 'Remote assessment: Creating, correcting and preventing collusion'. It was offered free of charge to members and non-members of Engineers Ireland.

A recording is available to members of Engineering Ireland on the Engineers Ireland YouTube channel:

The presentation slides are also available upon request by emailing sectorsupport@engineersireland.ie using the subject line 'Academic society second webinar – request for presentation slides'.

The webinar was chaired by Dr Aimee Byrne, lecturer in the School of Civil and Structural Engineering at TU Dublin, who welcomed 137 delegates at its peak, from all over Ireland and from overseas – in countries such as Switzerland, Australia, Canada, Kenya and the US.

Úna Parsons, CEng, FIEI, head of Faculty of Engineering & Design at IT Sligo and chair of the Academic Society, subsequently introduced each speaker in turn.

Ideas from key practitioners

The focus of the webinar was to showcase ideas from key practitioners on how to move forward in the context of remote assessment.

The findings from a survey conducted at the end of August received responses across 19 higher educational institutions, representing 16 engineering disciplines in Ireland and reporting the impact lockdown had on assessment processes at NFQI Levels 6, 7, 8 and 9. Survey responses in the context of assessment were presented by Irene Hayden, CEng MIEI at the webinar.

Before COVID-19 restrictions, survey respondents reported a 50% use of mixed-mode methods of assessment, that is, a combination of continuous assessment (CA) and terminal exams.

This was followed by 29% reporting the use of 100% CA only. The remainder reported the additional inclusion of multiple choice questions (MCQs), project work and research projects.

The modes of assessment reported pre-March 2020, are illustrated in Figure 1 below, and during the first lockdown in 2020 varied, as illustrated in Figure 2 below.

Those who used online proctored examination solutions used platforms such as Menti and MyForms used with MS Teams. However, professional proctoring organisations such as Examate were reported as not being available due to the global pandemic.

Figure 1: The modes of assessment reported before COVID-19 restrictions

Figure 2: The modes of assessment used instead of a terminal examination in an exam hall reported during the first lockdown in 2020

The method of assessment found most useful by 80% of survey respondents in assessing students remotely in the 2019-2020 academic year was exam-based assessments.

However, in planning for the next academic year, 2020-2021, survey respondents reported a significant change in planned assessment practices, with 42% now reporting a 100% continuous assessment strategy, closely followed by 32% reporting the planned use of mixed-mode assessments.

This reflects a noted change in engineering education pedagogy among survey respondents. The impact of this change will be interesting to see going forward, as it is arguable that this could alter the outlook of the next generation of engineers if the impact is significant.

The second speaker was Louise Lynch, an assistant lecturer in civil and structural engineering at TU Dublin and a member of the LEAF teaching fellowship (Further information on LEAF can be found by clicking here, and in the video).

Lynch delivered an overview of remote assessment options she incorporates into her engineering teaching practice including MCQs and video assessments.

In order to reduce collusion and plagiarism, she developed multiple versions of MCQs using Brightspace, with hand-written answers subsequently submitted to produce evidence of completed calculations.

Lynch reported using Numbas SCORM files incorporated into the VLE or Learning Management System (LMS), which facilitated complex mathematical questions online (Figure 3) and enabling different variations of the MCQs for each student. Numbas is an open source e-assessment system developed by Newcastle University.

Figure 3: Numbas used for structural analysis assessment

When using video assessments, Lynch reported that final-year structural analysis students were set a task to agree calculations in a group and subsequently record themselves on video explaining how they solved an exam paper question. She said that it provided a deeper understanding of how to carry out the procedure while developing teamwork, communication and presentation skills.

The third speaker, Dr Stephen Burnley, a senior lecturer from the School of Engineering and Innovation in the Open University (OU) in the UK, spoke about 'Industrial scale' remote assessment for class groups of up to 1,100 students in first-year engineering.

Each student is guided by a tutor who will typically look after 20 students per module. Each 30-credit module (Representing 300 teaching and learning hours) is assessed with two to three summative coursework assignments, formative computer-based assessments, a summative end of module assessment such as a mini-project or another synoptic assessment or else an unseen examination.

Even before Covid, the OU had seen a general move away from unseen examinations. Since spring 2020, all unseen exams have been replaced with time-limited assignments. All scripts are cross checked against Turnitin and Copycatch plagiarism checkers.

Engineering students in the OU have transitioned to temporary arrangements this year as a result of the coronavirus pandemic. The two-week residential schools were paused during 2020, with a plan to reintroduce them in 2022.

In the meantime, in 2021, students will be sent a 'Home Experiment Kit' with reports submitted through the assignment system. OU reports to also have useful resources within the Open STEM Laboratory, which can be accessed here.

Dr Burnley's conclusion was to note that remote assessment can provide an efficient means of assessing learning and of providing teaching and can incorporate traceable audit trails and quality assurance processes, but that reliable information technology systems are essential.

However, remote assessment is not a cheap alternative to conventional assessment methodologies as there is a high set-up cost which could mean that it might be better suited to large student numbers and be set up as a permanent system. To find out more information about the OU, click here.

The penultimate speaker for the afternoon was Dr Paul Young, who gave a presentation on the Online Examination of Open Problems for Engineering Mechanics.

Associate professor Dr Young is from the School of Mechanical and Manufacturing Engineering in DCU and uses Moodle extensively for assessments. For more information on Moodle, click here. 

Dr Young discussed the concept of a loop quiz on Moodle. A loop quiz has a container activity into which one can import a set of questions. The set-up of the quiz is completely independent of the questions.

Questions can be of different types, e.g. word-based, such as fill in the blanks, image-based using drag and drop, numerical, such as values with tolerance bands, mathematical type questions and essays.

Using a loop quiz on Moodle can be set up to automatically allocate summative marks and formative feedback. This can also be used as an online tutorial set-up, depending on the intention behind the quiz design. Grading of the assessment is per question and can be weighted to reflect the overall quiz learning outcomes. An example is illustrated in Figure 3.

Figure 3: Example of loop quiz on Moodle

The final speaker of the afternoon was Brian Coll, a lecturer in the Faculty of Engineering & Design at IT Sligo, who spoke about How students cheat –designing your assessments to reduce plagiarism, collusion and copying.

Coll won the National Teaching Expert award for his work in this area. He presented the results of a survey that found that as much as two-thirds of students surveyed cheated on an assignment and half of the students surveyed reported that they would be willing to purchase an assignment (McCabe, 2016).

Technology is providing inadvertent routes for cheating. For example, CourseHero with 10 million members and $100 million in annual revenue, provides access to course notes, exam questions and sample solutions uploaded by students (and professors).

Academic writing services are available on the internet for $2-$5 per page, with Kenya being a popular location to source this service. The PhotoMath app, with more than 100 million downloads to data is another example, and claims to have more than one billion monthly maths problems solved and explained for a fee.

There are several strategies to minimise the occurrences of plagiarism both when assessing online or in the traditional exam setting environment.  These include:

  • Online proctored exams, using organisations such as Proctor 360, (Figure 4);
  • Plagiarism detection software such as Turnitin, Urkund and Copycatch;
  • Setting clear guidelines for written assignments, such as requesting a separate table in the assignment which identifies where the student sourced their references, (Figure 5);
  • Setting individualised assignments and asking students to explain their choices;
  • Individualised quizzes on Moodle.

In using Moodle Quizzes it is advised to have a large question bank, to randomise the questions, to limit the use of MCQs as they are easily defeated, to set a time limit on quizzes, e.g. one hour, to open on a fixed day and time and to use a sequential format to present the quiz questions.

For numerical questions, use randomised variables, such as Wiris quizzes, which is an excellent quiz question generator. It is also recommended to use text-based questions to disguise the underlying equations required for numerical calculations online.

Figure 4: Online proctored exams

Figure 5: Table of references added as an appendix 

Further resources

Book detailing background and philosophy on the Computer Aided Assessment of Mathematics, Chris Sangwin, OUP, 2013 http://ukcatalogue.oup.com/product/9780199660353.do 

Chairs

Speakers

The Engineers Ireland Academic Society is a network of academic engineers and engineers in industry with an interest in teaching and learning. Our mission is to promote the professional development of academic engineers and the advancement of academic standards in engineering. Members share ideas and experiences on career development, the latest teaching and learning methods and the recognition of best practice. 

Feedback on the webinar and suggestions for future Engineers Ireland Academic Society webinars can be sent to sectorsupport@engineersireland.ie (please enter 'Academic Society' in the subject line).

Future webinars include:

  • 10am-11.30pm, February 18 2021: The Sustainable Development Goals (SDGs) on our campuses: Practice in teaching, research and operations
  • Registration Link: https://zoom.us/webinar/register/WN_3Or10AptRvyjjA1RH-ZRuA
  • Call for graphical abstracts – A lot of activity is ongoing to introduce the SDGs into engineering programmes across Ireland. We encourage academics to showcase their work and submit a graphical abstract showing how the rollout of the SDGs is embedded in what they do. Accepted submissions will be distributed to webinar participants and made available on the Engineers Ireland website. A template is available to guide your work. Prizes will be awarded to the best graphical abstracts submitted. Graphical abstracts must be submitted by Friday, January 29, 2021. Address all questions to: Kevin.Delaney@tudublin.ie  

Members of Engineers Ireland can join the Academic Society by logging into the Engineers Ireland Member Dashboard (here) and clicking 'My Preferences'. You can also record your CPD and you can browse Engineers Ireland's Events Calendar here.

Authors: Irene Hayden, BSc (Arch), MCIAT, BE, MSc, CEng MIEI, PG Dip, lecturer in the School of Engineering, Department of Building and Civil Engineering, Galway-Mayo Institute of Technology and PRO of the Engineers Ireland Academic Society. Úna Parsons, BSc, MSc, CEngFIEI, head of Faculty of Engineering & Design, Institute of Technology, Sligo and chairperson of Engineers Ireland Academic Society. Anne Morrissey, BE, MEngSc, PhD, associate professor in the School of Mechanical and Manufacturing Engineering at Dublin City University and committee member of the Engineers Ireland Academic Society.