Using flight simulator technology to safely operate machinery could take off in the COVID-19 era, writes former Engineers Ireland president Chris Horn.

Ten years ago, I had the privilege of serving as president of Engineers Ireland. During my 12-month term of office, I had an active schedule, meeting members across our regional network and representing the profession with partner organisations both at home and in the UK. I also received multiple invitations to interesting events and sites.

I recall a particularly impressive demonstration of a 360-degree simulated ship bridge at the National Maritime College in Cork, used for professional training of ship’s officers and pilots.

The bridge is full size, with standard equipment. The bridge windows are replaced with large digital screens that can give a panoramic view of myriad ports across the world.

Different weather conditions can also be simulated. Although the room is completely static, I definitely had 'sea legs' at the end of the demonstration, with my body quite convinced that we had been gently rolling in time to the undulating horizon projected on the screens.

A few years previously, I had the opportunity of 'flying' a simulator at Boeing in Seattle. Strapped into the co-pilot’s seat in a full-size B777 cockpit with digitally projected scenery – in a pod on hydraulic legs which moved the cockpit in three dimensions – was surreally authentic.

For hobbyists, the quality and realism of flight simulation took a major advance last August when Microsoft Flight Simulator was substantially updated; at 37 years old, it is the oldest Microsoft product.

The new release has delighted enthusiasts and, with a highly detailed and largely accurate scenery of the entire planet, gamers have explored the entire digital world.

To the community’s delight, some have discovered occasional data glitches in the scenery such as an airport buried deep in a bizarre chasm in Brazil, and a non-existent 212-storey super skyscraper in downtown Melbourne.

Joystick

Putting aside such curious digital edifices, a flight simulator on a home computer, even when complemented by a joystick and foot pedals, cannot compare to a full-size capsule used for professional flight training.

A full and faithful reproduction of a cockpit, or actually walking around a ship’s simulated bridge, with every switch and control guiding the tactile response of the machine, are difficult to reproduce in a hobbyist setting.

A number of machinery companies and startups are developing teleoperated equipment. At its most basic, an operator sits in an office in front of a standard desktop computer equipped with a hobbyist steering wheel, foot pedals and selector block, and controls a real-life heavy machine such as a fork-lift truck, bulldozer or excavator.

The actual plant is fitted with digital video cameras, microphone and speakers, and devices to remotely turn the steering wheel and to operate pedals and levers.

The commercial advantage is that almost any existing heavy equipment can be retrofitted for teleoperation rather than having to purchase a new specially designed model.

Furthermore, the operator can then be anywhere. Working from home during COVID-19 – and after the pandemic in due course – can now in principle even be an option for heavy plant operators.

It is important to note that the machinery is not operating autonomously. Rather, a professionally qualified operator controls the machine from a remote location, maintaining the same safety procedures and responsibilities that apply whether or not the real machine has a human in its driver's cabin.

Although artificial intelligence is not used to control the machine, teleoperation can produce substantial quantities of training data that can then be used for machine learning.

Driving seat

Nevertheless, a standard desktop machine with a hobbyist steering wheel and pedals is a poor and potentially unsafe substitute for the driving seat of a real plant.

Stereoscopic vision, tactile feedback from the controls, ambient vibration and noise are important ergonomic factors for safe operation. Thus, some equipment manufacturers are now exploring teleoperation with full-size reproductions of driving cabins, akin to professional flight and ship bridge simulators.

An operator is then given a much more authentic experience, but at a remote site away from the actual equipment. The investment necessary for full-size reproductions may make teleoperation of routine work prohibitive.

Nevertheless, teleoperation is becoming adopted in situations where the safety for an operator is of concern, for example, in using heavy equipment to clear landslide or earthquake debris.

Hazardous environments have also rapidly become a primary concern of medical professionals. A Chinese initiative is exploring whether teleoperated mobile robots could replace the risk for nursing and medical staff inside COVID-19 wards and ICUs.

Each self-propelled robot has a fully articulated arm and hand, cameras and a screen, speakers and microphone. The nurse or consultant, from outside of the ward or ICU, can see and talk to patients, who in turn can see and talk to the professional.

Wearing a suitable sensor-enabled sleeved glove with tactile feedback, the operator can move the robot arm and its hand to operate bedside equipment, administer drugs, and take temperature and other readings.

The acceptance and consent of patients to their care provided via a robot are improved by their acknowledgement of reduced infectious risk to their professional medical carers.

This article first appeared in The Irish Times on November 5, 2020.

Author: Dr Chris Horn, former president of Engineers Ireland, is the co-founder, CEO and chairman of IONA Technologies, industry expert on Irish technology development, trends, and business. As an honorary Doctor of Science from Trinity College Dublin and former TCD lecturer in computer science, Dr Horn is at the forefront of the Irish high-tech debate.