How do you train people in jobs where one wrong move can have dangerous consequences? Very carefully, is the traditional answer. But with virtual reality (VR), it is now possible to engage these learners in decision-making scenarios before they apply themselves in the real world. Dr Henry Y K Lau of the Department of Industrial and Manufacturing Systems Engineering (IMSE) is at the forefront in developing such VR technology in Hong Kong.

Over the past 15 years, Dr Lau has led the evolution of the imseCAVE, an immersive technology that began with a joy stick, large projectors and cloth screens to train cargo handlers at the container terminal, but is now a sophisticated training tool offering interactive stereo images and 360-degree video using high definition projections, wearable VR goggles, multi-player scenarios and the incorporation of real-life features, such as a fire-truck cab.

“There’s a lot of interest in the public about this field because of the fantasy and ‘wow’ factors of being able to perceive virtual worlds and to teleport yourself to scenarios that may never be easy to access. But we’ve been able to develop a lot of practical uses for this technology, too,” he said.

Apart from the container terminal, Dr Lau and his team were commissioned by Cathay Pacific to develop a system to train workers in loading cargo using the virtual system plus a real-life control panel, and by the Hong Kong Airport Authority to develop a system to train operators in connecting passenger airbridges to aircraft and drivers in operating vehicles on the airfield. The Fire Services Department recently asked them to create a system that trains its drivers.

“Fire-truck drivers need to practise how to do emergency driving in real road conditions. People jump out, or cars don’t move out of the way. Should they overtake them or go off the road? We’re going to simulate various road situations authentic to Hong Kong and see how the firemen react,” he said.

​ There’s a lot of interest in the public
about this field because of the fantasy and ‘wow’ factors of being able to perceive
virtual worlds… But we’ve been able to develop a lot of practical uses for this technology, too.

Dr Henry Y K Lau

Cyber sickness concerns

Another challenge is the need to better understand the effects of VR technology on users, such as cyber sickness. Dr Lau is working with academics and experts in the Department of Psychology to study these effects.

“For example, if you put on the VR goggles and the environment is changing or moving around you but you are not moving, you might feel unbalanced or dizzy after a few minutes and can’t stay there very long. Why is that? Are there pathological issues in human perceptions? Are their visual cues that cause dizziness? Or is it because of hardware performance, such as delay, latency, flashing lights, colour, or the speed of movement of virtual objects? If we can dig down to the more psychological parameters, then we can develop much more powerful, effective and usable VR systems.”

In the meantime, there are many more uses to be had from the technology in its present form. Dr Lau has been involved in start-ups in the field, such as Hactis Ltd, which received TSSSU@HKU funding for the maximum three years and has successfully commercialised the imseCAVE. The company is applying the technology to healthcare and entertainment, as well as end-use by consumers.

“My role in start-ups is more as a mentor, but it is good to see the technology being used, rather than appearing only in academic journals. So far we are doing both things very well,” he said.

Virtual crime scenes

Rather different from these operational kinds of training is their work with the Hong Kong Police to train crime investigators using mock crime scenes. The trainees can move around the scene and identify and label evidence, such as weapons left behind, shoe marks and blood stains. Several trainees can be in the same virtual space at the same time and interact with each other and their supervisors.

“They can have a dialogue, there can be teaching, and the supervisors can assess whether the trainees are doing the right thing interactively in real time,” Dr Lau said.

HKU’s School of Nursing and Faculty of Engineering are also training students in VR settings and the Correctional Services Department is using the technology to train prison guards in such things as night patrols. The latter has virtual prisoners that trainees can respond to, for instance in deciding whether to act on a request for a glass of water, although the technology is not quite able to allow real and virtual participants to speak to each other. That will require further development.

“Virtual reality is only one component in a cyber physical system. The others are artificial intelligence and data communication technology. Not until we bundle these three together can we create a more realistic virtual environment for people to interact. This remains a challenge,” he said.

The Chief Executive Mrs Carrie Lam (second from right) in the Hong Kong Airport Authority’s virtual reality simulator which has been developed to train operators in connecting passenger airbridges to aircraft and drivers in operating vehicles on the airfield.

The team also works with the Hong Kong Police to train crime investigators using mock crime scenes.

The imseCAVE team led by Dr Henry Y K Lau (centre).

WHEN THE VIRTUAL

MEETS REALITY

Prison guards, police officers, cargo handlers, nursing students and others are all benefitting from virtual reality technology developed at HKU.

The Correctional Services Department is using virtual reality technology to train prison guards in such things as night patrols.