Current Issue - Virtual Reality & Intelligent Hardware

Editorial

Virtual reality and augmented reality in medical simulation

Aimin HAO

2021, 3(4) : 1-1

DOI：10.3724/SP.J.2096-5796.2021.03.04

PDF (21)

HTML (558)

Article

Orthodontic simulation system with force feedback for training complete bracket placement procedures

Fan YE, Luwei LIU, Bin YAN, Xiaohan ZHAO, Aimin HAO

2021, 3(4) : 261-273

DOI：10.1016/j.vrih.2021.08.001

Abstract (594)

PDF (17)

HTML (684)

Background

A virtual system that simulates the complete process of orthodontic bracket placement can be used for pre-clinical skill training to help students gain confidence by performing the required tasks on a virtual patient.

Methods

The hardware for the virtual simulation system is built using two force feedback devices to support bi-manual force feedback operation. A 3D mouse is used to adjust the position of the virtual patient. A multi-threaded computational methodology is adopted to satisfy the requirements of the frame rate. The computation threads mainly consist of the haptic thread running at a frequency of >1000Hz and the graphic thread at >30Hz. The graphic thread allows the graphics engine to effectively display the visual effects of biofilm removal and acid erosion through texture mapping. Using the haptic thread, the physics engine adopts the hierarchy octree collision-detection algorithm to simulate the multi-point and multi-region interaction between the tools and the virtual environment. Its high efficiency guarantees that the time cost can be controlled within 1 ms. The physics engine also performs collision detection between the tools and particles, making it possible to simulate paint and removal of colloids. The surface-contact constraints are defined in the system; this ensures that the bracket will not divorce from or embed into the tooth during the adjustment of the bracket. Therefore, the simulated adjustment is more realistic and natural.

Results

A virtual system to simulate the complete process of orthodontic bracket bonding was developed. In addition to bracket bonding and adjustment, the system simulates the necessary auxiliary steps such as smearing, acid etching, and washing. Furthermore, the system supports personalized case training.

Conclusions

The system provides a new method for students to practice orthodontic skills.

Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems

Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG

2021, 3(4) : 274-286

DOI：10.1016/j.vrih.2021.08.002

Abstract (583)

PDF (9)

HTML (589)

Background

Compared with traditional thoracotomy, video-assisted thoracoscopic surgery (VATS) has less minor trauma, faster recovery, higher patient compliance, but higher requirements for surgeons. Virtual surgery training simulation systems are important and have been widely used in Europe and America. Augmented reality (AR) in surgical training simulation systems significantly improve the training effect of virtual surgical training, although AR technology is still in its initial stage. Mixed reality has gained increased attention in technology-driven modern medicine but has yet to be used in everyday practice.

Methods

This study proposed an immersive AR lobectomy within a thoracoscope surgery training system, using visual and haptic modeling to study the potential benefits of this critical technology. The content included immersive AR visual rendering, based on the cluster-based extended position-based dynamics algorithm of soft tissue physical modeling. Furthermore, we designed an AR haptic rendering systems, whose model architecture consisted of multi-touch interaction points, including kinesthetic and pressure-sensitive points. Finally, based on the above theoretical research, we developed an AR interactive VATS surgical training platform.

Results

Twenty-four volunteers were recruited from the First People's Hospital of Yunnan Province to evaluate the VATS training system. Face, content, and construct validation methods were used to assess the tactile sense, visual sense, scene authenticity, and simulator performance.

Conclusions

The results of our construction validation demonstrate that the simulator is useful in improving novice and surgical skills that can be retained after a certain period of time. The video-assisted thoracoscopic system based on AR developed in this study is effective and can be used as a training device to assist in the development of thoracoscopic skills for novices.

Topological distance-constrained feature descriptor learning model for vessel matching in coronary angiographies

Xiaojiao SONG, Jianjun ZHU, Jingfan FAN, Danni AI, Jian YANG

2021, 3(4) : 287-301

DOI：10.1016/j.vrih.2021.08.003

Abstract (600)

PDF (5)

HTML (526)

Background

Feature matching technology is vital to establish the association between virtual and real objects in virtual reality and augmented reality systems. Specifically, it provides them with the ability to match a dynamic scene. Many image matching methods, of which most are deep learning-based, have been proposed over the past few decades. However, vessel fracture, stenosis, artifacts, high background noise, and uneven vessel gray-scale make vessel matching in coronary angiography extremely difficult. Traditional matching methods perform poorly in this regard.

Methods

In this study, a topological distance-constrained feature descriptor learning model is proposed. This model regards the topology of the vasculature as the connection relationship of the centerline. The topological distance combines the geodesic distance between the input patches and constrains the descriptor network by maximizing the feature difference between connected and unconnected patches to obtain more useful potential feature relationships.

Results

Matching patches of different sequences of angiographic images are generated for the experiments. The matching accuracy and stability of the proposed method is superior to those of the existing models.

Conclusions

The proposed method solves the problem of matching coronary angiographies by generating a topological distance-constrained feature descriptor.

A virtual reality based surgical skills training simulator for catheter ablation with real-time and robust interaction

Haoyu WANG, Jianhuang WU

2021, 3(4) : 302-314

DOI：10.1016/j.vrih.2021.08.004

Abstract (710)

PDF (3)

HTML (622)

Background

Atrial fibrillation (AF) is the most common sustained cardiac arrhythmia that can cause severe heart problems. Catheter ablation is one of the most ideal procedures for the treatment of AF. Physicians qualified to perform this procedure need to be highly skilled in manipulating the relevant surgical devices. This study proposes an interactive surgical simulator with high fidelity to facilitate efficient training and low-cost medical education.

Methods

We used a shared centerline model to simulate the interaction between multiple surgical devices. An improved adaptive deviation-feedback approach is proposed to accelerate the convergence of each iteration. The periodical beating of the human heart was also simulated in real time using the position-based dynamics (PBD) framework to achieve higher fidelity. We then present a novel method for handling the interaction between the devices and the beating heart mesh model. Experiments were conducted in a homemade simulator prototype to evaluate the robustness, performance, and flexibility of the proposed method. Preliminary evaluation of the simulator was performed by medical students, residents, and surgeons.

Results

The interaction between surgical devices, static vascular meshes, and beating heart mesh was stably simulated in a frame rate suitable for interaction.

Conclusion

Our simulator is capable of simulating the procedure of catheter ablation with high fidelity and provides immersive visual experiences and haptic feedback.

Development and application of digital assistive teaching system for anatomy

Na ZHANG, Liwen TAN, Fengying LI, Bing HAN, Yifa XU

2021, 3(4) : 315-335

DOI：10.1016/j.vrih.2021.08.005

Abstract (776)

PDF (4)

HTML (615)

Background

Anatomy is a required course for all medicine-related industries. In recent decades, the teaching quality and effect of anatomy have been compromised by factors including a decrease in human body specimens, dampened enthusiasm for the discipline, reduced teaching hours of anatomy, scale expansion of medical education, and obstacles in performing field autopsies and observations.

Methods

Based on China's digitalized visible human research achievements, this article extracts the boundary information of anatomic structures from tomographic images, constructs three-dimensional (3D) digital anatomical models with authentic texture information, and develops an anatomy assistive teaching system for teachers and students based on the knowledge points of anatomy, to meet the anatomy teaching requirements of different majors at various levels.

Results

This scientific, complete, and holistic system has produced over 6000 3D digital anatomical models, 5000 anatomy knowledge points, 50 anatomical operation videos, and 150 micro demonstration classes, with teaching contents for different majors and levels, such as systematic anatomy, topographic anatomy, sectional anatomy, anatomy of motion, and virtual anatomical operation table. Ranging from network terminals, desktops, touchscreen 3D displays, desktops, and projection 3D volumetric displays to augmented reality, its diversified interactive forms meet the requirements for a learning environment in different settings.

Conclusions

With multiple teaching and learning links covered, such as teaching environment, teaching resources, instructional slides, autonomous learning, and learning effect evaluation, this novel teaching system serves as a vital component and a necessary resource in anatomy teaching and functions as an important supplement to traditional anatomy teaching. Applied and promoted in most medical colleges and schools in China, this system has been recognized and approved by anatomy teachers and students, and plays a positive role in guaranteeing the effect and quality of anatomy teaching.

A marching cube algorithm based on edge growth

Xin WANG, Su GAO, Monan WANG, Zhenghua DUAN

2021, 3(4) : 336-349

DOI：10.1016/j.vrih.2021.08.006

Abstract (553)

PDF (3)

HTML (761)

The marching cube algorithm is currently one of the most popular three-dimensional (3D) reconstruction surface rendering algorithms. It forms cube voxels based on an input image and then uses 15 basic topological configurations to extract isosurfaces from the voxels. The algorithm processes each cube voxel in a traversal-based manner, but it does not consider the relationship between the isosurfaces in adjacent cubes. Owing to ambiguity, the final reconstructed model may have holes. In this paper, we propose a marching cube algorithm based on edge growth. The algorithm first extracts seed triangles, grows these seed triangles, and then reconstructs the entire 3D model. According to the position of the growth edge, we propose 17 topological configurations with isosurfaces. The reconstruction results showed that the algorithm can reconstruct the 3D model well. When only the main contour of the 3D model is required, the algorithm performs well. In addition, when there are multiple scattered parts in the data, the algorithm can extract only the 3D contours of the parts connected to the seed by setting the region selected based on the seed.