Chinese
Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2020,  2 (1):   12 - 27   Published Date:2020-2-20

DOI: 10.1016/j.vrih.2019.12.002
1 Introduction2 System framework3 Cable bracket inspection 3.1 Portable visual inspection device 3.2 Cable bracket detection and recognition 3.2.1   Object detection of aircraft cable brackets 3.2.2   Recognition of aircraft cable bracket type 3.2.3   Keypoint extraction and 3D reconstruction for detecting the posture of the brackets 3.3 Determination of bracket state by matching the inspection data to the design model data 4 Cable text reading 4.1 Workflow of cable text reading 4.2 Reading the text in the image obtained by a wearable device 5 Assembly process guidance based on AR 5.1 Generation of the assembly process 5.2 Visualization of the assembly process 6 Results7 Discussion and future work

Abstract

Background
Assembly guided by paper documents is the most widespread type used in the process of aircraft cable assembly. This process is very complicated and requires assembly workers with high-level skills. The technologies of wearable Augmented Reality (AR) and portable visual inspection can be exploited to improve the efficiency and the quality of cable assembly.
Methods
In this study, we propose a smart assistance system for cable assembly that combines wearable AR with portable visual inspection. Specifically, a portable visual device based on binocular vision and deep learning is developed to realize fast detection and recognition of cable brackets that are installed on aircraft airframes. A Convolutional Neural Network (CNN) is then developed to read the texts on cables after images are acquired from the camera of the wearable AR device. An authoring tool that was developed to create and manage the assembly process is proposed to realize visual guidance of the cable assembly process based on a wearable AR device. The system is applied to cable assembly on an aircraft bulkhead prototype.
Results
The results show that this system can recognize the number, types, and locations of brackets, and can correctly read the text of aircraft cables. The authoring tool can assist users who lack professional programming experience in establishing a process plan, i.e., assembly outline based on AR for cable assembly.
Conclusions
The system can provide quick assembly guidance for aircraft cable with texts, images, and a 3D model. It is beneficial for reducing the dependency on paper documents, labor intensity, and the error rate.

Content

1 Introduction
Cables are an important medium for power and signal transmission in aircraft[1]. Owing to the complexity of aircraft avionic systems, the cable assembly process involves a large number of components in a complex environment. The main tasks of cable assembly include the inspection of cable brackets, the reading of cable texts, and the laying of cables. At present, paper documents of assembly outline (AO) are still primarily used in aircraft assembly, which is associated with three main drawbacks[2,3] . Firstly, there are specific inspection requirements for bracket assembly. After brackets are installed, it is necessary to review their correctness and completeness. However, this inspection process is complicated because of the many variety and quantity of brackets. Secondly, it is quite difficult to identify the correct cables given the wide variety of cable types. In addition, the number of cables assembled on the brackets is large. During the assembly process, workers need to repeatedly visually confirm the correctness of the identification text (the unique identity) on the cables. In this case, assembly workers need to focus intently, and the task is time-consuming and prone to error. Thirdly, unlike rigid parts that can be directly installed according to the instructions in a paper document, flexible cables need to be fixed step-by-step according to the assembly outline, and the process needs to be repeatedly confirmed using 2D drawings. Owing to these issues, the current paper AOs are not ideal for guiding the aircraft cable assembly process, and results in high labor intensity, low work efficiency and a high rework rate. Wearable AR and portable visual inspection provide a promising solution for the improvement of the efficiency and quality of cable assembly. On one hand, a portable binocular vision system combined with deep learning has been adopted for the inspection of the brackets after installation, to realize rapid detection and transmission of the inspection results to wearable devices in real-time. On the other hand, visual recognition and deep learning technologies have been adopted to quickly read and identify cable text and transmit the results to wearable devices in real-time, providing support for visual guidance of the cable assembly. The 3D model of the cable can be superimposed on the aircraft cabin using AR technology, and workers can quickly fix cables to their designated position according to the 3D cable model, thereby avoiding the multiple confirmations of paper AOs.
In the field of deep-learning-based object recognition, many scholars have conducted similar studies. The recognition of face and aerial images is a relatively active research field at present[3]. Matiz et al. proposed an inductive conformal predictor for convolutional neural networks (ICP-CNN), in which experiments conducted using face recognition databases demonstrated an improved performance[4]. Zhuang et al. extended an End-to-End deep convolutional neural network for automatic face image quality prediction[5]. Patra et al. presented a novel technique for the classification of hyperspectral images using limited labeled samples[6]. However, when compared to faces and aerial images, the research object used in this paper has many different features, i.e., complicated background, an unobtrusive feature, and the small scale of the region of interest (ROI). Therefore, existing approaches for face and aerial image recognition are not directly applicable to bracket inspection. In addition, to the best of our knowledge, there is still no specialized or related research on aircraft cable bracket inspection. In this study, deep learning is utilized for the inspection of the quality of the assembly of cable brackets.
Reading and identification of cable text belong to the research field of character recognition. At present, there are numerous reference studies. However, in most cases, the research objects are standard characters such as the recognition of license plate and plane printing characters[7,8,9,10]. Owing to character distortion and inconsistent spacing, existing character recognition methods have many shortcomings when directly applied to cable text reading.
Cable and harness assembly are examples of typical AR and VR applications in industry and manufacturing. Caudell and Mizell[11], employees at Boeing, first proposed the concept of AR. A wearable AR prototype that facilitated information overlay of virtual and real scenes was applied to aircraft cable assembly to provide real-time guidance for the user. Ritchie JM developed a cable assembly simulation system using virtual reality technology[12]. The user can assemble the cable model in a virtual environment, and the system captures the user's hand motion information via a data glove for interaction with a virtual model. However, this system cannot be used for the formal assembly process, but instead, for the assembly training process. Erkoyunc[13] applied AR to industrial maintenance and developed an AR assisted assembly system that can adaptively sense assembly semantics. The system has an AR process authoring function, which can be easily used by maintenance engineers without programming experience to construct an AR assisted assembly process.
This report presents an aircraft cable assistance assembly system that combines wearable AR with a portable visual inspection, by analyzing the deficiencies of previous approaches. Specifically, a portable visual inspection device based on binocular vision is proposed to realize rapid detection and recognition of cable brackets. Moreover, a deep learning model that is used for inspection is optimized according to the characteristics of the bracket to improve accuracy. Deep learning and visual recognition are used to read the text on cables, and the results are sent to wearable devices for visual assembly guidance. Wearable devices based on AR technology are used to guide the cable assembly process. To improve the efficiency of the process, a cable assembly process authoring software is developed.
2 System framework
The proposed system, which combines wearable AR and portable visual inspection technology, can be divided into three subsystems. They are bracket inspection, cable text reading, and AR assembly guidance.
In the subsystems for bracket inspection and cable text reading, the devices for data acquisition are portable binocular industrial cameras and a camera on the wearable AR device respectively. The process information for AR includes text, image, and 3D model. For data processing, both the bracket inspection subsystem, and cable text reading subsystem use the deep learning of a CNN Network as their data processing engines. To adapt to the characteristics of the inspected objects, the classic CNN Network was studied and improved to build a deep learning CNN for bracket inspection and cable text reading. In addition, the AR assembly guidance subsystem uses the AO authoring software developed in this study for data processing. The data expression of each subsystem is realized using an XML file into which the data are encapsulated. Then the XML file is sent to the wearable device for display, as shown in the data display phase in Figure 1. Finally, this system is utilized in a workshop for cable assembly.
In addition, the resources of the AR assembly guidance subsystem include texts, images, and 3D models. They are inputted into the wearable device by a process engineer during the data acquisition phase. An engineer without professional programming experience can manage the resources using authoring software in the data processing phase.
3 Cable bracket inspection
3.1 Portable visual inspection device
It is difficult for traditional inspection devices to be used on an aircraft assembly site because of the complex environment and limited space. Therefore, a portable visual inspection device is required. In this study, a portable device for cable bracket inspection is developed. It contains three parts, i.e., a computing unit, a movable load unit with four wheels, and binocular cameras. The computing unit and binocular cameras are carried by the movable load unit, and the binocular cameras can be adjusted up and down. Moreover, the system can be moved freely in the workshop, as shown in Figure 2.
3.2 Cable bracket detection and recognition
3.2.1   Object detection of aircraft cable brackets
Given the huge quantity, various types, and variable layout of the brackets used in complicated assembly environments, object detection of cable brackets in the assembly site is critical. In addition, aircraft cable brackets are small objects and the features of brackets are not obvious. These factors impose additional challenges. As a result, it is difficult to detect the object of aircraft cable brackets in the assembly site.
Deep learning technology has sufficient generalization for image-feature detection[14]. In this report, deep learning is used to detect brackets in different assembly scenes, and with the assistance of binocular vision, the coordinates of brackets can be obtained.
The cable brackets in the assembly scenario are relatively small compared to the field of view, therefore, small target detection is important. Most previous algorithms use top-level features to make predictions, which results in a loss of the underlying information. However, by utilizing the structure of the Feature Pyramid Net[16] that is used in this investigation, feature information obtained at different feature layers can be fused as shown in Figure 3. The core idea of this algorithm is that it includes a bottom-up pathway, a top-down pathway, and a lateral connection. The bottom-up pathway is the forward process of the network, which is identified as ResNet in Figure 3, and the top-down pathway called Feature Pyramid Net that facilitates up-sampling, is the backward process. The lateral connection merges the feature maps generated by the up-sampling and bottom-up pathway to obtain the feature map for the detection of aircraft cable brackets.
3.2.2   Recognition of aircraft cable bracket type
To satisfy the requirements for detection in aircraft cable bracket installation, it is necessary to recognize the type of cable bracket to confirm that the right type is installed.
Based on image analysis of the region of interest (ROI) of the cable brackets, the proportion of the image size occupied by the bracket in the ROI area is relatively large at approximately 90%. The cable bracket has a smooth surface, the color is basically uniform, the texture is missing, and there is no large color change. Comparing different types of images of cable brackets, it is evident that they have a lot in common. The main features of the different types are the position of the cable fixing holes relative to the bracket, the outline of the outer edge and the length-width ratio of the cable brackets.
In this investigation, we study the classification criteria of aircraft cable brackets based on shape, size, and color, and use the method of deep convolutional neural network to extract features from cable brackets and to classify these features to confirm the type of brackets in the installation scenario. In object detection, a sufficiently deep network is needed to locate the cable brackets, but these networks can cause over-fitting for the bracket classification. In this report, the VGG-8 network is used as the feature extraction network for aircraft cable brackets. Its basic structure is shown in Figure 4. The network includes three conv blocks (yellow boxes in Figure 4). Each conv block uses a single conv layer that is composed of a 3×3 Kernel, a Batch Normalization, and a ReLU-function, as shown in the dashed box. In addition, three pooling layers are set between the conv blocks and the first fully connected layer. The network includes three conv layers and five fully connected layers, hence the name VGG-8.
3.2.3   Keypoint extraction and 3D reconstruction for detecting the posture of the brackets
To confirm the location of the brackets, keypoint extraction and 3D reconstruction of the cable brackets should be implemented. This study utilizes binocular vision to realize this requirement. Binocular vision is based on the parallax between the same object image using left and right cameras. The triangulation principle is used to extract 3D geometric information from the object. In this process, it is necessary to calibrate the camera to determine its internal and external parameters. Prior to the completion of 3D inversion, it is necessary to collect the target scene image, extract the key feature points, and math the feature points.
The aim of feature extraction is to identify key features in the assembly scene for the subsequent determination of the 3D coordinates of the aircraft cable brackets. As shown in Figure 5, traditional feature extraction operators such as SIFT and HOG[16] cannot effectively extract the robust features of aircraft cable brackets.
To obtain effective robust features of aircraft cable supports in assembly scenarios, the ROI of the brackets obtained in 3.2.1 is utilized in this study to perform feature extraction, as shown in Figure 6.
Figure 7 shows the critical process of feature extraction for brackets based on binocular vision, which includes five steps. Step 1 and step 2 are designed to acquire and preprocess the bracket images. The specific method involves the grayscale conversion of the acquired images. Gaussian filtering is initially used to minimize noise. In addition, edge detection is performed on the images of the cable brackets using an edge detection algorithm as shown in step 3. The bounding box of the assembly holes and the entire bracket is screened by calculating the perimeter of the edge.
To determine the accurate coordinates of the bracket, it is necessary to solve for the center of bracket, for which images are simultaneously acquired using two cameras as shown in step 4. In addition, the surface textures are a single color, and the pixels of the surface are evenly distributed. Therefore, the invariant moment can be used to solve for the centers of the cable brackets and the assembly holes. In this report, we suppose that
f ( i ,   j )
is a digital image. The geometric moment[17] and central moment of rank p-q are
m p q
and
μ p q
.
m p q = i = 1 M j = 1 N i p j q f ( i , j )
μ p q = i = 1 M j = 1 N ( i - i ¯ ) p ( j - j ¯ ) q f ( i , j )
The position of the center is
C x = M 10 M 00 , C y = M 01 M 00
where
M 10
and
M 01
are the first moments, and
M 00
is the zero moment. In step 5, the left and right cameras simultaneously acquire the assembly scene image using the binocular vision subsystem. The epipolar-constraint can be used to compress the corresponding point matching problem from the entire image to a straight line.
After the aforementioned processes, the information of the corresponding matching point is obtained for 3D reconstruction, including the coordinates of the center and the assembly holes on the cable brackets. The actual 3D coordinates corresponding to the feature points are then obtained.
The two holes and the center of the bracket can be used to determine the state of the bracket. As shown in Figure 8, the position relationship between the two holes and the center position on the same cable bracket can form a spatial isosceles triangle, with a base edge that represents the connecting line of the two holes and the vertex as the center point. As a result, the installation state of a bracket is determined by the relationship between vertex and bottom edge.
3.3 Determination of bracket state by matching the inspection data to the design model data
The inspection data of a bracket, which includes type, center coordinates, and posture, is saved to the database as a record, and compared with the data of its theoretical model to determine whether the bracket is type error, missing, or reversed. The interpretation process is as follows:
Table 1 shows the matching process for bracket inspection; it mainly includes several cycles. The core idea of this algorithm is to use a loop nesting process to compare the inspection data to the design model data. The two matrices,
M m   
and
D n
, are the inputs of the algorithm. The matrix
M m
stores the data that includes information on the coordinate, type, and posture, which are acquired from the detection and recognition phases, as shown in section 3.2. The matrix
D n
corresponds to
M m
, which stores the information from the design model. The outputs of the algorithm are the assembly states of the brackets. After the matching process, the system can obtain and output the assembly states of the brackets to the assembly workers to end the cable brackets inspection task.
Determination of bracket states by matching inspection data with design model data
Algorithm Matching inspection data with design data for state detection of brackets
Input: Design Matrix
M m = C i , T i , P i
/*
C i
is the coordinates,
T i
is the type, and
P i
is the posture, m is the total number of designed */
Inspection Matrix
D n = c j , t j , p j
/*
c j
is the coordinates,
t j
is the type, and
p j
is the posture. n is the total number that was inspected */
Output: The assembly states of the brackets.
1: initialize
i 1
//The matching started from
M 1 a n d D [ 1 ]
.
2: function “Match-Data”
3:
j 1
4: for
i
to
m
//The matching ends only if all design matrixes are processed
5: do if
c j = C i
//verify the position of the bracket
6: then if
t j = T i
//verify the type of the bracket
7: then if
p j = P i
//verify if the bracket is missing
8: then print (“The
i
th bracket is OK!”) // Installed correctly
9: else print (“The posture of the
i
th bracket is wrong!”)
10: else print (“The type of the
i
th bracket is wrong!”)
11: if
j n
12: then
j j + 1
13: goto 5 //matching the inspected data with
M i
in-loop
14: else goto 3 //matching of the inspected data with
M i + 1
in-loop
15: else if
j n
16: then
j j + 1
17: goto 5 //matching of the inspected data with
M i
in-loop
18: else print (“The
i
th bracket is missing!”)
19: goto3 // matching of the inspected data with
M i + 1
in-loop
20: end function
4 Cable text reading
Owing to the complex background environment of the assembly site, it is difficult for existing character recognition methods to correctly identify the text printed on the aircraft cable. In this report, a new text recognition technology is proposed that can accurately detect and identify text including letters and Arabic Numbers.
4.1 Workflow of cable text reading
Figure 9 shows the workflow for reading the cable text, including the capture of images at the assembly site, detection of the text area, text recognition, and display on wearable devices. The camera on the wearable device is used to obtain images containing cable text in the industrial assembly site. These images are then inputted into the training model. A series of pre-processed predictive feature images are outputted after training. A bounding box of text is formed in the predicted feature images, and in its smallest area, a rectangular text region is formed. This region that contains a series of cable text is cropped for resizing and pixel padding. The processed text area is inputted into the text recognition training model to predict the character sequence. The location information and the recognized text in the image are eventually outputted.
4.2 Reading the text in the image obtained by a wearable device
The process of reading the text in the image acquired by the camera on the wearable device consists of two steps: detection of the ROI and recognition of the characters including letters and Arabic numerals.
As shown in Figure 10, the network architecture for the detection model uses the features of different levels to detect the texts of aircraft cable of various sizes. The model consists of a feature extractor stem, a feature-merging branch, and the output layer. The backbone of the feature extractor stem in the network is VGG16[18], which has deeper layers than PVANet[19] and can build the deep learning model using simple small modules. It includes five conv-blocks from shallow to deep. The feature merging branch combines the output of each conv-block with the module below, and it adopts an up-sampling approach similar to the FPN architecture. The output layer produces three per-pixel prediction maps including a text score map, a text border map, and a vertex regressor. The text score map represents the confidence score of each pixel as a text pixel. The text border map is a 4-channel confidence score of each pixel as the border sections of text. The vertex regressor contains four channels that are used to predict the coordinate shift using the closer two corner vertices of the quadrangle to the pixel location. The values of the four channels denote the four offsets.
To improve the performance of character recognition, post-processing is necessary before feeding the images into the recognition training model. The region prediction model outputs a text score map, a text border map, and a vertex regressor. Based on the first two maps, several rough-text blocks (orange), short text border blocks (yellow and red) and long text border blocks (green and blue) are determined based on global thresholding 8, as shown in Figure 11a. In Figure 11b, the inner-text blocks (orange) are delineated by removing the overlaps between the rough-text blocks and the two long-text border blocks. The overlaps between each inner-text block and two short text border blocks are regarded as side-pixels, including head-pixels (yellow) and tail-pixels (red).
Text regions are cropped from the images according to the bounding boxes obtained after the post-processing steps. The cropped text region is initially resized by setting the height of this image to 64 pixels and the width to N×16-16. The parameter N is an integer that is reversely derived from the recognition network CRNN, to prevent the calculation process of convolution and pooling in the CRNN model from crossing the boundary of the images. Thus we have N=max[2, (w×4⁄h+1.5)], where w is the width and h is the height of the image before resizing. To ensure that the text is in the center of the cropped image, 32 blank pixels are padded to two sides of the image. The final image of text region has a height of 64 pixels and width of 16+48 pixels. The CRNN model is employed as the text recognizer and the output text content is obtained by putting the cropped image after region pre-processing of the model (Figure 12).
5 Assembly process guidance based on AR
Currently, the assembly of aircraft cables depends on the skill level of the worker, which leads to significant limitations. For instance, the soft-structure of cables is not amenable to the recognition of the assembly path. Augmented reality allows this problem to be effectively addressed. In the case of aircraft cables, assembly based on AR contains two steps, i.e., generation and visualization of the assembly process.
5.1 Generation of the assembly process
Limited by expertise, it is usually challenging for an assembly process engineer to design a seamless process document based on AR. Consequently, a standard procedure and software for the generation of the assembly process are essential.
The assembly process engineer is familiar with process information, which includes the tool, material, text, 2D or 3D image, and the 3D model based on AR. To assist the assembly process engineer in organizing the process information based on AR, a software tool for assembly process authoring is useful. The interface of the software is shown in Figure 13. The engineer can then focus on the design of the assembly process, which becomes simpler and more efficient with the assistance of the software. In addition, an XML document is generated for the use of an assembly process in an AR device. It can be automatically generated by clicking on the SAVE button of the software after completion of the design.
There is a record in the XML file that corresponds to each step operation in the software. An example of the detail content of the XML file is presented in Figure 14. As shown in the left part of this figure, the assembly information includes products, processes, and steps, and they are each generated automatically using the software.
5.2 Visualization of the assembly process
The visual aided assembly contains two major parts, i.e., bracket inspection aided and cable assembly guidance, which includes cable text reading. Section 3 elaborates on the bracket inspection process in detail. It exports an XML document that can be utilized in a wearable AR device after inspection. The reading of cable text is described in Section 4. The result of the reading is packaged into an XML file and sent to the wearable AR device for display in real-time.
The content of the developed assistance system is managed by a control panel displayed in the wearable device, as shown in Figure 15. The icon “bracket inspection” connects to the bracket inspection subsystem, and by clicking on the “Cable Text Reading” icon, a cable text is acquired. The icon “Picture” shows a 3D model image. The “Tool” and “Material” display the tools and material used in the assembly. In addition, “Loading the scene” loads the AR model and matches it to the real scene to guide the assembly cable process.
The final interface of this AR assisted assembly system is shown in Figure 16. The product and process information are displayed on the upper left corner of the screen, and an image of a 3D model is displayed at the bottom corner of the screen. A control panel that can be moved by the user to any position on the screen is used to schedule assembly instructions. Most importantly, a 3D model based on AR is matched to the actual scene to assist workers in the assembly process.
6 Results
In this study, 7290 samples of brackets were collected from an assembly prototype. To prove that feature extraction using our method is more efficient, a set of experiments were performed, and the results were compared to traditional extracting operations. SIFT and HOG are separately used as the feature extractors for cable brackets. In addition, SVM is used as the classifier to identify the type of cable bracket. The results are shown in Table 2. Neural networks as a feature extractor are more efficient than traditional extraction operations. This report also presents a comparison experiment using Multi-Layer Percep-tion, and it is determined that the proposed method is superior to this approach. The proposed method saves more time in the retrain stage compared to the Multi-Layer Perception when the assembly scene is changed.
Comparison of different methods
Methods Mean accuracy (%)
SIFT+SVM 56
HOG+SVM 21.99
Multi-Layer Perception 76.97
Proposed approach 85.69
With respect to cable text recognition, correct recognition means all the characters of the text are correctly identified (i.e., word recognition). At present, a unified criterion is not available for the evaluation of the performance of aviation cable text detection and recognition. In this study, the evaluation criterion for scene text detection tasks in ICDAR 2015 is adopted. It mainly evaluates the detection model for the cable in case the overlap (IoU) between the detection and ground truth bounding box is greater than 0.5, i.e., 0.5, 0.6, and 0.7, and express the results of tests using different methods as shown in Table 3.
Recall (R), Precision (P), and f-score (F) of different detection methods under different IoUs
IoU 0.4 0.5 0.6 0.7
P R F P R F P R F P R F
East 33.33 65.28 44.13 8.63 16.91 11.43 6.11 11.96 8.09 - - -
AdvancedEAST - - - 96.26 97.01 96.64 93.81 94.55 94.18 87.76 88.44 88.10
Ours - - - 97.55 98.44 97.99 96.78 97.66 97.22 92.14 92.98 92.56
The detection network is trained with 736×736 images using the Adam optimizer with an initial learning rate of 10-3 and a batch size of 4, based on the VGG16 model pre-trained on the ImageNet data. In this study, new layers (e.g., the feature-merging branch and output layer) are initialized by using random weights and a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. The recognition network is fine-tuned based on a pre-trained CRNN model. As shown in Table 2, our method is superior to that of the East and Advanced East for all the indexes.
7 Discussion and future work
To address key problems in aircraft cable assembly, a smart assistance system that combines wearable augmented reality and portable visual inspection is proposed. It consists of three subsystems, i.e., bracket inspection, cable text reading, and assembly process guidance based on AR. The main findings are summarized as follows.
(1) The inspection of the bracket subsystem combines binocular vision and deep learning technology. In this report, a special deep learning model is designed based on the analysis of the assembly environment and the inherent characteristics of brackets. By matching the inspection information features of the bracket position and pose to the design model, detection of missing, wrong, and reverse installation of the bracket can be automatically and rapidly determined.
(2) Automatic reading of the text printed on cable, which is a preprocessing of AR assisted assembly, is realized based on special deep learning for the detection of cable text. The camera on an AR device gathers cable images containing text. These images are fed into the designed deep learning model. The output of the subsystem for cable text reading is an element of the assembly process guidance system.
(3) An assembly process engineer is typically competent in assembly process design, but not in AR programming. In this report, a standard procedure and authoring software are proposed to specifically generate the assembly process, which reduces the programming skill requirements of engineers.
This system was applied to the cable assembly process of an aircraft cabin simulator. The results show that it can assist workers to quickly inspect the state of brackets and cable text, and readily show the installation path of cables to be assembled on the bulkhead with numerous brackets. Evidently, it has improved the assembly efficiency and quality of the aircraft cable assembly process.
However, some parts of the system could be further improved. For example, this system uses Deep Learning for the inspection of cable brackets and the reading of cable text. It needs to collect a large number of training samples in advance, and this process is time-consuming and laborious. Few-shot learning or Meta-Learning could be introduced to the system in the future. In addition, due to the limitation of current AR devices, the field of view is very small, and it is heavy. As such, it is not convenient for use in tasks that require the device to be worn for a long time. In the future, a more lightweight AR wearable device should be developed.

Reference

1.

Mora N, Rachidi F, Pelissou P, Junge A. Numerical simulation of the overall transfer impedance of shielded spacecraft harness cable assemblies. IEEE Transactions on Electromagnetic Compatibility, 2015, 57(4): 894–902 DOI:10.1109/temc.2015.2404928

2.

Geng J H, Zhang S M, Yang B. A publishing method of lightweight three-dimensional assembly instruction for complex products. Journal of Computing and Information Science in Engineering, 2015, 15(3): 031004 DOI:10.1115/1.4029753

3.

Görgel P, Simsek A. Face recognition via deep stacked denoising sparse autoencoders (DSDSA). Applied Mathematics and Computation, 2019, 355: 325–342 DOI:10.1016/j.amc.2019.02.071

4.

Matiz S, Barner K E. Inductive conformal predictor for convolutional neural networks: Applications to active learning for image classification. Pattern Recognition, 2019, 90: 172–182 DOI:10.1016/j.patcog.2019.01.035

5.

Zhuang N, Zhang Q, Pan C H, Ni B B, Xu Y, Yang X K, Zhang W J. Recognition oriented facial image quality assessment via deep convolutional neural network. Neurocomputing, 2019, 358: 109–118 DOI:10.1016/j.neucom.2019.04.057

6.

Patra S, Bhardwaj K, Bruzzone L. A spectral-spatial multicriteria active learning technique for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017, 10(12): 5213–5227 DOI:10.1109/jstars.2017.2747600

7.

Guienko G, Doytsher Y. Geographic information system data for supporting feature extraction from high-resolution aerial and satellite images. Journal of Surveying Engineering, 2003, 129(4): 158–164 DOI:10.1061/(asce)0733-9453(2003)129:4(158)

8.

Yang F, Ma Z, Xie M. A Robust Character Segmentation Approach for License Plate. International Conference on Communications. IEEE Xplore, 2007

9.

Han J, Yao J, Zhao J, Tu J M, Liu Y H. Multi-oriented and scale-invariant license plate\r detection based on convolutional neural networks. Sensors, 2019, 19(5): 1175 DOI:10.3390/s19051175

10.

Zhou X Y, Yao C, Wen H, Wang Y Z, Zhou S C, He W R, Liang J J. EAST: an efficient and accurate scene text detector. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.283

11.

Caudell T P, Mizell D W. Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences. Kauai, HI, USA, IEEE, 1992 DOI:10.1109/hicss.1992.183317

12.

O'B Holt P, Ritchie J M, Day P N, Simmons J E L, Robinson G, Russell G T, Ng F M. Immersive virtual reality in cable and pipe routing: design metaphors and cognitive ergonomics. Journal of Computing and Information Science in Engineering, 2004, 4(3): 161–170 DOI:10.1115/1.1759696

13.

Erkoyuncu J A, del Amo I F, Dalle Mura M, Roy R, Dini G. Improving efficiency of industrial maintenance with context aware adaptive authoring in augmented reality. CIRP Annals-Manufacturing Technology, 2017, 6(1):465–468 DOI:10.1016/j.cirp.2017.04.006

14.

Hao X, Zhang G, Ma S. Deep learning. International Journal of Semantic Computing, 2016, 10(3):417–439 DOI:10.1142/S1793351X16500045

15.

Lin T Y, Dollar P, Girshick R, He K M, Hariharan B, Belongie S. Feature pyramid networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017 DOI:10.1109/cvpr.2017.106

16.

Zhong S H, Wu J X, Zhu Y Y, Liu P Q, Jiang J M, Liu Y. Visual orientation inhomogeneity based convolutional neural networks. In: 2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI). San Jose, CA, USA, IEEE, 2016 DOI:10.1109/ictai.2016.0079

17.

Xu D, Li H. Geometric moment invariants. Pattern Recognition, 2008, 41(1): 240–249 DOI:10.1016/j.patcog.2007.05.001

18.

Dung C V, Sekiya H, Hirano S, Okatani T, Miki C. A vision-based method for crack detection in gusset plate welded joints of steel bridges using deep convolutional neural networks. Automation in Construction, 2019, 102: 217–229 DOI:10.1016/j.autcon.2019.02.013

19.

Yang C X, Li W J, Lin Z Y. Vehicle object detection in remote sensing imagery based on multi-perspective convolutional neural network. ISPRS International Journal of Geo-Information, 2018, 7(7): 249 DOI:10.3390/ijgi7070249