Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2019,  1 (6):   611 - 621   Published Date:2019-12-20

DOI: 10.1016/j.vrih.2019.10.003
1 Introduction2 Overall schemes3 Key technology 3.1 Virtual-real mapping of spraying operation process 3.1.1   Total-factor modeling of spraying operation 3.1.2   Data-driven modeling of the spray robot 3.1.3   Spraying visualization technology 3.2 Online evaluation and optimization of spraying operation process 3.2.1   Online evaluation of spraying operation process 3.2.2   Online optimization of spraying operation process 3.3 Multi-robot efficient co-simulation in large scenes 4 Development and application of system5 Conclusion


This study aims at addressing the lack of closed-loop feedback optimization-enabling tool in aircraft automatic spraying systems; we systematically analyze a three-dimensional (3D) virtual-real mapping technique, namely the digital twin technique, used by the automatic spraying system.
With the sensors installed in the spraying system, the spraying working parameters are collected online and are used for driving the 3D virtual spraying system to realize the total-factor monitoring of the spraying operation. Furthermore, the operation-evaluation model is applied for analyzing and managing the key indexes of the spraying quality; once the data value of the key indexes exceeds a threshold, the operation shall be optimized automatically.
This approach can effectively support the high-efficiency analysis, evaluation, and optimization of the spraying process.


1 Introduction
Surface spraying or painting is the last step in the manufacturing of an aircraft; it is one of the most time-consuming steps in the modern aircraft-manufacturing process. The spraying process requires highly skilled operators. On one hand, these operators can master the task of spraying through training and practice. On the other hand, excellent practice ability is also required. Manual spraying fails to deliver consistent quality, and it is also harmful to the health of the worker. Surface spraying performed by robots offers unique advantages that include efficient spraying, quality consistency, safety, and environmental protection; it has the potential for future development and application in wider markets[1]. Typically, the size of an aircraft exceeds the working-space limit of common industrial robots. Therefore, they need to be specially designed, modified, and integrated; this, however, involves high technical complexity. The robotic spraying system has been introduced in the aviation industry, and initially, the surface painting of the entire machine was realized through robots. Automatic surface-painting of an aircraft involves many processes, and the reasonable arrangement of each process can effectively improve the automation level of the spraying system. Miao et al. studied the key technologies associated with the spraying-operation planning such as aircraft-pose calibration and spray-gun trajectory planning[2]. Based on the secondary development technology, the operation planning platform in the CATIA environment was developed.
Owing to the large size and complex shape of an aircraft, multi-robot cooperation is required in the automatic surface spraying system. Besides, the spraying-process parameters are complicated, and they change dynamically with the working space, time, and environment; thus, it is a typical and complex automated operating system that has high requirements for spraying process and robot collaboration. At present, the research in this field is focused on the development of automatic spraying systems and planning of spraying operations. The implementation of closed-loop feedback optimization has not yet been involved in the field of automatic spraying systems. The digital twin technology, also called virtual-real mapping technology, can link the physical world with the virtual model to realize online monitoring, simulation analysis, and automatic optimization of the production process[3]. Grieves et al. described the digital twin concept and its development; its application across the product lifecycle was also elucidated[4]. Tao et al. presented a new product-design-based digital twin method and proposed a framework for it[5]. Bilberg et al. discussed an object-oriented event-driven simulation as a digital twin of a flexible assembly cell coordinated with a robot to perform assembly tasks alongside humans[6]. Tao et al. proposed a novel concept of digital twin workshop to allow for the communication and interaction between the physical and virtual worlds of manufacturing[7]. Uhlemann et al. presented guidelines for the implementation of the digital twin method in production systems[8].
This study systematically analyzes the 3D digital twin modeling technology to realize the total-factor, full-view 3D monitoring of the spraying operation process in aircraft automatic spraying systems, and it comprehensively discusses the efficient analysis and evaluation optimization of the automatic spraying operation.
2 Overall schemes
The aircraft automatic spraying system primarily consists of three IRB-5500 robots and three sets of three-degree-of-freedom motion platforms. Before the spraying operation, the entire outer surface of the aircraft is divided into several surface blocks based on the size of the robot workspace; the size of each surface block must be smaller than the robot workspace. Then, the position of each moving platform is designed to ensure that the spraying range of the robot at each designated station entirely covers the corresponding surface block. Finally, each robot is delivered at the designated position by the motion platform in advance; then, each robot starts the spraying operation.
The overall scheme of the digital twin technology for the aircraft automatic surface spraying system is detailed in Figure 1. Firstly, based on the process flow, the entire spraying process is simulated considering all the associated factors under consideration in a virtual environment to verify the rationality of the path of motion of the robot. Thus, the interferences between the (1) robots and aircraft, (2) robots and surrounding shop floor environments and (3) robots and robots must be avoided to prevent major accidents during the spraying process. After the simulation is verified, the operation-planning data is transmitted to the on-site industrial computer to drive the robot to perform the spraying operation. During the spraying operation, the spraying system parameters are collected via sensors that are installed in the system; these include the spraying state dataset, motion parameter set of the robot, process parameter set, and the working environment dataset, and all these datasets are transmitted in real time to the virtual monitoring system. Thus, the entire operation process is mapped to the virtual environment in order to realize total-factor monitoring of the spraying operation. The key indicators of the spraying quality are monitored online through the operation-evaluation model. Once the value of the indicator exceeds the set threshold, based on the spraying operation optimization model, the system shall retrieve the process parameter knowledge base to find the optimal process parameters and adjust the process parameters of the on-site operation through control commands. If there is no suitable process parameter or if the value of the indicator still exceeds the threshold value after adjustment, the spraying is stopped by the control command, and an alarm is issued for notifying the technician to perform on-site processing. If the spraying quality of a local area is not satisfactory, the entire process parameter information, environment (temperature, humidity etc.) and spraying characteristics (thickness) can be traced through the spraying process. All this information is used to comprehensively analyze and re-adjust the process parameters. The optimal process parameters are then stored in the knowledge database to achieve closed-loop optimization that fully supports the construction of the spray optimization model under multiple complex factors.
In summary, the digital twin method for aircraft automatic spraying system primarily includes the virtual-real mapping technology and online evaluation and optimization of the spraying operation process; it also involves multi-robot co-simulation in the large complex spraying environment.
3 Key technology
3.1 Virtual-real mapping of spraying operation process
3.1.1   Total-factor modeling of spraying operation
Seven datasets included in the total-factor information of the spraying operation are listed in Table 1. The spraying-operation process is monitored online to support the virtual restoration of the real situation of the spraying operation for optimizing the process. The specific meanings of each parameter are as follows.
Total-factor information of the spraying operation
Type Symbol Description Specific Parameters
Quality parameter set Q

Characterizes quality-related variables such as spray thickness

and consistency

ΔHa(t),σHa ,ΔFHa,

ΔHb(t),σHb ,ΔFHb


parameter set


Characterizes production-related variables such as spraying schedule,

spraying area, and spraying ratio

S,Sa,Sb,PctS,Ta,Tb,PctT, Sp,Spa,Spb,PctSp
Process parameter set OPrt,P

Includes spraying objects and spraying process parameters that are

important variables affecting the spraying quality

Device parameter set K,ODEV Includes information that is closely related to the spray equipment itself such as motion information of spray system and operating status signals




parameter set


Includes the environmental variables that affect the spraying quality and

the selection of the spray process parameters

Parameter set P= {Type, Da, Db, α, v, f, Pa, Pb, W} includes the key parameters for the optimization of the spraying process and improvement of the spraying quality, where Type denotes the type of painting (owing to the differences in composition, viscosity, etc., the process parameters of automatic spraying are fairly different for different paints); Da denotes the spraying distance; Db denotes the lap distance; α denotes the angle between the direction of spraying and the normal to the aircraft skin (the method used to calculate Da and α is detailed in Section 3.2.1); v denotes the spraying speed; f denotes the paint flow rate; Pa denotes the atomization pressure; Pb represents the fan pressure; W represents the spray width (W = C×λ, where λ is determined by Da, Pa, Pb).
Parameter set K={RbtA, RbtB, RbtC} includes the basic motion data for realizing the synchronous mapping from the real robot to the virtual robot, where RbtA represents the motion parameter of robot A (RbtA={Pos, R, Ctrl}, where Pos={x, y, z} denotes the position of the robot; R={R1, R2, R3, R4, R5, R6, R7} denotes the joint posture of the robot); Ctrl denotes robot working state.
Parameter set E={Te, Hr} includes the environmental variables that affect the spray quality that, in turn, affects the selection of the spray process parameters. In E, Te represents the temperature and Hr represents the humidity in the shop floor.
Parameter set OPrt={PrtNo, PrtCur, PrtMat} includes the attribute data of the sprayed object, where PrtNo represents spraying object ID; PrtCur represents the curvature of the sprayed skin; PrtMat represents the material, including composite, metal, etc.
Parameter set ODEV={SigRbtA, SigRbtB, SigRbtC,...} includes the running status of the spraying system such as running, pausing, and breakdown, where SigRbtA, SigRbtB, and SigRbtC are the running states of robots A, B, and C, respectively; ODEV also includes other information.
Parameter set Rate={S, Sa, Sb, PctS, Ta, Tb, PctT, Sp, Spa, Spb, PctSp} characterizes the spraying progress, where S represents ​​the planned total spraying area; Sa denotes the total area that has been sprayed (calculated in real time based on the motion trajectory of the robot and the range of spray gun); Sb denotes the remaining spray area (Sb=SSa); PctS denotes the ratio of the sprayed area to the total spraying area (PctS=(Sa/S)×100%); Ta denotes the spraying time; Tb denotes the estimated spraying time left (Tb=Sb/v); PctT denotes the ratio of the spraying time to the total time (PctT=(Ta/(Ta+Tb))×100%) and helps in understanding the spraying progress with respect to time; Sp represents the spraying area of the current sprayed skin (automatically calculated during spraying planning); Spa denotes the area of the current skin that has been sprayed, And when spraying a new skin, spraying time t is recorded (Spa=W×t×v); Spb denotes the remaining spraying area of the current skin (Spb=SpSpa); PctSp denotes the ratio of the spraying area to the total area of the current skin (PctSp=(Spa/SP)×100%). The spraying process of the current skin can be intuitively understood from PctSp.
The significance of the spraying quality parameter set is detailed in Section 3.2.1.
3.1.2   Data-driven modeling of the spray robot
The process of data-driven modeling of the spray robot primarily involves three levels: (1) Establishing the parent-child relationship between joints of the robot model; (2) Collecting and analyzing the spraying data; and (3) Driving the robot according to the data. Figure 2 illustrates the process of data-driven modeling of the robot. First, an ABB IRB-5500 spray robot is modeled as a 6-joint tandem robot with three independent auxiliary axes. To model the kinematic chain of the end joint of the robot arm, a parent-child relationship is developed by connecting adjacent joint arms, whereby the transformation applied to the parent object can be simultaneously transferred to the child object, and the motion of the child object shall not affect the parent object.
Then, the data of the spraying process is collected by the OPC UA[9] standard protocol, and the data file is generated after processing. The spraying operation data includes system running time, XYZ coordinate of the three-degree-of-freedom platform, and angles of the seven joints of robots R1-R6. The state of the spray includes the opening and closing of the spray gun and the process parameters. The coordinates and the angles are absolute values (i.e., the coordinates are relative to the zero position of the platform, and the rotation angles are relative to the initial positions of the robot joints). The directly collected data files must be analyzed through a customized interface. Then, the corresponding data in the file is extracted and sorted into a single formatting instruction with respect to time. Each instruction includes the coordinate information of the platform, angle information of the robot joints, and status information of the spraying process at the current time.
Finally, the real scene of the aircraft is mapped into the virtual environment. There is a certain difference between the position of the aircraft in the real scene and that in the virtual environment; this can be observed in Figure 3. The position transformation relationship Mb between the planning environment and the actual environment can be derived through calibration. The position of the aircraft in the actual scene can be mapped into the virtual environment by setting the position matrix as Mb×Ma1. There is a one-to-one correspondence between the coordinate and angle information in the extracted format instruction with the objects in the established three independent auxiliary axis model and the robot model. Based on the system running time, the corresponding motion instruction is called, and then the position of the auxiliary axis and the corresponding rotation angle of each joint of the spray robot are controlled based on the position and angle information in the instruction, thereby realizing the data driving of the painting robot.
3.1.3   Spraying visualization technology
In the real process of spraying, the paint is sprayed via the spray gun mounted on the robot to form a mist cone that consists of small particles. Each particle has the following characteristics: (1) It has a certain life cycle, starting from the spray gun and ending at the disappearance after collision with the aircraft skin; (2) It has its own motion state, having a certain emission angle, initial velocity, and acceleration after spraying; (3) The motion is generally linear, irrespective of the rotational motion of the droplet itself; (4) It may collide with the surrounding environment; and (5) It has a certain appearance state.
To realize the visualization of the spray in a 3D virtual environment, a particle system is used to simulate the mist cone. The system is primarily used to simulate the generation and display of a large number of tiny substances moving or changing based on certain rules on a computer[10]. Each particle in the system has its own set of properties that are updated over time; these properties include the life cycle, velocity, acceleration, color, and position. The shape and position of the particle emission system are set to conical and the nozzle of the spray gun, respectively. The radius and cone angle of the particle emission cone are set at the same time. The life cycle of the particles must be adjusted based on the spraying distance and the speed of the particles. If the life cycle of the particles is too long, a considerable amount of memory and GPU consumption shall occur, and if it is too short, the particles will not be able to collide with the fuselage to complete the spraying. The particles can only disappear after colliding with the fuselage. By setting the mist cone parameters, the spray visualization scene is illustrated in Figure 4.
The format instruction extracted from the spray data contains information of the spraying process status that includes the start and pause time of the spray and the color and cone angle of the mist cone. In the particle system, the on and off status of the particle emission, the color of the particle, and the radius and angle of the cone in the conical emission shape can be controlled. If there is a one-to-one correspondence between the extracted spray state information and the properties of the particle system, the data can be effectively used to control the spraying process. In the 3D virtual environment, the mist cone consists of a particle system, and through the collision of each particle with the fuselage model, the point of collision is obtained via collision detection, and the color of the collision point is changed by using vertex coloring method to realize the visualization of the spraying process of the fuselage.
3.2 Online evaluation and optimization of spraying operation process
3.2.1   Online evaluation of spraying operation process
To ensure that the quality of the spraying is good, it is necessary to evaluate the quality efficiently. Traditional process of manual spraying primarily relies on the experience of the operators to observe the painted surface and identify and solve the problems in time. Therefore, in the automatic spraying process through robots, it is urgent to monitor the quality of spraying online for preventing large-scale spraying accidents. Spray uniformity is a key indicator to characterize the quality of spray. The use of online calculation of paint coating thickness is proposed to quantify the uniformity of spraying and to construct an online evaluation model of spray uniformity. The specific modeling process is detailed below.
The coating thickness Ha(t) of a skin at any time is calculated as follows:
Ha(t)=f×(t-t0 )/Spa (1)
Then, a single skin uniformity online monitoring model is established based on eq. (1):
ΔHa(t)=| Ha(t)-H |,ΔHa(t)≤K 1
σHa (t)=STDEVP(Ha (t), Ha (tN), Ha (t2N),……, Ha (t0 )), σHa (t)≤K 2
ΔFHa (t)=FitHa(t), ΔHa(t-N),ΔHa(t2N),……,ΔHa(tMN)), |ΔFHa’(t) |≤K 3
Here, ΔHa(t) is the deviation between the coating thickness and the design thickness H at any point in time. The deviation must be congruent with the design value K 1 that is the basic requirement for spraying; σB(t) is the fluctuation of the coating thickness that characterizes the stability of the spray quality; σB(t) must be less than K2 that is obtained from a series of process experiments; ΔFHa (t) is a linear fitting function of 0‒t; ΔFHa’(t) is the differentiation of ΔFHa (t); ΔFHa’(t) characterizes the variation trend of the spray thickness, theoretically, and ΔFHa’(t)=0 and |ΔFHa’(t)| must be less than K3 based on the process test. In the above model, N is the sampling frequency. To reduce the online calculation of online fitting, based on the actual needs of the process, MN is typically 600s. For all sprayed skins at any point in time, the thickness Hb (t) is calculated as follows:
Hb(t)=f×(tt0 )/Sa (2)
Based on eq.(2), a full-aircraft spray uniformity online monitoring model is established:
ΔHb (t)=|Hb (t)‒H |,ΔHb (t)≤K 1
σHb (t)=STDEVP(Hb (t), Hb (tN), Hb (t2N), ……, Hb (t0)),σHb (t) ≤K 2
ΔFHb (t)=FitHb (t), ΔHb (tN),ΔHb (t2N), ……,ΔHb (t-MN)), |ΔFHb’(t) |≤K 3
The parameters of the full-aircraft spray uniformity online monitoring model have the same meanings as those of the single skin spray uniformity online monitoring model. The sampling frequency is larger, and MN is typically three times that of the single skin. In addition to considering spray uniformity as a quantitative indicator of the spray quality, it is also necessary to monitor the key parameters of the spray quality online. The distance and angle between the spray robot nozzle and the aircraft surface significantly affect the spray quality. In theory, the distance between the spray head and the fuselage must be consistent, and the angle must be perpendicular to the surface of the aircraft. In the virtual environment, detection is performed by emitting radiation. The model of the robot nozzle is cylindrical. Starting from the center of the bottom, the vertical line is perpendicular to the bottom surface, and the direction is outward, thereby forming a ray. During the spraying operation, regardless of the posture of the nozzle, the ray is always perpendicular to the plane of the nozzle, thereby intersecting the collision model of the fuselage. The aircraft collision model comprises a mesh of the simplified body. When the ray intersects the model, it is equivalent to calculating the intersection of the straight line and the triangle. The distance between the collision point and the base point of the ray is the real spraying distance Da. The true spray angleαis calculated by multiplying the vector of the ray and the normal vector of collision point with the skin. By monitoring the value of Da and α online, the warning shall timely raise an alert when the threshold is exceeded.
3.2.2   Online optimization of spraying operation process
Certain factors cannot be completely considered in the planning stage of the spraying operation; thus, the actual execution of the spraying operation always has a gap with the ideal state. For example, the planning of the spraying path t and the spraying parameters are done under ideal conditions, but the real aircraft can suffer from deformations in different areas owing to its own weight during the assembly process. In addition, the shop floor environment, the robot motion accuracy, the paint properties, etc., shall affect the final spray quality. Therefore, it is necessary to comprehensively judge the abnormality of spraying through the total-factor online monitoring model and quality online monitoring model. Then, the system automatically analyzes and selects the process parameters in the knowledge base according to the analyzed result and the characteristic attributes of the current spraying object (e.g., to adjust the injection distance, spray angle, paint flow rate, spray speed, and other parameters for timely online optimization). Online optimization of the spray operation process is detailed in Figure 5.
3.3 Multi-robot efficient co-simulation in large scenes
In the actual spraying operation, the collision among the spray robot, the fuselage skin, and the shop floor facilities must be avoided. Because the entire virtual scene model consists of tens of millions of geometric patches, it is impossible to perform multi-robot real-time co-simulation using software tools such as DELMIA[11]. A simplified collision detection model[12] is established in Unity3D to replace the original object geometry model for collision detection. Using the spray robot as an example, the robot consists of multi-joints, and the suitable bounding box model is added to each joint of the robot. Figure 6 shows the bounding box model added to the robot. Because the aircraft model is much more complicated, if the bounding box components are used for collision detection, then the collision detection shall not be accurate enough; thus, the mesh body component is added to the aircraft for collision detection.
Then, a highly efficient collision detection method is required. The octree segmentation method is used to complete the segmentation of the collision model. The nodes of the octree are traversed starting from the root node. If the nodes intersect, the traversals continued, and if they do not intersect, the traversal of the subtree is abandoned to achieve real-time collision detection. Finally, the collision result can be obtained[13]. The results obtained from the collision detection include the time of occurrence, location, and direction of the collision. The system records the collected data and uses it for later analysis of the spray process.
4 Development and application of system
The aircraft automatic spraying digital twin system is developed using the. NET framework, and the underlying spraying process data is collected by the OPC UA standard protocol. The human-computer interaction interface uses virtual reality technology to establish a 3D virtual scene while providing user friendly operations such as rotation, positioning, and scaling. Using the particle system to develop the spraying visualization model, the spraying process is rendered in real time based on the spraying process parameters. The collision detection algorithm is used to detect the interference between the robot and the parts of the aircraft in real time. The primary functions of each module are detailed in Figure 7.
The virtual-real mapping system of aircraft automatic spraying uses WPF[14] to integrate the system function modules, and uses the Unity3D engine to realize the 3D visualization. The system integrates equipment management, human-computer interaction, visual display, logic calculation, and spray process knowledge base module to realize timely communication among the modules through the messaging mechanism. The platform visualization interface is presented in Figure 8.
5 Conclusion
This study aimed at addressing the lack of closed-loop feedback optimization enabling tool in the aircraft automatic spraying system; a digital twin model of the system was proposed. The key technology was studied systematically. The full-factor information model and spray visualization model of aircraft automatic spraying operation were constructed. The online virtual-real mapping was realized the OPC UA protocol. The online evaluation and closed-loop optimization of the spraying process were realized based on knowledge engineering. The collision detection method for tens of millions of virtual spraying particles in virtual scenes was studied. The practical application verified that the digital twin technology of the aircraft automatic spraying system can considerably improve the planning efficiency and quality of the spraying operation.



Wang G L, Wu D, Chen K. Current status and development trend of aviation manufacturing robot. Aeronautical Manufacturing Technology,2015, 10: 26–30


Miao D J, Wu L, Xu J, Chen K, Xie Y, Liu Z. Automatic spraying robot system for aircraft surfaces and spraying operation planning. Journal of Jilin University (Engineering and Technology Edition), 2015, 45(2): 547–553 (in Chinese) DOI:10.13229/j.cnki.jdxbgxb201502031


Tao F, Cheng J F, Qi Q L, Zhang M, Zhang H, Sui F Y. Digital twin-driven product design, manufacturing and service with big data. The International Journal of Advanced Manufacturing Technology, 2018, 94(9/10/11/12): 3563–3576 DOI:10.1007/s00170-017-0233-1


Grieves M, Vickers J. Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems. In: Transdisciplinary Perspectives on Complex Systems. Switzerland, Springer, Cham, 2017, 85–113 DOI:10.1007/978-3-319-38756-7_4


Tao F, Sui F Y, Liu A, Qi Q L, Zhang M, Song B Y, Guo Z R, Lu StephenC Y, Nee A Y C. Digital twin-driven product design framework. International Journal of Production Research, 2019, 57(12): 3935–3953 DOI:10.1080/00207543.2018.1443229


Bilberg A, Malik A A. Digital twin driven human–robot collaborative assembly. CIRP Annals, 2019, 68(1): 499–502 DOI:10.1016/j.cirp.2019.04.011


Tao F, Zhang M, Cheng J, Qi Q L. Digital twin workshop: a new paradigm for future workshop. Computer Integrated Manufacturing Systems, 2017, 23(1): 1–9 (in Chinese) DOI:10.13196/j.cims.2017.01.001


Uhlemann Thomas H J, Lehmann C, Steinhilper R. The digital twin: realizing the cyber-physical production system for industry 4.0. Procedia CIRP, 2017, 61: 335–340 DOI:10.1016/j.procir.2016.11.152


Wan J F, Chen B T, Imran M, Tao F, Li D, Liu C L, Ahmad S. Toward dynamic resources management for IoT-based manufacturing. IEEE Communications Magazine, 2018, 56(2): 52–59 DOI:10.1109/mcom.2018.1700629


Messaoudi F, Simon G, Ksentini A. Dissecting games engines: The case of Unity3D. In: 2015 International Workshop on Network and Systems Support for Games (NetGames). Zagreb, Croatia, IEEE, 2015DOI:10.1109/netgames.2015.7382990


Zhao L Z, Zhang Y H, Wu X H, Yan J H. Virtual assembly simulation and ergonomics analysis for the industrial manipulator based on DELMIA. In: Proceedings of the 6th International Asia Conference on Industrial Engineering and Management Innovation. Paris, Atlantis Press, 2015, 527–538 DOI:10.2991/978-94-6239-148-2_51


Liu Y X. Research and application of collision detection technology based on hybrid bounding box in virtual simulation of industrial robot. Guangdong: Guangdong University of Technology, 2017(in Chinese)


Liu X P, Weng X Y, Chen H, Cao L. An improved algorithm for octree-based exact collision detection. Journal of Computer Aided Design Computer Graphics, 2005, 17(12): 2631–2635(in Chinese)


Macdonald M. Pro WPF 4.5 in C#, Windows Presentation Foundation in. 2012