Adv Search
Home | Accepted | Article In Press | Current Issue | Archive | Special Issues | Collections | Featured Articles | Statistics

2019,  1 (2):   185 - 200   Published Date:2019-4-20

DOI: 10.3724/SP.J.2096-5796.2018.0012
1 Introduction 1.1 Temporal ventriloquism and perceptual organization 1.2 Haptic illusions 1.3 The space, direction and number perception of haptic events 2 Wearable haptics3 Discussion4 Conclusion


Multisensory lab based in Peking University, has carried out basic studies in multisensory space and time processing, intersensory binding and haptic/tactile perception. We exploited a typical paradigm of multisensory illusion-temporal ventriloquist effect and applied it in a wide range of multisensory interactions (mainly focused on temporal processing). In this work, we summarized how the tactile stimuli were exploited to compose tactile cues and as tactile apparent motion to interface with other sensory stimuli (visual and auditory stimuli) to examine the underlying perceptual organization in a multisensory context. Moreover, we introduced two examples of wearable haptic/tactile perception in our lab, by using two customized tactile devices and discussed the potential applications in this field.


1 Introduction
1.1 Temporal ventriloquism and perceptual organization
In a multisensory world, perceptual illusions such as the distortions of spatial localization and temporal illusions between cross-modal events are not exceptions but rules. We receive signals from different sensory channels in everyday life. Our brain builds a coherent representation to discriminate between the signals which come from the same object and the signals which are initially segmented[1]. Perceptual organization is the process which converts the complex sensory information into coherent units, so that we can easily identify the objects in the environment[2,3]. It was first proposed by Gestalt psychologists who suggested that perceptual organization was composed of two parts: grouping and segregation[4,5,6,7]. Many factors have influenced our ability to organize information, such as time[8,9,10], space[11,12] and featural relationships[13]. In temporal ventriloquism effect, the perceived time onsets/time intervals of target events-visual events (with lower temporal resolutions) could usually be biased with the presence of the distractors-auditory stimuli or tactile events (with higher temporal resolutions) when all the events are within the presumed temporal window[14]. By exploiting the temporal ventriloquism effect, we have implemented empirical psychophysical studies to show that the ambiguous percept of visual apparent motion could be resolved with the well-designed (alignments) of auditory inputs[15,16,17,18] and tactile inputs[19,20]. In telepresence, we showed that synchronous perception of cross-modal (visuo-haptic) events was very sensitive to the visual packet loss rate. The packet loss caused the impression of time delay and influenced the perception of the subsequent events. The perceived time was also influenced by the communication delay, which caused time to be slightly overestimated[21].
With those pieces of evidence, we have demonstrated the basic perceptual (temporal) organization principles in determining the outcomes (with potential perceptual bias) of cross-modal interactions. Typically, paradigm of cross-modal apparent motion (modulated by perceptual groupings) has been adopted[13,22]. For example, we demonstrated the auditory capture effect on tactile apparent motion by using long range tactile apparent motion in which we presented and alternated two tactile stimuli on both middle-fingers and aligned paired auditory beeps to the taps. One beep was always given in synchrony with one tap, but the other beep could lead or lag the presence of the other tap in the SOA (stimulus onset asynchrony) of 0-75ms. The two SOAs were found to produce opposite modulations of apparent motion, leading to dominant directional perception (leftward or rightward motion). This modulation effect indicates an influence of cross-modal grouping. Moreover, when only odd-numbered, but not even-numbered, taps were paired with auditory beeps, this configuration abolished the temporal-capture effect and, instead, a dominant percept of apparent motion from the audiotactile side to the tactile-only side was observed independently of the SOA variation (Figure 1). These findings suggest that asymmetric cross-modal grouping leads to an attentional modulation of apparent motion, which inhibits cross-modal temporal-capture effects[23].
1.2 Haptic illusions
Studies of haptic/tactile perception are challenging, compared with the psychophysical studies in other sensory channels including visual and auditory modalities. Compared to visual and auditory channels, illusions in haptic domain were less than those in the former. Indeed, in everyday life we were exposed to a very few haptic illusions[24].
Recent research focused on the illusions in which we perceive the virtual environments or real world via haptic interfaces[25,26]. One prominent reason is the complexity to design the suitable device/stimuli to accommodate and fulfill the research purposes by imposing the stimuli to different surfaces (such as finger, face and trunk/torso) of human body, in this direction, the compliances of the stimuli are critical and the rendering of appropriate stimuli are largely constrained by the materials and working mediums of the traditional haptic devices. With the burgeoning technique in virtual reality and marriage with the neuroimaging recording[27], the haptic devices that are compatible with the magnetic-free environment are hardly available. On the other hand, more interesting and sophisticated illusions have been observed in tactile modality, hence the investigating of the underpinning of neurocognitive mechanism poses a challenge to the academic circles. Among the illusions, three of them are conspicuous and have received wide attention. The funneling illusion, describes a phantom sensation midway between multiple stimuli when they are presented simultaneously at adjacent locations on the human skin[28,29]. In this context, spatial displacement of a vibrotactile stimulus can be deployed for information display. By exploiting "funneling illusion", in psychophysical experiments, human spatial perception ability on the human forearm for stationary and moving vibrotactile stimuli has been revealed[28,29,30]. The mislocalization of touches has also been robustly found in the Cutaneous rabbit effect, which was firstly discovered in the Cutaneous Communication Laboratory at Princeton University when Frank Geldard and his colleagues were testing cutaneous perception[31,32]. In detail, cutaneous rabbit illusion (also known as cutaneous saltation and sometimes the cutaneous rabbit effect or CRE) is a tactile illusion evoked by tapping two or more separate regions of the skin in rapid succession. The illusion is most readily evoked on regions of the body surface that have relatively poor spatial acuity, such as the forearm. A rapid sequence of taps delivered first near the wrist and then near the elbow creates the sensation of sequential taps hopping up the arm from the wrist towards the elbow, although no physical stimulus was applied between the two actual stimulus locations. Bayesian modeling work indicated that weaker stimuli results in greater saltation[33], but the direct neural mechanism of this illusion is far from clear[34]. The last one and important example is the flash-lag effect in tactile domain. In one psychophysical study, Chen reported an analogous tactile flash-lag illusion, compared with visual flash lag effect[35,36], in which a sequence of tactile taps with increasing (or decreasing) intensity was presented on one finger. The temporally-middle tap in the sequence was perceived to be weaker in both cases than a synchronous single tap on another finger. This illusion could be accounted by a mechanism of temporal compensation and tactile masking[37].
1.3 The space, direction and number perception of haptic events
The basic psychophysical studies on tactile/haptic spatial perception and numerosity perception have been implemented in previous studies[38,39]. In general, the perception of direction and location of tactile events on fingers was modulated by the roles of attentional shifts[40,41]. In a most recent study, we revealed the roles of attentional shifts and attentional reengagement in resolving the spatial compatibility effect in tactile Simon-like tasks[42]. The Simon effect refers to the acceleration of choice responses when the target position and response location are consistent compared with the cases when they are inconsistent, even if the target position is task-irrelevant. For the tactile direction task, the spatial compatibility effect was absent in the focused-attention condition but maintained in the divided-attention condition (Figure 2). For the tactile localization task, this pattern was reversed. We propose that the temporal course of resolving conflicts between spatial codes during attentional shifts, including attentional reengagement could explain the tactile Simon-like effect we observed.
In a recent study, we asked whether the same attentional mechanism subserves subitizing (with number of items less than or equal to 4) and numerosity estimation (with number of items equal to or larger than 5) in the tactile modality. Using tactile Braille displays, participants completed the unisensory task with focused attention, unisensory task with divided attention by reporting the numbers of pins across the upper and lower area of their left index fingers, in addition to reporting the number of tactile pins on their right index fingers, and cross-modal task with divided attention by reporting the number of tactile pins and compared the numbers of visual dots across the upper and lower part of a (illusory) rectangle that overlaid the tactile stimuli (Figure 3). Importantly, the precision of tactile subitizing, in the presence of a visual distractor, was correlated with the capacity of visual working memory, not of tactile working memory. We showed that revealed that tactile numerosity perception is accounted for by amodal attentional modulation yet by differential attentional mechanisms in terms of subitizing and estimation[43].
2 Wearable haptics
The human body surface has been considered as an additional means of presenting information using vibrotactile display devices. Wearability could significantly increase the usage of haptics in our everyday life[44,45]. In the following paragraphs, we presented recent two studies of wearable haptic displays-on the dorsum (with vibrotactile stimuli) and on the forearm (with stimuli induced by piezoelectric ceramic transducers, PZTs), to show how human observers performed the basic perceptual tasks including the directional perception, numerosity judgment and oddball detection with the wearable tactile display.
In previous studies, the haptic stimulation and physiological feedback system (HSPFS) has been prototyped to deliver the appropriate haptic cueing information, as well as to monitor the physiological signals[46,47], and hence to improve the human performance in various applications such as (tele) rehabilitation[48,49], industrial haptic design[50,51] and wearable sensing on psychophysiological states[52,53,54]. In one of the present studies, we built a haptic stimulation and physiological feedback system (HSPFS) (Figure 4). This system is wearable and mainly consists of a microcontroller (STM32, STMicroelectronics), an ECG Analog Front End (ADS1292, Texas Instruments, Dallas, USA), 4×4 flat vibration motors (universal 10×2.7mm flat vibration motor) array at human back and a Matlab-based user interface. The user interface communicates with the microcontroller through a couple of Bluetooth wireless modules. The ECG AFE is used to acquire the single lead ECG signal continuously with 125Hz sampling frequency and deliver the digital signal to the microcontroller through SPI (i.e., Serial Peripheral Interface). The microcontroller extracts heart rate from every three-second ECG signal and sends it to the user interface for further analysis. Besides receiving the heart rate, the user interface is also used to program the individual vibration parameters for each of the 16 flat motors, including start-up time and shut-down time. The microcontroller decodes these parameters and drives the motors to vibrate in order. Thus the 4×4 motor array can be controlled flexibly and apply various forms of stimulation sequences on the human back. The latency from sending command to startup vibration is less than 10ms. The minimum resolution of the duration between start-up time and shut-down time is 250ms, which is mainly limited by the slow response time of the adopted vibration motors.
We built another wearable tactile device which contained 6-channel piezo benders (PZB) (Figure 5). The haptic stimulation system mainly includes a user interface, a microcontroller (STM32, STMicroe-lectronics) and six piezo haptics drivers (DRV266x, Texas Instruments, Dallas, USA) to respectively control the six connected piezo actuators (PL112, PI Ceramic, Thuringia, Germany). The user interface based on Matlab (The MathWorks, Inc., Natick, MA) is used to program command that includes the intensity, duration, and start-up time of the vibration for each channel and also sends the command to the microcontroller through UART (i.e., universal asynchronous receiver-transmitter). When a complete command is received, the microcontroller decodes it and configures the piezo haptic drivers through IICs (i.e., Inter-Integrated Circuit). Once the configuration is completed, the drivers produce signals to vibrate the piezo actuators following the vibration parameters of the command. The latency from sending command to startup vibration is less than 10ms. The maximum vibration intensity (i.e., blocking force) of the piezo actuator can reach ±(2±20%)N. While the present vibration frequency is fixed at 125Hz, the minimum vibration duration can reach 8ms (i.e., 1/125ms), together with the 8ms-based configuration of start-up time on each channel, enabling flexible control for fine haptic stimulation.
Study 1: Vibration detection with HSPFS
In study 1, we used HSPFS to render different patterns of vibrations (Figure 6). We implemented many tests and hereby picked up one test of oddball detection. The purpose of the experiment was to examine how target behavior of detecting the oddball vibration was affected by the preceding workload. Participants wore a T-shirt with a tactile array containing 16 flat mobile phone vibrators (Figure 7). For the low-workload condition, they reported the oddball vibration (duration of 500ms) from the other three vibrations (durations of 1000ms each), the number of the trials was 20. All the task-relevant four vibrators were located at the corners in the display. For the high-workload condition, participants received a test of reporting the number of simultaneously presented visual dots and the tactile stimuli from the tactile array, they then attended the subsequent test of detecting the oddball stimuli as shown in the low-workload condition. For the high-workload condition, participants should receive 8 trials of the numerosity perception task, they then did the subsequent 4 trials for oddball detection.
In the numerosity perception task, for a typical trial, after a fixation point (lasted 500ms), 6 to 9 white round dots (with durations of 150ms) appeared on the screen (with viewing distance of 50cm). At the same time, 6-9 tactile dots (with durations of 1000ms) were triggered from the tactile display. Participants were encouraged to report the total numbers of the stimuli including both visual dots and tactile vibrations
To measure the efficacy of task load, we recorded real-time the heart rates with customized microcontroller and circuits (chips) and attached single-lead thoracic electrode, and used a Real-Time QRS Detection Algorithm to visualize the data of heart rates[55]. The results showed that for the current setup, the correct rates for reporting the oddball stimulus was 0.575(SE=0.053) in low taskload condition and was 0.267(SE=0.044) in high taskload condition, t(5)=6.290, p=0.001. However, we did not observe the difference in heart rates across the two conditions, the mean heart rate was 74(SE=6) in low taskload condition and was 70(SE=5) in high taskload condition, t(6)=1.496, p=0.195.
Study 2: Vibration detection and discrimination by PZB
Using PZB, we implemented three experiments. The tactile stimuli were given by four pieces of PZTs which encode the direction of tactile apparent motion, the numbers of tactile stimuli and the ‘oddball’ stimuli. Accordingly, we launched three experiments to examine the directional perception (clockwise and anti-clockwise, Experiment 1), the number perception (Experiment 2) as well as the detection of oddballs (Experiment 3).
Experiment 1: Direction discrimination
Ten college students (ages between 18 and 25 years old, right-handed 5 males) took part in Experiment 1. All participants reported normal tactile sensation and had no history of neurological diseases. They also self-reported right-handed, and were naïve to the purpose of the experiment. They were paid for their time, and gave written informed consent before the experiment. The study was conducted in accordance with the principles of Declaration of Helsinki and was approved by the human subject review committee of Peking University.
A customized device containing 4 channels of PZT actuators was used. The PZT benders were attached on the left forearm and the distance from the center of the benders to the border of wrist was 5cm. The four benders were located and fixed on the dorsal (up), volar (down), left and right sides of the forearm, with equivalent distances between adjacent benders. Participants sat before a table and the viewing distance was 50cm from the eye-level to the center of the screen. Each participant rested his/her palm of the left hand on the table, but the forearm was unsupported to prevent potential damage to the benders (Figure 8).
In Experiment 1, four benders produced the vibrations consecutively. Each bender vibrated one time with the duration of 8ms. The inter-stimulus-interval (ISI) was picked from one of the seven levels (8ms, 16ms, 32ms, 64ms, 96ms, 128ms or 160ms). The stimuli were presented in clockwise or counterclockwise. The initial vibration (randomly selected from the four positions) was repeated twice so that we largely prevented the response bias based on the first and last vibrations. In a typical trial, a fixation point of “+” was presented on the screen for 500ms, and the vibrations started after the gaze point disappeared. When all of the tactile stimuli were presented over, the instruction was given on the screen to prompt the perceptual response as ‘clockwise’ or ‘counterclockwise’. They pressed key “1” for clockwise and “2” for counterclockwise. The next trial was initiated when the press was issued (the inter-trial interval, i.e., ITI=500ms) (Figure 9). Before the formal experiment, participants received practice to be familiar with the task. Stimuli presentation and data collection were implemented by computer programs designed with Matlab and Psychophysics Toolbox extensions[56,57,58].
The proportions of reporting the direction of counterclockwise across seven ISIs were fitted to the psychometric curve using a logistic function[59,60]. The transitional threshold, that is, the point of subjective equality (PSE) at which the participant was likely to report the two percepts (clockwise and counterclockwise) equally, was calculated by estimating 50% of reporting of counterclockwise on the fitted curve. The just noticeable difference (JND), an indicator of the sensitivity of discrimination, was calculated as half of the difference between the lower (25%) and upper (75%) bounds of the thresholds from the psychometric curve (Figure 10). The difference of PSE was not significant, F(3,15)=1.025, p=0.409. The result implied that the judgment of direction was not affected by the starting point. The difference of JND was also not significant, F(3,15)=0.854, p=0.486 (Figure 11).
Experiment 2: Numerosity judgment
The participants in Experiment 1 also took part in Experiment 2 and Experiment 3. The procedure of Experiment 2 was similar to the one in Experiment 1, except that only 1, 2, 3 or 4 vibrations were generated from these bender (one bender at most vibrated once, with duration of 8ms). Participants were encouraged to report the number of tactile stimuli presented at the same time. Due to the different combinations of the 1-4 vibrations we balanced the repetitions of the each type of vibrations (1-4). For one-bender vibration, there were 60 times, 90 with 2-bender vibrations, 60 with 3-bender vibrations and 30 with 4-bender vibrations. All the stimuli combinations were presented randomly. The timing protocols of the experiment was the same as in Experiment 1, but the task now was to report the number of tactile stimuli presented. Participants issued the responses by clicking the appropriate numbers (1,2,3,4) to correspond the number of perceived tactile stimuli. Before the formal experiment, participants received a practice to be familiar with the task.
The mean correct rates were 0.788±0.041 (M±SE), 0.551±0.025, 0.330±0.041 and 0.110±0.059 for numbers 1-4 respectively. Overall, there was significant differences between the correct rates across the four conditions, F(3,27)=115.226, p<0.001. Further comparisons indicated the rate decreased from the numbers 1 to 4, ps<0.01.For one vibration condition, the correct rate was higher in ‘down’ (correct rate: 0.873±0.039 ) than the one in ‘right’ (0.740±0.052) condition, p<0.05. However, for the two vibrations and three vibrations, the difference of correct rates was not significant, F(5,45)=1.18, p=0.334 and F(3,27)=0.6, p=0.620 (Figure 12).
For the four vibrations, participants nearly had a floor effect. They reported the numbers as 1,2 or 3. They reported more “2” vibrations (frequencies: 13.838±2.888) than did “1” (4.571±1.395 ) p=0.028.
Experiment 3: Oddball detection
The procedure of Experiment 3 was similar to the one in Experiment 1, except that all the four benders vibrated but one of them was the ‘oddball’. In each trial, the four benders started to vibrate at the same time and lasted 1440ms. However, one of the benders vibrated with an on (160ms) and off (160ms, no vibration) cycle. The intensities of all the different vibrations were calibrated in a pilot test to ensure the perceived intensities of the four stimuli were the same for each individual. Participants were encouraged to report which bender generated the ‘oddball’ by issuing the following mappings: pressing “1” for the dorsal (up) bender, “2” for the right one, “3” for the volar (down) and “4” for the left one. Likewise, participants received a practice before the formal experiment (Figure 13).
The main effect of position was significant, F(3,27)=3.257, p=0.019<0.05. Further analysis indicated that the correct rate at ‘right’ position (correct rate: 0.405±0.069 (M±SE)) was higher than the one at ‘up’ (correct rate: 0.265±0.067 (M±SE)), p<0.01. The correct rate at down (correct rate: 0.415±0.070 (M±SE)) was higher than the one at ‘up’, p<0.05 (Figure 14).
3 Discussion
Previous studies on tactile display have looked into the numerosity perception[39,43,61,62,63,64] and tactile search[65,66]. However, the similar investigations with wearable (small) devices have been less addressed. Under the broad context of multisensory interaction, we presented two typical work using wearable haptics. The prototypes of the two wearable devices (with normal vibrations and PZT actuators) and the psychophysical tasks revealed that it is applicable to use those small wearable haptics devices to monitor the vital signs of the operators under some special working conditions and henceforth send out the tactile-encoded instructions to improve the task performance. The design of PZT actuators with the relevant tasks of directional discrimination, numerosity perception and oddball detection, indicates a convenient solution to provide effective tactile communication channels for streamlining the direct and information-protected human-machine (computer) interaction. With that said, the current studies suggest some characteristic findings (such as with PZT actuators) as well as some limitations. First, with the forearm as the effector, tactile sensitivities across different position of the forearm were not equal, in which the volar side was relatively more sensitive. When we are trying to use one or two vibrations as main communication channels, this position should be considered as the high priority to put up with tactile sensors. Secondly, with more tactile benders (vibrations), tactile masking effect has been observed when the individual vibratory stimuli were near in spatial locations and (or) close in timing. Those spatiotemporal factors will constrain the effectiveness of putting more tactile sensors on the human body effectors. Moreover, we should take into consideration of the working memory capacity of holding less than 4 items of stimuli. It was also mentioned in previous studies that the correct rate of tactile numerosity judgments decreased if the number tactile actuators on the body surface was increased[39,63]. Researchers also found a moderate increase in response time when the number of stimuli was up to four by using one hand[67]. Evidence also showed that the tactile subitizing range was up to 2 and the counting range was from 3 and on[68]. Those findings helped to explain why the correct rate of reporting three or four vibrations decreased in our current studies. With that said, there was still a question that how much tactile information could be tracked in parallel since it was affected by many factors, such as intensity, duration, the spatial distance of the tactors and individual difference[69]. Therefore, in future studies and potential applications, we should strike a balance to optimize the number of stimuli and the associated spatiotemporal parameters. For the complicated (combinations) of the tactile array, a relevant and appropriate perceptual/cognitive training is strongly recommended. Last but not least, our performance of perceptual discrimination with tactile array was influenced by a number of other factors, including the taskload. Taskload would influence the efficiency of task performance and human optimal behavior. It could be measured in a variety of ways, such as questionnaires (NASA-TLX), heart rate, electrocardiogram (ECG), electroencephalograms (EEG) and pupil diameters[70,71,72]. High taskload often entails dividing attention between complex or multiple tasks. For instance, after driving vehicles continuously for four hours on a highway, the correct discrimination responses and detection probability of participants significantly decreased[72]. To improve the performance, a decent taskload would be helpful. The performance would be hampered when human operators are under stress. However, in the worst case, we could still improve our perceptual expertise by receiving enough task-relevant training, and adaptively maintain our performances.
4 Conclusion
With the framework of wearable haptics and in a multisensory context, the PZT actuators system we developed can be applied and adapted to nearly any part on the body, and we can remotely control the tactile array through wireless network or bluetooth. Through well-defined tactile instructions and cueing information, including motion direction and number, people can obtain task-relevant information though tactile channels. The integration of wearable haptics with simultaneous monitoring of physiological signals could intelligently assists and adjusts/optimized human performance. The miniaturization and portability of the haptic system could be promisingly applied in many fields, such as education, sports, aviation, telerehabilitation and military operations.



Shams L, Beierholm U R. Causal inference in perception. Trends in Cognitive Sciences, 2010, 14(9): 425–432 DOI:10.1016/j.tics.2010.07.001


Kramer A F, Jacobson A. Perceptual organization and focused attention: The role of objects and proximity in visual processing. Perception & Psychophysics, 1991, 50(3): 267–284 DOI:10.3758/bf03206750


Wagemans J, Elder J H, Kubovy M, Palmer S E, Peterson M A, Singh M, von der Heydt R. A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychological Bulletin, 2012, 138(6): 1172–1217 DOI:10.1037/a0029333


King A J, Calvert G A. Multisensory integration: perceptual grouping by eye and ear. Current Biology, 2001, 11(8): R322–R325 DOI:10.1016/s0960-9822(01)00175-0


Gepshtein S, Kubovy M. Stability and change in perception: Spatial organization in temporal context. Experimental Brain Research, 2005, 160(4): 487–495 DOI:10.1007/s00221-004-2038-3


Sanabria D, Soto-Faraco S, Chan J S, Spence C. When does visual perceptual grouping affect multisensory integration? Cognitive, Affective, & Behavioral Neuroscience, 2004, 4(2): 218–229 DOI:10.3758/cabn.4.2.218


Sanabria D, Soto-Faraco S, Chan J, Spence C. Intramodal perceptual grouping modulates multisensory integration: Evidence from the crossmodal dynamic capture task. Neuroscience Letters, 2005, 377(1): 59–64 DOI:10.1016/j.neulet.2004.11.069


Fendrich R, Corballis P M. The temporal cross-capture of audition and vision. Perception & Psychophysics, 2001, 63(4): 719–725 DOI:10.3758/bf03194432


Vroomen J, de Gelder B. Temporal ventriloquism: sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception and Performance, 2004, 30(3): 513–518 DOI:10.1037/0096-1523.30.3.513


Freeman E, Driver J. Direction of visual apparent motion driven solely by timing of a static sound. Current Biology, 2008, 18(16): 1262–1266 DOI:10.1016/j.cub.2008.07.066


Slutsky D A, Recanzone G H. Temporal and spatial dependency of the ventriloquism effect. Neuroreport, 2001, 12(1): 7–10 DOI:10.1097/00001756-200101220-00009


Alais D, Burr D. The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 2004, 14(3): 257–262 DOI:10.1016/j.cub.2004.01.029


Roseboom W, Kawabe T, Nishida S. Direction of visual apparent motion driven by perceptual organization of cross-modal signals. Journal of Vision, 2013, 13(1): 6 DOI:10.1167/13.1.6


Chen L H, Vroomen J. Intersensory binding across space and time: A tutorial review. Attention, Perception, & Psychophysics, 2013, 75(5): 790–811 DOI:10.3758/s13414-013-0475-4


Chen L H, Zhou X L. Capture of intermodal Visual/Tactile apparent motion by moving and static sound. Seeing and Perceiving, 2011, 24(4): 369–389 DOI:10.1163/187847511x584434


Chen L H, Zhou X L, Müller H J, Shi Z H. What you see depends on what you hear: Temporal averaging and crossmodal integration. . Journal of Experimental Psychology: General, 2018, 147(12): 1851–1864 DOI:10.1037/xge0000487


Jiang Y S, Chen L H. Mutual influences of intermodal Visual/Tactile apparent motion and auditory motion with uncrossed and crossed arms. Multisensory Research, 2013, 26(1/2): 19–51 DOI:10.1163/22134808-00002409


Shi Z H, Chen L H, Müller H J. Auditory temporal modulation of the visual Ternus effect: The influence of time interval. Experimental Brain Research, 2010, 203(4): 723–735 DOI:10.1007/s00221-010-2286-3


Chen L H, Shi Z H, Müller H J. Influences of intra-and crossmodal grouping on visual and tactile Ternus apparent motion. Brain Research, 2010, 1354: 152–162 DOI:10.1016/j.brainres.2010.07.064


Yiltiz H, Chen L H. Corrigendum: Tactile input and empathy modulate the perception of ambiguous biological motion. Frontiers in Psychology, 2015, 6: 161 DOI:10.3389/fpsyg.2015.00161


Shi Z H, Zou H, Rank M, Chen L H, Hirche S, Muller H J. Effects of packet loss and latency on the temporal discrimination of visual-haptic events. IEEE Transactions on Haptics, 2010, 3(1): 28–36 DOI:10.1109/toh.2009.45


Lyons G, Sanabria D, Vatakis A, Spence C. The modulation of crossmodal integration by unimodal perceptual grouping: A visuotactile apparent motion study. Experimental Brain Research, 2006, 174(3): 510–516 DOI:10.1007/s00221-006-0485-8


Chen L H, Shi Z H, Müller H J. Interaction of perceptual grouping and crossmodal temporal capture in tactile apparent-motion. PLoS One, 2011, 6(2): e17130 DOI:10.1371/journal.pone.0017130


Wang Q, Hayward V. Tactile synthesis and perceptual inverse problems seen from the viewpoint of contact mechanics. ACM Transactions on Applied Perception, 2008, 5(2): 1–19 DOI:10.1145/1279920.1279921


Hogan N, Kay B A, Fasse E D, Mussa-Ivaldi F A. Haptic illusions: experiments on human manipulation and perception of “virtual objects”. Cold Spring Harbor Symposia on Quantitative Biology, 1990, 55: 925–931 DOI:10.1101/sqb.1990.055.01.086


Wang D X, Zhang Y R, Hou J X, Wang Y, Lv P, Chen Y G, Zhao H. Idental: A haptic-based dental simulator and its preliminary user evaluation. IEEE Transactions on Haptics, 2012, 5(4): 332–343 DOI:10.1109/toh.2011.59


Katzakis N, Tong J, Ariza O, Chen L, Klinker G, R B, Röder B. , Steinicke F. Stylo and handifact: modulating haptic perception through visualizations for posture training in augmented reality. In: Proceedings of the 5th Symposium on Spatial User Interaction. Brighton, United Kingdom, ACM, 2017: 58-67 DOI:10.1145/3131277.3132181


Von Békésy G. Neural funneling along the skin and between the inner and outer hair cells of the cochlea. The Journal of the Acoustical Society of America, 1959, 31(9): 1236–1249 DOI:10.1121/1.1907851


Von Békésy G. Funneling in the nervous system and its role in loudness and sensation intensity on the skin. The Journal of the Acoustical Society of America, 1958, 30(5): 399–412 DOI:10.1121/1.1909626


Barghout A, Cha J, Saddik A E, Kammerl J, Steinbach E. Spatial resolution of vibrotactile perception on the human forearm when exploiting funneling illusion. In: 2009 IEEE International Workshop on Haptic Audio visual Environments and Games, 2009, 19–23 DOI:10.1109/HAVE.2009.5356122


Geldard F A. Saltation in somesthesis. Psychological Bulletin, 1982, 92(1): 136–175 DOI:10.1037//0033-2909.92.1.136


Geldard F A, Sherrick C E. The cutaneous “rabbit”: A perceptual illusion. Science, 1972, 178(4057): 178–179 DOI:10.1126/science.178.4057.178


Tong J, Ngo V, Goldreich D. Tactile length contraction as Bayesian inference. Journal of Neurophysiology, 2016, 116(2): 369–379 DOI:10.1152/jn.00029.2016


Blankenburg F, Ruff C C, Deichmann R, Rees G, Driver J. The cutaneous rabbit illusion affects human primary sensory cortex somatotopically. PLoS Biology, 2006, 4(3): e69 DOI:10.1371/journal.pbio.0040069


Nijhawan R. Motion extrapolation in catching. Nature, 1994, 370(6487): 256–257 DOI:10.1038/370256b0


Sheth B R, Nijhawan R, Shimojo S. Changing objects lead briefly flashed ones. Nature Neuroscience, 2000, 3(5): 489–495 DOI:10.1038/74865


Chen L. Tactile flash lag effect: Taps with changing intensities lead briefly flashed taps. In: 2013 World Haptics Conference (WHC). Daejeon, South Korea, 2013, 253–258 DOI:10.1109/WHC.2013.6548417


Gallace A, Tan H Z, Spence C. Tactile change detection. In: First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems World Haptics Conference. 2005, Pisa, Italy, 2005: 12–16 DOI:10.1109/WHC.2005.122


Gallace A, Tan H Z, Spence C. Numerosity judgments for tactile stimuli distributed over the body surface. Perception, 2006, 35(2): 247–266 DOI:10.1068/p5380


Medina J, McCloskey M, Coslett H B, Rapp B. Somatotopic representation of location: Evidence from the Simon effect. Journal of Experimental Psychology: Human Perception and Performance, 2014, 40(6): 2131–2142 DOI:10.1037/a0037975


Salzer Y, Aisenberg D, Oron-Gilad T, Henik A. In touch with the Simon effect *The first two authors contributed equally. Experimental Psychology, 2014, 61(3): 165–179 DOI:10.1027/1618-3169/a000236


Zheng W T, Chen L H. The roles of attentional shifts and attentional reengagement in resolving the spatial compatibility effect in tactile simon-like tasks. Scientific Reports, 2018, 8: 8760 DOI:10.1038/s41598-018-27114-9


Tian Y, Chen L H. Cross-modal attention modulates tactile subitizing but not tactile numerosity estimation. Attention, Perception, & Psychophysics, 2018, 80(5): 1229–1239 DOI:10.3758/s13414-018-1507-x


Bortone I, Leonardis D, Mastronicola N, Crecchi A, Bonfiglio L, Procopio C, Solazzi M, Frisoli A. Wearable haptics and immersive virtual reality rehabilitation training in children with neuromotor impairments. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(7): 1469–1478 DOI:10.1109/tnsre.2018.2846814


Prattichizzo D, Chinello F, Pacchierotti C, Malvezzi M. Towards wearability in fingertip haptics: A 3-DoF wearable device for cutaneous force feedback. IEEE Transactions on Haptics, 2013, 6(4): 506–516 DOI:10.1109/toh.2013.53


Ceballos R, Ionascu B, Park W, Eid M. Implicit emotion communication. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018, 14(1): 1–18 DOI:10.1145/3152128


Schecter S, Lin W, Gopal A, Fan R, Rashba E. Haptics and the heart: Force and tactile feedback system for cardiovascular interventions. Cardiovascular Revascularization Medicine, 2018, 19(6): 36–40 DOI:10.1016/j.carrev.2018.05.017


Demain S, Metcalf C D, Merrett G V, Zheng D Y, Cunningham S. A narrative review on haptic devices: Relating the physiology and psychophysical properties of the hand to devices for rehabilitation in central nervous system disorders. Disability and Rehabilitation: Assistive Technology, 2013, 8(3): 181–189 DOI:10.3109/17483107.2012.697532


Navarro E, González P, López-Jaquero V, Montero F, Molina J P, Romero-Ayuso D. Adaptive, multisensorial, physiological and social: The next generation of telerehabilitation systems. Frontiers in Neuroinformatics, 2018, 12: 43 DOI:10.3389/fninf.2018.00043


Hale K S, Stanney K M. Haptic rendering-beyond visual computing-deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. IEEE Computer Graphics and Applications, 2004, 24(2): 33–39 DOI:10.1109/mcg.2004.1274059


Sato Y, Ueoka R: Investigating Haptic Perception of and Physiological Responses to Air Vortex Rings on a User's Cheek. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Denver, Colorado, USA, ACM, 2017: 3083–3094 DOI:10.1145/3025453.3025501


Taelman J, Adriaensen T, Horst C v d, Linz T, Spaepen A. Textile integrated contactless EMG sensing for stress analysis. In: 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2007, 3966–3969 DOI:10.1109/IEMBS.2007.4353202


Wang L, Loh K J, Chiang W H, Manna K. Micro-patterned graphene-based sensing skins for human physiological monitoring. Nanotechnology, 2018, 29(10): 105503 DOI:10.1088/1361-6528/aaa709


Xu H, Xiang J X, Lu Y F, Zhang M K, Li J J, Gao B B, Zhao Y J, Gu Z Z. Multifunctional wearable sensing devices based on functionalized graphene films for simultaneous monitoring of physiological signals and volatile organic compound biomarkers. ACS Applied Materials & Interfaces, 2018, 10(14): 11785–11793 DOI:10.1021/acsami.8b00073


Pan J P, Tompkins W J. A real-time QRS detection algorithm. IEEE Transactions on Biomedical Engineering, 1985, BME-32(3): 230–236 DOI:10.1109/tbme.1985.325532


Brainard D H. The psychophysics toolbox. Spatial Vision, 1997, 10(4): 433–436 DOI:10.1163/156856897x00357


Kleiner M, Brainard D, Pell D. What's new in Psychtoolbox-3? PERCEPTION. 2007, 36: 1–16 DOI:10.1068/v070821


Pelli D G. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 1997, 10(4): 437–442 DOI:10.1163/156856897x00366


Treutwein B, Strasburger H. Fitting the psychometric function. Perception & Psychophysics, 1999, 61(1): 87–106 DOI:10.3758/bf03211951


Wichmann F A, Hill N J. The psychometric function: I. Fitting, sampling, and goodness of fit. Perception & Psychophysics, 2001, 63(8): 1293–1313 DOI:10.3758/bf03194544


Gallace A, Auvray M, Tan H Z, Spence C. When visual transients impair tactile change detection: A novel case of crossmodal change blindness? Neuroscience Letters, 2006, 398(3): 280–285 DOI:10.1016/j.neulet.2006.01.009


Gallace A, Tan H Z, Haggard P, Spence C. Short term memory for tactile stimuli. Brain Research, 2008, 1190: 132–142 DOI:10.1016/j.brainres.2007.11.014


Gallace A, Tan H Z, Spence C. Multisensory numerosity judgments for visual and tactile stimuli. Perception & Psychophysics, 2007, 69(4): 487–501 DOI:10.3758/bf03193906


Gallace A, Tan H Z, Spence C. Can tactile stimuli be subitised? an unresolved controversy within the literature on numerosity judgments. Perception, 2008, 37(5): 782–800 DOI:10.1068/p5767


Assumpção L, Shi Z H, Zang X L, Müller H J, Geyer T. Contextual cueing: implicit memory of tactile context facilitates tactile search. Attention, Perception, & Psychophysics, 2015, 77(4): 1212–1222 DOI:10.3758/s13414-015-0848-y


Assumpção L, Shi Z H, Zang X L, Müller H J, Geyer T. Contextual cueing of tactile search is coded in an anatomical reference frame. . Journal of Experimental Psychology: Human Perception and Performance, 2018, 44(4): 566–577 DOI:10.1037/xhp0000478


Cohen Z Z, Naparstek S, Henik A. Tactile enumeration of small quantities using one hand. Acta Psychologica, 2014, 150: 26–34 DOI:10.1016/j.actpsy.2014.03.011


Cohen Z Z, Henik A. Effects of numerosity range on tactile and visual enumeration. Perception, 2016, 45(1): 83–98 DOI:10.1177/0301006615614662


Spence C. Multisensory attention and tactile information-processing. Behavioural Brain Research, 2002, 135(1/2): 57–64 DOI:10.1016/s0166-4328(02)00155-9


Jorna P G. Spectral analysis of heart rate and psychological state: A review of its validity as a workload index. Biological Psychology, 1992, 34: 237–257 DOI:10.1016/0301-0511(92)90017-O


Brookings J B, Wilson G F, Swain C R. Psychophysiological responses to changes in workload during simulated air traffic control. Biological Psychology, 1996, 42(3): 361–377 DOI:10.1016/0301-0511(95)05167-8


Recarte M A, Nunes L M. Mental workload while driving: Effects on visual search, discrimination, and decision making. Journal of Experimental Psychology: Applied, 2003, 9(2): 119–137 DOI:10.1037/1076-898x.9.2.119