Direction of Apparent Motion During Smooth Pursuit Is Determined Using a Mixture of Retinal and Objective Proximities

Many studies have investigated various effects of smooth pursuit on visual motion processing, especially the effects related to the additional retinal shifts produced by eye movement. In this article, we show that the perception of apparent motion during smooth pursuit is determined by the interelement proximity in retinal coordinates and also by the proximity in objective world coordinates. In Experiment 1, we investigated the perceived direction of the two-frame apparent motion of a square-wave grating with various displacement sizes under fixation and pursuit viewing conditions. The retinal and objective displacements between the two frames agreed with each other under the fixation condition. However, the displacements differed by 180 degrees in terms of phase shift, under the pursuit condition. The proportions of the reported motion direction between the two viewing conditions did not coincide when they were plotted as a function of either the retinal displacement or of the objective displacement; however, they did coincide when plotted as a function of a mixture of the two. The result from Experiment 2 showed that the perceived jump size of the apparent motion was also dependent on both retinal and objective displacements. Our findings suggest that the detection of the apparent motion during smooth pursuit considers the retinal proximity and also the objective proximity. This mechanism may assist with the selection of a motion path that is more likely to occur in the real world and, therefore, be useful for ensuring perceptual stability during smooth pursuit.

direction opposite to the smooth-pursuit direction was always seen. This observation disagrees with the prediction by retinal proximity (Ullman, 1979), as retinal displacements are the same in both directions. However, it agrees with Terao et al. (2015), except now the motion is apparent motion. To explain the observed phenomenon, we can suggest two possible effects of smooth pursuit on the process of apparent motion. One is that smooth pursuit modulates the strengths of motion signals, enhancing the direction opposite to that of eye movement. The other, which can only be tested with apparent motion, is that the proximity that determines the strength of motion correspondence is computed based not only on the retinal proximity but also on the objective "world" proximity. These two possibilities can be dissociated by investigating the perceived direction of two-frame square-wave gratings at various phase shift angles. In our experiment, we produced these seven phase shifts on the retina, but the patterns were the same on the objective "display" coordinate under the fixation condition. (C) The relationship between objective (top) and retinal (bottom) displacements for the fixation (left) and the pursuit (right) conditions. The white arrows indicate the eye position during fixation and the eye trajectory during rightward pursuit. During fixation, a phase shift of 180 degrees on the display corresponds to a phase shift of 180 degrees on the retina, and the perceived motion is ambiguous. During the pursuit to the right, the phase shift is 0 degree on the display but 180 degrees on the retina, and the dominant percept is a leftward movement direction.
In our experiments, we flashed a square-wave grating, comprising a horizontal array of bars, twice with a brief interval on a dark background (see Figure 1A). We varied the spatial phase of the square-wave grating, which could be described either with retinal coordinates or for the display with the objective coordinates. In either case, the spatial phase is defined as the shift angle of the second square wave from the first one, with the rightward direction being positive. According to the proximity principle of motion correspondence (Ullman, 1979), rightward motion is predominant when the retinal phase shift is smaller than 180 degrees, while leftward motion is predominant when the retinal phase shift is larger than 180 degrees. The frequencies of leftward and rightward motions are balanced when the phase angle is 180 degrees. Figure 1B depicts the seven different phase shifts on the retinal coordinates used in our experiment. We compared the perceived direction for the same retinal phase shift conditions between fixation and pursuit viewing conditions. Note that the objective phase shift angles were matched with the retinal ones under the fixation condition but were 180 degrees out of phase under the pursuit condition (see Figure 1C). Figure 2 depicts how the two hypothetical effects of pursuit on apparent motion will affect the probability of the perceived motion direction, as a function of the retinal displacement. The black lines in Figure 2A and B indicate the predicted direction of perceived motion during fixation. According to the proximity principle of motion correspondence (Ullman, 1979), we expected that the probability of perceiving leftward motion during fixation would change in a sinusoidal fashion as a function of the retinal displacement, with a negative peak at 90 degrees and a positive peak at 270 degrees.
In Figure 2A, the red dashed line shows the prediction from the first hypothesis in which smooth pursuit enhances motion signals in the direction opposite to that of eye movement. In this case, the probability of perceiving leftward motion will remain dependent on the retinal phase shift but generally increase at any phase angle (vertical upward shift), except near the floor (0%) or ceiling (100%). Figure 2B shows the prediction from the second hypothesis in which objective proximity affects the direction of apparent motion. In our experimental setup, the retinal displacement during pursuit was 180 degrees out of phase of the objective displacement (see Figure 1C). Thus, if the objective displacement determines the perceived direction of apparent motion entirely, the psychometric function, as a function of the "retinal" displacement, will horizontally shift 180 degrees (blue line). If the perceived direction of apparent motion is partially affected by the objective displacement, the psychometric function will shift horizontally in the opposite direction of smooth pursuit. The amount of shift depends on the degree of influence of the objective displacement. The red line in Figure 2B depicts the prediction of the case where the ratio of retinal versus objective influence is 1:1.
We tested these predictions in the following experiments.

Methods
Observers. The observers in this study included two of the authors, M. T. and S. N., and three volunteers who were unaware of the aim of the experiments. All observers had normal or corrected-to-normal vision. They provided written informed consent before the experiments were conducted. The experiments and the consent form were approved by the NTT Communication Science Laboratories Research Ethics Committee, on the basis of the principles mentioned in the Declaration of Helsinki.
Apparatus. Visual stimuli were displayed on a cathode ray tube (CRT) monitor (GDM-F520, Sony, Inc., Tokyo, Japan) at a refresh rate of 120 Hz. The monitor was connected to a visual stimulus generator (VSG2/5, Cambridge Research Systems, Rochester, UK) installed in a workstation (Dell Precision 350, Dell, Inc., Round Rock, TX, USA). The monitor was calibrated using a colorimeter (Color Cal, Cambridge Research Systems) to linearize the gamma relationship. The spatial resolution of the monitor was 800 Â 600 pixels, with each pixel subtending 1.5 minutes at a viewing distance of 113 cm. The observer sat with their head fixed on a chin rest and viewed the display binocularly. The room had no illumination, except for the stimulus presented on the CRT monitor. A keyboard was placed in front of the observer to record their responses. The movements of the dominant eye were monitored at 500 Hz using a video-based eye tracker (EyeLink II, SR Research, Ltd., Ottawa, ON, Canada). Analog output data obtained from the eye-tracker system were recorded on a disk for offline analysis, at a sampling rate of 1 kHz, using a data acquisition system (NR-2000; Keyence, Osaka-city, Osaka, Japan). To synchronize the timing of the stimulus presentation and eye data precisely, we chose a higher resampling rate than the original sampling rate of the eye tracker. In the offline analysis, the resampled eye position time series was low-pass filtered using a Butterworth filter with a cutoff frequency of 30 Hz.
Stimulus. The stimulus was a square-wave grating comprising a horizontal array of bars (see Figure 1A). Each array subtended 1.0 degree in height, 20 degrees in width, and 18 cd/m 2 . The background was a dark field. The height of each array, measured at the midline, was 1.5 degrees above the trajectory of the white marker. Notably, each bar subtended 1.52 deg in width and 1.0 degree in height.
Procedure. We tested three conditions: rightward pursuit condition, leftward pursuit condition, and fixation condition. Under the fixation condition, after the observer pressed a start key, the bar arrays were flashed twice on a dark background with an interval of 125 ms, while the observer fixated on the central stationary white marker. Under the pursuit condition, a key-press by the observer initiated the horizontal movement of the white point at a constant speed of 12.17 deg/s. The white marker moved a distance of 26.6 degrees either from the left end to the right end, or from the right end to the left end of the CRT screen. The observers were asked to pursue the white point as precisely as possible. When the marker approached the horizontal center of the screen, the square-wave grating was flashed twice with an interval of 125 ms when the marker reached the screen center. The procedure used for the leftward pursuit condition was similar to that used for the rightward pursuit condition, with the exception that all the stimuli were mirror reversed. We manipulated the interframe retinal displacement by varying the spatial phase of the second square wave from the first square wave on the display. The spatial phase was defined as the shift angle of the second square wave in the rightward direction. Therefore, a small (large) phase angle resulted in a small (large) displacement in the rightward direction and a large (small) displacement in the leftward direction. Under the fixation condition, the following seven phase angles were presented: 45, 90, 135, 180, 225, 270, and 315 degrees. Under the pursuit condition, the display was 180 degrees out of phase with respect to the displacement between the two flashed gratings during the fixation condition. This manipulation was used to cancel out the 180 degree phase shift introduced by smooth pursuit (see Figure 1C). Consequently, the retinal displacement during the smooth pursuit condition was identical to that during the fixation condition, given perfect tracking.
All three conditions were tested in separate blocks that were presented in random order. The blocks were separated by an interblock rest interval of at least 5 minutes. The interframe displacement was randomly changed within each block. The task of the observer was to judge the motion direction of the grating (leftward or rightward). Note that the observers were not instructed regarding specific coordinates such as retinal or objective coordinates.
Eye-Movement Analysis. The eye-movement system was calibrated at the beginning of each block. The eye position data for each trial were stored on a disk for offline analysis. Eye velocity was obtained by the digital differentiation of the eye position with respect to time. To minimize the difference in the retinal image among the three stimulus conditions, we excluded trials in which the eye position shift between the two flashes exceeded the intended shift of 1.52 AE 0.13 deg. In a preliminary experiment, we examined how much any deviations from the expected pursuit would affect the performance and found that the effect of AE0.13 deg deviations was negligibly small.
The trials were also excluded if they contained rapid eye movements that exceeded 30 deg/s, which could indicate catch-up saccades. We collected data until at least 12 valid trials (typically 24) were obtained within the aforementioned eye-velocity criteria for each stimulus condition. The median eye velocity of the trials used for analysis, along with the quartile deviation, was 12.47 AE 0.648 deg/s.

Results and Discussion
Figure 3 depicts the mean and individual results in separate panels. The perceived motion direction under all conditions was plotted as a function of retinal displacement. Note that the retinal displacement of the fixation condition is in-phase with the objective displacement and that the retinal displacement of the pursuit conditions is 180 degrees out of phase with the objective displacement. The probabilities of the perceived motion direction for the fixation condition (gray dashed line in Figure 3) followed the standard proximity principle, as predicted in Figure 2 (gray dashed line). When the phase angle was 180 degrees, the perceived direction was ambiguous. The rightward motion predominated when the phase angle was smaller than 180 degrees, and the leftward motion predominated when the phase angle was larger than 180 degrees.
A systematic difference to the aforementioned trend emerged for the pursuit conditions. Here, the perceived direction was opposite to the pursuit direction when the phase angle was 180 degrees (blue and red lines in Figure 3). This result is consistent with previous research (Terao et al., 2015) as well as our preliminary observations. The probabilities of the perceived motion direction at other phase angles were different than the prediction from motion enhancement in the direction opposite to that of eye movement ( Figure 2A). Instead, they are consistent with the hypothesis that objective proximity affects the direction of apparent motion ( Figure 2B). In the mean data, the probability of seeing rightward motion increased at 225, 270, and 315 phase degrees in the rightward pursuit condition and at 45, 90, and 135 phase degrees in the leftward pursuit condition. The probability of seeing leftward motion increased at 45, 90, 135, and 180 phase degrees in the rightward pursuit and at 180, 225, 270, and 135 phase degrees in the leftward pursuit condition. In addition, the motion direction judged to be ambiguous was not at 180 degrees, but at $110 degrees and $290 degrees for the rightward pursuit condition. The effects of pursuit can be characterized by a horizontal shift of the psychometric function.
If the perceived motion direction was determined entirely by the objective proximity, the psychometric function plotted as a function of the retinal displacement would shift 180 phase degrees toward the direction opposite of eye movement. However, such a large shift was not observed. Our results seem to be consistent to the prediction that the perceived direction of apparent motion is partially affected by the objective proximity, as depicted by the red line in right panel of Figure 2B. Although the actual data ( Figure 3) may look different from the sine-wave-based prediction, this is mainly because we did not collect any data at 0 and 360 degrees. The partial shift in the direction opposite to that of eye movement suggests that the direction of the apparent motion during smooth pursuit depends on both retinal and objective proximities.

Experiment 2
The first experiment suggests that the visual system's computation of motion correspondence during pursuit is based on the displacement of elements on a mixture of retinal and objective coordinates. To provide supportive evidence for this tentative conclusion, we additionally examined the perceived displacement, or jump size, in Experiment 2. If pursuit modulates the spatial coordinates, the apparent jump size may also change. Under fixation, where only retinal proximity matters, the physical jump size is largest at a 180 degrees phase shift, and accordingly, the perceptual jump size is expected to be largest at this phase shift. This is where the perceived motion direction is most ambiguous. If pursuit shifts the peak phase of the maximum motion ambiguity by changing the spatial coordinates, it may also shift the peak phase of the maximum perceived jump size in a similar way.

Methods
The methods were identical to those in Experiment 1, except for the following. At the beginning of the experiment, the observers saw two reference stimuli while fixating: One was maximum displacement (180 degrees), and the other was minimum displacement (45 degrees or 315 degrees). The observers were instructed to assign 3 to the maximum displacement and 1 to the minimum displacement. In each experimental trial, participants were asked to rate the perceived jump size using 1, 2, or 3, in addition to judging the motion direction by pressing the corresponding key. Three observers participated in Experiment 2. They also participated in Experiment 1. One of them (S. T.) was naive to the aim of the experiment.

Results and Discussion
Figure 4 depicts the proportion of the perceived motion direction and the evaluated jump size of the motion for each observer. The effects of pursuit on direction judgments show a similar pattern to Experiment 1. For the fixation condition, the curve of the perceived jump size shows a symmetric convex-upward function peaking at 180 degrees. The observers evaluated the apparent jump size to be maximum around the point of perceptual directional ambiguity (see black arrow in Figure 4). As the phase angle decreased or increased from 180 degrees, the estimated jump size gradually decreased. These results indicate that the observers were able to correctly report the jump size magnitude consistent with the retinal displacement under the fixation condition. For the pursuit conditions, the curves of the perceived jump size appeared to shift leftward for rightward pursuit or rightward for leftward pursuit, away from the curve of the fixation condition. If pursuit changes the spatial coordinates for both motion direction computation and jump size estimation in a similar way, the phase angle of the maximum perceived jump size is expected to coincide with the phase angle of the maximum directional ambiguity on the positive slope (red arrow). The observed curve shifts were consistent with this prediction with respect to the shift direction, but the magnitude of the curve shift was not as large as predicted. This suggests that during smooth pursuit, the perceived jump size is affected not only by the retinal displacements but also by physical displacements, although the contribution of physical displacements to perceived jump size is not as large as it is to direction judgments.

General Discussion
In this study, we considered the following two hypotheses regarding the determinant of the apparent motion direction during smooth pursuit. One is that the motion signals in the direction opposite to that of the smooth pursuit are enhanced. The other is that objective proximity affects the detection of apparent motion. To test which of these hypotheses is valid, we investigated the perceived direction of two-frame square-wave gratings with various spatial phases during smooth pursuit. In the case of the former hypothesis, the form of the  psychometric function of smooth pursuit plotted against retinotopic displacement will be different from that of fixation. In the case of the latter hypothesis, the psychometric function of smooth pursuit will shift horizontally from the psychometric function of fixation. In Experiment 1, the behavior of the psychometric function of smooth pursuit was consistent with the prediction from the latter hypothesis. Specifically, the psychometric function of smooth pursuit appeared to shift from the psychometric function of fixation in the direction opposite to smooth pursuit. In Experiment 2, pursuit also produced horizontal shifts of the point of maximum perceived jump size, though the shift magnitudes were smaller than observed with the point of maximum directional ambiguity. The results from both experiments indicate that objective proximity affects the detection of apparent motion. We showed that the psychometric function for motion direction shifted approximately 70 degrees toward the direction opposite to that of eye movement. This suggests that the direction of apparent motion during smooth pursuit was determined by 40% objective proximity and 60% retinal proximity, under the current experimental configuration (while the contribution ratio of the physical proximity may be smaller for jump size perception). This mixture ratio indicates that the visual system partially recovered the objective displacement to detect apparent motion. To recover the objective displacement, the visual system requires information regarding the motion vector that was added to the retinal displacement by smooth pursuit. Generally, extraretinal signals related to pursuit are considered to recover the objective world state during pursuit (e.g., Freeman, 2001;Freeman et al., 2009;Souman et al., 2006;Sperry, 1950;von Holst & Mittelstaedt, 1950;Wurtz, 2008). It is possible that the visual system takes extraretinal signals related to pursuit into account when estimating the additional motion vector by smooth pursuit. It is known that the additional motion vector by smooth pursuit is often underestimated in the case of the speed estimation process (see Freeman et al., 2010). Similarly, this underestimation might also occur in our phenomenon and might contribute to the partial recovery of objective displacement for the detection of apparent motion.
It is known that a mismatch between retinal and extraretinal signals leads to various perceptual phenomena. The Filehne illusion is a phenomenon in which an environmentally stationary object appears to move slightly in the opposite direction of pursuit (e.g., Filehne, 1922;Freeman & Banks, 1998;Mack & Herman, 1973;Wertheim, 1987). The Aubert-Fleischl phenomenon is a phenomenon in which a tracked object appears to move more slowly than when viewed during fixation (Aubert, 1886;Dichgans et al., 1975;Fleischl, 1882;Freeman & Banks, 1998;Wertheim & Van Gelder, 1990). These phenomena are considered products of imperfect velocity compensations caused by the integration of the underestimated pursuit velocity with the retinal velocity. Such errors in the velocity integration process may modify the apparent velocity of motion but will not change the pattern of motion correspondence. Thus, our effect and these classical phenomena presumably have different mechanisms, even though both phenomena might result from a mismatch between retinal and extraretinal signals.
The mismatch between retinal and extraretinal signals also causes localization errors (for review, see Schlag & Schlag-Rey, 2002). One might consider that our findings are related to the localization errors for shortly presented objects during smooth pursuit, given that proximity computation is based on the position information. However, this is likely because the mislocalization during pursuit has been ascribed to visual neural delays, which make visual information of an object combined with eye position information at different moments (e.g., Brenner et al., 2001). This mismatch presumably affects the two flashed gratings in our experiments in the same manner and thus produces no relative phase shift change between them.
Similarly, spatial and temporal properties of perception, memory, and representational momentum are suggested to contribute to localization errors during smooth pursuit (e.g., Brenner et al., 2001;Freyd & Finke, 1984;Kerzel, 2000Kerzel, , 2005Rotman et al., 2004), but they are unlikely to be responsible for our effect because they will affect the two gratings used in our experiments in the same manner.
Although this study was motivated by the findings of Terao et al. (2015), it remains controversial whether the contribution of objective proximity can also explain the motion direction perceived for the counterphase grating during pursuit. In our preliminary observation, we did not find a similar slower (shorter-displacement) bias in the judgment of the continuous counterphase grating direction. In other words, the direction of the slower component was not always judged as the dominant direction when two continuous drifting gratings along mutually opposite directions moving at different speeds were superimposed. Instead, two opposing motions were seen simultaneously for most of the time. Further investigation is necessary to clarify this issue.
Motion detection considering objective proximity during smooth pursuit may serve a useful function to ensure perceptual stability. Proximity, which is the preference to choose a smaller displacement, is an essential cue for the motion-correspondence problem. The motion-correspondence problem is a problem in which the visual system must determine which elements originate from the motion of which object in the world when multiple elements are presented in an apparent motion display. The elements that are close to each other have a stronger affinity than those that are farther apart. It has been considered that proximity computation is based on retinotopic displacement (Ullman, 1979). However, the proximity in the retinotopic displacement is useless for precisely estimating the object motion during eye movements, as the proximity in retinal coordinates does not reflect a physical one because of eye movements. If the proximity computation depends entirely on the proximity of the retinal image during eye movements, false matching might occur. Upon determining the matching of the motion path, it is almost impossible to rematch the motion path at a subsequent processing stage. Accordingly, the retinal proximity should be compensated at an early motion-processing stage as much as possible to achieve precise object-motion estimation. Our results indicate that the matching of motion path during smooth pursuits is determined not only by the retinal proximity but also by the objective proximity. Here, the objective proximity presumably helps reduce the false matching of motion paths. In addition, it may serve as a useful function for reducing image degradation due to motion. The neural integration of visual signals along the trajectory of moving objects can reduce motion-blur perception (Burr, 1981;Burr & Ross, 1986;Nishida, 2004;Nishida et al., 2007). This is because neural integration prevents signal mixtures among spatially adjacent inputs that induce image degradation (Terao et al., 2010;. Given that considering the objective proximity helps to obtain motion that might occur in the environment, the visual signals along the motion trajectory would be effectively integrated, and accordingly, we may obtain an impression of clear vision despite eye movements.
To summarize, we demonstrated that objective proximity affects the detection of apparent motion during smooth pursuit. Motion detection considering the objective displacement helps select a motion path that is more likely to occur in the real world and, therefore, may be useful for ensuring perceptual stability during smooth pursuit.

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.