Intelligent Carom Billiards Assistive System for automatic solution path generation and actual path prediction with principal component analysis-based ball motion detection

This article proposes an Intelligent Carom Billiards Assistive System for a novice at Carom Billiards. Since a novice player is very difficult to find the solution paths of the cue ball, the ICBAS automatically calculates the solution paths for given ball configuration which is detected by a monocular vision and displays them in a guidance monitor. For rapid and systematically calculating the solution paths, the five and half system that is the most representative method in the field of billiard is implemented. In addition, the paths of the cue ball changed by the direction of the cue stick are predicted in real time and projected onto the pool table bed. By comparing the solution paths of the cue ball with the projected paths, a player can hit the cue ball easily and successfully. For automatically providing the solution paths and the predicted paths during billiard game, the ball movement is detected by the principal component analysis under linear motion constraints robustly against image noise, occlusion, and so on. Whenever all of the balls stop, both paths are newly calculated and provided for a player. For verifying the feasibility of the proposed Intelligent Carom Billiards Assistive System, the experiments are conducted with the real pool table.


Introduction
According to the report of the Ministry of Culture, Sports and Tourism of the Republic of Korea in 2014, 62.2% of leisure activities are relaxation, but only 8.6% are sports activities. That is because most sports require an understanding of complex rules and a certain level of athletic ability. In addition, in the case of intense sports, the risk of injury can be felt. Unlike other sports, cue sports (or billiard sports) are rarely injured and are popular sports for all ages. However, there are still barriers to entry because it takes a lot of time to learn some complicated principles and billiards skills.
There are three major games in cue sports: Carom billiards, Pool (Pocket billiards), and Snooker. The difference between Carom Billiards and Pool and Snooker is that Carom Billiards has no pockets on the pool table whereas Pool and Snooker have pockets. There are many different types of games in Pool and Snooker, but the basic rule is for a player to hit the cue ball to put the object ball into the pocket. Because the rule is relatively simple, many assistive systems have been developed for Pool and Snooker. [1][2][3][4][5][6][7][8][9] Some study was conducted to provide a player with a simple top view of the pool table using intelligent glasses for self-training. 1 However, most of them focus on the cue ball's path prediction according to the cue stick's direction and provide the predicted path for a player. The predicted path is obtained by considering collisions with not only cushions but also object balls. In some research studies, physics-based cue ball's path prediction was studied for a billiards robot, not for a human being. The multiple neural network compensator was developed for a billiard robot to master the cue ball's trajectory with multiple cushion rebounds, 2 and the fuzzy neural network was proposed to predict the cue ball's final position after stroking and collision. 3 In another research, 4 the four frame long shortterm memory (LSTM) model was proposed for training the predictive dynamics model of billiards with sequential visual inputs. Thanks to the physics-based approach, the cue ball's trajectory and final position can be obtained more accurately, but it requires higher computational complexity due to neural network training and inference. In previous research studies, 5,6 a camera was attached to the player's head and body for ball and cue stick detection. During a billiards match, it may cause inconvenience to the player's movement. In other research studies, a band and a marker were used for the cue stick detection. 7,8 In this case, some additional settings are required in advance. Despite these inconveniences, the previous assistive systems seem to be very helpful for a player to find the cue ball path that can put the object ball into the pocket.
Unlike Pool and Snooker, the rule of the Carom billiards is more difficult and complicated. The rule of the three-cushion billiards among games within Carom billiards is that the cue ball hit by a player must hit two object balls and hit the second object ball after contacting at least three cushions. For a novice of the three-cushion billiards, it is challenging to find the solution path of the cue ball satisfying the three-cushion rule. In this article, we propose an Intelligent Carom Billiards Assistive System (ICBAS) for helping a novice player. The proposed ICBAS provides not only the solution paths of the cue ball satisfying the three-cushion rule but also the predicted paths of the cue ball that are changed in real time by the direction of the cue stick. To do so, the ICBAS first detects three balls and a cue stick using a monocular camera with no additional settings required in previous research studies. For robust detection to illumination change, the red-green-blue (RGB) image obtained from the camera is converted into the hue-saturation-value (HSV) image. By applying image preprocessing techniques, such as Gaussian filtering and binary thresholding, and contour extraction technique to the HSV image sequentially, the contour image can be obtained. For detecting three balls, the circle Hough transform (CHT) 10 is applied to the contour image. Finally, each ball color is determined considering the light reflection problem on the balls. Recently, many researchers use convolutional neural networks for edge or contour detection. In the research, 11 the fully convolutional encoder-decoder network is trained end-to-end to effectively highlight object contours with suppressing other edges. In the research, 12 the convolutional oriented boundaries are proposed by considering not only contour strength but also orientation for more accurate results. In the research, 13 the architecture-fusion network is trained end-to-end to effectively reduce the misrecognition of both internal and background edges inside objects without degradation of object detection precision. Deep neural networks (DNNs)-based methods are efficient and useful for object contour detection from various images, but training and inference of DNNs require powerful and expensive systems including graphics processing units (GPUs). Thus, the deep learning system with GPUs seems over-spec to apply to the proposed ICBAS as ball detection on the pool table bed is relatively simple.
With the configuration of the detected three balls, the solution paths of the cue ball can be systematically calculated by using the five and half system 14 that is the most representative method in the field of three-cushion billiards. The ICBAS displays the obtained solution paths in a guidance monitor with the tip-point for the desired spin of the cue ball and the overlap width between the cue ball and the first object ball. A novice player can refer to the solution paths in the monitor during billiard game. However, a player may be difficult to map the solution paths to the actual pool table. Thus, the ICBAS also predicts the cue ball paths changing in real time according to the direction of the cue stick and projects the paths on the actual pool table using a beam projector. That is a kind of augmented reality system. Consequently, a player can successfully play a billiard game conveniently comparing the solution paths and the predicted paths.
Next, it is very important to appropriately provide the solution paths and the predicted paths for a player without interfering with the player's game. In previous research, it was difficult for the billiard game to proceed smoothly because a player had to trigger using a keyboard, a mouse, and so on manually, to calculate the solution paths every time. 15 To solve this problem, the ICBAS automatically recalculates and provides the solution paths and the predicted paths whenever a player hits the cue ball and all of the balls stop. In this case, the ICBAS must detect the movement of the balls robustly against false detection caused by image noise, occlusion, and so on. Thus, for detecting the motion of the balls after the player hits the cue ball, the principal component analysis (PCA) 16,17 is used under linear motion constraints since the normal motion of the balls is linear. Of course, the motion of the cue ball may not be linear because of spin, but its motion is eventually linear after the spin disappears. Using the PCA with linear motion constraints, the ICBAS can exactly detect the motion of the balls robustly against the abnormal change of the ball position caused by image noise, occlusion, and so on.
The remaining part of this article is organized as follows. The second section describes the concept of the ICBAS and the flowchart of the proposed algorithm. In the third section, the image processing techniques for detecting three balls and a cue stick are described. In addition, the table bed is detected and the homography transformation from the image coordinates to the object coordinates is obtained from the four corner points of the table bed. In the fourth section, for automatic generation of the solution paths and the predicted paths of the cue ball, the detection method of the ball motion is introduced. In this case, the PCA is used for detecting whether all of the balls stop or not. Next, the solution paths of the cue ball are obtained by using the five and half system in the fifth section. In the sixth section, for verifying the feasibility of the proposed ICBAS, experiments are conducted with an actual pool table. Finally, concluding remarks are described in the seventh section.

Intelligent Carom Billiards Assistive System
The basic concept of the proposed ICBAS is shown in Figure 1. The ICBAS provides both the solution paths and the predicted paths for a player to easily make a successful billiard shot. The solution paths are calculated by using the five and half system that is the most representative technique in the field of billiards to systematically find the paths satisfying the three-cushion rule. The resultant solution paths are displayed together with the tip-point for the required spin of the cue ball and the overlap width between the cue ball and the first object ball in a guidance monitor. For player's convenience, the guidance monitor is continuously controlled to head for a player during billiard game. For automatic generation of the solution paths, the three ball positions are detected by a monocular camera. Next, the actual paths of the cue ball are predicted and projected by a beam projector onto the pool table bed. According to the change of the cue stick direction, the predicted paths are repeatedly recalculated and reprojected as shown in Figure 1. In this case, a player changes the direction of the cue stick to find the predicted path that is coincided with the solution path in the guidance monitor and then just hits the cue ball along the predicted path. In this way, the proposed ICBAS helps a player enjoy his or her own game even though he or she is a beginner of Carom Billiards.
In Figure 2, the algorithm of the ICBAS is depicted as the flowchart in detail. The image obtained from the camera is first calibrated by using the GML Cþþ Calibration Toolbox. 18 Next, the calibrated image is preprocessed for noise reduction and contour detection. From the detected contour image, the pool table bed and the three balls are detected. Using the obtained corner points of the table bed, the homography transformation matrix is calculated from the image coordinates to the object coordinates. Using the homography transformation matrix, the three ball positions in the object coordinates can be calculated from the points detected in the image coordinates. Next, the ICBAS detects the motion of the balls by applying the PCA to the trajectory points of the balls represented in the object coordinates. If all of the balls stop, the ICBAS automatically finds the solution paths of the cue ball and displays them in the guidance monitor. In addition, the cue ball path according to the direction of the cue stick is predicted and projected onto the table bed. If at least one of the balls begins to move, the ICBAS just receives the next image from the camera and continuously checks the ball motion until all of the balls stop again. In this way, the ICBAS appropriately provides the solution paths and the predicted paths for a player during billiard game. The detail methods are described in the following sections.

Ball and cue stick detection
In this section, the detection method of three balls and a cue stick is introduced based on the image processing techniques. For robust detection to illumination change and image noise, the ICBAS first detects the pool table bed from the contour image and sets the region of interest (ROI) as the quadrilateral of the table bed. Within the ROI, three ball positions are obtained by the circle Hough transform (CHT). Next, the color of each ball is detected by averaging color robustly against the light reflection on the surface of the ball. For detecting the direction of the cue stick, the ICBAS first sets the ROI around the cue ball and detects the center position of the cue stick within the ROI. Finally, the direction of the cue stick is determined as the vector from the center of the cue stick to the center of the cue ball. The detail process is described as follows.
The image from the monocular camera is preprocessed as shown in Figure 3. First, the original RGB image is converted into the HSV image, where V, S, and H values of the HSV color model are calculated from R, G, and B values of the RGB color model as in equations. (1), (2) and (3), respectively.
where H is set as (H þ 360) for H < 0. In this case, 0 H 179, 0 S 255, and 0 V 255. Next, the HSV image is thresholded for detecting the green table bed as in equation (4).
where T K l and T K u for K ¼ fH; S; V g are lower and upper threshold values, respectively. For detecting the green table bed, the actual threshold values of ðT H l ; T S l ; T V l Þ and ðT H u ; T S u ; T V u Þ are determined as (80, 100, 10) and (100, 255, 200), respectively. The hue value H is determined within the table green color range with a small offset value. However, the lower threshold of the saturation S is less than half and the upper threshold is the maximum, so the range of the saturation S is very wide. Furthermore, the range of the brightness V covers almost the entire range. As a result, it is possible to detect the pool table without being greatly affected by illumination change. From the binary image, the contours are detected by the border following technique 19 as shown in Figure 3.
For detecting the pool table bed, the contours are first approximated as polygons by using the Douglas-Peucker approximation. 20 Since the largest quadrilateral indicates the table bed as shown in the contour extraction image of Figure 3, detecting the table bed is like finding the largest quadrilateral among the polygons. The quadrilaterals with four corner points are extracted among the polygons and their sizes are compared to find the largest quadrilateral. In this case, we assume the size of the table bed in the camera image is larger than 1 4 of the whole image size and thus set the initial size of the quadrilateral as  (5).  where H is the 3 Â 3 homography transformation matrix, s i is the scale factor, and I P i and O P i for i ¼ 1; 2; 3; 4 are represented as ½ I x i I y i 1 T and ½ O x i O y i 1 T in homogeneous coordinates, respectively. The homography transformation matrix H is obtained by minimizing the back-projection error of E in equation (6). 21 where h ij for i ¼ 1; 2; 3 and j ¼ 1; 2; 3 are elements of H. In this case, the denominator, h 31 Þ , is the scale factor of s i in equation (5). The corresponding physical points are set as the corner points of the actual pool table without wooden rails as in equations (7)-(10).
where W T and L T are the width and length of the pool  (11).
where s is the scale factor. Next, we detect three balls from the contour image of Figure 3 as shown in Figure 5. We first apply the Gaussian smoothing filter to the contour image for reducing noise and small details, where the two-dimensional (2D) Gaussian filter is defined in equation (12).
where s is the standard deviation of the Gaussian distribution, and the mean is (0, 0). Using equation (12), the Gaussian kernel is computed as an n Â n matrix, where s is set as ðð n 2 À 1Þ Â 0:3 þ 0:8Þ. By convolving the contour image with the Gaussian kernel, the smoothed image can be obtained as in the first image of Figure 5. Next, we find circles from the smoothed image by using the circle Hough transform (CHT). 10 The CHT is basically based on the voting procedure like the original Hough transform (HT). 22,23 However, the CHT differs from the HT in that the parameter space of the CHT is 3D whereas that of the HT is 2D. Mathematically, the equation of a circle in x-y plane is given by equation (13).
ðx À aÞ 2 þ ðy À bÞ 2 ¼ r 2 (13) where ða; bÞ is the center of the circle, and r is the radius. For point ðx; yÞ in the image, the CHT finds all possible parameters, a, b, and r, according to equation (13), and votes each grid cell ða; b; rÞ in the 3D parameter space a-b-r (or Hough space), where voting means that the voting number of the cell increases by 1. Among the voted grid cells in the a-b-r space, the grid cell whose voting number is the local maxima is selected to determine the resultant circle parameters. In this case, we define the minimum distance D min among circles to remove the falsely detected circles around the true one. In other words, if the distances among the grid cells with the local maxima are less than D min , all of the grid cells are removed except the one grid cell with the largest voting number. We also define the minimum radius R min and the maximum radius R max of the circle, taking the actual radius of the ball into consideration. In other words, the radius of the detected circle should be in the range from R min to R max . As a result, the detected circles for three balls are shown in the second image of Figure 5, where I P B1 ; I P B2 ; and I P B3 are their center positions in fIg. Next, the colors of the balls are determined by averaging the colors of the ðN þ 1Þ 2 points around the center position ðx c ; y c Þ of the ball as in equation (14) for robust color detection to light reflection as shown in Figure 6. where where RðX i ; Y j Þ, GðX i ; Y j Þ, and BðX i ; Y j Þ are red, green, and blue colors at ðX i ; Y j Þ position, respectively, and where bc discards the decimal places. If R; G; andB at ðx c ; y c Þ satisfy the condition in equation (18), the circle with the center position ðx c ; y c Þ is determined as the red ball.
R ! G Â 1:5 and R ! B Â 1:5 If the condition in equation (18) is not satisfied but the condition in equation (19) is satisfied, the circle is determined as the yellow ball.
If both conditions in equations (18) and (19) are not satisfied, the circle is determined as the white ball. In this way, we can determine the positions of the red, yellow, and white balls, I P R ; I P Y ; and I P W , as shown in the third image of Figure 5.
Next, the three ball positions in the image coordinates of fIg are transformed to the physical positions in the object coordinates of fOg by the homography transformation matrix H in equation (5) as follows.
The resultant ball positions, O P W ; O P Y ; and O P R , in fOg are depicted in Figure 7 and will be used as the ball positions in the following sections.
Next, the cue stick is detected as shown in Figure 8. For detecting the cue stick, the HSV image of Figure 8(a) is thresholded by the thresholding values in equation (21). ð20; 20; 0Þ ðH; S; V Þ ð20; 80; 255Þ (21) Similar to the table detection, the binary image of the cue stick can be extracted without being greatly affected by illumination change using the HSV color model. The Gaussian smoothing filter with the size of 3 Â 3 and the operations of erosion and dilation are sequentially applied to the thresholded image for noise reduction. Next, we define the ROI with the width of W S at the cue ball position because the cue stick detection is needed only for predicting the path of the cue ball, where the cue ball is the white ball. In this case, the square with the width of W B is subtracted from the ROI to remove the falsely detected points of the cue ball because the color of the cue stick is almost same as the white ball. The center position of the object points in the ROI is obtained as the center position I P S of the cue stick. Using the homography transform in equation (11), we can obtain the cue center position of O P S and the cue ball position of O P W in the object coordinates of fOg, where O P W is obtained previously as in

PCA-based ball motion detection
For appropriately providing the solution paths and the predicted paths without interfering player's game, the new paths must be automatically recalculated and provided for a player whenever the player hits the cue ball and then all of the balls stop. In this section, the detection method for the moment when a player hits the cue ball and balls are moving and when all of the balls stop is proposed. In this case, we assume that the normal motion of the balls is linear. Of course, the motion of the cue ball may not be linear at the moment when a player hits the ball with spin, but the motion of the ball eventually becomes linear after the spin disappears. Thus, we define the linear motion constraints to detect the ball motion robustly against falsely detected ball positions. The false detection of the ball positions may occur due to image noise or occlusion by the player preparing the next billiard shot. Thus, the PCA is applied to the trajectory of the ball to robustly determine whether the ball motion satisfies the linear motion constraints. If the ball trajectory satisfies the linear motion constraints, it is determined that the ball has started to move. Next, it is determined that the ball has stopped if the variances of the ball positions are small enough. In this case, the variances of the ball positions can be obtained by the principal components. The detail method is described as follows.
where the mean X of fx 1 ; x 2 ; . . . ; x n g and the mean Y of fy 1 ; y 2 ; . . . ; y n g are defined in equations (23) and (24), respectively.
With these means, the center position of the n ball positions is determined as P c ðX ; Y Þ. Next, the covariance between two dimensions for X and Y can be obtained based on Bessel's correction as in equation (25).
where s xy is equal to s yx because the covariance satisfies the commutative law. The covariance between one dimension and itself is the variance. For each dimension, the variance is obtained as in equations (26) and (27). From equation (25) to equation (27), the covariance matrix C can be defined in equation (28).
where C is a 2 Â 2 symmetric matrix, and its eigenvalue l and the eigenvector V can be obtained from equation (29).
where two eigenvectors, V 1 and V 2 , are the principal components of X and are mutually perpendicular (or orthogonal) as shown in Figure 9. The eigenvalues l 1 and l 2 are variances in the directions of V 1 and V 2 respectively, where l 1 ! l 2 . In Figure 9, the center position P c ðtÞ at time t is determined as ðX ; Y Þ from equations (23) and (24). First, we define the condition for ball stop detection with the largest variance l 1 as in equation (30) when the ball has been moving.
ffiffiffiffi ffi where ffiffiffiffi ffi l 1 p is the largest standard deviation of X, and D S is defined by considering the maximum position error caused by image noise as shown in Figure 10. The fact that all of three balls satisfy the condition in equation (30) means that all of the balls have stopped. At this time, their center positions are prepared for the next steps such as finding the solution paths and predicting the real-time path of the cue ball. In this case, the obtained center positions for the next step are no longer updated because slight change of the ball positions may be caused by image noise or occlusion by a player preparing the next billiard shot. The next update will be performed when a player hits the cue ball and all of the balls stop again.
Next, as shown in Figure 11, we define the conditions for ball motion detection caused by player's hitting as in equations (31)

Systematic solution path acquisition of the cue ball
The basic rule of the three-cushion billiards among games within Carom billiards is that the cue ball hit by a player must hit two object balls and hit the second object ball after contacting at least three cushions. For finding the solution  path of the cue ball that satisfies the rule, we use the five and half system (or the diamond system) 14 that is the most representative technique for Carom billiards as shown in Figure 12, where the spin amount is basically set to the maximum at the middle height of the cue ball. If a player hit the cue ball, the cue ball begins to move from the starting point of S to the aiming point of A. After contacting the first cushion on the long rail, the cue ball contacts the second cushion on the short rail. After contacting the second cushion, the cue ball moves to the finishing point of F on the third cushion. After contacting the third cushion, the cue ball finally moves to the object balls and hits the balls. The five and half system defines the relationship among these three points of S, A, and F by the following formula in equation (33).
For the given starting point S and finishing point F, the aiming point of A can be calculated by equation (33), where S, A, and F are scalar.
In the five and half system, for calculating the formula in equation (33), the scalar values of the marker points on the wooden rail are defined nonlinearly and differently for S, A, and F as shown in Figure 13. In this case, ðS À FÞ must be greater than or equal to zero because A is greater than or equal to zero.
For finding the solution path of the cue ball, we first find the three cushion line to hit the object balls. For given finishing points, three cushion lines are determined as shown in Figure 14. In this case, we can find the three cushion line of the finishing point F of 20 to hit the object balls. The three cushion line between each two lines is calculated by linear interpolation.
Next, for the given position of the cue ball, we find the starting point S and the aiming point A that satisfy the formula in equation (33) as shown in Figure 15. For the given finishing point F of 20, the starting point S and the aiming point A are determined by equation (33) as 40 and 20, respectively. In this way, we can find the solution path of the cue ball. During billiard game, a player just hits the cue ball for the aiming point A and then the cue ball moves along the solution path to hit the object balls.
So far we find the solution path of the cue ball in the case of two object balls attached. In the case of two object balls separated, the solution path can be found as shown in Figure 16. In this case, we find the solution path for the cue ball to hit the first object ball and then to hit the second object ball after contacting three cushions. To do    so, we first find the three cushion line to hit the second object ball. Next, we define the ghost ball right next to the first object ball by considering the collision point between the cue ball and the first object ball. In this case, we regard the ghost ball as the cue ball and calculate the starting point S and the aiming point A similar to the attached object ball case.
Finally, we determine the direction of the cue ball so that its tangent after collision with the first object ball points to the aiming point A of 20. If we assume that the collision between balls is perfectly elastic, the collision can be described as shown in Figure 17 Using the angle q o between v a ! and v o ! , the angle q c between v a ! and v c ! can be calculated by equation (34).
When the vector v c ! is coincided with the tangent line, the overlap width d t between the two balls in the direction of v a ! can be calculated by geometry. According to the overlap width corresponding to q o and q c , the fractional aiming table can be defined as shown in Figure 18 based on the fractional aiming system. 26 During billiard game, if a player just hits the cue ball with the given overlap width, the cue ball moves along the solution path after hitting the first object ball and finally hits the second object ball. When applying the five and half system, we can find up to two solution paths depending on which ball is set as the first object ball. In addition, for each case, we can find up to four solution paths by symmetrically transforming the ball positions vertically or horizontally. Furthermore, we can find more paths by anticipating the path after contacting the fourth cushion, where the cue ball after the fourth cushion simply moves at the same incidence and reflection angles since the amount of the spin of the cue ball is almost gone. In this way, the solution paths can be found and displayed in the guidance monitor. Although we use only the five and half system for finding the solutions paths in this article, much more solution paths can be    found by various kinds of billiard techniques such as the plus system, the maximum English system, the double rail system, the reverse English system, and the no English system.

Experimental setup
For verifying the feasibility of the proposed ICBAS, experiments are conducted in a 60% reduced pool table as shown in Figure 19. For detecting the balls and the cue stick, a monocular camera, Logitech C615 webcam is installed 1.78 m above the pool table at a 27 from the ceiling with a beam projector. Using the camera detection results, the control PC calculates the predicted path and projects it onto the pool table bed by the beam projector. Also, the PC calculates the solution paths for three-cushion billiards and displays them in the guidance monitor which has a direction controller for the monitor to head for a player. The direction controller is implemented using a dinamixel AX-12þ servo motor with an Arduino nano. The Arduino receives commands to control the servo motor from the PC by serial communication. The detail parameters for experiments are described in Table 1.

Results of automatic solution path providing
The ICBAS calculates the solution paths of the cue ball and displays the paths in the guidance monitor. The trajectories of three balls during billiard game are shown in Figure 20, where the trajectories of the white, yellow, and red balls are depicted as blue, red, and green points, respectively. In Figure 20(a), all of the balls stop before the player hits the white ball as the cue ball. In Figure 20     sufficiently obtained to confirm their linear motion. In Figure 20(e), the whole trajectories of the three balls are shown until all of the balls stop after player's hitting. During billiard game in Figure 20, the ball motion is detected as shown in Figure 21. In this case, only N S latest positions for each ball are depicted where N S is 10 as in Table 1, and the circle indicates the center position P c ðtÞ. The black solid line from the center position indicates the vector V 1 of the first principal axis with variance l 1 , and the gray line indicates the vector V 2 of the second principal axis with variance l 2 although it is sometimes short and thus not well visible. Before all of the balls stop as in Figure 20(a), only the yellow ball moves whereas the other balls stop, and the trajectories of the balls are shown in Figure 21(a). In Figure 21(b), three balls are judged to be stopped because all of them satisfy the ball stop condition in equation (30). That is, if the variances of the three ball positions are small enough, we judge that there is no motion of the balls or all of the balls stop. After the player hits the white ball as shown in Figure 20(b), the white ball begins to move, and the red ball also begins to move due to the collision with the white ball. However, in this case, the motion of the white ball does not satisfy the linear ball motion conditions because the condition of ffiffiffiffi ffi l 2 p < D M 2 is not satisfied, where ffiffiffiffi ffi l 2 p and D M 2 are 77.27 and 1.5, respectively. In Figure 21 Figure 21(e). In this way, the ICBAS can detect the moment when all of the balls stop or when the motion of the balls occurs. Figure 22 shows the case that the white ball is falsely detected due to the occlusion by a player preparing the next billiard shot. In Figure 22  and is larger than D M 2 of 1.5 mm in Table 1. Thus, the ICBAS maintains the judgment that all of the balls stop. In this way, the ICBAS can prevent from judging incorrectly the ball motion due to the occlusion. In Figure 22(d), the white ball appears and thus its variance becomes small enough again.
Whenever all of the balls are judged to be stopped, the solution paths are calculated and displayed in the guidance monitor. In Figure 23, the result of two solution paths for the white cue ball is depicted. In the first case, the yellow ball is the first object ball and the red ball is the second object ball. By using the five and half system, the solution path can be found and displayed together with the tip-point for maximum spin and the overlap width between the cue ball and the first object ball. In the second case, the red ball is the first object ball and the yellow ball is the second object ball. In this case, the path after contacting the fourth cushion is first estimated to hit the second object ball, where the path after the fourth cushion satisfies the condition of the same angle of incidence and reflection of the ball because the spin of the cue ball almost gone. For the fourth cushion of the estimated path, the solution path is found by the five and half system. The resultant solution path is also displayed with the tip-point and the overlap width.
In Figure 24, two solution paths for the yellow cue ball are depicted. In the first case, the white ball is the first object ball and the red ball is the second object ball. The solution path can be found by using the five and half system and displayed with the tip-point and the overlap width like the first case in Figure 23. In the second case, although the white ball is the first object ball and the red ball is the second object ball as in the first case, we estimate the path after contacting the fourth cushion and find the solution path for the fourth cushion like the second case in Figure 23. The resultant solution path is also displayed with the tippoint and the overlap width. During billiard game, a player can refer to the solution paths with the tip-point for the desired ball spin and the overlap width displayed in the guidance monitor.

Results of real-time path projection
The ICBAS detects the three ball positions and the cue stick direction and projects the cue ball path according to the ball configuration and the cue stick direction as shown in Figure 25. In Figure 25(a), only the predicted path of the cue ball is projected because of no collision with the object balls before the third cushion. On the other hand, in Figure 25(b), both paths of the cue ball and the object ball are projected together because the cue ball collides with the object ball before the third cushion, where the collision occurs between the first cushion and the second cushion. Also, it can be confirmed that the direction of the guidance monitor displaying the solution paths of the cue ball is continuously controlled by detecting the cue stick position to head for the player during player's preparing the next billiard shot.
The actual motion of the balls after player's hitting is shown in Figure 26. In Figure 26(a), the predicted paths of the cue ball and the object ball are shown before player's hitting. In Figure 26(b), it can be confirmed that the actual movement of the balls from 1 to 9 are almost same with the predicted path.