Punching path optimization method for warp-knitted vamp based on machine vision and improved ant colony algorithm

Aiming at the current problems of duplicated paths and low work efficiency in machine punching of warp knitted vamp marker points, this paper proposes a punching path planning method of machine vision combined with intelligent algorithms. The method can improve the timeliness of visual recognition of punching location by limiting the search area and similarity function threshold, and improve the ability of global search and adaptive adjustment in punching path planning by combining with the improved ant colony algorithm to calculate a more accurate and optimized path more efficiently. Through the visual recognition test and the simulation test of the improved ant colony algorithm, the results show that the template matching can correctly identify the positioning hole marker points for different styles, rotation directions and lighting conditions, and the recognition accuracy is 0.43 mm and the repeat positioning accuracy is 0.09 mm; meanwhile, the improved ant colony algorithm can effectively avoid the local optimal solution, which can improve the optimal rate of the result by about 38% and the algorithm can reduce the number of iterations of the optimal solution within 60 times, which greatly saves the calculation time of path planning. The method can be used to improve the efficiency of punching in the actual warp knit vamp punching.


Introduction
Warp-knitted vamp has the advantages of large production, close fabric connection, and a strong supporting force. 1 The punching process is an important process in the shoemaking automatic line.The quality of punching is directly related to the positioning accuracy and accuracy of the subsequent links, because the contour of the vamp needs to be accurately cut by internal cutting in the pretreatment of the warp-knitted vamp.Since the warp-knitted vamp is thick and elastic, it is necessary to locate the vamp when using the cutting die.Whether the positioning hole punched out is accurate and the length of the punching process directly affects the quality of the shoe and the efficiency of shoemaking.However, according to the analysis of the current development status of most shoe-making enterprises.Most enterprises still use manual punching in the footwear punching process.The traditional contact measurement method has high work intensity, low work efficiency, and high labor cost, and the parallelism between workers' sight and measurement tools will also lead to reading errors, which are easy to produce waste products. 2mage target recognition is an essential prerequisite for active target recognition, static positioning mark recognition, and scene content recognition, especially in the application of natural scenes, such as image matching, 3 augmented reality, 4 image retrieval, 5 template matching, 6 3D reconstruction, 7 and many other fields.Template matching algorithms for rapid recognition of shoes with different style landmarks is constructed to improve the automation level of the current shoe industry.The traditional template matching algorithm has certain defects in calculation efficiency and robustness, and is sensitive to noise in the matching process.Therefore, it is essential to eliminate the interference of noise. 8In recent years, many scholars have conducted relevant research on it.For example, Salih et al. proposed a method of automatic segmentation of optic disk from color fundus images.The method includes three main stages: optic disk location, preprocessing, and segmentation.Template matching based on fast Fourier transform is used to locate the seed points located on the optical disk, and then it is used as the input of region growth technology for segmentation. 9Ye et al. proposed a fast and robust framework for multimodal geospatial data matching with integrated local features.Using a threedimensional fast Fourier transform technique, a similarity metric is defined based on the frequency domain feature representation, and then a template matching scheme is used to detect control points between multimodal data, which provides a more accurate matching performance of the framework compared with existing methods. 10Liu et al. proposed a generic preprocessor for accelerating the traditional template matching technique.The algorithm is to exclude regions that are impossible to match with minimal computational effort.The computational efficiency is improved by using a new octagonal star-shaped template and inclusion exclusion principle to extract and compare faceted features.In addition, it can efficiently handle arbitrary rotation and scaling of the reference image.The algorithm can greatly reduce the search space without losing the best match. 11Emlek et al. 12 proposed a new variable window size method for stereo matching that determines the window size based on edge information and matches variable regions of the image.Kim et al. 13 proposed a segmented Hausdorff distance that uses subregions of the image in matching, which has better face orientation recognition and higher processing speed.Although the above methods have been accelerated in a strong noise environment, the matching speed and accuracy are often not ideal, lacking precise positioning of the recognized target.Moreover, the similarity function has a certain complexity.At the same time, the detection accuracy is affected by the external environment, and the robustness is poor, which cannot meet the industrial scenes with changes in the external environment.Therefore, on the basis of the above research, it is crucial to improve the robustness of the detection algorithm to achieve the accuracy and efficiency of the template matching target.
The punching path optimization of warp-knitted vamp is a non-fixed optimization.In the punching process, which is a path planning problem in an uncertain condition because the placement angle of the vamp on the punching platform is arbitrary during the punching process, and the paths to be planned are different for each punching.Now many researchers use intelligent algorithms such as ant colony algorithm, 14 particle swarm algorithm, 15 and genetic algorithm to solve path planning problems. 16At present, there are many fields in which path planning is applied, such as robot welding and obstacle avoidance, and the goal is generally to achieve the task in the shortest time.In recent years, the application of path planning in the field of textiles has also gradually increased.In the three-dimensional path planning of fabrics, Li et al. 17 obtained the three-dimensional coordinates of the fabric stitching position based on binocular vision system, and used Frenner-Schell vector method for stitching path planning to improve the robot stitching efficiency.In terms of two-dimensional path planning of fabrics, Ding and Qiu 18 performed path planning for the cutting process of specialshaped full-page clothing patterns, and introduced an improved artificial bee colony algorithm to reduce redundant cutting paths.Dai 19 proposed the optimization scheme of vamp punching path based on collaborative cyclic neural network to overcome the drawbacks of single optimization algorithm, but the calculation amount was relatively increased.
In this paper, a new method is proposed to solve the above problems, and it is proved to be successfully applied to the punching operation of warp-knitted vamp positioning mark points in practical factory applications.The main work of this paper is as follows: In this paper, the gradient direction and sub-pixel coordinates of the edge are obtained by using the surface fitting method, the similarity is calculated by using the position and gradient direction of the edge points as the matching information, and the termination condition of the template matching search is established to quickly locate the punching mark points.The hole coordinates obtained from the positioning are used for traveling salesman path planning by using the improved ant colony algorithm.The algorithm updates the pheromone concentration of the ant colony in real time by citing the reward and punishment mechanism, which can avoid falling into the local optimal solution in the optimization process, and can optimally sort the punching point sequence of the vamp, improve the convergence speed in the calculation process and improve the punching efficiency.

Punching point recognition of warp-knitted vamp
In the process of warp-knitted vamp punching, the position and attitude of the vamp transmitted to the punching platform are random.If the fixed punching point coordinates are used to work, it will not be able to punch accurately, resulting in wasted vamp and worn punches.Therefore, machine vision is used to identify the coordinates of the punching points that need to be punched at each time.Together with the initial coordinates of the punching robot, the set of points to be optimized is formed, and then the path optimization algorithm is used for optimization.For the punching point recognition algorithm, firstly, the features of the punching point on the batch of warp-knitted vamp templates are trained as shown in Figure 1.The template matching algorithm based on geometric features is used to extract the geometric feature information of the template, calculate the similarity measure of the template of each target image, and after extracting the features, the punching point position in the camera coordinate system of the same vamp at any position and posture is obtained, and finally the path is optimized.
In the process of warp-knitted vamp punching point recognition, the template matching algorithm based on gray correlation is difficult to obtain accurate matching results in many cases, especially for the case that the gray difference between the recognition area and the surrounding is not large.The template matching algorithm based on geometric features can effectively avoid the influence of gray information by extracting the geometric feature information of the template, such as edge, corner, and template center of gravity, and it is not sensitive to the influence of illumination, so it can also get good matching results under the condition of uneven illumination.In this paper, the geometric template matching method is adopted, and the principle is shown in Figure 2. The position of demarcation points arranged by boundary contour lines between different regions and the direction vector perpendicular to the contour line are mainly extracted.The gradient direction and sub-pixel coordinates of the edge are obtained by surface fitting method, and the similarity is calculated by using the position and gradient direction of the edge points as matching information.
In this paper, the coordinates of edge points and gradient vectors are used as matching information when calculating the similarity measure.After non-sharpening guided filtering and DOG feature processing, the template highlights the boundary contour between different regions.The boundary contour transformation of the template consists of a series of points p x y , ,  , and each point has a gradient direction vector d t u . The target image is converted to the point ( , ) x y and the direction vector e v w x y x y x y T , , , ( , ) = corresponding to by the same processing.The gradient vector of the edge points in the template is transformed to sub-image of the target image, the similarity measure value is calculated with the transformed template and the sub-image of the target image search region, and the similarity measure function is shown in Formula (1).

Surface fitting and feature extraction
According to the similarity measure function, it is necessary to obtain the edge point coordinates and gradient direction of the template and the image to calculate the similarity measure value of the template and the target image sub-image.The Sobel operator or Canny operator is used to calculate the position and gradient direction of the edge points.The method is simple and fast, but further processing is needed to obtain the accuracy and precise gradient direction at the sub-pixel level, which increases the processing time.In this paper, the principle of surface fitting is used to derive the fast acquisition of the edge gradient direction and sub-pixel coordinates.Assuming (x, y) as the coordinate, and g(x, y) as the grayscale or gradient value, we assume that the fitting second-order surface parameter equation is shown in Formula (2).

G x y k x k y k xy k x k y k
(2) Then the sum of residual squares of fitting second-order surface and real surface is shown in Formula (3).
According to the principle of minimizing the sum of squares of residuals, the Formula (4) shows to minimize the residuals.
The matrix form is shown in Formula (5).
In the formula, A is the coefficient matrix, K is the coefficient vector, and H is the constant term vector.The coefficients of the second-order surface equation are calculated from K = A−1H.The method of using gradient operator is  simple and fast to calculate, but its accuracy is difficult to guarantee.In this paper, according to the principle of surface fitting, the accurate edge gradient direction can be quickly obtained, and the specific derivation process is shown in Formula (6).
It can be seen from the theoretical derivation process that the accurate gradient direction of the edge point can be obtained only by several additions and multiplications.Another advantage of using the fitting method is that the fitting process is equivalent to a Gaussian filtering, which obviously improves the anti-noise and anti-interference ability of the algorithm.
In this paper, when calculating the sub-pixel edge coordinates, the method used is similar to the gradient method for calculating the edge.It is only necessary to first derivative the image, then judge the threshold value of the image and suppress the maximum value to obtain the pixel-level edge position, and then the gradient surface is fitted with the gradient value in the 3 × 3 neighborhood of the pixellevel edge point.According to the obtained gradient direction, the sub-pixel coordinates of the edge point are solved.The linear equation is shown in Formula (7) passing the point (x 0 , y 0 ) on the image coordinate plane, along the gradient direction.
Any space curve can be expressed as the intersection of two surfaces.Thus, the space curve on the surface along the gradient direction and past the point (x 0 , y 0 ), G(x 0 , y 0 ) is shown in Formula (8).

G x y k x k y k xy k x k y k k y k x k k x k y k
The sub-pixel edge point coordinates (x, y) can be obtained by Formula (7) and Formula (8), and then the accurate coordinate value of the vamp punching point in the camera coordinate system can be obtained by calculating the pixels and transforming the coordinate system.

Design of punching path model
In the manual punching, the workers punch from top to bottom and from left to right row by row according to their working experience, which causes many unnecessary paths and increases the punching time, while the use of optimization algorithm to obtain the shortest path for punching can solve such disadvantages and improve work efficiency.In the vamp punching path planning problem, it has the following typical characteristics: (1) There is a fixed starting point, and the punching points on the vamp are placed arbitrarily, and the position of the punched point in the camera coordinate system is different each time.(2) The working time and efficiency of forward punching and reverse punching on the same punching path are the same.(3) Each punching point is regarded as a particle, and the coordinates are obtained by machine vision and applied to the optimization algorithm.
Based on the above characteristics, the mathematical model is established and analyzed by improved ant colony algorithm, so that the punching time of each punch is the shortest.
The punching path planning problem of warp-knitted vamp can be simply described as: the coordinates of punching points on the vamp are known, and the punch is operated from the external starting point, punching once at each point, and finally returning to the starting point to find the shortest path solution.Let V = {1, 2,. .., n} be the set of n working points including the starting point, and S ij be the distance between points i and j.That is, let the objective function be: In the formula, x ij is a model variable with a value of 0 or 1, and only if x ij is 1 on the optimal path, otherwise x ij is 0. Constraints: In the formula, the first and second constraints limit the bifurcation of the punching path, and the last constraint is to limit each point to be passed and return to the starting punching point.

Improved ant colony algorithm
Ant colony algorithm is a multi-agent intelligent bionic algorithm based on the behavior of ants searching for food in nature. 20Traditional ant colony algorithm has a good effect in solving the Traveling Salesman Problem (TSP). 21owever, due to its application in the optimization of the punching path of the warp-knitted vamp, combined with its own characteristics, there are some shortcomings in using the traditional ant colony algorithm to solve the problem: (1) Due to the dense distribution of warp-knitted vamp punching points, the increment gap of pheromone on each motion path calculated at the end of each iteration is not obvious according to the traditional ant colony algorithm update rules, which weakens the influence of pheromone concentration in the calculation of probability, resulting in the final ant selection path mainly depends on the length of S ij , making the movement of the ant blind, and unable to make the pheromone concentration on the optimal path significantly increased, which reduces the convergence speed and falls into local optimal solution.(2) The traditional ant colony algorithm is prone to premature convergence due to the high local pheromone, that is, it falls into a local optimal solution and is difficult to achieve the breadth search. 22owever, there are many punching points on the warp-knitted vamp, and the calculation results are not ideal.After the traditional ant colony algorithm falls into the local optimal solution, there is no mechanism that can break the cycle and cannot jump out of the local optimal solution.After many cycles, the pheromone accumulation on the path increases, and finally the non-optimal solution is obtained, which affects the efficiency of the punch movement.
Aiming at the shortcomings of the above traditional ant colony algorithm in the optimization of warp-knitted vamp punching path, the traditional ant colony algorithm is optimized according to the working characteristics of punching.Two mechanisms are introduced in the process of updating the pheromone to improve the efficiency of the algorithm.The first is the reward mechanism that can increase the convergence rate of the algorithm.The second is the adaptive punishment mechanism that breaks the local optimal solution.The flow chart of the optimization of warp-knitted vamp punching path based on the improved ant colony algorithm is shown in Figure 4.
In the traditional ant colony algorithm, the pheromone update is calculated according to Formula (13) and Formula ( 14), and does not take into account the difference in the contribution of excellent paths compared to other paths to the process of solving the optimal solution.
In the formula, ∆τ ij k represents the pheromone increment of the kth ant in the corresponding path; Q is a constant, indicating the total amount of pheromone left by ants; l k represents the path that ants k moves during this work, and L k represents the path length.To this end, the reward mechanism is introduced, and the details are as follows: (1) At the end of each iteration, the total path length of each ant crawling is calculated and ranked, and the reward is given to the top x, and the pheromone concentration on the path is increased, so that the path is easy to be selected by subsequent ants.(2) Set the maximum and minimum pheromone concentration to avoid falling into local optimum due to too much emphasis on pheromone concentration on certain paths.
In this way, the pheromone increment of the excellent path can be improved in a short time to achieve the purpose of using the obvious concentration difference to guide the algorithm to converge to the excellent path, and to accelerate its convergence speed.According to the above ideas, the pheromone updating formula of traditional ant colony algorithm is optimized as follows: In the formula, r represents the number after the ant colony reranks by the length of the path; x represents the number of ants rewarded; ∆τ ij r represents the pheromone increment left by the ants ranked r on the path; l r represents the path that the ant r moved during this work, and L r represents the path length; [τ min , τ max ] denotes the interval of pheromone concentration.
In the searching process of using traditional ant colony algorithm, it is easy to form a path locked state when encountering relatively complex vamp punching point, that is, the ants "die," and the final solution is the local optimal solution, which affects the punching efficiency.Therefore, in order to make the pheromone concentration on the deadlock path decrease rapidly, an adaptive punishment mechanism is introduced.The details are as follows: (1) When the iterative results have not changed N times, the pheromone concentration on the path is forced to refresh.(2) The pheromone concentration on the other paths is averaged and assigned to each segment of the path, in order to adjust the pheromone concentration on this path in the pheromone matrix does not differ much from that on other paths, and the deadlock state is broken.(3) Since the pheromone concentration matrix is not emptied, the algorithm does not need to recalculate the pheromone concentration of each path, and the probability of re-selecting the path will not be affected.On the basis of the original, it can continue to calculate to ensure the calculation efficiency.
According to the above ideas, the pheromone punishment formula is: In the formula, l best represents the same path 10 times; l n means that all ants walk through the remaining paths that exclude l best ; τ a represents the average pheromone concentration of the path in l best .
In the optimization algorithm, the most important two parameters are the number of rewarded ants x and the number of repeated paths N.
The function of parameter x is as follows.If x = 0, no ants are selected to reward, which is calculated according to the pheromone update formula of traditional ant colony algorithm.On the contrary, if x = m, all ants are rewarded, which cannot play a role in highlighting the excellent path.Through the st70 test route, the influence of x is shown in Figure 5.
The data in Figure 5 is obtained under the conditions of α = 1, β = 3, ρ = 0.1, Q = 10, m = 50.It can be concluded that the inner path length of the optimal solution is the shortest when the number of rewarded ants is 1-3.
The function of parameter N is as follows.If the results obtained by iterations are the same for consecutive N times, the current local optimal solution is broken, that is, the pheromone concentration on the path of the solution is reduced.If N = 0, the concentration on the optimal solution obtained by each iteration is reduced, and the pheromone accumulation cannot be formed.If N = 10, the results of 10 iterations are consistent, and the pheromone concentration on the solution path is reduced to escape the local optimal solution.Through the st70 test route, the influence of N is shown in Figure 6.
The data in Figure 6 are obtained under the conditions of α = 1, β = 3, ρ = 0.1, Q = 10, m = 50.It can be concluded that the shortest path length of the optimal path is obtained by punishing at 25-30 repetitions of the optimal path.

Experiment verification
Through the template matching verification of different styles of vamp, the results are shown in Figure 7.In different types, the matching target points can be correctly recognized.Under different rotation angles, shapes, and sizes, the proposed algorithm can effectively identify and meet the practical field application requirements.In this experiment, Daheng Industrial MER-2000-5GM digital camera, lens Computar V1228-MPY, light source DHK-TL36030-W, industrial computer processor I5-6500, memory 16 GB, software platform MATLAB_ R2018a, VisualStudio 2015, OpenCV3.0.0 were used for verification.The experimental platform is shown in Figure 7.
The movement vamp is placed on the bearing platform by the operator, which cannot ensure the consistency of the position and angle of placing the vamp, and the vision system is in the environment of external light changes.Through the rotation angle and illumination intensity change experiment, as shown in Figure 8. Figure 8(a) the upper is inclined in the upright, left, and right directions, the position and direction of the visual coordinate system relative to the bearing platform are unchanged, and the feature recognition of the upper coordinate system is correct.The upper coordinate system is consistent in the relative position of the upper, and the position on the bearing table is related to the position of the upper.The 13 punching points of the upper can be correctly identified under the upper coordinate system without misjudging the identification target.Figure 8(b) when the vamp is imaged at the same position under different illumination, the feature recognition of vamp coordinate system is accurate, the relative position is consistent, and the 12 punching points of vamp can be accurately identified.
The pattern of sports vamp is updated quickly.The pattern printing is formed by different printing screens.There are certain differences in the punching points of vamp according to the process requirements.The characteristics of vamp coordinate system of different vamps are quite different, so it is difficult to formulate a template to adapt to all vamps.The verification for different models is shown in Figure 9.
In this study, the path planning of punching work on the first warp-knitted vamp in Figure 7 is taken as an example.There are 24 punching points on the vamp, plus the starting point of the punching robot for a total of 25 coordinate points need to be planned.After accurately recognizing the position of each punching point, the punch starts to move from the origin in the shortest time, passes through each point for punching operation and finally returns to the origin.Except the starting point, each point can only be selected once.The position coordinates of the punching point obtained by machine vision are compared with the theoretical values.The position error and root mean square error are shown in Figure 10.It can be seen from (a) that the maximum deviation in x direction is 0.43 mm, and the maximum deviation in y direction is 0.38 mm.In (b), the    In order to verify the iterative process comparison between the improved ant colony algorithm and the traditional algorithm applied to the punching optimization of warp-knitted vamp, the two algorithms are simulated with the same parameters (n = 23, x = 3).The results are shown in Figure 11.The horizontal axis is the number of iterations, the vertical axis is the length of the solution path, and the data are the average value of 10 experiments.It can be seen from the figure that the improved algorithm has high solution efficiency and can calculate the motion path within 60 iterations, Compared with the traditional algorithm, the number of iterations is reduced by about 45%; At the same time, the accuracy of the improved algorithm is high, and the optimal rate of the result is increased by about 38%.
Figure 12 shows the optimal solution path obtained by the improved ant colony algorithm, which returns to the origin after passing through all the vamp punching points once, and finally the value of the shortest path solution is 1079.9966mm.
In order to verify the superiority of the algorithm in this paper, the traditional path planning algorithm (traditional ant colony algorithm, adaptive ant colony algorithm, and genetic algorithm) to solve the traveling salesman problem is selected for comparative test.Eight examples of TSPLIB are selected, and each of them is subjected to 50 independent simulation experiments.Some of the data are derived from the simulation results    1 that the quality of the optimal solution and the average solution obtained by the improved algorithm in this paper is higher than that of the other algorithms, indicating that this algorithm can effectively jump out of the local optimal solution.
In the application of traditional ant colony algorithm in warp-knitted vamp punching path optimization, it can be seen that the iterative time is long in the optimization process, and it is easy to fall into local optimal solution, which cannot significantly improve the efficiency of shoemaking.After using the improved ant colony algorithm, the results are more accurate, efficient, and have stronger global search ability, which can largely avoid the local optimal solution.Under the same other conditions, the accuracy and efficiency of warp-knitted vamp manufacturing can be greatly improved.

Conclusions
In the processing of warp-knitted vamp, the accurate recognition of punching points and the optimization of punching path are the key to control the quality and efficiency of shoe production.Traditional manual punching has low efficiency and high labor intensity.In this paper, machine vision technology and intelligent optimization algorithm are used to realize automatic punching, and some shortcomings of traditional optimization algorithm in warpknitted vamp punching are optimized.
(1) Aiming at the problem that template matching algorithm is difficult to obtain accurate matching results in many cases, the template matching algorithm based on geometric features is used to replace the traditional template matching algorithm based on gray correlation to improve the recognition accuracy.
(2) Aiming at the problem of low efficiency of traditional ant colony algorithm, the reward mechanism is introduced to make full use of the positive feedback characteristics, and the convergence speed is improved by about 45%.(3) Aiming at the problem that traditional ant colony algorithm is easy to fall into local optimal solution when applied in warp-knitted vamp punching path optimization, an adaptive punishment mechanism is introduced to jump out of the local solution cycle and improve the optimal solution rate by about 38%.
From the experimental results, the improved ant colony algorithm is more suitable than the traditional ant colony algorithm in warp-knitted vamp punching work, which can effectively improve the overall production efficiency.With the deepening of research, improved ant colony algorithm will be increasingly applied in the process of footwear manufacturing.

Figure 1 .
Figure 1.Schematic diagram of matching sports shoes template.

Figure 4 .
Figure 4. Calculation process of improved ant colony algorithm.

Figure 5 .
Figure 5.Effect of parameter x.

Figure 6 .
Figure 6.Effect of parameter N.

Figure 11 .
Figure 11.Comparison of iterative processes for improving ACOs and traditional ACOs.

Figure 12 .
Figure 12.Optimization path of the improved ant colony algorithm.

Table 1 .
Improved ACO, traditional ACO, adaptive ACO, and genetic algorithms for optimal and average solutions of TSP. of different algorithms given in Lei and Wang 23 for corresponding examples.The simulation results of corresponding examples are shown in Table1.It can be found from Table