WO2022137494A1

WO2022137494A1 - Shoe appearance inspection system, shoe appearance inspection method, and shoe appearance inspection program

Info

Publication number: WO2022137494A1
Application number: PCT/JP2020/048681
Authority: WO
Inventors: 淳也平柴; 剛史小川
Original assignee: 株式会社アシックス
Priority date: 2020-12-25
Filing date: 2020-12-25
Publication date: 2022-06-30
Also published as: JPWO2022137494A1; CN116802483A

Abstract

This invention provides shoe appearance inspection technology that is capable of enhancing inspection efficiency and inspection accuracy. In this shoe appearance inspection system, an imaginary line extraction unit 128 extracts, from an image of a shoe under inspection, a plurality of imaginary lines joining a reference point and a plurality of element points. A model storage unit 136 stores a trained model generated through machine learning using, as training data, imaginary lines and contours extracted from images of a plurality of shoes that are acceptable products. A pass/fail determination unit 134 determines whether a shoe under inspection is an acceptable product by inputting, into the trained model, a plurality of imaginary lines and a contour extracted from an image of the shoe under inspection, and outputs the determination.

Description

Shoe appearance inspection system, shoe appearance inspection method and shoe appearance inspection program

The present invention relates to a technique for inspecting the appearance of shoes.

Conventionally, for standard products such as metal products whose shape is fixed while remaining constant, inspection efficiency and inspection accuracy may be improved by performing visual inspection by sensors or image processing (see, for example, Patent Document 1). Even in the case of amorphous products whose shapes can change significantly, there is also known a technique of determining the correspondence and consistency between parts between products having completely different shapes by image processing (for example, non-patent documents). 1). Both the technique of Patent Document 1 and the technique of Non-Patent Document 1 are common in that they are characterized in that the identity of the product is determined, and the determination content is whether or not the image processing substantially matches 100%. There is.

Japanese Unexamined Patent Publication No. 2019-076819

The manufacturing process of shoe products includes an inspection process in order to maintain the quality of the product. Here, although the shoe product is a product whose shape is fixed to some extent, the upper material is a mesh fiber material or a leather material, and the shape is not completely fixed and is easily deformed. Depending on the situation, there may be slight differences in shape. Further, since the manufacturing process such as the attachment of the upper to the sole and the application of the adhesive is performed manually by a human being, the attachment position and the application position may vary slightly. Due to these properties of shoes, the appearance inspection of shoe products has traditionally been performed by human visual inspection. However, visual inspection cannot deny the possibility of work variation and inspection omission, and the load of inspection is heavy. Therefore, it is desired to establish a technique capable of improving inspection efficiency and inspection accuracy.

The present invention has been made in view of these problems, and an object thereof is to provide a shoe appearance inspection technique capable of improving inspection efficiency and inspection accuracy.

In order to solve the above problems, the shoe appearance inspection system according to an aspect of the present invention is an image acquisition unit that acquires an image of the shoe to be inspected, and an appearance feature point in the image of the shoe to be inspected. A reference point extraction unit that extracts a reference point by a predetermined reference point extraction method, and an element that extracts a plurality of element points that are appearance feature points in an image of shoes to be inspected by a predetermined element point extraction method. The point extraction unit, the virtual line extraction unit that extracts multiple virtual lines connecting the reference point and multiple element points from the image of the shoe to be inspected, and the virtual line extracted from the images of multiple shoes that are acceptable products. The model storage unit that stores the learning model generated by machine learning using lines as teacher data, and the shoes to be inspected by inputting multiple virtual lines extracted from the image of the shoes to be inspected into the training model. It is provided with a pass / fail judgment unit for determining whether or not the product has passed.

The acquired image of the shoe is an image of the shoe in a state of being suspended by the last, and the reference point extraction unit extracts the appearance feature points of the last exposed from the shoe as the reference point in the image. You may.

The acquired shoe image is composed of a plurality of images taken from a plurality of angles, and the model storage unit uses machine learning using virtual lines extracted from the plurality of images for one shoe as teacher data. The generated learning model is stored, and the pass / fail judgment unit determines whether or not the shoe to be inspected is a pass product by inputting virtual lines extracted from a plurality of images for the shoe to be inspected into the learning model. You may.

A contour extraction unit that extracts the contour of the shoe from the image of the shoe to be inspected may be further provided. The model storage unit stores a learning model generated by machine learning using virtual lines and contours extracted from images of a plurality of acceptable shoes as teacher data, and the pass / fail judgment unit stores the learning model of the shoe to be inspected. By inputting a plurality of virtual lines and contours extracted from the image into the learning model, it may be determined whether or not the shoe to be inspected is a pass product.

Another aspect of the present invention is a shoe appearance inspection method. In this method, a process of acquiring an image of shoes to be inspected by a predetermined image acquisition means and a method of extracting a reference point, which is a feature point on the appearance of the image of shoes to be inspected, by a computer. The process of extracting a plurality of element points, which are the appearance feature points in the image of the shoe to be inspected, by a predetermined element point extraction method by a computer, and the reference point and the plurality of element points, respectively. The process of extracting multiple virtual lines to be connected by a computer from the image of the shoe to be inspected, and the learning model generated by machine learning using the virtual lines extracted from the images of multiple shoes that are acceptable products as teacher data. The process of reading from a predetermined storage means, the process of inputting a plurality of virtual lines extracted from the image of the shoe to be inspected into the learning model, and the process of determining whether or not the shoe to be inspected is a pass product by a computer. To prepare for.

It should be noted that any combination of the above components, or the components and expressions of the present invention are mutually replaced between a method, a device, a program, a temporary or non-temporary storage medium in which the program is stored, a system, and the like. Is also effective as an aspect of the present invention.

According to the present invention, it is possible to provide a shoe appearance inspection technique capable of improving inspection efficiency and inspection accuracy.

It is a block diagram of the shoe appearance inspection system which concerns on this embodiment. It is a figure which compares the upper shape of the accepted product and the upper shape of the rejected product with the image taken from the side of the outer instep side of the shoe product. It is a figure which compares the inclination of the axis in the accepted product and the inclination of the axis in a rejected product with a rear image. It is a figure which shows the positional deviation of the midfoot high hardness material in the midsole. It is a functional block diagram which shows the basic structure of a shoe appearance inspection apparatus. It is a figure which shows typically the extraction method of the reference point, the element point, and the virtual line in the image which took the outer instep side of the shoe product with the last from the side. It is a figure which shows typically the extraction method of the reference point, the element point, and the virtual line in the image which took the outer instep side of the shoe product without last from the side. It is a figure which shows typically the extraction method of the reference point, the element point, and the virtual line in the rear image of the shoe product with the last. It is a figure which shows typically the method of extracting the contour in the image which took the outer instep side of the shoe product with the last from the side. It is a functional block diagram which shows the basic structure of the shoe appearance inspection learning apparatus. It is a flowchart which shows the procedure which extracts a plurality of virtual lines and contours from the image of a shoe product, and estimates whether or not it is a pass product based on a learning model.

FIG. 1 is a configuration diagram of a shoe appearance inspection system 100 according to the present embodiment. The shoe appearance inspection system 100 includes a shoe appearance inspection device 110 and a shoe appearance inspection learning device 112. The shoe appearance inspection device 110 and the shoe appearance inspection learning device 112 are derived from a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a RAM (RandomAccessMemory), a ROM (ReadOnlyMemory), an auxiliary storage device, a communication device, and the like. It may consist of a computer. The shoe appearance inspection device 110 and the shoe appearance inspection learning device 112 may be configured by separate computers, or may be realized by one computer having both functions. In this embodiment, an example realized by a separate computer will be described.

The shoe appearance inspection device 110 is communicated and connected with a plurality of photographing devices that capture images of a plurality of shoe products 10. Since the shoe product 10 is easily deformed when held by the operator and it is difficult to perform an accurate inspection, for example, the shoe product 10 is photographed while being placed on a table as shown in the figure. Not limited to. The plurality of photographing devices include a left side photographing device 50 for photographing the shoe product 10 from the left side, a right side photographing device 52 for photographing from the right side, an upper photographing device 54 for photographing from directly above, and a front photographing device 56 for photographing from the front. It is composed of a rear photographing device 58 for photographing from the rear and a lower photographing device 59 for photographing the bottom surface from below. The left side photographing device 50 photographs the outer instep side of the shoe product 10 for the left foot and photographs the inner instep side of the shoe product 10 for the right foot. The right side photographing device 52 photographs the inner instep side of the shoe product 10 for the left foot and photographs the outer instep side of the shoe product 10 for the right foot. The images taken by the left side photographing device 50, the right side photographing device 52, the upper photographing device 54, the front photographing device 56, the rear photographing device 58, and the lower photographing device 59 are transmitted to the shoe appearance inspection device 110. The shoe appearance inspection device 110 inspects the appearance of the shoe product 10 based on the received image. The shoe appearance inspection learning device 112 communicates with and connects with the shoe appearance inspection device 110, and machine-learns an image of the shoe product 10 to generate a learning model. The learning model is used for inspection by the shoe appearance inspection device 110.

FIG. 2 is a diagram comparing the upper shape of the accepted product and the upper shape of the rejected product with images taken from the side of the outer instep side of the shoe product. FIG. 2A is an example of a accepted product, and FIG. 2B is an example of a rejected product. In the example shoe product 10, the sole is formed by laminating the outsole 12, the lower midsole 14, and the upper midsole 16 in this order from the bottom to the top. The shoe product 10 is configured in a so-called suspended state in which the upper 20 is attached to the vicinity of the instep of the last (foot type) 30 placed on the upper midsole 16 and adhered to the sole. The lower midsole 14, upper midsole 16, and outsole 12 made of resin such as EVA (Ethylene-Vinyl Acetate) are hard to deform after manufacturing and the shape is almost fixed, except for manufacturing variations. On the other hand, the upper 20 is made of a material such as a mesh fiber material or leather whose shape is not necessarily fixed even after manufacturing. When the last 30 is pulled out from the shoe product 10, the height of the toes is slightly lowered due to the repulsive force of the sole, so that the upper 20 is easily deformed and the shape of each individual is likely to vary. In this state, it is easy to keep the shape of the upper 20 constant. Therefore, the state with the last 30 is more suitable for the inspection, but the inspection accuracy is improved by machine learning even without the last 30.

The upper is attached to the sole in the suspended state, but since the attaching process is done manually by the operator, there are variations in the work, and the variations may cause variations in the shape. In the accepted product of FIG. 2A, the upper shape 22a has a curved shape that slightly warps from the instep to the toe, whereas in the rejected product of FIG. 2B, the upper shape 22b has an instep to the toe. It has a shape that is almost straight. These differences are so small that they may be overlooked by human eyes, and it is difficult to distinguish them unless they are observed from an appropriate direction. In the whole shoe, there is a clear difference in the positional relationship of the feature points and the balance of length between the passed product and the rejected product. Therefore, the error range of the ratio of the length to the accepted product is modeled by image processing and machine learning, and if the error is judged to be within the allowable range by the learning model, it is estimated as the accepted product, and if it exceeds the allowable range, it is not acceptable. Estimated to be a passing product.

FIG. 3 is a diagram comparing the tilt of the axis in the accepted product and the tilt of the axis in the rejected product in the rear image. FIG. 3A is an example of a accepted product, and FIG. 3B is an example of a rejected product. In the accepted product of FIG. 3 (a), the horizontal axis 18a is almost horizontal and the vertical axis 24a is almost vertical, whereas in the rejected product of FIG. 3 (b), the horizontal axis 18b is horizontal. It is tilted slightly downward to the left, and the vertical axis 24b is also tilted slightly downward from the vertical. These differences in inclination are so small that they may be overlooked by human eyes, and it is difficult to distinguish them unless they are properly placed on a horizontal table. There is a clear difference in the rejected products. Therefore, the range of the inclination ratio with the accepted product is modeled by image processing and machine learning, and if the error is judged to be within the allowable range by the learning model, it is estimated as a accepted product, and if it exceeds the allowable range, it is rejected. I presume.

In the present embodiment, running shoes are exemplified and described as shoe products 10, and various sports shoes including running shoes, leather shoes, and various other shoe products such as those manufactured by attaching an upper to a sole are used. , Can be used for inspection of footwear including sandals and slippers without an upper.

There may be an error in the resin molding process not only in the upper but also in the sole. FIG. 4 shows the misalignment of the midfoot high hardness material in the midsole. In the lower midsole 14, a high-hardness material may be partially used at the position corresponding to the midfoot of the foot to ensure rigidity, but at the position where the high-hardness material is used in the sole molding process. Errors can occur. However, since the portion where the high hardness material is used does not appear in the appearance of the lower midsole 14, it is difficult for the operator to visually detect the error. The figure shows the outline of the sole. The midfoot portion M1 shown by a diagonal line pattern from the upper left to the lower right indicates the high hardness material portion in the accepted product. The midfoot portion M2 shown by a diagonal line pattern from the upper right to the lower left indicates a high hardness material portion in the rejected product. The difference in position is clear when comparing the two as shown in the figure, but when viewed from the inside of the shoe (left side of the figure), there is no difference in the position between the midfoot part M1 and the midfoot part M2, and the outside of the shoe (figure). There is a difference in the positions of the midfoot portion M1 and the midfoot portion M2 only when viewed from the right side of). However, when a worker visually inspects, it is difficult to find a positional deviation such as a difference in the positions of the midfoot portion M1 and the midfoot portion M2 just by looking at a single shoe.

Inside the shoe product 10, the virtual lines La and La'from the reference point Ra at the apex on the toe side to the start points of the midfoot portions M1 and M2 have the same length. On the outside of the shoe product 10, the virtual line Lb from the reference point Ra to the start point of the midfoot portion M1 and the virtual line Lb'from the reference point Ra to the start point of the midfoot portion M2 have different lengths. The width Lc of the midfoot portion M1 and the width Lc'of the midfoot portion M2 are substantially the same length in the example of this figure. In this case, the magnitude of the error can be expressed by the ratio (Ln'/ Ln) of the lengths of the virtual lines of the accepted product and the rejected product. Therefore, after acquiring a plurality of images of the inspection target taken from a plurality of shooting directions, the ratio of the length of the virtual line extracted from each image and the length of the virtual line extracted from the image of the accepted product is calculated and learned. When compared with the result, it is theoretically possible to determine whether or not the magnitude of the misalignment error is within the allowable range.

FIG. 5 is a functional block diagram showing the basic configuration of the shoe appearance inspection device 110. In this figure, functional blocks focusing on functions are drawn, and these functional blocks can be realized in various forms by hardware, software, or a combination thereof. The shoe appearance inspection device 110 includes an image acquisition unit 120, an image storage unit 122, a reference point extraction unit 124, an element point extraction unit 126, a virtual line extraction unit 128, a contour extraction unit 130, an extraction data storage unit 132, and a pass / fail determination unit 134. , The model storage unit 136 is included.

The image acquisition unit 120 acquires an image of the shoe product 10 to be inspected from each of the left side photographing device 50, the right side photographing device 52, the upper photographing device 54, the front photographing device 56, and the rear photographing device 58, and stores the image. Store in unit 122. The image storage unit 122 is classified and stored together with attribute information such as the product model name and size of the shoe product 10 to be inspected, and whether it is for the left foot or the right foot.

The reference point extraction unit 124 extracts a reference point, which is an appearance feature point in the image of the shoe product 10 to be inspected, by a predetermined reference point extraction method. The element point extraction unit 126 extracts a plurality of element points, which are appearance feature points in the image of the shoe product 10 to be inspected, by a predetermined element point extraction method. The virtual line extraction unit 128 extracts a plurality of virtual lines connecting the reference point and the plurality of element points from the image of the shoe product 10 to be inspected. The contour extraction unit 130 extracts the contour of the shoe product 10 from the image of the shoe product 10 to be inspected.

FIG. 6 schematically shows a method of extracting reference points, element points, and virtual lines in an image taken from the side of the outer instep side of a shoe product with a last. In the present embodiment, a reference point and an element point are extracted as feature points from the image of the shoe product 10, and a virtual line connecting the reference point and the element point is extracted. Multiple such virtual lines are extracted, and the data of these multiple virtual lines are input to a predetermined machine learning model, and the position, inclination, inclination ratio of each virtual line to the accepted product, the length, and the accepted product are used. By determining whether the length ratio is within the allowable error range, it is estimated whether or not the shoe product 10 is a acceptable product.

In addition, by inspecting a plurality of images of one shoe product 10 taken from a plurality of shooting directions, it is not possible to estimate whether or not the product is a pass product based on only the images from one direction, but the entire shoe is comprehensive. Estimate whether or not the product has passed. In the image from one side, even if the position, inclination, inclination ratio with the accepted product, length, and length ratio with the accepted product are within the allowable range of the multiple virtual lines extracted from the image. This is because there may be an error exceeding the permissible range in the positional relationship between a plurality of virtual lines extracted from a plurality of images taken from a plurality of directions.

Further, by extracting a contour from the image of the shoe product 10, inputting the contour into a predetermined machine learning model, and determining whether the overall balance of the shape and position of the contour is within the allowable error range, the shoe It is estimated whether or not the product 10 is a acceptable product. Even in the contour inspection, it is possible to determine when an error exceeding the permissible range occurs in the positional relationship and balance of the contour between a plurality of images taken from a plurality of directions.

Reference points and element points are feature points on the appearance with a stable shape that can be extracted based on a predetermined extraction method by image processing. The reference point extraction unit 124 extracts the end point on the heel side of the upper edge, which is a part of the last 30 exposed from the opening of the shoe product 10 and whose edge is detected by image processing for the last 30, as the first reference point R1. do. The upper edge of the last 30 has little shape change in the shoemaking process, and the last 30 used even when inspecting other product models has a common shape, so that it is suitable as a reference point for easy extraction. In addition, since one reference point is to be extracted from one image and a plurality of element points are connected one-to-many with a common reference point as a starting point to extract a plurality of virtual lines, the number of feature points to be extracted is determined. It is possible to avoid an increase in processing load without increasing it indiscriminately. In this respect, it is more advantageous in terms of processing load than the method of extracting a plurality of virtual lines by connecting a plurality of element points from a plurality of different feature points in a many-to-many manner. As a modification, a feature point that can be extracted by using a pattern or a character attached to a part of the last 30 as a mark may be extracted as the first reference point R1.

The element point extraction unit 126 extracts the tip of the toe side of the outsole 12 whose edge is detected by image processing on the outsole 12 as the first element point P1. The first element point P1 is the apex that overhangs the most forward in the arcuate contour on the toe side of the outsole 12. The virtual line extraction unit 128 extracts the first virtual line L1 connecting the first reference point R1 and the first element point P1.

The element point extraction unit 126 is a point at which the curvature of the winding portion on the toe side whose edge is detected by image processing with respect to the outsole 12 changes, that is, a starting point for winding from the ground plane of the outsole 12 toward the toe (“toe spring start point”). ”) As the second element point P2. The virtual line extraction unit 128 extracts the second virtual line L2 connecting the first reference point R1 and the second element point P2.

The element point extraction unit 126 is a point where the curvature of the heel-side winding portion detected by image processing on the outsole 12 changes, that is, a starting point of winding from the ground plane of the outsole 12 toward the heel (“heel cut start point”). ”) As the third element point P3. The virtual line extraction unit 128 extracts the third virtual line L3 connecting the first reference point R1 and the third element point P3.

The element point extraction unit 126 extracts the rearmost end portion on the heel side whose edge is detected by image processing on the sole as the fourth element point P4. The fourth element point P4 is the apex that protrudes most rearward in the arcuate contour on the heel side of the lower midsole 14. The virtual line extraction unit 128 extracts the fourth virtual line L4 connecting the first reference point R1 and the fourth element point P4.

The element point extraction unit 126 extracts the uppermost end portion whose edge is detected by image processing for the shoe opening as the fifth element point P5. The shoe product 10 has a wavy shape on both the outside and the inside, and the element point extraction unit 126 extracts the apex or the highest point in the arc of the wave shape as the fifth element point P5. The virtual line extraction unit 128 extracts the fifth virtual line L5 connecting the first reference point R1 and the fifth element point P5. If the shoe product 10 to be inspected does not have a wavy shape such as boots, for example, the front end or the rearmost end of the shoe opening may be extracted as the fifth element point P5.

In the example of FIG. 6, an example of extracting five element points and extracting five virtual lines has been described, but the number of element points and virtual lines is not limited to this, and the stability of the shoe shape is determined. Depending on the situation, another point may be used as an element point.

FIG. 7 schematically shows a method of extracting reference points, element points, and virtual lines in an image taken from the side of the outer instep side of a shoe product without a last. The shoe product 10 in this figure is an example in which the state after removing the last 30 is inspected. Unlike the case with the last 30, the reference point is extracted from a part of the shoe product 10. The reference point extraction unit 124 extracts the lowermost end portion whose edge is detected by image processing for the wearing opening as the second reference point R2. The opening of the shoe product 10 has a wave shape both on the outside and inside, and the reference point extraction unit 124 extracts the lowest point or the lowest point in the arc of the wave shape as the second reference point R2. If the shoe product 10 to be inspected does not have a wavy shape such as boots, for example, the rearmost end or the frontmost end of the shoe opening may be used as a reference point for extraction. As a modification, a specification may be made in which any one of a plurality of element points is set as a reference point.

The virtual line extraction unit 128 extracts the first virtual line L1 connecting the second reference point R2 and the first element point P1. The virtual line extraction unit 128 extracts the second virtual line L2 connecting the second reference point R2 and the second element point P2. The virtual line extraction unit 128 extracts the third virtual line L3 connecting the second reference point R2 and the third element point P3. The virtual line extraction unit 128 extracts the fourth virtual line L4 connecting the second reference point R2 and the fourth element point P4. The virtual line extraction unit 128 extracts the fifth virtual line L5 connecting the second reference point R2 and the fifth element point P5.

FIG. 8 schematically shows a method of extracting reference points, element points, and virtual lines in a rear image of a shoe product with a last. The reference point extraction unit 124 extracts the apex or the highest point at the upper edge detected by the image processing for the last 30 as the third reference point R3. The third reference point R3 in the rear image is set to a point different from the first reference point R1 in the side image. For example, even in the side image, the upper edge is detected by the image processing for the last 30. By using the apex or the highest point in the above as a reference point, common feature points may be used as reference points.

The element point extraction unit 126 extracts the leftmost end detected by the image processing for the lower midsole 14 as the sixth element point P6. The sixth element point P6 is the leftmost apex of the arcuate contour on the left side of the lower midsole 14. The virtual line extraction unit 128 extracts the sixth virtual line L6 connecting the third reference point R3 and the sixth element point P6.

The element point extraction unit 126 extracts the lowermost end where the edge is detected by the image processing for the outsole 12 as the seventh element point P7. The seventh element point P7 is the lowest point or the lowest point in the outsole 12 arcuate contour. The virtual line extraction unit 128 extracts the seventh virtual line L7 connecting the third reference point R3 and the seventh element point P7.

The element point extraction unit 126 extracts the rightmost edge detected by image processing on the lower midsole 14 as the eighth element point P8. The eighth element point P8 is the apex that overhangs to the right in the arcuate contour on the right side of the lower midsole 14. The virtual line extraction unit 128 extracts the eighth virtual line L8 connecting the third reference point R3 and the eighth element point P8.

As shown in FIGS. 6 to 8, reference points and element points are also extracted from the right side, front, and upper images of the shoe product 10, and virtual lines are extracted.

FIG. 9 schematically shows a method of extracting contours in an image taken from the side of the outer instep side of a shoe product with a last. The contour extraction unit 130 extracts the contour SL of the entire shoe product 10 by edge detection in the image processing for the shoe product 10. Since only one contour SL can be obtained from one image of the shoe product 10, the number of items that can be inspected is less than that of inspection using virtual lines, and the number of objects to be machine-learned is smaller than that of virtual lines. There is an aspect that it is difficult to improve the detection accuracy by learning compared to the virtual line that can learn teacher data. On the other hand, the edge detection of the contour SL does not require the feature of the shape unlike the extraction of the reference point and the element point, so that it can be extracted more easily. Further, the contour inspection can be used not only in the inspection of the finished product but also in the inspection in each of the plurality of processes included in the manufacturing process.

As shown in FIG. 9, the contour is extracted from the right side, front, rear, upper, and lower images of the shoe product 10 and the image of the shoe product 10 without the last.

Refer to Fig. 1 again. The model storage unit 136 stores a learning model that has been pre-generated and learned by machine learning using a plurality of virtual lines and contours extracted from images of a plurality of passing shoe products as teacher data. The model storage unit 136 stores a learning model generated by machine learning using a plurality of virtual lines and contours extracted from a plurality of images for each shoe product 10 as teacher data. This training model has a tolerance for data errors such as virtual line position, slope, slope ratio with pass product, length, length ratio with pass product, contour position, and balance in many pass products. Is formed, the position and inclination of the virtual line extracted from the image to be inspected, the ratio of inclination to the accepted product, the length, the ratio of the length to the accepted product, the position of the contour, and the balance are compared. Therefore, it is possible to estimate whether or not the error is within the allowable range, that is, whether or not the product is acceptable. The learning model is generated in advance by the shoe appearance inspection learning device 112 and stored in the model storage unit 136 as described later.

The pass / fail determination unit 134 inputs a plurality of virtual lines and contours extracted from the image of the shoe product 10 to be inspected into the learning model, and the position, inclination, ratio of inclination to the accepted product, and length of the virtual line, By comparing the length ratio, the position of the contour, and the balance with the accepted product, it is possible to estimate whether the error is within the allowable range, that is, whether the shoe product 10 to be inspected is the accepted product. The pass / fail determination unit 134 inputs a plurality of virtual lines and contours extracted from a plurality of images for the shoe product 10 to be inspected into the learning model, and determines the position, inclination, and inclination ratio of the virtual lines to the accepted products. Comprehensively compare the length, the ratio of the length to the accepted product, the position of the contour, and the balance of the shoe as a whole, and estimate whether the shoe to be inspected is the accepted product. The pass / fail determination unit 134 outputs the estimation result by a method such as screen display, and feeds it back to the learning model stored in the model storage unit 136 as data of a pass product or a fail product.

FIG. 10 is a functional block diagram showing the basic configuration of the shoe appearance inspection learning device 112. The image acquisition unit 220, the image storage unit 222, the reference point extraction unit 224, the element point extraction unit 226, the virtual line extraction unit 228, the contour extraction unit 230, and the extraction data storage unit 232 are the image acquisition unit 120 and the image storage unit 122, respectively. , Reference point extraction unit 124, element point extraction unit 126, virtual line extraction unit 128, contour extraction unit 130, and extraction data storage unit 132, each of which has the same function.

The machine learning unit 234 uses the data of a plurality of virtual lines and the contour data stored in the extracted data storage unit 232 as teacher data, and uses a learning model for determining whether or not the error between the virtual lines and the contour falls within an allowable range. It is generated by machine learning and stored in the model storage unit 236. The learning model is transmitted to the shoe appearance inspection device 110 and used for the appearance inspection of the shoe product 10.

The teacher data includes information on a plurality of virtual lines and contours extracted from shoes as shown in FIGS. 6 to 9. The virtual line data is the position and inclination of the virtual line obtained from a plurality of images of a large number of accepted products, the ratio of the inclination to the accepted product, the length, and the ratio of the length to the accepted product. The contour data is the position and balance of the contour obtained from a plurality of images of a large number of accepted products. Since the virtual lines and contours have variations in position, inclination, length, balance, etc., machine learning is performed to model the allowable range as an error. The shoe product 10 has a plurality of types of product models, and even in one product model, there are a plurality of sizes, and the shoe product 10 is divided into one for the left foot and one for the right foot. The machine learning unit 234 machine-learns a plurality of virtual lines and contours for each attribute such as product model, size, left foot and right foot. By performing machine learning separately for each attribute, the determination accuracy can be further improved.

In the present embodiment, an example of generating a learning model by machine learning only the images of the passing shoe products 10 has been described. In the modified example, the virtual lines and contours extracted from the image of the accepted product are learned and labeled as pass, and the virtual lines and contours extracted from the image of the rejected product are further learned and labeled as rejected. It may be that. Although more teacher data is required than when only passing products are trained, the accuracy of judgment for classifying pass and fail can be improved accordingly. Further, in another modification, a learning model may be generated only by machine learning of virtual lines without using machine learning of contours.

FIG. 11 is a flowchart showing a procedure for extracting a plurality of virtual lines and contours from an image of a shoe product and estimating whether or not the product is acceptable based on a learning model.

The image acquisition unit 120 captures images of the shoe product 10 from a plurality of imaging directions by a plurality of imaging devices such as the left side photographing device 50 and the right side photographing device 52 (S10), and the image acquisition unit 120 captures a plurality of images. Acquire (S11). The reference point extraction unit 124 and the element point extraction unit 126 extract the reference point and a plurality of element points from the image (S12), and the virtual line extraction unit 128 extracts a plurality of virtual lines from the image based on the reference point and the plurality of element points. Is extracted and stored in the extracted data storage unit 132 (S14). The pass / fail determination unit 134 inputs virtual line data into the learning model stored in the model storage unit 136, and determines whether the error of the virtual line is within the allowable range (S16). The contour extraction unit 130 extracts a contour from the image of the shoe product 10 and stores it in the extraction data storage unit 132 (S18). The pass / fail determination unit 134 inputs contour data into the learning model stored in the model storage unit 136, and determines whether the contour error is within the permissible range (S19). The pass / fail determination unit 134 estimates whether the shoe product 10 is a pass product or not by comprehensively determining whether the error of the virtual line is within the allowable range and the error of the contour is within the allowable range. (S20).

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that the embodiments are exemplary and that various modifications are possible for each of these components and combinations of processing processes, and that such modifications are also within the scope of the present invention. ..

The present invention can provide a shoe appearance inspection technique that can improve inspection efficiency and inspection accuracy.

R1 1st reference point, P1 1st element point, L1 1st virtual line, R2 2nd reference point, P2 2nd element point, L2 2nd virtual line, R3 3rd reference point, P3 3rd element point, L3 1st 3 virtual lines, P4 4th element point, L4 4th virtual line, P5 5th element point, L5 5th virtual line, L6 6th virtual line, P7 7th element point, L8 8th virtual line, 12 outsole, 14 lower midsole, 16 upper midsole, 20 upper, 30 last, SL contour, 100 shoe appearance inspection system, 110 shoe appearance inspection device, 112 shoe appearance inspection learning device, 120 image acquisition section, 124 reference point extraction section, 126 Element point extraction unit, 128 virtual line extraction unit, 130 contour extraction unit, 134 pass / fail judgment unit, 136 model storage unit, 220 image acquisition unit, 224 reference point extraction unit, 226 element point extraction unit, 228 virtual line extraction unit, 230 Contour extraction unit, 236 model storage unit.

Claims

The image acquisition unit that acquires the image of the shoes to be inspected,
A reference point extraction unit that extracts a reference point, which is an appearance feature point in the image of the shoe to be inspected, by a predetermined reference point extraction method, and a reference point extraction unit.
An element point extraction unit that extracts a plurality of element points, which are appearance feature points in the image of the shoe to be inspected, by a predetermined element point extraction method.
A virtual line extraction unit that extracts a plurality of virtual lines connecting the reference point and the plurality of element points from the image of the shoe to be inspected, and a virtual line extraction unit.
A model storage unit that stores a learning model generated by machine learning using the virtual lines extracted from images of a plurality of acceptable shoes as teacher data, and a model storage unit.
A pass / fail determination unit that determines whether or not the shoe to be inspected is a pass product by inputting a plurality of virtual lines extracted from the image of the shoe to be inspected into the learning model.
A shoe appearance inspection system characterized by being equipped with.
The acquired image of the shoe is an image of the shoe in a state of being suspended by the last.
The shoe appearance inspection system according to claim 1, wherein the reference point extraction unit extracts the appearance feature points of the foot shape exposed from the shoes in the image as the reference points.
The acquired shoe image is composed of a plurality of images taken from a plurality of angles.
The model storage unit stores a learning model generated by machine learning using virtual lines extracted from a plurality of images for one shoe as teacher data.
The pass / fail determination unit is characterized in that it determines whether or not the shoe to be inspected is a pass product by inputting virtual lines extracted from a plurality of images of the shoe to be inspected into the learning model. The shoe appearance inspection system according to claim 1 or 2.
Further, a contour extraction unit for extracting the contour of the shoe from the image of the shoe to be inspected is provided.
The model storage unit stores a learning model generated by machine learning using the virtual line and the contour as teacher data extracted from images of a plurality of shoes that are acceptable products.
The pass / fail determination unit is characterized in that it determines whether or not the shoe to be inspected is a pass product by inputting a plurality of virtual lines and contours extracted from the image of the shoe to be inspected into the learning model. The shoe appearance inspection system according to any one of claims 1 to 3.
The process of acquiring an image of shoes to be inspected by a predetermined image acquisition means, and
A process of extracting a reference point, which is an appearance feature point in the image of the shoe to be inspected, by a predetermined reference point extraction method using a computer, and a process of extracting the reference point.
A process of extracting a plurality of element points, which are appearance feature points in the image of the shoe to be inspected, by a predetermined element point extraction method by a computer, and
A process of extracting a plurality of virtual lines connecting the reference point and the plurality of element points from the image of the shoe to be inspected by a computer, and a process of extracting the plurality of virtual lines from the image of the shoe to be inspected.
A process of reading a learning model generated by machine learning using the virtual lines extracted from images of a plurality of acceptable shoes as teacher data from a predetermined storage means, and
A process of determining whether or not the shoe to be inspected is a pass product by inputting a plurality of virtual lines extracted from the image of the shoe to be inspected into the learning model, and a process of determining whether or not the shoe to be inspected is a pass product.
A shoe appearance inspection method characterized by comprising.
The function to acquire the image of the shoe to be inspected and
A function of extracting a reference point, which is an appearance feature point in the image of the shoe to be inspected, by a predetermined reference point extraction method, and
A function of extracting a plurality of element points, which are appearance feature points in the image of the shoe to be inspected, by a predetermined element point extraction method, and
A function of extracting a plurality of virtual lines connecting the reference point and the plurality of element points from the image of the shoe to be inspected, and
A function to store a learning model generated by machine learning using the virtual lines extracted from images of multiple shoes that are acceptable products as teacher data, and
A function of determining whether or not the shoe to be inspected is a pass product by inputting a plurality of virtual lines extracted from the image of the shoe to be inspected into the learning model.
A shoe appearance inspection program characterized by the realization of a computer.