JP6041156B2 - Information processing system, information processing method, and information processing program - Google Patents

Description

  The present invention relates to a technique for recognizing products and parts in captured images.

  In the above technical field, Patent Document 1 discloses a technique for confirming whether or not components on a board are definitely attached by image recognition.

JP 2006-245483 A

  However, the technique described in the above-mentioned document has not been able to achieve sufficient quality in terms of accuracy and speed of confirmation.

  The objective of this invention is providing the technique which solves the above-mentioned subject.

In order to achieve the above object, a system according to the present invention provides:
M number of feature vectors, each consisting of 1-dimensional to i-dimensional feature vectors, generated for each of the m local regions including each of the m feature points in the image of the product and the part attached to the product. A first local feature quantity storage means for storing one local feature quantity in association with the product or the part;
Imaging means for imaging the entire product with the parts attached thereto;
N feature points are extracted from the image picked up by the image pickup means, and n local regions including the n feature points are respectively provided with n pieces of feature vectors from 1 to j dimensions. Second local feature quantity generating means for generating a second local feature quantity;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognition means for recognizing that the object exists;
It is provided with.

In order to achieve the above object, the method according to the present invention comprises:
An imaging step for imaging the entire product with the parts attached;
N feature points are extracted from the image captured in the imaging step, and n local feature regions each including the n feature points are each composed of feature vectors from 1 to j dimensions. A second local feature generation step of generating the second local feature of
Each of the m local regions including each of the m feature points in the image of the object is made up of feature vectors from 1D to iD that are generated in advance and stored in the first local feature storage unit. a reading step of reading m first local feature values from the first local feature value storage means;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognizing that the object is present;
It is characterized by including.

In order to achieve the above object, a program according to the present invention provides:
An imaging step for imaging the entire product with the parts attached;
N feature points are extracted from the image captured in the imaging step, and n local feature regions each including the n feature points are each composed of feature vectors from 1 to j dimensions. A second local feature generation step of generating the second local feature of
Each of the m local regions including each of the m feature points in the image of the object is made up of feature vectors from 1D to iD that are generated in advance and stored in the first local feature storage unit. a reading step of reading m first local feature values from the first local feature value storage means;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognizing that the object is present;
Is executed by a computer.

  According to the present invention, the product can be checked accurately and immediately at the production stage, and the product quality can be improved.

It is a block diagram which shows the structure of the information processing system which concerns on 1st Embodiment of this invention. It is a figure which shows the structure of the information processing system which concerns on 2nd Embodiment of this invention. It is a sequence diagram for demonstrating the flow of the process of the whole information processing system which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the communication terminal which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the factory computer which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the production | generation process of the local feature-value based on 2nd Embodiment of this invention. It is a figure for demonstrating the production | generation process of the local feature-value based on 2nd Embodiment of this invention. It is a figure for demonstrating the production | generation process of the local feature-value based on 2nd Embodiment of this invention. It is a figure for demonstrating the production | generation process of the local feature-value based on 2nd Embodiment of this invention. It is a figure for demonstrating the production | generation process of the local feature-value based on 2nd Embodiment of this invention. It is a figure for demonstrating the production | generation process of the local feature-value based on 2nd Embodiment of this invention. It is a figure for demonstrating the collation process which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the collation process which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the content of the local feature-value database 310 which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the content of the product / part database 340 which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the content of the inventory management database 330 which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the hardware constitutions of the communication terminal which concerns on 2nd Embodiment of this invention. It is a figure which shows the local feature-value production | generation table in the communication terminal which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the communication terminal which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the communication terminal which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the communication terminal which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the communication terminal which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the factory computer which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the factory computer which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the factory computer which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the factory computer which concerns on 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the process in the factory computer which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the process of the information processing system which concerns on 3rd Embodiment of this invention. It is a figure for demonstrating the process of the information processing system which concerns on 4th Embodiment of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. However, the constituent elements described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention only to them.

[First Embodiment]
An information processing system 100 as a first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of the information processing system 100. As illustrated in FIG. 1, the information processing system 100 includes a first local feature quantity storage unit 110, an imaging unit 120, a second local feature quantity generation unit 130, and a recognition unit 140.

  The first local feature amount storage unit 110 is generated for each of the m local regions including each of the m feature points in the image of the product and the part 111 attached to the product. The m first local feature quantities 112 composed of feature vectors up to dimensions are stored in association with the product or part 111.

  The imaging unit 120 images the entire product to which the component is attached. The second local feature value generation unit 130 extracts n feature points 131 from the video imaged by the imaging unit 120, and each of the n local regions 132 including the n feature points 131 is one-dimensional. To j-dimensional feature vectors n second local feature values 133 are generated.

  The recognizing unit 140 adds the n second local feature amounts 133 including the feature vectors up to the j dimension generated by the second local feature amount generating unit 130 to the j dimension stored in the first local feature amount storage unit 110. When it is determined that a predetermined ratio or more of the m first local feature values 112 composed of the feature vectors corresponds, it is recognized that the product or part 111 exists in the video.

  According to the above embodiment, the product can be checked accurately and immediately at the production stage, and the product quality can be improved.

[Second Embodiment]
Next, an information processing system according to the second embodiment of the present invention will be described. In the information processing system according to the present embodiment, the communication terminal generates a local feature amount from the captured video and transmits the local feature amount to the factory computer. The factory computer assembles the local feature amount in association with the assembled product or component. Identify products and parts, and manage assembly products and parts.

  According to this embodiment, since the direction and angle of each part can be recognized with high accuracy in real time, it is possible to quickly check a precise assembly product or part.

<Configuration of information processing system>
FIG. 2 is a block diagram illustrating a configuration of the information processing system 200 according to the present embodiment. An information processing system 200 in FIG. 2 is a production system that assembles parts to create a product.

  The product assembly area 202 in the factory 201 includes a component receiving gate 210, a component check line 220, assembly check lines 230-1 and 230-1, an assembled product check line 240, and a product shipment gate 250. . Further, it is assumed that a pre-repair check line 260 and a post-repair check line 270 exist in the product repair area 203. The present embodiment is not limited to the above processing, and is useful in real-time checking of assembling a plurality of parts.

(Parts receipt)
First, in the component storage gate 210, the communication terminal 291 images the storage component 212 that is unloaded from the transport vehicle and stored in the factory 201, and generates a local feature amount from the image. The generated local feature amount is sent to the factory computer 280. In the factory computer 280, the warehousing product 212 is identified from the local feature amount and the warehousing process is performed. In the parts storage gate 210, the storage part 212 to be imaged is a packing case such as a cardboard box.

(Parts check)
Next, in the component check line 220, the communication terminal 292 captures the assembly component 222 or 223 carried by the belt conveyor to generate a local feature amount, and the factory computer 280 includes a wrong component. Check for it. According to the present embodiment, it is possible to recognize whether the individual parts 222 and 223 are facing in any direction, at any angle, and even if a part is hidden by other parts.

(Assembly check)
Next, in the assembly check line 230-1, the communication terminal 293 captures the assembly parts 232 and 233 in which the parts 222 and 223 are assembled, generates local feature amounts, and does not mistake the parts 222 and 223. Check in real time whether it is assembled in the correct positional relationship. Next, in the assembly check line 230-2, the communication terminal 294 captures the product and assembly part being assembled to generate local feature amounts, and the assembly of the product from the assembly parts 232 and 233 is performed without error. It is checked from the direction and arrangement of the assembly parts.

(Product check)
In the product check line 240, the communication terminal 295 captures the product and its parts to generate local features, and the factory computer 280 checks the product assembled from the assembly parts 232 and 233. The factory computer 280 checks in real time whether the configuration of each part is correct, the assembly part is correct, and whether the product and its part arrangement are correct.

(Product shipment)
In the product shipment gate 250, the communication terminal 296 takes a picture of the product 252 that is OK in the product check 240, generates a local feature, and manages the product shipment. Here again, the product 252 to be shipped is a packaging case such as a cardboard box, as in the case of parts receipt.

(Check before repair)
On the other hand, in the product and assembly part repair area 203, first, in the pre-repair check 260, the communication terminal 297 captures a product or assembly part 262 brought in due to a failure or periodic inspection and generates a local feature amount. Check whether there is a failure or where the failure is due to the appearance or distortion of the part, or the position movement of the part. In this way, it is possible to determine a faulty part or the like that is difficult for a worker to make a normal product or a normal assembly part by photographing. In addition, since the failure location can be recognized in real time, it is possible to automatically process what parts are required for repair, whether the parts are in stock, and how long it is possible to receive parts if they are not in stock. .

(Check after repair)
In the post-repair check 270, similar to the product check 240, the communication terminal 298 captures the product and its parts 272, generates local features, and checks the repaired product and the assembly parts and parts for repair. Determine that has succeeded.

  As described above, the parts receiving gate 210, the parts check line 220, the assembly check lines 230-1 and 230-2, the product check line 240, the product shipment gate 250, the pre-repair check line 260, and the post-repair check. The line 270 can be realized in a unified manner in real time only by imaging by the communication terminals 291 to 298, respectively.

<< Operation procedure of information processing system >>
FIG. 3A is a sequence diagram showing an operation procedure of the information processing system 200 according to the present embodiment.

  First, in step S300, if necessary, an application and / or data for executing this embodiment is downloaded from the factory computer 280 to the communication terminals 291 to 298 in each area. In the operation of the present embodiment, the application is activated and initialized in step S301.

(Parts check)
The communication terminal 292 that performs the component check 220 photographs the components 222 and 223 in step S303. In step S305, a local feature amount is generated. Next, in step S307, the generated local feature and feature point coordinates are encoded. In step S309, the encoded local feature value and feature point coordinates are transmitted from the communication terminal 292 to the factory computer 280.

  In the factory computer 280, in step S311, whether or not the local feature received from the communication terminal 292 has a local feature stored in the local feature database 310 in association with a part, an assembly part, or a product is checked. Is done. A part is recognized from the result of such collation, and it is checked whether it is a correct part. In step S313, if a recognition result and an error are detected from the factory computer 280 to the communication terminal 292, error information is returned.

  In step S315, the communication terminal 292 displays that the error information is displayed if there is an error and the recognition result received from the factory computer 280. It is also notified which part is.

  On the other hand, the factory computer 280 refers to the inventory management database 330 in step S317 to perform parts inventory management processing. The inventory management database 330 may be installed in the factory in association with the factory computer 280, or may be set in association with a management server (not shown) that manages a plurality of factories via a network. .

(Assembly check)
Next, the communication terminals 293 and 294 that perform the assembly checks 230-1 and 230-2 photograph the assembly parts 232 and 233 or the assembly product 235 in step S321. In step S323, a local feature amount is generated. Next, in step S325, the generated local feature and feature point coordinates are encoded. In step S327, the encoded local feature amount and feature point coordinates are transmitted from the communication terminals 293 and 294 to the factory computer 280.

  In the factory computer 280, whether or not the local feature amount received from the communication terminal 293 or 294 in step S329 has a local feature amount stored in the local feature amount database 310 in association with the component, the assembly component, or the product. Are matched. Further, the product / part relationship is checked with reference to the product / part database 340 in which the compatibility and positional relationship between the product and the part are stored. A part is recognized from the result of such collation, and it is checked whether it is a correct part, whether the arrangement of the part is correct, whether the direction is correct, or whether the combination of the parts is correct in software. In step S331, if a recognition result and an error are detected from the factory computer 280 to the communication terminals 293 and 294, error information is returned.

  In step S333, the communication terminals 293 and 294 display error information if there are any recognition results and errors received from the factory computer 280, and the provided parts are not normal parts that are wrong or have scratches. In addition to being notified of which part it is, it is also notified of an error in the arrangement of the parts in the product.

(Product check)
Next, the communication terminal 295 that performs the product check 240 photographs the product 242 in step S341. In step S343, a local feature amount is generated. Next, in step S345, the generated local feature and feature point coordinates are encoded. In step S347, the encoded local feature and feature point coordinates are transmitted from the communication terminal 295 to the factory computer 280.

  In step S349, the factory computer 280 checks whether or not the local feature received from the communication terminal 295 has a local feature stored in the local feature database 310 in association with each part, assembly component, or product. Is done. The feature of the image of the entire product is recognized from the result of such collation, and it is checked whether it is a correct product, whether the arrangement of the parts is correct, the direction is correct, and the combination of parts is correct in software. In step S351, if a recognition result and an error are detected from the factory computer 280 to the communication terminal 295, error information is returned.

  In step S353, the communication terminal 295 displays error information if there is an error and the recognition result received from the factory computer 280, and is notified that there is a slight difference between the product and the normal part.

<Functional configuration>
FIG. 3B is a block diagram illustrating an internal functional configuration of the communication terminals 291 to 298. The communication terminals 291 to 298 have an imaging unit 311 such as a digital video camera. The video acquired by the imaging unit 311 is sent to the image development unit 313. The image expanding unit 313 expands the acquired video image, and then outputs the expanded image to the local feature value generating unit 320. The local feature quantity generation unit 320 extracts n feature points from the developed image, and includes n 1-dimensional to j-dimensional feature vectors for n local regions each including the n feature points. n local feature values are generated and sent to the encoding unit 321. The encoding unit 321 encodes the local feature amount and passes it to the communication control unit 323. The communication control unit 323 sends the encoded local feature amount to the factory computer 280. The communication control unit 323 also receives the product and component recognition results from the factory computer 280 and passes them to the recognition result notification unit 325. The recognition result notification unit 325 notifies the received recognition result. Specifically, when a defect or a component installation error is detected as a product, an alarm is issued to notify the worker of which part of which product has which problem.

  FIG. 3C is a block diagram showing an internal functional configuration of the factory computer 280. The factory computer 280 includes a local feature database 310 and a product / part database 340. In addition, the factory computer 280 includes a communication control unit 331, a local feature amount decoding unit 332, a product / component recognition unit 333, a recognition result generation unit 334, and a component arrangement determination unit 335.

  The communication control unit 3311 receives the local feature amount encoded from the communication terminals 291 to 298 and passes it to the local feature amount decoding unit 332. The local feature amount decoding unit 332 decodes the encoded local feature amount and passes it to the product / part recognition unit 333. The product / part recognition unit 333 compares the local feature received via the network with the local feature stored in the local feature database 310 in advance and performs collation. The local feature values stored here are each composed of feature vectors from 1-dimensional to i-dimensional generated for each of m local regions including each of m feature points in the image of the object. There are m local features.

  The product / part recognition unit 333 selects a smaller number of dimensions from the dimension numbers i and j of the feature vectors of the two local feature values acquired from the local feature value decoding unit 332 and the local feature value database 310. Then, the product / part recognition unit 333 is the local feature amount acquired from the local feature amount decoding unit 335, n local feature amounts including feature vectors up to the selected number of dimensions, and the local feature amount database 310. Are compared with m local feature values composed of feature vectors up to the selected number of dimensions.

  If the product / part recognition unit 333 determines that these local feature quantities match at a predetermined ratio or more, the product / part recognition unit 333 recognizes that the product and the part are properly present in the captured image, and sends the recognition result to the recognition result generation unit 334. hand over. On the other hand, the component placement determination unit 335 also determines whether the placement of the components is correct based on the feature amount matching. The recognition result generation unit 334 generates a recognition result according to the determination result of the component placement determination unit 335 and transmits the recognition result to the communication terminals 291 to 298 via the communication control unit 331.

  For example, it is possible to determine whether an appropriate part has been received, whether a product in which the parts are properly combined can be shipped, or whether an appropriate part is sequentially placed and attached at an appropriate position in the product assembly process. Alternatively, it is possible to determine whether the type, number, arrangement, etc. of the parts are correct before and after the repair for the requested product.

<Configuration of local feature generator and encoder>
FIG. 4A is a block diagram showing configurations of local feature value generation section 320 and encoding section 321. The local feature quantity generation unit 320 includes a feature point detection unit 401, a local region acquisition unit 402, a sub region division unit 403, a sub region feature vector generation unit 404, and a dimension selection unit 405. The feature point detection unit 401 detects a number of characteristic points (feature points) from the image data, and outputs the coordinate position, scale (size), and angle of each feature point. The local region acquisition unit 402 acquires a local region for performing feature amount extraction from the detected coordinate value, scale, and angle of each feature point.

  The sub area dividing unit 403 divides the local area into sub areas. For example, the sub-region dividing unit 403 can divide the local region into 16 blocks (4 × 4 blocks) or divide the local region into 25 blocks (5 × 5 blocks). The number of divisions is not limited. Hereinafter, a case where the local area is divided into 25 blocks (5 × 5 blocks) will be described as a representative.

  The sub region feature vector generation unit 404 generates a feature vector for each sub region of the local region. As the feature vector of the sub-region, for example, a gradient direction histogram can be used.

  The dimension selection unit 405 selects (for example, thins out) a dimension to be output as a local feature amount based on the positional relationship between the sub-regions so that the correlation between feature vectors of adjacent sub-regions becomes low. In addition, the dimension selection unit 405 can determine not only the dimension but also the selection priority. That is, for example, the dimension selecting unit 405 can select dimensions with priorities so that the dimensions in the same gradient direction are not selected between adjacent sub-regions. Then, the dimension selection unit 405 outputs a feature vector composed of the selected dimensions as a local feature amount. Note that the dimension selection unit 405 can output the local feature amount in a state where the dimensions are rearranged based on the priority order.

  The encoding unit 321 includes a coordinate value scanning unit 407 that inputs the coordinates of feature points from the feature point detection unit 401 of the local feature quantity generation unit 320 and scans the coordinate values. The coordinate value scanning unit 407 scans the image according to a specific scanning method, and converts the two-dimensional coordinate values (X coordinate value and Y coordinate value) of the feature points into one-dimensional index values. This index value is a scanning distance from the origin according to scanning. There is no restriction on the scanning direction.

  The encoding unit 321 also includes a sorting unit 408 that sorts the index values of feature points and outputs permutation information after sorting. Here, the sorting unit 408 sorts, for example, in ascending order. You may also sort in descending order.

  In addition, the encoding unit 321 includes a difference calculation unit 409 that calculates a difference value between two adjacent index values in the sorted index value and outputs a series of difference values.

  The encoding unit 321 further includes a difference encoding unit 410 that encodes a sequence of difference values in sequence order. The sequence of the difference value may be encoded with a fixed bit length, for example. When encoding with a fixed bit length, the bit length may be specified in advance, but this requires the number of bits necessary to express the maximum possible difference value, so the encoding size is small. Don't be. Therefore, when encoding with a fixed bit length, the differential encoding unit 410 can determine the bit length based on the input sequence of difference values. Specifically, for example, the difference encoding unit 410 obtains the maximum value of the difference value from the input series of difference values, obtains the number of bits (expression bit number) necessary to express the maximum value, A series of difference values can be encoded with the obtained number of expression bits.

  On the other hand, it has a local feature amount encoding unit 406 that encodes local feature amounts of corresponding feature points in the same permutation as the index values of the sorted feature points. By encoding with the same permutation as the sorted index value, it is possible to associate the coordinate value encoded by the differential encoding unit 410 and the corresponding local feature amount in a one-to-one relationship. The local feature encoding unit 406 may encode a local feature selected from the 150-dimensional local feature for one feature point, for example, one dimension with one byte and a number of dimensions. it can.

<< Local feature generation process >>
Next, the process of the local feature-value production | generation part 320 which concerns on this embodiment is demonstrated in detail using FIG. 4B-FIG. 4F.

  First, FIG. 4B is a diagram showing a series of processing of feature point detection / local region acquisition / sub-region division / sub-region feature vector generation in the local feature quantity generation unit 320. Such a series of processing is described in US Pat. No. 6,711,293, David G. Lowe, “Distinctive image features from scale-invariant key points” (US), International Journal of Computer Vision, 60 (2), 2004. See Years, p. 91-110.

(Feature point detection)
The feature point detection unit 401 first detects a feature point 421 from an image in the video as shown in the upper left of FIG. 4B. Hereinafter, the generation of local feature amounts will be described by using one feature point 421 as a representative. Each feature point 421 is accompanied by an arrow having a direction and a magnitude of a change in value of luminance, saturation, hue, and the like of surrounding pixels as elements. Moreover, although the example of FIG. 4B demonstrates the example quantized to any of six directions of 60 degree | times intervals, it is not limited to this.

(Local area acquisition)
Next, as shown in the upper right of FIG. 4B, the local area acquisition unit 402 generates a Gaussian window 422a around the feature point 421 and generates a local area 422 that substantially includes the Gaussian window 422a, for example. . In the example of FIG. 4B, the local area acquisition unit 402 generates a square local area 422, but the local area may be circular or have another shape. This local region is acquired for each feature point. If the local area is circular, there is an effect that the robustness is improved with respect to the imaging direction.

(Sub-region division)
The state sub-region dividing unit 403 divides the scale and angle of each pixel included in the local region 422 of the feature point 421 into sub-regions 423 as shown in the lower left of FIG. 4B. In FIG. 4B, 4 × 4 = 16 pixels are used as one sub-region, and the local region 422 is divided into a total of 5 × 5 = 25 sub-regions. However, the method of dividing into sub-regions and the number thereof are not limited to this.

(Sub-region feature vector generation)
As shown in the lower right of FIG. 4B, the sub-region feature vector generation unit 404 generates and quantizes a histogram of the scale of each pixel in the sub-region in units of angles in six directions to obtain a sub-region feature vector 424. . That is, the direction is normalized with respect to the angle output by the feature point detection unit 401. Then, the sub-region feature vector generation unit 404 aggregates the frequencies in the six directions quantized for each sub-region, and generates a histogram. In this case, the sub-region feature vector generation unit 404 outputs a feature vector constituted by a histogram of 25 sub-region blocks × 6 directions = 150 dimensions generated for each feature point. In addition, the gradient direction is not limited to 6 directions, but may be quantized to an arbitrary quantization number such as 4 directions, 8 directions, and 10 directions. When the gradient direction is quantized in the D direction, if the gradient direction before quantization is G (0 to 2π radians), the quantized value Qq (q = 0,..., D−1) in the gradient direction is For example, although it can obtain | require by Formula (1), Formula (2), etc., it is not restricted to this.

Qq = floor (G × D / 2π) (1)
Qq = round (G × D / 2π) mod D (2)
Here, floor () is a function for rounding off the decimal point, round () is a function for rounding off, and mod is an operation for obtaining a remainder. Further, when generating the gradient histogram, the sub-region feature vector generation unit 404 may add up the magnitudes of the gradients instead of adding up the simple frequencies. In addition, when the sub-region feature vector generation unit 404 adds up the gradient histogram, the sub-region feature vector generation unit 404 assigns weight values not only to the sub-region to which the pixel belongs, but also to sub-regions adjacent to each other (such as adjacent blocks) according to the distance between the sub-regions. You may make it add. Further, the sub-region feature vector generation unit 404 may add weight values to gradient directions before and after the quantized gradient direction. Note that the feature vector of the sub-region is not limited to the gradient direction histogram, and may be any one having a plurality of dimensions (elements) such as color information. In the present embodiment, it is assumed that a gradient direction histogram is used as the feature vector of the sub-region.

(Dimension selection)
Next, the dimension selection process performed by the dimension selection unit 405 will be described with reference to FIGS. 4C to 4F.

  The dimension selection unit 405 selects (decimates) a dimension (element) to be output as a local feature amount based on the positional relationship between the sub-regions so that the correlation between feature vectors of adjacent sub-regions becomes low. More specifically, the dimension selection unit 405 selects a dimension so that, for example, at least one gradient direction is different between adjacent sub-regions. In the present embodiment, the dimension selection unit 405 mainly uses adjacent subregions as adjacent subregions. However, the adjacent subregions are not limited to adjacent subregions, for example, from the target subregion. A sub-region within a predetermined distance may be a nearby sub-region.

  FIG. 4C shows an example of selecting a dimension from a feature vector 431 of a 150-dimensional gradient histogram generated by dividing a local region into 5 × 5 block sub-regions and quantizing gradient directions into six directions 431a. FIG. In the example of FIG. 4C, dimensions are selected from feature vectors of 150 dimensions (5 × 5 = 25 sub-region blocks × 6 directions).

  As illustrated in FIG. 4C, the dimension selection unit 405 selects a feature vector 432 of a half 75-dimensional gradient histogram from a feature vector 431 of a 150-dimensional gradient histogram. In this case, dimensions can be selected so that dimensions in the same gradient direction are not selected in adjacent left and right and upper and lower sub-region blocks.

  In this example, when the quantized gradient direction in the gradient direction histogram is q (q = 0, 1, 2, 3, 4, 5), a block for selecting elements of q = 0, 2, 4 and , Q = 1, 3, and 5 are alternately arranged with sub-region blocks for selecting elements. In the example of FIG. 4C, when the gradient directions selected in the adjacent sub-region blocks are combined, there are six directions.

  Further, the dimension selection unit 405 selects a feature vector 433 of a 50-dimensional gradient histogram from the feature vector 432 of the 75-dimensional gradient histogram. In this case, the dimension can be selected so that only one direction is the same (the remaining one direction is different) between the sub-region blocks positioned at an angle of 45 degrees.

  In addition, when the dimension selection unit 405 selects the feature vector 434 of the 25-dimensional gradient histogram from the feature vector 433 of the 50-dimensional gradient histogram, the gradient direction selected between the sub-region blocks located at an angle of 45 degrees. Dimension can be selected so that does not match. In the example shown in FIG. 4C, the dimension selection unit 405 selects one gradient direction from each sub-region from the first dimension to the 25th dimension, selects two gradient directions from the 26th dimension to the 50th dimension, and starts from the 51st dimension. Three gradient directions are selected up to 75 dimensions.

  In this way, it is desirable that the gradient directions are selected uniformly so that the gradient directions do not overlap between adjacent sub-region blocks. At the same time, as in the example shown in FIG. 4C, it is desirable that dimensions be selected uniformly from the entire local region. Note that the dimension selection method illustrated in FIG. 4C is an example, and is not limited to this selection method.

(Local area priority)
FIG. 4D is a diagram illustrating an example of the selection order of feature vectors from sub-regions in the local feature value generation unit 320.

  The dimension selection unit 405 can determine the priority of selection so as to select not only the dimension but also the dimension that contributes to the feature point feature in order. That is, for example, the dimension selecting unit 405 can select dimensions with priorities so that dimensions in the same gradient direction are not selected between adjacent sub-area blocks. Then, the dimension selection unit 405 outputs a feature vector composed of the selected dimensions as a local feature amount. Note that the dimension selection unit 405 can output the local feature amount in a state where the dimensions are rearranged based on the priority order.

  That is, the dimension selection unit 405 adds dimensions in the order of the sub-region blocks as shown in the matrix 441 in FIG. 4D, for example, between 1 to 25 dimensions, 26 dimensions to 50 dimensions, and 51 dimensions to 75 dimensions. It may be selected. When the priority order shown in the matrix 441 in FIG. 4D is used, the dimension selection unit 405 can select the gradient direction by increasing the priority order of the sub-region blocks close to the center.

  A matrix 451 in FIG. 4E is a diagram illustrating an example of element numbers of 150-dimensional feature vectors in accordance with the selection order in FIG. 4D. In this example, 5 × 5 = 25 blocks are represented by numbers p (p = 0, 1,..., 25) in raster scan order, and the quantized gradient direction is represented by q (q = 0, 1, 2, 3). , 4, 5), the element number of the feature vector is 6 × p + q.

  A matrix 461 in FIG. 4F is a diagram showing that the 150-dimensional order according to the selection order in FIG. 4E is hierarchized in units of 25 dimensions. That is, the matrix 461 in FIG. 4F is a diagram illustrating a configuration example of local feature amounts obtained by selecting the elements illustrated in FIG. 4E according to the priority order illustrated in the matrix 441 in FIG. 4D. The dimension selection unit 405 can output dimension elements in the order shown in FIG. 4F. Specifically, when outputting a 150-dimensional local feature amount, for example, the dimension selection unit 405 can output all 150-dimensional elements in the order shown in FIG. 4F. When the dimension selection unit 405 outputs, for example, a 25-dimensional local feature amount, the element 471 in the first row (76th, 45th, 83rd,..., 120th) shown in FIG. Can be output in the order shown in (from left to right). For example, when outputting a 50-dimensional local feature amount, the dimension selection unit 405 adds the elements 472 in the second row shown in FIG. 4F in the order shown in FIG. To the right).

  Incidentally, in the example shown in FIG. 4F, the local feature amount has a hierarchical structure. That is, for example, in the 25-dimensional local feature value and the 150-dimensional local feature value, the arrangement of the elements 471 to 476 in the first 25-dimensional local feature value is the same. In this way, the dimension selection unit 405 selects a dimension hierarchically (progressively), and thereby, according to the application, communication capacity, terminal specifications, etc., the local feature quantity of any number of dimensions, that is, the local size of any size. Feature quantities can be extracted and output. In addition, the dimension selection unit 405 can hierarchically select dimensions, rearrange the dimensions based on the priority order, and output them, thereby performing image matching using local feature amounts of different dimensions. . For example, when images are collated using a 75-dimensional local feature value and a 50-dimensional local feature value, the distance between the local feature values can be calculated by using only the first 50 dimensions.

  Note that the priorities shown in FIG. 4D from the matrix 441 to FIG. 4F are examples, and the order of selecting dimensions is not limited to this. For example, the order of blocks may be the order shown in the matrix 442 in FIG. 4D or the matrix 443 in FIG. 4D in addition to the example of the matrix 441 in FIG. 4D. Further, for example, the priority order may be set so that dimensions are selected from all the sub-regions. Also, the vicinity of the center of the local region may be important, and the priority order may be determined so that the selection frequency of the sub-region near the center is increased. Further, the information indicating the dimension selection order may be defined in the program, for example, or may be stored in a table or the like (selection order storage unit) referred to when the program is executed.

  In addition, the dimension selection unit 405 may select a dimension by selecting one sub-region block. That is, 6 dimensions are selected in a certain sub-region, and 0 dimensions are selected in other sub-regions close to the sub-region. Even in such a case, it can be said that the dimension is selected for each sub-region so that the correlation between adjacent sub-regions becomes low.

  Further, the shape of the local region and the sub-region is not limited to a square, and can be an arbitrary shape. For example, the local region acquisition unit 402 may acquire a circular local region. In this case, the sub-region dividing unit 403 can divide the circular local region into, for example, nine or seventeen sub-regions into concentric circles having a plurality of local regions. Even in this case, the dimension selection unit 405 can select a dimension in each sub-region.

As described above, as illustrated in FIGS. 4B to 4F, according to the local feature value generation unit 320 of this embodiment, the dimensions of the feature vectors generated while maintaining the information amount of the local feature values are hierarchically selected. The This processing enables real-time object recognition and recognition result display while maintaining recognition accuracy. Note that the configuration and processing of the local feature value generation unit 320 are not limited to this example. Naturally, other processes that enable real-time object recognition and recognition result display while maintaining recognition accuracy can be applied.
(Product / Parts Recognition Department)
4G and 4H are diagrams showing processing of the product / part recognition unit 333 according to the present embodiment.

  FIG. 4G is a diagram showing processing of the product / component recognition unit 333 at the time of the component check 220 in FIG. Local feature amounts generated according to the present embodiment from an image 478 of one component 222 or 223 shown in FIG. 4G in advance are stored in the local feature amount database 310. On the other hand, a local feature amount is generated according to the present embodiment from the video screen 471 captured by the communication terminal 292 in the left diagram of FIG. 4G. And it is collated whether the local feature-value stored in the local feature-value database 310 exists in the local feature-value produced | generated from the video screen 471.

  As shown in FIG. 4G, the product / part recognition unit 333 associates each feature point where the local feature quantity stored in the local feature quantity database 310 matches the local feature quantity like a thin line. Note that the product / part recognition unit 333 determines that the feature points match when a predetermined ratio or more of the local feature amounts match. The product / part recognition unit 333 recognizes the same product or the same part if the positional relationship between the associated feature point sets is a linear relationship. If such recognition is performed, it is possible to recognize by size difference, orientation difference (difference in viewpoint), or inversion. In addition, since the recognition accuracy can be obtained if there are a predetermined number or more of associated feature points, the product can be recognized even if a part of the feature points is hidden from view.

  In FIG. 4G, feature points that match a plurality of parts in the part tray 472 are connected from one local feature quantity generated from the video 478 and stored in the local feature quantity database 310. In FIG. 4G, thin lines are drawn for two parts, but if the same part is provided in the part tray 472, all parts are related like a thin line. In FIG. 4G, it can be determined that the component 475 is the wrong component among the plurality of symbols provided in the component tray 472.

  FIG. 4H is a diagram illustrating processing of the product / part recognition unit 333 at the time of the assembly check 230-2 and the product check 240 in FIG. Local feature amounts generated according to the present embodiment from videos 483 to 485 of one component 486 to 488 shown in FIG. 4H in advance are stored in the local feature amount database 310. On the other hand, a local feature amount is generated according to the present embodiment from the video screen 481 of the product 482 captured by the communication terminal 294 or 295 in the left diagram of FIG. 4H. Then, it is checked whether or not the local feature amount stored in the local feature amount database 310 is in the local feature amount generated from the video screen 481.

  As shown in FIG. 4H, the product / part recognition unit 333 associates each feature point where the local feature quantity stored in the local feature quantity database 310 matches the local feature quantity like a thin line. Note that the matching conditions of the feature points of the product / part recognition unit 333 are the same as those in the case of FIG. 4G, but may be adjusted in consideration of the precision and usage. Similar to FIG. 4G, if recognition is performed as in the present embodiment, recognition can be performed by size, direction difference (difference in viewpoint), or inversion. In addition, since the recognition accuracy can be obtained if there are a predetermined number or more of associated feature points, the product can be recognized even if a part of the feature points is hidden from view.

  In FIG. 4H, features that match a plurality of parts mounted on the product 482 from one local feature quantity generated from the images 483 to 485 of the parts 486 to 488 and stored in the local feature quantity database 310. Connect the dots. In FIG. 4H, thin lines are drawn from three parts to the parts of the product 482. However, all the parts on the product 482 or important parts are associated with each other as shown by thin lines. In FIG. 4H, if the three parts are correctly mounted at the correct position and the direction thereof is correct, it is determined that the product is a normal product.

  FIG. 5 is a diagram showing the contents of the local feature quantity database 310. The local feature quantity database 310 stores product / part IDs, product names, part names, and first to nth local feature quantities in association with each other.

  FIG. 6 is a diagram showing the contents of the product / part database 340. The product / part database 340 registers the product name, various parts used for the product, and their positions in association with the product ID.

  FIG. 7 is a diagram showing the contents of the inventory management database 330. The inventory management database 330 manages a part name, a warehousing history, a usage history, a stock quantity, a usage prediction, necessity of warehousing, and the number thereof in association with the part ID.

<< Hardware configuration and each process >>
FIG. 8 is a diagram illustrating a hardware configuration of the communication terminals 291 to 298. The CPU 810 is a processor for calculation control, and implements each functional component of the communication terminal by executing a program. The ROM 820 stores fixed data and programs such as initial data and programs. In addition, the communication control unit 440 communicates with other devices via a network. Note that the number of CPUs 810 is not limited to one, and may be a plurality of CPUs or may include a graphics processing unit (GPU) for image processing.

  The RAM 840 is a random access memory that the CPU 810 uses as a work area for temporary storage. The RAM 840 has an area for storing data necessary for realizing the present embodiment. The developed image data 841 is data input by the imaging unit 311. The feature point data 842 is data including feature point coordinates, scales, and angles detected from the developed image data. The local feature value generation table 843 is a table that stores data related to generation of local feature values.

  The storage 850 stores a database, various parameters, or the following data or programs necessary for realizing the present embodiment. The communication terminal control program 851 is a program that controls the entire communication terminal. The local feature value generation module 852 generates a local feature value from the input video according to FIGS. 4B to 4F. The encoding module 853 is a module that encodes local feature amounts, and functions as the encoding unit 321 when executed by the CPU 810. The result notification module 854 is a module for notifying the recognition result of the product / part, and functions as the recognition result notification unit 325 when executed by the CPU 810.

  The input / output interface 860 relays input / output data with the input / output device. A display unit 861, a touch panel 862, a speaker 864, a microphone 865, and an imaging unit 311 are connected to the input / output interface 860. The input / output device is not limited to the above example. In addition, a GPS (Global Positioning System) position generation unit 866 acquires a current position based on a signal from a GPS satellite.

(Local feature generation data)
FIG. 9 is a diagram showing a local feature value generation table 843 according to the present embodiment. The local feature amount generation table 843 stores a plurality of detected feature points, feature point coordinates, and local region information corresponding to the feature points in association with the input image ID. A selection dimension including a plurality of sub-region IDs, sub-region information, feature vectors corresponding to each sub-region, and priority order is stored in association with each detected feature point, feature point coordinates, and local region information. Further, the local feature amount derived for the input image is stored.

(Processing flow in communication terminals)
FIG. 10 is a flowchart for explaining the flow of processing in the communication terminal 211. First, in step S1011, when an image is input, local feature generation processing is performed in step S1013, and in step S1014, the local feature is encoded. Furthermore, in step S1015, the encoded local feature is transmitted to the factory computer 280. If authentication result data is received instead of inputting an image, the process advances from step S1011 to step S1021 to step S1025 to perform processing for notifying workers of product and component recognition results.

  FIG. 11A is a flowchart for explaining the flow of local feature generation processing performed in step S1013 of FIG. In step S1101, the feature point detection unit 401 first detects feature points. Next, in step S1103, the local region acquisition unit 402 acquires a local region of one feature point. Further, in step S1105, the sub-region dividing unit 403 divides the local region into sub-regions. In step S1107, the sub-region feature vector generation unit 404 generates a sub-region feature vector. Further, the dimension selection unit 405 selects a dimension in step S1109. In step S1111, it is determined whether the dimension selection processing has been completed for all feature points. If not, the process returns to step S1103.

  FIG. 11B is a flowchart illustrating the processing procedure of the encoding processing S1014 according to the present embodiment. First, in step S1131, the coordinate values of feature points are scanned in a desired order. Next, in step S1133, the scanned coordinate values are sorted. In step S1135, a coordinate difference value is calculated in the sorted order. In step S1137, the difference value is encoded. In step S1139, the local feature values are encoded in the coordinate value sorting order. The difference value encoding and the local feature amount encoding may be performed in parallel.

(Difference processing)
FIG. 11C is a flowchart illustrating a detailed processing procedure of difference value encoding processing S1137. First, in step S1141, it is determined whether or not the difference value is within a range that can be encoded. If it is in the range which can be encoded, it will progress to step S1147, will encode a difference value, and will transfer to step S1149. If it is not within the range that can be encoded (outside the range), the process proceeds to step S1143. After the escape code is encoded, the difference value is encoded in step S1145 by an encoding method different from the encoding in step S1147. Then, control goes to a step S1149. In step S1149, it is determined whether the processed difference value is the last element of the difference value series. If it is the last, the process ends. When it is not the last, it returns to step S1141 again and performs the process with respect to the next difference value of the series of difference values.

(Processing in the factory computer 280)
FIG. 12 is a flowchart for explaining the flow of processing in the factory computer 280. First, when a new local feature database needs to be generated (updated), such as when a new product is received, local feature database generation processing is performed from step S1211 to step S1213. If the local feature amount is received from the communication terminals 291 to 298 in the factory, the process proceeds from step S1221l to step S1223, product / part recognition processing is performed, and further, component placement recognition processing is performed in step S1225. In step S1227, the recognition result is returned to the communication terminals 291 to 298.

(Local feature database generation processing)
FIG. 13 is a flowchart showing a processing procedure of local feature database generation processing S1213 according to the present embodiment.

  First, in step S1301, images of products and parts are acquired. In step S1303, the position coordinates, scale, and angle of the feature point are detected. In step S1305, a local region is acquired for one of the feature points detected in step S1303. Next, when a local region of one feature point is acquired in step S1305, the local region is divided into sub-regions in step S1307. In step S1309, a feature vector for each sub-region is generated to generate a feature vector for the local region.

  Next, in step S1311, dimension selection is performed on the feature vector of the local region generated in step S1309. In generating the local feature quantity database 310, hierarchization is performed in dimension selection, but it is desirable to store all generated feature vectors.

  In step S1313, it is determined whether local feature generation and dimension selection have been completed for all feature points detected in step S1303. If not completed, the process returns to step S1305 to repeat the process for the next one feature point. If all feature points have been completed, the process advances to step S1315 to register local feature values and feature point coordinates in the local feature value database 310 in association with the recognized product.

  In step S1317, it is determined whether there is another product. If there is another product, the process returns to step S1301, and an image of the recognition target object is acquired and the process is repeated.

(Product / part recognition processing)
FIG. 14 is a flowchart showing the processing procedure of the product / part recognition processing S1223 according to this embodiment.

  First, in step S1401, the local feature amount of one product or part is acquired from the local feature amount database 310. In step S1403, the local feature amount of the product or part is compared with the local feature amount received from the communication terminal (see FIG. 15).

  In step S1405, it is determined whether or not they match. If they match, the process proceeds to step S1411 to store the matched commercial product or part and transmit it to the communication terminal.

  If they do not match, it is determined in step S1407 whether or not all products and parts have been collated, and if there is any remaining, the process returns to step S1401 to repeat collation of the next product or part. In such verification, the field may be limited in advance.

(Verification process)
FIG. 15 is a flowchart showing the processing procedure of the collation processing S1403 according to the present embodiment.

  First, in step S1531, parameters p = 1 and q = 0 are set as initialization. Next, in step S1533, the dimension number j of the local feature amount generated in step S1531 is acquired.

  In the loop of steps S1535 to S1545, the collation of each local feature amount is repeated until p> m (m = number of product feature points). First, in step S <b> 1535, the data of the dimension number j of the p-th local feature of the product stored in the local feature database 310 is acquired. That is, the j dimension is acquired from the first one dimension. Next, in step S1537, the p-th local feature amount acquired in step S1535 and the local feature amounts of all feature points generated from the input video are sequentially checked to determine whether or not they are similar. In step S1539, it is determined whether or not the similarity exceeds the threshold value α from the result of matching between the local feature quantities. If so, in step S1541, the local feature quantity and the matched feature points in the input video and the product are determined. The combination with the positional relationship is stored. Then, q, which is a parameter for the number of matched feature points, is incremented by one. In step S1543, the feature point of the product is advanced to the next feature point (p ← p + 1). If all feature points of the product have not been matched (p ≦ m), the process returns to step S1535 to match Repeat the feature verification. Note that the threshold value α can be changed according to the recognition accuracy required for each product. Here, if the recognition object has a low correlation with other products, accurate recognition is possible even if the recognition accuracy is lowered.

  When collation with all feature points of the product is completed, the process proceeds from step S1545 to S1547, and in steps S1547 to S1553, it is determined whether the product exists in the input video. First, in step S1547, it is determined whether or not the ratio of the feature point number q that matches the local feature amount of the feature point of the input video among the feature point number p of the product exceeds the threshold value β. If it exceeds, the process proceeds to step S1549, and it is further determined as a product candidate whether the positional relationship between the feature point of the input video and the feature point of the product has a relationship that allows linear transformation. That is, whether the positional relationship between the feature point of the input video and the feature point of the product stored as the local feature amount is matched in step S1541 is a positional relationship that can be achieved by changes such as rotation, inversion, and change of the viewpoint position. Determine if the positional relationship is impossible. Since such a determination method is geometrically known, detailed description thereof is omitted. If it is determined in step S1551 that the linear conversion is possible, the process proceeds to step S1553, where it is determined that the verified product exists in the input video. Note that the threshold value β can be changed according to the recognition accuracy required for each product. Here, accurate recognition is possible even if there are few matching feature points as long as the product has a low correlation with other products or a product whose characteristics can be judged even from a part. That is, even if a part is hidden and cannot be seen, or a characteristic part can be seen, the product can be recognized.

  In step S 1555, it is determined whether or not unmatched products remain in the local feature database 310. If there is still a product, the next product is set in step S1557, parameters p = 1 and q = 0 are initialized, and the process returns to step S1535 to repeat the verification.

  As is clear from the description of the collation processing, the processing for storing products in all fields in the local feature amount database 310 and collating all the products is very heavy. Therefore, for example, it is conceivable that the user selects a product field from a menu before product recognition from the input video, and searches and collates the field from the local feature amount database 310. Also, the load can be reduced by storing only the local feature amount of the field used by the user in the local feature amount database 310.

(Part placement recognition process)
FIG. 16 is a flowchart showing a processing procedure of component arrangement recognition processing S1225 according to the present embodiment.

  First, in step S1601, the arrangement information of the parts in the product is acquired from the product / parts DB 320, and collated with the positional relationship between the product and the part recognized from the local feature amount. In step S1603, it is determined whether or not they match. If they match, the process advances to step S1605 to set that the component arrangement in the product is OK.

  If they do not match, in step S1607, a correct component arrangement is searched from the product / component DB 320.

  According to the above embodiment, since the direction and angle of each part can be recognized with high accuracy in real time, it is possible to quickly check a precise assembly product or part.

[Third Embodiment]
FIG. 17 is a diagram for explaining the operation of the information processing system according to the third embodiment of the present invention. This embodiment differs from the second embodiment in that the matching accuracy is controlled according to the situation. Other configurations and operations are the same as those of the second embodiment, and thus description of the same configurations and operations is omitted.

  In FIG. 17, an electronic board 1711 that is a captured product is displayed on a display screen 1710 of the communication terminal 1701.

  A display screen 1720 in the center of FIG. 17 is a diagram when a local feature amount of the electronic substrate 1711 is generated with low accuracy (a limit with a small number of feature points and a small number of dimensions). A feature point and a local region 1721 including the feature point are indicated by a black circle. Note that the feature point and the local region 1721 including the feature point may or may not be displayed on the screen. It is assumed that the type, type, and the like of the electronic substrate 1711 are recognized by the accurate local feature amount shown on the central display screen 1720.

  As a result of the recognition of the type and type of the electronic substrate 1711, the display screen 1730 narrows the range of components such as IC chips mounted on the electronic substrate 1711. Under the condition that the number of points is large and the number of dimensions is large, local feature amounts are generated, and components such as an IC chip mounted on the electronic substrate 1711 are recognized. Since the component placement position on the product is also known as in the product / part DB described above, only the region of the placement position may be recognized with higher accuracy.

  By controlling in this way, it is possible to significantly reduce the local feature generation time and collation time.

[Fourth Embodiment]
Next, an information processing system according to the fourth embodiment of the present invention will be described. Compared with the third embodiment, the information processing system according to the present embodiment generates a limited and highly accurate local feature amount for a part having an abnormality or a part that needs to be checked, and checks the part. It is different in point. Other configurations and operations are the same as those of the second embodiment, and thus description of the same configurations and operations is omitted.

<Processing of information processing system>
FIG. 18 is a diagram illustrating the operation of the information processing system according to the fourth embodiment of the present invention.

  In FIG. 18, the processing from the display screen 1810 of the communication terminal 1801 to the third emergency screen 1830 in the left diagram is the same as in the third embodiment, and thus the description thereof is omitted.

  It is assumed that a problem has been found in the upper right IC chip in the electronic substrate 1811 due to the highly accurate local feature amount in the electronic substrate 1811 shown on the third display screen 1830 from the left.

  The fourth display screen 1840 of FIG. 18 includes feature points when local feature amounts are generated with maximum accuracy (maximum feature points and maximum dimensions) with respect to the IC chip on the upper right of the electronic substrate 1811, and the feature points. The local area 1841 is indicated by a black circle. In this way, it is possible to specify the IC chip and check with the maximum accuracy.

[Other Embodiments]
Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In addition, a system or an apparatus in which different features included in each embodiment are combined in any way is also included in the scope of the present invention.

  In addition, the present invention may be applied to a system composed of a plurality of devices, or may be applied to a single device. Furthermore, the present invention can also be applied to a case where a control program that realizes the functions of the embodiments is supplied directly or remotely to a system or apparatus. Therefore, in order to realize the functions of the present invention on a computer, a control program installed in the computer, a medium storing the control program, and a WWW (World Wide Web) server that downloads the control program are also included in the scope of the present invention. include.

  This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2011-276523 for which it applied on December 16, 2011, and takes in those the indications of all here.

A part or all of the present embodiment can be described as in the following supplementary notes, but is not limited thereto.
(Appendix 1)
M number of feature vectors, each consisting of 1-dimensional to i-dimensional feature vectors, generated for each of the m local regions including each of the m feature points in the image of the product and the part attached to the product. A first local feature quantity storage means for storing one local feature quantity in association with the product or the part;
Imaging means for imaging the entire product with the parts attached thereto;
N feature points are extracted from the image picked up by the image pickup means, and n local regions including the n feature points are respectively provided with n pieces of feature vectors from 1 to j dimensions. Second local feature quantity generating means for generating a second local feature quantity;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognition means for recognizing that the object exists;
An information processing system comprising:
(Appendix 2)
The information according to claim 1, wherein the recognizing unit recognizes that the product or the part is defective when a specific portion of the first local feature amount does not correspond to the second local feature amount. Processing system.
(Appendix 3)
The recognizing unit further recognizes the arrangement of the recognized part in the recognized product based on position information of the recognized product in the image and position information of the recognized part in the image. The information processing system according to appendix 1 or 2, characterized in that:
(Appendix 4)
The recognizing means has an arrangement storage means for storing the normal arrangement of the parts in the product, and when the arrangement of the parts in the recognized product is different from the normal arrangement stored in the arrangement storage means, The information processing system according to any one of appendices 1 to 3, wherein the information processing unit is recognized as an arrangement error of the component in the product or a selection error of the component.
(Appendix 5)
The information processing system includes a communication terminal, and an information processing apparatus connected to the communication terminal via a communication line,
The communication terminal includes the imaging means, the second local feature quantity generation means, and the notification means, and transmits the n second local feature quantities from the communication terminal to the information processing apparatus;
The supplementary note 4, wherein the information processing apparatus includes the first local feature amount storage means and the recognition means, and transmits the recognition result of the recognition means from the information processing apparatus to the communication terminal. Information processing system.
(Appendix 6)
The information processing system according to any one of appendices 1 to 5, further comprising management support means for supporting inventory management of the parts based on recognition of the parts by the recognition means.
(Appendix 7)
The first local feature amount and the second local feature amount include a histogram of gradient directions in the plurality of sub-regions obtained by dividing a local region including feature points extracted from an image or video into a plurality of sub-regions. The information processing system according to any one of supplementary notes 1 to 6, wherein the information processing system is generated by generating a multidimensional feature vector.
(Appendix 8)
The first local feature and the second local feature are generated by deleting a dimension having a larger correlation between adjacent sub-regions from the generated multi-dimensional feature vector. 8. The information processing system according to 7.
(Appendix 9)
The first local feature amount and the second local feature amount are generated by deleting feature points determined to have less feature information from the plurality of feature points extracted from an image. The information processing system according to appendix 7 or 8.
(Appendix 10)
A plurality of dimensions of the feature vector can be selected in order from the dimension that contributes to the feature of the feature point, and from the first dimension according to the improvement in accuracy required for the local feature amount, for each predetermined number of dimensions The information processing system according to any one of appendices 8 and 9, wherein the information is arranged so as to make a round in the local region.
(Appendix 11)
The second local feature quantity generation means corresponds to the level of correlation between the product or the part, and the product or part having the higher correlation with respect to another product or part has a dimension number of The information processing system according to appendix 11, wherein more second local feature quantities are generated.
(Appendix 12)
The first local feature storage means has a higher number of dimensions for a product having the higher correlation with respect to other products or parts, corresponding to the level of correlation between the products or the parts. 12. The information processing system according to appendix 10 or 11, wherein the first local feature is stored.
(Appendix 13)
An identification means for identifying the component by reading an identifier added to the component;
The recognizing unit determines the direction of the recognized part and recognizes the position of the identifier added to the part based on the direction. Information processing system.
(Appendix 14)
An imaging step for imaging the entire product with the parts attached;
N feature points are extracted from the image captured in the imaging step, and n local feature regions each including the n feature points are each composed of feature vectors from 1 to j dimensions. A second local feature generation step of generating the second local feature of
Each of the m local regions including each of the m feature points in the image of the object is made up of feature vectors from 1D to iD that are generated in advance and stored in the first local feature storage unit. a reading step of reading m first local feature values from the first local feature value storage means;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognizing that the object is present;
An information processing method comprising:
(Appendix 15)
An imaging step for imaging the entire product with the parts attached;
N feature points are extracted from the image captured in the imaging step, and n local feature regions each including the n feature points are each composed of feature vectors from 1 to j dimensions. A second local feature generation step of generating the second local feature of
Each of the m local regions including each of the m feature points in the image of the object is made up of feature vectors from 1D to iD that are generated in advance and stored in the first local feature storage unit. a reading step of reading m first local feature values from the first local feature value storage means;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognizing that the object is present;
An information processing program for causing a computer to execute.

Claims (13)

M number of feature vectors, each consisting of 1-dimensional to i-dimensional feature vectors, generated for each of the m local regions including each of the m feature points in the image of the product and the part attached to the product. A first local feature quantity storage means for storing one local feature quantity in association with the product or the part;
Imaging means for imaging the entire product with the parts attached thereto;
N feature points are extracted from the image picked up by the image pickup means, and n local regions including the n feature points are respectively provided with n pieces of feature vectors from 1 to j dimensions. Second local feature quantity generating means for generating a second local feature quantity;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognition means for recognizing that the object exists;
Equipped with a,
The first local feature amount and the second local feature amount include a histogram of gradient directions in the plurality of sub-regions obtained by dividing a local region including feature points extracted from an image or video into a plurality of sub-regions. Generated by generating multi-dimensional feature vectors,
The first local feature quantity and the second local feature quantity are generated by deleting a dimension having a larger correlation between adjacent sub-regions from the generated multi-dimensional feature vector. Processing system.   The said recognition means recognizes that the said product or the said part has a defect when the specific part of the said 1st local feature-value does not respond | correspond with the said 2nd local feature-value. Information processing system.   The recognizing unit further recognizes the arrangement of the recognized part in the recognized product based on position information of the recognized product in the image and position information of the recognized part in the image. The information processing system according to claim 1 or 2.   The recognizing means has an arrangement storage means for storing the normal arrangement of the parts in the product, and when the arrangement of the parts in the recognized product is different from the normal arrangement stored in the arrangement storage means, The information processing system according to any one of claims 1 to 3, wherein the information processing system recognizes a misplacement of the parts in the product or a misselection of the parts. The information processing system includes a communication terminal, and an information processing apparatus connected to the communication terminal via a communication line,
The communication terminal includes the imaging means, the second local feature quantity generation means, and a notification means, and transmits the n second local feature quantities from the communication terminal to the information processing apparatus.
The information processing apparatus includes the first local feature amount storage unit and the recognition unit, and transmits a recognition result of the recognition unit from the information processing apparatus to the communication terminal. The information processing system described.   6. The information processing system according to claim 1, further comprising management support means for supporting inventory management of the parts based on recognition of the parts by the recognition means. The first local feature amount and the second local feature amount are generated by deleting feature points determined to have less feature information from the plurality of feature points extracted from an image. The information processing system according to claim 1 . A plurality of dimensions of the feature vector can be selected in order from the dimension that contributes to the feature of the feature point, and from the first dimension according to the improvement in accuracy required for the local feature amount, for each predetermined number of dimensions. The information processing system according to claim 7 , wherein the local area is arranged so as to make a round. The second local feature quantity generation means corresponds to the level of correlation between the product or the part, and the product or part having the higher correlation with respect to another product or part has a dimension number of The information processing system according to claim 8 , wherein more second local feature quantities are generated. The first local feature storage means has a higher number of dimensions for a product having the higher correlation with respect to other products or parts, corresponding to the level of correlation between the products or the parts. The information processing system according to claim 8 or 9 , wherein the first local feature is stored. An identification means for identifying the component by reading an identifier added to the component;
The recognition means determines the direction of the component recognized, according to any one of claims 1 to 10, characterized in that to recognize the position of the identifier added to the component on the basis of the direction Information processing system. An imaging step for imaging the entire product with the parts attached;
N feature points are extracted from the image captured in the imaging step, and n local feature regions each including the n feature points are each composed of feature vectors from 1 to j dimensions. A second local feature generation step of generating the second local feature of
Each of the m local regions including each of the m feature points in the image of the object is made up of feature vectors from 1D to iD that are generated in advance and stored in the first local feature storage unit. a reading step of reading m first local feature values from the first local feature value storage means;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognizing that the object is present;
Only including,
The first local feature amount and the second local feature amount include a histogram of gradient directions in the plurality of sub-regions obtained by dividing a local region including feature points extracted from an image or video into a plurality of sub-regions. Generated by generating multi-dimensional feature vectors,
The first local feature quantity and the second local feature quantity are generated by deleting a dimension having a larger correlation between adjacent sub-regions from the generated multi-dimensional feature vector. Processing method. An imaging step for imaging the entire product with the parts attached;
N feature points are extracted from the image captured in the imaging step, and n local feature regions each including the n feature points are each composed of feature vectors from 1 to j dimensions. A second local feature generation step of generating the second local feature of
Each of the m local regions including each of the m feature points in the image of the object is made up of feature vectors from 1D to iD that are generated in advance and stored in the first local feature storage unit. a reading step of reading m first local feature values from the first local feature value storage means;
A smaller dimension number is selected from among the dimension number i of the feature vector of the first local feature quantity and the dimension number j of the feature vector of the second local feature quantity, and the feature vector includes up to the selected dimension number. When it is determined that the n second local feature values correspond to a predetermined ratio or more of the m first local feature values composed of feature vectors up to the selected number of dimensions, Recognizing that the object is present;
To the computer ,
The first local feature amount and the second local feature amount include a histogram of gradient directions in the plurality of sub-regions obtained by dividing a local region including feature points extracted from an image or video into a plurality of sub-regions. Generated by generating multi-dimensional feature vectors,
Wherein the first local feature quantity and the second local feature quantity is information from a multi-dimensional feature vectors described above generated, wherein Rukoto generated by the correlation between adjacent sub-regions to remove a larger dimension Processing program.

Images