JP5351084B2 - Image recognition apparatus and image recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image recognition device capable of accurately detecting an object without an increase in calculation amount even in the recognition of a complicated object, to perform subsequent processing such as the extraction of a feature quantity or pattern recognition. <P>SOLUTION: The image recognition device 1 includes an imaging unit 21 for imaging a recognition object to generate recognition image data; a display unit 23 which displays a recognition image based on the recognition image data generated by the imaging unit 21; an input unit 22 for a user to input recognition position input data to instruct a position of an element to be recognized to the recognition image; a recognition feature quantity extraction unit 31 which extracts a recognition image feature quantity from the recognition image data based on the recognition position input data inputted to the input unit 22; and a pattern recognition unit 32 which recognizes the recognition object based on the recognition image feature quantity extracted by the extraction unit 31. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image recognition apparatus and an image recognition method, and more particularly to an image recognition apparatus and an image recognition method for recognizing an object shown in a captured image.

  2. Description of the Related Art Conventionally, an image recognition device that analyzes an image taken by an imaging device such as a digital camera and recognizes an object shown there is widely known. For example, a code reader such as a QR code reader or a device that recognizes what it is from the image of a book cover is known.

  In general, these apparatuses perform processing on an image in the order of (1) target detection (segmentation), (2) feature amount extraction, and (3) pattern identification (recognition processing). A series of these processes are automated. When an input image is given, the above procedures (1) to (3) are performed by the software of the computer, and the recognition result is output. Here, when the shape of the recognition target is regular and the image recognition process can be easily applied as in a code reader, the above-described “target detection” process is easy.

  On the other hand, in recent years, a system for recognizing more complicated objects such as birds, flowers, dogs, cats, and insects has been proposed (for example, see Non-Patent Document 1 for a system that recognizes flowers). ). In such a system, in order to improve recognition accuracy, it is necessary to accurately perform “target detection”.

Further, if the target can be detected for each individual part, recognition with higher accuracy can be performed. For example, when recognizing a bird, if the position of the head, wings, trunk, eyes, and beak can be detected individually in the “target detection” process, The accuracy of the recognition result after “feature amount extraction” and “pattern identification” processing can be significantly increased. Therefore, in order to recognize a complex target with high accuracy, it is indispensable to first perform “target detection” appropriately.
As prior art documents related to the present invention, there are the following documents.

Nilsback, M-E. And Zisserman, A. `` A Visual Vocabulary for Flower Classification '' Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2006) G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray.Visual categorization with bags of keypoints.In Workshop on Statistical Learning in Computer Vision, ECCV, pages 1-22, 2004.

  However, when recognizing a complex object such as a bird, a flower, a dog, a cat, or an insect, for example, it is not easy to detect the object and its individual parts with a computer for the following reason.

  The first reason is the variety of appearances of recognition objects. That is, when recognizing a QR code, a book cover, etc., the recognition target is a flat surface, and by photographing it from the front, the appearance of the captured image becomes uniform, but the recognition target is a bird, flower, dog, etc. In the case of a three-dimensional object such as a cat or an insect, the appearance of the recognition target in the captured image varies in various ways depending on the angle at the time of shooting. Further, the appearance of the recognition target in the photographed image varies in various ways depending on the photographing conditions such as illumination conditions and zooming. Furthermore, when the recognition target is an animal or plant such as a bird, a flower, a dog, a cat, or an insect, there are individual differences. For example, even if the birds are the same species, the shape, color, and the like are different. Therefore, it is not easy to detect a recognition target or individual parts thereof from a photographed image by computer software.

  The second reason is that the boundary to be detected as the recognition target or its individual parts on the image may be unclear. In other words, when performing “target detection” with a computer, in general, the recognition target or its individual parts are detected by dividing the region by extracting the edge portion of the image, but the recognition target is photographed. In an image, there may be little color change at the boundary between the recognition target and its background, or between adjacent parts of the recognition target. In such a case, the computer is adjacent to the recognition target and its background or recognition target. It becomes difficult to detect the boundary between parts.

  The third reason is that for objects such as birds, flowers, dogs, cats, and insects, there is not necessarily a true solution in “target detection”. For example, when the recognition target is a bird, eyes and beaks can be defined relatively clearly, and it is clear from where the eyes are in the bird's body and where the beak is from. However, for example, for the wings and trunks, their boundaries cannot be uniquely defined in the bird's body, and their solutions vary from person to person and do not exist uniquely. Thus, a task for which no true solution exists is not suitable for a computer.

  The fourth reason is that the calculation amount becomes enormous. In the processes (1) to (3) described above, (1) “target detection” has a particularly large amount of calculation. In general, “detecting an object” for a complex object requires complicated processing, and thus consumes a large amount of computing resources. Therefore, it is difficult to realize “target detection” for complex objects such as birds, flowers, dogs, cats, and insects on hardware with low resources.

  Therefore, the present invention provides an image recognition apparatus and image recognition that can detect a target accurately without increasing the amount of calculation and perform subsequent processing such as feature amount extraction and pattern identification even when a complex target is recognized. Provide a method.

  The present invention does not automatically perform “target detection”, but allows the user to input the position of the recognition target (and its individual parts) via the input unit, so that the task that the computer is not good at is substituted for humans. Then, the detection result of this recognition target (and its individual parts) is used for subsequent processing to recognize the target.

  One aspect of the present invention is an image recognition apparatus, which includes an imaging unit for capturing a recognition target and generating recognition image data, and a recognition image generated by the imaging unit. A display unit that displays an image for recognition based on data, an input unit for a user to input position input data for recognition indicating the position of the element to be recognized with respect to the image for recognition, and A recognition feature amount extraction unit that extracts a recognition image feature amount from the recognition image data based on the recognition position input data input to the input unit, and a recognition feature extracted by the recognition feature amount extraction unit A recognition unit for recognizing the recognition target based on the image feature amount.

  According to this configuration, the user performs “target detection” among the processes of “target detection”, “feature amount extraction”, and “pattern identification” necessary for recognition. By performing “target detection” based on the user's cognitive ability, even if the appearance of the target is diverse, and even if the edges of the boundary of the target or individual parts of the target are unclear, it is possible to accurately detect the target. Can be detected. Further, even when a true solution does not necessarily exist for “target detection”, “target detection” can be performed. Furthermore, the amount of calculation does not increase because of these.

  In the image recognition device of the present invention, the recognition unit is configured to recognize the recognition image extracted by the recognition feature amount extraction unit based on a learning result obtained using a relationship between the learning image feature amount and the correct label. The recognition target is recognized by specifying the correct label of the recognition target from the feature amount. Here, the learning image feature amount is an image feature amount extracted from the learning image data corresponding to the learning position input data based on the learning position input data.

  With this configuration, by using the learning result, the boundary cannot be uniquely defined, such as a bird's wing or a trunk when the recognition target is a bird, and the position specified by the position input data Even when the user is different depending on the user, the target can be appropriately recognized according to the learning result. In particular, when recognition based on learning is performed, the accuracy of recognition using the learning result increases as the number of learning image feature quantities for generating the learning result increases. Therefore, by using learning results generated from a large number of learning image feature amounts prepared in advance, it becomes possible to appropriately recognize an object whose boundary is not easily determined.

  The image recognition apparatus of the present invention is further based on the learning database in which the learning image data, the learning position input data, and the correct label are stored in association with each other, and the learning position input data. A learning feature amount extraction unit that extracts the learning image feature amount from the learning image data associated therewith, and the learning result using the relationship between the learning image feature amount and the correct answer label. And a learning unit for acquiring

  With this configuration, learning results are not stored in advance in the image recognition device, or learning results based on a sufficient number of learning image feature quantities are not obtained in advance in the image recognition device, and the image recognition device However, even if the learning result cannot be obtained from the outside by the communication function, the learning result can be obtained on the image recognition apparatus side, and therefore the recognition target can be appropriately recognized using the learning result.

  In the image recognition apparatus of the present invention, a plurality of learning position input data is input to one learning image data, and the one learning image data is based on the plurality of learning position input data. A plurality of learning image feature amounts are extracted by extracting image feature amounts respectively from the above, and the recognition unit is obtained using a relationship between the plurality of learning image feature amounts and the correct answer label. The recognition target is recognized based on the learning result.

  With this configuration, since the recognition target is recognized based on a plurality of learning results, the recognition accuracy can be improved.

  In the image recognition apparatus according to the present invention, the plurality of learning image feature quantities may be the combined learning image generated by combining the plurality of learning position input data input to the one learning image data. An image feature amount extracted from learning image data corresponding to the combined learning position input data based on the position input data is included.

  With this configuration, learning results can be acquired from more learning image feature quantities, and the recognition accuracy can be further improved.

  The image recognition apparatus of the present invention further includes a position for generating the combined learning position input data by combining the plurality of learning position input data input to the one learning image data. An input data synthesis unit is provided.

  With this configuration, even when the user shoots a subject to generate learning image data and inputs learning position input data thereto, a learning result is obtained based on such learning position input data. Therefore, the learning image feature amount can be increased.

  In the image recognition device of the present invention, the input unit is a touch panel.

  With this configuration, the user can input recognition position input data by tracing or touching the display unit with a fingertip or the like on the recognition image displayed on the display unit.

  In the image recognition apparatus of the present invention, the recognition position input data is data indicated by a closed curve, a point, a line, or a combination thereof.

  With this configuration, an area, a point, or a line can be designated as the recognition position input data.

  In the image recognition apparatus of the present invention, the recognition target elements are the whole recognition target and individual parts of the recognition target.

  With this configuration, not only the entire recognition target but also individual parts of the recognition target can be specified by the recognition position input data, so that the recognition accuracy can be improved.

  In the image recognition device according to the present invention, the recognition feature quantity extraction unit is configured to generate a plurality of the recognition image data by performing a plurality of times of imaging for the one recognition target by the imaging unit. When the plurality of recognition position input data for the plurality of recognition image data is input from the user, the plurality of recognition image feature quantities are extracted based on the plurality of recognition position input data. The recognizing unit recognizes the one recognition target based on the plurality of recognition image feature quantities.

  With this configuration, even if the characteristic part of the recognition target cannot be photographed in a single photographing, if the characteristic part can be photographed in any of a plurality of photographing, the feature Since the recognition target can be recognized based on the specific part, the recognition accuracy can be improved.

  In the image recognition apparatus of the present invention, the recognition unit recognizes the one recognition target based on one recognition image feature amount obtained by integrating the plurality of recognition image feature amounts.

  With this configuration, it is possible to recognize one recognition target by a plurality of shootings.

  In the image recognition apparatus of the present invention, the one recognition image feature amount is an average of the plurality of recognition image feature amounts.

  With this configuration, it is possible to recognize one recognition target by a plurality of shootings with a simple process.

  Another aspect of the present invention is an image recognition system that recognizes a recognition target from image data obtained by photographing, and this image recognition system includes a user interface device and a recognition device. The user interface device includes an imaging unit that captures a recognition target and generates recognition image data, a display unit that displays a recognition image based on the recognition image data generated by the imaging unit, An input unit for a user to input recognition position input data that indicates the position of the element to be recognized with respect to the recognition image; recognition image data generated by the imaging unit; and A data transmitting unit that transmits the input position input data for recognition to the recognition device, and a recognition result receiving unit that receives a recognition result by the recognition device for the image data for recognition transmitted by the data transmitting unit. I have. The recognition device includes: a data receiving unit that receives recognition image data and recognition position input data transmitted from the data transmission unit; and the data based on the recognition position input data received by the data receiving unit. A recognition feature amount extraction unit that extracts a recognition image feature amount from the recognition image data received by the reception unit; and the recognition based on the recognition image feature amount extracted by the recognition feature amount extraction unit. A recognition unit for recognizing a target; and a recognition result transmission unit for transmitting a recognition result by the recognition unit to the user interface device.

  Even with this configuration, as in the image recognition apparatus described above, by performing “target detection” based on the user's cognitive ability, even if the appearance of the target is diverse, the boundary of the target or the part of the target Even if the edges are unclear, it is possible to detect the target accurately, and even if there is not necessarily a true solution for “target detection”, “target detection” can be performed. In addition, the calculation amount does not increase.

  Still another aspect of the present invention is a mobile terminal, which includes an imaging unit for capturing a recognition target and generating recognition image data, and recognition image data generated by the imaging unit. A display unit that displays a recognition image based on the image, an input unit for a user to input recognition position input data that indicates a position of the recognition target element with respect to the recognition image, and the imaging Recognition data generated by the unit and a data transmission unit that transmits the recognition position input data input thereto to the recognition device, and the recognition device for the recognition image data transmitted by the data transmission unit And a recognition result receiving unit for receiving the recognition result by.

  Even with this configuration, as in the image recognition apparatus described above, by performing “target detection” based on the user's cognitive ability, even if the appearance of the target is diverse, the boundary of the target or the part of the target Even if the edges are unclear, it is possible to detect the target accurately, and even if there is not necessarily a true solution for “target detection”, “target detection” can be performed. In addition, the calculation amount does not increase.

  Yet another aspect of the present invention is a program, which stores a recognition image based on recognition image data generated by photographing a recognition target in the imaging unit on a portable terminal including the imaging unit. A display step for displaying; an input step for allowing a user to input recognition position input data for instructing a position of the recognition target element with respect to the recognition image; and the recognition image data and the input for the recognition image data. A data transmission step for transmitting the position recognition position input data to the recognition device; and a recognition result reception step for receiving a recognition result by the recognition device for the recognition image data transmitted in the data transmission step.

  Even with this configuration, as in the image recognition apparatus described above, by performing “target detection” based on the user's cognitive ability, even if the appearance of the target is diverse, the boundary of the target or the part of the target Even if the edges are unclear, it is possible to detect the target accurately, and even if there is not necessarily a true solution for “target detection”, “target detection” can be performed. In addition, the calculation amount does not increase.

  Still another aspect of the present invention is an image recognition method, which includes an imaging step for capturing a recognition target and generating image data for recognition, and recognition generated in the imaging step. A display step of displaying a recognition image based on the image data; an input step in which the user inputs recognition position input data for indicating a position of the element to be recognized with respect to the recognition image; A recognition feature quantity extracting step for extracting a recognition image feature quantity from the recognition image data based on the recognition position input data inputted in the input step, and a recognition extracted in the recognition feature quantity extraction step. And a recognition step of recognizing the recognition target based on the image feature quantity.

  Even with this configuration, as in the image recognition apparatus described above, by performing “target detection” based on the user's cognitive ability, even if the appearance of the target is diverse, the boundary of the target or the part of the target Even if the edges are unclear, it is possible to detect the target accurately, and even if there is not necessarily a true solution for “target detection”, “target detection” can be performed. In addition, the calculation amount does not increase.

  According to the present invention, during image recognition, the user performs “target detection”, so that the target can be detected accurately even for complex recognition targets, and the amount of calculation required for “target detection” is reduced. be able to.

The figure which shows the whole structure of the image recognition apparatus of embodiment of this invention 1 is a front external view of an image recognition apparatus according to an embodiment of the present invention. 1 is a rear external view of an image recognition apparatus according to an embodiment of the present invention. The figure explaining the example of the some image for recognition of embodiment of this invention, and the position input data for recognition with respect to it The figure which shows the image for learning of embodiment of this invention, and several position input data for learning with respect to it The figure which shows the example of a synthesis | combination of the position input data for learning of embodiment of this invention The figure which shows the structure of the learning database of embodiment of this invention The flowchart which shows the operation | movement of the image recognition in the image recognition apparatus of embodiment of this invention. The flowchart which shows the operation | movement which inputs the position input data for recognition of embodiment of this invention The figure which shows the example of the display screen at the time of imaging | photography in embodiment of this invention The figure which shows the example of the display screen at the time of inputting the position input data for recognition (whole body) in embodiment of this invention The figure which shows the example of the display screen at the time of inputting the position input data for recognition (torso) in embodiment of this invention The figure which shows the example of the display screen at the time of inputting the position input data for recognition (eye) in embodiment of this invention The figure which shows the example of the display screen at the time of inputting the position input data for recognition (foot) in embodiment of this invention The figure which shows the example of the display screen at the time of selecting whether imaging again in embodiment of this invention

  Hereinafter, an image recognition apparatus according to an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a recognition target is a bird, and an image recognition apparatus that captures a bird and obtains the name of the type of the bird will be described as an example. However, the present invention can be similarly applied to recognition objects other than birds. The image recognition apparatus of the present invention is also effective when used for recognizing animals and plants such as flowers, dogs, cats and insects, as well as artificial objects such as temples and cars.

  FIG. 1 is a diagram showing an overall configuration of an image recognition apparatus according to the present embodiment. The image recognition apparatus 1 includes a user interface unit 2 and a learning recognition unit 3. In the present embodiment, among the processes of “target detection”, “feature amount extraction”, and “pattern identification” necessary for image recognition, “target detection” is performed by the user interface unit 2. Extraction ”and“ pattern identification ”are performed by the learning recognition unit 3.

  The user interface unit 2 includes an imaging unit 21, an input unit 22, a display unit 23, an image storage unit 24, and a position input data storage unit 25. The imaging unit 21 captures a subject, generates display image data, and outputs the display image data to the display unit 23. When imaging is instructed by the user (shutter is pressed), the imaging unit 21 generates image data and outputs the image data to the image storage unit 22 (hereinafter, this image data is referred to as “recognition image data”. An image displayed based on the recognition image data is referred to as a “recognition image”). The input unit 23 receives position input and other inputs from the user.

  When the user inputs position input data to the recognition image displayed on the display unit 23, the input unit 22 outputs the position input data to the display unit 23 (hereinafter, the position input data is recognized as “recognition”. "Position input data"). In the image recognition apparatus 1 according to the present embodiment, the position input data for recognition has six elements: the whole bird body, the trunk, two legs, and two eyes. The input unit 22 outputs the position input data for recognition to the position input data storage unit 25 when the end of the position input data for recognition is instructed by the user. The display unit 23 displays an image based on the display image data, the recognition image data, and the position input input to the input unit 22. The display unit 23 also displays various input buttons. The image storage unit 24 stores the recognition image data generated by the imaging unit 21. The position input data storage unit 25 stores the recognition position input data received from the input unit 22.

  The learning recognition unit 3 includes a recognition feature value extraction unit 31, a pattern identification unit 32, a learning unit 33, a learning feature value extraction unit 34, and a learning database 35. The recognition feature quantity extraction unit 31 is based on the recognition position input data stored in the position input data storage unit 25, and the recognition image data corresponding to the recognition position input data stored in the image storage unit 24. The image feature amount is extracted from the image feature amount (hereinafter, the image feature amount extracted from the recognition image data is referred to as “recognition image feature amount”). In the learning database 35, for each of the plurality of pieces of learning image data, correct labels, a plurality of position input data by a plurality of input persons, and position input data created by morphing are stored in association with each other ( Hereinafter, the position input data stored in the learning database 35 is referred to as “learning position input data”).

  The learning feature amount extraction unit 34 extracts image feature amounts from the learning image data based on the learning position input data stored in the learning database 35 (hereinafter, extracted from the learning image data). The image feature amount is referred to as “learning image feature amount”). The learning unit 33 associates each learning image feature amount extracted by the learning feature amount extraction unit 34 with the corresponding correct answer label extracted from the learning database 35 to acquire a learning result. The pattern identification unit 32 refers to the learning result obtained by the learning unit 33, performs pattern identification based on the recognition image feature amount extracted by the recognition feature amount extraction unit 31, and performs the recognition image feature. The correct label for the quantity is output as the recognition result.

  FIG. 2A is a front external view of the image recognition apparatus 1 of the present embodiment, and FIG. 2B is a rear external view of the image recognition apparatus 1 of the present embodiment. As shown in FIGS. 2A and 2B, the image recognition apparatus 1 is a mobile terminal (for example, a mobile phone). As shown in FIG. 2A, the image recognition apparatus 1 includes a touch panel 101 and a plurality of buttons 102 on the front. The touch panel 101 corresponds to the display unit 23 in FIG. 1 and also corresponds to the input unit 22. The button 102 also corresponds to the input unit 22. As shown in FIG. 2B, the image recognition device 1 includes a lens 103 of the imaging unit 21 on the back surface.

  FIG. 3 is a diagram illustrating an example of a plurality of recognition images and recognition position input data corresponding thereto. The user captures the target O by pressing a capture button (shutter button) displayed on the touch panel 102 while the capturing unit 21 is capturing the target O. When the recognition image data is generated by shooting, the display unit 23 displays the recognition image based on the recognition image data. The user inputs recognition position input data to the recognition image displayed on the display unit 23 with a fingertip (others such as a stylus may be used. The same applies hereinafter). As described above, in the present embodiment, the user uses the recognition image displayed on the display unit 23 for recognition of the six elements such as the whole bird body, the trunk, two legs, and two eyes. Input position input data.

  When the user wants to recognize a certain target, the user can prepare a plurality of recognition image data having different shooting angles by shooting the target from a plurality of different angles. In FIG. 3, an image P1 is a recognition image obtained by photographing the object O from the front direction, and an image P2 is a recognition image obtained by photographing the object O from the diagonally left front direction, and the image P3. Is a recognition image obtained by photographing the object O from the diagonally backward left direction.

  In FIG. 3, D1 to D3 indicate recognition position input data input from the user to the recognition images P1 to P3, respectively. The user indicates the recognition image displayed on the display unit 23 by enclosing the entire body of the bird with one closed curve, indicating the trunk with one closed curve, and indicating the two legs with lines. Point and point each two eyes with a point. These closed curves, lines, and points serve as recognition position input data.

  If some elements such as feet and eyes do not appear in the recognition image, only the appearing elements are specified. In the example of FIG. 3, only one bird's eye is shown in the recognition images P2 and P3. Therefore, the recognition position input data D2 and D3 have only one eye recognition position input data. . However, each element of the recognition position input data is specified in at least one of a plurality of recognition position input data for a plurality of recognition image data.

  Even if the user cannot shoot the characteristic part of the object by one shooting, the user can shoot a plurality of times by changing the viewpoint. By capturing the object from a plurality of viewpoints and inputting the position input data for recognition of each element from each viewpoint, it is possible to improve the accuracy of the extraction of the feature amount in the subsequent stage and the pattern identification unit. For example, when a beak and a wing are included in the elements of the position input data for recognition, when recognizing a bird having an image feature amount that is very characteristic in the beak and wing pattern, the first recognition image is displayed. Even if the beak is visible, even if the wing pattern is not reflected well, if you take a picture so that the wings appear well in the second recognition image, you can get both characteristic image features. can do. In this way, the user can repeat the shooting many times until the characteristic part of the target can be sufficiently shot.

  The recognition position input data of each element input from the input unit 22 for one recognition image is stored in the position input data storage unit 25 as recognition position input data for the recognition image. Such recognition position input data is input to each of the plurality of recognition images, and the position input data storage unit 25 stores a plurality of recognition position input data for the plurality of recognition image data. When the input of the user's recognition position input data for the plurality of recognition images is completed, the user interface unit 2 associates the recognition image data with the recognition position input data, and sends them to the recognition feature quantity extraction unit 31. Output.

  The recognition feature quantity extraction 31 obtains a plurality of recognition image feature quantities for one recognition target by using a plurality of recognition image data and recognition position input data corresponding thereto. The recognition feature quantity extraction unit 31 takes the average of the plurality of recognition image feature quantities and sets it as the recognition image feature quantity of the recognition target. As described above, when inputting the position input data for recognition, the position input data for recognition is not input for some elements, and part of the position input data for recognition may be missing. In this case, the recognition image feature amount of that portion is also missing. In this case, the image feature quantity for recognition of the lacking part is not taken into consideration at the time of averaging.

  In the above description, a plurality of recognition image feature quantities are averaged to obtain one recognition image feature quantity. However, a plurality of recognition image feature quantities are integrated by a method other than averaging to obtain one recognition feature. The image feature amount may be obtained.

  Next, learning will be described. As described above, in the present embodiment, the user is made to input the recognition position input data for the six elements of the whole bird body, the trunk, the two legs, and the two eyes with respect to the recognition image. Depending on the input element, various interpretations may exist for each user. For example, when the user inputs the bird's trunk (chest and belly), the boundary line between the trunk, the head, and the back is ambiguous, and an accurate boundary cannot be defined. Therefore, the user inputs recognition position input data in various patterns according to each interpretation. Therefore, in the image recognition device 1 of the present embodiment, learning position input data by a plurality of input persons is collected in advance for one learning image and stored in the learning database 35 as independent learning samples. Keep it. Thereby, the learning feature amount extraction unit 34 can extract different learning image feature amounts using the learning position input data rich in variations.

  FIG. 4 is a diagram showing a learning image and a plurality of learning position input data corresponding thereto. In the example of FIG. 4, the body learning position input data DA is input by the input person A to the learning image SP, the body learning position input data DB is input by the input person B, and the input person C inputs the learning image SP. The body learning position input data DC is input. The input persons A, B, and C interpret the same area for the bird's torso with respect to the same learning image. In this way, a plurality of learning image feature quantities are acquired by preparing a plurality of learning position input data by a plurality of input persons for one learning image data.

  FIG. 5 is a diagram illustrating an example of synthesis of learning position input data. In general, the more learning samples, the better the recognition accuracy. However, collecting learning position input data manually is a difficult task. Therefore, in the present embodiment, in order to increase the number of learning position input data, the learning position input data input by a plurality of input persons are combined to obtain combined learning position input data. The example of FIG. 5 shows an example in which two pieces of learning position input data, that is, learning position input data DA by the input person A and learning position input data DB by the input person B are synthesized for one piece of learning image data. ing. In this way, by combining two of the plurality of learning position input data input by a plurality of input persons, a plurality of combined learning position input data is obtained.

  As a method for synthesizing two closed curves (or lines), a morphing technique for generating an intermediate shape by associating characteristic portions of these closed curves (or lines) is adopted. The technique for matching the characteristic parts of two closed curves is Scott, C .; Nowak, R. “Robust Contour Matching via the Order Preserving Assignment Problem”, IEEE Transactions on Image Processing, Volume 15, Issue 7, July 2006 Page ( s): 1831-1838.

  FIG. 6 is a diagram showing the configuration of the learning database 35. As shown in FIG. 6, the learning database 35 includes a plurality of learning position input data input by a plurality of input persons and a plurality of learning positions synthesized by morphing for each of the plurality of learning image data. Input data is stored in advance. Further, in the learning database 35, a correct answer label is attached to each learning image data.

  Next, extraction of image feature amounts in the recognition feature amount extraction unit 31 and the learning feature amount extraction unit 34 will be described. The recognition feature quantity extraction unit 31 receives the recognition position input data and the recognition image data as inputs, extracts a feature vector, and sets the feature vector as a recognition image feature quantity. Similarly, the learning feature quantity extraction unit 34 extracts the feature vector by using the learning position input data and the learning image data as input, and uses the feature vector as the learning image feature quantity. For the feature vector extraction, any feature extraction technique used in the field of image processing can be employed. In this embodiment, the histogram expression of Visual Word proposed in Non-Patent Document 2 described above is used. By using this method, it is possible to extract a feature vector having a certain length independent of the area of the region from the partial region on the image.

At this time, the area to be converted into the Visual Word histogram representation is set as follows for the position input data. When the position input data is a closed curve, the partial area surrounded by the closed curve is converted into a histogram expression of Visual Word. Thereby, an N _one- dimensional feature vector is obtained. When the position input data is a point, a partial region having a radius r around the point is converted into a histogram expression of Visual Word. Thereby, an N _two- dimensional feature vector is obtained. As the radius r, a fixed value may be used. When a closed curve indicating the whole body is input as in the bird example, a value normalized by the size of the closed curve is set as r. Good. By normalizing in this way, a feature vector that does not depend on the scale of the object in the image can be obtained. When the position input data is a line, a partial area having a width w centered on the line is converted into a histogram expression of Visual Word. Thereby, an N _three- dimensional feature vector is obtained. For the width w, a fixed value may be used. When a closed curve indicating the whole body is input as in the bird example, the value normalized by the size of the closed curve is w. Good. By normalizing in this way, a feature vector that does not depend on the scale of the object in the image can be obtained.

The recognition feature quantity extraction unit 31 and the learning feature quantity extraction unit 34 obtain the feature vectors of each element as described above, and arrange these feature vectors vertically to constitute a high-dimensional feature vector. In the case of the bird recognition according to the present embodiment, there are two closed curves, two points, and two lines. Therefore, an N = 2 × N ₁ + 2 × N ₂ + 2 × N _three- dimensional vector is used as a feature vector. Is obtained. The recognition feature quantity extraction unit 31 and the learning feature quantity extraction unit 34 output the N-dimensional feature vector as an image feature quantity.

  Note that the method of obtaining the image feature amount based on the position input data is not limited to the above example. For example, a method other than the method using the histogram expression of Visual Word may be used. Further, for example, information (shape and size) of the shape of the closed curve may be included in the image feature amount. In addition, as described above, if there are two closed curves as the position input data, a feature amount representing a relative positional relationship between them may be included in the image feature amount. Similarly, the relative positional relationship between a closed curve, a line, and a point may be included as an image feature amount.

  The learning unit 33 and the pattern identification unit 32 obtain a learning result by a support vector machine or a neural network, and recognize the N-dimensional feature vector extracted by the recognition feature amount extraction unit 31 by using the learning result. Get results. When the support vector machine is employed, the learning result by the learning unit 33 is a support vector, and when the neural network is employed, the learning result by the learning unit 33 is a weight of the neural network.

  The operation of the image recognition apparatus 1 configured as described above will be described. FIG. 7 is a flowchart showing an image recognition operation in the image recognition apparatus 1. When the user captures a recognition target, the user operates the user interface unit 2 to input recognition position input data using the image data of the captured image as recognition image data (step S71). The image recognition apparatus 1 extracts a recognition image feature amount from the recognition image data based on the recognition position input data (step S72). Next, the image recognition apparatus 1 performs pattern identification using the extracted image feature quantity for recognition, and recognizes a recognition target (step S73).

  FIG. 8 is a flowchart showing an operation of inputting recognition position input data. 9A to 9F are display examples of the display screen of the image recognition apparatus 1 when inputting the position input data for recognition. First, the imaging unit 21 of the image recognition apparatus 1 captures a subject, and the display unit 23 displays a display image (step S81). FIG. 9A is an example of the display screen at this time. A shooting button B1 is displayed on the display screen. The image recognition device 1 determines whether the shooting button B1 has been pressed (step S82). The display unit 23 displays the display image captured by the imaging unit 21 until the shooting button B1 is pressed. When the photographing button B1 is pressed (YES in step S82), the imaging unit 21 generates recognition image data and stores it in the image storage unit 24, and the display unit 23 displays the recognition image based on the recognition image data. indicate.

  When the recognition image is displayed on the display unit 23, the user inputs recognition position input data for the recognition image. First, the user inputs a closed curve that includes the entire body and a closed curve that includes the trunk (step S83). FIG. 9B is an example of a display screen when a closed curve including the entire body is input, and FIG. 9C is an example of a display screen when a closed curve including the body is input. On the display screen when the user inputs the position input data for recognition, buttons B2 to B5 for specifying an element for inputting the position input data for recognition are displayed. When inputting the position input data for recognition of the whole body, the user selects the whole body by pressing the whole button B2 and inputs a closed curve including the whole body, as shown in FIG. 9B. Specifically, when the whole button B2 is pressed, a line can be drawn by tracing the screen with a fingertip, so that the user surrounds the whole body of the bird with a line. When the fingertip is removed from the screen, the start point and end point of the curve are connected and a closed curve is automatically created. This closed curve becomes the position input data for recognition of the whole body.

  When the user inputs the body recognition position input data, as shown in FIG. 9C, the user selects the body by pressing the body button B3 and inputs a closed curve including the body. Specifically, when the torso button B3 is pressed, a line can be drawn by tracing the screen with a fingertip, so the user surrounds the bird's torso (the part including the belly and chest) with a line. When the fingertip is removed from the screen, the start point and end point of the curve are connected and a closed curve is automatically created. This closed curve becomes the position input data for body recognition.

  The user presses the eye button B4 to select an eye, and designates the two eyes of the bird with points (step S84). FIG. 9D is an example of the display screen at this time. Specifically, when the eye button B4 is pressed, a point is drawn by touching the screen with a fingertip, and thus the user touches both the bird's eyes with the fingertip. These two points serve as eye recognition position input data.

  The user further presses the foot button B5 to select the foot, and designates the two feet of the bird by lines (step S85). FIG. 9E is an example of the display screen at this time. Specifically, when the foot button B5 is pressed, a line can be drawn by tracing the screen with a fingertip, so the user traces both of the bird's legs with a line. These two lines become foot recognition position input data. Note that the input of recognition position input data for each element in steps S83, S84, and S85 may be performed in any order.

  When the input of the recognition position input data for the recognition image is completed, it is determined whether or not the same subject is to be shot again (step S86). FIG. 9F is an example of the display screen at this time. On the display screen of FIG. 9F, the user presses the “Yes” button B6 when shooting again, and presses the “No” button B7 when not shooting again. If another shooting is instructed (YES in step S86), the process returns to step S81 and the imaging unit 21 captures the subject and waits for the shooting button B1 to be pressed. When the image capturing is not performed again (NO in step S86), the input of the recognition position input data is terminated, and the recognition position input data input so far is output (step S87).

  As mentioned above, although embodiment of this invention was described, said description is only an illustration and this invention is implemented also in other embodiment.

  For example, in the above embodiment, the buttons B2 to B5 for designating elements for inputting the position input data for recognition are displayed on the screen for inputting the position input data for recognition, and the user presses any button. Although it is configured to select an element and input position input data for recognition of the element, for example, the image recognition apparatus may designate an element for inputting position input data for recognition to the user. In this case, the image recognition device can, for example, “enclose the entire body with a closed curve”, “enclose the torso with a closed curve”, “touch the eyes (two)”, “foot (2 “Please trace this book with a line” or the like, and display the text on the display unit 23 or read it out using a device such as a speaker to input the position input data for recognizing the element corresponding to the user. Can be urged. In addition, the screen for inputting the position input data for recognition may include a user interface for assisting the input work such as a clear button for initializing the input contents and an end button for ending the input work.

  In the above embodiment, the learning recognition unit 3 includes the learning database 35 and the learning feature amount extraction unit 34 used for learning. However, the image recognition apparatus of the present invention may not include these. Good. That is, in the above embodiment, the learning image 35, the learning position input data, and the correct label are stored in the learning database 35 in association with each other, and the learning feature amount extraction unit 34 based on the learning position input data. The image feature amount is obtained from the learning image, and the learning unit 33 obtains the learning result by associating the image feature amount with the correct answer label. However, these processes are performed outside the image recognition apparatus 1 in advance, thereby The obtained learning result may be stored in the learning unit 33 in advance. Moreover, the image recognition apparatus 1 may acquire such a learning result from the outside by communication. With this configuration, the image recognition apparatus 1 does not need to include the learning database 35 and the learning feature amount extraction unit 34.

  On the other hand, in the above-described embodiment, the learning database 35 stores learning images and learning position input data input by a plurality of input persons in advance, but these data are stored in the image recognition device. 1 may be generated. In this case, the user interface unit 2 shoots a subject to be recognized in the same manner as described above to generate image data, and the user inputs position input data and a correct answer label for the image data. This image data becomes learning image data, and this position input data becomes learning position input data. The learning image data and the learning position input data are input to the learning feature amount extraction unit 34 or stored in the learning database 35 and then input to the learning feature amount extraction unit 34. The learning feature amount extraction unit 34 extracts image feature amounts in the same manner as described above. The user further gives a correct answer label to the learning image data via the input unit 21. The correct answer label may be given by a method of selecting from correct answer labels in the learning database 35 or may be directly input by the user. The learning unit 33 acquires the learning result by associating the extracted image feature quantity with the correct answer label.

  The position recognition data for learning may be synthesized by the image recognition apparatus 1. In this case, the image recognition apparatus 1 includes a position input data synthesis unit. The position input data combining unit combines a plurality of learning position input data stored in the learning database 35 for the same learning image as described above, and generates new learning position input data.

  In the above-described embodiment, the example in which the user interface unit 2 and the learning recognition unit 3 are both provided in the mobile terminal to configure the image recognition device 1 has been described. However, the embodiment of the present invention is not limited thereto. Not limited. For example, the user interface unit 2 may be provided in a mobile terminal, and the learning recognition unit 3 may be provided in another computer that can communicate with the mobile terminal. In this case, the image data for recognition obtained by the user using the imaging unit 21 and the position input data for recognition input from the input unit 22 by the user are transmitted to the other computer via the communication network. Sent. The other computer performs image recognition based on the recognition image data and the recognition position input data transmitted from the portable terminal, and transmits the recognition result to the portable terminal. The portable terminal receives this recognition result.

  In the above embodiment, the user interface unit 2 asks the user whether or not to take another shot after inputting the recognition position input data for one shot, and whether or not the user himself takes another shot. I decided. Thus, the user can repeat the input of the shooting and recognition position input data until he / she thinks that he / she is sufficient. However, the user may not be able to determine how long to repeat the input of shooting and recognition position input data. Therefore, the user interface unit 2 is notified to the user whether or not sufficient position input data for recognition has been obtained, or if sufficient position input data for recognition has not been obtained, the user is forced to repeat the operation. You may add the function to perform imaging | photography. As described above, in order to obtain a recognition image feature quantity and perform pattern identification, at least one recognition position input data is required for each element to be recognized. Therefore, in this function, the user is guided whether or not at least one recognition position input data is obtained for all elements, or is compulsory until at least one recognition position input data is obtained for all elements. Can cause the user to take another picture.

  In the above embodiment, the image recognition apparatus for recognizing birds has been described as an example. However, the image recognition apparatus of the present invention may recognize other animals and plants such as insects, flowers, and fish. For example, when recognizing an insect, the position input data may be data in which a foot is designated by six lines and the head, chest and abdomen are designated by closed curves. When the recognition target is a flower, the petal can be designated by a closed curve, the center of the flower can be designated by a point, and the stem can be designated by a line as position input data. When the recognition target is a fish, the whole body, the back and tail fins can be designated by closed curves, and the data designated by the eyes can be used as position input data. The image recognition apparatus according to the present invention may further recognize an artifact such as a temple or a car.

  The present invention allows the user to perform “target detection” during image recognition, so that the target can be accurately detected even for complex recognition targets, and the amount of calculation required for “target detection” can be reduced. The present invention has an excellent effect that it can be applied, and can be applied to an image recognition device that recognizes an object in a captured image.

DESCRIPTION OF SYMBOLS 1 Image recognition apparatus 2 User interface part 21 Imaging part 22 Input part 23 Display part 24 Image preservation | save part 25 Position input data preservation | save part 3 Learning recognition part 31 Recognizing feature-value extraction part 32 Pattern identification part 33 Learning part 34 Learning feature-value Extraction unit 35 Learning database 101 Touch panel 102 Button 103 Lens

Claims (12)

An imaging unit for capturing a recognition target and generating image data for recognition;
A display unit for displaying a recognition image based on the recognition image data generated by the imaging unit;
An input unit for a user to input recognition position input data that indicates the position of the recognition target element with respect to the recognition image;
A recognition feature quantity extraction unit that extracts a recognition image feature quantity from the recognition image data based on the recognition position input data input to the input unit;
A recognition unit for recognizing the recognition target based on the recognition image feature amount extracted by the recognition feature amount extraction unit ;
The recognition unit corrects the correct answer of the recognition target from the recognition image feature amount extracted by the recognition feature amount extraction unit based on a learning result obtained using the relationship between the learning image feature amount and the correct answer label. By identifying the label, the recognition target is recognized,
The learning image feature amount is an image feature amount extracted from the learning image data corresponding to the learning position input data based on the learning position input data.
A plurality of pieces of learning position input data are input to one piece of the learning image data, and image feature amounts are extracted from the one piece of learning image data based on the plurality of pieces of learning position input data, respectively. By doing so, a plurality of the learning image feature quantities are extracted,
The recognition unit recognizes the recognition target based on a learning result obtained by using a relationship between the plurality of learning image feature quantities and the correct answer label.
Image recognition apparatus according to claim and this. A learning database in which the learning image data, the learning position input data, and the correct label are stored in association with each other;
Based on the learning position input data, a learning feature amount extraction unit that extracts the learning image feature amount from the learning image data associated therewith;
A learning unit that acquires the learning result using the relationship between the learning image feature quantity and the correct answer label;
The image recognition apparatus according to claim 1 , further comprising: The plurality of learning image feature amounts are based on the combined learning position input data generated by combining the plurality of learning position input data input to the one learning image data. The image recognition apparatus according to claim 1 , comprising an image feature amount extracted from learning image data corresponding to the combined learning position input data.   A position input data combining unit configured to generate the combined learning position input data by combining the plurality of learning position input data input to the one learning image data. The image recognition apparatus according to claim 3. Wherein the input unit, the image recognition apparatus according to any one of claims 1 to 4, characterized in that a touch panel. The recognition position input data, closed curve, a point, or line, or the image recognition apparatus according to any one of claims 1 to 5, characterized in that the data indicated by the combination thereof. The elements of the recognition target, the image recognition apparatus according to any one of claims 1 to 6, characterized in that an overall and individual parts of the recognition target of the recognition target. The recognition feature quantity extraction unit is configured to generate a plurality of pieces of recognition image data by performing a plurality of times of image capturing by the imaging unit with respect to one recognition target, and the input unit performs the plurality of recognitions from the user. When a plurality of recognition position input data for image data is input, a plurality of recognition image feature quantities are extracted based on the plurality of recognition position input data,
The image recognition apparatus according to claim 1, wherein the recognition unit recognizes the one recognition target based on the plurality of recognition image feature amounts. The image recognition apparatus according to claim 8 , wherein the recognition unit recognizes the one recognition target based on one recognition image feature amount obtained by integrating the plurality of recognition image feature amounts. The image recognition apparatus according to claim 9 , wherein the one recognition image feature amount is an average of the plurality of recognition image feature amounts. An image recognition system comprising a user interface device and a recognition device and recognizing a recognition target from image data obtained by photographing,
The user interface device includes:
An imaging unit for capturing a recognition target and generating image data for recognition;
A display unit for displaying a recognition image based on the recognition image data generated by the imaging unit;
An input unit for a user to input recognition position input data that indicates the position of the recognition target element with respect to the recognition image;
A data transmission unit that transmits the image data for recognition generated by the imaging unit and the position input data for recognition input thereto to the recognition device;
A recognition result receiving unit that receives a recognition result by the recognition device for recognition image data transmitted by the data transmission unit;
The recognition device is
A data receiving unit for receiving the recognition image data and the recognition position input data transmitted from the data transmission unit;
A recognition feature amount extraction unit that extracts a recognition image feature amount from recognition image data received by the data reception unit based on the recognition position input data received by the data reception unit;
A recognition unit for recognizing the recognition target based on the recognition image feature amount extracted by the recognition feature amount extraction unit;
A recognition result transmission unit that transmits a recognition result by the recognition unit to the user interface device ;
The recognition unit corrects the correct answer of the recognition target from the recognition image feature amount extracted by the recognition feature amount extraction unit based on a learning result obtained using the relationship between the learning image feature amount and the correct answer label. By identifying the label, the recognition target is recognized,
The learning image feature amount is an image feature amount extracted from the learning image data corresponding to the learning position input data based on the learning position input data.
A plurality of pieces of learning position input data are input to one piece of the learning image data, and image feature amounts are extracted from the one piece of learning image data based on the plurality of pieces of learning position input data, respectively. By doing so, a plurality of the learning image feature quantities are extracted,
The recognition unit is configured to recognize the recognition target based on a learning result obtained by using a relationship between the plurality of learning image feature quantities and the correct answer label . An imaging step for capturing a recognition target and generating image data for recognition;
A display step for displaying a recognition image based on the recognition image data generated in the imaging step;
An input step in which a user inputs recognition position input data for indicating the position of the recognition target element with respect to the recognition image;
A recognition feature amount extraction step for extracting a recognition image feature amount from the recognition image data based on the recognition position input data input in the input step;
On the basis of the recognition image feature quantity extracted by the recognition feature extraction step, and a recognition step of recognizing the recognition target,
In the recognition step, based on a learning result obtained by using a relationship between the learning image feature quantity and the correct answer label, the recognition target correct answer is extracted from the recognition image feature quantity extracted by the recognition feature quantity extracting unit. By identifying the label, the recognition target is recognized,
The learning image feature amount is an image feature amount extracted from the learning image data corresponding to the learning position input data based on the learning position input data.
A plurality of pieces of learning position input data are input to one piece of the learning image data, and image feature amounts are extracted from the one piece of learning image data based on the plurality of pieces of learning position input data, respectively. By doing so, a plurality of the learning image feature quantities are extracted,
The recognition step recognizes the recognition target based on a learning result obtained using a relationship between the plurality of learning image feature quantities and the correct answer label.
Image recognition method which is characterized a call.