JP2012203823A - Image recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image recognition device capable of reducing erroneous recognition in the case of recognizing an object, and suppressing the costs of recognition processing.SOLUTION: A recognition area selection part 12 selects a characteristic area in an image obtained by an image photographing part 11. A selection area validation part 13 validates only the characteristic area as the object of recognition. An image featured value calculation part 14 calculates an image featured value in the validated area. An image featured value comparison part 16 compares the calculated image featured value with the image featured value of a preliminarily prepared reference image. A result output part 17 outputs the identifier of the reference image having the image featured value determined to be the most approximate to the calculated image featured value as a recognition result. Media associated with the recognition result may be displayed.

Description

  The present invention relates to an image recognition apparatus, and more particularly, to an image recognition apparatus that can recognize an image captured by a camera such as a portable terminal with a low calculation load.

  An image recognition device that recognizes an image captured by a camera is applied to various fields. For example, various managements can be performed based on the image recognized by the image recognition apparatus, and information related to the recognized image can be provided to the user.

  In Patent Document 1, even when the shape of an object is recognized, even if its shape is affected by deformation or the like, a linear component or a curve component corresponding to the contour is extracted so that it can be compared with a predetermined object. A possible image recognition device is disclosed.

  In Patent Document 2, in an image including a plurality of objects, an attention area such as a window function is set, a histogram of the RGB color space in the attention area is obtained, and the position and size of the attention area are changed in advance. An image recognition / retrieval device that determines the presence of a registered object is disclosed.

JP-A-7-73317 JP-A-9-128542

  The image recognition apparatus disclosed in Patent Document 1 performs image recognition using all of the image information acquired by the imaging unit. For this reason, for example, when the input image includes a complicated background in addition to the recognition target, there is a possibility that a large number of straight line components and curved line components may be extracted, which causes a problem of erroneous recognition of the target.

  The image recognition / retrieval device disclosed in Patent Literature 2 recognizes only an attention area. However, in this case, it is necessary to calculate the image feature amount while changing the position and size of the region of interest, so that the processing cost required for calculating the image feature amount increases, and in particular, a mobile phone with limited calculation resources. There was a possibility that reasonable performance could not be obtained with terminals.

  The objective of this invention is providing the image recognition apparatus which can suppress the cost of recognition processing while solving the said subject and reducing the misrecognition in recognition of a target object.

    In order to solve the above problems, the present invention provides an image capturing unit that captures an image of the outside world and acquires an image, a recognition region selecting unit that selects a characteristic region from the image acquired by the image capturing unit, and the recognition A selection region enabling unit that enables the characteristic region selected by the region selection unit to be recognized as a recognition target, and disables other regions so as to be excluded from the recognition target; and the selection region validation An image feature amount calculation unit that calculates an image feature amount in an area validated by the unit, and an image feature amount that compares the image feature amount calculated by the image feature amount calculation unit with an image feature amount of a reference image prepared in advance According to the comparison result by the amount comparison unit and the image feature amount comparison unit, the result of recognizing the identifier of the reference image having the image feature amount most approximated to the image feature amount calculated by the image feature amount calculation unit It is basically characterized by including the result output section which to output.

  According to the present invention, when image recognition is performed with a camera-equipped mobile phone or the like, erroneous recognition can be reduced even if an image obtained by capturing a recognition target includes a complicated background.

  In addition, since image recognition can be performed at a lower processing cost, the image recognition process can be operated without any problem even in a terminal with limited resources such as a mobile phone.

  Furthermore, since the speed of image recognition is expected to be reduced by reducing processing costs, the time from recognition of the target image to playback of the media associated with the recognition result can be shortened. Reasonable performance can be obtained even when the frequency changes frequently.

1 is a block diagram showing a first embodiment of an image recognition apparatus according to the present invention. It is a figure which shows the specific example of an input image and an effective input image. It is a figure explaining operation | movement of the image recognition apparatus when two characteristic area | regions are selected from the input image. It is a block diagram which shows the structure at the time of applying this invention to the image recognition apparatus which recognizes an image especially for advertising video.

  The present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a first embodiment of an image recognition apparatus according to the present invention.

  The image recognition apparatus 10 of the present embodiment is, for example, a camera-equipped mobile phone, and includes an image photographing unit 11, a recognition region selection unit 12, a selection region validation unit 13, an image feature amount calculation unit 14, and an image feature amount comparison unit 15. A recognition result output unit 16, a recognition result display unit 17, a reference image feature amount storage unit 18, and a media file storage unit 19.

  The image capturing unit 11 is, for example, a camera included in a mobile phone, and acquires an image of the outside world as an input image of the image recognition device 10.

  The recognition area selection unit 12 selects a characteristic area from the input image (screen) from the image photographing unit 11. Here, a technique of detecting a feature in the input image and directly selecting a characteristic region based on the feature is used. By using such a method, processing is performed in comparison with the process of searching for the attention area from the image while calculating the image feature amount while setting the attention area in advance and changing the position and size of the attention area. The burden can be reduced. Note that the number of characteristic regions selected here is not limited to one, but may be an arbitrary number. Also, the size and shape of the characteristic region to be selected are arbitrary, and may be, for example, a rectangle or other shapes. A specific method for selecting a characteristic region from the input image will be described in detail later.

  The selection area validation unit 13 generates an input image (hereinafter referred to as an effective input image) in which the characteristic area selected by the recognition area selection unit 12 is validated. That is, the selection area validation unit 13 performs an validation process for limiting the characteristic area selected by the recognition area selection unit 12 to be a recognition process target. The validation of the characteristic area is such that the pixel values of pixels belonging to areas other than the characteristic area are ignored in the feature quantity calculation in the subsequent image feature quantity calculation unit 14, for example, The pixel value of the pixel belonging to the characteristic region selected by the recognition region selection unit 12 is left as it is, and the pixel value of the pixel belonging to the region other than the characteristic region is not affected by the recognition process ( For example, it can be realized by overwriting with 0).

  The image feature amount calculation unit 14 calculates an image feature amount for each characteristic region selected by the recognition region selection unit 12 with the effective input image from the selection region validation unit 13 as a target. When N characteristic regions are selected by the recognition region selection unit 12, the image feature amount may be calculated for each of N effective input images. An arbitrary feature amount can be used as the image feature amount, and the image feature amount already obtained when the recognition region selection unit 12 selects a characteristic region can also be used as it is.

  The reference image feature amount storage unit 15 stores image feature amounts of reference images in advance. The image feature amount comparison unit 16 compares the image feature amount of the effective input image calculated by the image feature amount calculation unit 14 with the image feature amount of the reference image stored in the reference image feature amount storage unit 15.

  The recognition result output unit 17 uses, as a recognition result, an identifier (such as an object name) of a reference image having an image feature amount that is closest to the image feature amount of the effective input image according to the comparison result in the image feature amount comparison unit 16. Output.

  The recognition result display unit 18 displays a character string and an image including the recognition result output from the recognition result output unit 17, the shape of the selected region, and the like. Further, if an arbitrary medium is stored in the media file storage unit 19 in association with the identifier of the reference image, the media related to the identifier output by the recognition result output unit 17 is acquired from the media file storage unit 19. The recognition result display unit 18 can display the result image.

  Next, the operation of the image recognition apparatus 10 in FIG. 1 will be described with a specific example. FIG. 2 is a diagram illustrating specific examples of an input image and an effective input image.

  If the input image from the image photographing unit 11 is, for example, FIG. 2A, the recognition area selecting unit 12 selects the inside of the broken line in FIG. FIG. 2B in which the above is valid is output as an effective input image. Here, pixel values of pixels belonging to regions other than the characteristic region are overwritten with 0 (black).

The selection of a characteristic area by the recognition area selection unit 12 can be performed by using, for example, a feature point distribution. The feature point distribution is calculated by detecting feature points such as corners in the image and using the distance between the detected feature points. For example, when attention is paid to one feature point F _N , a set of feature points existing around the feature point F _N and having a distance from the feature point F _N smaller than a predetermined threshold TH is a feature point set {F _N }. A feature point set is obtained for all feature points or arbitrarily sampled feature points, and when the region in the image composed of the feature point sets has a predetermined number or more, the region is selected as a feature region. be able to.

  A characteristic region can also be selected by using an edge feature. When using an edge feature, edge enhancement by an edge enhancement filter is applied to an input image from the image capturing unit 11, and when the edge obtained thereby constitutes a closed region, the region is characterized. It can be regarded as an area.

  It is also possible to select a characteristic region from the number of straight lines, color distribution, texture complexity, texture similarity, and the like. For example, when selecting a characteristic region from the number of straight lines, after applying straight line detection such as Hough transform to the input image from the image capturing unit 11, a predetermined direction and a predetermined number of lines for the entire image are applied. In addition, a region where the detected straight line and a predetermined direction and a predetermined number of straight lines intersect at a certain density or more can be selected as a characteristic region.

  Furthermore, a characteristic region can be selected by appropriately combining the above characteristic region selection methods.

  The image feature amount calculation unit 14 calculates an image feature amount for each characteristic region selected by the recognition region selection unit 12 with the effective input image from the selection region validation unit 13 as a target. If N characteristic regions are selected from the input image by the region selection unit 12, image feature amounts are calculated for N effective input images.

  FIG. 3 is a diagram for explaining the operation of the image recognition apparatus 10 when two characteristic regions are selected from the input image. Here, the input image is shown in FIG. 3A by the region selection unit 12, and two characteristic regions A and B are selected from the input image.

  The recognition area selection unit 12 outputs an effective input image (FIG. 3B) in which the image of the area A is effective and an effective input image (FIG. 3C) in which the image of the area B is effective. Again, the pixel values of pixels belonging to areas other than the characteristic area are overwritten with 0 (black).

  The image feature amount calculation unit 14 calculates an image feature amount for each of the effective input images (FIGS. 3B and 3C). That is, first, the image feature amount is calculated from the region A by leaving only the region A and excluding the region B. Next, the image feature amount is calculated from the region B by leaving only the region B and excluding the region A.

  As the image feature amount calculated by the image feature amount calculation unit 14, an arbitrary feature amount can be used. For example, scale invariant feature transformation (SIFT) or image moment can be used.

  Scale invariant feature transformation (SIFT) can be extracted by the method described in “Distinctive Image Feature from Scale-invariant Keypoints” (David Lowe, International Journal of Computer Vision).

When the two-dimensional image is F (x, y), the (m + n) next image moment M _mn is calculated by the equation (1). At this time, the two-dimensional image may be binarized.

Further, (M ₁₀ / M ₀₀ , M ₀₁ / M ₀₀ ) represents the center of gravity, and the center of gravity moment μ _{mn with} respect to this center of gravity is invariant to translation. The center of gravity moment μ _mn is calculated by the equation (2).

Further, the center-of-gravity moment η _ij (i + j ≧ 2) normalized by μ ₀₀ is invariant to enlargement / reduction. The normalized centroid moment η _ij is calculated by the equation (3).

  SIFT is robust against noise and the like, but has a large number of feature dimensions. On the other hand, the image moment has a small dimension of the feature quantity, but is vulnerable to noise. Considering the advantages and disadvantages of SIFT and image moment, you can use any combination of these features according to the recognition target and application.

  The image feature amount comparison unit 16 compares the image feature amount calculated from the effective input image with the image feature amount of the reference image stored in advance in the reference image feature amount storage unit 15. The image feature amount of the reference image can be stored in an arbitrary format, but when comparing with the image feature amount of the effective input image, both formats are unified and compared.

For the comparison by the image feature quantity comparison unit 16, for example, measurement of the distance between the image feature quantities can be used. For example, in order to measure the distance d between η ₀₂ and η ₂₀ as the normalized centroid moment using the image moment as the image feature amount, the equation (4) may be used.

Here, qη _ij represents the normalized centroid moment of the effective input image, and rη _ij represents the normalized centroid moment of the reference image. Note that any one of i, j (i + j ≧ 2) can be used.

  The recognition result output unit 17 is an identifier of a reference image (such as an object name) having an image feature amount that is closest to the image feature amount of the effective input image according to the comparison result of the image feature amount in the image feature amount comparison unit 16. For example, the identifier of the reference image that gives the minimum value among the distances d in Expression (4) is output as the recognition result.

  The recognition result display unit 18 displays a character string and an image including the identifier of the reference image output from the recognition result output unit 17 and the shape of the selected region. Instead of an identifier or the like, a medium associated with the identifier or the like may be reproduced in the reference image.

  As shown in FIG. 1, the media file storage unit 19 for storing the media associated with the identifier of the reference image is provided for reproduction of the media associated with the identifier or the like in the reference image, and the recognition result output unit 17 outputs the media. This can be realized by acquiring the media associated with the identifier from the identifier of the reference image from the media file storage unit 19 and sending the media to the recognition result display unit 18.

  As media to be played back by the recognition result display unit 18, still images, moving images, animations, sounds, sounds, and the like can be used, and these can be stored in the media file storage unit 19 in advance.

  FIG. 3D shows a result display image when two characteristic areas A and B are selected from the input image. Here, the result display image is shown when the distance d in Equation (4) is calculated and the recognition results for “XX beer details” and “YY snacks for beer” are obtained for regions A and B, respectively. ing. As shown in the figure, the recognition results for a plurality of areas may be superimposed and displayed as one screen, or may be sequentially switched and displayed at a predetermined timing. Further, as will be described later, it is possible to guide to the “XX beer” or “YY snack” website from the display of the recognition result.

  In the above, the image recognition device 10 has been described as including the reference image feature amount storage unit 15 and the image feature comparison unit 16, but instead of including the reference image feature amount storage unit 15 and the image feature comparison unit 16, a network It is also possible to use a server connected via the Internet. In this case, the server is provided with functions as the reference image feature amount storage unit 15 and the image feature comparison unit 16, and the image feature amount of the effective input image calculated by the image feature amount calculation unit 14 is converted into an arbitrary format. The image feature quantity of the effective input image is compared with the image feature quantity of the reference image in the server, and the identifier of the reference image having the image feature quantity closest to the image feature quantity of the effective input image according to the comparison result Or the like may be notified to the image recognition apparatus 10 via the network.

  Further, instead of providing the image feature quantity calculation unit 14, a server may be used. In this case, the server is provided with a function as the image feature amount calculation unit 14, the effective input image itself is transmitted from the image recognition apparatus 10 to the server, the image feature amount is calculated from the effective input image in the server, and the effective What is necessary is just to compare the image feature-value from an input image with the image feature-value of a reference image. According to this, since it is not necessary for the image recognition device 10 to calculate the image feature amount, the processing load on the image recognition device 10 can be reduced when the image feature amount calculation cost is high.

  Further, the media file storage unit 19 may be connected to a network, and instead of including the recognition result output unit 17, a server can be used. According to this, for example, a server is provided with a function as the recognition result output unit 17, and a medium associated with the recognition result is combined with the recognition result output by the function as the recognition result output unit 17 in the server. Can be acquired from the network and transmitted to the image recognition apparatus 10, and the image recognition apparatus 10 can reproduce the recognition result and media. The media may be reproduced by being superimposed on the input image by the screen photographing unit 11.

  FIG. 4 is a block diagram showing a configuration in a case where the present invention is applied to an image recognition apparatus that performs image recognition particularly for advertisement video.

  The image recognition device 40 is, for example, a camera-equipped mobile phone, and includes an image photographing unit 41, an advertisement product region selection unit 42, an advertisement product region validation unit 43, an image feature amount calculation unit 44, an image feature amount comparison unit 46, A related URL acquisition unit 47, a related web display unit 48, and a storage medium 45 are provided.

  The image capturing unit 41 captures a screen of the television 50 that receives the television broadcast. When the television screen of the advertising video is captured by the image capturing unit 41, the advertising product area selection unit 42 selects the advertising product area as a characteristic area from the input image.

  When the image photographing unit 41 shoots the screen of the television 50, it may be notified from the outside which advertising video screen is displayed on the television 50. For example, if the screen of TV 50 is the screen of an advertising video or TV program that is currently being broadcast, information about which advertising video or TV program is being broadcast is delivered from the server of the TV station or a provider representing the TV station. . The image recognition device 40 receives the information distributed from the server, and uses the advertisement video or TV program to be recognized from the information as prior knowledge, thereby including the advertisement product included in the screen captured by the image capturing unit 41 A limited area can be selected.

  Further, if the screen of the television 50 is a screen of an advertisement video or TV program once recorded and played back by the recording / playback device, information on which advertising video or TV program is played back from the recording / playback device to the image recognition device. Just notify 40. This notification can be made, for example, by using short-range wireless communication between the recording / playback apparatus and the image recognition apparatus 40, and transmitting / receiving information related to the advertisement video or TV program being played back.

  Furthermore, if the target advertisement video or TV program is analyzed in advance and the image feature quantities of a plurality of reference images that are continuous in time are stored, the advertisement is displayed from one screen shot by the image shooting unit 41. It becomes easy to specify the product area. Note that the advertisement product area may be specified in advance in the reference image.

  The advertisement product area validation unit 43 generates an input image in which the characteristic area selected by the advertisement product area selection unit 42 is validated. Regions other than the characteristic region selected by the advertisement product region selection unit 42 may be simply ignored or converted into a single value.

  The image feature amount calculation unit 44 calculates the image feature amount of the characteristic region selected by the advertisement product region selection unit 42 for the effective input image from the advertisement product region validation unit 43 as a target.

  The storage medium 45 uses an advertisement product image registered in advance as a reference image, and stores image feature amounts calculated in advance from the reference image. Further, the storage medium 45 stores a file in which the identifier of the advertisement product of the reference image and the URL of the website associated with each identifier are described. The image feature amount of the reference image and the URL of the website can be acquired from the advertisement aggregator 60 or the like.

  The image feature amount comparison unit 46 compares the image feature amount of the characteristic area selected by the advertisement product region selection unit 42 with the feature amount of the reference image stored in the storage medium 45. The related URL acquisition unit 47 stores the URL of the website associated with the identifier of the reference image having the image feature amount most similar to the image feature amount of the effective input image according to the comparison result in the image feature amount comparison unit 46. Get from 45.

  The related WEB display unit 48 activates a WEB browser and accesses the WEB page of the server 70 such as an advertiser using the URL of the WEB site acquired by the related URL acquisition unit 47. Thereby, the WEB site associated with the advertisement product of the advertisement video is opened, and the user can be easily guided to a site where the detailed information of the advertisement product can be viewed, a site where the advertisement product can be purchased, and the like.

  As mentioned above, although embodiment was described, this invention is not limited to the said embodiment. For example, FIG. 4 is an example of the case where the present invention is applied to an image recognition apparatus that performs image recognition on an advertising video. For example, the present invention is applied to an image recognition apparatus that performs image recognition on a TV performer. Can also be applied. That is, the performer face area is selected as a characteristic area from the video on the TV screen, and information such as a website associated with the performer recognized based on the image feature amount in the characteristic area is displayed. Can do.

  Further, instead of the image acquired by the image capturing unit as it is, an image processed so as to be easily processed by the selection unit, the image feature amount extraction unit, or the image feature amount comparison unit may be input to the recognition region selection unit. . For example, the distortion of the image acquired by the image capturing unit may be estimated, and an image corrected for distortion based on the estimated distortion may be input to the recognition area selection unit. May be input to the recognition area selector. Note that the techniques for image distortion correction, enlargement, and rotation processing are known, and the techniques used here are not limited.

  10, 40 ... Image recognition device, 11, 41 ... Image capturing unit, 12 ... Recognition region selection unit, 13 ... Selection region validation unit, 14, 44 ... Image feature amount calculation unit , 15, 46 ... image feature amount comparison unit, 16 ... recognition result output unit, 17 ... recognition result display unit, 18 ... reference image feature amount storage unit, 19 ... media file storage unit 42 ... Advertisement product area selection part 43 ... Advertisement product area validation part 47 ... Related URL acquisition part 48 ... Related web display part 45 ... Storage medium

Claims (14)

An image shooting unit for shooting the outside world and acquiring images;
A recognition region selection unit that selects a characteristic region from the image acquired by the image photographing unit;
A selection region enabling unit that enables the characteristic region selected by the recognition region selection unit to be a recognition target and disables the other region to be excluded from the recognition target;
An image feature amount calculation unit for calculating an image feature amount in the region validated by the selection region validation unit;
An image feature amount comparing unit that compares the image feature amount calculated by the image feature amount calculating unit with an image feature amount of a reference image prepared in advance;
A result output unit that outputs, as a recognition result, an identifier of a reference image having an image feature amount that is closest to the image feature amount calculated by the image feature amount calculation unit according to a comparison result by the image feature amount comparison unit; An image recognition apparatus characterized by that.   The image recognition apparatus according to claim 1, wherein the selection area validation unit sets a value of a pixel belonging to an area other than the characteristic area selected by the recognition area selection unit as a single value. .   The image recognition apparatus according to claim 1, wherein the image feature amount calculation unit calculates at least one of scale invariant feature conversion and image moment as an image feature amount.   4. The image recognition apparatus according to claim 1, wherein a server connected via a network is used instead of including the image feature calculation unit. 5.   The feature amount comparison unit calculates a distance between the feature amount in the characteristic region selected by the recognition region selection unit and the feature amount of the reference image. The image recognition apparatus described in one.   5. The image recognition apparatus according to claim 1, wherein the image feature amount of the reference image is stored in a storage medium in the image recognition apparatus.   6. The image feature quantity of the reference image is stored in a server connected via a network, and the server is used instead of including the feature quantity comparison unit. The image recognition apparatus as described in any one of these.   8. Instead of providing the result output unit, a server connected via a network is used, and a recognition result thereby is received via the network. Image recognition device.   The image recognition apparatus according to claim 1, further comprising a media playback unit that acquires media associated with the recognition result and plays back the media.   The image recognition apparatus according to claim 9, wherein the media reproduction unit reproduces the medium superimposed on the image acquired by the image photographing unit.   The outside world photographed by the image photographing unit is a television screen, the characteristic region selected by the recognition region selecting unit is an advertising product region, and the media reproducing unit displays media associated with the advertising product. The image recognition apparatus according to claim 9 or 10, wherein reproduction is performed.   The outside world photographed by the image photographing unit is a television screen, the characteristic region selected by the recognition region selecting unit is a performer face region, and the media reproduction unit is configured to use a WEB associated with the performer. The image recognition apparatus according to claim 9 or 10, wherein information on a site is displayed. And a distortion estimation unit that estimates distortion of the image acquired by the image capturing unit, and a distortion correction unit that corrects the image acquired by the image capturing unit based on the distortion estimated by the distortion estimation unit. And
The image recognition apparatus according to claim 1, wherein the image feature amount calculation unit calculates an image feature amount in a characteristic region of the image corrected by the distortion correction unit. . Furthermore, it comprises at least one of an image enlarging unit for enlarging the image acquired by the image capturing unit and an image rotating unit for rotating the image acquired by the image capturing unit,
The image feature amount calculation unit calculates an image feature amount from a characteristic region of the image enlarged and rotated by the image enlargement unit and the image rotation unit. The image recognition apparatus described in one.