JP2014134856A - Subject identification device, subject identification method, and subject identification program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the accuracy of recognizing an object existing in an image.SOLUTION: A subject identification device detects a plane from a depth image acquired by depth image acquisition means, cuts out an image area corresponding to the plane detected by plane detection means from within an image acquired by image acquisition means, extracts a feature amount from the cut-out image area, and collates the extracted feature amount with a feature amount extracted from an image to be collated.

Description

  The present invention relates to a technique for accurately identifying an object present in a captured image or video.

  In the field of computer vision, a technique is known in which a plurality of feature points are extracted from an image acquired by a camera or the like, and subject recognition is performed by comparing the extracted feature points with the feature points of a subject image registered in a database. ing. An example of such a technique will be described with reference to FIG. The image acquisition unit 601 included in a device such as the camera device 610 inputs the acquired image to the feature amount extraction unit 602. The feature amount extraction unit 602 extracts a plurality of feature points from the input image. The collation unit 603 performs subject recognition by collating the feature points extracted by the feature amount extraction unit 602 with the feature points of the subject image registered in the database 604.

  As an algorithm for feature quantity extraction used in such subject recognition technology, many feature points (feature points) in an image are used in order to be able to identify robustly against changes in shooting size, angle, and occlusion. A method for detecting and extracting feature quantities (local feature quantities) in a local region around each feature point has been proposed. As typical methods, SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features) are known. Details of SIFT are described in, for example, Patent Document 1 and Non-Patent Document 1.

US Pat. No. 6,711,293

David G. Lowe, "Distinctive image features from scale-invariant keypoints" (USA), International Journal of Computer Vision, 60 (2), 2004, p. 91-110

  However, in a subject identification apparatus that extracts feature amounts by the technique described in the above-mentioned document, the feature amounts of the entire captured image are usually extracted and collated. Therefore, it can be used without considering where the recognition object exists in the image. On the other hand, when a recognition target exists in a part of the photographed image, it is necessary to extract feature amounts from the entire image and collate them. In this case, since a similar feature amount is detected from a region where it is not necessary to perform feature amount extraction, recognition accuracy may deteriorate.

  The present invention has been made to solve the above problems. An object of the present invention is to provide a technique for improving the recognition accuracy of subject identification.

  A subject identification device according to the present invention includes a depth image acquisition unit that acquires a depth image of an object, an image acquisition unit that acquires an image of the object, and a surface from the depth image acquired by the depth image acquisition unit. A surface detecting means for detecting; an image processing means for cutting out an image area corresponding to the surface detected by the surface detecting means from the images acquired by the image acquiring means; and an image area cut out by the processing means A feature amount extracting unit that extracts a feature amount; and a collation unit that collates the feature amount extracted by the feature amount extraction unit with a feature amount extracted from an image to be collated. .

  In the subject identification method according to the present invention, a depth image acquisition step of acquiring a depth image by capturing an object, an image acquisition step of acquiring an image by capturing the object, and detecting a surface from the depth image When a surface is detected in the surface detection step, and in the surface detection step, an image processing step of cutting out an image area corresponding to the detected surface from the image acquired in the image acquisition step, and the image processing step A feature amount extracting step for extracting a feature amount from the image area obtained, a matching step for comparing the feature amount extracted in the feature amount extraction step with a feature amount extracted from an image to be collated, and As a result, when it is determined that the feature amounts match, a step of outputting information for identifying the subject of the image to be collated is provided. And wherein the Rukoto.

  The program according to the present invention allows a computer to detect a surface from a depth image acquisition unit that acquires a depth image of an object, an image acquisition unit that acquires an image of the object, and a depth image acquired by the depth image acquisition unit. A surface detecting unit that performs image processing on the image acquired by the surface detecting unit from the image acquired by the image acquiring unit, an image processing unit that extracts an image region corresponding to the surface detected by the surface detecting unit, A feature amount extracting unit that extracts a quantity, and a feature unit that functions as a collating unit that collates a feature amount extracted by the feature amount extracting unit with a feature amount extracted from an image to be collated.

  According to the present invention, the recognition accuracy of subject identification can be improved.

1 is a block diagram illustrating a configuration of a subject identification device according to a first embodiment of the present invention. It is a flowchart which shows the procedure of the to-be-photographed object identification method which concerns on 1st Embodiment of this invention. It is a flowchart which shows the procedure of the process of the image process part which concerns on 1st Embodiment of this invention. It is a flowchart which shows the procedure of the process of the image process part which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the to-be-photographed object identification apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows the principle of a prior art.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. However, the constituent elements described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention only to them.

[First Embodiment]
FIG. 1 is a block diagram showing a functional configuration of a subject identification device as a first embodiment of the present invention. The subject identification device according to the present invention has, as main functional configurations, a depth image acquisition unit 101, a surface detection unit 102, an image processing unit 103, an image acquisition unit 104, a feature amount extraction unit 105, a collation unit 106, and a feature amount (subject ) The database 107 is included. The depth image acquisition unit 101 and the image acquisition unit 104 can be configured as a part of the camera device 110, but can also be configured as independent configurations.

  The subject identification device has a hardware configuration including a control unit (CPU (Central Processing Unit), etc.), a memory (RAM (Random Access Memory), ROM (Read Only Memory), etc.), a storage unit (hard disk drive, etc.), The computer has the configuration of a general computer, such as a communication unit, a display unit, and an operation unit. Further, each functional configuration described above can be realized by, for example, a control unit included in the subject identification device developing a program from the storage unit into a memory and executing the program.

  The depth image acquisition unit 101 acquires a depth image by imaging a subject (target object) or by communicating with an external device. Here, the depth image is an image in which a distance zd from the camera device is given to the two-dimensional image (xd, yd). Each pixel constituting the depth image is a point on a three-dimensional space and can be expressed by three-dimensional coordinates (xd, yd, zd). That is, the depth image may be treated as point cloud data representing a set of points in the three-dimensional space. As a method for acquiring a depth image, for example, a three-dimensional image acquisition method by pattern projection is used in which a laser beam having a pattern is applied to an object, the distortion of the pattern of reflected light is measured by triangulation, and the distance is measured. . Also, using the Time of Flight method that measures the distance from the time it takes for the light to reciprocate, the light is pulse-modulated and irradiated into the angle of view, and the phase delay of this pulse is measured on the image sensor side 3 A dimensional image acquisition method may be used. Furthermore, a three-dimensional image acquisition method using a stereo camera may be used in which the binocular parallax is reproduced by simultaneously capturing images of the object from a plurality of different directions, and information in the depth direction zd can be recorded.

  The image acquisition unit 104 acquires a normal two-dimensional image (hereinafter also simply referred to as an image) that is not a depth image by imaging a subject (target object) or by communicating with an external device. The image acquisition unit 104 acquires an image obtained by photographing the subject from approximately the same direction as the depth image acquisition unit 101. It is assumed that the correspondence between the acquired depth image and the image is determined in advance for each pixel unit. For example, the image (x, y) can be expressed by a calculation formula such as xd = x + α, yd = y + β.

  The depth image acquired by the depth image acquisition unit 101 is input to the surface detection unit 102 that detects a surface.

  The surface detection unit 102 performs processing for detecting a surface region from the depth image. Here, the surface is a continuous region in the three-dimensional space, and may be a flat surface or a curved surface, for example. One or a plurality of surface areas may be detected by the surface detection unit 102. The detected surface area is input to the image processing unit 103.

  The surface detection unit 102 can detect a surface area by dividing (segmenting) the point group of the depth image into a subset of points belonging to the same surface. Specifically, for example, for each pixel (each point in the point group) of the depth image, a parameter value such as a normal vector is calculated from the three-dimensional coordinate values of the surrounding pixels (points), and each pixel (each Clustering is performed using the coordinate values and parameter values of (points), and the surface area can be divided (segmented). By doing so, it is possible to detect a region where the normal vector direction of each pixel is continuous as a surface. If it is a condition that the direction of the normal vector is continuously changing and is aligned in a certain direction, a plane can be detected particularly among the surfaces.

  Here, a method of obtaining a normal vector for each pixel (each point in the point group) of the depth image will be described. As a method of calculating the normal vector from the coordinate information of the pixels around each pixel of the depth image, there is a method of obtaining by a principal component analysis of a covariance matrix. Specifically, for a given pixel of interest, a covariance matrix (3 × 3 matrix) is calculated using the coordinate values (xd, yd, zd) of the surrounding pixels. Then, eigenvalues and eigenvectors of the covariance matrix are calculated. Among the eigenvectors, the eigenvector having the smallest eigenvalue is set as the normal vector of the target pixel. As another method for obtaining a normal vector, there is a method for forming a triangle from the coordinate values of surrounding pixels for a certain target pixel, obtaining several outer products thereof, and using the average as a normal vector. .

  Next, a method for dividing (segmenting) a surface area using a normal vector obtained for each pixel (each point of the point group) of the depth image will be described. As one method, there is a method called Region Growing. This method is a method of dividing a surface by sequentially merging peripheral pixels (peripheral points) having similar normal vectors (that is, having a large inner product between the normal vectors). First, a certain pixel (a point) is set as the initial value of the region, and the normal vector of the pixel (point) of the initial value is compared with the normal vector of the surrounding pixel (peripheral point), and similar normal lines are compared. A pixel (point) having a vector is added to the region. By sequentially performing this, a set of pixels (points) whose normal vector continuously changes can be divided (segmented) as a surface area.

  In addition, by using a three-dimensional coordinate value of each pixel (each point of the point group) and a parameter value such as a normal vector, a cutting method based on graph theory is applied to divide the surface area (segmentation). Good. Specifically, for example, the similarity is calculated by calculating the similarity between pixels (between points) based on the distance between coordinate values and the similarity of normal vectors, and a method such as Normalized Cut is used. A set of pixels (points) may be divided.

  In addition, the method for detecting a plane uses a three-dimensional Hough transform, which is a three-dimensional extension of the Hough transform, which has been used for straight line recognition in two-dimensional image processing. There is a way to detect. By passing 3D coordinate information obtained from the depth image through 3D Hough transform, it becomes possible to detect the position and orientation of the planar area. In addition to the Hough transform, a robust estimation method such as RANSAC (RANdom SAmple Consensus) may be used. Moreover, you may estimate using the least squares method.

  By the method as described above, the surface detection unit 102 detects a surface region from the depth image. The surface detection unit 102 inputs information specifying the detected surface area to the image processing unit 103. Here, the information for specifying the surface area is, for example, a set of pixel values (set of points) of the depth image constituting the surface area. Further, for example, the information specifying the surface area may be information such as a contour line of the surface area or a bounding box including the surface area.

  Further, the surface detection unit 102 may calculate and output not only information for specifying the surface area but also information indicating the orientation of the surface detected at the same time. The information indicating the orientation of the surface may be, for example, the normal vector (a, b, c) when the surface is a plane. The normal vector representing the orientation of the plane may be calculated, for example, as the average of the normal vectors of each pixel (point) of the pixel set (point set) that constitutes the plane. Note that the information indicating the orientation of the surface is used by the image processing unit 103 of the second embodiment to be described later. In the first embodiment, it is not necessary to output information indicating the orientation of the surface.

The image acquired by the image acquisition unit 104 is input to the image processing unit 103.
In the image processing unit 103, an image region corresponding to the surface region detected by the surface detection unit 102 is cut out from the image acquired by the image acquisition unit 104, and the cut-out image region is input to the feature amount extraction unit 105. To do. Here, a method of giving information for specifying an image area in the image, such as giving coordinate values of four corners of the image area corresponding to the detected area of the surface, may be used without physically cutting out.

  For example, when the surface area information input from the surface detection unit 102 is a set of pixel values of a depth image (a set of points), the image processing unit 103 is, for example, on a two-dimensional image including the set of pixel values. The bounding box may be obtained and an image area from which the bounding box is cut out may be used. At this time, for example, pixels in a cut image region that does not correspond to the surface region detected by the surface detection unit 102 may be painted with a single color to remove image information.

  The feature amount extraction unit 105 extracts a feature amount from the image region input from the image processing unit 103. For example, local feature amounts are extracted from feature points of an image using SIFT feature amounts. The feature amount of the image area extracted by the feature amount extraction unit 105 is input to the matching unit 106. Here, the local feature amount such as the SIFT feature amount is taken as an example, but the present invention is not limited to this, and a global feature amount may be extracted. For example, visual features such as Color Histogram, Color Layout, Edge Histogram, Image Signature, and Video Signature defined in the international standard ISO / IEC 15938-3 (MPEG-7 Visual) may be used.

  The feature quantity (subject) database 107 stores feature quantities extracted in advance from subject images to be identified. Note that the feature amount of the image registered in advance in the feature amount database 107 is the feature amount extracted by the same method as the feature amount extraction used in the feature amount extraction unit 105. For example, the feature amount registered in the feature amount database 107 may be extracted from an image obtained by photographing a subject to be identified from the front.

  The collation unit 106 collates the feature amount of the image area generated by the feature amount extraction unit 105 with the feature amount of the subject image stored in the feature amount (subject) database 107 in advance, and whether there is matching data. It is determined whether or not (that is, whether or not the same subject is included between the two images). When it is determined that there is matching data, information that identifies the recognized object is output as a comparison result based on the data.

  As a collation method, for example, when the feature quantity input from the feature quantity extraction unit 105 is a local feature quantity (SIFT feature quantity or the like) extracted from a large number of feature points, the following method can be used. . First, corresponding points are detected using the distance between local feature amounts between the feature points of the input image region and the feature points of the feature amount data of the subject image registered in the feature amount (subject) database 107 in advance. . Next, it is determined whether or not a common subject is included between the two images by verifying the geometric consistency from the positional relationship of the corresponding points.

  In the detection of corresponding points, for example, the Euclidean distance between the local feature values of any feature point of the input image and all the feature points of the reference image is calculated, and the minimum distance value and the second smallest distance value are calculated. If the ratio is equal to or smaller than the threshold, a method of determining the feature point with the minimum distance as the corresponding point may be used. By using the ratio of the distances between the two local feature amounts, it is possible to detect only corresponding points having a sufficiently small distance between the local feature amounts as compared to other feature points, and thus it is possible to suppress miscorrespondence.

  In the determination of geometric consistency, for example, a geometric transformation parameter between images is estimated using a positional relationship between corresponding points. As a geometric transformation model to be estimated, a basic matrix or a projective transformation matrix (homography matrix) is used, and as a method for estimating a geometric transformation parameter, there is a method using RANSAC (RANdom SAmple Consensus) capable of robust estimation.

  The corresponding points that match the geometric transformation parameters estimated by RANSAC are inliers, and the other corresponding points are outliers. If there are a sufficient number of inliers relative to the outliers, it can be determined that the reliability of the estimated geometric transformation parameter is high. In this way, the subject can be identified by using RANSAC and local feature quantities such as SIFT feature quantities.

  As a result of identifying the subject, if an image including the same subject is found, identification information for identifying the identification target (subject) is output as the identification result. In addition, when using local feature quantities such as SIFT feature quantities, geometrical consistency information is obtained as a result of verification of geometric consistency in collation. In addition to the information, the position (the exact position of the subject) and direction of the identification target in the image region cut out by the image processing unit 103 can be obtained, and the information (for example, a homography matrix) is output. be able to.

  A subject identification processing method as described above will be described with reference to the flowchart of FIG. This process is controlled by, for example, a control unit included in the subject identification device expanding and executing a program stored in the storage unit in the memory.

  First, the depth image acquisition unit 101 and the image acquisition unit 104 capture a subject surface (step S201). Next, the surface detection unit 102 performs surface detection processing on the depth image output from the depth image acquisition unit 101 (step S202). Next, it is determined whether a surface is detected by the surface detection unit 102 (step S203). If no surface is detected, the process ends. When a surface is detected, the image processing unit 103 performs a process of cutting out image areas corresponding to all the surfaces detected by the surface detection unit 102 from the image acquired by the image acquisition unit 104. (Step S204).

  Next, the feature amount extraction unit 105 performs a process of extracting a feature amount for the largest image region from the image regions output from the image processing unit 103 (step S205). Next, the collation unit 106 collates the extracted feature quantity with the feature quantity registered in the feature quantity database 107, thereby registering the feature quantity extracted from the image region in the feature quantity database. It is determined whether there is a feature quantity that matches the feature quantity data that is present (step S206).

  If it is determined that there is a collation, the collation unit 106 outputs information that identifies the object recognized as the collation result (step S207), and if it is determined that there is no collation, the process of outputting the information is skipped. . Thereafter, it is determined whether or not an image area from which the feature amount has not been extracted remains (step S208). When the image area remains, the process returns to the process of extracting the feature amount with respect to the maximum image area again (step S205), and thereafter, steps S205 to S208 are performed until there is no image area that has not been collated. repeat. When there is no more image area, the process ends.

  The above is the method of subject identification processing in this embodiment. The above describes the case of image processing. However, when processing a moving image, it can be handled by repeating the above processing every frame.

  The processing of the image processing unit 103 will be described with reference to the flowchart of FIG. First, the image processing unit 103 obtains an image from the image acquisition unit 104 (step S301). Next, the image processing unit 103 acquires a surface area from the surface detection unit (step S302). Finally, the image processing unit 103 cuts out an image area corresponding to the surface area (step S303), and ends the process.

  According to the present embodiment, a surface whose surface is detected without performing feature amount extraction / collation for an image region corresponding to a surface that is not detected by the surface detection unit 102 with respect to a recognition target in the video. Since the feature amount extraction / collation can be performed only on the image region corresponding to the object area, the recognition accuracy of the subject identification can be improved.

  In addition, except for the case where the entire screen is detected, it is possible to limit the image area where feature amount extraction / collation is performed, compared to the case where the feature amount of the entire entire image is extracted and collation is performed. Can be speeded up.

  In addition, it is possible to change to the step of determining whether or not an image area of a certain size or more remains in the determination of whether or not the image area from which the feature amount is not extracted remains (step S208). A threshold is set for the size of the image region, and for a region having a smaller size, for example, an image region that is so small that feature extraction or collation processing is difficult, results are not obtained even if steps S205 to S206 are performed. Or, accurate results may not be obtained. By skipping feature amount extraction and collation processing for such a small-sized image region, the identification accuracy can be further improved. In addition, the useless feature amount extraction / collation process can be skipped, so the processing can be speeded up.

[Second Embodiment]
Next, a video processing apparatus according to the second embodiment of the present invention will be described. In the image processing apparatus according to the present embodiment, the image processing unit 103 performs orientation correction on the cut-out image area. This is different from the first embodiment in which the image processing unit 103 only cuts out an image region. Other configurations and operations are the same as those in the first embodiment, and thus description of the same configurations and operations is omitted.

  An image region corresponding to the region of the surface detected by the surface detection unit 102 is cut out, and the cut-out image is geometrically corrected using information on the detected surface direction. Here, the geometric correction is processing for correcting an image by geometric transformation including, for example, enlargement / reduction, affine transformation, projective transformation, and the like. For example, the image processing unit 103 may geometrically correct the image of the clipped image region so that the direction of the surface detected by the surface detection unit 102 is the front direction. Here, the front direction means a direction parallel to the image plane acquired by the image acquisition unit 104, that is, a direction perpendicular to the optical axis (z axis) of the camera.

  A specific example in which the image processing unit 103 geometrically corrects an image using the orientation of the surface detected by the surface detection unit 102 will be described. The orientation of the detected surface can be expressed as, for example, the normal vector (a, b, c) of the surface. The method for calculating the normal vector of the surface has been described in the surface detection unit 102 of the first embodiment. In the second embodiment, the surface detection unit 102 also supplies information indicating the orientation of the surface, such as a normal vector (a, b, c) of the surface, to the image processing unit 103.

  As an example of the orientation correction method, a case where the surface detected by the surface detection unit 102 is a flat surface and a case where the image is directed to the front in the image processing 103 will be described below.

  Using the normal vector (a, b, c) representing the orientation of the surface output by the surface detection unit 102, the image is corrected by geometrically converting the image of the image region corresponding to the surface into an image photographed from the front. (Geometric correction). As a result, it is possible to generate an image with the surface facing the front. Here, an image corrected by geometric correction is referred to as a corrected image.

  An example of a method for generating a corrected image will be described. Here, an xyz coordinate system for the camera device of the image acquisition unit 104 is defined. The x-axis is the left-right direction of the image plane (image acquired by the image acquisition unit 104) (rightward is positive), the y-axis is the vertical direction of the image plane (downward is positive), and the z-axis is the optical axis of the camera device. The direction (the direction of the captured image plane is positive). Further, the optical center of the camera device is defined as an origin 0 and the focal length is defined as f.

First, consider rotating the z-axis of the xyz coordinate system in the direction of the normal vector (a, b, c) of the detected surface. That is, the following formula
Consider a rotation matrix R (3 × 3 matrix) that satisfies By doing so, it is possible to obtain a front-facing viewpoint with respect to the detected surface. Note that the normal vector (a, b, c) is normalized to a unit vector. Here, the rotation is performed by the procedure of first rotating about φ around the y axis and then rotating about θ about the x axis. Then, the rotation angles θ and φ are calculated by the following equations using a, b, and c.

Accordingly, the rotation matrix R is obtained as the following equation.

Next, consider the coordinate transformation on the surface where the image coordinates (x, y) are detected. That is, when the three-dimensional vector (x, y, f) from the origin 0 corresponding to one point on the image plane is extended, the point p that intersects the detected plane is set as the image coordinate after coordinate conversion. Assuming that the three-dimensional vector corresponding to the point p is P with respect to the xyz coordinate system, P is expressed by the following equation.

Here, k is k> 0, and is an expansion coefficient that represents the distance from the origin 0 to the surface. Here, the three-dimensional vector P ′ when facing the detected plane is obtained by performing coordinate transformation on P using the rotation matrix R.

By using the x-coordinate and y-coordinate of the three-dimensional vector P ′ in Expression 6 as the image coordinates after coordinate conversion of the point (x, y), the image coordinates (x, y) after geometric correction can be acquired. In other words, the following formula is obtained.

  A corrected image can be generated by converting a certain coordinate (x, y) on the image plane to the coordinate (X, Y) of the corrected image using Expression 7. Although the image coordinates (x, y) before conversion corresponding to the image coordinates (X, Y) of the corrected image are not generally integers, interpolation methods such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, and B-spline interpolation are used. May be used.

  A method of image clipping and orientation correction performed by the image processing unit 103 as described above will be described with reference to the flowchart of FIG. First, an image is obtained from the image acquisition unit (step S401). Next, a surface area is acquired from the surface detection unit, and three parameters representing the direction of the surface are obtained from the surface area (step S402). Next, an image region corresponding to the surface region is cut out (step S403). Next, based on the obtained (a, b, c), the rotation matrix R is calculated by the procedure described above (step S404). Finally, based on (a, b, c) obtained by the above procedure and the rotation matrix R, a direction-corrected image is created by performing coordinate transformation of Formula 7 (step S405), and the process ends.

  In the above example, since the image obtained from the feature quantity (subject) database 107 is often registered in the front direction, it is described as correction in the front direction, but it is not always necessary to correct in the front direction. However, it may be corrected in the same direction as the database image.

  According to the present embodiment, the image processing unit 503 applies the image area of the image acquired by the image acquisition unit 504 corresponding to the surface detected by the surface detection unit 102 to the recognition target in the video. On the other hand, feature amount extraction / collation can be performed by correcting the orientation of the image. As a result, the accuracy of recognition of subject identification can be increased.

  In addition, except for the case where the entire screen is detected, it is possible to limit the image area where feature amount extraction / collation is performed, compared to the case where the feature amount of the entire entire image is extracted and collation is performed. Can be speeded up.

[Third Embodiment]
Next, a video processing apparatus according to the third embodiment of the present invention will be described. Compared with the first and second embodiments, the video processing apparatus according to the present embodiment performs subject identification, and then uses the information indicating the subject identification result and the position and orientation of the subject to identify the subject. An image corresponding to the result is different in that the image is projected according to the identified position and the orientation of the surface. Other configurations and operations are the same as those in the first embodiment and the second embodiment, and thus the description of the same configurations and operations is omitted.

  The collation unit 506 collates the feature amount extracted by the feature amount extraction unit 505 with the feature amount registered in the feature amount (subject) database 507. When the collation unit 506 determines that there is matching data in the feature quantity (subject) database 507 as a result of the collation, the collation result includes identification information (identification result (ID)) of the data and the image processing unit 503. Information on the position and orientation of the subject in the cut image area (identification result (position)) is output. The identification result (ID) and the identification result (position) are input to the projection source image selection unit 508 and the projection image correction unit 511, respectively.

  The projection source image selection unit 508 selects a projection source image and a projection relative position associated with the input collation result (ID) from the projection source image database 509 and outputs them to the projection image correction unit 511. The projection source image is an image previously registered in the projection source image database 509 in association with the subject image. The projection relative position is a relative position when the projection image is projected by the image projection unit 512 described later.

  The projection image correction unit 511 uses information indicating the orientation of the surface obtained from the surface detection unit 502 and information on the position and orientation of the subject (identification result (position)) obtained by the collation unit 506. Then, the projection source image from the projection source image database 509 is corrected. As for the projection position, the projection position is determined from the identification result (position) from the collating means and the projection relative position from the projection source image database 509. Note that it is assumed that the imaging area of the image acquisition unit 504 and the projection area of the image projection unit 512 are calibrated and the correspondence is determined in advance for each pixel.

  Here, a specific processing method of the projection image correction unit 511 in the case of subject identification using a local feature amount (SIFT feature amount or the like) will be described. As one method, there is a method in which correction is performed using a homography matrix obtained when verification of geometric consistency in verification is performed. The homography matrix expresses the position (exact position of the subject) and orientation of the identification target in the image area.

First, each pixel (X, Y) of the projection source image is moved by the projection relative position. Thereafter, geometric transformation is performed using a homography matrix to obtain (Xn, Yn). Thereafter, the coordinates (Xn, Yn) of each pixel of the projection source image geometrically transformed by the homography matrix using Equation 8 which is the inverse transformation of Equation 7 obtained from the information indicating the orientation of the surface is a two-dimensional image. A corrected image can be obtained by converting the coordinates to the upper coordinates (xn, yn). Note that (Xn, Yn) for obtaining (xn, yn) is generally not an integer, but the nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, B-spline interpolation method, etc., exemplified in the second embodiment. An interpolation method may be used.
The corrected image is output to the image projection unit 512.

  The image projection unit 512 projects the image corrected by the projection image correction unit 511. The image projection unit 512 is configured by, for example, a projector.

  According to the present embodiment, a projection image corresponding to a subject can be selected by subject identification, and an image subjected to projection image correction corresponding to the detected surface can be projected. For example, by using this embodiment, it is possible to identify a subject image printed on a paper surface and project a projection image corresponding to the subject image on the same paper surface at the same angle as the paper surface.

[Other Embodiments]
As mentioned above, although this invention was demonstrated with reference to some embodiment, this invention is not limited to the said embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In addition, a system or an apparatus in which different features included in each embodiment are combined in any way is also included in the scope of the present invention.

  In addition, the present invention may be applied to a system composed of a plurality of devices, or may be applied to a single device. Furthermore, the present invention can also be applied to a case where a control program that realizes the functions of the embodiments is supplied directly or remotely to a system or apparatus. Therefore, in order to realize the functions of the present invention on a computer, a control program installed in the computer, a medium storing the control program, and a WWW (World Wide Web) server that downloads the control program are also included in the scope of the present invention. include.

  Some or all of the above-described embodiments can be described as in the following supplementary notes, but the present invention is not limited to the following.

(Supplementary note 1) Depth image acquisition means for acquiring a depth image of an object;
Image acquisition means for acquiring an image of the object;
Surface detection means for detecting a surface from the depth image acquired by the depth image acquisition means;
Image processing means for cutting out an image area corresponding to the surface detected by the surface detection means from the images acquired by the image acquisition means;
Feature amount extraction means for extracting a feature amount from the image region cut out by the processing means;
A subject identification apparatus comprising: a collation unit that collates a feature amount extracted by the feature amount extraction unit with a feature amount extracted from an image to be collated.
(Supplementary note 2) The subject identification device according to supplementary note 1, wherein the image processing means geometrically corrects the image of the cut out image region using information on the orientation of the detected surface.
(Supplementary note 3) The subject identification device according to supplementary note 1, wherein the image processing unit geometrically corrects the image of the clipped image area so that the detected orientation of the surface is a front orientation. .
(Supplementary Note 4) When it is determined as a result of the collation, a projection source image selection unit that selects a projection source image and a projection relative position associated with the corresponding collation target from a database;
The projection using the information on the orientation of the surface obtained by the detection of the surface, the position and orientation of the object in the cut-out image region obtained by the collation, and the selected projection relative position. Correction means for correcting the original image;
The subject identification device according to claim 1, further comprising: a projecting unit that projects the corrected image.
(Supplementary Note 5) The subject identification device according to Supplementary Note 4, wherein the correction of the projection source image by the correction unit uses a homography matrix obtained by the collation.
(Appendix 6) Depth image acquisition step of acquiring a depth image by photographing an object;
An image acquisition step of acquiring an image by photographing the object;
A surface detection step of detecting a surface from the depth image;
When a surface is detected in the surface detection step, an image processing step of cutting out an image area corresponding to the detected surface from the image acquired in the image acquisition step;
A feature amount extracting step for extracting a feature amount from the image region cut out in the image processing step;
A collation step for collating the feature quantity extracted in the feature quantity extraction step with the feature quantity extracted from the image to be collated;
And a step of outputting information for identifying a subject of the image to be collated when it is determined that the feature amounts match as a result of the collation.
(Supplementary note 7) The subject identifying method according to supplementary note 6, wherein the feature amount extracting step extracts a feature amount from an image region having a predetermined size or larger among the image regions cut out in the image processing step. .
(Appendix 8)
A depth image acquisition means for acquiring a depth image of the object;
Image acquisition means for acquiring an image of the object;
A surface detecting means for detecting a surface from the depth image acquired by the depth image acquiring means;
Image processing means for cutting out an image area corresponding to the surface detected by the surface detection means from the images acquired by the image acquisition means;
Feature amount extraction means for extracting feature amounts from the image region cut out by the processing means;
A program for functioning as a collating unit that collates a feature amount extracted by the feature amount extracting unit and a feature amount extracted from an image to be collated.

Claims (8)

A depth image acquisition means for acquiring a depth image of the object;
Image acquisition means for acquiring an image of the object;
Surface detection means for detecting a surface from the depth image acquired by the depth image acquisition means;
Image processing means for cutting out an image area corresponding to the surface detected by the surface detection means from the images acquired by the image acquisition means;
Feature amount extraction means for extracting a feature amount from the image region cut out by the processing means;
A subject identification apparatus comprising: a collation unit that collates a feature amount extracted by the feature amount extraction unit with a feature amount extracted from an image to be collated.   The subject identification apparatus according to claim 1, wherein the image processing unit geometrically corrects the image of the clipped image region using the detected orientation information of the surface.   The subject identification device according to claim 1, wherein the image processing unit geometrically corrects the image of the clipped image region so that the detected surface is directed to the front. As a result of the collation, if it is determined that they match, a projection source image selection unit that selects a projection source image and a projection relative position associated with the corresponding collation target from a database;
The projection using the information on the orientation of the surface obtained by the detection of the surface, the position and orientation of the object in the cut-out image region obtained by the collation, and the selected projection relative position. Correction means for correcting the original image;
The subject identifying apparatus according to claim 1, further comprising: a projecting unit that projects the corrected image.   The subject identification device according to claim 4, wherein the correction of the projection source image by the correction unit uses a homography matrix obtained by the collation. A depth image acquisition step of capturing an object and acquiring a depth image;
An image acquisition step of acquiring an image by photographing the object;
A surface detection step of detecting a surface from the depth image;
When a surface is detected in the surface detection step, an image processing step of cutting out an image area corresponding to the detected surface from the image acquired in the image acquisition step;
A feature amount extracting step for extracting a feature amount from the image region cut out in the image processing step;
A collation step for collating the feature quantity extracted in the feature quantity extraction step with the feature quantity extracted from the image to be collated;
And a step of outputting information for identifying a subject of the image to be collated when it is determined that the feature amounts match as a result of the collation.   The object identifying method according to claim 6, wherein the feature amount extracting step extracts a feature amount from an image region having a predetermined size or larger among the image regions cut out in the image processing step. Computer
A depth image acquisition means for acquiring a depth image of the object;
Image acquisition means for acquiring an image of the object;
A surface detecting means for detecting a surface from the depth image acquired by the depth image acquiring means;
Image processing means for cutting out an image area corresponding to the surface detected by the surface detection means from the images acquired by the image acquisition means;
Feature amount extraction means for extracting feature amounts from the image region cut out by the processing means;
A program for functioning as a collating unit that collates a feature amount extracted by the feature amount extracting unit and a feature amount extracted from an image to be collated.