JP7061092B2 - Image processing equipment and programs - Google Patents

Description

The present invention relates to an image processing apparatus and a program capable of estimating information regarding a mirror image relationship in an image by simple processing.

When three-dimensional measurement is performed using a captured image, it is impossible to perform three-dimensional measurement because a scene is reflected in a reflective object such as a mirror. That is, since it is not possible to distinguish whether the scene is wide or the scene is reflected, it is misunderstood that the depth exists in the reflective region. As a solution, a technique for excluding the measurement from the reflective region in the three-dimensional measurement is required. Patent Document 1 discloses a region specifying device for specifying a reflective region. Patent Document 1 projects a pattern having a predetermined motion, calculates a motion vector of a virtual image of the pattern reflected in a mirror from a captured image of the projected landscape, and then calculates a motion different from the motion of the pattern. The area where the vector is generated is specified as the reflective area.

Japanese Unexamined Patent Publication No. 2014-048276

However, the above-mentioned conventional techniques have the following problems. Patent Document 1 has a problem that the device becomes large because a projection unit for projecting a pattern is required. Further, even if the projection unit can be made smaller, there is a problem that the positional relationship in which the image pickup unit can image the projection unit reflected in the reflective region becomes limited and local.

This problem of specifying the reflective region can be reduced to the problem of estimating the mirror image relationship caused by the presence of the reflective region in the captured image. The mirror image relationship itself can occur even in the absence of a reflective region.

In view of the above problems of the prior art, it is an object of the present invention to provide an image processing apparatus and a program capable of estimating information regarding a mirror image relationship in an image by simple processing.

In order to achieve the above object, the present invention is an image processing apparatus, and the feature point and the feature amount at the feature point and / or the inverted feature amount obtained by mirror-inverting the feature amount are used as feature information from the image. When the calculation unit to be calculated, the collation unit that collates the feature information with each other and obtains the collation information as the collation information that is determined to match the feature amount and the inverted feature amount with each other, and when the collation information is obtained. It is characterized by including an estimation unit for estimating that there is a mirror image relationship in the image. Further, the program is characterized in that the computer functions as the image processing device.

According to the present invention, it is possible to estimate information regarding a mirror image relationship in an image by a simple process called a collation process.

It is a functional block diagram of the image processing apparatus which concerns on one Embodiment. It is a figure which shows typically the mode of the image pickup in this Embodiment by showing the example of the arrangement of the image pickup part and a reflective area. It is a figure which shows the schematic example for demonstrating the inversion in a feature space in the case where a marker is a square marker or the like and the feature points are four corners. It is a figure which shows the schematic example when the captured image is processed by the image processing apparatus. It is a figure which shows the schematic example of the processing in a processing part. It is a figure which shows the hardware configuration in a general computer device.

FIG. 1 is a functional block diagram of an image processing apparatus according to an embodiment. The image processing device 10 includes an image pickup unit 1, a calculation unit 2, a collation unit 3, an estimation unit 4, and a processing unit 5. The processing contents of each part are as follows.

The image pickup unit 1 is composed of a camera as hardware, receives a user operation or the like to capture a scene including a reflective region by a mirror or the like, and outputs the captured image to the calculation unit 2 as an image pickup image.

FIG. 2 is a diagram schematically showing an embodiment of imaging in the present embodiment by showing an example of arrangement of an imaging unit 1 and a reflective region. Although FIG. 2 is shown in two dimensions (cross section) for ease of explanation, it is actually configured in three dimensions. The point A in FIG. 2 represents the camera coordinates of the imaging unit 1, and the line segment CD indicates the reflective region. When the point B1 is present in the scene of the captured image, it is reflected as the point E on the mirror surface CD, and the image pickup unit A can see that a virtual image exists at the symmetry point B1'with respect to the line segment CD. (Here, according to the law of reflection, ∠B1-E-C = ∠B1'-E-C, and line segment length B1-E = line segment length B1'-E.) That is, point B1 and point B1'in the captured image. Is imaged. Similarly, when points B2 and B3 are present in the scene of the captured image, B2'and B3' are also captured as virtual images in the captured image.

In the calculation unit 2, the calculation unit uses the feature point p _i (i = 1,2, ...) (feature point p _i = specified by the position coordinates of the pixels in the captured image) from the captured image captured by the image pickup unit. (u _i , v _i )), the feature quantity f _i around this feature point p _i , and the inverted feature quantity R (f _i ), which is a mirror image of this feature quantity f _i , calculated. The feature information (p _i , f _i , R (f _i )) (i = 1, 2, ...) Is output to the collation unit as a link between the point, the feature amount, and the inverted feature amount.

A known marker or the like (a square marker or the like in augmented reality) may be used for the feature points and feature quantities to be calculated. When a marker or the like is used, the feature points correspond to the points around the marker (for example, the four corners of a square marker), and the feature amount corresponds to the type of marker. Alternatively, a pattern that can be detected as a natural feature amount is further provided in the marker, and a local image feature amount such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Features) can be used.

Further, for both the feature points and the feature quantities to be calculated, the feature points and the feature quantities in the local image feature quantities such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) can be used.

Regarding the method of obtaining the inverted feature amount R (f _i ) from the feature amount f _i , the following first to third embodiments are possible.

In the first embodiment, the feature amount inversion process is performed in the feature space. Since high-speed calculation is possible, the effect of shortening the processing time can be obtained.

For example, when a marker or the like is used, the projective transformation coefficient is calculated and applied by arranging the feature points in the reverse order to specify the marker type as the feature amount. That is, the marker specification (unlike the present embodiment, in a normal case where the inversion of the marker is not considered) generally comprises the following steps 1 and 2.
(Procedure 1) List possible correspondence patterns between the feature points of the marker detected from the captured image and the feature points of the marker (reference points surrounding the marker) registered in advance as a reference. (If N feature points are lined up on the outer circumference, the pattern will be N ways.)
(Procedure 2) In each of the above-listed mapping patterns, the reference marker is transformed by planar projective transformation, superimposed on the marker region in the captured image, and the difference between the regions (evaluate by the sum of squares of differences, etc.). The matching pattern is the correct answer (correspondence to feature points and the correct answer for markers determined by this).

In the present embodiment, only one reference marker in a normal state where the shape and pattern of the reference marker are not inverted is prepared, and the case of being inverted as the pattern of (Procedure 1) is additionally considered (feature). When N points are lined up in an outer shape, a total of 2N patterns of N patterns when not inverted and N patterns when inverted are taken into consideration), and the marker type (including identification of the presence or absence of inversion is included) at high speed. ) Can be specified. In (Procedure 2), it can be determined that the marker is in the non-inverted state if the correct answer is obtained from the non-inverted pattern, and the marker is in the inverted state if the correct answer is obtained from the inverted pattern.

FIG. 3 is a diagram showing a schematic example for explaining inversion in the feature space when the marker is a square marker or the like and the feature points are four corners (N = 4). The reference marker RL defines a reference pattern (not shown) when the vertices r1, r2, r3, and r4 are defined in the order of upper left, upper right, lower right, and lower left (clockwise). On the other hand, it is assumed that p1, p2, p3, and p4 are detected as feature points in the clockwise direction as candidates for the four vertices of the marker from the captured image. In this case, there are the following four ways of normal rotation NO that does not reverse.
(p1 → p2 → p3 → p4), (p2 → p3 → p4 → p1), (p3 → p4 → p1 → p2) and (p4 → p1 → p2 → p3),
In addition, the following 4 ways of reversing RO
(p2 → p1 → p4 → p3), (p1 → p4 → p3 → p2), (p4 → p3 → p2 → p1) and (p3 → p2 → p1 → p4),
Corresponds to the vertices (r1 → r2 → r3 → r4) in the order of upper left, upper right, lower right, and lower left, respectively. All you have to do is allocate it to the area and find the one that best matches. (Conversely, the marker area in the captured image may be deformed and assigned to the pattern of the reference marker RL.)

In the case of plane projection conversion, if the conversion is performed by associating the inversion rotation RO, the pattern of the reference marker RF will also be mirror-inverted, so for the reference marker, only the pattern of the normal pattern that does not invert should be registered. good.

In the above, the case where the marker (pattern) is used as one type and the feature point assignment of the marker is performed in the feature point space in consideration of inversion has been described. However, when there are two or more markers, the same procedure is performed for each marker. And the most matching marker type may be used as the calculation result.

The inversion in the feature space in the case of a marker or the like has been described above, but when SIFT is used as the feature amount, the inversion feature is inverted from the feature amount f _i in the feature amount by the method disclosed in Non-Patent Documents 1 and 2 below. The quantity R (f _i ) can be obtained.
[Non-Patent Document 1] X. Gat, et al., `` MIFT: A Mirror Reflection Invariant Feature Descriptor,'' ACCV, pp. 536-545, 2009.
[Non-Patent Document 2] M. Su, et al., `` MBR-SIFT: A mirror reflected invariant feature descriptor using a binary representation for image matching,'' PloS One, vol. 12, no. 5, 2017.

In the second embodiment, in the case of using a marker or the like, in addition to the marker M _j (j = 1,2, ...), the inverted marker R (M _j ) (j = 1,2, ...) Register so that it can be recognized. At this time, by associating the inversion marker R (M _j ) with the original marker M _j before inversion, the feature amount is uniformly calculated from the captured image, and the feature amount in the area corresponding to the inversion marker is calculated. Inverted features. At this time, the feature amount (marker type) has a one-to-one correspondence with the feature points such as markers, but information on whether or not this marker type is inverted is given.

That is, (Procedure 1) and (Procedure 2) described in the first embodiment are carried out as usual in (Procedure 1) without considering inversion, and as a candidate marker, the normal marker M _j In addition, the inversion marker R (M _j ) may be used.

In this case, the feature information (p _i , f _i , R (f _i )) already described may be obtained by modifying the data format as follows, for example. If a feature point is detected as a normal marker, the feature information (p _i , f _i , ・) (i = 1,2,…) (“・” is the inversion feature R (f _i ) and the normal feature f It is assumed that only _i is detected, that is, it is a feature point corresponding to a normal marker), and if a feature point is detected as an inversion marker, feature information (p _i , ·, R (f _i )). (“・” Indicates that only the inversion feature R (f _i ) is detected without the corresponding feature f _i , that is, it is a feature point corresponding to the inversion marker).

When the inversion marker R (M _j ) is obtained from the marker M _j , it may be inverted with respect to a predetermined axis on the image, for example, it may be inverted horizontally with respect to the central axis (vertical axis) in the horizontal direction, or vertically. It may be turned upside down with respect to the central axis (horizontal axis) in the width direction, or may be turned upside down with respect to any diagonal axis.

In the third embodiment, the feature point p _i and the feature amount f _i of the captured image are obtained, and then the corresponding inverted feature amount R (f _i ) is obtained from the mirror image inverted image of the captured image (referred to as an inverted image). be able to.

Here, the position of the feature point in the inverted image is obtained as the inverted position R (p _i ) of the original feature point p _i by using the transformation used when obtaining the inverted image from the captured image. The feature point detection process in the above can be omitted. _Similarly , when local features such as _SIFT are used, the _inverted region R (N (N (N (N p _i )) (R (N (p _i )) ⊂ inverted image) is obtained, and the inverted feature R (f _i ) is detected for this inverted neighborhood region R (N (p _i )). , It is possible to save the trouble of inverting the entire image. When a marker or the like is used as a feature amount, only the marker region (the region containing the marker feature points) surrounded by the marker feature points in the original captured image is inverted, and the marker is detected from this inverted marker region. You can do it.

As described above, the feature point p _i and the feature quantity f _i are obtained, and the inversion feature quantity R (f _i ) is further obtained from the information such as the inversion position R (p _i ), and (inversion position R (p _i ) etc. Feature information (p _i , f _i , R (f _i )) can be obtained without including the information of. Regarding the conversion for obtaining the inverted image from the captured image, as in the case of obtaining the inverted marker R (M _j ) from the marker M _j in the second embodiment, the conversion is inverted with respect to a predetermined axis on the image or the region. Should be used.

The collation unit 3 collates the feature information obtained from the calculation unit 2, and outputs the feature information determined to be in a mirror image relationship to the estimation unit 4 as collation information.

In this collation process, each feature information (p _i , f _i , R (f _i )) is used as a query, and all other feature information (p _j , f _j , R (f _j )) seen from this query. Using (j ≠ i) as a reference, search for a reference that is judged to have a mirror image relationship with the query. Specifically, the corresponding query feature amount f _i is obtained by searching for the inversion feature amount R (f _j ) of the reference that is determined to match the feature amount f _i of the query. It suffices to obtain a judgment that the reference feature quantity f _j and the reference feature quantity f j are in a mirror image relationship (expressed as MR (f _i , f _j )).

Judgment is consistent when the distance d (f _i , R (f _j )) between these feature quantities is less than the predetermined threshold value TH for judgment (when the following equation (1) holds). It may be determined that.
d (f _i , R (f _j )) <TH… (1)

Alternatively, among the feature quantities f _j of the reference, those that are determined to match by satisfying the following equation (2) with respect to the inversion feature quantity R (f _i ) of the query are mirror image-related MRs ( You may search as if it is in f _i , f _j ).
d (R (f _i ), f _j ) <TH… (2)

Alternatively, a common index i, j that satisfies both of the above equations (1) and (2) may be determined to be in the mirror image relation MR (f _i , f _j ).

Here, in the collation process, the following first additional process and / or second additional process may be performed.

(1st additional processing)
Of each feature information (p _i , f _i , R (f _i )), the distance d (f _i , R (f _i )) between the feature amount f _i and the inverted feature amount R (f _i ) is set in advance. For those smaller than the threshold value TH2 (those for which the following equation (3) holds), they are excluded from the target used as a query and reference, that is, the collation target to suppress the occurrence of erroneous collation. You may do so.
d (f _i , R (f _i )) <TH2… (3)

(Second additional processing)
The captured image is subjected to region division processing by any existing method (for example, region division by the Mean shift method using the color histogram, region division by object detection using deep learning such as a convolutional neural network, etc.) and querying. When the feature point p _i of each feature information (p _i , f _i , R (f _i )) is in the region V in the region division result (p _i ∈ V), it corresponds to the same region V. The feature information (p _j , f _j , R (f _j )) (j ≠ i) to which the feature point p _j belongs (p _j ∈ V) may be excluded from the reference.

When the mirror image relation MR (f _i , f _j ) occurs by the principle as explained in FIG. 2, the situation where the corresponding feature points p _i and p _j are included in the same region V (p _i ∈ V and p _j ∈). Since the situation of V) is considered to be rare, the second additional processing can reduce unnecessary collation processing and speed up collation.

The above explanation is based on the premise that the feature information is obtained as a combination of the feature point, the feature amount and the inverted feature amount (p _i , f _i , R (f _i )). By using the second embodiment in the calculation unit 2, the inversion marker R (M _j ) is registered in advance in addition to the normal marker M _j , and the feature information (p _i , f _i , ·) (only the normal feature amount). ) Or the feature information (p _i , ·, R (f _i )) (only the inverted feature amount), the same can be done as follows. That is, the marker M _i calculated as a normal feature is used as a query, and from the inversion marker R (M _j ) calculated as an inversion feature, the inversion marker, that is, the corresponding normal marker R (R (M _j )). If = M _j matches (M _j = M _i ), it can be judged that there is a mirror image relation MR (M _i , R (M _j )).

The estimation unit 4 indicates that there is a mirror image relationship in the captured image when at least one collation information is obtained by the collation unit 3 (that there is at least one mirror image relationship MR (f _i , f _j )). Is output as the estimation result.

The estimation unit 4 further estimates the information of the points belonging to the reflective region and / or the region information of the reflective region as the information of the reflective region in the captured image from the collation information obtained by the collation unit 3, and as the estimation result. You may output it.

Specifically, the world of the captured image corresponding to the feature points p _i = (u _i , v _i ) and p _j = (u _j , v _j ) (pixel coordinates) of the mirror image-related MR (f _i , f _j ). Find the three-dimensional spatial coordinates X _i = (x _i , y _i , z _i ) and X _j = (x _j , y _j , z _j ) as coordinates, and divide these two points into two perpendicular planes P (X _i) . , X _j ) (plane in three-dimensional space) is calculated. Here, in order to obtain the three-dimensional spatial coordinates X _i and X _j , three-dimensional restoration by any existing method may be used. The cameras constituting the image pickup unit 1 may be calibrated in advance to obtain the camera parameters. For example, two feature points p _i = (u _i , v _i ) and p _j = (u _j , v _j ) are tracked on the video, and these two feature points are obtained in at least two images, and the movement is performed. Three-dimensional spatial coordinates X _i and X _j may be obtained by parallax or stereo camera method (epipolar geometry method). Alternatively, if the feature points are configured as vertices of a square marker or the like, the three-dimensional spatial coordinates X _i and X _j may be obtained by obtaining the planar projective transformation matrix.

Furthermore, of the three-dimensional space coordinates X _i and X _j in the mirror image-related MR (f _i , f _j ), the side that is far away in the three-dimensional space at the position of the center of the camera lens constituting the image pickup unit 1 is a virtual image. Assuming that the near side is the real image, it can be estimated that the intersection of the straight line connecting the virtual image and the center of the camera lens and the vertical bisection plane P (X _i , X _j ) belongs to the reflective region. can. If it is known in advance that there is a reflective region of one plane in the captured image by a mirror or the like, the vertical bisection plane P (X _i , X _j ) may be estimated as the reflective region.

In the example of FIG. 2, the point A is the center of the camera lens, the point B1 is the real image, the point B1'is the imaginary image, and the vertically bisected plane is given as the plane to which the sides C-D belong. Estimate that the intersection E of the dividing surface and the straight line A-B1'(the straight line connecting the center A of the camera lens and the imaginary image B1') belongs to the reflective region (as a point in the three-dimensional space). Can be done.

If there are multiple mirror image-related MRs (f _i , f _j ) and multiple vertical bisection planes P (X _i , X _j ) are also obtained, maximum likelihood estimation is performed on these multiple vertical bisection planes. The fitted surface may be estimated as the reflective region.

If a marker M _j such as a square marker and an inverted marker R (M _j ) obtained by reversing the marker M j are obtained in the collation result, the marker M is calculated by calculating the plane projection transformation matrix from the feature points of the markers. _Find the in-space plane PL _j to which _j belongs and the in-space plane RPL _j to which the inversion marker R (M _j ) belongs _. It may be estimated as a region. Here, it is assumed in advance that there is a reflective region of one plane in the captured image by a mirror or the like.

FIG. 4 is a diagram showing a schematic example when the captured image P is processed by the image processing device 10. A reflective region RF exists as a plane of a mirror or the like in the captured image P, and the lower end thereof is indicated by a line L. The marker M1 is captured as a real image and the marker M2 is captured as a virtual image on the captured image P. The feature points p1 to p6 are calculated from within the marker M1, and the feature points p7 to p12 are calculated from within the marker M2. As shown by ~ L6, they are judged to be in a mirror image relationship.

In the processing unit 5, the estimation unit 4 estimates that at least one mirror image-related MR ( _{fi, f j )} exists, and the image is used as a trigger to perform arbitrary processing. It is possible to do.

FIG. 5 is a diagram showing a schematic example of processing in the processing unit 5, in which the first superimposed display AR1 is added to the marker M1 as a real image and the marker M2 as a virtual image is added to the captured image P in FIG. The second superimposed display AR2 is added. These displays AR1 and AR2 may be displayed as being symmetrical with respect to the reflective region RF, that is, as a relationship between a real image and a virtual image by an actual mirror. Any existing method may be used as the method for performing the superimposed display. Further, for example, it is possible to add only the second superimposed display AR2 to only the marker M2 as a virtual image without superimposing the marker M1 which is a real image.

Which is the real image side and which is the virtual image side may be determined by the method already described (determination by the size of the spatial distance from the center of the camera lens) in the mirror image-related MR (f _i , f _j ). In this way, the display AR1 of the first aspect is performed for the marker M1 (the feature point group determined as a real image), and the display of the second aspect is performed for the marker M2 (the feature point group determined as a virtual image). AR2 can be implemented in the processing unit 5. When such displays AR1 and AR2 are superimposed and displayed in the processing unit 5, an optical see-through method may be used or a video see-through method may be used.

The supplementary matters will be described below.

(1) When using only local features such as SIFT without using markers, it is not necessary to prepare reference information (pre-registration information in the case of markers) regarding features in advance, and the calculation unit 2 It is possible to perform collation processing only for the feature information obtained in. Further, when using a local feature amount such as SIFT, reference information regarding the feature amount and / or the inverted feature amount is prepared, and in the calculation unit 2, among all the feature information calculated from the captured image, the feature amount and / or the inverted feature amount are prepared. Only the feature information whose feature amount and / or the inverted feature amount match with any of the reference information may be output to the collation unit 3. As the feature points and features, skeleton data (skeletal joint data) of the human body or the like detected by the existing method such as OpenPose may be used.

(2) FIG. 6 is a diagram showing a hardware configuration in a general computer device 70, and the image processing device 10 can be realized as one or more computer devices 70 having such a configuration. The computer device 70 is a CPU (central processing unit) 71 that executes a predetermined instruction, and a dedicated processor 72 (GPU (graphic calculation device)) that executes a part or all of the execution instructions of the CPU 71 on behalf of the CPU 71 or in cooperation with the CPU 71. And deep learning dedicated processor, etc.), RAM73 as the main storage device that provides a work area for the CPU71 and the dedicated processor 72, ROM74 as the auxiliary storage device, communication interface 75, display 76, camera 77, mouse, keyboard, touch panel, etc. It includes an input interface 78 that accepts user input, and a bus B for exchanging data between them.

Each part of the image processing apparatus 10 can be realized by a CPU 71 and / or a dedicated processor 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74. Here, when the shooting-related processing is performed, the camera 77 further operates in conjunction with the display, and when the display-related processing is performed, the display 76 further operates in conjunction with the communication-related data transmission / reception. When the processing of is performed, the communication interface 75 further operates in conjunction with it.

For example, the captured image may be acquired from the network via the communication interface 75, or may be directly captured and acquired by the camera 77 operated by the user. The processing result (superimposed display) obtained by the processing unit 5 may be displayed on the display 76. When the image processing device 10 is realized as a system by two or more computer devices 70, information necessary for each processing may be transmitted and received via a network.

10 ... image processing device, 1 ... imaging unit, 2 ... calculation unit, 3 ... collation unit, 4 ... estimation unit, 5 ... processing unit

Claims (11)

From the image, a calculation unit that calculates the feature points, the feature amount at the feature point, and the inverted feature amount obtained by mirror-inverting the feature amount as feature information.
A collation unit that collates the feature information with each other and obtains as collation information what is determined that the feature amount and the inverted feature amount match each other.
It is provided with an estimation unit that estimates that there is a mirror image relationship in the image when the collation information is obtained .
The collation unit is an image processing apparatus characterized in that, among the feature information, those whose difference between the corresponding feature amount and the inverted feature amount is determined to be small is excluded from the collation target . From the image, a calculation unit that calculates the feature points, the feature amount at the feature point, and the inverted feature amount obtained by mirror-inverting the feature amount as feature information.
A collation unit that collates the feature information with each other and obtains as collation information what is determined that the feature amount and the inverted feature amount match each other.
It is provided with an estimation unit that estimates that there is a mirror image relationship in the image when the collation information is obtained .
In the estimation unit, the three-dimensional space coordinates in the world coordinate system corresponding to the image are obtained for the feature points corresponding to each other in the collation information, and then the three-dimensional space is obtained from the lens center of the camera that captured the image. An image processing device characterized in that the side close to the coordinates is used as a feature point of a real image and the side far from the coordinates is used as a feature point of a virtual image. In the estimation unit, the intersection of the perpendicular bisected surface of the feature point of the real image and the feature point of the virtual image and the straight line passing through the center of the camera lens of the image and the feature point of the virtual image belongs to the reflective region. The image processing apparatus according to claim 2 , wherein the image processing apparatus is estimated as a thing. The image processing apparatus according to claim 2 or 3 , further comprising a processing unit that performs a first display based on the feature points of the real image and a second display based on the feature points of the virtual image. The image processing apparatus according to any one of claims 1 to 4, wherein the calculation unit calculates the inverted feature amount by mirror-inverting the feature amount in the feature space. The image processing apparatus according to any one of claims 1 to 4 , wherein the calculation unit calculates the inverted feature amount from an inverted image obtained by reversing a mirror image of the image. The image processing apparatus according to claim 6 , wherein the calculation unit obtains a feature point in the inverted image for calculating the inverted feature amount as a mirror image inverted position of the feature point in the image. The calculation unit is characterized in that the inverted image is obtained only in the vicinity of the position where the feature points in the image are mirror-inverted, instead of inverting the entire region of the image to obtain the inverted image. The image processing apparatus according to 6 or 7 . From the image, a calculation unit that calculates the feature points, the feature amount at the feature point, and the inverted feature amount obtained by mirror-inverting the feature amount as feature information.
A collation unit that collates the feature information with each other and obtains as collation information what is determined that the feature amount and the inverted feature amount match each other.
It is provided with an estimation unit that estimates that there is a mirror image relationship in the image when the collation information is obtained .
The feature points are defined as reference points surrounding the planar marker.
The feature amount is a type of the planar marker, and is
The calculation unit allocates a region surrounded by a plurality of feature points extracted from the image in a predetermined order of reference points surrounding the planar marker, and obtains a planar projective transformation between the planar marker and the surrounding region. , The feature amount is calculated by detecting the surrounding area at the time of the best match by the conversion as the area of the planar marker, and
The calculation unit allocates a region surrounded by a plurality of feature points extracted from the image in the reverse order of a predetermined order of reference points surrounding the planar marker, and performs a planar projective transformation between the planar marker and the surrounding region. The image processing apparatus is characterized in that the inverted feature amount is calculated by detecting the surrounding region at the time of the most matching by the transformation as the inverted region of the marker . From the image, a calculation unit that calculates the feature points, the feature amount at the feature point, and the inverted feature amount obtained by mirror-inverting the feature amount as feature information.
A collation unit that collates the feature information with each other and obtains as collation information what is determined that the feature amount and the inverted feature amount match each other.
It is provided with an estimation unit that estimates that there is a mirror image relationship in the image when the collation information is obtained .
The feature points are defined as reference points surrounding the planar marker.
The feature amount is a type of the planar marker, and is
An inverted planar marker that is mirror image inverted is defined for each of the planar markers.
When the planar marker is detected from the image, the calculation unit calculates the type of the planar marker as the feature amount.
The image processing unit is characterized in that when an inverted planar marker is detected from the image, the calculation unit calculates the type of the inverted planar marker as the inverted feature amount . A program characterized in that a computer functions as the image processing device according to any one of claims 1 to 10 .

Images