JP2003223639A - Object identification information recognition system and method, image input device, and object identification information recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object identification information recognition system prevented from being restricted by a reading direction and a distance from an object identification information code when reading the object identification information code. <P>SOLUTION: The object identification information code comprising a plurality of feature points peculiar to the object and a first geometric invariant corresponding to the object identification code are produced, an image input device provides a plurality of images of the object added with the object identification information codes from a plurality of viewpoints, a plurality of independent images are produced out of the plurality of images, and feature points positions constituting the object identification information code are selected for the plurality of independent images. A second geometric invariant is calculated from the selected feature points positions and the object identification information code being the closest to the calculated second geometric invariant and corresponding to the first geometric invariant is detected. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to recognition of object identification information, and more particularly to recognition of object identification information used in an article management system in retail, distribution, advertisement and the like.

[0002]

2. Description of the Related Art Identification of an object is generally performed using an object identification information code such as a tag for identifying the object. In particular, recently, a tag that is a non-contact type tag and that reads information wirelessly, such as an RFID that allows rewriting of information in the tag, has been used. Also,
As another tag, a tag such as a bar code which uses a dedicated reading device to read information is also used. Further, as an object identification information code for reading information using an image input device, an object identification information code using CyberCode, a digital watermark or the like is also used.

[0003]

However, a tag such as RFID that reads information wirelessly has a problem that it has a narrow range in which it can communicate with the tag and is expensive. On the other hand, there is a problem that a dedicated reading device is required to read the bar code, and the bar code cannot be read unless it is close to the tag. Furthermore, for a tag in the form of embedding an ID (identification) code in an image using CyberCode, a digital watermark, etc., there is a constraint that the image must be acquired almost from the front.

The present invention has been made in view of the above problems, and when an object identification information code such as a tag is read, the object is not restricted by the reading direction or the distance from the object identification information code. The main object is to provide an identification information recognition system and an object identification information recognition method.

[0005]

In order to solve the above-mentioned problems, the present invention is an object identification information recognition system for specifying an object from an image acquired by an image input device,
Unique to the object, means for generating an object identification information code composed of a plurality of feature points and a first geometric invariant corresponding to the object identification code, and an object identification information code generated by the generating means, A means for storing the first geometric invariant for each object, a means for acquiring a plurality of images of the object to which the object identification information code is added from a plurality of viewpoints by the image input device, and an image Object identification information code for each of the means for generating a plurality of independent images from the plurality of images acquired by the means and the plurality of independent images generated by the means for generating an image. Means for extracting the position of the feature point forming the second geometrical invariant from the feature point position extracted by the extracting means, and the second geometry calculated by the calculating means. Invariant Nearest provides object identification information recognition system having means for detecting the object identification information code corresponding to the first geometric invariant stored in the means for storing.

According to the present invention, when an object identification information code such as a tag is read, it is possible to realize an object identification information recognition system which is not restricted by the reading direction or the distance from the object identification information code.

[0007]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing the principle of the present invention. Figure
1 to generate the object identification information code and its geometric
Calculation of invariants and multiple images acquired by the image input device
Regarding the principle of the present invention for obtaining a geometric invariant from an image
Explain. Reference numeral 101 indicates a physical space
Each point O in space 101₁， O_Two， O_Three， O_Four， O₅And O ₆
Is the object identification information code on the identified object.
It is the characteristic points that are geometrically arranged to configure. See also
The number 102 is an object identification information code attached to the object
, Showing the projection plane acquired by the image input device,
Each point P on the plane 102₁， P_Two， P_Three， P_Four， P₅And P
₆Are the points O in the physical space 101, respectively.₁， O_Two， O
_Three， O_Four， O₅And O₆Corresponding to.

First, the calculation of the geometric invariant of the geometrically arranged feature points forming the object identification information code will be described. The geometric invariants of the geometrically arranged feature points forming the object identification information code are calculated as follows.

First, the feature point O in the physical space 101.
₁， O_Two， O_Three， O_Four， O₅About, O ₁Physical space 10
The coordinate within 1 is (1, 0, 0, 0), and O_TwoPhysical space
The coordinates within 101 are (0, 1, 0, 0), and O_ThreePhysics
The coordinates in space 101 are set to (0,0,1,0), and O_Fourof
The coordinates in the physical space 101 are changed to (0,0,0,1)
O₅The coordinates in the physical space 101 of (1, 1, 1,
Spatial collinear transformation matrix A to be transformed into 1)_4x4Calculate
Put out. That is, the spatial collinear transformation matrix A_4x4Is the following relationship
Is a matrix that satisfies.

[0011]

[Equation 1] Next, the coordinates of O _{6 in} the physical space 101 are transformed by this space collinear transformation matrix A _4x4 , and (X, Y, Z, T)
To get That is, the spatial collinear transformation matrix A _{4 × 4} gives O ₆
The coordinates of are converted as follows.

[0012]

[Equation 2] By these X, Y, Z, and T, geometric invariants α, β, γ
Can be calculated as α = X / T, β = Y / T, γ = Z / T. In this way, the first geometric invariant at the time of generating the object identification information code composed of the geometrically arranged feature points can be obtained. The object identification information code and the first geometrical invariant generated in this way are compared with the second geometrical invariant calculated from the image acquired by the image input device described later. , Is stored in the object identification information recognition system together with the object identification information code for each object.

On the other hand, the number of images acquired by the image input device is n, and the image ID (identifier) of each image is k = 1,
2 ,. ． , N. In this embodiment, for example, n = 4.

Here, for each image, the features on the projection surface
Point P₁， P_Two， P_Three， P_FourAgainst P ₁In the projection plane 102 of
Set the coordinates at (1,0,0) to P_TwoWithin the projection plane 102 of
Coordinate of (0,1,0), P_ThreeIn the projection plane 102 of
Set the coordinates to (0,0,1) and P_FourProjection plane 102
The plane to convert the coordinates inside to (1,1,1) respectively
Collinear transformation matrix A_3x3To calculate. That is, spatial collinear transformation
Matrix A_4x4Is a matrix satisfying the following relation.

[0015]

[Equation 3] Next, using this plane collinear transformation matrix A _3x3 , P ₅ and P
By converting the coordinates of _{6 on} the projection plane 102, (u ₅ ,
v ₅ , w ₅ ) and (u ₆ , v ₆ , w ₆ ) are obtained. That is, the coordinates of P ₅ and P ₆ are transformed as follows by the spatial collinear transformation matrix A _{3 × 3} .

[0016]

[Equation 4] Here, I ₁ = XY, I ₂ = XZ, I ₃ = XT and I ₄ = YZ, I ₅ = YT, I ₆ = ZT are defined.

Further, i₁ ^(K)= (U₅ ^(K)-V₅ ^(K)) W₅ ^(K) i_Two ^(K)= (W₅ ^(K)-U₅ ^(K)) V₅ ^(K) i_Three ^(K)= (V₅ ^(K)-W₅ ^(K)) U₅ ^(K) i_Four ^(K)= (U₆ ^(K)-V₆ ^(K)) W₆ ^(K) i₅ ^(K)= (W₆ ^(K)-U₆ ^(K)) V₆ ^(K) i₆ ^(K)= (V₆ ^(K)-W₆ ^(K)) U₆ ^(K) To define. Here, in this embodiment, k = 1 to 4
is there.

As defined above, I ₁ , I ₂ , I ₃ ,
I ₄ , I ₅ , I ₆ and i ₁ ^(k) , i ₂ ^(k) , i ₃
^{Between (k)} , i ₄ ^(k) , i ₅ ^(k) , and i ₆ ^(k) , the following invariant relations, Σ (i _j ^(k) I _j ) = 0 and Σ
(I _j ^(k) ) = 0 holds. Here, j is a feature point ID, j = 1 to 6 in the present embodiment, and k = 1 to 4 in the present embodiment.

From the above, the plane collinear transformation matrix A_3x3To
Therefore, P on the projection plane 102₅And P ₆Transform the coordinates of
Respectively (u₅, V₅, W₅) And (u₆，
v₆, W₆), Then satisfy the above invariant relation
I₁, I_Two, I_Three, I_Four, I₅, I₆If you ask
For example, the following equation, α = I₁/ I₅, Β = I₁/ I_Three, Γ = I
_Two/ I_ThreeTo obtain the invariants α, β, γ
It Note that I₁, I_Two, I_Three, I _Four, I₅, I₆From
The relations for obtaining the invariants α, β, γ are limited to the above relations.
Not of.

For example, the following equation,

[0021]

[Equation 5] According to, I1 ₁ to I1 ₄ and I2 ₁ to I2 ₄ are calculated. Then, α = I2 ₂ −I2 ₁ , β = − (I2 ₁ / I1 ₂ ) I1 ₄ + I2 ₄ and γ = (I2 ₁ I1 ₂ −I1 ₂ I2 ₂ ) / (I2 ₁ I1 _3).
The invariants α, β, and γ can be obtained by −I1 ₂ I2 ₃ ).

In this way, the second geometric invariant can be calculated from each image of the plurality of images acquired by the image input device.

FIG. 2 shows a conceptual block diagram of one embodiment of the overall system of the present invention. The system includes object identification information codes 201, 202 generated according to the present invention.
And 211 to which objects 203 and 203 are attached respectively
And 213, a portable information terminal 220 having an image input device 221, a mobile communication network 230, a processing device 241, and an object identification ID database 242. The processing device 241 is not limited to the illustrated configuration, but may be a personal computer main body 243 and a monitor 244.
And the like. The object identification ID database 242 stores the object identification information code and the first geometric invariant, as described above.

Next, the operation of one embodiment of the entire system of the present invention shown in FIG. 2 will be described with reference to FIG.

FIG. 3 shows a flow chart of an embodiment of the invention, in particular the above-mentioned objects 211, 21 to be identified.
Object identification information codes 201 and 2 attached to
This is an example in which the geometrically arranged feature points forming 02 and 203 are read into the portable information terminal 220 by the image input device 221 to identify the object.
The outline of the identification processing of the present invention will be described with reference to FIG.

In FIG. 3, step 301 is performed in this embodiment.
Then, in step 302, the image input device 22 is started.
According to 1, a plurality of image frames are acquired. Next, in step 303, the correlation between adjacent image frames between the acquired plurality of image frames is calculated, and the image frames are extracted. In this embodiment, for example, three image frames are extracted in order from the one having the smallest correlation value between the image frames. Then, with these three image frames as a delimiter,
In this embodiment, a plurality of consecutive image frames are
Divide into four consecutive image frame groups.

Next, in step 304, step 3
For the image frames grouped in 03, an average image is calculated for each group. By calculating the average image, a new image frame whose pixel value is the average value of the corresponding pixels of the image frame is generated. In this way
By generating four average images of four groups, it is possible to obtain images from four independent viewpoints.

Next, in step 305, the positions of the geometrically arranged feature points forming the object identification information code are detected for each image from four independent viewpoints. Then, in step 306, the geometric invariant is calculated based on the position information of each feature point of each image from the four independent viewpoints detected in this way, and the second geometry described above is calculated. The invariant is calculated.

Finally, in step 307, the second geometric invariant calculated in step 306 is compared with the aforementioned first geometric invariant registered in the object identification information recognition system, The best match is selected to determine the object identification code of the object having that first geometric invariant.

Next, the detailed operation of the embodiment of the present invention will be described.

FIG. 4 is a conceptual diagram for calculating an image from four independent viewpoints based on a series of a plurality of images acquired by the image input unit.

The object identification information code 201 attached to the surface of the object shown in FIG. 1 is set to the position of the image input device 221 as shown in step 302 of FIG.
It is acquired by the image input device 221 while changing from 1 to 414. The images denoted by reference numerals 421 to 424 are
The object identification information code 201 observed at positions 411 to 414 is shown. Continuous image frames 431 to 4
42 indicates a plurality of acquired image frames.

Next, as shown in step 303 of FIG. 3, the correlation between adjacent image frames between the obtained plurality of image frames is calculated, and the correlation value between the three image frames is minimized. Extract the image frame of. In FIG. 4, the image frames 433, 436, and 439 correspond to the above-mentioned three image frames. Then, the three image frames 433, 436, and 439 are used as delimiters,
The continuous image frames 431 to 442 are divided into four continuous image frame groups. In FIG. 4, the first
The image frame groups of are image frames 431, 432, 4
It consists of 33. The second image frame group is composed of image frames 434, 435, 436. The third image frame group includes image frames 437, 438, 43.
It is composed of 9. Then, the fourth image frame group is
It is composed of image frames 440, 441, 442.
The number of images forming the image frame group is not limited to this.

Next, as shown in step 304 of FIG. 3, an average image is calculated for each of the first to fourth groups for the three image frames grouped as described above. By calculating the average image, new image frames 451, 452, 453, and 454 having the average value of the corresponding pixels of the three image frames as the pixel value are generated. In this way, images from four independent viewpoints are obtained by generating four average images of four groups.

FIG. 5 shows a flow chart of an embodiment for calculating the position information of the feature points in the image. The flowchart shown in FIG. 5 is an example of the method of calculating the position information of the feature points in the image shown in step 305 of FIG.

The following processing is performed on each of the images from the four independent viewpoints generated as described above.

In step 501, color conversion processing is performed on the image input to this processing. The color conversion processing is performed on the RGB of the image composed of the input RGB signals.
An expression whose value is

[0038]

[Equation 6] According to the above, color conversion into I ₁ I ₂ I ₃ values is performed.

In step 502, preprocessing is performed on the image input to this processing. In the preprocessing, the image is enhanced by smoothing the image with a Gaussian filter to remove noise existing in the image and by expanding the dynamic range of the histogram.

Next, in step 503, extraction processing of the tag area is performed. Before detecting the tag area, I
_{In the 1} I ₂ I ₃ color space, a representative color indicating the boundary line of the tag is defined. The boundary line of the tag is extracted by selecting a pixel having a color close to this representative color in the image. Then, the inside of the boundary line is determined as the tag area.

In step 504, color segmentation processing is performed. In the color segmentation process, six representative colors of each feature point are defined in advance in the I ₁ I ₂ I ₃ color space. Then, by selecting a pixel having a color close to the representative color in the tag area in the image, each pixel is separated and extracted for each color.

In step 505, SUSAN filter processing is performed on each pixel separated and extracted in the above step to perform corner detection. In the corner detection using the SUSAN filter, the number of pixels in which a difference between adjacent pixel values has a large difference of a predetermined value or more is counted within a circular small area centered on each extracted pixel. And
A place having a large count value, that is, a place having a high corner strength is detected as a corner. The detected information is corner position information and the corner strength value.

Next, in step 506, the center of gravity is calculated by weighting the position information of the corner detected in the above step with the corner strength at that position. Then, the position of this center of gravity is determined as the position of the feature point.

By carrying out the above processing for each of the six feature points in each image from four independent viewpoints, the position information of the feature points in the image can be calculated.

Next, as described in step 306 in FIG. 3, the six feature points in each image from the four independent viewpoints calculated in this way are used to describe in FIG. Geometric invariants α, β, γ are calculated. As a result, a set of four geometric invariants is calculated for each image from four independent viewpoints.

Next, as described in step 307 in FIG. 3, the second geometrical invariant calculated as described above is registered in the object identification information recognition system, and the second geometric invariant described above is registered. One of the geometric invariants is compared, and the best match is selected to determine the object identification information code of the object having the first geometric invariant. In this process, two geometric invariants are compared using the Mahalanobis distance as a criterion.

The Mahalanobis distance is calculated as follows. For a pattern belonging to a certain cluster, if the average pattern is μ and the covariance matrix is Σ, the unknown pattern Pi
The Mahalanobis distance D between a and its cluster is calculated by D = (Pi-μ) ^T (Σ ⁻¹ ) (Pi-μ). Here, Σ ⁻¹ is the inverse matrix of the covariance matrix Σ, and (Pi−μ) ^T is the transposed matrix of (Pi−μ).

In the present embodiment, the patterns forming the cluster are 4 for each image from four independent viewpoints.
A set of two geometric invariants, that is, the above-mentioned second geometric invariant (α, β, γ). Unknown pattern Pi
Is the first geometric invariant at the time of generating the above-described object identification information code, which has been described with reference to FIG. And first of all,
For a set of four geometric invariants (α, β, γ),
Their average pattern μ and the covariance matrix Σ are calculated.

Next, the Mahalanobis distance D is calculated from the first geometric invariant at the time of generating the object identification information code, the above-mentioned average pattern μ, and the covariance matrix Σ.

In the calculation of the Mahalanobis distance D, the first geometric invariant is registered in the database 242 of FIG. 2 together with those object identification information codes. Therefore, by calculating the first geometric invariant having the smallest Mahalanobis distance D and searching the object identification information code corresponding to the first geometric invariant in the database 242, the object identification information is obtained. The code can be detected.

In this embodiment, the Mahalanobis distance is used as a criterion to compare two geometric invariants. However, for example, another distance such as Euclidean distance is used as a criterion, and two geometric invariants are used as the criterion. It is also possible to make a comparison of the geometric invariants.

The object identification information code thus extracted can be returned from the processing device 241 to the portable information terminal 220 via the mobile communication network 230 in the system of the present invention shown in FIG. .

[0053]

According to the present invention, when an object identification information code such as a tag is read, it is possible to realize an object identification information recognition system which is not restricted by the reading direction or the distance from the object identification information code. The effect can be obtained.

Further, according to the present invention, images from four independent viewpoints necessary for calculating the geometric invariant can be automatically acquired based on the correlation of consecutive frames.

Further, according to the present invention, the geometrical arrangement of the feature points expressing the object identification information code is arbitrary 6 in space.
Since it is composed of dots, various expressions can be generated depending on the application.

[Brief description of drawings]

FIG. 1 is a diagram showing the principle of the present invention.

FIG. 2 is a diagram showing a conceptual block configuration of one embodiment of the entire system of the present invention.

FIG. 3 is a diagram showing a flowchart of an embodiment of the present invention.

FIG. 4 is a conceptual diagram of calculating images from four independent viewpoints based on a series of a plurality of images acquired by an image input unit.

FIG. 5 is a diagram illustrating a flow of an example of calculating position information of feature points in an image.

[Explanation of symbols]

201, 202, 203 Object identification information code 211, 212, 213 objects 221 image input device 220 Personal Digital Assistant 230 mobile communication network 241 processor 243 Personal computer body 244 monitor

Continued front page    (72) Inventor Toshiaki Sugimura             2-11-1, Nagatacho, Chiyoda-ku, Tokyo Stock             Ceremony company NTT Docomo F term (reference) 5B057 AA02 CH01 DA11 DC34 DC36                 5B058 CA40 KA02 KA08 YA18                 5L096 BA20 CA03 DA02 FA09 FA34                       FA69 HA09 JA03

Claims (7)

[Claims] 1. An object identification information recognition system for identifying an object from an image acquired by an image input device, the object identification information code being unique to the object, the object identification information code including a plurality of feature points, and the object identification. Means for generating a first geometric invariant corresponding to the code, means for storing the object identification information code and the first geometric invariant generated by the means for generating, for each object A means for acquiring a plurality of images of the object to which the object identification information code is added from a plurality of viewpoints by the image input device; and a plurality of the images acquired by the means for acquiring the image. From the means for generating an independent plurality of images, and before configuring the object identification information code for each of the independent plurality of images generated by the means for generating the image Means for extracting a feature point position; means for calculating a second geometric invariant from the feature point position extracted by the extracting means; and the second geometry calculated by the calculating means. Object identification information recognition system having means for detecting the object identification information code corresponding to the first geometrical invariant stored in the storing means, which is closest to the geometric invariant. 2. The object identification information recognition system according to claim 1, wherein the plurality of independent images are four independent images. 3. A means for acquiring a plurality of images of the object to which the object identification information code is added from a plurality of viewpoints, and an independent image from the plurality of images acquired by the means for acquiring the image. Image input device having means for generating a plurality of different images. 4. The image input device according to claim 3, wherein the plurality of independent images are four independent images. 5. An object identification information code, which is unique to an object and is composed of a plurality of feature points, and means for generating a first geometrical invariant corresponding to the object identification code; The object identification information code and the first geometric invariant, means for storing each object, means for inputting a plurality of independent images of the object to which the object identification information code is added, A second geometric invariant is calculated from the feature point position forming the object identification information code for each of the plurality of independent images, and the feature point position extracted by the extracting unit. Calculating means, and the object closest to the second geometric invariant calculated by the calculating means and corresponding to the first geometric invariant stored in the storing means. Identification code Object identification information recognition device having means for detecting. 6. An object identification information recognition method for identifying an object from an image acquired by an image input device, the object identification information code being unique to the object, the object identification information code including a plurality of feature points, and the object identification. Generating a first geometric invariant corresponding to the code, storing the object identification information code and the first geometric invariant generated by the generating step, for each object A step of acquiring a plurality of images of the object to which the object identification information code is added from a plurality of viewpoints by the image input device; and a step of acquiring the images from the plurality of images. From the step of generating a plurality of independent images, and the object identification for each of the plurality of independent images generated in the step of generating the image. Calculating a second geometric invariant from the feature point positions extracted by the extracting step; calculating the feature point position forming the information code; Object identification information that is closest to the second geometric invariant and that detects the object identification information code corresponding to the first geometric invariant stored in the storing step. Recognition method. 7. The object identification information recognition method according to claim 6, wherein the plurality of independent images are four independent images.