JP6302427B2 - Image data processing method, image data processing apparatus, and image data processing program - Google Patents

Description

  The present invention relates to an image data processing method, an image data processing device, and an image data processing program.

  External calibration for estimating the position and orientation relationship between a camera and a subject is a technology that is the basis of computer vision, and many studies have been made so far (Non-Patent Document 1). Many of these existing external calibration methods assume that the camera can directly observe the reference object.

  In recent years, with the spread of laptop computers, smartphones, digital signage, and the like, various studies using a display-camera system have attracted attention. One example is interest estimation using digital signage. This is realized by taking a correspondence between the user's line-of-sight direction and the content presented on the display. Furthermore, application to three-dimensional shape restoration is also being studied. This is based on a change in luminance of an observation target accompanying a change in the presentation contents, using the display as a controllable surface light source. In this way, external calibration is performed to determine the positional orientation between the display and the camera in order to geometrically associate the gaze estimated from the video captured by the camera and the luminance change of the observation target with the display content. There is a need. Usually, a chess pattern etc. is presented on the display, and it is photographed with a camera for external calibration. However, in a display-camera system such as a laptop computer or a smartphone, there is generally no display (reference object) in the field of view of the camera, and it has been difficult to apply a conventional external calibration method.

  There has been proposed a technique for performing external calibration by using a mirror in a situation where such a reference object does not exist in the field of view of the camera. In the following, a reference point exists on the reference object, and external calibration is performed using the reference point. In external calibration, research aimed at a simpler configuration is underway from the viewpoint of labor and calculation amount. Non-Patent Document 2 and Non-Patent Document 3 describe a technique that uses three plane mirrors with respect to three reference points, which is a minimum configuration using a plane mirror. On the other hand, from the viewpoint of reducing the number of mirror postures, Non-Patent Document 4 describes a technique that uses a spherical mirror with one posture with respect to eight reference points. Further, Non-Patent Document 5 describes a technique using specular reflection of an eyeball as an attempt to eliminate an additional device. This technique uses binocular eyes that are regarded as spherical mirrors with respect to three reference points, that is, spherical mirrors with two postures. Non-Patent Document 6 describes a technique that uses an eyeball of one posture with respect to three reference points by limiting the positional relationship between the eyeball center, the camera center, and the reference point.

Z. Zhang, "A Flexible New Technique for Camera Calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, Nov. 2000, Volume 22, Issue 11, p.1330-1334 K. Takahashi, S. Nobuhara, and T. Matsuyama, "A New Mirror-based Extrinsic Camera Calibration Using an Orthogonality Constraint", 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2012, p.1051-1058 JAHesch, AIMourikis, and SIRoumeliotis, "chapter: Mirror-Based Extrinsic Camera Calibration", "Algorithmic Foundation of Robotics VIII", (Germany), Springer-Verlag, 2009, Volume 57 of Springer Tracts in Advanced Robotics, p. 285-299 A. Agrawal, "Extrinsic Camera Calibration Without A Direct View Using Spherical Mirror", 2013 IEEE International Conference on Computer Vision (ICCV), Dec. 2013, p.2368-2375 C. Nitschke, A. Nakazawa, and H. Takemura, "Display-camera calibration using eye reflections and geometry constraints", "Computer Vision and Image Understanding", Elsevier BV, June 2011, Volume 115, Issue 6, p.835- 853 Kosuke Takahashi, Amami Mikami, Akira Kojima, “Minimum configuration external calibration of camera and objects outside the field of view using eye characteristics and equidistant constraints”, IEICE Technical Report, IEICE, 20151 Month, Vol. 114, no. 410, p. 277-282

  However, in the techniques described in Non-Patent Documents 2, 3, and 4, there is a trouble that a mirror must be prepared for each calibration. In the technique described in Non-Patent Document 5, calibration is performed from a reference object reflected in a minimal region called an eyeball. It is desirable to do. However, since the technique described in Non-Patent Document 5 uses both eyes, it is necessary to separate the camera and the eyeball to such a distance that both eyes can be photographed simultaneously. In the technique described in Non-Patent Document 6, there is a restriction that the distance between the eyeball center and each reference point must be equal to the distance between the eyeball center and the camera center.

  In view of the above circumstances, the present invention is a data processing capable of performing external calibration between a camera and an object existing outside the field of view without imposing restrictions on the positional relationship between the eyeball center, the reference point, and the camera center. It is an object to provide a method, an image data processing apparatus, and an image data processing program.

  One embodiment of the present invention includes an image captured by an imaging device, internal parameters of the imaging device, a radius of a corneal sphere that considers the cornea as a part of a sphere, and three-dimensional coordinates of a plurality of reference points in a reference object And the positions of the imaging device and the reference object located outside the field of view of the imaging device based on the two-dimensional coordinates in the image of primary reflection images of the plurality of reference points by the cornea included in the image An image data processing method for estimating a posture relationship, the corneal sphere coordinate estimation step for estimating a center coordinate of the corneal sphere based on the image, the internal parameter, and the radius, the internal parameter, and the plurality of the plurality of parameters Based on the three-dimensional coordinates of the reference point, the two-dimensional coordinates of the plurality of reference points, and the central coordinates, the external parameters of the imaging device when the coordinate system of the reference object is used as a reference are estimated And a part parameter estimation step, an image data processing method.

  According to another aspect of the present invention, in the image data processing method, in the corneal sphere coordinate estimation step, an elliptic parameter for a projection image of the corneal sphere included in the image is estimated, and the estimated elliptic parameter Based on this, the center coordinates are estimated.

  According to another aspect of the present invention, in the image data processing method, in the external parameter estimation step, a first vector is calculated for each reference point from the two-dimensional coordinates of the reference point and the internal parameter. And calculating the three-dimensional coordinates of the reflection points on the corneal sphere of each of the reference points based on the center coordinates, the radius, and the first vector, and the three-dimensional coordinates of the plurality of reference points; By solving the simultaneous equations obtained based on the three-dimensional coordinates of the reflection point corresponding to each of the plurality of reference points, the first vector corresponding to each of the plurality of reference points, and the external parameter, The external parameter is estimated.

  According to another aspect of the present invention, in the image data processing method, in the external parameter estimation step, the simultaneous equations are obtained by using basis vector notation, and the external parameters are estimated linearly.

  In one embodiment of the present invention, in the image data processing method, there are at least six reference points, or at least five reference points when the reference points are located on the same plane.

  Further, according to one embodiment of the present invention, an image captured by an imaging device, an internal parameter of the imaging device, a radius of a corneal sphere that considers the cornea as a part of a sphere, and a tertiary of a plurality of reference points in a reference object Based on the original coordinates and two-dimensional coordinates in the image of primary reflection images of the plurality of reference points included in the image by the cornea, the reference object located outside the field of view of the imaging device, and the imaging device An image data processing device for estimating a position and orientation relationship of the corneal sphere, a corneal sphere coordinate estimation unit for estimating a central coordinate of the corneal sphere based on the image, the internal parameter, and the radius; the internal parameter; Based on the three-dimensional coordinates of a plurality of reference points, the two-dimensional coordinates of the plurality of reference points, and the center coordinates, an external parameter of the imaging device is estimated based on the coordinate system of the reference object. And a part parameter estimation unit, an image data processing apparatus.

  One embodiment of the present invention is an image data processing program for causing a computer to execute the above image data processing method.

  According to the present invention, it is possible to perform external calibration between a camera and an object existing outside the field of view without imposing restrictions on the positional relationship between the eyeball center, the reference point, and the camera center.

It is a figure which shows the outline | summary of the environment assumed in embodiment. It is a block diagram which shows the structure of the image data processing apparatus 1 in embodiment. It is a flowchart which shows the image data process which the image data processing apparatus 1 in embodiment performs. 3 is a diagram illustrating a projection of a corneal sphere E.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. First, an environment assumed by the image data processing method, the image data processing apparatus, and the image data processing program according to the embodiment of the present invention will be described. FIG. 1 is a diagram illustrating an outline of an environment assumed in the embodiment. In the present embodiment, a camera C, a reference object X, and an eyeball (corneal ball) E as an imaging device are used. A corneal sphere is a sphere when the cornea is regarded as a part of the sphere. On the reference object X, there are reference points p _i (i = 0, 1, 2,..., N _p −1). N _p is the number of reference points on the reference object X. The reference object X does not exist in the field of view of the camera C, and is observed from the camera C by the primary reflection of the corneal sphere E. Here, as shown in FIG. 1, the center of the eyeball is the center S of the corneal sphere E, and the camera center is the optical center O of the camera C used for photographing. In the present embodiment, as shown in FIG. 1, three coordinate systems of a camera coordinate system {C}, an image coordinate system {I}, and a reference object coordinate system {X} are used.

The purpose of the image data processing method, the image data processing apparatus, and the image data processing program in the present embodiment is that the coordinate value of the projection image q _i of the reference point p _i in the image coordinate system {I} It is to obtain external parameters R and T between the reference object coordinate system {X}. The external parameter R is a rotation matrix that represents a change in posture in a different coordinate system between the camera coordinate system {C} and the reference object coordinate system {X}. The external parameter T is a translation vector representing a position change in a different coordinate system between the camera coordinate system {C} and the reference object coordinate system {X}. Hereinafter, the coordinate value of the point a in the coordinate system ^{b} is represented as a ^{b} .

  FIG. 2 is a block diagram showing a configuration of the image data processing apparatus 1 in the present embodiment. As shown in FIG. 2, the image data processing device 1 includes an input unit 11, a corneal sphere coordinate estimation unit 12, and an external parameter estimation unit 13. The image data processing device 1 includes an image captured by a camera, camera internal parameters, a radius of a corneal sphere that considers the cornea as a part of a sphere, three-dimensional coordinates of a plurality of reference points in a reference object, and an image The position / orientation relationship between the reference object and the camera located outside the field of view of the camera is estimated based on the two-dimensional coordinates in the image of the primary reflection image of the plurality of reference points included in the cornea.

The input unit 11 inputs an image I, an internal parameter K, a corneal sphere radius r, a reference point two-dimensional coordinate q _i ^{I}, and a model coordinate p _i ^{X} from an external device or a host device. The image I is an image captured by the camera C to be processed and includes a primary reflection image of a reference point by a corneal sphere. The internal parameter K is an internal parameter of the camera C and is a parameter representing the focal length and the image center of the camera C. A value determined at the time of camera shipment or a value output from the camera C may be used as the internal parameter K, or a value estimated using the technique described in Non-Patent Document 1 is used as the internal parameter K. Also good. Note that the value output by the camera C is used only when the camera C has a measurement function. The internal parameter K is given by, for example, a 3 × 3 matrix. As the corneal sphere radius r, a value obtained by measuring a user's corneal sphere radius in advance may be used, or an average value of human corneal sphere radii (r = 5.6 mm) may be used. Good.

The reference point two-dimensional coordinates q _i ^{I} are coordinates in the image I when the reference point p _i is reflected by the corneal sphere E and projected onto the image I. The reference point two-dimensional coordinate q _i may be detected by human eyes. Alternatively, a prior image when the reference object X is not reflected and projected by the corneal sphere E is acquired in advance, and an image and the previous image when the reference object X is reflected and projected by the corneal sphere E are obtained. The reference point two-dimensional coordinates q _i ^{I} may be detected using the background difference method.

The model coordinates p _i ^{X} are three-dimensional coordinates of the reference point p _i in the reference object coordinate system {X}, and can be determined in advance. Note that an intersection of chess patterns or the like is generally used as a reference point for ease of detection, but various things can be used as a reference object as long as the model coordinates p _i ^{{X} are} known. For example, a three-dimensional object having a known shape or video content displayed on a display can be used as a reference object. Video content can be used as a reference object since the shape of an object included in the video and the display color of pixels in the video are known.

The corneal sphere coordinate estimation unit 12 estimates the center coordinate S ^{C} of the corneal sphere E based on the image I, the internal parameter K, and the corneal sphere radius r input by the input unit 11. The external parameter estimation unit 13 includes the internal parameter K, the model coordinates p _i ^{X} , the reference point two-dimensional coordinates q _i ^{I} input by the input unit 11, and the corneal sphere estimated by the corneal sphere coordinate estimation unit 12. The external parameters R and T are estimated based on the center coordinates S ^{C} .

FIG. 3 is a flowchart illustrating image data processing performed by the image data processing apparatus 1 according to the present embodiment. When the image data processing is started in the image data processing apparatus 1, the input unit 11 receives the image I, the internal parameter K, the corneal sphere radius r, the reference point two-dimensional coordinates q _i ^{I}, and the model coordinates p _i ^{X}. Is input (step S101).

The corneal sphere coordinate estimation unit 12 estimates the elliptic parameter of the corneal sphere E with respect to the image I input in step S101 (step S102). FIG. 4 is a diagram illustrating the projection of the corneal sphere E. As shown in the figure, the ellipse parameters of the corneal sphere E include the ellipse center coordinate i _L ^{I} (x coordinate: cx, y coordinate: cy), the ellipse major axis length 2r _max , and the ellipse minor axis. 5 variables including length 2r _min and slope φ. The estimation of the ellipse parameter is performed by the method described below. The estimation of the ellipse parameter may be performed by a person measuring the region of the corneal sphere E in the image I. Note that the distance d from the image I to the corneal sphere E in FIG. 4 is determined by the focal length f from the camera C to the image I and the elliptic parameter.

  For example, the corneal sphere coordinate estimation unit 12 performs edge detection after binarizing the image I, and estimates an elliptic parameter of the corneal sphere E with respect to the detected edge. A known technique can be used for the binarization of the image I. For example, fixed threshold processing or adaptive threshold processing can be used. A known technique can also be used for edge detection. For example, a Canny filter can be used. Furthermore, an arbitrary method can be used for estimating the ellipse parameter at the detected edge. For example, reference 1 (AWFitzgibbon, RBFisher, "A Buyer's Guide to Conic Fitting", BMVC '95 Proceedings of the 6th British conference on Machine vision (Vol.2), (UK), BMVA Press Surrey, 1995, p. .513-522) can be used.

The corneal sphere coordinate estimation unit 12 estimates the center coordinates S ^{C} of the corneal sphere E from the estimated ellipse parameters (step S103). When the angle τ formed by the average depth plane located at the center L of the corneal ring portion and the plane including the corneal ring portion is obtained by Expression (1), the line-of-sight direction g is expressed by Expression (2).

Further, when the distance between the camera center and the center L of the cornea ring portion is d, d is expressed as d = f · (r _L / r _max ) using the focal length f (pixel value). Incidentally, _{r L} denotes the radius of the limbus, the average value of _{r L} is the 5.6 mm. Further, when the internal parameter K of the camera is given by a matrix, the coordinates L ^{C} of the center L of the corneal limbus are expressed by Expression (3).

The central coordinate S ^{C} of the corneal sphere is obtained by Expression (4) because it is in the direction advanced by d _LS (= 5.6 mm) in the −g direction from the center L of the corneal limbus.

The external parameter estimation unit 13 calculates a coefficient a _j ^{i {C}} in base vector notation (step S104). In the present embodiment, the basis vector notation means that a basis vector spanning a certain _Ne- dimensional space is e _j (j = 0, 1, 2,..., N _e −1). It shall be expressed by a polynomial, that is, expressed by equation (5). Note that a _j is a coefficient for each basis vector, and each basis vector e _j is linearly independent.

Here, using the basis vector notation, the coordinates of the reference point p _i are expressed by the equations (6) and (7) in the reference object coordinate system {X} and the camera coordinate system {C}.

Here, when Expression (8) is satisfied, the coordinates p _i ^{C} of the reference point p _{i in} the camera coordinate system {C} are expressed by Expression (9).

In addition, the matrix R in Formula (9) is the external parameter R and is an orthogonal matrix. Since the transformation by the external parameter R is a linear transformation having an equal length, it can be said that a _j ^{i {X}} = a _j ^{i {C}} . Since p _j ^{X} is known, the coefficient a _j ^{i {X}} (and a _j ^{i {C}} ) can be obtained by appropriately determining e _j ^{X} . The coefficient a _j ^{i {X}} may be obtained in any way. For example, when e _i ^{X} (i = 0, 1, 2) is defined as in equation (10) and p _i ^{X} is defined as in equation (11), each coefficient a _j ^{i {X}} (J = 0, 1, 2) is given by equation (12). As a result, each a _j ^{i {C}} is obtained.

The external parameter estimation unit 13 calculates a reflection point on the corneal sphere E for each reference point p _i (i = 0, 1, 2,..., N _p −1) (step S105). The process of step S105 is performed for each reference point p _i . That is, the external parameter estimating unit 13 repeats the processing in step S105 the _{N p} times. As described above, the projection point in the image I of the reference point p _i is defined as q _i . Here, vector v _i is defined as equation (13) using reference point two-dimensional coordinates q _i ^{I} and external parameter K.

The reference point _{p i,} when was reflected at the reflection point _{m i} on the cornea sphere E, the distance _{k Omi} from the camera center O to the reflection point _{m i} is represented by the formula (14).

Thus, the coordinate _m ⁱ of the reflection points _{m i} ^{C} is calculated by the equation (15).

The external parameter estimation unit 13 solves the linear equation related to the external parameter and acquires the external parameters R and T (step S106). In the basis vector notation, when e _j ^{X} (j = 0, 1, 2) is defined as in equation (10) and the external parameter R is defined as equation (16), it is expressed as equation (17). The coordinates p _i ^{C} of the point p _i are expressed by equation (18).

On the other hand, the coordinates _p i of the reference point _{p i} {C}, using the distance _{k i} between the unit vectors _{u i} and _m i -p _i connecting the reference point _{p i} and the reflection points _{m i,} the formula ( 19).

The unit vector u _i is expressed by the equation (20) from the law of reflection.

Here, _{n i} is the normal vector at reflection point _{m i.} The normal vector n _i at the reflection point m _i, since a unit vector connecting the center of the spherical mirror and reflection points m _i, is calculated by equation (21).

Here, Expression (22) is obtained from Expression (18) and Expression (19).

Furthermore, Equation (22) is prepared for N _p points and summarized as Equation (23).

However, the matrices A, X, and B in the equation (23) are given by the following equations (24-1) to (24-4), (25), and (26).

When N _p points are used as reference points p _i , the number of unknown variables in equation (23) is (12 + N _p ), and 3N _p constraint equations are obtained. Therefore, there are 6 or more reference points p _i. When (N _p ≧ 6), unknown variables including the external parameters R and T are obtained by Expression (27). That is, an unknown variable including external parameters R and T can be obtained by solving simultaneous equations composed of constraint equations obtained from six or more reference points p _i . Note that the matrix A ^* in Equation (27) is a pseudo inverse matrix of the matrix A.

Moreover, it is also possible to solve as follows by using the basis vector notation. When p ₀ ^X is the origin of the reference object coordinate system {X}, since p ₀ ^{C} can be regarded as a translation vector itself, Expression (28) is established.

Formula (28) is prepared for N _p points, and summarized as Formula (29).

However, the matrices C, Y, and D in Expression (29) are given by Expression (30-1) to Expression (30-3), Expression (31), and Expression (32).

When using a reference point _{p i N p} points, the number of unknown variables in equation (29) (9 + _{N p)} a number, from the constraint equation _{3 (N} p -1) by the obtained reference point _{p i} is When 6 points or more (N _p ≧ 6), unknown variables including the external parameters R and T are obtained by Expression (33).

In Expression (33), the matrix C ^* is a pseudo inverse matrix of the matrix C. At this time, the external parameter T is obtained by Expression (34).

Further, since the vector r _j is a component of the rotation matrix R, the norm is 1. However, in practice, it is not always 1 due to the influence of noise, error, and the like. Therefore, r ′ _j obtained by Expression (35) may be a component of the rotation matrix R estimated. The norm need not be corrected.

Further, when the reference point exists on the same plane, that is, when the vector p ₂ ^X is a zero vector, since the vector r ₂ = (000) ^T , the relationship of Expression (36) is established.

However, in Expression (36), the matrices A ′ and X ′ are given by Expression (37) and Expression (38).

When N _p points are used as the reference points p _i , the number of unknown variables in the equation (36) is (9 + N _p ), and 3N _p constraint equations are obtained. Therefore, the reference points p _i are 5 or more (N _{When p} ≧ 5), unknown variables including the external parameters R and T are obtained by Expression (39).

In Expression (39), the matrix A ′ ^* is a pseudo inverse matrix of the matrix A ′. Further, since the vector r ₂ is orthogonal to the vectors r ₀ and r ₁ , it is obtained as Expression (40). As described above, each vector r _j may or may not be corrected for the norm.

Moreover, it is also possible to solve by Formula (41) by using a basis vector table.

However, in Expression (41), the matrices C ′, Y ′, and D ′ are given by Expression (42) and Expression (43).

When using a reference point _{p i N p} points, the number of unknown variables in equation (41) (6 + _{N p)} a number, from the constraint equation _{3 (N} p -1) by the obtained reference point _{p i} is When 5 points or more (N _p ≧ 5), unknown variables including the external parameters R and T are obtained by Expression (44).

In Expression (44), the matrix C ′ ^* is a pseudo inverse matrix of the matrix C ′. At this time, the external parameter T is obtained by Expression (45).

Further, since the vector r ₂ is orthogonal to the vectors r ₀ and r ₁ , it is obtained as Expression (46). As described above, each vector r _j may or may not be corrected for the norm.

The external parameter estimation unit 13 corrects the obtained rotation matrix R (step S107). The vector r _j (j = 0, 1, 2) obtained in step S106 does not always satisfy the constraints (Equation (47) and Equation (48)) that the rotation matrix should satisfy due to the influence of noise and the like. .

Therefore, by using the Orthogonal Procrustes Problem for the rotation matrix R = (r ₀ r ₁ r ₂ ), it is possible to obtain a rotation matrix R ′ that satisfies Expressions (47) and (48). The Orthogonal Procrustes Problem is a known technique, such as Reference 2 (Gene H. Golub and Charles F. Van Loan, "Matrix Computations", (USA), The Johns Hopkins University Press, Third Edition, 1996). It is described in. If the influence of noise or the like can be ignored, the correction in step S107 may be omitted.

  The external parameter estimation unit 13 optimizes the obtained external parameters R and T (step S108), outputs the optimized external parameters R and T to an external device, and ends the image data processing. Any cost function may be used as the cost function for optimization. For example, a term that minimizes a reprojection error that is generally used in optimizing external parameters may be used. At this time, the external parameters R and T are optimized so that the projection points calculated using the various parameters coincide with the actually detected projection points. It can be expected to be optimized with high accuracy in a rough environment. If the influence of noise can be ignored, the external parameters R and T need not be optimized.

According to the image data processing device 1 of the present embodiment, the corneal sphere coordinate estimation unit 12 estimates the center coordinate S ^{C} of the corneal sphere E based on the image I, the internal parameter K, and the corneal sphere radius r, and the external parameter estimating unit 13 based on the internal parameter K and the three-dimensional coordinates _p ^{i {X}} of the reference point _{p i} reference point two-dimensional coordinates _q ^{i {I}} the center coordinates and the ^{S {C},} the reference object coordinate system { The external parameters R and T of the camera C when X} is used as a reference are estimated. Thus, a simple configuration in which one eyeball (corneal sphere) is used for five reference points distributed on the same plane or six reference points not distributed on the same plane is used. External calibration for estimating external parameters of the camera can be realized without imposing restrictions on the positional relationship between the point, eyeball position, and camera center.

  In the image data processing shown in FIG. 3, the configuration in which step S105 is performed after step S104 has been described. However, step S104 may be performed after step S105.

  You may make it implement | achieve all or one part of the image data processing apparatus 1 in embodiment mentioned above with a computer. For example, it is realized by recording a program for realizing each component included in the image data processing apparatus 1 on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. May be. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the above-described components, or may be realized by combining the above-described components with a program already recorded in the computer system. It may be realized by using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention can also be applied to applications where it is indispensable to perform external calibration between a camera and an object existing outside the field of view without imposing restrictions on the positional relationship between the eyeball center, the reference point, and the camera center.

DESCRIPTION OF SYMBOLS 1 ... Image data processing apparatus 11 ... Input part 12 ... Corneal sphere coordinate estimation part 13 ... External parameter estimation part

Claims (7)

Included in the image are the image captured by the imaging device, the internal parameters of the imaging device, the radius of the corneal sphere that considered the cornea as part of the sphere, the three-dimensional coordinates of a plurality of reference points in the reference object Image data for estimating a position and orientation relationship between the reference object and the imaging device located outside the field of view of the imaging device based on two-dimensional coordinates in the image of primary reflection images of the plurality of reference points by the cornea A processing method,
A corneal sphere coordinate estimation step for estimating a center coordinate of the corneal sphere based on the image, the internal parameter, and the radius;
Based on the internal parameters, the three-dimensional coordinates of the plurality of reference points, the two-dimensional coordinates of the plurality of reference points, and the center coordinates, the external parameters of the imaging apparatus when the coordinate system of the reference object is used as a reference An external parameter estimation step for estimating
An image data processing method comprising: In the corneal sphere coordinate estimation step,
Estimating an ellipse parameter for the projected image of the corneal sphere contained in the image;
Estimating the central coordinates based on the estimated ellipse parameters;
The image data processing method according to claim 1. In the external parameter estimation step,
For each reference point, calculate a first vector from the two-dimensional coordinates of the reference point and the internal parameters;
Based on the central coordinates, the radius, and the first vector, calculate the three-dimensional coordinates of the reflection point on the corneal sphere of each of the reference points,
The three-dimensional coordinates of the plurality of reference points, the three-dimensional coordinates of the reflection points corresponding to the plurality of reference points, the first vector corresponding to each of the plurality of reference points, and the external parameter Estimating the external parameters by solving simultaneous equations obtained from
The image data processing method according to claim 1. In the external parameter estimation step,
Obtaining the simultaneous equations by using basis vector notation to estimate the external parameters linearly;
The image data processing method according to claim 3. The reference points exist at least 6 points, or at least 5 points when the reference points are in the same plane,
The image data processing method according to any one of claims 1 to 4. Included in the image are the image captured by the imaging device, the internal parameters of the imaging device, the radius of the corneal sphere that considered the cornea as part of the sphere, the three-dimensional coordinates of a plurality of reference points in the reference object Image data for estimating a position and orientation relationship between the reference object and the imaging device located outside the field of view of the imaging device based on two-dimensional coordinates in the image of primary reflection images of the plurality of reference points by the cornea A processing device comprising:
A corneal sphere coordinate estimator for estimating a center coordinate of the corneal sphere based on the image, the internal parameter, and the radius;
Based on the internal parameters, the three-dimensional coordinates of the plurality of reference points, the two-dimensional coordinates of the plurality of reference points, and the center coordinates, the external parameters of the imaging apparatus when the coordinate system of the reference object is used as a reference An external parameter estimation unit for estimating
An image data processing apparatus comprising:   An image data processing program for causing a computer to execute the data processing method according to any one of claims 1 to 5.

Images