JP6276713B2 - Image data processing method, image data processing apparatus, and image data processing program - Google Patents

Description

  The present invention relates to an image data processing method, an image data processing apparatus, and an image data processing program that estimate the position and orientation relationship between a camera and a subject (execute external calibration).

  External calibration for estimating the position and orientation relationship between a camera and a subject is a problem that is the basis of computer vision, and many studies have been made so far (for example, see Non-Patent Document 1). Many of these existing external calibration methods assume that the camera can directly observe the reference object.

  In recent years, with the spread of laptop computers, smartphones, digital signage, and the like, various studies using display-camera systems have attracted attention. One example is interest estimation using digital signage. This is realized by taking a correspondence between the user's line-of-sight direction and the content presented on the display. Furthermore, application to three-dimensional shape restoration is also being studied. This is based on a change in luminance of an observation target accompanying a change in the presentation contents, using the display as a controllable surface light source.

  In this way, external calibration is performed to determine the positional orientation between the display and the camera in order to geometrically associate the gaze estimated from the video captured by the camera and the luminance change of the observation target with the display content. There is a need. Usually, a chess pattern or the like is presented on the display, and this is photographed with a camera to perform external calibration.

  However, in a display-camera system such as a laptop computer or a smartphone, there is generally no display as a reference object in the field of view of the camera, and it is difficult to apply a conventional external calibration method.

  There has been proposed a method of performing external calibration by using a mirror in a situation where such a reference object does not exist in the field of view of the camera. In the following description, a reference point exists on the reference object, and external calibration is performed using the reference point. In external calibration, research aimed at a simpler configuration is underway from the viewpoint of labor and calculation amount. Takahashi et al. (For example, see Non-Patent Document 2) and Hesch et al. (For example, Non-Patent Document 3) and others propose a method of using three orientations of a plane mirror with respect to three reference points, which is the minimum configuration using a plane mirror. did.

  On the other hand, from the viewpoint of reducing the number of mirror postures, Agrawal (for example, see Non-Patent Document 4) has proposed a method of using a spherical mirror with one posture for eight reference points. Furthermore, Nitschke et al. Used specular reflection of the eyeball as an attempt to eliminate an additional device (see Non-Patent Document 5, for example). This uses binoculars regarded as spherical mirrors with respect to three reference points, that is, spherical mirrors with two postures.

Z. Zhang.A flexible new technique for camera calibration.IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000. K. Takahashi, S. Nobuhara, and T. Matsuyama.A new mirror-based extrinsic camera calibration using an orthogonality constraint.In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 1051-1058, June 2012. J. A. Hesch, A. I. Mourikis, and S. I. Roumeliotis. Algorithmic Foundation of Robotics VIII, volume 57 of Springer Tracts in Advanced Robotics, chapter Mirror-Based Extrinsic Camera Calibration, pages 285-299.Springer-Verlag, Berlin, Germany, Dec. 2009. A. Agrawal.Extrinsic camera calibration without a direct view using spherical mirror.In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 2368-2375, Dec 2013. C. Nitschke, A. Nakazawa, and H. Takemura.Display-camera calibration using eye refections and geometry constraints.Computer Vision and Image Understanding, 115 (6): 835-853, 2011.

  However, in Non-Patent Document 2, Non-Patent Document 3, and Non-Patent Document 4, the trouble that a mirror must be prepared each time calibration is performed occurs. Further, in Non-Patent Document 5 using an eyeball, since calibration is performed from a reference object reflected in a minimal region called an eyeball, it is desirable to photograph with a close distance between the camera and the eyeball from the viewpoint of accuracy. However, since Non-Patent Document 5 uses both eyes, it is necessary to keep both eyes away from the camera to a distance that allows both eyes to photograph simultaneously.

  The present invention has been made in view of such circumstances, and an image data processing method and image data processing capable of performing external calibration for estimating the position and orientation relationship between a camera and an object as a subject with a simple configuration. An object is to provide an apparatus and an image data processing program.

  The present invention provides an image captured by a camera, an internal parameter of the camera, a radius of a corneal sphere in which the cornea is regarded as a part of a sphere, a coordinate value of a predetermined reference point in the image, and the reference point An image data processing method for performing external calibration for estimating a position and orientation relationship between the camera and a reference object existing outside the field of view using a model coordinate system, wherein the cornea estimates center coordinates of the corneal sphere A spherical coordinate estimation step, a reference point 3D coordinate estimation step for estimating a three-dimensional coordinate value of the reference point in the camera coordinate system, and an external parameter estimation step for estimating an external parameter between the camera and the reference object. It is characterized by.

  The present invention is characterized in that, in the corneal sphere coordinate estimation step, center coordinates of the corneal sphere are estimated by estimating an elliptic parameter of a projection image of the corneal sphere with respect to the image.

  The present invention is characterized in that, in the reference point 3D coordinate estimation step, the three-dimensional coordinate value is estimated on the assumption that a positional relationship between the corneal sphere, the reference object, and the camera coordinate system satisfies a predetermined condition.

  The present invention provides an image captured by a camera, an internal parameter of the camera, a radius of a corneal sphere in which the cornea is regarded as a part of a sphere, a coordinate value of a predetermined reference point in the image, and the reference point An image data processing apparatus that performs external calibration using a model coordinate system to estimate a position and orientation relationship between the camera and a reference object that is outside the field of view, the cornea that estimates the center coordinates of the corneal sphere A spherical coordinate estimation unit; a reference point 3D coordinate estimation unit that estimates a three-dimensional coordinate value of the reference point in a camera coordinate system; and an external parameter estimation unit that estimates an external parameter between the camera and the reference object. It is characterized by.

  The present invention is an image data processing program for causing a computer to execute the image data processing method.

  According to the present invention, it is possible to perform an external calibration between a camera and an object that is a subject existing outside the field of view with a simple configuration.

It is a figure showing the environment which this invention assumes. It is a block diagram showing the structure of the image data processing apparatus by one Embodiment of this invention. It is a flowchart which shows the processing operation of the image data processing apparatus shown in FIG. 1 when a corneal sphere is used as a mirror ball. It is a figure showing projection of a corneal sphere.

  Hereinafter, an image data processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. First, an environment assumed by the present invention is shown in FIG. FIG. 1 is a diagram illustrating an environment model assumed in the present embodiment. In the present invention, one camera, one reference object, and one eyeball (corneal ball) are used. There is a reference point on the reference object. The reference object does not exist in the field of view of the camera, and is observed from the camera by the primary reflection of the corneal sphere. In the present invention, the corneal sphere is installed at a position that satisfies the following conditions.

  The condition of the corneal sphere is that the distance between the corneal sphere center and the camera center is equal to the distance between the corneal sphere center and each reference point. The corneal sphere center is the center S of the eyeball of FIG. 1 (cornea sphere: a sphere whose surface is the corneal surface), and the camera center is the optical center O of the camera used for photographing. Note that it is not necessary to strictly satisfy (condition) if an error of some accuracy is allowed. This condition can be easily satisfied when the display and the camera are on the same plane, such as when a web camera is installed on the top of a smartphone or display. Specifically, as shown in FIG. 1, the reference points are displayed so that each reference point and the camera center exist on a certain circle, and the eyeball center is set on a straight line l that vertically passes through the circle center. do it.

  In the present embodiment, as shown in FIG. 1, three coordinate systems (camera coordinate system {C}, image coordinate system {I} of an image captured by the camera, and model coordinate system {X}) are used. The purpose of this embodiment is to obtain external parameters R and T between the camera coordinate system {C} and the model coordinate system {X} from the coordinate values of the projected image of the reference point in the image coordinate system {I}. The parameter R is a rotation matrix that represents an attitude change in a different coordinate system, and the parameter R is a translation vector that represents a position change in a different coordinate system. In the following, “coordinate {a}” represents a coordinate value in the coordinate system {a}.

  Next, the configuration of the image data processing apparatus will be described. FIG. 2 is a block diagram showing a detailed configuration of an image data processing apparatus constituted by a computer apparatus or the like. The image data processing apparatus 1 includes an input unit 2, a corneal sphere center coordinate estimation unit 3, a reference point 3D coordinate estimation unit 4, and an external parameter estimation unit 5.

  The input unit 2 inputs an image, an internal parameter, a corneal sphere radius, a reference point 2D coordinate {I}, and a model coordinate system {X} from the outside. In response to this, the input unit 2 inputs the image to be processed, the camera internal parameters, the corneal sphere radius, the reference point 2D coordinates, and the model coordinates, and the image, the internal parameters, the corneal sphere radius, the reference point 2D coordinates { I} and model coordinates {X} are output.

  The corneal sphere coordinate estimation unit 3 inputs an image, internal parameters, and corneal sphere radius. In response to this, the corneal sphere coordinate estimation unit 3 estimates the corneal sphere center coordinates using the projected image of the corneal sphere, the internal parameters of the camera, and the corneal sphere radius. Then, the corneal sphere coordinate estimation unit 3 outputs the corneal sphere center coordinates {C}.

  The reference point 3D coordinate estimation unit 4 inputs corneal sphere center coordinates {C}, reference point 2D coordinates {I}, and internal parameters. In response to this, the reference point 3D coordinate estimation unit 4 estimates the reference point 3D coordinates using the reference point 2D coordinates {I} and the corneal sphere center coordinates {C}. Then, the reference point 3D coordinate estimation unit 4 outputs the reference point 3D coordinate {C}.

  The external parameter estimation unit 5 inputs model coordinates {X} and reference point 3D coordinates {C}. In response to this, the external parameter estimation unit 5 estimates the external parameter using the model coordinates and the reference point 3D coordinates. Then, external parameters (R, T) are output.

  Next, the processing operation of the image data processing apparatus 1 shown in FIG. 2 will be described with reference to FIG. FIG. 3 is a flowchart showing the processing operation of the image data processing apparatus shown in FIG. First, the input unit 2 inputs an image to be processed, camera internal parameters, corneal sphere radius, reference point 2D coordinates, and model coordinates (step S101).

  It is assumed that the image to be processed includes a primary reflection image of a reference point by a corneal sphere. The internal parameters of the camera are parameters representing the focal length and the image center of the camera, and are estimated in advance using a known technique described in Non-Patent Document 1 or the like. Regarding the corneal sphere radius, an average value of the human corneal sphere radius may be used.

  The reference point 2D coordinate is a coordinate when the image of the reference point reflected by the corneal sphere is projected on the image plane. This point may be detected visually. Alternatively, a situation in which the reference point is not displayed in advance may be taken, and then the reference point may be displayed with a time difference and automatically detected using the background difference method. The model coordinates represent the three-dimensional coordinates of the reference point in the model coordinate system, which is determined in advance.

  Next, the corneal sphere coordinate estimation unit 3 estimates an elliptic parameter of the corneal sphere with respect to the input image (step S102). The ellipse parameter refers to five variables: the center coordinates (x coordinate, y coordinate) {I} of the ellipse (including a circle), the major axis and minor axis of the ellipse, and the inclination thereof. The method for estimating the ellipse parameter may be directly measured by hand or automatically estimated. In the following description, the case of automatic estimation will be described.

Any method may be used for automatically estimating the ellipse parameter. For example, it is possible to apply a method of performing edge detection after binarizing an image and estimating an ellipse parameter of an ellipse that is an image of the cornea with respect to the detected edge. At this time, any method may be used for binarization. For example, it is possible to use fixed position processing or adaptive threshold processing. Further, any method for performing edge detection may be used. For example, a Canny filter can be used. Furthermore, any method may be used for estimating the ellipse parameter at the detected edge. For example, a known technique described in Reference 1 can be used.
Reference 1: Andrew W. Fitzgibbon, RBFisher. A Buyer's Guide to Conic Fitting. Proc. 5th British Machine Vision Conference, Birmingham, pp. 513-522, 1995.

Next, the corneal sphere coordinate estimation unit 3 estimates the center coordinates of the corneal sphere from the ellipse parameters (step S103). FIG. 4 is a diagram showing projection of a corneal sphere. As shown in FIG. 4, the ellipse parameters are the ellipse major axis length: r _max , the minor axis length: r _min , the ellipse inclination Φ, and the ellipse center coordinate i _L (x coordinate, y coordinate). Is a variable. In addition, τ = ± arccos (r _min / r _max ). At this time, the line-of-sight direction is expressed as g = [sin τ sin φ−sin τ cos φ cos τ] ^T.

Further, when the distance between the camera center on the image plane and the center L of the corneal ring portion is d, d is expressed as d = f · r _L / r _max using the focal length f (pixel value). R _L represents the radius of the corneal limbus, and the average of human r _L is 5.6 mm. Furthermore, if the matrix representing the internal parameters of the camera is K, the center L of the corneal limbus is represented as L {C} = dK ⁻¹ i _L {I}. Since the corneal sphere center S is in a direction d _LS (= 5.6 mm) advanced in the −g direction from the center L of the corneal limbus, it is calculated by S {C} = L {C} −d _LS {C} g. Can do.

Next, the process is repeated for the three reference points p _i (i = 1, 2, 3) shown in FIG. 1 (step S104). The reference point 3D coordinate estimation unit 4 estimates the 3D coordinates of the reference point p _i (step S105). Let q _{i be} the projection point of the reference point p _{i on} the image plane. The vector ^{_{v i v i {C} =}} (K -1 q i {I} / |.. And ^K -1 _q i {I} At this time, _{q i} is the reference point 2D coordinates The reference point p _i when was reflected at m _i on the cornea sphere, the distance from the camera center to m _i is represented by the following values.
k _0mi = (v _i ^T {C} S {C) −√ ((v _i ^T {C} S {C}) ² − | S {C} | ² −r ² ))) / | v _i {C } ｜ ²

In this case, the reflection point _{m i} is determined by _{m i {C} = k 0mi} V i {C}. Further, since the normal _{n i} at the reflection point _{m i} is a unit vector connecting the reflection point with the center of the spherical _{mirror, n i {C} = (} m i {C} -S {C}) / | m i {C} - S {C} | Further, _{u i} the unit vector connecting the reference point _{p i} and the reflection points _{m i} is the law of reflection _{u i {C} = v i} {C} +2 (-v i T {C} · n i {C} ) N _i {C}. When the distance between the reference point _{p i} and the reflection points _{m i} and _{k MIPI,} a _{k mipi = k 0mi} when satisfying (Condition). From the above, the reference point p _i is obtained by the following equation.
p _i {C} = k _mipi u _i {C} + m _i (C}
The above processing is performed for all reference points (step S106).

Next, the external parameter estimating unit 5 calculates the external parameter using the 3D coordinates of the obtained reference point _{p i} (R, T) (step S107). Any method for obtaining the external parameters (R, T) may be used. The problem of obtaining an external parameter between coordinate systems using the correspondence of points between two different coordinate systems is known as the Absolute Orientation Problem, and it is possible to obtain an external parameter by solving this problem. In the present embodiment, the model coordinates and (step S104) given as input ~ By using the 3D coordinates of the reference point p _i in the camera coordinate system obtained in (step S106), the model coordinate system and between the camera coordinate system External parameters are determined. Since the method of solving the Absolute Orientation Problem is a known method described in Reference Document 2, detailed description thereof is omitted here.
Reference 2: D. Eggert, A. Lorusso, and R. Fisher. Estimating 3-d rigid body transformations: a comparison of four major algorithms. Machine Vision and Applications, 9 (5-6): 272 {290, 1997.

  Next, the external parameter estimation unit 5 optimizes the obtained external parameters (R, T) (step S108). Any cost function may be used as the cost function for optimization. For example, a term that minimizes a reprojection error that is generally used in optimizing external parameters may be used. At this time, the external parameters (R, T) are optimized so that the projection points calculated using the various parameters coincide with the actually detected projection points. It can be expected to be optimized with high accuracy in a rough environment.

  Alternatively, terms such that the distance between the corneal sphere center and the camera center and the distance between the corneal sphere center and each reference point may be the same so as to satisfy the above-described conditions. In an environment where noise exists, the position of the reference point that can be calculated using the estimated external parameter does not always satisfy (condition). Since this optimization term is optimized to satisfy these physical constraints, it is stable and highly accurate without optimizing for an extremely inaccurate local solution even in the presence of noise at the projection point. It can be expected to optimize. Both of these terms may be used. If there is no need for optimization, optimization does not necessarily have to be performed.

  As described above, when camera parameters are estimated (external calibration), the introduction of a geometrical constraint that the camera and the reference point are at the same distance from the center of the spherical mirror and the physical model of the eyeball are compared with the conventional method. Since the number of spherical mirror postures and the number of reference points required for calculating external parameters can be reduced, a simple configuration in which one eyeball (corneal ball) is used for three reference points. From the obtained image, external calibration between the camera and an object existing outside the field of view can be performed.

  The image data processing apparatus in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can be applied to applications where it is indispensable to perform external calibration between a camera and an object existing outside the field of view with a simple configuration.

  DESCRIPTION OF SYMBOLS 1 ... Image data processing apparatus, 2 ... Input part, 3 ... Corneal sphere coordinate estimation part, 4 ... Reference point 3D coordinate estimation part, 5 ... External parameter estimation part

Claims (5)

An image captured by a camera, an internal parameter of the camera, a radius of a corneal sphere in which the cornea is regarded as a part of a sphere, a coordinate value of a predetermined reference point in the image, and a model coordinate system of the reference point Using an image data processing method for performing external calibration to estimate a position and orientation relationship between the camera and a reference object existing outside the field of view;
A corneal sphere coordinate estimation step for estimating a center coordinate of the corneal sphere;
A reference point 3D coordinate estimation step for estimating a three-dimensional coordinate value of the reference point in the camera coordinate system;
An external data estimation step for estimating an external parameter between the camera and the reference object.   2. The image data processing according to claim 1, wherein in the corneal sphere coordinate estimation step, center coordinates of the corneal sphere are estimated by estimating an elliptic parameter of a projection image of the corneal sphere with respect to the image. Method.   3. The reference point 3D coordinate estimation step estimates the three-dimensional coordinate value assuming that a positional relationship among the corneal sphere, the reference object, and the camera coordinate system satisfies a predetermined condition. 2. An image data processing method described in 1. An image captured by a camera, an internal parameter of the camera, a radius of a corneal sphere in which the cornea is regarded as a part of a sphere, a coordinate value of a predetermined reference point in the image, and a model coordinate system of the reference point An image data processing apparatus that performs external calibration to estimate a position and orientation relationship between the camera and a reference object existing outside the field of view,
A corneal sphere coordinate estimation unit for estimating the center coordinates of the corneal sphere,
A reference point 3D coordinate estimation unit for estimating a three-dimensional coordinate value of the reference point in the camera coordinate system;
An image data processing apparatus comprising: an external parameter estimation unit that estimates an external parameter between the camera and the reference object.   An image data processing program for causing a computer to execute the image data processing method according to any one of claims 1 to 3.

Images