JP2011070579A - Captured image display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically form and display a two-dimensional image observed from an observation direction, regarding an object whose observation direction is defined. <P>SOLUTION: The same object is captured by a camera from imaging points Q1, Q2 to obtain captured images P1, P2. The image P1 is analyzed to recognize a reference plane R of interest on the object, and points S11-S14 corresponding to four points S31-S34 on the reference plane R are obtained on the image P1. Features of the images P1, P2 are compared to obtain points S21-S24 corresponding to the points S11-S14 on the image P2. Three-dimensional coordinate values of the points S31-S34 on the reference plane R are obtained from eight points S11-S24 on the images P1, P2 and information on the location and direction of the camera. Based on the coordinate values of the points S11-S24 and points S31-S34, a transformation formula for transforming the images P1, P2 on an xy coordinate system into images on a uv coordinate system is derived, and the transformation formula is used to obtain transformed images of the images P1, P2 on the uv coordinate system. These transformed images are composed to obtain a display image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a photographed image display device, and in particular, generates and displays an image obtained by observing a subject from a predetermined virtual direction based on photographed images of subjects photographed from different directions using a plurality of cameras. The present invention relates to a captured image display device.

  If a plurality of cameras are used to photograph the same subject from different directions, different two-dimensional captured images can be obtained. As described above, in the three-dimensional space, the mutual relationship between the captured image obtained when the same object is observed from different directions and the object is analyzed by geometry called epipolar geometry. By using this epipolar geometry, it is possible to restore a three-dimensional subject image based on a plurality of two-dimensional captured images, and to create a two-dimensional image in which the subject is observed from an arbitrary virtual viewpoint. Is possible.

  For example, in Patent Document 1 below, a plurality of cameras are arranged at predetermined positions to obtain a plurality of photographed images from which blind spots with respect to a subject are eliminated, and based on these photographed images, a complete image of the subject is obtained. A technique for restoring a virtual three-dimensional image is disclosed. By using this technique, it is possible to create a two-dimensional image in which a subject is observed from an arbitrary viewpoint according to a user's request. Non-Patent Document 1 below proposes a technique for creating a two-dimensional image obtained by observing a person from an arbitrary direction based on two face photographs taken of the same person from two directions.

Japanese Patent No. 3300334

Yasuhiro Mukai et al. “Generating Facial Images in Arbitrary Direction Using Two Face Photos”: IPSJ Journal Vol. 37, No. 4 pp. 635-644, Apr. 1996

  As described above, if a two-dimensional photographed image taken from a plurality of different directions can be obtained for the same subject, a three-dimensional image of the subject can be restored based on these photographed images. It is possible to create and display a two-dimensional image of a subject observed from an arbitrary virtual viewpoint designated by. As described above, the conventional technique can display a two-dimensional image observed from the position of an arbitrary virtual viewpoint desired by the user. Therefore, there is an advantage that an image display with a very high degree of freedom is possible. However, since it is necessary for the user himself (or some external device) to specify the position of the virtual viewpoint, an operation burden for specifying the observation direction is imposed even in the case of a subject whose appropriate observation direction is predetermined. There is a problem that convenience is impaired.

  For example, when a license plate portion of a vehicle is a subject, in a general application of “recognizing characters on the license plate”, “the direction in which the license plate is observed from the front” is an appropriate observation direction. In a system that monitors the intrusion of a suspicious person, “the direction in which the face of the person who is the subject is observed from the front” is the appropriate observation direction.

  As described above, even when an appropriate observation direction for a specific subject is determined in advance according to the application, the conventionally proposed technique instructs the observation direction by the user himself or some external device. Since it is necessary, a complicated designation operation is required.

  For example, suppose you want to display a two-dimensional image of a vehicle license plate viewed from the front. In this case, in an environment where the orientation of the subject vehicle is fixed, such as a road or a parking lot gate, it is sufficient to perform photographing from almost the front of the vehicle with one camera. However, in places such as plazas where the direction of the vehicle is not fixed, after shooting with multiple cameras, it is necessary for the user to perform an operation to specify the “direction in which the license plate can be observed from the front” using the conventionally proposed technology. There is. Further, in the case of a suspicious person monitoring system, the user needs to perform an operation of designating “a direction in which a person can be observed from the front”.

  Therefore, the present invention can automatically generate a two-dimensional image showing a state observed from an appropriate observation direction for a subject whose appropriate observation direction is determined in advance according to the application, and display the image. An object is to provide an image display device.

(1) According to a first aspect of the present invention, in a captured image display device that generates and displays an image obtained by observing a subject from a predetermined virtual direction based on a captured image of the subject captured from different directions,
A first camera and a second camera installed at different positions and oriented in a direction in which the same subject can be photographed;
A digital image obtained by photographing with the first camera or an image obtained by applying a predetermined conversion process to the digital image is input as the first photographed image, and the digital image obtained by photographing with the second camera or the digital image An image input unit for inputting an image obtained by performing a predetermined conversion process on the image as a second captured image;
A captured image storage unit for storing the input first captured image and second captured image;
By analyzing the first captured image, four points on the first captured image corresponding to four reference points that are preset as points on the reference plane of interest on the subject are displayed. A first corresponding point extracting unit for extracting as one corresponding point;
By comparing the features of the first photographed image and the features of the second photographed image, the correspondence between the points on both images is obtained, and the four first corresponding points are obtained on the second photographed image. A second corresponding point extracting unit that extracts four corresponding second corresponding points,
On the subject corresponding to each corresponding point based on the information on the installation positions and orientations of the first camera and the second camera and the two-dimensional coordinate values of the four first corresponding points and the four second corresponding points A reference point coordinate calculation unit for obtaining three-dimensional coordinate values of the four reference points at
A two-dimensional uv coordinate system is defined on a reference plane including four reference points or a plane obtained by rotating the reference plane in a predetermined direction by a predetermined rotation angle, and the first captured image and the second When at least one of the captured images is selected as a conversion target image and a two-dimensional xy coordinate system is defined on the conversion target image, an arbitrary coordinate value (u, v) on the two-dimensional uv coordinate system, and Coefficient values of coordinate transformation formulas that define the relationship between the coordinate values (x, y) of the points on the two-dimensional xy coordinate system corresponding to the points indicated by the coordinate values, the coordinate values of the four reference points, A coefficient value calculation unit for calculating using a correspondence relationship with the coordinate values of the first corresponding point or the four second corresponding points;
An image conversion unit that converts a conversion target image on the two-dimensional xy coordinate system into a display target image on the two-dimensional uv coordinate system using a coordinate conversion formula defined by the coefficient value calculated by the coefficient value calculation unit;
A display target image storage unit for storing the display target image converted by the image conversion unit;
An image output unit for displaying a display target image on a monitor screen;
Is provided.

(2) According to a second aspect of the present invention, in the captured image display device according to the first aspect described above,
The first corresponding point extraction unit searches for a straight line portion on the first captured image, recognizes a quadrangle surrounded by the searched straight line portion as a reference plane, and sets four vertices of the quadrangle as four first corresponding points. It is made to extract as.

(3) According to a third aspect of the present invention, in the captured image display device according to the first aspect described above,
The first corresponding point extraction unit searches the human face from the first photographed image, and uses the surface including the right eye center point, left eye center point, lip right end point, and lip left end point in the searched face as a reference. A plane is recognized, and each of these points is extracted as four first corresponding points.

(4) According to a fourth aspect of the present invention, in the captured image display device according to the first aspect described above,
The first corresponding point extraction unit includes information on installation positions and orientations of the first camera and the second camera, two-dimensional subject information on the first photographed image, and two-dimensional subject information on the second photographed image. Based on the above, a three-dimensional shape model of the subject is created, a predetermined plane on the three-dimensional shape model is recognized as a reference plane, and four points corresponding to arbitrary four points on the reference plane are determined. This is extracted as the first corresponding point.

(5) According to a fifth aspect of the present invention, in the captured image display device according to the first to fourth aspects described above,
The reference point coordinate calculation unit has a function of storing information on the installation positions and orientations of the first camera and the second camera by a setting operation from the outside. The original coordinate value is obtained.

(6) According to a sixth aspect of the present invention, in the captured image display device according to the first to fourth aspects described above,
Based on the first photographed image and the second photographed image obtained by photographing the test subject stored in the photographed image storage unit in the reference point coordinate calculation unit, the installation positions of the first camera and the second camera And a function to automatically acquire orientation information.

(7) According to a seventh aspect of the present invention, in the captured image display device according to the first to sixth aspects described above,
The coefficient value calculation unit arbitrarily sets a scaling factor between the two-dimensional uv coordinate system and the two-dimensional xy coordinate system, so that the relationship between the coordinate value (u, v) and the coordinate value (x, y) is obtained. As a coordinate transformation formula to be defined,
x = (h11 · u + h12 · v + h13) / (h31 · u + h32 · v + 1)
y = (h21 · u + h22 · v + h23) / (h31 · u + h32 · v + 1)
The two formulas are defined, and the two formulas are coordinate values (u, v) of the two-dimensional uv coordinate system for the four reference points and the coordinate values (x of the two-dimensional xy coordinate system of the four first corresponding points) , Y) are used to calculate eight coefficients h11, h12, h13, h21, h22, h23, h31, h32 using eight simultaneous equations obtained by substituting respectively.

(8) According to an eighth aspect of the present invention, in the captured image display device according to the first to seventh aspects described above,
The coefficient value calculation unit selects both the first photographed image and the second photographed image as conversion target images, defines first and second two-dimensional xy coordinate systems on these conversion target images, respectively, An arbitrary coordinate value (u, v) on the dimension uv coordinate system, and a coordinate value (x, y) of a point on the first and second two-dimensional xy coordinate systems corresponding to the point indicated by the coordinate value, The coefficient value of the coordinate transformation formula that defines the relationship between is calculated using the correspondence between the coordinate values of the four reference points and the coordinate values of the four first corresponding points and the four second corresponding points,
The image conversion unit converts the first photographed image on the first two-dimensional xy coordinate system into the two-dimensional uv coordinate system using the first coordinate conversion formula defined by the coefficient value calculated by the coefficient value calculation unit. The second imaging on the second two-dimensional xy coordinate system using the second coordinate conversion formula defined by the coefficient value calculated by the coefficient value calculation unit while being converted into the first preparation image above. The display target image is generated by converting the image into a second preparation image on the two-dimensional uv coordinate system and synthesizing the first preparation image and the second preparation image.

(9) According to a ninth aspect of the present invention, in the photographed image display device according to the eighth aspect described above,
When the image conversion unit combines the first preparation image and the second preparation image, the pixel value of the pixel at the position indicated by the coordinate value (u, v) on the first preparation image is set to I1 (u , V), the pixel value of the pixel at the position indicated by the coordinate value (u, v) on the second preparation image is indicated by I2 (u, v), and the coordinate value (u, v) on the display target image. When the pixel value of the pixel at the position is I (u, v), using a predetermined weight coefficient α (0 ≦ α ≦ 1),
I (u, v) = (1−α) · I1 (u, v) + α · I2 (u, v)
The composition operation is performed.

(10) According to a tenth aspect of the present invention, in the captured image display device according to the ninth aspect described above,
An acute angle θ1 formed by a straight line connecting a predetermined representative point on the uv plane and the shooting point where the first camera is installed and the uv plane, and a straight line connecting this representative point and the shooting point where the second camera is installed Compare the acute angle θ2 made with the uv plane,
If θ1> θ2, α <0.5 is set, and the larger the difference between the two angles, the smaller α is set.
If θ1 <θ2, α is set to 0.5, and the larger the difference between the two angles, the larger α is set.
If θ1 = θ2, α = 0.5 is set.

(11) According to an eleventh aspect of the present invention, in the captured image display device according to the ninth aspect described above,
When determining the pixel value I (u, v) of the pixel of interest at the position indicated by the coordinate value (u, v) on the display target image, shooting with the arrangement point of the pixel of interest and the first camera installed The acute angle θ1 formed by the straight line connecting the points and the uv plane is compared with the acute angle θ2 formed by the straight line connecting the arrangement point and the shooting point where the second camera is installed and the uv plane, and the pixel value I (u , V) is used to determine the weighting factor α
If θ1> θ2, α <0.5 is set, and the larger the difference between the two angles, the smaller α is set.
If θ1 <θ2, α is set to 0.5, and the larger the difference between the two angles, the larger α is set.
If θ1 = θ2, α = 0.5 is set.

(12) According to a twelfth aspect of the present invention, in the photographed image display device according to the tenth or eleventh aspect described above,
α = (θ2−θ1) / π + 1/2
The weighting factor α is set by the following arithmetic expression.

(13) According to a thirteenth aspect of the present invention, in the captured image display device according to the ninth aspect described above,
The image converter
Based on the installation position and orientation information of the first camera and the second camera, the two-dimensional image information on the first photographed image, and the two-dimensional image information on the second photographed image, the subject and Create a 3D shape model of the obstacle, perform the process of recognizing the obstacle existing in front of the reference plane,
When determining the pixel value I (u, v) of the pixel of interest at the position indicated by the coordinate value (u, v) on the display target image, the pixel of interest is displayed on the first preparation image by the obstacle. If the pixel is in a region that is only a hidden surface, the weighting factor α used to determine the pixel value I (u, v) is set to α = 1, and the second preparation image is created by the obstacle. In the case of a pixel in a region that is a hidden surface only above, the weighting coefficient α used when determining the pixel value I (u, v) is set to α = 0.

(14) According to a fourteenth aspect of the present invention, in the captured image display device according to the first to thirteenth aspects described above,
When the image conversion unit converts a conversion target image on the two-dimensional xy coordinate system into a display target image or a preparation image on the two-dimensional uv coordinate system, the coordinate value (x, x) is used as an argument. y) is used to calculate an interpolation value obtained by interpolation using pixel values of a plurality of pixels existing in the vicinity of the position indicated by the coordinate value (x, y) on the image to be converted. The pixel value of the pixel arranged at the position indicated by the value (u, v) is used.

(15) According to a fifteenth aspect of the present invention, in the captured image display device according to the first to fourteenth aspects described above,
The coefficient value calculation unit defines a two-dimensional uv coordinate system on a plane obtained by rotating the reference plane around a predetermined axis about a predetermined rotation angle γ. .

(16) According to a sixteenth aspect of the present invention, in the captured image display device according to the first to fifteenth aspects described above,
A plurality of M cameras (M ≧ 3) installed at different positions and directed in a direction in which the same subject can be photographed, are provided.
A combination of any two of these M cameras is adopted as the first camera and the second camera so as to have a function of creating a display target image.

(17) According to a seventeenth aspect of the present invention, in the captured image display device according to the first to sixteenth aspects described above,
Using a camera equipped with a fisheye lens or an omnidirectional mirror as the first camera and the second camera,
A fish-eye image conversion unit that converts an image near a predetermined cut-out center point in a distorted circular image obtained by photographing with the first camera and the second camera into a planar regular image using a predetermined coordinate conversion formula Further provided,
The image input unit inputs the planar regular image converted by the fisheye image conversion unit as a first captured image and a second captured image, respectively.

  (18) According to an eighteenth aspect of the present invention, components other than the cameras in the captured image display device according to the first to seventeenth aspects described above are configured by incorporating a program into a computer. is there.

  (19) According to a nineteenth aspect of the present invention, the constituent elements other than the cameras in the photographed image display device according to the first to seventeenth aspects are configured by a semiconductor integrated circuit.

(20) According to a twentieth aspect of the present invention, a photographic image display device according to the second aspect described above is used to constitute a photographic display device for a license plate of a vehicle.
Install each camera in a position and orientation where you can shoot the license plate of the subject vehicle,
The first corresponding point extraction unit extracts the four vertices of the quadrangle constituting the license plate as four first corresponding points, using the surface of the license plate as a reference plane to be focused,
The image output unit outputs an image obtained by observing the license plate from the front direction as a display target image.

(21) According to a twenty-first aspect of the present invention, a monitoring system is configured using the captured image display device according to the third aspect described above.
Install each camera in a position and orientation that allows shooting the human face as the subject,
An image output unit outputs an image obtained by observing a human face from the front direction as a display target image.

(22) According to a twenty-second aspect of the present invention, in the captured image display method for generating and displaying an image obtained by observing the subject from a predetermined virtual direction based on the captured image of the subject captured from different directions,
A camera installation stage in which the first camera and the second camera are installed at different positions and in the direction in which the same subject can be photographed;
A computer inputs a digital image obtained by photographing with the first camera or an image obtained by applying a predetermined conversion process to the digital image as a first photographed image, and a digital image obtained by photographing with the second camera Alternatively, an image input step of inputting an image obtained by performing a predetermined conversion process on the digital image as a second captured image;
When the computer analyzes the first captured image, points corresponding to the four reference points set in advance as points on the reference plane to be focused on the subject are displayed on the first captured image. A first corresponding point extraction stage for extracting as four first corresponding points;
The computer compares the features of the first photographed image and the features of the second photographed image to obtain the correspondence between the points on the two images, and the four first images are displayed on the second photographed image. A second corresponding point extracting step of extracting four second corresponding points corresponding to the corresponding points;
The computer corresponds to each corresponding point based on the information on the installation positions and orientations of the first camera and the second camera, and the two first corresponding points and the two-dimensional coordinate values of the four second corresponding points. A reference point coordinate calculation step for obtaining three-dimensional coordinate values of four reference points on the subject to be
The computer defines a two-dimensional uv coordinate system on a reference plane including four reference points or on a plane obtained by rotating the reference plane in a predetermined direction by a predetermined rotation angle, and the first captured image When at least one of the second photographed image and the second photographed image is selected as a conversion target image and a two-dimensional xy coordinate system is defined on the conversion target image, arbitrary coordinate values (u, v) on the two-dimensional uv coordinate system And the coordinate value (x, y) of the point on the two-dimensional xy coordinate system corresponding to the point indicated by the coordinate value, the coefficient value of the coordinate conversion formula that defines the relationship between the coordinate value of the four reference points and A coefficient value calculating stage for calculating using a correspondence relationship with coordinate values of four first corresponding points or four second corresponding points;
Image conversion in which a computer converts a conversion target image on a two-dimensional xy coordinate system into a display target image on a two-dimensional uv coordinate system using a coordinate conversion formula defined by the coefficient value calculated in the coefficient value calculation stage Stages,
An image output stage in which a computer displays a display target image converted in the image conversion stage on a monitor screen;
Is to do.

  According to the photographed image display device of the present invention, a reference plane to be focused on is recognized on the subject based on two photographed images obtained by photographing the same subject from two different directions with two cameras. Since a two-dimensional image obtained by observing the reference plane from a predetermined direction set in advance is generated, a state in which an appropriate observation direction is determined in advance according to the application is observed from the appropriate observation direction. The two-dimensional image shown can be automatically generated and displayed.

It is a block diagram which shows the structure of the picked-up image display apparatus which concerns on fundamental embodiment of this invention. It is a top view which shows an example of the 1st captured image P1 and the 2nd captured image P2 which were stored in the captured image storage part 120 of the apparatus shown in FIG. FIG. 3 is a plan view showing a state in which first corresponding points S11 to S14 and second corresponding points S21 to S24 are extracted from the first captured image P1 and the second captured image P2 shown in FIG. It is a spatial layout diagram showing a basic concept of general epipolar geometry. It is a front view which shows an example of reference | standard points S31-S34 obtained on the to-be-photographed object. FIG. 6 is a layout diagram showing a positional relationship between a first captured image P1, a second captured image P2, and a reference plane R. It is a perspective view which shows the relationship between the position of the projection surface of a picked-up image, and the scaling of a picked-up image. It is a perspective view which shows the state which has arrange | positioned the projection surface (pi) of a picked-up image on the plane shown by Z = 1. It is a perspective view which shows the positional relationship of the three-dimensional XYZ coordinate system arrange | positioned on three-dimensional space, and a three-dimensional UVW coordinate system. It is a figure which shows the coordinate transformation type | formula which shows the relationship between 1 point p (x, y, 1) on the XYZ coordinate system shown in FIG. 9, and 1 point q (u, v, 1) on a UVW coordinate system. It is a figure which shows the deformation | transformation aspect of the coordinate transformation type | formula shown in FIG. It is a top view which shows four corresponding points obtained on xy coordinate system, and four reference points obtained on uv coordinate system. It is a figure which shows the determinant obtained based on the coordinate transformation formula shown in FIG. It is a figure which shows the coordinate transformation formula obtained by determining the coefficient h11-h32 in the determinant shown in FIG. It is a top view which shows the conversion process by the image conversion part 170 of the apparatus shown in FIG. It is a top view which shows the conversion process with the synthesis | combination of two picked-up images by the image conversion part 170 of the apparatus shown in FIG. FIG. 17 is a plan view illustrating an example of a method for combining two captured images in the conversion process illustrated in FIG. 16. FIG. 17 is a plan view illustrating another example of a method for combining two captured images in the conversion process illustrated in FIG. 16. FIG. 19 is a plan view showing two captured images to be combined by the combining method shown in FIG. 18. It is a top view which shows another example of the 1st corresponding point extracted by the 1st corresponding point extraction part 130 of the apparatus shown in FIG. It is a perspective view which shows the example which defined the uv coordinate system on the plane which rotated the reference | standard plane R only the predetermined rotation angle (gamma). It is a perspective view which shows the relationship between the distortion circular image S obtained by imaging | photography with a fisheye camera, and the planar regular image T. FIG. It is a figure which shows the coordinate conversion type | formula for converting the distortion circular image S obtained by imaging | photography with a fisheye camera into the planar regular image T. FIG. It is a block diagram which shows the structure of the picked-up image display apparatus which concerns on embodiment using a fisheye camera. It is a perspective view which shows arrangement | positioning of each camera in embodiment using four cameras.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic embodiment of the present invention >>
FIG. 1 is a block diagram showing a configuration of a captured image display apparatus according to a basic embodiment of the present invention. In this embodiment, an example in which a subject 30 is photographed by two cameras 10 and 20 is shown. Here, when the subject 30 is a vehicle, an embodiment will be described in which a license plate portion of the vehicle is taken as a subject, and this is photographed and displayed on a monitor screen.

  In the general application of “recognizing characters on the license plate” on the monitor screen, the “observation direction of the license plate from the front” is naturally an appropriate observation direction. Therefore, the apparatus shown in FIG. 1 moves the subject 30 from a predetermined virtual direction, that is, from the front of the license plate, based on the images taken by the two cameras 10 and 20 from different directions. This means an apparatus that generates and displays an observed image.

  In the case of the embodiment shown here, the first camera 10 and the second camera 20 are digital cameras capable of capturing a normal planar still image, and are installed at different positions as shown in the figure and are the same. The subject 30 is directed in a direction in which the subject 30 can be photographed. In the figure, the vehicle to be the subject 30 is arranged in the front direction, the first camera 10 is installed at a position where the front image is photographed from the left oblique direction, and the second camera 20 is installed at the position where the subject image is photographed from the right oblique direction. However, in actuality, it is not always necessary to be arranged in such a positional relationship.

  A system that automatically reads the letters on the license plate of a vehicle has already been put into practical use, and is used as a device that automatically reads the number of a vehicle traveling on a highway or the number of a vehicle passing through a parking lot gate. Has been. Thus, in an environment where the traveling direction of the vehicle is fixed, it is sufficient to read the number if an image obtained by photographing the front of the vehicle from a predetermined direction with one camera is obtained. The apparatus according to the present invention exhibits its main features when used in an environment where the orientation of the subject 30 is not determined.

  In this apparatus, first, a digital image obtained by photographing with the first camera 10 and a digital image obtained by photographing with the second camera 20 are input by the image input unit 110 as shown in FIG. In the unit 120, the first captured image P1 and the second captured image P2 are stored, respectively. If necessary, a predetermined conversion process may be applied to a digital image obtained by photographing with each camera when inputting the image. For example, when a fisheye camera is used, a process of converting a distorted circular image into a planar regular image is necessary (details will be described in the modification of §4). When a digital video camera is used, frame images taken at the same point are taken in as a first shot image P1 and a second shot image P2, respectively. It will be executed continuously along the series.

  FIG. 2 is a plan view showing an example of the first photographed image P1 and the second photographed image P2 stored in the photographed image storage unit 120. A first photographed image P1 shown in FIG. 2 (a) is an image obtained by photographing the vicinity of the license plate in front of the vehicle to be the subject 30 from the diagonal left direction, and a second photographed image P2 shown in FIG. FIG. 6 is an image obtained by photographing the vicinity of the license plate in front of the vehicle as the subject 30 from the diagonally right direction. Note that, in the actual photographed image, not only the license plate but also the front part of the vehicle body is shown, but for convenience of explanation, only the part of the license plate is shown here.

  An important feature of the present invention is that information on “display a two-dimensional image obtained by observing which part of a subject from which direction” is set in advance for the captured image display device, and display according to the setting is performed. There is in point to let you do. Specifically, in the embodiment illustrated here, information “display a two-dimensional image obtained by observing the license plate portion from the front direction” is set. For this purpose, the present invention defines a reference plane that is a target of interest on the subject. In the above example, the surface of the license plate is defined as the reference plane.

  Further, four reference points are defined on this reference plane. Since these four reference points are points that define the reference plane, they may be arbitrary points on the reference plane (however, they should not include three points on a straight line). However, as described later, since it is necessary to perform processing for extracting corresponding points corresponding to these four reference points from the photographed image, in practice, it is characteristic that such extraction processing can be easily performed. Preferably, the point is defined as a reference point. For example, if the surface of the license plate is used as a reference plane, it is optimal to define four vertices of a rectangle constituting the license plate as reference points. Therefore, in the following description, an example in which these four vertices are defined as reference points will be described in advance.

  Since the plane can be defined by three points that are not on a straight line, it is sufficient to define three reference points in order to define the reference plane. However, as described in detail in §2, in the present invention, four reference points are required in order to solve the simultaneous equations for obtaining the coefficients of the coordinate conversion equation.

  The first corresponding point extraction unit 130 illustrated in FIG. 1 analyzes each of the first captured images P <b> 1, so that each of the four reference points set in advance as points on the reference plane that is the target of interest on the subject 30 is obtained. A process of extracting corresponding points as four first corresponding points on the first captured image P1 is performed. That is, in the first corresponding point extraction unit 130, information on four reference points is set in advance as points on the reference plane to be focused on the subject 30, and the first captured image P1 is analyzed. As a result, processing for recognizing points corresponding to the four reference points is performed.

  Here, it is assumed that the setting is made so that the four vertices of the rectangle constituting the license plate are recognized as reference points. In this case, by analyzing the first photographed image P1 shown in FIG. 2A, four corresponding points S11 to S14 as shown in FIG. 3A are extracted. Specifically, the first corresponding point extraction unit 130 first searches for a straight line portion on the first captured image P1, recognizes a rectangle surrounded by the searched straight line portion as a reference plane, A process of extracting the four vertices as the four first corresponding points S11 to S14 may be performed.

  In general, as a method for recognizing a straight line included in an image, various methods such as a method of calculating a pixel value histogram and a method of calculating a density gradient of pixel values are known and put into practical use. A method of recognizing a quadrangle using such a straight line recognition method is also a known method, and is actually used in various fields. Therefore, in this specification, a description of a specific processing method for extracting four vertices of a quadrangle from a captured image is omitted.

  In short, the first corresponding point extraction unit 130 performs an analysis on the first captured image P1 based on a predetermined analysis algorithm set in advance, and executes a process of extracting the four first corresponding points S11 to S14. That's fine. The important point here is that the four extracted first corresponding points S11 to S14 correspond to the four reference points located on the reference plane to be focused on the subject. In the above example, the reference plane is the surface of the license plate of the subject 30, and the four reference points are the four vertices of the rectangle constituting the license plate, so the four first corresponding points S11 to S14 are the first ones. That is, the four vertices of a quadrangle constituting the license plate shown in the photographed image P1.

  On the other hand, the second corresponding point extraction unit 140 performs a process of extracting four corresponding points (second corresponding points) corresponding to the four reference points on the second captured image P2. As a result, four second corresponding points S21 to S24 as shown in FIG. 3B are extracted. However, a method different from the extraction of the first corresponding points S11 to S14 described above is adopted for the extraction of the second corresponding points S21 to S24. That is, the second corresponding point extracting unit 140 obtains the correspondence between the points on the two images by comparing the feature of the first photographed image P1 and the feature of the second photographed image P2, and the second On the photographed image P2, four second corresponding points S21 to S24 corresponding to the four first corresponding points S11 to S14 are extracted.

  In general, if there are two photographed images obtained by photographing the same subject from different directions, the correspondence between the pixels constituting both images can be obtained. For example, when two photographed images as shown in FIGS. 2 (a) and 2 (b) are given, since the photographed subject is the same, for example, when each image is scanned in the horizontal direction, If features such as changes in the density gradient of the pixel value and changes in the density gradient of the pixel value when scanned in the vertical direction are analyzed, it is possible to recognize that two points with similar neighboring features correspond to each other. is there. As described above, a technique for obtaining the correspondence between points constituting two images of two captured images obtained by photographing the same subject from different directions is a known technique and is used in various fields. (For example, "OpenCV Programming Book" published by Mainichi Communications Inc. (2009), "Practical OpenCV-Video Processing & Analysis" published by Cut Systems Inc. (2009), "Detailed OpenCV" published by O'Reilly Japan (2009), etc. See the book). Therefore, in this specification, a description of a more detailed processing method is omitted.

  After all, the four first corresponding points S11 to S14 shown in FIG. 3 (a) are subjected to analysis (rectangular recognition) based on a predetermined algorithm for the first captured image P1 shown in FIG. 2 (a). In contrast, the four second corresponding points S21 to S24 shown in FIG. 3 (b) compare the characteristics of the images P1 and P2 shown in FIGS. 2 (a) and 2 (b). As a result, a correspondence relation indicating which point on the image P1 corresponds to which point on the image P2 is obtained, and using the correspondence relation, four first correspondence points S11 shown in FIG. ˜S14 are extracted as corresponding points on the image P2. The reason for adopting such a method is that the second corresponding points S21 to S24 can be extracted as more accurate points corresponding to the first corresponding points S11 to S14.

  In this way, as shown in FIGS. 3A and 3B, four first corresponding points S11 to S14 are determined on the first photographed image P1, and four second points on the second photographed image P2. When the corresponding points S21 to S24 are determined, the information on these eight points is given to the reference point coordinate calculation unit 150. The reference points are four points defined on the reference plane on the subject 30, but the information is two-dimensional information as the positions of corresponding points S11 to S14 and S21 to S24 on the captured images P1 and P2. It is only recorded as. Based on this two-dimensional information, the reference point coordinate calculation unit 150 performs processing for restoring the positions of the four reference points in the three-dimensional space, that is, the three-dimensional information of each reference point.

  In general, in a three-dimensional space, the correspondence between two captured images obtained when the same subject is observed from different directions and the subject can be obtained by using geometry called epipolar geometry.

  FIG. 4 is a spatial layout diagram showing the basic concept of general epipolar geometry. In this figure, two projected images obtained by observing the feature point ξ on the subject from two different viewpoints Q1 and Q2 are shown. The projection image obtained on the projection plane π1 is an image obtained when observed from the first viewpoint Q1, and the feature point ξ is projected onto the projection point ξ1. On the other hand, the projection image obtained on the projection plane π2 is an image obtained when observed from the second viewpoint Q2, and the feature point ξ is projected onto the projection point ξ2.

  Here, the straight line B connecting the two viewpoints Q1 and Q2 is called a base line, and the intersections of the base line B and the projection planes π1 and π2 are called epipoles e1 and e2. The straight line E1 connecting the projection point ξ1 and the epipole e1 and the straight line E2 connecting the projection point ξ2 and the epipole e2 are called epipolar lines, and the plane passing through the three points ξ, Q1, and Q2 is the epipolar plane EP. being called.

  If FIG. 4 is considered as a photographing system using two cameras for the same subject, the viewpoints Q1 and Q2 are photographing points where each camera is installed. Strictly speaking, the viewpoint Q1 is a first shooting point where the lens of the first camera 10 is focused, and the viewpoint Q2 is a second shooting where the lens of the second camera 20 is focused. It becomes a point. The first projection plane π1 is an imaging plane orthogonal to the optical axis of the first camera 10, and the subject image projected on the imaging plane π1 corresponds to the first captured image P1. Similarly, the second projection plane π2 is an imaging plane orthogonal to the optical axis of the second camera 20, and the subject image projected on the imaging plane π2 corresponds to the second captured image P2.

  Eventually, the projection point ξ1 shown in the figure becomes a corresponding point on the first photographed image P1 corresponding to the feature point ξ on the subject, and the projection point ξ2 is a second photographed image corresponding to the feature point ξ on the subject. Corresponding point on P2. Here, if the positions of the photographing points Q1 and Q2 are fixed, the position of the feature point ξ on the subject in the three-dimensional space is determined based on the positions of the projection points ξ1 and ξ2 based on the principle of triangulation. Can be sought. That is, the position of the feature point ξ in the three-dimensional space can be determined as the intersection of the straight line connecting the two points Q1 and ξ1 and the straight line connecting the two points Q2 and ξ2.

  Actually, the positions of the projection points ξ1 and ξ2 are given as coordinate values on a two-dimensional coordinate system defined on the imaging surfaces π1 and π2, but the imaging surfaces π1 and π2 are orthogonal to the optical axis of each camera. If the installation position of each camera, its orientation, and the zoom magnification are determined, the positions of the imaging surfaces π1, π2 in the three-dimensional space can be determined, and the three-dimensional projection points ξ1, ξ2 are determined. Coordinate values can be determined. If the zoom magnifications of the two cameras are set to be the same (usually, such a setting will be performed), the camera zoom magnification simply determines the size of the image of the subject to be restored. It is only a parameter to be determined, and is not essential information for restoring the shape of the subject. Therefore, in practice, if information on the installation positions and orientations of the first camera 10 and the second camera 20 is obtained, the cubic of the feature points ξ is based on the information on the positions of the corresponding projection points ξ1, ξ2. Coordinates in the original space can be determined.

  The reference point coordinate calculation unit 150 illustrated in FIG. 1 includes information on the installation positions and orientations of the first camera 10 and the second camera 20, and four first corresponding points S11 given from the first corresponding point extraction unit 130. -S14 and the two-dimensional coordinate values of the four second corresponding points S21 to S24 given from the second corresponding point extraction unit 140, on the subject 30 corresponding to these corresponding points. The process of obtaining the three-dimensional coordinate values of the four reference points S31 to S34 at is performed. FIG. 5 is a front view showing an example of the reference points S31 to S34 obtained on the subject 30. FIG. In this figure, for convenience of explanation, the entire image of the subject 30 is drawn. However, what is actually calculated by the reference point coordinate calculation unit 150 is the three-dimensional of the four reference points S31 to S34 on the subject 30. It is only a coordinate value. As described above, the four reference points S31 to S34 are points on the reference plane R set as the target surface of interest on the subject 30 (in this example, the surface of the license plate).

  Four reference points S31 to S34 based on the information on the installation position and orientation of each camera, the two-dimensional coordinate values of the first corresponding points S11 to S14, and the two-dimensional coordinate values of the second corresponding points S21 to S24. The reason why the three-dimensional coordinate value is obtained can be easily understood from the principle of epipolar geometry shown in FIG. For example, the captured images P1 and P2 shown in FIGS. 3 (a) and 3 (b) correspond to each other as shown in FIGS. 3 (a) and 3 (b) as images obtained on the imaging surfaces π1 and π2 shown in FIG. If the points S11 and S21 to be performed are the projection points ξ1 and ξ2 shown in FIG. 4, the three-dimensional coordinate value of the reference point S31 shown in FIG. 5 is obtained as the feature point ξ shown in FIG. The same applies to the other reference points S32, S33, S34.

  Note that information on the installation positions and orientations of the first camera 10 and the second camera 20 is obtained by the user from the reference point coordinate calculation unit 150 after the operation of installing each camera at a fixed position is completed. Can be stored in the reference point coordinate calculation unit 150. When such an operation is adopted, the reference point coordinate calculation unit 150 has a function of storing information on the installation positions and orientations of the first camera 10 and the second camera 20 by an external setting operation. Using the stored information, the three-dimensional coordinate values of the four reference points may be obtained. Such an operation method is generally called “calibration”.

  On the other hand, based on the first captured image P1 and the second captured image P2 obtained by capturing the test subject stored in the captured image storage unit 120 in the reference point coordinate calculation unit 150, the first camera 10 and It is also possible to have a function of automatically acquiring information on the installation position and orientation of the second camera 20. That is, a program for executing an algorithm for estimating the installation positions and orientations of the two cameras used for photographing based on two photographed images of a predetermined test subject is incorporated in the reference point coordinate calculating unit 150. After completing the work of installing each camera at a fixed position, if you take a test shoot of the test subject, you can automatically obtain information on the installation position and orientation of the two cameras using the results of the test shoot. It becomes possible to do.

  Such an operation method is generally called “non-calibration”. This operation method is convenient when the camera installation position and orientation are frequently changed and used. It should be noted that an algorithm for automatically acquiring information on the installation position and orientation of the camera based on the photographed image of the test subject is also a well-known algorithm that has already been put into practical use, and will not be described in detail here.

  Now, when the three-dimensional coordinate values of the four reference points S31 to S34 are obtained in this way, the four first corresponding points S11 to S14 and the second on the first photographed image P1 on the global three-dimensional coordinate system. The positions of 12 points of the four second corresponding points S21 to S24 on the captured image P2 and the four reference points S31 to S34 on the reference plane R are determined. FIG. 6 is an arrangement diagram showing the positional relationship between the first photographed image P1, the second photographed image P2, and the reference plane R arranged in the three-dimensional space.

  Here, the points S11, S21, S31 are points corresponding to each other, the points S12, S22, S32 are points corresponding to each other, the points S13, S23, S33 are points corresponding to each other, and the points S14, S24, S34 is a point corresponding to each other. Coordinates indicating a general correspondence between an arbitrary point p on the first photographed image P1 and a point q on the reference plane R are obtained by using the correspondence of each point whose position on the space is known. A conversion equation can be defined, and similarly, a coordinate conversion equation indicating a general correspondence between an arbitrary point p on the second captured image P2 and a point q on the reference plane R is defined. Can do.

  Here, as shown in the figure, a first two-dimensional xy coordinate system is defined on the first captured image P1 (imaging plane π1 of the first camera), and an arbitrary point p on the first captured image P1 is defined. The position is indicated by coordinates (x, y). Similarly, a second two-dimensional xy coordinate system is defined on the second photographed image P2 (imaging plane π2 of the second camera), and the second photograph The position of an arbitrary point p on the image P2 is indicated by coordinates (x, y). The first two-dimensional xy coordinate system and the second two-dimensional xy coordinate system are both local coordinate systems defined only on each captured image, and the coordinates of both are completely independent. On the other hand, as shown in the figure, a local two-dimensional uv coordinate system is defined also on the reference plane R including the four reference points S31 to S34, and the position of an arbitrary point q on the reference plane R is expressed by coordinates (u, v).

  The three local two-dimensional coordinate systems are coordinate systems arranged at predetermined positions of the global three-dimensional coordinate system described above, and the positions of the respective coordinate systems are known on the global coordinate system. . That is, the imaging plane π1 of the first camera is a plane defined as a projection plane orthogonal to the optical axis of the first camera 10, and the position of the shooting point Q1 where the first camera 10 is installed. If the orientation of the optical axis is determined, the position of the imaging plane π1 in the global coordinate system can be determined. Theoretically, the position of the imaging plane π1 translates along the optical axis according to the zoom magnification of the first camera 10, but as described in detail in §2, the zoom magnification of the camera is Since it becomes a scaling factor that determines the size of the image, it may be set arbitrarily in practice.

  Similarly, regarding the imaging plane π2 of the second camera, if the position of the photographing point Q2 where the second camera 20 is installed and the direction of the optical axis thereof are determined, the position in the global coordinate system can be determined. . Further, since the reference plane R is a plane including four reference points S31 to S34, the reference plane R in the global coordinate system is based on the three-dimensional coordinate values of the reference points S31 to S34 calculated by the reference point coordinate calculation unit 150. The position can be determined.

  In FIG. 6, a reference plane R is a plane that is a target of interest on the subject restored in the three-dimensional space, and corresponds to the surface of the license plate in the above example. Therefore, if the image of the subject can be restored on the two-dimensional uv coordinate system defined on the reference plane R, the restored image becomes an “image obtained by observing the license plate from the front”. This is a target display target image (a two-dimensional image showing a state in which an object with an appropriate observation direction determined in advance according to the application is observed from the appropriate observation direction). That is, a front image when the license plate is observed from a virtual viewpoint E (a point on a line orthogonal to the reference plane R) shown in FIG. 6 is obtained.

  In order to restore the image of the subject 30 on the two-dimensional uv coordinate system, the first captured image P1 may be used, or the second captured image P2 may be used. When the first photographed image P1 is used, the first photographed image P1 is used as a conversion target image, and the point p (x, y) on the first two-dimensional xy coordinate system and the point on the two-dimensional uv coordinate system are used. When the conversion using the coordinate conversion formula “(u, v) = f (x, y)” indicating the correspondence with q (u, v) is performed and the second captured image P2 is used, the second Coordinate conversion showing the correspondence between the point p (x, y) on the second two-dimensional xy coordinate system and the point q (u, v) on the two-dimensional uv coordinate system Conversion using the expression “(u, v) = g (x, y)” is performed.

  The coefficient value calculation unit 160 shown in FIG. 1 has a function of calculating the value of the coefficient included in the function f or g. That is, when the first captured image P1 or the second captured image P2 is used as a conversion target image and a two-dimensional xy coordinate system is defined on the conversion target image, an arbitrary coordinate value on the two-dimensional uv coordinate system ( u, v) and the coordinate value (x, y) of the point on the two-dimensional xy coordinate system corresponding to the point indicated by the coordinate value, the coefficient value of the coordinate transformation formula that defines the relationship between the four reference points Is calculated using a correspondence relationship between the coordinate values of the four first corresponding points or the coordinate values of the four second corresponding points.

  In short, when the first captured image P1 is a conversion target image, “an arbitrary coordinate value (u, v) on the two-dimensional uv coordinate system” and “a point q (u, v) indicated by the coordinate value” are displayed. ) And a coordinate conversion formula “(u, v) = f (x, y,)” that defines the relationship between the coordinate value (x, y) of the point p (x, y) on the first two-dimensional xy coordinate system corresponding to y) ”is calculated using the correspondence between the coordinate values of the four reference points S31 to S34 and the coordinate values of the four first corresponding points S11 to S14. When the second captured image P2 is the conversion target image, “arbitrary coordinate values (u, v) on the two-dimensional uv coordinate system” and “point q (u, v) indicated by the coordinate values” are displayed. ) Corresponding to the coordinate value (x, y) of the point p (x, y) on the second two-dimensional xy coordinate system corresponding to the coordinate conversion expression “(u, v) = g (x, y) ”is calculated using the correspondence relationship between the coordinate values of the four reference points S31 to S34 and the coordinate values of the four second corresponding points S21 to S24. The coefficient value calculation method for a specific function formula performed by the coefficient value calculation unit 160 will be described in detail in Section 2.

  The image conversion unit 170 illustrated in FIG. 1 has a coordinate conversion formula “(u, v) = f (x, y)” or “(u, v) = defined by the coefficient value calculated by the coefficient value calculation unit 160. g (x, y) ”is used to convert the image to be converted on the first two-dimensional xy coordinate system or the second two-dimensional xy coordinate system, that is, the first photographing stored in the photographed image storage unit 120. A process of converting the image P1 or the second captured image P2 into a display target image P3 on the two-dimensional uv coordinate system is performed. By this conversion processing, a display target image P3 is obtained on the two-dimensional uv coordinate system shown in FIG. As described above, the display target image P3 is an image obtained by restoring the reference plane R of the original subject 30, and an image showing a state in which the reference plane R is observed from the front.

  As shown in FIG. 1, the display target image P3 thus created by the conversion processing by the image conversion unit 170 is stored in the display target image storage unit 180, output to the monitor device by the image output unit 190, and displayed on the monitor screen. Is displayed. Eventually, the front image of the license plate portion shown in FIG. 5 is displayed on the monitor screen. As described above, according to the photographed image display device according to the present invention, the user can perform the appropriate observation for the subject whose appropriate observation direction is determined in advance according to the use without performing any instruction operation regarding the observation direction. A two-dimensional image showing a state observed from the observation direction is automatically generated and displayed on the monitor screen.

  The configuration of the captured image display device according to the basic embodiment of the present invention has been described above with reference to the block diagram of FIG. 1, but actually, the components other than the cameras 10 and 20 in this captured image display device. That is, the image input unit 110, the captured image storage unit 120, the first corresponding point extraction unit 130, the second corresponding point extraction unit 140, the reference point coordinate calculation unit 150, the coefficient value calculation unit 160, the image conversion unit 170, the display target The image storage unit 180 and the image output unit 190 can be configured by incorporating a dedicated program into the computer. Moreover, it can also be comprised by semiconductor integrated circuits, such as LSI.

<<< §2. Specific processing in the coefficient value calculation section >>
Here, specific processing performed by the coefficient value calculation unit 160 in the apparatus shown in FIG. 1 will be described. As described in §1, the coefficient values calculated here are the point p (x, y) on the first two-dimensional xy coordinate system and the point q (u, v) on the two-dimensional uv coordinate system shown in FIG. )) Or the value of the coefficient included in the function f of the coordinate conversion expression “(u, v) = f (x, y)” indicating the correspondence relationship or the point p (x on the second two-dimensional xy coordinate system , Y) and the value of the coefficient included in the function g of the coordinate conversion formula “(u, v) = g (x, y)” indicating the correspondence between the point q (u, v) on the two-dimensional uv coordinate system. It is. Since both relate to the conversion formula between the coordinate value (x, y) and the coordinate value (u, v), the function f will be described here.

  If the function f is determined, the coordinates (x, y) of an arbitrary point p (x, y) on the first two-dimensional xy coordinate system are converted into coordinates (u, v) on the two-dimensional uv coordinate system. Therefore, a pixel located at an arbitrary point p (x, y) on the first photographed image P1 is plotted at the position of the coordinate conversion point q (u, v) on the two-dimensional uv coordinate system. The first captured image P1 can be converted into a display target image on the two-dimensional uv coordinate system. This conversion process is executed by the image conversion unit 170 as described in §1.

  As the coefficient values included in the function f, the known coordinate values of the four points S11 to S14 on the first two-dimensional xy coordinate system and the known coordinate values of the four points S31 to S34 on the two-dimensional uv coordinate system are used. Calculated by calculation. In theory, however, the coordinate values of these eight points are not sufficient to determine all the coefficient values included in the function f. Therefore, in the present invention, by devising the premise that “the scaling factor between the two-dimensional uv coordinate system and the two-dimensional xy coordinate system can be arbitrarily set”, the coordinate values of the eight points are obtained. The coefficient value can be calculated by using this.

  In FIG. 6, if the positions of the photographing points Q1 and Q2, the positions of the imaging planes π1 and π2, and the position of the reference plane R on the global coordinate system can be completely determined, the subject 30 on the actual scale is placed on the reference plane R. Since the image can be restored, an image on the actual scale can be obtained as a display target image on the two-dimensional uv coordinate system. However, in practical terms, there is no technical significance in obtaining an actual display image. Actually, since the display target image is obtained as digital image data, it is obtained as a pixel array having a predetermined pitch. Therefore, the size of the obtained image data (the number of vertical and horizontal pixels) changes depending on the setting of the pixel pitch. In addition, when the display target image is displayed on the monitor screen, the final size of the displayed image changes depending on conditions such as the pixel size corresponding to the monitor device to be used.

  After all, since it is meaningless to restore the subject image on the reference plane R, the scaling factor between each captured image on the two-dimensional xy coordinate system and the display target image on the two-dimensional uv coordinate system is It can be set arbitrarily.

  FIG. 7 is a perspective view showing the relationship between the position of the projection plane of the captured image and the scaling of the captured image in the three-dimensional XYZ coordinate system. The projection planes πa and πb shown in the figure are both projection planes parallel to the XY plane, the projection plane πa is arranged on a plane indicated by Z = za, and the projection plane πb is indicated by Z = zb. It is arranged on a plane. Here, when the viewpoint Q1 is taken at the position of the origin O of this coordinate system and the feature point ξ on the subject is observed, a projection point ξa is formed on the projection surface πa as shown in the figure, and the projection point ξa is formed on the projection surface πb. Forms a projection point ξb. In this way, each point of the subject is projected on each of the projection planes πa and πb to form a projection image of the entire subject, but the projection image formed on the projection plane πa The projected image formed on the projection plane πb has a similar shape. In other words, if the projection image formed on the projection plane πa is enlarged zb / za times, a projection image formed on the projection plane πb can be obtained.

  After all, translating the projection plane π in the Z-axis direction is equivalent to changing the scaling factor of the projection image. If the scaling factor can be arbitrarily set, the position of the projection plane π in the Z-axis direction Can also be set arbitrarily. Therefore, as shown in FIG. 8, the projection plane π is arranged on a plane indicated by Z = 1, and a two-dimensional xy coordinate system is defined on the projection plane π. An arbitrary point p on the two-dimensional xy coordinate system is indicated by a two-dimensional coordinate value p (x, y). In the three-dimensional XYZ coordinate system, a three-dimensional coordinate value p (x, y, 1 ).

  On the other hand, as shown in FIG. 9, let us define a three-dimensional XYZ coordinate system and a completely different three-dimensional UVW coordinate system (origin O ′) in the three-dimensional global space. The XYZ coordinate system and the UVW coordinate system are completely independent coordinate systems, and the positions of the origins O and O ′ and the orientations of the coordinate axes are set independently. Then, on the XYZ coordinate system, as shown in FIG. 8, a projection plane π arranged on the plane indicated by Z = 1 is defined, and a two-dimensional xy coordinate system is defined on the projection plane π. . On the other hand, on the UVW coordinate system, a reference plane R arranged on a plane indicated by W = 1 is defined, and a two-dimensional uv coordinate system is defined on the reference plane R. An arbitrary point q on the two-dimensional uv coordinate system is indicated by a two-dimensional coordinate value q (u, v), but in the three-dimensional UVW coordinate system, the three-dimensional coordinate value q (u, v, 1). ).

  As described above, the XYZ coordinate system and the UVW coordinate system are completely independent coordinate systems in which positions and orientations can be arbitrarily set. Therefore, the first two-dimensional xy coordinate system ( A two-dimensional xy coordinate system and a two-dimensional model shown in the model of FIG. 9 represent a two-dimensional uv coordinate system (coordinate system on the reference plane R) on the projection plane π1 on which the first photographed image P1 is formed. It can be considered by replacing with the uv coordinate system. However, there is no perfect consistency regarding the scaling factor. The distance between the projection plane π1 shown in FIG. 6 and the viewpoint Q1 is determined based on the zoom magnification of the first camera 10 used for shooting, whereas the projection plane π shown in the model of FIG. And the viewpoint Q1 (origin O) are fixed at 1. However, as described above, since the scaling factor can be set arbitrarily in practice, there is no problem even if the coordinate conversion formula is defined assuming the coordinate system shown in the model of FIG.

The coordinate transformation formula to be defined here is an arbitrary point p (x, y, 1) on the three-dimensional XYZ coordinate system shown in FIG. 9 and a point q (u, v, 1) on the three-dimensional UVW coordinate system. Is an expression showing the correspondence relationship In general, a coordinate conversion formula between two different three-dimensional coordinate systems is expressed using a homography matrix H, and a coordinate value (x, y, z) on a three-dimensional XYZ coordinate system and a three-dimensional UVW coordinate system. The coordinate conversion formula between coordinate values (u, v, w) is
[X y z] ^T = H [u v w] ^T
It is expressed by the determinant Here, H is a 3 × 3 homography matrix, and a matrix with T on the right shoulder indicates a transposed matrix (in the specification, a column vector is used for the sake of convenience of performing matrix expression only with character strings). Is shown in the row vector transpose form).

In the model shown in FIG. 9, in the above formula, z = 1 and w = 1 can be fixed.
[X y 1] ^T = H [u v 1] ^T
The following formula may be used. Here, further, an arbitrary scaling factor σ is introduced by utilizing the advantage that the scaling factor between the xy coordinate system and the uv coordinate system can be arbitrarily set, as shown in Expression (1) in FIG. [X y 1] ^T = H [u v 1] ^T · σ
I will formulate Here, if each element of the 3 × 3 homography matrix H is h11, h12, h13, h21, h22, h23, h31, h32, h33, the above equation can be expressed as equation (2) in FIG. Can be expressed as

Dividing the determinant shown in equation (2) into three equations,
x = h11 · uσ + h12 · vσ + h13 · σ
y = h21 · uσ + h22 · vσ + h23 · σ
1 = h31 · uσ + h32 · vσ + h33 · σ
It becomes. Solving this third equation for σ,
σ = 1 / (h31 · u + h32 · v + h33)
Therefore, when this σ is substituted into the first expression and the second expression and rearranged, as shown in FIG. 10 as the expressions (3) and (4),
x = (h11 · u + h12 · v + h13) / (h31 · u + h32 · v + h33)
y = (h21 · u + h22 · v + h23) / (h31 · u + h32 · v + h33)
The following formula is obtained.

Dividing the denominator numerator of these two formulas by h33,
x = (h11 / h33 · u + h12 / h33 · v + h13 / h33)
/ (H31 / h33 · u + h32 / h33 · v + h33 / h33)
y = (h21 / h33 · u + h22 / h33 · v + h23 / h33)
/ (H31 / h33 · u + h32 / h33 · v + h33 / h33)
Therefore, h11 / h33, h12 / h33, h13 / h33, h21 / h33, h22 / h33, h23 / h33, h31 / h33, h32 / h33 are designated as h11 ′, h12 ′, h13 ′, h21 ′, If you say h22 ', h23', h31 ', h32'
x = (h11'.u + h12'.v + h13 ') / (h31'.u + h32'.v + 1)
y = (h21'.u + h22'.v + h23 ') / (h31'.u + h32'.v + 1)
And can be transformed. Here, if h11 ′ to h32 ′ are newly set as h11 to h32,
x = (h11 · u + h12 · v + h13) / (h31 · u + h32 · v + 1)
y = (h21 · u + h22 · v + h23) / (h31 · u + h32 · v + 1)
The following formula is obtained. After all, in the formulas (3) and (4) in FIG. 10, for convenience, it is possible to derive the formulas (5) and (6) shown in FIG. 10 by setting h33 = 1.

If both sides are multiplied by the denominator of Expression (5) in FIG. 10, Expression (7) is obtained as shown in FIG. 11, and Expression (8) is obtained by shifting a part of the left side to the right side. Similarly, if both sides are multiplied by the denominator of equation (6) in FIG. 10, equation (9) is obtained as shown in FIG. 11, and when a part of the left side is moved to the right side, equation (10) is obtained. It is done. After all, when the model shown in FIG. 9 is used, the coordinate conversion equations between the coordinate values (x, y) and the coordinate values (u, v) are shown in equations (8) and (10) in FIG. like,
x = h11.u + h12.v + h13-h31.x.u-h32.x.v
y = h21 * u + h22 * v + h23-h31 * y * u-h32 * y * v
It becomes.

  These two expressions are expressions showing the relationship between an arbitrary coordinate value (x, y) on the two-dimensional xy coordinate system and an arbitrary coordinate value (u, v) on the two-dimensional uv coordinate system. Is an expression derived on the premise of two three-dimensional coordinate system models arranged on the global coordinate system shown in FIG. In this model, the two-dimensional xy coordinate system is a two-dimensional coordinate system defined at a predetermined position on the three-dimensional XYZ coordinate system, and the two-dimensional uv coordinate system is defined at a predetermined position on the three-dimensional UVW coordinate system. It is a two-dimensional coordinate system. The two-dimensional xy coordinates are coordinates defined on the imaging plane π1 of the first camera, and the first captured image P1 is arranged there. The two-dimensional uv coordinates are coordinates defined on the reference plane R (the target target surface on the subject 30), and the display target image P3 is formed there.

  Therefore, if these two expressions are used, mutual conversion between the coordinate values (u, v) and the coordinate values (x, y) can be performed, and the first captured image P1 on the two-dimensional xy coordinates can be obtained. , It can be converted into a display target image P3 on the two-dimensional uv coordinates. However, in order to use these equations as coordinate transformation equations, it is necessary to determine the values of the eight coefficients h11 to h32. As described in §1, the coefficient value calculation unit 160 performs a process of calculating these eight coefficient values using the coordinate values of the eight points.

  FIG. 12 is a plan view showing eight points used for this calculation process. FIG. 12A is a diagram showing four first corresponding points S11 to S14 arranged on the first two-dimensional xy plane. Here, these corresponding points are points indicated by coordinates of S11 (x1, y1), S12 (x2, y2), S13 (x3, y3), S14 (x4, y4), respectively, in the xy coordinate system. Let's have it. These four first corresponding points S11 to S14 are extracted by the first corresponding point extracting unit 130, and the coordinate values thereof are obtained.

  On the other hand, FIG. 12B is a diagram showing four reference points S31 to S34 arranged on the two-dimensional uv plane. Here, these reference points are points indicated by coordinates S31 (u1, v1), S32 (u2, v2), S33 (u3, v3), S34 (u4, v4), respectively, in the uv coordinate system. Let's have it. The coordinate values of these four reference points S31 to S34 are calculated by the reference point coordinate calculation unit 150.

  Moreover, the corresponding point S11 (x1, y1) is a point corresponding to the reference point S31 (u1, v1), and the corresponding point S12 (x2, y2) is a point corresponding to the reference point S32 (u2, v2). Corresponding point S13 (x3, y3) is a point corresponding to reference point S33 (u3, v3), and corresponding point S14 (x4, y4) is a point corresponding to reference point S34 (u4, v4). Therefore, the combination of the coordinate values (x, y) and the coordinate values (u, v) in the correspondence relationship satisfies the equations (8) and (10) shown in FIG. Therefore, if the above four combinations are substituted into the equations (8) and (10), a total of eight simultaneous equations can be obtained.

  FIG. 13 is a diagram showing these eight simultaneous equations in the form of determinants. For example, each term in the first row of the determinant of 8 rows and 8 columns on the left is a term obtained by substituting x = x1, u = u1, v = v1 into equation (8) in FIG. Yes, each term in the second row is a term obtained by substituting y = y1, u = u1, v = v1 into equation (10) in FIG. That is, each term in the first and second rows of this determinant is a term indicating the correspondence between the corresponding point S11 (x1, y1) and the reference point S31 (u1, v1). Similarly, each term in the third row and the fourth row is a term indicating a correspondence relationship between the corresponding point S12 (x2, y2) and the reference point S32 (u2, v2). Each term in the sixth row is a term indicating a correspondence relationship between the corresponding point S13 (x3, y3) and the reference point S33 (u3, v3), and each term in the seventh and eighth rows is a corresponding item. This is a term indicating the correspondence between the point S14 (x4, y4) and the reference point S34 (u4, v4).

  In the eight simultaneous equations shown by the determinant in FIG. 13, each coordinate value is a known number, so the unknown number is eight of eight coefficients h11 to h32. Therefore, if the operation for solving these eight simultaneous equations is executed, the values of the eight coefficients h11 to h32 can be calculated. The coefficient value calculation unit 160 calculates the coefficient values h11 to h32 by such calculation.

In short, the coefficient value calculation unit 160 arbitrarily sets a scaling factor between the two-dimensional uv coordinate system and the two-dimensional xy coordinate system, so that the coordinate value (u, v), the coordinate value (x, y), and As a coordinate transformation formula that defines the relationship, as shown in formula (5) and formula (6) in FIG.
x = (h11 · u + h12 · v + h13) / (h31 · u + h32 · v + 1)
y = (h21 · u + h22 · v + h23) / (h31 · u + h32 · v + 1)
The two formulas are defined, and the two formulas are coordinate values (u, v) of the two-dimensional uv coordinate system for the four reference points and the coordinate values (x of the two-dimensional xy coordinate system of the four first corresponding points) , Y) are used to calculate eight coefficients h11, h12, h13, h21, h22, h23, h31, h32 using eight simultaneous equations obtained by substituting respectively.

  In the above description, in the three two-dimensional coordinate systems shown in FIG. 6, the coordinate values (x, y) on the first two-dimensional xy coordinate system, the coordinate values (u, v) on the two-dimensional uv coordinate system, Is calculated based on the coordinate values of the four first corresponding points S11 to S14 and the coordinate values of the four reference points S31 to S34, and is calculated by this method. If the coordinate conversion formula “(u, v) = f (x, y)” is used, the first captured image P1 can be converted into the display target image P3.

  On the other hand, the coefficient of the coordinate conversion equation between the coordinate value (x, y) on the second two-dimensional xy coordinate system and the coordinate value (u, v) on the two-dimensional uv coordinate system is calculated in exactly the same manner. It is also possible to calculate based on the coordinate values of the four second corresponding points S21 to S24 and the coordinate values of the four reference points S31 to S34, and the coordinate conversion formula “(u, v ) = G (x, y) ”, the second captured image P2 can be converted into the display target image P3.

  Practically, the coefficient value calculation unit 160 executes the above two methods, and uses the coordinate conversion formula “(u, v) = f (x, y)” as described in §3. Conversion from the captured image P1 to the display target image P3 and conversion from the second captured image P2 to the display target image P3 using the coordinate conversion formula “(u, v) = g (x, y)” It is preferable to be able to do both.

<<< §3. Specific processing in the image conversion unit >>>
Here, specific processing performed by the image conversion unit 170 in the apparatus illustrated in FIG. 1 will be described. As described in §1, the image conversion unit 170, as shown as the equation (12) in FIG. 14, is a coordinate conversion formula “(u, v) defined by the coefficient value calculated by the coefficient value calculation unit 160. = F (x, y) "or" (u, v) = g (x, y) "using the first captured image P1 or the second captured image P2 stored in the captured image storage unit 120. Is converted into the display target image P3. FIG. 15 is a plan view showing this conversion process.

  As illustrated, the first captured image P1 is arranged on the first two-dimensional xy coordinate system defined on the imaging plane π1 of the first camera, and the coordinate conversion formula “(u, v) = f ( x, y) "can be converted into a display target image P3 arranged on the two-dimensional uv coordinate system (on the reference plane R). Similarly, the second captured image P2 is arranged on the second two-dimensional xy coordinate system defined on the imaging plane π2 of the second camera, and the coordinate conversion formula “(u, v) = g (x , Y) ”can be converted into the display target image P3 arranged on the two-dimensional uv coordinate system (on the reference plane R). This is because an arbitrary point p (x, y) on the two-dimensional xy coordinate system can be converted to a point q (u, v) on the two-dimensional uv coordinate system by the coordinate conversion formula. .

  However, since both the captured images P1 and P2 and the display target image P3 are configured by a pixel array in which pixels are arranged at a predetermined pitch, in practice, any one pixel on the captured images P1 and P2 is displayed on the display target. It is not always possible to accurately correspond to one pixel on the image P3. Therefore, in practice, first, a pixel array (pixel array for configuring the display target image P3) is defined on the two-dimensional uv coordinate system at a desired pitch, and pixel values of individual pixels configuring the pixel array are determined. Then, a method of determining by referring to the pixel value of the captured image P1 or P2 on the two-dimensional xy coordinate system is adopted.

  That is, when the image conversion unit 170 converts the conversion target image (captured image P1 or P2) on the two-dimensional xy coordinate system into the display target image P3 on the two-dimensional uv coordinate system, the coordinate value (u, v). Interpolation calculation using the pixel values of a plurality of pixels existing in the vicinity of the position indicated by the coordinate value (x, y) on the image to be converted using a coordinate conversion formula for obtaining the coordinate value (x, y) using as a parameter The interpolation value obtained by (1) may be processed to be the pixel value of the pixel arranged at the position indicated by the coordinate value (u, v).

  In this way, as a method of performing the interpolation calculation for reading out the pixel values of a plurality of pixels existing in the vicinity of the predetermined coordinate value (x, y) and obtaining the interpolation values for these pixel values, for example, the bilinear interpolation method Since various methods such as bicubic spline interpolation are known, detailed description thereof is omitted here. Of course, it is also possible to adopt a method in which the pixel value of the pixel closest to the position indicated by the coordinate value (x, y) is used as it is without performing such interpolation.

After all, in practice, the image conversion processing performed by the image conversion unit 170 uses a coordinate conversion formula in the form of obtaining the coordinate value (x, y) using the coordinate value (u, v) as an argument. Instead of the form of “(u, v) = f (x, y)” or “(u, v) = g (x, y)”, inverse functions f ⁻¹ and g ^{−1 of the} functions f and g are changed. The coordinate arithmetic expression in the form of “(x, y) = f ⁻¹ (u, v)” or “(x, y) = g ⁻¹ (u, v)” is used. The coordinate conversion formulas shown by the formulas (8) and (10) in FIG. 11 are in this format.

  As shown in FIG. 15, the display target image P3 can be obtained by converting the first captured image P1 by the coordinate conversion formula “(u, v) = f (x, y)”. It can also be obtained by converting the second captured image P2 by the coordinate conversion formula “(u, v) = g (x, y)”. Geometrically, there is no difference in the display target image P3 obtained regardless of which conversion method is employed. Therefore, theoretically, the image conversion unit 170 selects one of the first captured image P1 and the second captured image P2 as a conversion target image, and the conversion target image is converted using the coordinate conversion formula. Processing may be performed to obtain the display target image P3.

  However, in practice, each of the captured images P1 and P2 has a resolution based on physical constraints, and these images contain only information on a finite number of pixels. For this reason, actually, the conversion results based on both the captured images P1 and P2 do not exactly match. Therefore, practically, both the first photographed image P1 and the second photographed image P2 are selected as the conversion target images, and the conversion processing using the coordinate conversion formula is performed on both the conversion target images, and the display is performed. It is preferable to perform processing for obtaining the target image P3.

  For this purpose, first, the coefficient value calculation unit 160 selects both the first captured image P1 and the second captured image P2 as conversion target images, and the first and second two images are respectively displayed on the conversion target images. A dimension xy coordinate system is defined, an arbitrary coordinate value (u, v) on the two-dimensional uv coordinate system, and a point on the first and second two-dimensional xy coordinate systems corresponding to the point indicated by the coordinate value The coordinate values of the coordinate transformation equations f and g that define the relationship between the coordinate values (x, y) of the two, the coordinate values of the four reference points S31 to S34, the four first corresponding points S11 to S14, and the four first values. The calculation is performed using the correspondence with the coordinate values of the two corresponding points S21 to S24.

  Then, as shown in FIG. 16, the image conversion unit 170 uses the first coordinate conversion formula (function f) defined by the coefficient value calculated by the coefficient value calculation unit 160, and the first The first captured image P1 on the one two-dimensional xy coordinate system is converted into the first preparation image Pa on the two-dimensional uv coordinate system shown above, and the coefficient value calculated by the coefficient value calculation unit 160 The second photographed image P2 on the second two-dimensional xy coordinate system shown at the lower right of the figure is converted into the two-dimensional uv coordinate shown above using the second coordinate conversion formula (function g) defined by A process of generating the display target image P3 shown in the upper part of the figure by converting the second preparation image Pb on the system and finally combining the first preparation image Pa and the second preparation image Pb. You just have to do it.

As described above, actually, the conversion process from the first captured image P1 to the first preparation image Pa is performed by using coordinate values (u, v) on the pixel array constituting the first preparation image Pa. The coordinate conversion formula “(x, y) = f ⁻¹ (u, v) for the pixel q1 (u, v) arranged at the position where the coordinate value (u, v) is used as an argument. v) ”to obtain the corresponding coordinate value (x, y), and an interpolation operation using pixel values of a plurality of pixels existing in the vicinity of the position indicated by the coordinate value p (x, y). A process of setting the obtained interpolation value as the pixel value of the pixel q1 (u, v) is performed.

Similarly, in the conversion process from the second captured image P2 to the second preparation image Pb, the pixel q2 (u) arranged at the coordinate value (u, v) on the pixel array constituting the second preparation image Pb. , V) by an operation using a coordinate conversion expression “(x, y) = g ⁻¹ (u, v)” in a format for obtaining the coordinate value (x, y) using the coordinate value (u, v) as an argument. A corresponding coordinate value (x, y) is obtained, and an interpolation value obtained by an interpolation operation using pixel values of a plurality of pixels existing in the vicinity of the position indicated by the coordinate value p (x, y) is obtained as a pixel q2. Processing is performed to obtain a pixel value of (u, v).

  In this way, when the first preparation image Pa and the second preparation image Pb are obtained, a process for generating the display target image P3 shown in the upper part of the figure may be performed by combining the two images. Any method may be used for synthesizing both images as long as the features of both images are fused. Generally, a method called α blending may be used. .

Specifically, as shown in FIG. 16, the pixel value of an arbitrary pixel q1 (u, v) at the position indicated by the coordinate value (u, v) on the first preparation image Pa is set to I1 (u, v ), The pixel value of the pixel q2 (u, v) at the position indicated by the same coordinate value (u, v) on the second preparation image Pb is set to I2 (u, v), and the same coordinate value on the display target image P3. When the pixel value of the pixel q (u, v) at the position indicated by (u, v) is I (u, v), a predetermined weight coefficient α (0 ≦ α ≦ 1) is used, and FIG. As shown in equation (13),
I (u, v) = (1−α) · I1 (u, v) + α · I2 (u, v)
And the pixel value I (u, v) of the pixel q (u, v) may be determined.

  Here, the weighting factor α is a parameter that determines which of the first preparation image Pa and the second preparation image Pb is important in image synthesis, and is a value in the range of 0 ≦ α ≦ 1. It becomes. The smaller the weight coefficient α is, the more the composition is made with emphasis on the first preparation image Pa, and the larger the composition is, the emphasis is on the second preparation image Pb. When α = 0 is set, the display target image P3 is equal to the first preparation image Pa, and when α = 1 is set, the display target image P3 is equal to the second preparation image Pb. If α is set to 0.5, an average image of both images is obtained as the display target image P3.

  The value of the weighting factor α may be fixed such that α is always set to 0.5, for example, but more preferably, the uv coordinate plane from which the display target image P3 is obtained and the imaging surface of each camera It is preferable to determine the optimum value each time based on the positional relationship with the (projection plane of the subject).

  For example, in the case of the example shown in the upper part of FIG. 1, the license plate that is the subject of interest of the subject 30 is taken by the two cameras 10 and 20 from an angle of about 45 ° to the left and 45 ° to the right. As shown in FIGS. 2A and 2B, captured images P1 and P2 that are substantially symmetrical are obtained. In such a case, α = 0.5 may be set and the weights of both images may be set equal. However, in the example shown in the upper part of FIG. 1, if the direction of the subject 30 is slightly leftward, the first captured image P1 is an image captured from a direction close to the front of the vehicle, and the second captured image P2 is Since the image is taken from a direction close to the side surface of the vehicle, the taken image P1 is higher than the taken image P2 with respect to the resolution of the license plate to be focused. In such a case, it is preferable to set α <0.5 so as to perform composition with more importance on the captured image P1.

  FIG. 17 is a top view showing an example of a specific method for setting the value of the weighting factor α. Here, the points Q1 and Q2 are photographing points of the two cameras, and the projection planes π1 and π2 are imaging surfaces of the two cameras. The first captured image P1 is formed on the projection plane π1, and the second captured image P2 is formed on the projection plane π2. On the other hand, the uv plane is a coordinate plane defined on the reference plane R on the subject 30, and is a plane on which the display target image P3 is formed.

  Here, a predetermined representative point K (for example, the center point of the first preparation image Pa or the second preparation image Pb) is defined on the uv plane, and as shown in the figure, the representative point K and the first point K are defined. The acute angle θ1 formed between the straight line L1 connecting the shooting point Q1 where the camera 10 is installed and the uv plane, and the acute angle formed between the straight line L2 connecting the representative point K and the shooting point Q2 where the second camera 20 is installed and the uv plane. θ2 is obtained. Then, the angles θ1 and θ2 are parameters indicating the direction in which the representative points K of the respective cameras 10 and 20 are photographed, and the larger the angle, the more the camera is disposed in front of the representative points K. Therefore, it is considered that a more preferable result can be obtained if the angles θ1 and θ2 are compared, and composition is performed with more emphasis on the image captured by a camera having a larger angle (a camera arranged closer to the front).

  Therefore, the angles θ1 and θ2 are calculated based on the coordinate positions in the three-dimensional space of the photographing points Q1, Q2, the representative points K, and the uv plane, and if the weighting coefficient α is θ1> θ2, α <0.5. Α is set to be smaller as the difference between the two angles is larger, α is set to 0.5 when θ1 <θ2, and α is larger as the difference between the two angles is larger. It may be set so as to increase, and if θ1 = θ2, α = 0.5 may be set. Specifically, for example, as shown in the lower part of FIG. 17, the weighting factor α may be set by an arithmetic expression of α = (θ2−θ1) / π + 1/2. According to this arithmetic expression, α = 0 in the extreme case of θ1 = 90 ° and θ2 = 0 °, and α = 1 in the extreme case of θ1 = 0 ° and θ2 = 90 °.

  For convenience of illustration, FIG. 17 shows the arrangement of each element as a two-dimensional plan view. However, in actuality, each element is arranged in a three-dimensional space, and the angles θ1 and θ2 are represented by this cubic. It is given as the angle between a straight line and a plane in the original space.

  As described above, in the method of setting the weighting factor α according to the orientation of the camera with respect to the representative point K, the same value of the weighting factor α is used for the synthesis operation for obtaining the pixel values of all the pixels constituting the display target image P3. However, instead, the value of the weight coefficient α to be used may be changed for each individual pixel.

  For example, in the example shown in FIG. 17, when the value of the common weighting coefficient α is determined based on the representative point K, both the pixel arranged at the point ξ10 on the uv plane and the pixel arranged at the point ξ20 Thus, a composite operation using a common α is performed. However, as shown in the figure, with respect to the arrangement point ξ10, the acute angle θ1 formed by the straight line connecting the two points Q1 and ξ10 and the uv plane is compared with the acute angle θ2 formed by the straight line connecting the two points Q2 and ξ10 and the uv plane. , Θ1> θ2, and therefore, based on the pixel value of the pixel arranged at the projection point ξ11 on the projection plane π1 and the pixel value of the pixel arranged at the projection point ξ12 on the projection plane π2, the point ξ10 When determining the pixel values of the pixels arranged on the top, the former should be combined with greater importance.

  On the other hand, regarding the arrangement point ξ20, when comparing the acute angle θ1 formed by the straight line connecting the two points Q1 and ξ20 and the uv plane with the acute angle θ2 formed by the straight line connecting the two points Q2 and ξ20 and the uv plane, Since θ1 <θ2, on the point ξ20 based on the pixel value of the pixel arranged at the projection point ξ21 on the projection plane π1 and the pixel value of the pixel arranged at the projection point ξ22 on the projection plane π2. When determining the pixel values of the pixels arranged in, the synthesis should be performed with more emphasis on the latter.

  Therefore, if an independent weighting factor α is determined for each pixel and the synthesis operation is performed, the pixel value I (u) of the pixel of interest at the position indicated by the coordinate value (u, v) on the display target image P3. , V), the acute angle θ1 formed by the straight line connecting the arrangement point (for example, ξ10 or ξ20) of the pixel of interest and the photographing point Q1 where the first camera 10 is installed, and the uv plane, and the arrangement The straight line connecting the point and the shooting point Q2 where the second camera 20 is installed is compared with the acute angle θ2 formed by the uv plane, and the weighting coefficient α used when determining the pixel value I (u, v) is θ1. If> θ2, α <0.5 is set, and the larger the difference between the two angles, the smaller α is set. If θ1 <θ2, α> 0.5 is set. The larger the difference is, the larger α is set. If θ1 = θ2, α = 0. It may be set to 5. Also in this case, specifically, as shown in the lower part of FIG. 17, for example, the weighting coefficient α may be set by an arithmetic expression of α = (θ2−θ1) / π + 1/2.

  Here, another factor to be considered when determining the weighting factor α will be described. It is an obstacle that exists in front of the reference plane R that is the subject of interest on the subject 30. FIG. 18 is a plan view illustrating an example in which an obstacle 40 is present in front of the uv plane. Since the display target image P3 is an image of the reference plane R on the subject 30 located behind the obstacle 40, the image of the obstacle 40 must be originally excluded. However, in the illustrated example, a projection image of the obstacle 40 is formed on the projection plane π1, and therefore, in the plan view illustrated, the original object is not located between the projection points ξ31 and ξ41. Thus, an image of the obstacle 40 is obtained. On the other hand, the projection image of the obstacle 40 is not formed on the projection plane π2, and an original subject image is obtained between the projection points ξ32 and ξ42 in the plan view shown in the drawing. Become.

  In such a case, in determining the pixel value of the pixel located between the arrangement points ξ30 and ξ40 on the uv plane in the illustrated plan view, the first captured image P1 on the projection plane π1 should be used. is not. The area corresponding to the position between the subject placement points ξ30 and ξ40 is a hidden surface by the obstacle 40 on the first photographed image P1 obtained on the projection surface π1, whereas it is on the projection surface π2. It is not a hidden surface on the second captured image P2 obtained. Therefore, in this example, when the pixel value I (u, v) of the pixel located in the region corresponding to the arrangement point ξ30 and ξ40 in the display target image P3 formed on the uv plane is determined. Is set to α = 1 and only the second photographed image P2 may be considered.

  FIG. 19 is a plan view showing two captured images to be combined by the combining method shown in FIG. FIG. 19A shows the first captured image P1, and FIG. 19B shows the second captured image P2. In any of the captured images, the license plate that is the original subject 30 and the cat as the obstacle 40 are shown. However, in the first photographed image P1, the obstacle 30 positioned in front of the subject 30 causes the subject 30 to move. A part is a hidden surface, and image information relating to the part is missing. On the other hand, in the second captured image P2, the subject 30 is not hidden by the obstacle 40, and all image information is maintained.

  In such a case, for the area that is a hidden surface in the first captured image P1, the weighting coefficient α used when determining the pixel value I (u, v) is set to α = 1, and the second The pixel value may be determined using only the information of the captured image P2.

  In short, when the image conversion unit 170 determines the pixel value I (u, v) of the pixel of interest at the position indicated by the coordinate value (u, v) on the display target image P3, In the case where the pixel is in a region that is a hidden surface only on the first preparation image Pa due to the object 40, the weighting coefficient α used when determining the pixel value I (u, v) is set to α = 1. On the contrary, when the pixel is in a region hidden by the obstacle 40 only on the second preparation image Pb, the weight used when determining the pixel value I (u, v) The coefficient α may be set to α = 0.

  In FIG. 18, in order to recognize a region that is a hidden surface due to an obstacle on each of the prepared images Pa and Pb formed on the uv plane, it exists before the reference plane in the three-dimensional space. It is necessary to recognize the obstacle 40. The position recognition of the obstacle 40 in the three-dimensional space includes information on the installation positions and orientations of the first camera 10 and the second camera 20, two-dimensional image information on the first captured image P1, A three-dimensional shape model of the subject 30 and the obstacle 40 may be created based on the two-dimensional image information on the second captured image P2. Such a method for creating a three-dimensional shape model is known as a method using epipolar geometry, as already described.

  The specific image conversion processing method in the image conversion unit 170 has been described above for the case where each of the captured images P1 and P2 is a monochrome image. However, if these images are color images, the same processing is performed. This can be done for each primary color. For example, when one pixel has pixel values of three primary colors of RGB, if the above-described processing is performed for each of the pixel value of the primary color R, the pixel value of the primary color G, and the pixel value of the primary color B, color display The target image P3 can be obtained.

<<< §4. Other variations >>
Finally, some modifications of the present invention will be described.

<4-1. Recognition of various subjects>
So far, the example in which the subject of interest on the subject is the vehicle license plate, the surface of the license plate is set to the reference plane R, and the four vertices of the quadrangle constituting the license plate are the four reference points has been described. . However, the present invention is not limited to such a case where the license plate of the vehicle is used as a target of attention, but an object that can define any reference plane R and four reference points located on the reference plane R. Any object can be applied.

  The most typical example of the reference plane R and the four reference points is a quadrangle such as the license plate in the above example and its four vertices. In general vehicles, there are structure parts that can be grasped as a quadrangle, such as a body and a window, in addition to a license plate. It is possible to automatically display an image showing a state observed from the front. Since many industrial products, not limited to vehicles, include a rectangular structure, the present invention can be widely used in fields where such industrial products are used as subjects.

  As described above, when the four vertices of the quadrangle are set as the four reference points, the first corresponding point extracting unit 130 searches for the straight line part on the first captured image P1, and is surrounded by the searched straight line part. The quadrilateral can be recognized as a reference plane, and the four vertices of the quadrilateral can be extracted as the four first corresponding points. When a plurality of rectangles are recognized, it is possible to identify the target object by recognizing the aspect ratio and area of each rectangle.

  The present invention can also be used in a suspicious person monitoring system that automatically displays an image obtained by observing a face of a human face from the front. In general, as shown in the example shown in FIG. 20, the human face may define four feature points: a right eye center point S1, a left eye center point S2, a lip right end point S3, and a lip left end point S4. In addition, a method for recognizing four feature points from a two-dimensional photographed image of a person has already been put to practical use in a face recognition process such as a digital camera as a known technique. Further, since the human face is almost symmetrical, the four feature points are located on the same plane. Therefore, if the four feature points are defined as four reference points on the subject, and a plane including the four reference points is defined as a reference plane R, the present invention can be applied to a human face as the subject. Is possible.

  In this case, the first corresponding point extraction unit 130 searches for a human face from the first captured image P1 using a known face recognition technique, and the right eye center point S1 and the left eye center in the searched face. A surface including the point S2, the right end point S3 of the lips, and the left end point S4 of the lips may be recognized as the reference plane R, and each of these points may be extracted as four first corresponding points S11 to S14. In the three-dimensional space, the surface of the human face is not exactly on the reference plane R, but is almost a plane along the reference plane R. Therefore, the display target image is on the reference plane R. Even if P3 is formed, there is no practical problem.

  If the embodiment described so far is applied to the human face in this way, it includes four feature points: right eye center point S1, left eye center point S2, right end point S3 of the lips, and left end point S4 of the lips. Since the display target image P3 observed from the direction perpendicular to the reference plane R can be obtained, a front image of the person's face can be displayed after all. This is very convenient for use such as identification of a suspicious person. In general, the orientation in the human space, particularly the orientation of the face, has a very high degree of freedom compared to the orientation of the vehicle, and the face taken in various directions is reflected in the photographed image of the person by the camera. According to the present invention, no matter which direction the person is facing, the front image of the person is automatically displayed if the faces are reflected on the two cameras. Therefore, the present invention is also suitable for use in a suspicious person monitoring system.

  Each of the embodiments described so far is an example in which the reference plane R is defined based on the unique characteristics of a specific subject. For example, a license plate has a unique feature of “a plate plate having a rectangular outer shape”, and four reference points (four vertices of a rectangle) that constitute a reference plane are utilized by using the unique feature. ) Could be defined in advance. Similarly, a human face has a unique feature of “both eyes and lips”, and using the unique feature, the four reference points (the center point and both eyes of the eyes) that constitute the reference plane are used. (Lip edge) could be defined in advance.

  However, in practicing the present invention, it is not always necessary to define the reference plane R based on the unique characteristics of such a specific subject, and simply “the plane recognized on the subject is defined as the reference plane R. It is also possible to perform an operation of “using any four points on the plane as reference points”. That is, as already described, if epipolar geometry is used, information on the installation positions and orientations of the first camera 10 and the second camera 20, the two-dimensional subject information on the first captured image P1, and the second A three-dimensional shape model of the subject 30 can be created based on the two-dimensional subject information on the captured image P2. Therefore, the three-dimensional shape model is created in the first corresponding point extraction unit 130, a predetermined plane on the three-dimensional shape model is recognized as the reference plane R, and the arbitrary four points on the reference plane R are set. If a function for extracting the corresponding four points as the first corresponding points is provided, any subject can have a plane portion, and the plane portion can be focused on from the front. An observed image can be displayed.

<4-2. Observation from an oblique direction>
In all the embodiments described so far, it has been assumed that an image obtained by observing the target object on the subject from the front direction is displayed. For this reason, in the model shown in FIG. 9, a two-dimensional uv coordinate system is defined on the reference plane R that is the subject of interest of the subject 30, and calculation is performed to obtain a display target image on the coordinate system. In general, in the case of a vehicle license plate or a human face, it is preferable to display an image obtained by observing the license plate from the front. That is, in applications such as vehicle number identification or person identification, it is appropriate to display the state of the license plate or the person's face observed from the front, and the front direction, that is, the direction orthogonal to the reference plane R is appropriate. It will be a different observation direction.

  However, depending on the application, the direction orthogonal to the reference plane R is not always an appropriate observation direction. For example, even if the subject is the same vehicle, for the purpose of confirming the vehicle type, it is better to display an image observed from diagonally forward or diagonally behind than an image observed from the front of the vehicle. Easy to understand the characteristics of the entire vehicle. Therefore, for the purpose of vehicle type confirmation, the appropriate observation direction is diagonally forward or diagonally backward. In this case, if the surface of the license plate is set as the reference plane R, an appropriate observation direction is not a direction orthogonal to the reference plane R, for example, a direction intersecting with the reference plane R at an angle of 45 °. It turns out that.

  Therefore, as in the above example, if the oblique direction with respect to the reference plane R is set as an appropriate observation direction, the model shown in FIG. 21 may be used instead of the model shown in FIG. In the model shown in FIG. 21, a uv coordinate system is defined on a plane obtained by rotating a reference plane R (for example, the surface of a license plate) by a predetermined rotation angle γ (for example, γ = 45 °) around the v axis. ing. Of course, the rotation axis is not necessarily the v-axis, and any axis may be set as the rotation axis.

  As described above, when the two-dimensional uv coordinate system is rotated and defined, the obtained display target image P3 is an image obtained by projecting the three-dimensional image of the subject onto the uv plane. Such a projection image can be obtained by creating a three-dimensional shape model of the subject. In this case, the coordinate value (u, v) on the display target image P3 formed on the two-dimensional uv coordinate system and the two-dimensional xy coordinate system corresponding to the point on the display target image P3 indicated by the coordinate value. The coordinate conversion formula that defines the relationship between the coordinate values (x, y) of the points in FIG. 4 is slightly different from the formulas described in the previous embodiments. However, as shown in FIG. 21, the relationship between the two-dimensional coordinate system defined on the reference plane R and the two-dimensional uv coordinate system obtained by rotating the two-dimensional coordinate system by the rotation angle γ can be determined geometrically. The coordinate conversion formula is also obtained by slightly modifying the coordinate conversion formula described so far. The coefficient value calculation unit 160 may calculate each coefficient value of the coordinate conversion formula after such correction.

<4-3. Adoption of fisheye camera>
The embodiment described so far is an example in which a general planar digital camera is used as the first camera 10 and the second camera 20, but in order to ensure a wider field of view with one camera. Can use a fish-eye camera equipped with a fish-eye lens (a camera equipped with an omnidirectional mirror instead of a fish-eye lens is also referred to as a fish-eye camera here).

  When a fisheye lens is used, a circular image showing all directions of a hemisphere can be obtained without a mechanical operation mechanism, and a considerably wide field of view can be covered with one camera. However, the image captured by the fisheye camera is different from the image captured by the normal flat camera, and becomes a distorted circular image. Therefore, when using this, a part of the distorted circular image is cut out and the normal planar regularity is obtained. Conversion processing to convert to an image is required.

  FIG. 22 is a perspective view for explaining the general characteristics of a distorted circular image obtained by photographing using a fisheye lens and the basic principle of processing for cutting out a part of the image and converting it into a planar regular image. A basic model in which a distorted circular image S is formed by photographing using a projection type fisheye lens is shown. In general, fish-eye lenses are classified into a plurality of types depending on the projection method. The model shown in FIG. 22 is for a fish-eye lens of an orthogonal projection method.

  Here, an example in which a distorted circular image S is formed on an XY plane in a three-dimensional XYZ orthogonal coordinate system is shown. As shown in the figure, the Z axis is taken upward in the figure, and the dome is located on the positive region side of the Z axis. A virtual phantom spherical surface H (hemisphere) is defined. A two-dimensional uv orthogonal coordinate system is arranged on the upper left of the phantom spherical surface H. The three-dimensional XYZ orthogonal coordinate system and the two-dimensional uv orthogonal coordinate system shown here are for explaining coordinate conversion for converting a distorted circular image S obtained by photographing with a fisheye camera into a planar regular image T. The coordinate system is another coordinate system that is completely unrelated to the three-dimensional XYZ coordinate system and the two-dimensional uv coordinate system shown in the model shown in FIG.

  The distorted circular image S formed on the XY plane is an image forming a circle with a radius R centered on the origin O of the coordinate system, and is an area having an angle of view of 180 ° on the positive area side of the Z axis. Is equivalent to the image recorded with distortion. In the distorted circular image S, all the images existing on the positive region side of the Z axis are recorded, but the scale portion of the image is different between the central portion and the peripheral portion, and the recorded images are recorded. The shape of the image is distorted.

  An actual fisheye lens is constituted by an optical system that combines a plurality of convex lenses and concave lenses, and it is known that its optical characteristics can be modeled by a virtual spherical surface H as shown in FIG. That is, considering a model in which a dome-shaped virtual spherical surface H (hemisphere) having a radius R is arranged on the upper surface of the distorted circular image S, the optical characteristics of the orthographic fish-eye lens are arbitrary on the virtual spherical surface H. A light ray L (a one-dot chain line in the figure) incident from the normal direction to the point Q (x, y, z) behaves as a light ray parallel to the Z axis toward the point S (x, y) on the XY plane. You can think of it. In other words, in FIG. 22, the pixel located at the point S (x, y) on the distorted circular image S indicates one point on the object existing on the extension line of the incident light beam L shown in FIG. 22. It will be.

  Of course, an optical phenomenon occurring in an actual fisheye lens is that a specific point of an object to be imaged is connected to a specific point S (x, y) on the XY plane due to refraction by a plurality of convex lenses and concave lenses. Although this is an image phenomenon, there is no problem in performing an image conversion process or the like even if the discussion is replaced with a model using a virtual spherical surface H as shown in FIG. The purpose of the image conversion performed here is to perform a process of cutting out a part on the distorted circular image S and converting it into a planar regular image T.

Here, for example, let's consider converting an image in a neighboring area on the distorted circular image S into a planar regular image with the point P (x ₀ , y ₀ ) shown in the figure as a cut center point. Now, as shown in the figure, the intersection of a straight line parallel to the Z axis and the phantom spherical surface H passing through the cut center point P (x ₀ , y ₀ ) is defined as Q (x ₀ , y ₀ , z ₀ ). This intersection point Q is, so to speak, a point directly above the cut-out center point P (x ₀ , y ₀ ), and is referred to as a corresponding point on the sphere. Similarly, the point Q (x, y, z) is the corresponding point on the sphere for the point S (x, y).

Subsequently, a line-of-sight vector n is defined as shown in the direction connecting the origin O and the corresponding point Q (x ₀ , y ₀ , z ₀ ) on the spherical surface, and a two-dimensional uv coordinate system is formed on the line-of-sight vector n. Define the origin G of. Here, the coordinates of the origin G are (xg, yg, zg), and the distance between the two points OG is m · r. r is the radius of the distorted circular image S (the radius of the phantom spherical surface H), and m is a magnification parameter indicating how many times the radius r is the distance between the origin G and the origin O. The two-dimensional uv coordinate system is defined on a plane that passes through the origin G and is perpendicular to the line-of-sight vector n. As shown in the figure, the two-dimensional uv coordinate system has a u-axis and a v-axis that are orthogonal to each other, but the angle formed between the rotation reference axis J and the u-axis is φ. Here, the rotation reference axis J is defined as an axis that passes through the origin G, is parallel to the XY plane, and is orthogonal to the line-of-sight vector n.

When the angle φ is increased or decreased, the two-dimensional uv coordinate system rotates about the line-of-sight vector n as the central axis, and the angle φ is generally called a plane tilt angle. Further, the corresponding point Q (x ₀ , y ₀ , z ₀ ) on the spherical surface is a point that determines the direction of the line-of-sight vector n, and can be specified by the azimuth angle α and the zenith angle β as shown in the figure. Here, the azimuth angle α (0 ≦ α <360 °) is an angle formed by a straight line connecting the cut-out center point P (x ₀ , y ₀ ) and the origin O of the XY coordinate system and the Y axis, and the zenith The angle β (0 ≦ β ≦ 90 °) is an angle (acute angle) formed by the line-of-sight vector n and the Z axis.

Eventually, the position and orientation of the two-dimensional uv coordinate system are uniquely determined by determining the azimuth angle α, zenith angle β, plane tilt angle φ, and magnification parameter m. The conversion process performed here converts a partial image in the vicinity of the cut-out center point P (x ₀ , y ₀ ) of the distorted circular image S formed on the XY plane into a planar regular image T on the uv plane. It is to be. For this purpose, a coordinate conversion formula is necessary that indicates the relationship between an arbitrary point S (x, y) on the distorted circular image S and a corresponding point T (u, v) on the planar regular image T. .

  It is known that such coordinate conversion formulas are formulas (14) and (15) shown in FIG. In these equations, r is the radius of the distorted circular image S (the radius of the virtual spherical surface H), and is a value specific to the fisheye lens used for imaging. A to F are coefficients obtained based on the azimuth angle α, the zenith angle β, and the plane inclination angle φ as shown in the equations (16) to (21), and are generally called rotation coefficients. Then, w is a value given by w = m · r as shown in Expression (22), and is a value corresponding to the distance between the origin O and the origin G shown in FIG. When m is increased or decreased, the uv coordinate system is translated in the direction along the line-of-sight vector n, and the value of m becomes a scaling factor that determines the magnification of the obtained planar regular image T.

Eventually, when a fisheye camera equipped with a fisheye lens or an omnidirectional mirror is used instead of using a general flat digital camera when implementing the present invention, a predetermined circular image S in the distorted circular image S obtained by photographing is used. An image in the vicinity of the cut-out center point P (x ₀ , y ₀ ) using a coordinate transformation formula shown in FIG. 23 (this formula is a transformation formula in the case of the orthographic projection method, In this case, it is only necessary to first convert to the orthographic projection method and then use this conversion formula) to convert the planar regular image T into the first regular captured image in the embodiments described so far. What is necessary is just to utilize as P1 and the 2nd picked-up image P2.

  FIG. 24 is a block diagram showing a configuration of a captured image display apparatus according to an embodiment using such a fisheye camera. The difference from the apparatus shown in FIG. 1 is that a first fisheye camera 15 and a second fisheye camera 25 are used as two cameras, and a fisheye image conversion unit 200 is newly provided. It is.

The fish-eye image conversion unit 200 has a predetermined cut-out center point P (x ₀ , y ₀ ) in the distorted circular image S obtained by photographing with the first fish-eye camera 15 and the second fish-eye camera 25. A process of converting a nearby image into a planar regular image T using a coordinate conversion formula shown in FIG. The image input unit 110 inputs the planar regular image T thus converted by the fisheye image conversion unit 200 as the first captured image P1 and the second captured image P2, respectively.

<Use of 4-4.3 or more cameras>
The embodiments described so far are examples in which the captured image display device is configured using only two cameras, but the device according to the present invention can also be configured using three or more cameras. is there. That is, a plurality of M cameras (M ≧ 3) that are installed at different positions and are directed in the direction in which the same subject can be photographed are prepared, and any two of these M cameras can be combined. The first camera 10 and the second camera 20 in the embodiments described so far may be adopted to have a function of creating a display target image.

  FIG. 25 is a perspective view showing the arrangement of the cameras in the embodiment using four cameras. In this example, four support posts 61 to 64 are arranged around a circular photographing target area 50 into which a vehicle as the subject 35 enters, and fisheye cameras 71 to 74 are provided on the support posts 61 to 64, respectively. . These four fisheye cameras 71 to 74 are all directed in the direction in which the same subject 35 can be photographed.

  In the case of this example, for each of the distorted circular images obtained by photographing with the four fisheye cameras 71 to 74, a position corresponding to a predetermined cut-out center point (for example, the center point of the circular photographing target area 50). ) And a plane regular image is created by the method described above. Then, a pair of adjacent cameras is set as one group, and the processing of the embodiment described so far is executed to create a display target image.

  Specifically, the cameras 71 and 72 are the first group, the cameras 72 and 73 are the second group, the cameras 73 and 74 are the third group, and the cameras 74 and 71 are the fourth group. The display target image is created by the method described above. As a result, a total of four display target images are obtained. Therefore, all four display target images may be displayed on the monitor screen by dividing the screen, or the screens may be displayed in time division. You may make it display while switching.

  Alternatively, one or several of the four display images may be selected and displayed. For example, when it is desired to automatically display an image of the license plate observed from the front with the license plate portion of the vehicle as a target of interest, in some of the above groups, license plate recognition (four correspondences) (Point extraction) may fail. As described above, when the recognition fails, the display of the group is stopped, and only the result of the group that is successfully recognized may be selected and displayed. Alternatively, by extracting four corresponding points for each group and recognizing the quadrangle constituting the license plate, the group having the largest area of the quadrangle is selected and only the result of the group is displayed. It is also possible.

  FIG. 25 shows an example in which four fisheye cameras are used. Of course, the number of cameras is not limited to four, and any number can be installed as necessary. Any two of these cameras can be combined to form a group. In the above example, a fish-eye camera is used. Of course, the same processing as in the above example can be performed when a plurality of normal planar cameras are arranged.

10: 1st camera 15: 1st fisheye camera 20: 2nd camera 25: 2nd fisheye camera 30, 35: Subject 40: Obstacle 50: Shooting target area 61-64: Posts 71-74 : Fisheye camera 110: image input unit 120: captured image storage unit 130: first corresponding point extraction unit 140: second corresponding point extraction unit 150: reference point coordinate calculation unit 160: coefficient value calculation unit 170: image conversion unit 180 : Display target image storage unit 190: Image output unit 200: Fisheye image converters A to F: Rotation coefficient B: Baseline E: Virtual viewpoint E1, E2: Epipolar line EP: Epipolar plane e1, e2: Epipole:
f: function G (xg, yg, zg) for converting coordinates (x, y) to coordinates (u, v): origin of two-dimensional uv coordinate system g: coordinates (x, y) to coordinates (u, v) Function H to convert to: Homography matrix / virtual sphere h11 to h33: coefficients (elements of homography matrix)
I (u, v): Pixel value of pixel q (u, v) I1 (u, v): Pixel value of pixel q1 (u, v) I2 (u, v): Pixel of pixel q2 (u, v) Value J: rotation reference axis K: representative point on uv plane L: incident ray L1: straight line connecting two points (K, Q1) L2: straight line connecting two points (K, Q2) m: magnification parameter n: line-of-sight vector O: Origin of the three-dimensional XYZ coordinate system O ′: Origin of the three-dimensional UVW coordinate system P1: First photographed image P2: Second photographed image P3: Display target image P (x ₀ , y ₀ ): Center of extraction Point p: One point on the xy plane Q1: First photographing point (viewpoint)
Q2: Second shooting point (viewpoint)
Q (x ₀ , y ₀ , z ₀ ): corresponding point on spherical surface Q (x, y, z): corresponding point on spherical surface q: one point q (u, v) on uv plane: on display target image P3 Pixel q1 (u, v): Pixel q2 (u, v) on the first preparation image Pa: Pixel R on the second preparation image Pb: Reference plane r: Radius of the distorted circular image S (of the phantom spherical surface H radius)
S: Distorted circular images S1 to S4: Face reference points S11 to S14: First corresponding points S21 to S24: Second corresponding points S31 to S34: Reference points S (x, y) on the subject: Distorted circular image S Point T: Planar regular image T (u, v): Point in the planar regular image T: Coordinate axis of the three-dimensional UVW coordinate system u: Coordinate axis / coordinate values u1 to u4 of the two-dimensional uv coordinate system: Four reference points u coordinate value V: coordinate axis of 3D UVW coordinate system v: coordinate axis of 2D uv coordinate system / coordinate values v1 to v4: v coordinate value of 4 reference points W: coordinate axis of 3D UVW coordinate system w: on W axis Coordinate value X: coordinate axis of three-dimensional XYZ coordinate system x: coordinate axis of two-dimensional xy coordinate system / coordinate values x1 to x4: x coordinate value of four first corresponding points Y: coordinate axis y of three-dimensional XYZ coordinate system Coordinate axes of two-dimensional xy coordinate system / coordinate values y1 to y4: y positions of four first corresponding points Value Z: coordinate axis z, za, zb of the three-dimensional XYZ coordinate system: coordinate value α on the Z axis α: weighting factor / azimuth angle β: zenith angle γ: rotation angle θ11 to θ22: angle ξ: feature point ξ1 on the subject : Projection point on projection plane (imaging plane) π 1: Projection point on projection plane (imaging plane) π 2 ξ 10, ξ 20, ξ 30, ξ 40: Arrangement points of target pixel on uv plane ξ41: projection point on the first projection plane π1, ξ12, ξ22, ξ32, ξ42: projection point on the second projection plane π2 ξa: projection point on the projection plane πa ξb: projection point on the projection plane πb π, πa, πb: projection plane parallel to the XY plane π1: first projection plane / first camera imaging plane π2: second projection plane / second camera imaging plane σ: scaling coefficient φ: plane Angle of inclination

Claims (22)

A photographed image display device that generates and displays an image obtained by observing a subject from a predetermined virtual direction based on a photographed image of the subject photographed from different directions,
A first camera and a second camera installed at different positions and oriented in a direction in which the same subject can be photographed;
A digital image obtained by photographing with the first camera or an image obtained by applying a predetermined conversion process to the digital image is input as a first photographed image, and a digital image obtained by photographing with the second camera or An image input unit that inputs an image obtained by performing a predetermined conversion process on the digital image as a second captured image;
A captured image storage unit for storing the input first captured image and the second captured image;
By analyzing the first photographed image, four points on the first photographed image corresponding to four reference points preset as points on the reference plane as a target of interest on the subject are displayed. A first corresponding point extracting unit that extracts the first corresponding points;
By comparing the feature of the first photographed image with the feature of the second photographed image, a correspondence relationship between each point on both images is obtained, and the four second images are obtained on the second photographed image. A second corresponding point extracting unit that extracts four second corresponding points corresponding to one corresponding point;
Each corresponding point based on information on the installation position and orientation of the first camera and the second camera, and the two-dimensional coordinate values of the four first corresponding points and the four second corresponding points A reference point coordinate calculation unit for obtaining three-dimensional coordinate values of four reference points on the subject corresponding to
A two-dimensional uv coordinate system is defined on a reference plane including the four reference points or a plane obtained by rotating the reference plane in a predetermined direction by a predetermined rotation angle, and the first captured image and When at least one of the second captured images is selected as a conversion target image and a two-dimensional xy coordinate system is defined on the conversion target image, arbitrary coordinate values (u, v) on the two-dimensional uv coordinate system are defined. ) And the coordinate value (x, y) of the point on the two-dimensional xy coordinate system corresponding to the point indicated by the coordinate value, the coefficient value of the coordinate transformation formula that defines the relationship between the four reference points A coefficient value calculation unit that calculates a coordinate value and a correspondence relationship between the coordinate values of the four first corresponding points or the four second corresponding points;
An image for converting the conversion target image on the two-dimensional xy coordinate system into a display target image on the two-dimensional uv coordinate system using a coordinate conversion formula defined by the coefficient value calculated by the coefficient value calculation unit. A conversion unit;
A display target image storage unit for storing the display target image converted by the image conversion unit;
An image output unit for displaying the display target image on a monitor screen;
A photographed image display device comprising: The captured image display device according to claim 1,
The first corresponding point extraction unit searches for a straight line portion on the first captured image, recognizes a quadrangle surrounded by the searched straight line portion as a reference plane, and sets four vertices of the quadrangle as four first corresponding points. A photographed image display device characterized in that it is extracted as: The captured image display device according to claim 1,
The first corresponding point extraction unit searches the human face from the first photographed image, and uses the surface including the right eye center point, left eye center point, lip right end point, and lip left end point in the searched face as a reference. A captured image display device that recognizes a plane and extracts each of the points as four first corresponding points. The captured image display device according to claim 1,
The first corresponding point extraction unit includes information on installation positions and orientations of the first camera and the second camera, two-dimensional subject information on the first photographed image, and two-dimensional on the second photographed image. Based on the subject information, a three-dimensional shape model of the subject is created, a predetermined plane on the three-dimensional shape model is recognized as a reference plane, and four corresponding to any four points on the reference plane A captured image display device, wherein a point is extracted as a first corresponding point. In the captured image display device according to any one of claims 1 to 4,
The reference point coordinate calculation unit has a function of storing the installation position and orientation information of the first camera and the second camera by setting operation from the outside, and using the stored information, A photographed image display device characterized by obtaining a three-dimensional coordinate value. In the captured image display device according to any one of claims 1 to 4,
Based on the first captured image and the second captured image in which the reference point coordinate calculation unit captures the test subject stored in the captured image storage unit, the installation positions of the first camera and the second camera And a captured image display device having a function of automatically acquiring orientation information. In the captured image display device according to any one of claims 1 to 6,
The coefficient value calculation unit arbitrarily sets a scaling factor between the two-dimensional uv coordinate system and the two-dimensional xy coordinate system, so that the relationship between the coordinate value (u, v) and the coordinate value (x, y) is obtained. As a coordinate transformation formula to be defined,
x = (h11 · u + h12 · v + h13) / (h31 · u + h32 · v + 1)
y = (h21 · u + h22 · v + h23) / (h31 · u + h32 · v + 1)
The two formulas are defined, and the two formulas are coordinate values (u, v) of the two-dimensional uv coordinate system for the four reference points and the coordinate values (x of the two-dimensional xy coordinate system of the four first corresponding points) , Y) and eight simultaneous equations obtained by substituting for each of them, eight coefficients h11, h12, h13, h21, h22, h23, h31, h32 are calculated. apparatus. In the captured image display device according to any one of claims 1 to 7,
The coefficient value calculation unit selects both the first photographed image and the second photographed image as conversion target images, defines first and second two-dimensional xy coordinate systems on these conversion target images, respectively, Arbitrary coordinate values (u, v) on the dimension uv coordinate system and the coordinate values (x, y) of the points on the first and second two-dimensional xy coordinate systems corresponding to the points indicated by the coordinate values And a coefficient value of a coordinate transformation formula that defines the relationship between the four reference points and the correspondence values of the four first corresponding points and the four second corresponding points.
An image conversion unit converts the first captured image on the first two-dimensional xy coordinate system into the second image using a first coordinate conversion formula defined by the coefficient value calculated by the coefficient value calculation unit. The second two-dimensional xy coordinate system is converted into a first preparation image on a dimensional uv coordinate system and a second coordinate conversion formula defined by the coefficient value calculated by the coefficient value calculation unit is used. The second photographed image above is converted into a second preparation image on the two-dimensional uv coordinate system, and the first preparation image and the second preparation image are combined to form a display target image. A photographic image display device characterized by generating. The captured image display device according to claim 8,
When the image conversion unit combines the first preparation image and the second preparation image, the pixel value of the pixel at the position indicated by the coordinate value (u, v) on the first preparation image is set to I1 ( u, v), the pixel value of the pixel at the position indicated by the coordinate value (u, v) on the second preparation image is I2 (u, v), and the coordinate value (u, v) on the display target image. When the pixel value of the pixel at the indicated position is I (u, v), using a predetermined weight coefficient α (0 ≦ α ≦ 1),
I (u, v) = (1−α) · I1 (u, v) + α · I2 (u, v)
A photographed image display device characterized by performing a composition operation. The captured image display device according to claim 9,
An acute angle θ1 formed by a straight line connecting a predetermined representative point on the uv plane and a shooting point where the first camera is installed and a uv plane, and a straight line connecting the representative point and a shooting point where the second camera is installed Compare the acute angle θ2 made with the uv plane,
If θ1> θ2, α <0.5 is set, and the larger the difference between the two angles, the smaller α is set.
If θ1 <θ2, α is set to 0.5, and the larger the difference between the two angles, the larger α is set.
If θ1 = θ2, α = 0.5 is set as a captured image display device. The captured image display device according to claim 9,
When determining the pixel value I (u, v) of the pixel of interest at the position indicated by the coordinate value (u, v) on the display target image, shooting with the arrangement point of the pixel of interest and the first camera installed An acute angle θ1 formed by a straight line connecting a point and the uv plane is compared with an acute angle θ2 formed by a straight line connecting the arrangement point and the photographing point where the second camera is installed and the uv plane, and the pixel value I ( The weighting factor α used in determining u, v) is
If θ1> θ2, α <0.5 is set, and the larger the difference between the two angles, the smaller α is set.
If θ1 <θ2, α is set to 0.5, and the larger the difference between the two angles, the larger α is set.
If θ1 = θ2, α = 0.5 is set as a captured image display device. The captured image display device according to claim 10 or 11,
α = (θ2−θ1) / π + 1/2
A photographed image display device characterized in that a weighting factor α is set by the following equation. The captured image display device according to claim 9,
The image converter
Based on the installation position and orientation information of the first camera and the second camera, the two-dimensional image information on the first photographed image, and the two-dimensional image information on the second photographed image, the subject and Create a 3D shape model of the obstacle, perform the process of recognizing the obstacle existing in front of the reference plane,
When determining the pixel value I (u, v) of the pixel of interest at the position indicated by the coordinate value (u, v) on the display target image, the pixel of interest is moved onto the first preparation image by the obstacle. If the pixel is in a region that is a hidden surface only, the weighting coefficient α used when determining the pixel value I (u, v) is set to α = 1, and the second is caused by the obstacle. If the pixel is in a region that is a hidden surface only on the prepared image, the weighting coefficient α used when determining the pixel value I (u, v) is set to α = 0. A captured image display device. In the picked-up image display device according to any one of claims 1 to 13,
When the image conversion unit converts a conversion target image on the two-dimensional xy coordinate system into a display target image or a preparation image on the two-dimensional uv coordinate system, the coordinate value (x, x) is used as an argument. Using the coordinate conversion equation for obtaining y), an interpolation value obtained by an interpolation operation using pixel values of a plurality of pixels existing in the vicinity of the position indicated by the coordinate value (x, y) on the image to be converted is A captured image display device characterized in that a pixel value of a pixel arranged at a position indicated by the coordinate value (u, v) is used. In the picked-up image display apparatus in any one of Claims 1-14,
The coefficient value calculation unit defines a two-dimensional uv coordinate system on a plane obtained by rotating a reference plane around a predetermined axis by a predetermined rotation angle γ. Image display device. In the photographic image display device according to any one of claims 1 to 15,
Provided with a plurality of M cameras (M ≧ 3) installed in different positions and directed in the direction in which the same subject can be photographed,
A photographed image display device having a function of adopting a combination of any two of these M cameras as a first camera and a second camera and creating a display target image. In the picked-up image display apparatus in any one of Claims 1-16,
Using a camera equipped with a fisheye lens or an omnidirectional mirror as the first camera and the second camera,
Fish-eye image conversion for converting an image in the vicinity of a predetermined cut-out center point in a distorted circular image obtained by photographing with the first camera and the second camera into a planar regular image using a predetermined coordinate conversion formula Further comprising
A photographic image display device, wherein the image input unit inputs the planar regular image converted by the fisheye image conversion unit as a first photographic image and a second photographic image, respectively.   The program for functioning a computer as components other than each camera in the picked-up image display apparatus in any one of Claims 1-17.   A semiconductor integrated circuit that functions as a component other than each camera in the captured image display device according to claim 1. A captured image display device according to claim 2,
Each camera is installed in a position and orientation where the license plate of the vehicle that is the subject can be photographed,
The first corresponding point extracting unit extracts the four vertexes of the quadrangle constituting the license plate as four first corresponding points, with the surface of the license plate as a reference plane to be focused on,
An imaging output device for a license plate of a vehicle, wherein the image output unit outputs an image obtained by observing the license plate from the front direction as a display target image. Including the captured image display device according to claim 3;
Each camera is installed in a position and orientation that can shoot the human face as the subject,
An image output unit outputs an image obtained by observing the human face from the front direction as a display target image. A captured image display method for generating and displaying an image obtained by observing a subject from a predetermined virtual direction based on a photographed image of the subject photographed from different directions,
A camera installation stage in which the first camera and the second camera are installed at different positions and in the direction in which the same subject can be photographed;
A computer inputs a digital image obtained by photographing with the first camera or an image obtained by applying a predetermined conversion process to the digital image as a first photographed image, and is obtained by photographing with the second camera. An image input step of inputting a digital image or an image obtained by applying a predetermined conversion process to the digital image as a second captured image;
When the computer analyzes the first captured image, points corresponding to four reference points set in advance as points on the reference plane as a target of interest on the subject are determined as the first captured image. A first corresponding point extraction stage for extracting as four first corresponding points above;
The computer obtains the correspondence between each point on both images by comparing the feature of the first photographed image and the feature of the second photographed image, and on the second photographed image, A second corresponding point extracting step of extracting four second corresponding points respectively corresponding to the four first corresponding points;
The computer, based on the information on the installation position and orientation of the first camera and the second camera, and the two-dimensional coordinate values of the four first corresponding points and the four second corresponding points, A reference point coordinate calculation step for obtaining three-dimensional coordinate values of four reference points on the subject corresponding to each corresponding point;
The computer defines a two-dimensional uv coordinate system on a reference plane including the four reference points or on a plane obtained by rotating the reference plane by a predetermined rotation angle in a predetermined direction. When at least one of the photographed image and the second photographed image is selected as a conversion target image and a two-dimensional xy coordinate system is defined on the conversion target image, an arbitrary coordinate value on the two-dimensional uv coordinate system ( u, v) and the coordinate value (x, y) of the point on the two-dimensional xy coordinate system corresponding to the point indicated by the coordinate value, the coefficient value of the coordinate transformation formula that defines the relationship between the four A coefficient value calculating step for calculating using a correspondence relationship between the coordinate value of the reference point and the coordinate values of the four first corresponding points or the four second corresponding points;
The computer converts the conversion target image on the two-dimensional xy coordinate system into a display target image on the two-dimensional uv coordinate system using a coordinate conversion formula defined by the coefficient value calculated in the coefficient value calculation step. An image conversion stage to convert,
An image output step in which the computer displays the display target image converted in the image conversion step on a monitor screen;
A captured image display method characterized by comprising:

Images