JP2011101265A - Method and device for calculation of calibration information, and wide angle image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and device for calculation of calibration information, along with a wide angle image processing apparatus, capable of correcting distortion of a wide angle image with high accuracy. <P>SOLUTION: The wide angle image processing apparatus 1 captures a calibration image with a plurality of calibration patterns displayed by a calibration image display unit 120 by a wide angle imaging unit 110. Then, a plurality of lattice points shown by the calibration patterns on the picked up wide angle image are detected, and positions on a three-dimensional space are calculated using the principle of a space encoding method for the plurality of detected lattice points. Then, calibration information for correcting the distortion of the wide angle image is calculated from geometric relation between the positions of the plurality of lattice points on the wide angle image and the calculated positions of the plurality of lattice points on the three-dimensional space. In addition, distortion correction of the wide angle image picked up by the wide angle imaging unit 110 is performed on the basis of the calculated calibration information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention provides a calibration information calculation method for calculating calibration information for calibrating distortion of a wide-angle image captured by a wide-angle imaging device, a calibration information calculation device, and a wide-angle image based on the calibration information calculated by the method. The present invention relates to a wide-angle image processing apparatus to be corrected.

  A conventional wide-angle image display device includes a wide-angle imaging device that acquires a wide-angle image, an image processing device that corrects wide-angle distortion of the acquired wide-angle image, and a display device that displays the corrected wide-angle image. There is something. As a method for correcting wide-angle distortion by this image processing apparatus, a pattern composed of a plurality of markers arranged at equal angular intervals at the elevation angle of the wide-angle imaging apparatus is imaged, and the pattern is positioned at the marker position in the captured image. A correction condition is acquired based on this, and an annular wide-angle image is developed into a panoramic image based on the correction condition (see, for example, Patent Document 1).

JP 2003-308526 A

  However, in the actual product of the wide-angle lens used in the wide-angle imaging device as described in Patent Document 1, the light beam incident on the wide-angle lens is 1 with the lens optical axis as assumed in the conventional correction method. Regardless of the point, the point of intersection with the lens optical axis changes with respect to the incident angle to the lens. Therefore, the conventional wide-angle image display device has a problem that the wide-angle distortion of the wide-angle image acquired by the wide-angle imaging device is not correctly corrected.

  The present invention has been made in view of such problems, and includes a calibration information calculation method, a calibration information calculation device, and calibration information calculated by the method capable of correcting distortion of a wide-angle image with high accuracy. An object of the present invention is to provide a wide-angle image processing apparatus that corrects a wide-angle image based on the image.

In order to achieve the above object, a calibration information calculation method according to the first aspect of the present invention includes:
A calibration information calculation method for calculating calibration information for correcting distortion of a wide-angle image captured by a wide-angle imaging unit,
A calibration image for calculating a plurality of grid points arranged at predetermined intervals is displayed on a display unit surrounding the wide-angle imaging unit, and a wide-angle image obtained by capturing the calibration image by the wide-angle imaging unit is acquired. Based on the acquired wide-angle image and the calibration image displayed on the display unit, the storage unit associates the positions of the plurality of lattice points on the three-dimensional space and the positions on the wide-angle image. A grid point detection step stored in
An optical axis detection step of extracting a plurality of straight lines based on a plurality of lattice points detected by the lattice point detection step, and detecting an optical axis of the wide-angle imaging unit based on an intersection of the plurality of straight lines;
Based on the optical axis detected by the optical axis detection step, the position on the three-dimensional space of the plurality of lattice points stored in the storage means, and the position on the wide-angle image, A calibration information calculation step for calculating calibration information;
It is characterized by providing.

In order to achieve the above object, a calibration information calculating apparatus according to the second aspect of the present invention provides:
A calibration image for calculating a plurality of grid points arranged at predetermined intervals is displayed on a display unit surrounding the wide-angle imaging unit, and a wide-angle image obtained by capturing the calibration image by the wide-angle imaging unit is acquired and acquired. Based on the wide-angle image and the calibration image displayed on the display unit, the position of the plurality of grid points in the three-dimensional space and the position on the wide-angle image are associated with each other in the storage unit. Grid point detection means for storing;
An optical axis detection means for extracting a plurality of straight lines based on a plurality of lattice points detected by the lattice point detection means, and detecting an optical axis of the wide-angle imaging unit based on intersections of the plurality of straight lines;
Based on the optical axis detected by the optical axis detection means, the position on the three-dimensional space of the plurality of lattice points stored in the storage means, and the position on the wide-angle image, Calibration information calculating means for calculating calibration information for correcting distortion of the wide-angle image;
It is characterized by providing.

In order to achieve the above object, a wide-angle image processing device according to the third aspect of the present invention provides:
A calibration image for calculating a plurality of grid points arranged at predetermined intervals is displayed on a display unit surrounding the wide-angle imaging unit, and a wide-angle image obtained by capturing the calibration image by the wide-angle imaging unit is acquired and acquired. Based on the wide-angle image and the calibration image displayed on the display unit, the position of the plurality of grid points in the three-dimensional space and the position on the wide-angle image are associated with each other in the storage unit. Grid point detection means for storing;
An optical axis detection means for extracting a plurality of straight lines based on a plurality of lattice points detected by the lattice point detection means, and detecting an optical axis of the wide-angle imaging unit based on intersections of the plurality of straight lines;
Based on the optical axis detected by the optical axis detection means, the position on the three-dimensional space of the plurality of lattice points stored in the storage means, and the position on the wide-angle image, Calibration information calculating means for calculating calibration information for correcting distortion of the wide-angle image;
Wide-angle image correction means for correcting a wide-angle image captured by the wide-angle imaging unit based on calibration information calculated by the calibration information calculation means;
It is characterized by providing.

  According to the present invention, a calibration information calculation method and a calibration information calculation device capable of correcting distortion of a wide-angle image with high accuracy, and a wide-angle image process for correcting a wide-angle image based on the calibration information calculated by the method. Equipment can be provided.

1 is a block diagram illustrating a schematic configuration of a wide-angle image display device according to an embodiment of the present invention. It is a figure which shows an example of a wide angle imaging part. (A) is a figure before correction | amendment, (b) is a figure which shows an example of the wide-angle image after correction | amendment. It is a figure which shows an example of a calibration image display part. It is a block diagram which shows the function of a control part. (A)-(c) is a figure which shows the example of the horizontal stripe pattern of a calibration image, (d)-(f) is a figure which shows the example of the vertical stripe pattern of a calibration image. It is a figure for demonstrating coordinate transformation. It is a figure for deriving the relational expression of r and r '. It is a flowchart for demonstrating the operation | movement which the wide angle image processing apparatus which concerns on this embodiment calibrates a wide angle image. It is a flowchart for demonstrating the operation | movement which the wide angle image processing apparatus which concerns on this embodiment acquires calibration information. (A) is a figure which shows an example of the input image at the time of imaging a horizontal stripe pattern, (b) is a figure which shows an example of the input image at the time of imaging a vertical stripe pattern, (c) is (a) and The figure which shows the lattice point extracted from (b), (d) is a figure which shows the lattice point displayed three-dimensionally. (A) And (b) is a figure for demonstrating the detection method of an image center point. It is a figure for demonstrating the detection method of an optical axis. (A) And (b) is a figure for demonstrating the calculation method of an elevation angle and the position of a vertex. It is a figure which shows an example of calibration information DB. 5 is a flowchart for explaining an operation in which the wide-angle image processing apparatus according to the present embodiment corrects a wide-angle image. (A) And (b) is a figure which shows the modification of a calibration image display part.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a wide-angle image processing apparatus 1 according to the present embodiment. As shown in FIG. 1, the wide-angle image processing apparatus 1 according to the present embodiment includes a wide-angle imaging unit 110, a calibration image display unit 120, a wide-angle image display unit 130, an operation unit 140, and a control unit 150. Composed.

  The wide-angle imaging unit 110 captures a wide-angle image that is an image with a wide-angle visual field. For example, as shown in FIG. 2, the wide-angle imaging unit 110 includes a wide-angle lens 111 having a reflecting surface 113 (curved mirror), a known imaging device 112 such as a CCD (Charged Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). including. The wide-angle imaging unit 110 projects incident light incident on the wide-angle lens 111 onto the imaging surface of the imaging device 112 by reflection by the curved mirror 113 and refraction at the entrance surface and the exit surface. The image sensor 112 outputs an image projected on the imaging surface to the control unit 150. Further, in the calibration information acquisition process described later, the wide-angle imaging unit 110 is arranged inside the calibration image display unit 120 arranged as shown in FIG. 4, for example, and captures the calibration image displayed on the calibration image display unit 120. To do.

  FIG. 3A shows an example of a wide angle image captured by the wide angle imaging unit 110. In general, a wide-angle image captured using a wide-angle lens having a curved mirror having a conical shape or the like has a ring shape like a wide-angle image A1 shown in FIG. The image a1 is an image obtained by capturing a square surface with the wide-angle imaging device 110, and each side of the square is deformed into a curve due to distortion by the wide-angle lens. A wide-angle image A2 shown in FIG. 3B is a wide-angle image corrected by a wide-angle image correction process described later. In the image a2 in the wide-angle image A2, each side of the distorted square of the image a1 shown in FIG. 3A is corrected to a straight line and displayed. Hereinafter, the uncorrected wide-angle image captured by the wide-angle imaging unit 110 is referred to as an “input image”, and the corrected wide-angle image is referred to as an “output image”.

  The calibration image display unit 120 displays a calibration image output from the control unit 150. As shown in FIG. 4, the calibration image display unit 120 is combined in a triangular prism shape so that, for example, three flat displays 121 (121 a, 121 b, 121 c) composed of an LCD (Liquid Crystal Display) surround the wide-angle imaging unit 110. In accordance with an instruction from the control unit 150, the calibration images are individually displayed. The calibration image will be described later.

  The wide-angle image display unit 130 displays an output image output from the control unit 150. The wide-angle image display unit 130 is composed of, for example, an LCD (Liquid Crystal Display), and displays an output image according to an instruction from the control unit 150.

  The operation unit 140 outputs an operation mode signal to the control unit 150 by a user operation. Here, the operation mode signal is a signal for determining whether the calibration information calculation process or the wide-angle image correction process is executed in the later-described wide-angle image calibration process, and the wide-angle image correction process is executed. A wide-angle image correction mode signal and a calibration information acquisition mode signal for executing a calibration information acquisition process are included.

  The control unit 150 acquires calibration information for correcting wide-angle distortion based on the input image output from the wide-angle imaging unit 110, and corrects the input image based on the acquired calibration information. The control unit 150 includes a so-called microcomputer, and as shown in FIG. 1, a CPU (Central Processing Unit) 151, a ROM (Read-Only Memory) 152, a general-purpose memory 153, a video memory 154, and an external device An interface (I / F) 155 and a calibration information DB (database) 156 are provided. The CPU 151 reads and executes a predetermined program for a wide-angle image calibration process, which will be described later, stored in advance in the ROM 152. The CPU 151 temporarily stores the input image acquired from the wide-angle imaging unit 110, the output image after the wide-angle image calibration process, the calculation result, and the like in the general-purpose memory 153. In addition, the CPU 151 transfers the output image stored in the general-purpose memory 153 to the video memory 154 and outputs it to the wide-angle image display unit 130. Further, the wide-angle imaging unit 110, the calibration image display unit 120, the wide-angle image display unit 130, and the operation unit 140 are connected to each unit of the control unit 150 via the external device I / F 155.

  The calibration information DB 156 includes a rewritable storage device such as a hard disk, and stores calibration information for correcting distortion of the wide-angle image captured by the wide-angle imaging unit 110. The calibration information will be described later.

  Next, a process in which the control unit 150 of the wide-angle image processing apparatus 1 according to the present embodiment functions will be described. As shown in FIG. 5, the control unit 150 functions as a lattice point detection unit 150a, an image center detection unit 150b, an optical axis detection unit 150c, a calibration information calculation unit 150d, and a wide-angle image correction unit 150e. .

  The lattice point detection unit 150a acquires a wide-angle image (input image) obtained by capturing the calibration image displayed on the flat display 121 of the calibration image display unit 120 by the wide-angle imaging unit 110, and displays the input image and the flat display 121 on the input image. Based on the obtained calibration image, the position in the three-dimensional space and the position in the wide-angle image of the plurality of lattice points calculated from the calibration image are associated with each other and stored in a storage unit such as the general-purpose memory 153, for example. It functions as something to do. Here, the calibration image is an image for calculating a plurality of grid points arranged at predetermined intervals. In the present embodiment, the lattice point detection unit 150a performs the above processing using the principle of the spatial coding method which is a well-known three-dimensional measurement technique (reference document 1; Masashi Iguchi, Kosuke Sato, “3D image measurement”). "Akiyodo, (1990)).

  Specifically, the lattice point detection unit 150a acquires a wide-angle image captured by the wide-angle imaging unit 110 for each pattern of the calibration image having a plurality of patterns, and a plurality of the plurality of patterns based on the acquired wide-angle images. Calculate grid points. And the position in the three-dimensional space of the some grid point calculated from the some pattern of the calibration image displayed on the flat display 121 corresponding to the calculated some grid point is calculated. FIGS. 6A to 6F show examples of calibration images. As shown in FIG. 6, the calibration image in the present embodiment is a diagram in which bright areas higher than a predetermined brightness (for example, white) and dark areas lower than a predetermined brightness (for example, black) are alternately arranged in the vertical direction. 6 (a) to 6 (c), and a vertical stripe pattern as shown in FIGS. 6 (d) to 6 (f) in which bright areas and dark areas are alternately arranged in the horizontal direction. Consists of In the horizontal stripe pattern of FIGS. 6A to 6C and the vertical stripe pattern of FIGS. 6D to 6F, the pitches of the bright area and the dark area change twice. Placed in. The lattice point detection unit 150a displays the calibration image on the flat display 121. The lattice point detection unit 150a detects a boundary intersection between a bright region and a dark region in the horizontal stripe pattern and the vertical stripe pattern on the input image as a lattice point by edge detection which is a well-known image processing technique. Then, the position of the lattice point on the input image and the position of the lattice point on the three-dimensional space obtained by using the principle of the spatial coding method are stored in association with each other.

  Returning to FIG. 5, the image center detection unit 150 b extracts a plurality of straight lines based on the plurality of grid points detected by the grid point detection unit 150 a, and inputs the input image based on the intersections of the extracted plurality of straight lines. Functions to detect the image center point. In the present embodiment, the image center detection unit 150b uses a Hough transform, which is a well-known digital image processing technique, to select a plurality of grid points corresponding to each set from a plurality of sets of grid points aligned in a straight line in a three-dimensional space. Is extracted from the input image (Reference Document 2: “Digital Image Processing”, CG-ARTS Association, (2006)). Then, the intersection of these straight lines is detected as the center point of the annular region of the input image by the least square method which is a well-known intersection detection method.

  The optical axis detection unit 150c functions as a unit that detects the optical axis of the wide-angle imaging unit 110 based on the intersection of a plurality of straight lines extracted by the image center detection unit 150b. Specifically, each set of lattice points corresponding to a plurality of straight lines extracted by the image center detection unit 150b is on a plane that intersects the calibration image displayed on the flat display 121 in the actual three-dimensional space. It is in. From these planes, one intersection line is calculated as an optical axis by a known least square method. By using such a method, the optical axis can be calculated even when the imaging element surface and the optical axis do not intersect perpendicularly. When it is assumed that the image sensor surface and the optical axis intersect perpendicularly, a straight line that passes through the image center point detected by the image center detection unit 150b and is perpendicular to the image sensor surface is calculated as the optical axis. May be.

  The calibration information calculation unit 150d includes the optical axis detected by the optical axis detection unit 150c, the position of the plurality of grid points stored by the grid point detection unit 150a in the three-dimensional space, and the position on the input image. And function as calculating calibration information. In this embodiment, the calibration information calculation unit 150d is on a three-dimensional space of the optical axis detected by the optical axis detection unit 151c and lattice points on the circumference of a predetermined radius from the image center point on the input image. Is extracted from the cone, and the incident angle of the light beam corresponding to the predetermined radius and the position of the apex of the cone are calculated from the cone as calibration information. The calibration information calculation unit 150d stores the calculated calibration information in the calibration information DB 156 in association with the predetermined radius.

  The wide-angle image correction unit 150e corrects the input image acquired from the wide-angle imaging unit 110 based on the calibration information stored in the calibration information DB 156, and outputs an output image that is the corrected input image to the wide-angle image display unit 130. It functions as something to do.

  Next, the wide-angle image calibration process executed by the CPU 151 of the control unit 150 will be described.

この置き換えを入力画像の全画素について実行する、すなわちｘ＝０，１，２，…，Ｎ−１、ｙ＝０，１，２，…，Ｍ−１について反復実行することで出力画像を取得することができる。 First, an outline of the wide-angle image calibration process executed by the CPU 151 of the control unit 150 in the present embodiment will be described. An object to be corrected by the CPU 151 is an input image acquired from the wide-angle imaging unit 110, and this input image is an array of luminance value digital data of M rows and N columns as shown in FIG. It is assumed that the corrected output image has the same format. In the wide-angle image correction process, the luminance value data I (x, y) before correction in the xy coordinate system defined as shown in FIG. 7 is converted into coordinates (x ′, y ′) calculated by coordinate transformation described later. ) To obtain the corrected luminance value data I ′ (x, y). Here, the luminance values I and I 'is 0, 1, _..., shall take the value of _{I max.}

This replacement is executed for all the pixels of the input image, that is, the output image is obtained by repeatedly executing x = 0, 1, 2,..., N−1, y = 0, 1, 2,. can do.

ここで、ｒ、ｒ’、θは、それぞれ後述する広角画像較正処理において算出される画像中心ｃ（ｘ_０、ｙ_０）から画素ｂ１までの距離、画像中心ｃ（ｘ_０、ｙ_０）から画素ｂ２までの距離、画像中心ｃ（ｘ_０、ｙ_０）から画素ｂ１又はｂ２へ引いた線分とｘ軸との成す角度（反時計回りを正方向とする）である。また、変換関数ｆは、ｒとｒ’の関係を表す関数であり、以下に具体的な算出方法を説明する。
図８に示すように、広角撮像部１１０が、基準面ｄからｚ_ｈの高さに設置されたとする。基準面ｄ上の点Ｂの像が広角画像上で画素ｂ１に対応し、点Ｂと広角撮像部１１０の光軸との距離がＲである場合、ｒとｒ’の関係は下記式で表される。

ここで、α（ｒ）とｚ_０（ｒ）は、それぞれ、広角撮像部１１０に対して入射する光線の角度と、広角撮像部１１０の基準点からの高さのオフセット値であり、後述する較正情報取得処理によってｒと一対一に対応する値が算出され、較正情報ＤＢ１５６に記憶される。また、ｋは、長さの単位変換を行う定数である。従って、広角画像較正処理において、較正情報取得処理では、較正情報ＤＢ１５６の仰角α（ｒ）と高さのオフセット値ｚ_０（ｒ）とを算出し、広角画像補正処理では、較正情報取得処理で算出されたα（ｒ）とｚ_０（ｒ）に基づいて上記の式からＩ’（ｘ、ｙ）を算出し、歪補正を実行する。 Next, a specific coordinate conversion method will be described with reference to FIGS. For example, as shown in FIG. 7, consider a case where the luminance value of the pixel b1 on the input image is replaced with the luminance value of the pixel b2 after coordinate conversion. Here, assuming that the coordinates of the pixels b1 and b2 are (x, y) and (x ′, y ′), respectively, their relationship is given by the following equation.

Here, r, r ′, and θ are respectively the distance from the image center c (x ₀ , y ₀ ) to the pixel b 1 calculated in the later-described wide-angle image calibration process, and the image center c (x ₀ , y ₀ ). The distance to the pixel b2 is the angle formed by the line drawn from the image center c (x ₀ , y ₀ ) to the pixel b1 or b2 and the x axis (counterclockwise is the positive direction). The conversion function f is a function representing the relationship between r and r ′, and a specific calculation method will be described below.
As shown in FIG. 8, the wide-angle imaging unit 110, and installed in the height z _h from the reference plane d. When the image of the point B on the reference plane d corresponds to the pixel b1 on the wide-angle image and the distance between the point B and the optical axis of the wide-angle imaging unit 110 is R, the relationship between r and r ′ is expressed by the following equation. Is done.

Here, α (r) and z ₀ (r) are the angle of light incident on the wide-angle imaging unit 110 and the offset value of the height from the reference point of the wide-angle imaging unit 110, which will be described later. A value corresponding to r on a one-to-one basis is calculated by the calibration information acquisition process and stored in the calibration information DB 156. K is a constant for performing unit conversion of length. Therefore, in the wide-angle image calibration process, the calibration information acquisition process calculates the elevation angle α (r) and the height offset value z ₀ (r) of the calibration information DB 156, and the wide-angle image correction process uses the calibration information acquisition process. Based on the calculated α (r) and z ₀ (r), I ′ (x, y) is calculated from the above formula, and distortion correction is executed.

  Next, an example of a flowchart of the wide-angle image calibration process of the control unit 151 in the present embodiment will be described with reference to FIG. The wide-angle image calibration process is started when power is supplied to the CPU 151 of the control unit 150, for example, by a user turning on the power of the wide-angle image processing apparatus 1 or the like.

  The controller 150 determines whether an operation mode signal has been received from the operation unit 140 (step S11). When it is determined that the operation mode signal has not been received (step S11; No), the control unit 150 is in a waiting state until the operation mode signal is received.

  When determining that the operation mode signal has been received (step S11; Yes), the controller 150 determines whether the received operation mode signal is a calibration information acquisition mode signal (step S12).

  When it is determined that the received operation mode signal is a calibration information acquisition mode signal (step S12; Yes), the control unit 150 executes a calibration information acquisition process (step S13). In addition, when it is determined that the received operation mode signal is not a signal indicating the calibration information acquisition mode, that is, the wide-angle image correction mode signal (step S12; No), the control unit 150 executes a wide-angle image correction process ( Step S14).

  Next, calibration information acquisition processing executed by the CPU 151 of the control unit 150 will be described with reference to FIG.

  The lattice point detection unit 150a causes the flat display 121 of the calibration image display unit 120 to display a plurality of horizontal stripe patterns that are calibration images, and the wide-angle imaging unit 110 captures each pattern (step S21). Specifically, for example, the lattice point detection unit 150a sequentially displays horizontal stripe patterns as shown in FIGS. Each pattern is imaged by the wide-angle imaging unit 110 and stored in the general-purpose memory 153. FIG. 11A shows an example of an input image of a horizontal stripe pattern imaged by the wide-angle imaging unit 110. An input image e1 illustrated in FIG. 11A is an input image when a horizontal stripe pattern displayed on one flat display 121 is captured as illustrated in FIG. 6B. As shown in the figure, in the input image e1, the bright areas and the dark areas of the horizontal stripe pattern are alternately arranged on a concentric circle.

  The lattice point detection unit 150a causes the flat display 121 of the calibration image display unit 120 to display a plurality of vertical stripe patterns that are calibration images, and the wide-angle imaging unit 110 captures each pattern (step S22). Specifically, for example, the lattice point detection unit 150a displays vertical stripe patterns as illustrated in FIGS. 6D to 6F on each flat display 121 in order. Each pattern is imaged by the wide-angle imaging unit 110 and stored in the general-purpose memory 153. FIG. 11B shows an example of an input image of a vertical stripe pattern imaged by the wide-angle imaging unit 110. An input image e2 illustrated in FIG. 11B is an input image when a vertical stripe pattern displayed on one flat display 121 is captured as illustrated in FIG. As shown in the figure, in the input image e2, the bright areas and the dark areas of the vertical stripe pattern are positioned alternately in the circumferential direction.

  Next, the grid point detection unit 150a grids the boundary intersections of the bright region and the dark region in the horizontal striped pattern image and the vertical striped pattern image of the plurality of input images captured by edge detection which is a well-known image technology process. As points, the coordinates of lattice points on the input image are calculated (step S23). For example, when the input image e1 of the horizontal stripe pattern shown in FIG. 11A and the input image e2 of the vertical stripe pattern shown in FIG. 11B are acquired, a plurality of grid points shown in FIG. Calculate the coordinates of g.

  The lattice point detection unit 150a calculates the coordinates of the lattice point on the input image calculated in step S23 in the three-dimensional space using the principle of the spatial coding method that is a well-known three-dimensional measurement technique. For each grid point, the coordinates on the input image and the coordinates on the three-dimensional space are stored in association with each other (step S24). FIG. 11D shows, as an experimental example, a three-dimensional display of lattice points based on the coordinates in the three-dimensional space calculated in step S24. A cross in FIG. 11D indicates the extracted lattice points.

Next, the image center detection unit 150b calculates the coordinates of the input image center point c based on the lattice points on the input image calculated in step S23 (step S25). For example, when lattice points as shown in FIG. 12 (a) are calculated, the positions on the input image are determined from a plurality of lattice points arranged in a straight line in the vertical direction in the three-dimensional space by the Hough transform. Two sets h1 and h2 are extracted. Then, a straight line h passing through the lattice points of the sets h1 and h2 is extracted. Similarly, two sets l1 and l2 and a straight line l passing through the lattice points of the sets l1 and l2 are extracted. Then, the coordinates (x ₀ , y ₀ ) are obtained from the straight line h and the straight line 1 by using, for example, the intersection of the straight line h and the straight line 1 as the center point c of the input image by the method of least squares. The image center detection unit 150b stores the calculated coordinates of the image center point c in the calibration information DB 156. FIG. 12B shows an example of an experiment for detecting the image center point from the input image. Crosses in FIG. 12B indicate lattice points extracted on the input image. In FIG. 12B, it can be seen that three straight lines are extracted, and the image center point is detected from their intersection.

  Next, the optical axis detection unit 150c calculates the optical axis O from the set of lattice points on the straight line extracted in step S25 (step S26). Specifically, as shown in FIGS. 12A and 13, the set of lattice points corresponding to the straight lines h and l extracted in step S25 intersects the calibration image in the actual three-dimensional space. On planes H and L. From these planes, one intersection line is calculated as the optical axis O by the method of least squares.

Next, the calibration information calculation unit 150d calculates calibration information. Specifically, for example, an input image as shown in FIG. 14A and a case where the coordinates of the lattice points are calculated will be described. The calibration information calculation unit 150d repeats the following processing at r = r _min ,..., R _max . Note that r represents the distance from the image center point c to a predetermined pixel, and r _min and r _max are a minimum radius and a maximum radius that are distortion correction targets in the annular region of the input image, respectively.

First, the calibration information calculation unit 150d extracts grid points located on the circumference of the radius r (r _min ≦ r ≦ r _max ) on the input image (step S27). Specifically, as shown in FIG. 14A, for a circle p having a radius r on the input image, lattice points located on the circumference of the circle p are extracted.

  Then, the calibration information calculation unit 150d extracts a cone that includes lattice points on the circumferential surface in the three-dimensional space corresponding to the lattice points extracted in step S27 (step S28). Specifically, the circumference of a circle p having a radius r on the input image corresponds to three parabolas P1 to P3 shown in FIG. 14B in a three-dimensional space. The lattice points on the circumference of the circle p with the radius r on the input image correspond to the lattice points on the parabolas P1 to P3 in the three-dimensional space. Therefore, one virtual cone Q in which the parabolas P1 to P3 correspond to the intersection line with the calibration image is extracted.

Then, the calibration information calculation unit 150d uses the well-known geometric analysis based on the coordinates of the lattice points on the circumference of the radius r on the input image calculated in step S23 to extract the cone Q extracted in step S28. , That is, the elevation angle α (r) viewed from the vertex Q ₀ shown in FIG. 14B and the vertex position z ₀ (r) are calculated (step S29). The calibration information calculation unit 150d stores the calculated α (r) and z ₀ (r) in the calibration information DB 156 in association with the radius r. FIG. 15 shows an example of the calibration information DB. Then, this process is repeated at r = r _min ,..., R _max to complete the calibration information acquisition process, and the process proceeds to step S11 in FIG.

  Next, wide-angle image correction processing executed by the CPU 151 of the control unit 150 will be described with reference to FIG.

  The wide-angle image correction unit 150e captures a wide-angle image with the wide-angle imaging unit 110 (step S31).

  Next, the wide-angle image correction unit 150e performs the following processing for all pixels of the received wide-angle image (input image). As an example, a case will be described in which the input image is a matrix of luminance value digital data of M rows and N columns, as shown in FIG.

The wide-angle image correction unit 150e calculates the center point coordinate (x ₀ ) of the input image stored in the equation (5) and the calibration information DB 156 for the pixel located at the coordinate (x, y) = (i, j) on the input image. , Y ₀ ) to calculate (r, θ) to perform coordinate conversion (step S32). Here, 0 ≦ i ≦ N−1 and 0 ≦ j ≦ M−1.

Next, the wide-angle image correcting unit 150e acquires α (r) and z ₀ (r) corresponding to r calculated in step S32 from the calibration information DB 156, and r and the center point coordinates (x ₀ , By substituting y ₀ ), the acquired α (r) and z ₀ (r) into equation (6), r ′ is calculated (step S33).

  Then, the wide-angle image correcting unit 150e calculates (x ′, y ′) from r ′ calculated in Step S33 using Expressions (2) and (3) (Step S34).

  Then, the control unit 150 replaces the luminance value I of (x ′, y ′) with the luminance value of (x, y) and stores it (Step S <b> 35).

The above processing is repeated for i = 0, 1, 2,..., N−1, j = 0, 1, 2,. Then, the obtained output image is output to the video memory 154 and is output from the video memory 154 to the wide-angle image display unit 130. Note that in steps S33 to S35, for a radius r for which calibration information is not stored in the calibration information DB 156, such as a radius r not in the range of r _min ≦ r ≦ r _max, an image of an input image from the wide-angle imaging unit 110 is used. The processing in steps S33 to S35 may be ignored on the assumption that there is no information. Specifically, in steps S33 to S35, the processing is performed only for the radius r in the range of r _min ≦ r ≦ r _max , and the output value is obtained by setting the pixel value to 0 for pixels that have not been processed. Good.

  The CPU 151 of the control unit 150 repeats the above wide-angle image calibration process until the power of the wide-angle image processing apparatus 1 is turned off by a user operation, for example.

  As described above, the wide-angle image processing device 1 according to the present embodiment acquires calibration information based on a wide-angle image obtained by capturing a calibration image displayed on the flat display 121 of the calibration image display unit 120. As described above, since a large number of calibration images can be displayed on the plurality of three-dimensionally configured flat displays 121, lattice points can be measured with high accuracy and in a short time. Therefore, calibration information can be acquired with high accuracy and in a short time.

  Further, the calibration information can be acquired with high accuracy even if the light beam incident on the wide-angle lens of the wide-angle imaging unit 110 does not intersect the lens optical axis at one point.

  In addition, this invention is not limited to said embodiment, A various deformation | transformation and application are possible. For example, in the above embodiment, a mirror reflection type wide-angle imaging device is used as an example of the wide-angle imaging unit 110, but a wide-angle imaging device using a fisheye lens may be used.

  In the above embodiment, the three flat displays 121a to 121c are configured in a triangular prism shape, but more flat displays may be used. For example, the flat display may be arranged in a polygonal column shape as shown in FIG. Moreover, you may make it comprise a polyhedron as shown in FIG.17 (b). In this way, the use of many flat displays reduces variations in the distance between the grid points of the calibration image and the wide-angle image sensor, and allows the grid points to be arranged in a wider range of the camera's field of view. Calibration can be performed with accuracy.

  In the present embodiment, the wide-angle image processing apparatus 1 executes both the calibration information acquisition process and the wide-angle image correction process as the wide-angle image calibration process, but each process may be performed in different apparatuses. For example, the wide-angle image processing device may perform calibration information acquisition processing, and the wide-angle imaging device may perform wide-angle image correction processing. In this case, the wide-angle image processing device performs wide-angle image calibration processing on the wide-angle imaging device, and stores the calibration information acquired as a result in the storage unit included in the wide-angle imaging device. Then, the wide-angle imaging device corrects the wide-angle image captured by the wide-angle image correction process based on the calibration information stored in the storage unit. By configuring as described above, it is possible to provide a wide-angle imaging device capable of highly accurate image correction with a compact configuration.

DESCRIPTION OF SYMBOLS 1 Wide-angle image processing apparatus 110 Wide-angle imaging part 120 Calibration image display part 121 (121a-121c) Planar display 130 Wide-angle image display part 140 Operation part 150 Control part 150a Grid point detection part 150b Image center detection part 150c Optical axis detection part 150d Calibration Information calculation unit 150e Wide-angle image correction unit 151 CPU
152 ROM
153 General-purpose memory 154 Video memory 155 External device I / F
156 Calibration information DB

Claims (5)

A calibration information calculation method for calculating calibration information for correcting distortion of a wide-angle image captured by a wide-angle imaging unit,
A calibration image for calculating a plurality of grid points arranged at predetermined intervals is displayed on a display unit surrounding the wide-angle imaging unit, and a wide-angle image obtained by capturing the calibration image by the wide-angle imaging unit is acquired. Based on the acquired wide-angle image and the calibration image displayed on the display unit, the storage unit associates the positions of the plurality of lattice points on the three-dimensional space and the positions on the wide-angle image. A grid point detection step stored in
An optical axis detection step of extracting a plurality of straight lines based on a plurality of lattice points detected by the lattice point detection step, and detecting an optical axis of the wide-angle imaging unit based on an intersection of the plurality of straight lines;
Based on the optical axis detected by the optical axis detection step, the position on the three-dimensional space of the plurality of lattice points stored in the storage means, and the position on the wide-angle image, A calibration information calculation step for calculating calibration information;
A calibration information calculation method comprising: The optical axis detection step includes
A plurality of straight lines are extracted based on the plurality of lattice points detected by the lattice point detection step, and the wide-angle imaging is performed based on intersections of the extracted plurality of straight lines and a plurality of planes corresponding in a three-dimensional space. Detect the optical axis of the part,
The calibration information calculation method according to claim 1, wherein: The optical axis detecting step extracts a plurality of straight lines based on the plurality of lattice points detected by the lattice point detecting step, and detects an image center point of the wide-angle image based on the intersection of the plurality of straight lines. A center detection step;
The calibration information includes predetermined information for calculating distances between the plurality of lattice points on the corrected wide-angle image and the image center point detected by the image center detection step.
3. The calibration information calculation method according to claim 1, wherein the calibration information is calculated. A calibration image for calculating a plurality of grid points arranged at predetermined intervals is displayed on a display unit surrounding the wide-angle imaging unit, and a wide-angle image obtained by capturing the calibration image by the wide-angle imaging unit is acquired and acquired. Based on the wide-angle image and the calibration image displayed on the display unit, the position of the plurality of grid points in the three-dimensional space and the position on the wide-angle image are associated with each other in the storage unit. Grid point detection means for storing;
An optical axis detection means for extracting a plurality of straight lines based on a plurality of lattice points detected by the lattice point detection means, and detecting an optical axis of the wide-angle imaging unit based on intersections of the plurality of straight lines;
Based on the optical axis detected by the optical axis detection means, the position on the three-dimensional space of the plurality of lattice points stored in the storage means, and the position on the wide-angle image, Calibration information calculating means for calculating calibration information for correcting distortion of the wide-angle image;
A calibration information calculation device comprising: A calibration image for calculating a plurality of grid points arranged at predetermined intervals is displayed on a display unit surrounding the wide-angle imaging unit, and a wide-angle image obtained by capturing the calibration image by the wide-angle imaging unit is acquired and acquired. Based on the wide-angle image and the calibration image displayed on the display unit, the position of the plurality of grid points in the three-dimensional space and the position on the wide-angle image are associated with each other in the storage unit. Grid point detection means for storing;
An optical axis detection means for extracting a plurality of straight lines based on a plurality of lattice points detected by the lattice point detection means, and detecting an optical axis of the wide-angle imaging unit based on intersections of the plurality of straight lines;
Based on the optical axis detected by the optical axis detection means, the position on the three-dimensional space of the plurality of lattice points stored in the storage means, and the position on the wide-angle image, Calibration information calculating means for calculating calibration information for correcting distortion of the wide-angle image;
Wide-angle image correction means for correcting a wide-angle image captured by the wide-angle imaging unit based on calibration information calculated by the calibration information calculation means;
A wide-angle image processing apparatus comprising:

Images