JP5363872B2 - Image correction apparatus and program thereof - Google Patents

Description

  The present invention corrects a deviation between a visible light image captured in a visible light wavelength region through a shared zoom lens capable of outputting a zoom value and an invisible light image captured in an invisible light wavelength region through a shared zoom lens. The present invention relates to an image correction apparatus and a program thereof.

  Conventionally, a visible light image captured by a visible light camera such as a color camera or a black and white camera and an invisible light image captured by an invisible light camera such as an infrared camera or an ultraviolet camera have been synthesized. At this time, a deviation may occur between the visible light image and the invisible light image. Specifically, for example, when the positions of the visible light image and the invisible light image are simply shifted, when the visible light image and the invisible light image are rotated and shifted, the parts of the visible light image and the invisible light image are complicated. There may be deviation.

  Here, for example, a technique described in Non-Patent Document 1 has been proposed as a method for correcting the deviation. With the technique described in Non-Patent Document 1, it is possible to correct the relationship between the position of the subject, the position on the captured image, and the position of the camera.

“A Flexible New Technique for Camera Calibration”, Zhengyou Zhang, December 2,1998, Technical Report MSR-TR-98-71, Microsoft Corporation

  However, it is difficult to apply the conventional technique to a field such as production of a television program that requires a visible light image and an invisible light image to be synthesized with very high accuracy (deviation is very small). Specifically, in the prior art, it is difficult to arrange the visible light camera and the invisible light camera in an optically conjugate positional relationship and to ensure the lens mounting accuracy. Further, since the lens distortion changes due to the zoom frequently used in the production of television programs, the conventional technology is difficult to apply to the zoom lens.

  Moreover, if a visible light image is image | photographed with a visible light camera, an invisible light image is image | photographed with an invisible light camera, and the technique of a nonpatent literature 1 is applied with respect to these images, object of each will be imaged. It is possible to correct the relationship between the position and the position on the image. However, even in this case, correction in consideration of the positions of the visible light camera and the invisible light camera cannot be performed, and it is difficult to ensure the accuracy required in the field of television program production and the like.

  SUMMARY An advantage of some aspects of the invention is that it provides an image correction apparatus that can be applied to a zoom lens and can be easily operated, and a program therefor.

  The inventors of the present application have focused on the fact that the correction coefficient gradually changes according to the zoom value, and found that the correction coefficient can be approximated by a mathematical expression of a quadratic function corresponding to the zoom value, thereby completing the present invention. Specifically, the image correction apparatus according to the first invention of the present application includes a visible light image captured in a visible light wavelength region through a shared zoom lens capable of outputting a zoom value, and an invisible light wavelength region through a shared zoom lens. The image correction apparatus that corrects the deviation from the invisible light image photographed in step 1 includes a correction coefficient calculation unit and a correction unit.

  According to such a configuration, the image correction apparatus receives a visible light image and an invisible light image obtained by capturing a predetermined pattern, and a zoom value obtained when the pattern is captured, by the correction coefficient calculation unit, and a position shift of the image center. Is calculated in accordance with the zoom value, and a correction coefficient indicating the rotational deviation of the image is calculated. Here, the image correction apparatus detects the feature points from the visible light image and the invisible light image obtained by photographing the pattern by the correction coefficient calculation unit, and indicates the rotation deviation of the image so that the sum of the feature point distances is minimized. The correction coefficient and the correction coefficient indicating the positional deviation of the center of the image according to the zoom value are calculated by a function modeling the lens distortion of the common zoom lens.

Further, the image correction apparatus receives a zoom value when the subject is photographed by the correction unit, a correction coefficient indicating the positional deviation of the center of the image calculated by the correction coefficient calculation unit according to the zoom value, and an image rotation deviation. Is used to determine and correct the positional relationship between the visible light image and the invisible light image obtained by photographing the subject. At this time, the image correction apparatus includes a zoom value when the subject is photographed by the correction unit, a correction coefficient indicating the positional deviation of the image center calculated by the correction coefficient calculation unit according to the zoom value, and a correction coefficient calculation unit. The positional relationship between the visible light image and the invisible light image is obtained by substituting the calculated correction coefficient indicating the rotational deviation of the image into the function.

Accordingly, the image correction apparatus can easily calculate a correction coefficient indicating the positional deviation of the center of the image according to the zoom value by the correction coefficient calculation means.
Also, for example, when a television program is shot with a visible light camera and an invisible light camera placed at a new location, the image correction device corrects using the invisible light image and the invisible light image taken at that location. What is necessary is just to calculate a coefficient. For this reason, the image correction apparatus does not require high mounting accuracy of the shared zoom lens.
Furthermore, the image correction apparatus can obtain the positional relationship between the visible light image and the invisible light image by a simple calculation simply by substituting the correction coefficient into the function by the correction means.

Further, the present image correction apparatus according to the second invention, the correction coefficient calculation unit, so that the sum of the feature point distance l represented by the below-described formula (5) becomes a minimum, the correction coefficient indicating the rotational displacement of the image θ and correction coefficients a ₁ , b ₁ , c ₁ , a ₂ , b ₂ , c ₂ , a ₃ , b ₃ , c ₃ that indicate the positional deviation of the image center according to the zoom value Z are modeled. It is calculated by the formulas (1) to (4) indicating functions.

  The image correction program according to the third invention of the present application is a visible light image photographed in a visible light wavelength region through a shared zoom lens capable of outputting a zoom value, and a non-visible light wavelength region photographed through a shared zoom lens. An image correction program for correcting a deviation from an invisible light image includes a correction coefficient calculation unit and a correction unit.

  According to such a configuration, the image correction program receives, by the correction coefficient calculation means, a visible light image and an invisible light image obtained by photographing a predetermined pattern, and a zoom value obtained when the pattern is photographed. Is calculated in accordance with the zoom value, and a correction coefficient indicating the rotational deviation of the image is calculated. Here, the image correction program detects the feature points from the visible light image and the invisible light image obtained by photographing the pattern by the correction coefficient calculation means, and indicates the rotation deviation of the image so that the sum of the feature point distances is minimized. The correction coefficient and the correction coefficient indicating the positional deviation of the center of the image according to the zoom value are calculated by a function modeling the lens distortion of the common zoom lens.

Further, the image correction program receives the zoom value when the subject is photographed by the correction means, the correction coefficient indicating the positional deviation of the center of the image calculated by the correction coefficient calculation means according to the zoom value, and the image rotation deviation. Is used to determine and correct the positional relationship between the visible light image and the invisible light image obtained by photographing the subject. At this time, the image correction program includes a zoom value when the subject is photographed by the correcting means, a correction coefficient indicating the positional deviation of the image center calculated by the correction coefficient calculating means according to the zoom value, and a correction coefficient calculating means. The positional relationship between the visible light image and the invisible light image is obtained by substituting the calculated correction coefficient indicating the rotational deviation of the image into the function.

Thus, the image correction program can easily calculate a correction coefficient indicating the positional deviation of the center of the image according to the zoom value by the correction coefficient calculation means.
Also, for example, when a television program is shot with a visible light camera and an invisible light camera placed at a new location, the image correction program corrects using the invisible light image and the invisible light image taken at that location. What is necessary is just to calculate a coefficient. For this reason, the image correction program does not require high mounting accuracy of the shared zoom lens.

According to the present invention, the following excellent effects can be obtained.
According to the first and third inventions of the present application, the correction coefficient indicating the positional deviation of the center of the image according to the zoom value is simply calculated, so that it can be applied to a common zoom lens. Further, according to the first and third inventions of the present application, high mounting accuracy of the shared zoom lens is not required, and operation is easy.
In addition, according to the first and third aspects of the present invention, the positional relationship between the visible light image and the invisible light image can be obtained with a simple calculation, and the correction process can be speeded up.

It is a block diagram which shows the structure of the image correction apparatus which concerns on this embodiment of this invention. It is a figure explaining photography of a subject in this embodiment of the present invention. It is a figure explaining the image which the image composition system of FIG. 1 synthesize | combines. It is a figure which shows the 1st example of a pattern image in this embodiment of this invention. It is a figure which shows the 2nd example of a pattern image in this embodiment of this invention. It is a flowchart which shows the correction coefficient calculation process by the image correction apparatus 4 of FIG. It is a flowchart which shows the correction process by the image correction apparatus 4 of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

[Outline of image composition system]
With reference to FIG. 1, the outline of the image composition system 1 in this embodiment of this invention is demonstrated. Here, the image composition system 1 synthesizes an image (moving image) used for producing a television program, for example, and includes a photographing camera device 2, a photographing camera control device 3, an image correction device 4, and a CG image generation. A device 5 and an image composition device 6 are provided. First, the image composition system 1 captures a predetermined pattern with the color camera 23 and the infrared camera 25 (see FIGS. 4 and 5). In the image composition system 1, the image correction device 4 calculates a correction coefficient using a color image (visible light image) and an infrared image (invisible light image) obtained by capturing a pattern. Details of the correction coefficient will be described later. In FIG. 1, input / output between a color image obtained by photographing a pattern and an infrared image is indicated by broken-line arrows.

  Next, the image composition system 1 photographs the subject Ob (see FIG. 2) with the color camera 23 and the infrared camera 25, respectively. Here, in the image composition system 1, as shown in FIG. 2, a dark screen Sc is arranged behind the subject Ob, and photographing is performed while illuminating the subject Ob with the infrared light L arranged in the vicinity of the photographing camera device 2. . The screen Sc reflects incident light in the direction of the infrared light L, in other words, reflects the infrared light irradiated by the infrared light L to the photographing camera device 2 (recursive reflector). For this reason, as shown in FIG. 3A, the color camera 23 outputs a color image in which the screen Sc is the background (the dotted area in FIG. 3A) and the subject Ob. On the other hand, as shown in FIG. 3B, the infrared camera 25 has a silhouette of the subject Ob such that the luminance of the screen Sc is high and the luminance of the subject Ob is low because the screen Sc reflects infrared rays. An infrared image indicating is output.

  In the image composition system 1, the image correction device 4 uses the calculated correction coefficient to correct the deviation of the infrared image obtained by photographing the subject Ob with respect to the color image obtained by photographing the subject Ob. Then, the image synthesis system 1 uses the CG image generation device 5 to generate, for example, a CG image of the moon or stars. Furthermore, in the image composition system 1, the image composition device 6 detects a portion of the infrared image shown in FIG. 3B where the luminance is high, that is, the screen Sc portion, and composes a CG image on this portion. On the other hand, in the image composition system 1, the image composition device 6 captures the subject Ob in FIG. 3 (a) in a portion having a low luminance in the infrared image in FIG. Combine the color images. Therefore, in the image composition system 1, the image composition device 6 outputs a CG composite image in which the background of the moon or stars and the subject Ob are combined as shown in FIG. At this time, the image composition system 1 has no difference between the color image in FIG. 3A and the infrared image in FIG. 3B, so that CG composition can be easily performed even when the shared zoom lens 21 is used. It can be carried out.

  Note that the chroma key processing used in the CG synthesis needs to arrange a single-color screen such as blue or green, and uniformly illuminate the screen with illumination light. For this reason, in the chroma key process, a constraint condition of the illumination light occurs, or the illumination light irradiated on the screen is reflected by the subject and the subject is colored in the color of the screen (for example, blue or green), so that the television program CG synthesis may not be possible for images intended by the creator. In such a case, CG composition may be performed using the image composition system 1.

Hereinafter, each device constituting the image composition system 1 will be briefly described.
The photographing camera apparatus 2 in FIG. 1 photographs a subject Ob and a pattern with a color camera 23 and an infrared camera 25, and includes a shared zoom lens 21, a color camera 23, an infrared camera 25, a hot mirror 27, and the like. Is provided.

  The common zoom lens 21 is a zoom lens capable of photographing in the visible light wavelength region and the infrared wavelength region. Here, the common zoom lens 21 measures the zoom value, the focus amount, and the iris amount with a rotary encoder (not shown), and outputs it as lens data. The shared zoom lens 21 can control a zoom value, a focus amount, and an iris amount by driving a lens drive motor (not shown) based on a lens control signal from the photographing camera control device 3.

  The visible light wavelength region is a wavelength region in which a general broadcast camera has sensitivity (for example, a wavelength is 400 nm to 700 nm). The invisible light wavelength region is a wavelength region in which this broadcast camera has no sensitivity, and includes, for example, an infrared wavelength region and an ultraviolet wavelength region. Here, in an image composition method using an infrared image, in order to reduce the influence of axial chromatic aberration, infrared light having a wavelength of about 760 nm close to the visible light wavelength region is frequently used. Accordingly, the infrared wavelength region is, for example, a wavelength region of 700 nm to 1000 μm including the wavelength of 760 nm. Further, this ultraviolet wavelength region is, for example, a wavelength region of 10 nm to 400 nm.

  The color camera 23 is a video camera, a CCD camera, or a CMOS camera that photographs the subject Ob and the pattern in the visible light wavelength region via the common zoom lens 21. Here, the color camera 23 outputs a color image obtained by photographing the pattern to the photographing camera control device 3. Further, a color image obtained by photographing the subject Ob is output to the image correction device 4. The photographing camera device 2 may include a monochrome camera (not shown) that captures a monochrome image instead of the color camera 23.

  The infrared camera 25 is a camera that photographs the subject Ob and the pattern in the infrared wavelength region via the common zoom lens 21. The infrared camera 25 converts an image taken in the infrared wavelength region into an image that can be recognized with the naked eye, such as white on the side with a short infrared wavelength and black on the side with a long infrared wavelength. Here, the infrared camera 25 outputs an infrared image obtained by photographing the pattern to the photographing camera control device 3. Further, an infrared image obtained by photographing the subject Ob is output to the image correction device 4. The photographic camera device 2 may include an ultraviolet camera (not shown) that shoots an ultraviolet image instead of the infrared camera 25.

  The hot mirror 27 is an optical element that reflects infrared light to the infrared camera 25 and transmits visible light to the color camera 23. As such an optical element, there is a half mirror or a prism in addition to the hot mirror 27.

  The photographing camera control device 3 includes operation means (not shown) such as a lever and a button, generates a lens control signal according to the operation of the cameraman, and outputs the lens control signal to the photographing camera device 2. Further, the photographing camera control device 3 outputs the zoom value included in the lens data from the photographing camera device 2, the color image obtained by photographing the pattern, and the infrared image to the image correction device 4.

  The image correction device 4 corrects a shift of an infrared image obtained by photographing the subject Ob with reference to a color image obtained by photographing the subject Ob. Here, the image correction device 4 outputs a color image obtained by photographing the subject Ob and an infrared image in which the shift is corrected to the image composition device 6. Details of the image correction device 4 will be described later.

  The CG image generation device 5 is a computer that generates a CG image of the moon or stars, for example, in accordance with an operator command. Further, the CG image generation device 5 outputs the generated CG image to the image composition device 6.

  The image composition device 6 performs CG composition of a color image obtained by photographing the subject Ob, an infrared image whose displacement has been corrected, and a CG image generated by the CG image generation device 5. The details of the CG synthesis are omitted as described above.

(First embodiment)
[Configuration of image correction apparatus]
Hereinafter, the configuration of the image correction apparatus 4 according to the present embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, the image correction apparatus 4 includes a correction coefficient calculation unit 41, a storage unit 42, a correction unit 43, and a delay unit 44.

The correction coefficient calculation means 41 receives a color image and an infrared image obtained by photographing a pattern, and a zoom value when the pattern is photographed from the photographing camera control device 3, and indicates a positional deviation of the image center according to the zoom value. The coefficients a _{1 to} c ₃ and the correction coefficient θ indicating the image rotation deviation are calculated.

  Here, the photographing camera device 2 captures a pattern in advance while changing the zoom of the common zoom lens 21 at a predetermined interval. At this time, it is preferable that the photographic camera device 2 shoots the pattern by shortening the zoom interval in the vicinity of the zoom value where the deviation becomes large. That is, the correction coefficient calculating means 41 receives a plurality of pairs of a color image obtained by photographing a pattern and an infrared image so that the zoom values of the pairs are different.

  Further, the correction coefficient calculation means 41 detects feature points from the color image and the infrared image obtained by photographing the pattern. Hereinafter, detection of feature points will be described with reference to a first example and a second example.

<Detection of feature points: first example>
As shown in FIG. 4, this first example is a technique for detecting feature points from a pattern image Pt whose pattern is a checkered pattern. At this time, the correction coefficient calculating unit 41 detects feature points by applying corner detection such as Moravec corner detection and Harris-Steps corner detection to the checkered pattern image Pt. Here, the correction coefficient calculation means 41 detects the four corners of each quadrangle as feature points from the checkered pattern in FIG.

  When such corner detection is used, there is an advantage that the correction coefficient calculation means 41 can reduce the amount of calculation for detecting feature points. On the other hand, depending on the resolution of the color camera 23 or the infrared camera 25 when the common zoom lens 21 is zoomed (when enlarged shooting is performed), as shown in FIG. 4, the four corners of the black square are crushed in the pattern image Pt. Therefore, the feature point may not be detected (enlarged portion in FIG. 4). In such a case, the following second example is used.

<Detection of feature points: second example>
As shown in FIG. 5, the second example is a method of detecting feature points from a pattern image Pt having a pattern of polka dots arranged so that black circles do not overlap each other on a white background. At this time, the correction coefficient calculating means 41 extracts a black circle area (black area) from the polka dot pattern image Pt, and detects the center of gravity of each black circle area as a feature point. This second example detection method has the advantage of high feature point detection accuracy.

  In the first example and the second example, the correction coefficient calculation unit 41 calculates feature points from both a color image (pattern image Pt) obtained by photographing a pattern and an infrared image (pattern image Pt) obtained by photographing the pattern. It goes without saying that it is detected.

  Note that the shift between the color image and the infrared image (that is, the calculated correction coefficient) is the same regardless of which of the first and second examples described above. Therefore, for example, the image correction apparatus 4 may select either one of the methods of the first example and the second example in accordance with a command from the operator and detect the feature points.

Returning to FIG. 1, the description of the configuration of the image correction apparatus 4 will be continued.
Further, the correction coefficient calculation means 41 calculates the correction coefficient θ and the correction coefficients a _{1 to} c ₃ so that the sum of the feature point distances 1 expressed by the following expression (5) is minimized. -It calculates by following formula (4). Here, the feature point distance l is a feature point corresponding to a color image obtained by photographing a pattern and an infrared image, that is, a paired distance (coordinate deviation) of detection points detected from the same position.

However, according the formula (1) to _(5), the _{a 1, b 1, c 1} , a 2, b 2, c 2, a 3, b 3, c 3 is positional deviation of the image center to the zoom value a correction coefficient showing Te, theta is a correction factor indicating the rotational displacement of the image, S is a scale value, Z is the zoom value, C _x is the horizontal displacement of the image center, C _y is a vertical displacement of the image center, V _x is the horizontal position of the visible light image (coordinates), V _y is the vertical position of the visible light image (coordinates), I _x is invisible light image Is the horizontal position (coordinates), and I _y is the vertical position (coordinates) of the invisible light image.

In Expression (5), i _x is the horizontal position of the invisible light image calculated by substituting V _x into Expression (1), and i _y is calculated by substituting V _y into Expression (1). This is the vertical position of the invisible light image.

Here, the feature point distance 1 represented by the equation (5) is calculated for each zoom value Z. Therefore, the correction coefficient calculation means 41 may calculate the correction coefficients a _{1 to} c ₃ that minimize the sum of the feature point distances l for each zoom value. For example, the correction coefficient calculation means 41 uses an optimization method such as the Levenberg-Markert method, the Gauss-Newton method, or the like to correct the correction coefficient θ and the correction coefficient a that minimize the sum of the feature point distances l for each zoom value Z. to calculate the _{1 ~c 3.} Then, the correction coefficient calculation unit 41 writes the correction coefficients a _{1 to} c ₃ and the correction coefficient θ in the storage unit 42 in association with the zoom value Z.

The storage unit 42 is a storage unit such as a hard disk or a memory that stores the correction coefficients a _{1 to} c ₃ and the correction coefficient θ in association with the zoom value Z. In the present exemplary embodiment, the storage unit 42 is described. However, the image correction apparatus 4 may not include the storage unit 42. In this case, the correction coefficient calculation unit 41 outputs the correction coefficients a _{1 to} c ₃ and the correction coefficient θ associated with the zoom value Z to the correction unit 43.

The correction means 43 receives the zoom value Z ′ when the subject Ob is photographed from the photographing camera control device 3, and uses the correction coefficients a _{1 to} c ₃ and the correction coefficient θ calculated by the correction coefficient calculation means 41. The positional relationship between the color image obtained by photographing the subject Ob and the infrared image is obtained. First, the correction unit 43 refers to the storage unit 42 and reads out the correction coefficients a _{1 to} c ₃ and the correction coefficient θ corresponding to the same zoom value Z as the zoom value Z ′. Here, when the correction coefficients a _{1 to} c ₃ and the correction coefficient θ corresponding to the zoom value Z are not stored in the storage means 42, the correction means 43 corresponds to the zoom value Z closest to the zoom value Z ′. The corrected correction coefficients a _{1 to} c ₃ and the correction coefficient θ are read out.

Further, the correcting means 43 substitutes the zoom value Z ′ as the zoom value Z into the equations (3) to (4), and sets the correction coefficients a _{1 to} c ₃ and the correction coefficient θ to the equations (2) to (4). ) To calculate the scale value S and the image center misalignment C _x , C _y . Then, the correcting unit 43 substitutes the calculated scale value S, the calculated image center positional deviations C _x and C _y, and the correction coefficient θ calculated by the correction coefficient calculating unit 41 into the equation (1). Then, the deviation between the color image obtained by photographing the subject Ob and the infrared image is calculated. In other words, the expression (1) indicates the positional relationship between the color image obtained by photographing the subject Ob and the infrared image (that is, the coordinate conversion expression).

Further, since the positional relationship between the color image obtained by photographing the subject Ob and the infrared image is known from the equation (1), the correcting unit 43 corrects the deviation of the infrared image input from the photographing camera device 2. At this time, the correction unit 43 substitutes the coordinates IC _x and IC _y of the pixel whose deviation is to be corrected in the infrared image into V _x and V _y of the equation (1), respectively. Further, the correction means 43 calculates I _x , I calculated by substituting the scale value S, the image center misalignment C _x , C _y, and the correction coefficient θ into the equation (1) in the infrared image obtained by photographing the subject Ob. The pixel value of the pixel indicated by _y is acquired. Thereafter, the correcting unit 43 sets the pixel values at the coordinates I _x and I _{y as} the pixel values of the pixels IC _x and IC _y .

  At this time, the correction unit 43 performs image processing such as enlargement and reduction on the infrared image in order to eliminate the shift of the infrared image with respect to the color image by, for example, a 3-tap Lanczos method. In addition, the correction means 43 may use a bilinear method, a nearest neighbor method, or a bicubic method. Then, the correcting unit 43 outputs the corrected infrared image to the image composition device 6.

In general, the pixels are arranged in a lattice pattern, and the coordinates of the pixels are expressed as integers. On the other hand, the coordinates I _x and I _{y on the} left side of Expression (1) are not always the sampling positions of the image, and are expressed including decimal numbers. In this case, the correction unit 43 may perform pixel interpolation processing.

  The delay means 44 is a frame memory that receives a color image obtained by photographing the subject Ob and accumulates the color image. In addition, the delay unit 44 delays the correction processing time of the correction unit 43 and outputs the accumulated color image to the image composition device 6.

[Modeling of lens distortion of common zoom lens]
Hereinafter, modeling of the lens distortion of the common zoom lens in the present invention will be described.
When the shared zoom lens 21 is used in the color camera 23 (visible light camera) and the infrared camera 25 (invisible light camera) as in the present invention, a shift occurs between the color image and the infrared image for the following reason. . The first reason is due to the mounting accuracy of the shared zoom lens 21, the hot mirror 27, and the like. The second reason is that lateral chromatic aberration occurs due to the wavelength characteristics of the shared zoom lens 21.

First, the displacement amount of the image that these grounds are placed positional deviation C _x of the image center of the color image and the infrared _image, and C _y, a rotational shift θ in the vertical direction with respect to the optical axis of the common zoom lens 21. Here, according to the experiments of the present inventors, for example, in the case of a shared lens designed based on visible light, the chromatic aberration of magnification in the infrared wavelength region and the ultraviolet wavelength region is a simple scale with respect to the visible wavelength region. It was found to correspond to the value. From this, Formula (1) is materialized. Note that equation (1) may not apply to extreme wide-angle lenses.

Further, according to experiments by the inventors of the present application, in the case of the shared zoom lens 21, the optical axis of the shared zoom lens 21 is not accurately perpendicular to the imaging surface, and thus the correction coefficients a _{1 to} c ₃ described above are obtained. It has also been found that it gradually changes according to the zoom value Z. As a result of experiments using various common zoom lenses 21, it has been found that the above-described correction coefficients a _{1 to} c ₃ can be approximated by a quadratic function formula corresponding to the zoom value Z. From this, Formula (2)-Formula (4) are materialized. In addition, Formula (1)-Formula (4) are the functions which modeled the lens distortion of the common zoom lens as described in a claim.

[Operation of Image Correction Device: Correction Coefficient Calculation Processing]
Hereinafter, with reference to FIG. 6, the correction coefficient calculation processing by the image correction apparatus 4 in FIG. 1 will be described (see FIG. 1 as appropriate). First, the image correction device 4 inputs a color image (pattern image Pt) obtained by photographing a pattern and an infrared image (pattern image Pt) obtained by photographing the pattern to the correction coefficient calculating means 41 (step S1). Then, the image correction device 4 inputs the zoom value Z when this pattern is photographed to the correction coefficient calculation means 41 (step S2).

  Subsequent to the processing in step S2, the image correction apparatus 4 uses the correction coefficient calculation means 41 to extract feature points from each of the color image input in step S1 and the infrared image obtained by capturing the pattern, for example, by corner detection. Detect (step S3).

Subsequent to the processing in step S3, the image correction device 4 causes the correction coefficient calculation means 41 to correct the correction coefficient θ and the correction coefficient a so that the sum of the feature point distances 1 expressed by the equation (5) is minimized. and ₁ to c ₃ is calculated by the formula (1) to (4). Here, the image correction device 4 uses the correction coefficient calculation unit 41 to optimize the correction coefficient θ that minimizes the sum of the feature point distances l using, for example, an optimization method such as the Levenberg-Markert method or the Gauss-Newton method. And correction coefficients a _{1 to} c ₃ are calculated (step S4).

Subsequent to the processing in step S4, the image correction apparatus 4 uses the correction coefficient calculation means 41 to convert the correction coefficient θ calculated in step S4 and the correction coefficients a _{1 to} c ₃ into the zoom value Z input in step S2. Corresponding data are written in the storage means 42 (step S5).

  Here, for example, the image correction device 4 repeatedly performs the operations of steps S1 to S5 by the correction coefficient calculation means 41, and corrects the correction coefficient from the color image and the infrared image obtained by shooting the patterns shot at different zoom values Z. Is calculated. At this time, it is preferable that the image correction device 4 calculates the correction coefficient by the correction coefficient calculation unit 41 in the vicinity of the zoom value where the deviation becomes large and in the vicinity of the zoom value frequently used during shooting. Thereby, the image correction apparatus 4 can accurately reflect the change in the lens distortion of the shared zoom lens 21 due to the zoom by the correction coefficient, and the correction accuracy of the deviation is improved.

[Operation of Image Correction Device: Correction Processing]
Hereinafter, with reference to FIG. 7, the correction processing by the image correction apparatus 4 in FIG. 1 will be described (see FIG. 1 as appropriate). First, the image correction apparatus 4 inputs a color image obtained by photographing the subject Ob and an infrared image obtained by photographing the subject Ob to the correction unit 43 (step S11). Further, the image correction device 4 inputs the zoom value Z ′ when the subject Ob is photographed to the correction unit 43 (step S12).

Following the processing in step S12, the image correction apparatus 4 reads out the correction coefficients a _{1 to} c ₃ and the correction coefficient θ corresponding to the zoom value Z ′ from the storage means 42 by the correction means 43 (step S13).

Subsequent to the processing in step S13, the image correction apparatus 4 uses the correction unit 43 to substitute the zoom value Z ′ as the zoom value Z into the equations (3) to (4) to correct the correction coefficients a _{1 to} c _3. The coefficient θ is substituted into the equations (2) to (4), and the scale value S and the image center misalignments C _x and C _y are calculated (step S14).

Subsequent to the processing in step S14, the image correction apparatus 4 uses the correction unit 43 to calculate the scale value S calculated in step S14, the image center misalignment C _x , C _y, and the correction coefficient θ read in step S13. Substituting into the equation (1), the positional relationship between the color image obtained by photographing the subject Ob and the infrared image is obtained (step S15).

As described above, the image correction apparatus 4 according to the present embodiment of the present invention uses the correction coefficients a _{1 to} c ₃ that indicate the positional deviation of the image center according to the zoom value, and the expressions (1) to (5). Since it can be easily calculated by using it, it can be applied to a zoom lens. In addition, the image correction apparatus 4 according to the present embodiment of the present invention uses a color image and an infrared image captured at a place when the photographing camera apparatus 2 is placed at a new place to shoot a television program. The correction coefficients a _{1 to} c ₃ and the correction coefficient θ may be calculated. For this reason, the image correction apparatus 4 according to the present embodiment of the present invention does not require high mounting accuracy of the zoom lens. Furthermore, the image correction device 4 according to the present embodiment of the present invention can correct a deviation between a color image and an infrared image with accuracy in units of pixels even when zooming is performed during shooting. It can withstand practical use even in fields where extremely high accuracy is required.

  Further, the application of the image correction apparatus 4 according to the present embodiment of the present invention is not limited, but in particular, when chroma key processing is difficult, when using a retroreflecting material such as the screen Sc in FIG. This is effective when acquiring information about a subject included in an image with a marker that reacts only to invisible light.

  Further, for example, the image correction device 4 according to the present invention corrects a deviation between a thermographic image obtained by photographing a person with an infrared thermographic camera and a color image. Then, the image synthesis device 6 may synthesize the background portion of the color image and the thermographic image. As described above, the image correction apparatus 4 according to the present invention can also be used when generating an image in which a body temperature display of a subject is combined with a color live-action background.

  In the present embodiment of the present invention, the image correction apparatus 4 has been described as an independent apparatus, but the present invention is not limited to this. In the present invention, hardware resources such as a general computer arithmetic device and a storage device can also be realized by a program that performs cooperative operation as each of the above-described means. This program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

DESCRIPTION OF SYMBOLS 1 Image composition system 2 Shooting camera apparatus 21 Shared zoom lens 23 Color camera 25 Infrared camera 27 Hot mirror 3 Shooting camera control apparatus 4 Image correction apparatus 41 Correction coefficient calculation means 42 Storage means 43 Correction means 44 Delay means 5 CG image generation apparatus 6 Image synthesizer

Claims (3)

An image correction apparatus that corrects a deviation between a visible light image captured in a visible light wavelength region via a shared zoom lens capable of outputting a zoom value and an invisible light image captured in an invisible light wavelength region via the shared zoom lens. In
The visible light image and the invisible light image obtained by photographing a predetermined pattern, the zoom value obtained when the pattern is photographed, and a correction coefficient indicating a positional deviation of the center of the image according to the zoom value, and an image Correction coefficient calculating means for calculating a correction coefficient indicating the rotational deviation of
The zoom value when the subject is photographed is input, and the correction coefficient indicating the positional deviation of the center of the image calculated by the correction coefficient calculating unit according to the zoom value and the correction coefficient indicating the rotational deviation of the image are used. Correction means for obtaining and correcting the positional relationship between the visible light image and the invisible light image obtained by photographing the subject,
The correction coefficient calculating means detects a feature point from the visible light image and the invisible light image obtained by photographing the pattern, and indicates a correction coefficient indicating a rotational deviation of the image so that the sum of the feature point distances is minimized. And a correction coefficient indicating the positional deviation of the center of the image according to the zoom value is calculated by a function modeling the lens distortion of the shared zoom lens ,
The correction means calculates the zoom value when the subject is photographed, a correction coefficient indicating the positional deviation of the image center calculated by the correction coefficient calculation means according to the zoom value, and the correction coefficient calculation means calculates by substituting the correction coefficient indicating the rotational displacement of the image on the function, the image correction apparatus according to claim Rukoto obtain the position relationship between the invisible light image and the visible light image. The correction coefficient calculating means calculates the correction coefficient θ indicating the rotational deviation of the image and the positional deviation of the center of the image so as to minimize the sum of the feature point distances 1 represented by Expression (5). The correction coefficients a ₁ , b ₁ , c ₁ , a ₂ , b ₂ , c ₂ , a ₃ , b ₃ , and c ₃ shown in accordance with the formulas (1) to (4) indicating the modeled function The image correction apparatus according to claim 1, wherein
However, in Expressions (1) to (5), S is a scale value, C _x is a horizontal shift of the image center, _Cy is a vertical shift of the image center, and V _x is The horizontal position of the visible light image, V _y is the vertical position of the visible light image, I _x is the horizontal position of the invisible light image, and I _y is the vertical position of the invisible light image. I _x is the horizontal position of the invisible light image calculated by substituting V _x into Equation (1), and i _y is calculated by substituting V _y into Equation (1). It is the position in the vertical direction of the invisible light image.

In order to correct the deviation between the visible light image captured in the visible light wavelength region through the shared zoom lens that can output the zoom value and the invisible light image captured in the invisible light wavelength region through the shared zoom lens, Computer
The visible light image and the invisible light image obtained by photographing a predetermined pattern, the zoom value obtained when the pattern is photographed, and a correction coefficient indicating a positional deviation of the center of the image according to the zoom value, and an image correction coefficient calculating means to calculate a correction coefficient indicating the rotational displacement of
The zoom value when the subject is photographed is input, and the correction coefficient indicating the positional deviation of the center of the image calculated by the correction coefficient calculating unit according to the zoom value and the correction coefficient indicating the rotational deviation of the image are used. A correction means for obtaining and correcting the positional relationship between the visible light image and the invisible light image obtained by photographing the subject,
The correction coefficient calculating means detects a feature point from the visible light image and the invisible light image obtained by photographing the pattern, and indicates a correction coefficient indicating a rotational deviation of the image so that the sum of the feature point distances is minimized. And a correction coefficient indicating the positional deviation of the center of the image according to the zoom value is calculated by a function modeling the lens distortion of the shared zoom lens ,
The correction means calculates the zoom value when the subject is photographed, a correction coefficient indicating the positional deviation of the image center calculated by the correction coefficient calculation means according to the zoom value, and the correction coefficient calculation means calculates An image correction program for substituting a correction coefficient indicating a rotational shift of the image into the function to obtain a positional relationship between the visible light image and the invisible light image .

Images