KR101455071B1 - Method for enhancing night time image using digital compositing - Google Patents

Abstract

Disclosed is a method for enhancing a night time image using digital compositing which obtains the image of an object to distinguish the outline and color of the object at night when the object is difficult to recognize. For this, a method for enhancing a night time image includes a step of collecting an optical image with regard to an imaging object region through a low brightness camera, acquiring a thermal image through a thermal image camera, and storing the same; a step of digital-synthesizing the optical image and the thermal image and compositing an output image; and a step of storing the output image. According to the present invention, an image having a clear outline and color can be acquired without separate lighting even in a dark environment.

Description

METHOD FOR ENHANCING NIGHT TIME IMAGE USING DIGITAL COMPOSITING [0002]

The present invention relates to a method of improving nighttime photographing performance using digital synthesis, and more particularly, to a method of improving nighttime photographing performance by using digital synthesis capable of obtaining an image of a subject that allows identification of a contour and color of a subject at night, Night shooting performance improvement method.

Night shot cameras using an infrared illuminator provide a high magnification, but this high magnification naturally results in a narrow field of view.

The narrow field of view is also due to the fact that the beam branch of the fixture that passes the set angle in a given range involves insufficient beam intensity to produce useful images. Such visual field limitation makes it difficult for a user of such a camera for night vision to direct himself / herself to the night image capturing area, especially when the area required for the cover is wide and complex or when the instant night view photographing field has a poor information content.

One way to overcome this limitation is to scan the relevant area in detail to look for a visible mark that helps to locate the current field of view. Due to the nature of night-time shooting, such a search process is generally not practical because it takes a considerable amount of time in an emergency.

Some night vision systems use a laser infrared illuminator. Most of these systems use gated imaging techniques to overcome the backscattering around the periphery, which is considered a dominant noise mechanism. If the gate-type imaging technique is used, complex synchronization of an expensive imaging device and a laser pulse having such an imaging device is required. Also, the gated image based on the night imaging system is likely to be saturated due to the strong light source in the view of the imager.

The use of a night vision system using a conventional laser illuminator causes safety problems with respect to the eye, and it is desirable that there be a laser safety personnel in the area where the night vision system is located.

Another night vision system uses active infrared illumination generated by an LED bulb with a camera installed to pick up the projected light source. Such a system only provides useful images in a short range, the bulbs are relatively short in life and add maintenance and operating costs to the system. Also, such a system is prone to saturation due to intense light sources in the field of view of the imager.

Another night imaging system uses an enhanced CCD (ICCD) configuration that relies on ambient light (e.g., starlight and moonlight) picked up by a sensitive CCD camera. Such a system is capable of capturing color images in a cloudy or moony environment, but the contours of the captured images are unclear, making it difficult to use them for broadcasting and security purposes.

The most common night-time imaging system is based on a thermal image where the heat of the object is used to generate an image of the product. This is because although the final image can be used for detection by operator training, it is not a true identification because the character and some image detail can not be viewed. In addition, the uncooled thermal imaging system has a short range, and the cooled thermal imaging system has a long range but is very expensive. Finally, adding a zooming capability to a thermal imaging system increases the cost of the facility because it requires the use of a special lens, which causes many thermal imaging systems to have a fixed focal length.

On the other hand, in digital compositing technology, a key is used to cover another screen on a background screen. It is used to synthesize a screen, and separates a part to be overlapped and a part to be removed.

For example, suppose you have a transparent vinyl of the same size with a picture of a flower on a picture of Mona Lisa. If you erased other parts of the picture except the flower of Vinyl, the effect is the same as that of a flower on the picture of Mona Lisa. These keys include a luminance key, a chrominance key, a mat key, and the like.

All the colors of the luminance key have respective luminance values. By adjusting the cutline of the luminance value using the difference of the luminance values, the value below the cut line is removed, and the residual value is left on the screen, and only the remaining partial screen is synthesized on the other background screen.

The effect of removing the arbitrary color adjusted by the key using the above chrominance key color difference to obtain a necessary screen portion. Generally, in scenes shot with people and things on a blue background, only the background color is removed, and the remaining source is synthesized and processed with another screen.

When the key operation is performed on the matte key, a remaining part and a removed part of the screen are generated. This is called a key hole. The keyhole is made separately (with black and white only) Quot; is filled with another video screen.

Korean Patent Publication No. 10-2011-0114096 (published on October 19, 2011) Korean Patent Laid-Open No. 10-2007-0106709 (Published Nov. 5, 2007) Korean Registered Patent No. 10-0623835 (Announced on September 19, 2006)

Accordingly, it is an object of the present invention to provide a method of improving night-time photographing performance by using digital synthesis capable of generating a color image having a sharp outline even in a low-illuminance environment through mapping of optical image data and thermal image data.

According to an embodiment of the present invention, there is provided a method of acquiring an optical image of a region to be imaged through a low-illuminance camera, acquiring and storing a thermal image through a thermal imaging camera, A step of digitally synthesizing an image and a thermal image to form an output image, and a step of storing the output image.

The nighttime imaging performance improving method according to the present invention can acquire an image having sharp contours and colors by using the mapping between the optical image data generated by the low-illuminance camera and the thermal image data generated by the thermal imaging camera.

 In addition, the present invention can acquire image information in real time at night, and allows an operator to precisely monitor an area to be photographed on the basis of the image information.

In addition, since the present invention is based on thermal image data, it is possible to easily confirm the position of the object to be photographed even in an environment where it is difficult to confirm the position of the object to be photographed by the human vision, and the surrounding situation can be easily recognized.

FIG. 1 is a flowchart illustrating an embodiment of a method for improving a night photography performance according to the present invention.
2 is a block diagram illustrating a camera system for night photography according to the present invention.
3 is a flowchart showing another embodiment of the night photography performance improving method according to the present invention.

Hereinafter, a night imaging performance improving method using digital synthesis according to preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating an embodiment of a method for improving a night photography performance according to the present invention.

Referring to FIG. 1, a method of improving nighttime photographing performance according to the present invention includes an image acquisition step (S100) of acquiring an optical image and a thermal image, a step of synthesizing the optical image and the thermal image using a digital synthesis technique, (S200), and an output image storing step (S300) for storing the output image. Here, the optical image means an image obtained by imaging a visible light or a light in the near-infrared region condensed by the condenser lens with a normal image sensor.

Hereinafter, each component will be described in more detail with reference to the drawings.

Referring to FIG. 1, a nighttime photographing performance improving method according to the present invention includes an image acquiring step (S100).

The image acquiring step S100 captures an image of a region to be photographed through the low light camera 100, acquires optical image data of the region to be photographed, captures an image of the region to be photographed through the thermal image camera 200, And obtaining and storing the thermal image data for the region.

In this step S100, the low-illuminance camera 100 compresses the optical image data by a predetermined compression method, and transmits the compressed optical image data to the image processing apparatus connected to the wired / wireless network. Likewise, the thermal imaging camera 200 also compresses the thermal image data by a predetermined compression method, and transmits the compressed thermal image data to the image photographing apparatus connected to the wired / wireless network. At this time, the image photographing apparatus stores the received image data in the storage section.

Here, as the predetermined compression method, a compression method such as Moving Picture Experts Group Phase 4 (MPEG-4) method, H.264 method, or JPEG 2000 (Joint Photographic Experts Group 2000) method can be used.

The low-illuminance camera 100 is capable of photographing a color image even in an illuminance of 1 to 0.01 lux. Any digital camera may be used as long as it can achieve this purpose, but an electron-multiplying CCD (EM-CCD) It is preferable to use a digital camera (for example, EM-120H manufactured by Flovel) equipped with a camera. Since the low-illuminance camera 100 has a structure known in the art, a more detailed description will be omitted.

The thermal imaging camera 200 is a camera for outputting thermal image data for visualizing objects by radiation energy by sensing a temperature difference between the objects in the shooting target area and converting the temperature difference into an electrical signal.

2, the image processing apparatus includes a control unit 310, a storage unit 320, and a video output unit 320. The control unit 310 is connected to the low-illuminance camera 100 and the thermal video camera 200 via a wireless or data cable, (330).

If necessary, the image processing apparatus may further include an input unit (not shown). The input unit includes at least one of a keyboard, a mouse, a joystick, and a touch screen, and allows a user to operate the system and input control commands.

3 is a flowchart showing another embodiment of the night photography performance improving method according to the present invention.

Referring to FIG. 3, the night shooting performance improving method according to the present invention may further include a step S50 of adjusting a shooting object size.

In order to match the size of the object to be photographed observed through the low light camera 100 and the thermal image camera 200, the low light camera 100 (step S50) 100 and the front view angle of the thermal imaging camera 200 are matched with each other.

At this time, the front view angle can be matched to any one of the low-illuminance camera 100 and the thermal imaging camera 200, but the low-illuminance camera 100, It is also possible to adjust the front viewing angle of the front lens.

The step of adjusting the size of the object to be photographed (S50) includes the steps of calculating a distance value of the object by irradiating the object to be imaged with a laser range finder 400, and controlling the controller 310 Calculating a position value and a pan-tilt value of the low-illuminance camera 100 and the thermal imaging camera 200, and calculating a pan-tilt value of the low-illuminance camera 100 and the thermal- Controlling the position moving device 600 installed in the video camera 200 and controlling the pan-tilt driving device 500 installed in the low-illuminance camera 100 and the thermal video camera 200 according to the pan-tilt value Lt; / RTI >

More specifically, the characteristics of the lens mounted on the low-illuminance camera 100 are determined through a camera calibration technique, and the camera calibration is performed by extracting observation characteristic parameters of the camera using the algorithm proposed by Tsai et al. The most important parameters are the focal length and rotation and translation parameters of the lens.

In the case of the infrared thermal imaging camera 200, a lens specification and a field of view (FOV) specification provided by the manufacturer are used. The front viewing angle of the thermal imaging camera 200 is converted to the front viewing angle of the low-illuminance camera 100 to match the front viewing angles of the two cameras.

For example, the front view angle of the 25 mm infrared lens mounted on the thermal imaging camera 200 and the 25 mm lens mounted on the low illumination camera 100 are different. This is because the infrared thermal image sensor and the color CCD sensor have different sizes.

More specifically, in the case of a thermal imaging camera 200 equipped with a 25 mm lens, since the sensor size is 320 × 240 pixels and the size of one pixel is 48.5 μm × 48.5 μm, the actual size of the sensor is 15.52 × 11.64 mm. In contrast, when the sensor size of the low-illuminance camera 100 is 768 × 494 pixels, the actual size of the color CCD sensor is 6.45 × 4.84 mm since the size of one pixel is 8.4 μm × 9.8 μm.

When the FOV is converted into the FOV, the observed FOV of the thermal imaging camera 200 is 34.5 × 26.2 ° and the FOV of the low-illuminance camera 100 equipped with the same 25 mm lens is 14.7 × 11.1 °. Therefore, in order to match the sizes of the objects observed by the two cameras, the FOVs of the two cameras are matched to the FOV of the low-illuminance camera 100, the FOV of the thermal imaging camera 200, or the like.

 The pan-tilt driving device 500 is installed in the low-illuminance camera 100 and the thermal imaging camera 200 to fix the camera. The pan-tilt driving device 500 controls the low light intensity camera 100, (200) can be rotated in the left-right direction or the up-down direction, respectively.

 The position moving device 600 is provided in the low light camera 100 and the thermal image camera 200 and moves the low light camera 100 and the thermal image camera 200 so that the distance between each camera and the shooting target area is adjusted. .

The control unit 310 of the image processing apparatus controls the thermal imaging camera 200 and the low light intensity camera 100 to acquire thermal image data and optical image data of a region to be photographed according to a user's selection, The apparatus 500 and the position moving device 600 are controlled to provide a function of arranging the thermal imaging camera 200 and the low light intensity camera 100 so as to face a specific direction.

More specifically, the control unit 310 transmits a control signal for controlling the pan / tilt drive to the pan / tilt driver 500 in response to the control command input through the input unit. Here, the pan / tilt driving may be performed automatically according to a predetermined pattern by the program, or according to the result of the detection of the subject or the subject movement in the image.

The controller 310 controls the position and pan-tilt values of the low-illuminance camera 100 according to the distance values measured from the laser range finder 400 from the preset storage unit 320, Tilt driving device 500 and the position moving device 600 by extracting a tilt value. To this end, the storage unit 320 may store a look-up table indicating the corresponding relationship between the distance value, the position value, and the pan-tilt value. In other words, the control unit 310 refers to the look-up table to control the position and direction of the camera.

The image output unit 330 is configured to output an output image generated through the control unit 310. For this, the video output unit 330 may be connected to the controller 310 in a wired manner.

Referring to FIG. 1 and FIG. 2, a method of improving night image pickup performance according to the present invention includes an output image forming step S200.

The output image forming step S200 is a step of forming an output image by mapping the optical image data and the column image data to the controller 310. The matching image extracting step S210 and the digital combining step S220 .

In the matching data extracting step (S210), color data for an object to be imaged is extracted from the optical image data, and contour data for the object to be imaged is extracted from the thermal image data.

In the digital synthesis process (S220), the color data extracted through the matching data extracting process and the contour data are digitally combined through a mat key method or the like to form one output image.

For example, when an arbitrary object having a color F and an alpha mat is placed in front of the background of color B in a digital synthesis in general, the resulting color C is expressed by a mathematical equation as shown in [Equation 1].

[Equation 1]

C = F + (1 -?) B

Here, each mat or alpha represents the state and transparency of the pixel by the placed object.

Therefore, in general synthesis, the color of each element can be represented by a collection of several other elements. That is, an object can have an element that emits light, reflects some of the light coming from the light source, or passes or even refracts light from the environment in other parts of the environment.

Thus, an environment mat technique considering how the light of the surrounding environment is refracted and reflected by the placed object is introduced and expanded to the environment equation as shown in the following [Equation 2].

[Mathematics 2]

C = F + (1 -?) B +?

Where? Represents the contribution of the light of the surrounding environment refracted or reflected through the displaced object.

At this time, the ambient environment is defined as light E (?) Coming from all directions?, And the sum? Of light emitted at a portion f of the placed object shown by a given pixel is calculated from the surrounding environment contributing to the pixel point p All incoming light can be expressed by integrating it, which is also reduced by a certain reflectance, R (ω → p), and can be expressed by the following equation (3).

&Quot; (3) "

Φ = ∬R (ω → p) E (ω) dωdp

Also, it can be seen that the reduction function R (? -?) Includes all the effects of absorption and dispersion, and that?, R, and E all depend on the wavelength of light.

Since the reflectivity function E is a constant passing through a region of a given pixel, it is possible to simplify the equation as shown in Equation (4) by using a new function R (?) That is independent of the position in the pixel.

&Quot; (4) "

陸 = ∫ R (ω) E (ω) dω

Here, if the integration is simplified by using the texture map T _i (x) composed of a set of m, which represents a texture made up of different parts of the surrounding environment, it can be expressed as the following equation (5).

&Quot; (5) "

Here, in order to simplify the equation, it is assumed that the contribution from the texture map T _i can be approximated by a certain constant K _i times the amount of total light from the rectangular area A _i aligned on the axis.

The most standard texture-mapping technique actually calculates the average value of texture areas aligned on an axis. Therefore, since R _i = K _i A _i , it can be expressed as the following equation (6).

&Quot; (6) "

From this, the finally obtained environment mat equation is expressed by the following equation (7).

&Quot; (7) "

Referring to FIG. 1 and FIG. 2, a method of improving night image pickup performance according to the present invention includes an output image storage step S300.

The output image storing step S300 is a step of the control unit 310 storing the output image generated through the output image forming step S200 in the storage unit 320. In response to the user's selection, And output it through the video output unit 330.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention as defined in the appended claims. It can be understood that it is possible.

100: low light camera 200: thermal image camera
310: Control section 320:
330: video output unit 400: laser distance meter
500: Pan-tilt driving device 600: Position shifting device

Claims (5)

Adjusting the size of the object to be photographed by aligning the front view angle with the other one of the low-illuminance camera and the thermal image camera or adjusting the front view angle of the low-illuminance camera and the thermal image camera based on the distance to the object;
A low-illuminance camera equipped with an electronically amplifying type CCD is used to collect color images in an environment of 1 to 0.2 lux, and an optical image at night is acquired for a region to be photographed, and a thermal image is acquired and stored through a thermal image camera An image acquisition step;
A matching data extracting step of extracting color data of an object to be photographed from the optical image and extracting contour data of the object to be photographed from the thermal image, and a step of digitally synthesizing the color data and the contour data, An output image constructing step of synthesizing a digital image; And
And an output image storing step of storing the output image. delete delete delete 2. The method according to claim 1,
Calculating a distance value of an object to be imaged by irradiating a laser to a region to be imaged with a laser distance measuring instrument,
Calculating a position value and a pan-tilt value of the low-illuminance camera and the thermal imaging camera in order to match the front viewing angle according to the distance value, and
The control unit controls the low-illuminance camera and the position-moving device installed in the thermal image camera according to the position value, and controls the pan-tilt driving device installed in the low-illuminance camera and the thermal image camera according to the pan-tilt value. How to improve night photography performance.

Images