JP6221682B2 - Image processing apparatus, imaging system, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus that corrects an image to remove disturbance such as fog and yellow sand, an imaging system including the imaging apparatus and the image processing apparatus, an image processing method, and a program for causing a computer to execute the method.

  Surveillance cameras are used to monitor the amount of water in the dam, check the status of getting on and off the station platform, monitor the store, airport and school. This surveillance camera is also used when detecting an intruder into a house or the like, detecting a suspicious object, face authentication of a criminal or the like by image processing.

  When a surveillance camera is installed outdoors, the contrast (brightness / darkness difference) of the image decreases if an external disturbance such as fog or yellow sand occurs that affects the captured image. There was a problem that the image processing failed.

  In view of this problem, a visible light image and an infrared light image are acquired from a lens, and a dark portion in which the contrast is reduced due to fog in the visible light image is replaced with an infrared light image, thereby correcting the image with clear light and darkness. A technique has been proposed (see Patent Document 1).

  Also, it is not intended to remove disturbances such as fog and yellow sand, but by acquiring brightness from infrared light images and saturation from visible light images, it is possible to acquire sharp color images even during long-distance shooting. A technique that can be used has also been proposed (see Patent Document 2).

  In addition to contrast, the captured image has reduced saturation (color) due to disturbance. In the above-described conventional technique, since the process of restoring the saturation is not performed, the color information is lost when compared with an image captured before a disturbance occurs.

  Also, when a disturbance occurs, the sharpness (sharpness) of the image is also lost. However, since the above-mentioned conventional technique has not restored this sharpness, it is compared with an image taken before the disturbance occurs. When you look at it, it becomes a blurred image.

  Visible light is refracted by fog or the like, but infrared light is hardly refracted even if fog or the like is generated. For this reason, if a visible light image and an infrared light image are combined as they are, the position is shifted. However, the above conventional technique does not consider that this position is shifted. This reduces the accuracy of image processing such as detection and face authentication.

  Therefore, it is desired to provide an apparatus and a method that can restore saturation and sharpness and correct an image from which disturbance has been removed in consideration of positional deviation.

  In view of the above problems, the present invention is an image processing device for processing an image picked up by at least one image pickup device, and obtains an infrared light image and a visible light image as an image from the at least one image pickup device. The image acquisition unit, the image correction unit that performs saturation correction on the visible light image acquired by the image acquisition unit, the lightness information of the infrared light image acquired by the image acquisition unit, and the image correction unit Using the saturation information and hue information of the visible light image thus corrected, the misalignment between the infrared light image and the corrected visible light image is corrected, and the infrared light image and the visible light after correcting the misalignment are corrected. An image processing apparatus is provided that includes a composite image generation unit that generates a composite image by combining the images.

  By providing the apparatus, system, method, and program of the present invention, it is possible to restore saturation and sharpness, and to correct the image from which disturbance has been removed in consideration of misalignment. The accuracy of image processing can be improved.

The figure which showed schematic structure of the imaging system of this embodiment. The figure which showed the 1st hardware constitutions of the imaging device shown in FIG. The figure which showed the 2nd hardware constitutions of the imaging device shown in FIG. The figure which showed the 3rd hardware constitutions of the imaging device shown in FIG. The figure which showed the 4th hardware constitutions of the imaging device shown in FIG. The figure which illustrated the hardware constitutions of the image processing apparatus shown in FIG. 7 is a functional block diagram of the image processing apparatus shown in FIG. The figure explaining the saturation correction | amendment in RGB color space matched with the factor of atmospheric disturbance. The figure explaining the saturation correction | amendment in another color space. The figure explaining edge detection. The figure explaining hue division. The figure explaining parallax. The figure explaining the bad influence by atmospheric disturbance. The figure explaining the synthesis | combination of an infrared-light image and a visible light image. The figure explaining the 1st example of position shift correction. The figure explaining the 2nd example of position shift correction. The figure explaining the 3rd example of position shift correction. The figure explaining the 4th example of position shift amendment. The figure explaining the 5th example of position shift correction. The figure explaining the 6th example of position shift correction. The figure explaining the 7th example of position shift correction. 2 is a flowchart showing a flow of image processing executed by the image processing apparatus shown in FIG. 1.

  FIG. 1 is a diagram illustrating a configuration example of an imaging system including an imaging apparatus and an image processing apparatus according to the present embodiment. The imaging system shown in FIG. 1 includes two cameras 10 and 11 as imaging devices and a PC 12 as an image processing device. Here, the PC 12 is used as the image processing apparatus, but it is also possible to mount an image processing engine on the cameras 10 and 11 and use the image processing engine as the image processing apparatus. In addition, it is possible to use a device other than the PC 12 as an apparatus for performing image processing. The cameras 10 and 11 and the PC 12 are directly connected via a cable or via a network, and image data of captured images can be transmitted from the cameras 10 and 11 to the PC 12.

  When the cameras 10 and 11 and the PC 12 are connected, they may be connected by either wired connection or wireless connection. For wireless connection, a wireless LAN such as Bluetooth (registered trademark) or Wi-Fi can be used. When using a wireless LAN, communication can be performed via a relay device that relays wireless communication called an access point connected to a network.

  The camera 10 is an infrared camera that acquires an image (infrared light image) formed by infrared light in the infrared wavelength region. The camera 11 is a visible light camera that acquires an image (visible light image) formed by visible light in the visible light wavelength region. The detailed configuration of these cameras will be described later. In addition, although the structure which uses the two cameras 10 and 11 is illustrated in FIG. 1, as long as both an infrared light image and a visible light image can be acquired, you may use one camera. .

  The PC 12 receives the image data of the infrared light image and the image data of the visible light image transmitted from the two cameras 10 and 11, and performs image processing using the image data. In this way, what is actually exchanged is image data. However, in order to facilitate the description, it will be described as an image.

  In the image processing in the PC 12, each image is corrected, the position of each image is adjusted, the two images are combined, and a combined image is output. The output composite image is collated with, for example, a plurality of images registered in the database, and used for image processing such as intruder detection, suspicious object detection, and criminal face authentication described above. This database may be provided separately or may be mounted on the PC 12.

  The hardware configuration of the cameras 10 and 11 will be described with reference to FIGS. In FIG. 2, the cameras 10 and 11 include lenses 20 and 21 to which light is input and sensors 22 and 23 that receive light that has passed through the lenses 20 and 21. Although not shown, the cameras 10 and 11 may further include a diaphragm disposed in the vicinity of the lenses 20 and 21, a shutter disposed between the diaphragm and the sensors 22 and 23, and the like.

  The lenses 20 and 21 may be single focal lenses having a fixed focal length, or may be zoom lenses capable of continuously changing the focal length. The sensor 22 can be an RGrGbB sensor using a CCD image sensor or a CMOS image sensor and a color filter.

  In FIG. 2, the green (Gr, Gb) pixels are set to N / 2 and the red (R) and blue (B) pixels are set to N / 4 with respect to the total number N of pixels of the sensor 22. Bayer-arranged sensors arranged every other in the scanning direction are used. The sensor 22 receives light in the visible region (visible light) and converts it into an electrical signal (Bayer conversion), thereby obtaining a visible light image of the three primary colors RGB. Here, the main scanning direction is, for example, the horizontal direction (the direction from left to right) toward the image, and the sub-scanning direction is the vertical direction (the direction from top to bottom) perpendicular to the main scanning direction. It is.

  The sensor 23 may be a dedicated infrared sensor (Ir sensor) that receives light in the infrared region (infrared rays) and converts the light into an electrical signal. An infrared light image can be acquired by converting to an electrical signal by the Ir sensor. The electrical signals output from these sensors 22 and 23 are analog signals. Therefore, an analog signal can be converted into digital data using an A / D converter, and the digital data can be acquired as image data of a visible light image or image data of an infrared light image.

  FIG. 2 illustrates a configuration in which two cameras are used to separately acquire an infrared light image and a visible light image. If two cameras are used, the cost of the apparatus is increased. Therefore, if one of the two images can be acquired by one camera, one is preferable. FIG. 3 shows an example of a configuration that uses only one camera 10 and acquires two images. In this configuration, in addition to the lens 20 and the sensors 22 and 23, three mirrors 24 to 26, an infrared light cut filter 27, and a visible light cut filter 28 are provided.

  The mirror 24 has two mirrors connected in a V shape so that incident light can be separated in two directions. For this reason, after the light incident on the lens 20 is separated into light in two directions by the mirror 24, one is sent to the infrared light cut filter 27 through the mirror 25, and the other through the mirror 26. It is sent to the visible light cut filter 28. The infrared light cut filter 27 blocks the transmission of incident light in the infrared region (wavelength of about 0.7 μm to about 1 mm), and visible light having a shorter wavelength (wavelength of about 380 nm to about 810 nm). ). Visible light transmitted through the infrared light cut filter 27 is received by the sensor 22 and converted into an electrical signal. And it converts with an A / D converter and a visible light image is acquired as image data.

  The visible light cut filter 28 blocks visible light from the incident light and transmits infrared light having a longer wavelength. The infrared light transmitted through the visible light cut filter 28 is received by the sensor 23 and converted into an electrical signal. And it converts with an A / D converter and an infrared-light image is acquired as image data.

  As a configuration using only one camera, a configuration as shown in FIG. 4 or FIG. 5 can be adopted. In the configuration shown in FIG. 4, only the lens 20 and the sensor 29 are used. The sensor 29 is a special sensor, and R, G, B, and Ir are mounted in one sensor, not a Bayer array, and a visible light image and an infrared light image can be acquired by this one sensor. It has become. However, since the sensor 29 has substantially a 1/4 pixel number as compared with the dedicated Ir sensor, the resolution is substantially 1/4.

  The configuration shown in FIG. 5 includes a lens 20, sensors 22 and 23, an infrared light cut filter 27, a visible light cut filter 28, and a mirror 30 that are switched and used in units of frames. 2 to 4, the infrared light image and the visible light image can be acquired at the same timing. However, in the configuration shown in FIG. I can't do it.

  Therefore, at a certain timing, the infrared light cut filter 27 is used to block the transmission of infrared light, transmit visible light, and enter the sensor 22. Thereby, in the sensor 22, the incident light is converted into an electric signal, and a visible light image is acquired. At the next timing, the visible light cut filter 28 and the mirror 30 are used to block transmission of visible light, transmit infrared light, and enter the sensor 23 through the mirror 30. Thereby, in the sensor 23, the incident light is converted into an electric signal and an infrared light image is acquired. For example, the visible light image and the infrared light image can be acquired by using the infrared light cut filter 27 for the even frame and the visible light cut filter 28 and the mirror 30 for the odd frame.

  In the configuration shown in FIG. 5, only one image can be obtained for each of the two frames. Therefore, in order to obtain the same shooting speed as the configuration shown in FIGS. I need to shoot. The configurations shown in FIGS. 2 to 5 are only examples, and any configuration other than this may be adopted as long as a visible light image and an infrared light image can be acquired. Also good.

  FIG. 6 is a diagram illustrating a hardware configuration of the PC 12. The PC 12 acquires the infrared light image and the visible light image output from the cameras 10 and 11 as image data, performs image processing on the two images, and combines them to generate a combined image. For this reason, a storage device in which a program for executing these processes is stored, and a CPU 40 that reads and executes the program from the storage device are provided. The storage device includes a ROM 41 that stores a boot program, firmware, and the like, a RAM 42 that provides a work area to the CPU 40, and an HDD 43 that stores the above-described program, OS, and other applications. An SSD can be used instead of the HDD 43.

  The PC 12 can include an input / output interface 44 for connecting to the cameras 10 and 11 by a cable or the like, and a communication interface (not shown) that enables communication via a network. In addition, the PC 12 includes an external storage interface 46 that enables reading and writing to the recording medium 45, a display device 47 for displaying processing statuses, an input device 48 for inputting various data and setting values, selecting icons, and the like. Can be provided. As the input device 48, a keyboard, a mouse, or the like can be used.

  The CPU 40, ROM 41, RAM 42, HDD 43, input / output interface 44, etc. are connected to a bus 49 so that data can be exchanged with each other via the bus 49.

  FIG. 7 is a functional block diagram of the PC 12. When the CPU 40 reads out and executes the program stored in the HDD 43, it can function as the following functional units. The PC 12 functions as an image processing device for processing an image captured by at least one imaging device such as the cameras 10 and 11. The PC 12 includes an image acquisition unit 50 that acquires an infrared light image and a visible light image from the cameras 10 and 11, and an image correction unit 51 that performs saturation correction on the visible light image acquired by the image acquisition unit 50. A composite image generation unit 52.

  The composite image generation unit 52 uses the lightness information of the infrared light image and the saturation information and hue information of the corrected visible light image to shift the position of the infrared light image from the corrected visible light image. Correct. Then, the composite image generation unit 52 generates a composite image by combining the infrared light image and the visible light image after correcting the positional deviation.

  In addition, the PC 12 can include a filter unit 53, an edge detection unit 54, and a hue division unit 55. These functional units can be provided as necessary. The filter unit 53 performs filter processing, removes noise from the input image, and performs edge enhancement. The noise removal is a process for removing noise generated due to a defect of an image sensor provided in the cameras 10 and 11. For example, noise can be removed using a moving average filter, a median filter, or the like. Edge enhancement is a process for sharpening an image by making the density gradient of the image contour steep. For this process, for example, an unsharp mask or the like can be used. These processes are well-known processes and will not be described in detail here.

  The edge detection unit 54 detects an edge where the brightness of the image is steep, that is, discontinuously changes. For example, it can be determined whether it is an edge by determining whether the brightness exceeds a threshold. This edge detection can be performed using, for example, a Laplacian filter, a Sobel filter, a Prewitz filter, or the like.

  The hue dividing unit 55 divides the hue of the visible light image after the saturation correction, and determines which hue is present for each pixel. This hue may be, for example, 8 divisions or 16 divisions. The number of divisions can be set arbitrarily.

  The image acquisition unit 50 acquires an infrared light image and a visible light image from the cameras 10 and 11. Here, the characteristics of each image will be described. Infrared light images are less susceptible to disturbances such as fog and yellow sand, have the advantages of high sharpness and high contrast, but have the disadvantage of becoming monochrome images. Visible light images are susceptible to disturbance, sharpness is likely to be reduced, and contrast is likely to be reduced. Since the visible light image is a color image, it has a certain degree of saturation. However, when a disturbance occurs, the saturation and the like decrease due to the influence of the disturbance.

  The image acquisition unit 50 outputs the acquired infrared light image to the filter unit 53, and the filter unit 53 performs a filtering process on the infrared light image. In the filter processing, the above-described noise processing and edge enhancement are performed. After performing the filtering process, the filter unit 53 outputs the infrared image after the filtering process to the edge detection unit 54. The filter unit 53 also outputs the infrared light image after the filtering process to the composite image generation unit 52.

  The edge detection unit 54 performs edge detection on the infrared light image. In the edge detection, it is determined for each pixel of the infrared light image whether it is an edge or a non-edge, and is recorded as 1-bit edge information. The edge information can be recorded by being stored in a storage device such as the RAM 42 shown in FIG. The edge detection unit 54 can output the recorded edge information to the composite image generation unit 52. Note that the edge information may be in a form acquired by the composite image generation unit 52 instead of being output by the edge detection unit 54 to the composite image generation unit 52.

  The image acquisition unit 50 passes the acquired visible light image to the image correction unit 51, and the image correction unit 51 performs saturation correction on the visible light image. In the saturation correction, the saturation of the visible light image that is lowered due to the influence of disturbance is corrected, and the saturation is improved. Saturation correction can be performed using a LUT as a correction parameter after conversion to another color space called YUV in the case of a visible light image having the three primary colors of RGB, or converted to YUV. It can also be carried out in the masking process.

  Here, YUV is a color space that expresses a color by three elements: a luminance signal (Y), a difference signal (U) between the luminance signal and the blue component, and a difference signal (V) between the luminance signal and the red component. The LUT is a table describing color conversion sources and conversion destinations. The masking process is a technique for color conversion, and is a technique for obtaining an output color space by performing a matrix operation on the input color space.

  In the saturation correction, the LUT can be switched according to the type of disturbance such as fog or yellow sand. Details thereof will be described later. Then, the hue division unit 55 divides the hue of the visible light image after the saturation correction. By this hue division, the hue is determined according to the balance of the two components (UV version) of UV among the three components YUV related to the color of the visible light image. The hue division unit 55 outputs the hue information thus obtained and the visible light image to the subsequent composite image generation unit 52.

  The composite image generation unit 52 combines the infrared light image and the visible light image after performing positional deviation correction that matches the position coordinates. The composite image has a YUV version, where Y is a Y version of an infrared light image output from the filter unit 53, and UV is a UV version of a visible light image converted into YUV. At the time of synthesis, first, the positional relationship (parallax) of the lens is corrected. Even if this parallax correction is performed, there is a high possibility that the positions of the pixels of the infrared light image and the pixels of the visible light image are shifted due to the influence of refraction of the disturbance. Incidentally, infrared light hardly refracts. If the YUV does not overlap, it looks like an abnormal image. Therefore, the position of the edge in the infrared light image and the hue boundary in the visible light image are aligned and combined.

  By generating a composite image by synthesizing in this way, an image with high sharpness, contrast, and saturation can be output even under the influence of disturbance such as fog or yellow sand. Hereinafter, the contents of each process executed by each functional unit will be described in detail one by one in order.

  FIG. 8 is a diagram for explaining saturation correction in the RGB color space according to the cause of atmospheric disturbance as disturbance. Here, processing using an LUT will be described as an example of saturation correction. FIG. 8A is a diagram for explaining saturation correction when fog occurs as an example of atmospheric disturbance, and FIG. 8B is a diagram for explaining saturation correction when yellow sand occurs. is there.

  The horizontal axis of the graph shown in FIG. 8A indicates the input pixel value, and the vertical axis indicates the output pixel value. The straight line shown in the graph indicates the relationship between input and output pixel values when saturation correction is not performed. The curve represents the LUT set for each color when fog occurs. For example, it is assumed that the input pixel value before saturation correction is 100 for all of R, G, and B. When saturation correction is not performed, a vertical line is drawn upward from a position corresponding to 100 on the horizontal axis, and the vertical axis coordinate of the intersection with the straight line becomes the pixel value of the output. That is, the output pixel value is (R, G, B) = (100,100,100).

  On the other hand, when saturation correction is performed, a vertical line is drawn upward from a position corresponding to 100 on the horizontal axis, and the vertical axis coordinate of the intersection with each curve becomes the pixel value after saturation correction. That is, the output pixel value is (R, G, B) = (R100, G100, B100).

  When fog is generated, light is refracted, scattered, and absorbed by the fog, and the influence is more likely as the wavelength of the light is shorter. In the RGB version of the visible light image, it is easily affected by fog in the order of B, G, and R. For this reason, the saturation of the B version tends to decrease, but the saturation of the R version does not easily decrease. Further, the low saturation portion and the high saturation portion indicated by the oblique lines in FIG. 8A tend to be relatively less susceptible to fog. Therefore, the shape of the curve representing the LUT is changed for each RGB color, so that the saturation of the intermediate saturation portion that is easily affected is improved. The shape of the LUT is not limited to the shape shown in FIG. 8A, and may be another shape. Here, the shape of the LUT is changed for each color, but the same shape may be used.

  In the graph shown in FIG. 8B, the horizontal axis indicates the input pixel value, and the vertical axis indicates the output pixel value. The straight line shown in the graph indicates the relationship between input and output pixel values when saturation correction is not performed. The curve represents the LUT set for each color when yellow sand is generated.

  When yellow sand is generated, light is refracted, scattered, and absorbed by the yellow sand. Blue (B), which is a yellow complementary color that is the main component of yellow sand, is strongly affected by this, and the saturation is lowered. In contrast, green (G) and red (R) are not easily affected. Therefore, only the B plate has an LUT shape in which the saturation is greatly increased, and the G plate and R plate have an LUT shape in which the saturation is not extremely increased. Again, the shape of the LUT is not limited to the shape shown in FIG. 8B, but may be other shapes. Here, the shape of the LUT is changed for each color, but the same shape may be used. In FIG. 8, fog and yellow sand are taken as an example of the atmospheric disturbance, but it is also possible to apply to other atmospheric disturbances.

  FIG. 9 is a diagram for explaining saturation correction in another color space. In FIG. 8, saturation correction is performed in the RGB color space. In FIG. 9, however, RGB is converted into the YUV color space using a general conversion formula, and processing is applied to the Y version that is lightness. Alternatively, only the UV version having the saturation and hue may be processed. If the atmospheric disturbance is fog, increase the slope of the U version LUT that expresses blue (B) that is susceptible to fog, and increase the slope of the V version LUT that expresses red (R) that is less susceptible to fog. It is small. Here, the YUV color space is taken as an example, but saturation correction may be performed in another color space, and the slope of the LUT can be freely changed. Further, the disturbance may be yellow sand other than fog or other things.

  Since the saturation correction has been described so far, the edge detection will be described in detail with reference to FIG. Edge detection of an infrared image captured by an infrared camera with the subject “A” is an image obtained by applying general-purpose filter processing such as edge enhancement and noise removal to the infrared image. Do about. FIG. 10A shows an infrared light image after the filtering process. This infrared light image is an image having no saturation (saturation 0) and composed only of lightness information.

  FIG. 10B is a diagram showing a Laplacian filter as an example of a filter used for performing edge detection on the infrared light image shown in FIG. The Laplacian filter is a general filter, and the coefficients and matrix size are not limited to these, and other coefficients and matrix sizes may be used. Further, the filter is not limited to the Laplacian filter, and other filters capable of performing edge detection such as a Sobel filter can also be used.

  FIG. 10C is a diagram illustrating a result of edge detection performed on the infrared light image illustrated in FIG. 10A using the filter illustrated in FIG. In the edge detection, matrix calculation is performed using the filter, it is determined whether or not a preset threshold value is exceeded, a portion exceeding the threshold value is determined as an edge, and the edge is detected. In FIG. 10C, a portion detected as an edge is represented by a white line.

  The hue division will be described in detail with reference to FIG. The hue division is performed on the image after the saturation correction is performed on the visible light image. In this hue division, the image is divided to determine which hue is in each pixel. In FIG. 11, the visible light image after the saturation correction is converted into the YUV version, and is divided into eight hues using the UV version information that is the saturation information. In FIG. 11, with V and U as axes, the comparison result of V and U and the determination result of positive and negative are used to divide the hue into (1) to (8). The result of the division is held as 3-bit hue information in units of pixels, and this hue information is output to the composite image generation unit 52.

  Hue division is not limited to eight divisions as shown in FIG. 11, and the number of divisions can be freely set, such as 16 divisions or 24 divisions, and the division method can also be set freely. Can do. Further, the color space to be divided is not limited to YUV, and hue division may be performed in another color space such as RGB.

  When two cameras are used for simultaneous imaging and an infrared light image and a visible light image are acquired, the position of each camera is different from the subject as shown in FIG. The shooting range is changed. For this reason, parallax occurs in the image obtained by imaging as shown in FIG. This parallax can be removed using a generally known parallax correction technique. Specifically, it is possible to obtain an image as shown in FIG. 12C by performing correction using information such as the shooting range of the camera, the distance between the cameras, and the distance to the subject. As described above, by aligning the position of the image of the subject in units of pixels, it is possible to easily and accurately perform image composition in later composition processing.

  The parallax correction may be performed when images are simultaneously captured using two cameras as shown in FIG. For this reason, it is not necessary to perform parallax correction when taking an image using one camera or using two cameras instead of simultaneously taking an image and placing the camera at the same position. . Further, even when two cameras are used to capture images simultaneously, it may be performed only when the parallax is large, and when the parallax is small, only positional deviation correction described later may be performed.

  With reference to FIG. 13, the reason why an image is adversely affected by atmospheric disturbance will be described. Here, fog will be described as an example of atmospheric disturbance. FIG. 13A shows one water droplet constituting the mist, and shows how light is refracted in the water droplet. Since the water droplet has a density different from that of the surrounding air, refraction as shown in FIG. 13A occurs when light passes through the water droplet. The refractive index varies depending on the wavelength of light, and the longer the wavelength, the harder it is to refract. For this reason, infrared light having a long wavelength hardly refracts, and blue (B) having a longer wavelength tends to be refracted. In this example, the ease of refraction is in the order of B, G, B, and infrared light.

  FIG. 13B is a diagram showing the influence of the water droplets gathering to form a mist, and the mist affecting the imaging by the camera. In FIG. 13A, light is only refracted by water droplets, but in FIG. 13B, reflection, absorption, and scattering of light also occur. Thus, the refraction, reflection, absorption, and scattering of light cause the image of the subject that normally enters the camera to blur, cause color shift, and reduce sharpness. Such blurring or the like causes a decrease in contrast, saturation, and sharpness.

  As shown in FIG. 13A, infrared light hardly refracts even in the case of fog, so that reflection, absorption, scattering, and the like hardly occur. For this reason, there is almost no decrease in contrast or sharpness. However, since the color information is missing, the saturation is zero.

  FIG.13 (c) is the figure which showed the infrared-light image and visible light image which were obtained by imaging in the state where fog has not generate | occur | produced. Note that these images are images after performing parallax correction. Since these images are not affected by fog, almost no refraction occurs even in the visible light, and the image of the subject in the infrared light image after the parallax correction and the visible light image are almost at the same position, and there is a displacement. Absent.

  FIG. 13D is a diagram showing an infrared light image and a visible light image obtained by imaging in a state where fog is generated. Infrared light is less likely to be refracted even in the presence of fog, so the subject image in the infrared light image is located at the center, but visible light is refracted, so the subject image is deviated from the center. In addition, since the influence of refraction is different when the color is different, the displacement of the position is different at a place where the color is different (a halftone dot and a hatched portion in FIG. 13D).

  Such a misalignment is corrected, and a combined image is generated by combining the infrared light image and the visible light image after alignment, and the combining process will be described in detail with reference to FIG. Here, fog will be described as an example of atmospheric disturbance. FIGS. 14A and 14B are diagrams illustrating an infrared light image and a visible light image captured in a state where fog is not generated. Since these images have a substantially centered position of the subject image, and it is not necessary to correct misalignment or the like, they can be combined as they are to generate a combined image. Then, the composite image becomes an image as shown in FIG.

  In image synthesis, first, the color space of an infrared light image is converted into a YUV color space, and only Y (lightness) information is extracted. Next, the color space of the visible light image is also converted to the YUV color space, and only UV (saturation / hue) information is extracted. Then, using the lightness information (Y value) of the infrared light image and the saturation information and hue information (UV value) of the saturation-corrected visible light image, a combined YUV image is generated as a composite image. ,Output.

  When fog is not generated, as shown in FIG. 14C, there is no YUV misalignment, and therefore the edge position and the hue boundary that represents the location where the color changes coincide. Therefore, a synthesized image can be generated by synthesizing the position and the hue boundary so as to overlap each other. The infrared light image used for composition is filtered, and the visible light image is an image that has been subjected to saturation correction. It is said.

  14D and 14E are diagrams illustrating an infrared light image and a visible light image captured in a state where fog is generated. FIG. 14D is the same as the state shown in FIG. 13D, and the positions of the subject image in the infrared light image and the subject image in the visible light image are shifted. For this reason, when YUV is extracted and combined, the edge boundary (solid line) and the hue boundary of the visible light image are shifted as shown in FIG. In order to correct this deviation, the position deviation is corrected.

  With reference to FIG. 15, the correction of the positional deviation will be described in detail. FIG. 15 shows an example of correcting the positional deviation in the main scanning direction, which is the lateral direction, as indicated by the arrow. For this reason, only misregistration correction in the main scanning direction will be described, but misregistration in the sub-scanning direction perpendicular to the main scanning direction can also be corrected by a similar method.

  When correcting the positional shift in the main scanning direction, as shown in FIG. 15A, the upper left corner toward the image is set as a reference point (for example, coordinates (0, 0)), and the reference point is moved from the reference point to the main scanning direction. Scan one line at a time. At this time, the edge detection result and the result of the hue division are referred to for each pixel. As shown in FIG. 15B, the edge detection result is a portion indicated by a white line as to whether the edge portion is an edge portion or a non-edge portion for each pixel of the line being scanned. It is the information shown. When a plurality of pixels determined to be edge portions are adjacent to each other on the left and right, the adjacent blocks are handled as one edge portion.

  As a result of the hue division, the region is distinguished for each color as shown in FIG. Here, a distinction is made between three areas: a white background, a halftone dot, and a diagonal line. A portion where the region switches to another region is set as a hue boundary.

  The positional deviation is corrected on a pixel-by-pixel basis based on whether or not the pixel determined as the edge portion matches the position of the pixel determined as the hue boundary. For example, the positional deviation correction can be performed when the position of the pixel determined to be the edge portion and the position of the pixel determined to be the hue boundary are greater than or equal to the threshold th1 and less than the threshold th2. The thresholds th1 and th2 can be set arbitrarily.

  A specific procedure for correcting misregistration will be described with reference to FIGS. These examples all show the case of misalignment correction in the main scanning direction, and is a case of scanning from the left side to the right side toward the image. The scanning direction may be performed from the opposite right side to the left side. Further, the positional deviation correction in the sub-scanning direction can be performed in the same manner by scanning from the upper side to the lower side in the direction.

  FIG. 16 is a diagram illustrating the first example, in which an edge, a hue boundary, an edge, and a hue boundary are alternately detected when detecting an edge position or a hue boundary in the main scanning direction. FIG. One edge and one hue boundary are processed as a set. The same applies to the subsequent processing.

  As shown in FIG. 16A, in this example, the edge and the hue boundary are detected in the order of edge 1, hue boundary 1, edge 2, and hue boundary 2. First, the edge 1 and the hue boundary 1 are set (one set). The UV value of the pixel (halftone dot) immediately to the right of hue boundary 1 is recorded. As shown in FIG. 16B, in order to align the hue boundary 1 with the edge 1, for all pixels between the edge 1 and the hue boundary 1, all the UV values are replaced with the recorded UV values. . Accordingly, as shown in FIG. 16C, the UV values of all the pixels between the edge 1 and the hue boundary 1 become the recorded UV values. As a result, the edge 1 and the hue boundary 1 are in the same position, and there is no sense of incongruity and color shift.

  Next, edge 2 and hue boundary 2 are set. The UV value of the pixel (halftone dot) immediately to the right of hue boundary 2 is recorded. As shown in FIG. 16C, all the UV values of the pixels between the edge 2 and the hue boundary 2 are replaced with the recorded UV values. Thereby, as shown in FIG. 16D, the UV values of all the pixels between the edge 2 and the hue boundary 2 become the recorded UV values. Then, the edge 2 and the hue boundary 2 are in the same position, and there is no apparent discomfort and color shift.

  If the edge and the hue boundary are alternately continued, the same process may be repeated thereafter. When the set edge and the hue boundary are repeated in the order of the edge and the hue boundary, the UV value of the pixel between the position of the edge and the hue boundary is replaced with the UV value of the pixel on the right side of the hue boundary.

  FIG. 16 shows an example in which the edges are detected alternately in the order of the hue boundary, but FIG. 17 shows an example in which the detection is alternately performed in the order of the hue boundary and the edges. In this example, as shown in FIG. 17A, edges and hue boundaries are detected in the order of hue boundary 1, edge 1, hue boundary 2, and edge 2. First, edge 1 and hue boundary 1 are set. Record the UV value of the pixel (hatched) immediately to the left of Hue Boundary 1. As shown in FIG. 17B, in order to match the hue boundary 1 to the edge 1, all the UV values for the pixels between the edge 1 and the hue boundary 1 are replaced with the recorded UV values. . Thus, as shown in FIG. 17C, the UV values of all the pixels between the edge 1 and the hue boundary 1 become the recorded UV values. Then, the edge 1 and the hue boundary 1 are in the same position, and there is no apparent discomfort or color shift.

  Next, edge 2 and hue boundary 2 are set. The UV value of the pixel (halftone dot) immediately to the left of the hue boundary 2 is recorded. As shown in FIG. 17C, for all pixels between the edge 2 and the hue boundary 2, all the UV values are replaced with the recorded UV values. Thereby, as shown in FIG. 17D, the UV values of all the pixels between the edge 2 and the hue boundary 2 become the recorded UV values. Then, the edge 2 and the hue boundary 2 are in the same position, and there is no apparent discomfort or color shift.

  Also in this case, when the edge and the hue boundary are alternately continued, the same process can be repeated thereafter to correct the positional deviation. Further, when the set edge and the hue boundary are alternately detected in the order of the hue boundary and the edge, the pixel between the edge position and the hue boundary is changed to the UV value of the pixel on the right side of the hue boundary. Replace the UV value of.

  16 and 17, the case where the edge and the hue boundary are alternately continued has been described. Actually, in addition to being alternately continuous, there may be cases where discontinuity is detected, such as an edge, a hue boundary, a hue boundary, and an edge. Hereinafter, the case where such discontinuity is detected will be described.

  FIG. 18 is a diagram illustrating an example in which an edge and a hue boundary such as an edge, a hue boundary, a hue boundary, and an edge are switched halfway. As shown in FIG. 18A, in this example, edges and hue boundaries are detected in the order of edge 1, hue boundary 1, hue boundary 2, and edge 2. First, edge 1 and hue boundary 1 are set. The UV value of the pixel (halftone dot) immediately to the right of hue boundary 1 is recorded. As shown in FIG. 18B, in order to align the hue boundary 1 with the edge 1, all the UV values for the pixels between the edge 1 and the hue boundary 1 are set to the recorded UV values. replace. Thereby, as shown in FIG. 18C, the UV values of all the pixels between the edge 1 and the hue boundary 1 become the recorded UV values. Then, the edge 1 and the hue boundary 1 are in the same position, and there is no apparent discomfort or color shift.

  Next, edge 2 and hue boundary 2 are set. The UV value of the pixel (halftone dot) immediately to the left of the hue boundary 2 is recorded. As shown in FIG. 18C, all the UV values of the pixels between the edge 2 and the hue boundary 2 are replaced with the recorded UV values. Accordingly, as shown in FIG. 18D, the UV values of all the pixels between the edge 2 and the hue boundary 2 become the recorded UV values. Then, the edge 2 and the hue boundary 2 are in the same position, and there is no apparent discomfort or color shift.

  Even in such a discontinuous state, it is possible to perform positional deviation correction by setting the edge and the hue boundary as a set and performing the same processing in consideration of the order.

  FIG. 19 is a diagram illustrating an example in which the edge and the hue boundary are switched in the order of the hue boundary, the edge, the edge, and the hue boundary, contrary to the embodiment illustrated in FIG. As shown in FIG. 19A, in this example, edges and hue boundaries are detected in the order of hue boundary 1, edge 1, edge 2, and hue boundary 2. First, edge 1 and hue boundary 1 are set. Record the UV value of the pixel (hatched) immediately to the left of Hue Boundary 1. As shown in FIG. 19 (b), in order to match the hue boundary 1 to the edge 1, for all pixels between the edge 1 and the hue boundary 1, all the UV values are the recorded UV values. replace. Accordingly, as shown in FIG. 19C, the UV values of all the pixels between the edge 1 and the hue boundary 1 become the recorded UV values. Then, the edge 1 and the hue boundary 1 are in the same position, and there is no apparent discomfort or color shift.

  Next, edge 2 and hue boundary 2 are set. Record the UV value of the pixel (horizontal stripe) just to the right of hue boundary 2. As shown in FIG. 19C, for all pixels between the edge 2 and the hue boundary 2, all the UV values are replaced with the recorded UV values. Thereby, as shown in FIG. 19D, the UV values of all the pixels between the edge 2 and the hue boundary 2 become the recorded UV values. Then, the edge 2 and the hue boundary 2 are in the same position, and there is no apparent discomfort or color shift.

  FIG. 18 and FIG. 19 have described an example in which the arrangement order of the set of edges and hue boundaries is changed in the middle. As an example of discontinuous detection, when there are three or more consecutive edges or hue boundaries such as edges, hue boundaries, hue boundaries, hue boundaries, and edges, and the edges and hue boundaries cannot be set as a set There is also.

  FIG. 20 is a diagram illustrating an example in which three hue boundaries such as an edge, a hue boundary, a hue boundary, a hue boundary, and an edge are detected in the middle. As shown in FIG. 20A, in this example, edges and hue boundaries are detected in the order of edge 1, hue boundary 1, hue boundary 1 ', hue boundary 2, and edge 2. Since the edge 1 and the hue boundary 1 can be set, first, these are set. Then, the UV value of the pixel (halftone dot) immediately on the right side of the hue boundary 1 is recorded. As shown in FIG. 20B, all the UV values of the pixels between the edge 1 and the hue boundary 1 are replaced with the recorded UV values. Thus, as shown in FIG. 20C, the UV values of all the pixels between the edge 1 and the hue boundary 1 become the recorded UV values. Then, the edge 1 and the hue boundary 1 are in the same position, and there is no apparent discomfort or color shift.

  Next, when forming a set, the hue boundary 1 ′ and the hue boundary 2 are set, but two of the hue boundaries are set and no processing is performed. Therefore, the next hue boundary 2 and edge 2 are set and processed. Therefore, when four hue boundaries are continuous, the two hue boundaries in the middle are set and processing is not performed. When an edge is detected, the edge and the hue boundary detected immediately before the edge are set. Process.

  In the next process, edge 2 and hue boundary 2 are set. Record the UV value of the pixel (high-density halftone dot) immediately to the left of hue boundary 2. As shown in FIG. 20C, all the UV values of the pixels between the edge 2 and the hue boundary 2 are replaced with the recorded UV values. As a result, as shown in FIG. 20D, the UV values of all the pixels between the edge 2 and the hue boundary 2 become the recorded UV values. Then, the edge 2 and the hue boundary 2 are in the same position, and there is no apparent discomfort or color shift.

  As an example in which three or more such hue boundaries are continuously detected, a gradation can be mentioned. In the case of such gradation, even if there is no edge on the middle hue boundary, the positional deviation is not noticeable.

  Contrary to the example shown in FIG. 20, three or more edges may be detected in succession. FIG. 21 is a diagram illustrating an example in which three edges such as a hue boundary, an edge, an edge, an edge, and a hue boundary are continuously detected in the middle. As shown in FIG. 21A, in this example, the edge and the hue boundary are detected in the order of the hue boundary 1, the edge 1, the edge 1 ', the edge 2 and the hue boundary 2. Since edge 1 and hue boundary 1 can be set, first, they are set. Then, the UV value of the pixel (halftone dot) immediately to the left of the hue boundary 1 is recorded. As shown in FIG. 21 (b), in order to match the hue boundary 1 to the edge 1, for all pixels between the edge 1 and the hue boundary 1, all the UV values are the recorded UV values. replace. Thus, as shown in FIG. 21C, the UV values of all the pixels between the edge 1 and the hue boundary 1 become the recorded UV values. Then, the edge 1 and the hue boundary 1 are in the same position, and there is no apparent discomfort or color shift.

  Next, when a set is formed, the edge 1 'and the edge 2 are used. However, as in the case of the two hue boundaries described above, the processing is not performed with the two edges set. For this reason, the hue boundary next to the continuous edge is detected. Then, the detected hue boundary 2 and the edge 2 detected immediately before are set.

  Record the UV value of the pixel (horizontal stripe) just to the right of hue boundary 2. As shown in FIG. 21C, all the UV values of the pixels between the edge 2 and the hue boundary 2 are replaced with the recorded UV values. Thereby, as shown in FIG. 21D, the UV values of all the pixels between the edge 2 and the hue boundary 2 become the recorded UV values. Then, the edge 2 and the hue boundary 2 are in the same position, and there is no apparent discomfort or color shift.

  An example in which three or more such edges are detected in succession is when there is an edge of the same color as the background. In such a case, even if there is no hue boundary on the middle edge, the positional deviation is not noticeable.

  The configuration and functions of the image processing apparatus and the imaging system have been described in detail with examples. The processing executed by each functional unit has also been described. Here, the overall processing flow executed by the image processing apparatus is summarized.

  FIG. 22 is a flowchart showing the flow of the entire process executed by the image processing apparatus. Starting from step 2200, in step 2210, the image acquisition unit 50 acquires an infrared light image and a visible light image output from at least one camera. In step 2220, the image acquisition unit 50 passes the infrared light image of the two acquired images to the filter unit 53 and passes the visible light image to the image correction unit 51.

  In step 2230, the filter unit 53 performs the filter processing such as noise removal and edge enhancement described above. In step 2240, the edge detection unit 54 performs edge detection for detecting the edge of the infrared light image from which noise is removed and the edge is enhanced by the method described with reference to FIG. 10. . The edge detection unit 54 outputs the detection result to the composite image generation unit 52. In step 2230 described above, the filter unit 53 outputs the infrared light image subjected to noise removal and edge enhancement to the edge detection unit 54 and also to the composite image generation unit 52.

  In step 2250, the image correction unit 51 performs saturation correction on the visible light image. The saturation correction can be performed by the method described with reference to FIGS. The image correcting unit 51 outputs the visible light image after the saturation correction to the hue dividing unit 55. In step 2260, the hue dividing unit 55 divides the hue of the visible light image by the method described with reference to FIG. 11, and determines which hue it is. The hue division unit 55 outputs the determined result to the composite image generation unit 52 as a result of the hue division. At this time, the image correction unit 51 outputs the visible light image after the saturation correction to the composite image generation unit 52. In step 2250, the image correction unit 51 outputs the visible light image after the saturation correction to the hue division unit 55 and also outputs it to the composite image generation unit 52.

  In FIG. 22, the processing of step 2230 and step 2240 is performed prior to the processing of step 2250 and step 2260, but the order may be reversed or may be performed concurrently.

  In Step 2270, the composite image generation unit 52 performs parallax correction by the method described with reference to FIG. 12 as necessary, corrects the positional deviation between the infrared light image and the visible light image, and combines them. Generate a composite image. The synthesis can be performed by the method described with reference to FIG. Further, the alignment of the two images performed at the time of combining can be performed by the method described with reference to FIGS. 16 to 21 by correcting the positional deviation by the method described with reference to FIG. .

  The composite image generation unit 52 generates the composite image, and then outputs the composite image. In step 2280, this process ends. The output composite image is used for intruder detection, suspicious object detection, criminal face authentication, and the like.

  In this way, an infrared light image and a visible light image are acquired, saturation correction is performed on the visible light image, and the infrared light image and the corrected visible light image are combined so as not to be misaligned. Thus, it is possible to improve image processing accuracy such as detection and authentication.

  The present invention has been described with the above-described embodiments as an image processing apparatus, an imaging system, and an image processing method, but the present invention is not limited to the above-described embodiments. Therefore, other embodiments, additions, changes, deletions, and the like can be changed within a range that can be conceived by those skilled in the art, and as long as the effects and advantages of the present invention are exhibited in any aspect, the present invention It is included in the range. Therefore, it is possible to provide a program for causing a computer to execute the image processing method, a recording medium on which the program is recorded, and the like.

DESCRIPTION OF SYMBOLS 10, 11 ... Camera, 12 ... PC, 20, 21 ... Lens, 22, 23, 29 ... Sensor, 24-26, 30 ... Mirror, 27 ... Infrared light cut filter, 28 ... Visible light cut filter, 40 ... CPU 41 ... ROM, 42 ... RAM, 43 ... HDD, 44 ... input / output interface, 45 ... recording medium, 46 ... external storage interface, 47 ... display device, 48 ... input device, 49 ... bus, 50 ... image acquisition unit, DESCRIPTION OF SYMBOLS 51 ... Image correction part, 52 ... Composite image generation part, 53 ... Filter part, 54 ... Edge detection part, 55 ... Hue division part

JP 2012-054904 A JP 2008-252639 A

Claims (10)

An image processing apparatus for processing an image captured by at least one imaging apparatus,
An image acquisition unit that acquires an infrared light image and a visible light image as the image from the at least one imaging device;
An image correction unit that performs saturation correction on the visible light image acquired by the image acquisition unit;
Using the brightness information of the infrared light image acquired by the image acquisition unit and the saturation information and hue information of the visible light image corrected by the image correction unit, the infrared light image and the corrected image are corrected. An image processing apparatus comprising: a composite image generation unit configured to correct a positional deviation with respect to the visible light image and to synthesize the infrared light image after correcting the positional deviation and the visible light image to generate a composite image.   The image processing apparatus according to claim 1, wherein the image correction unit corrects the saturation of the visible light image, which has been lowered due to the influence of disturbance, using a correction parameter.   The image correction unit corrects the saturation of the visible light image using the correction parameter set for the element having color information among the elements of the color space expressing the color of the visible light image. The image processing apparatus according to claim 2.   The image processing apparatus according to claim 3, wherein the image correction unit switches the correction parameter according to a type of disturbance.   The composite image generation unit uses a position of an edge where the lightness obtained from the lightness information changes discontinuously, and a hue boundary representing a location where the color obtained from the saturation information and the hue information changes, The image processing apparatus according to claim 1, wherein the positional deviation is corrected.   The image processing device according to claim 5, wherein the composite image generation unit corrects the positional shift so that the position of the edge and the hue boundary overlap each other.   The image processing according to claim 5 or 6, wherein the composite image generation unit corrects the positional deviation with respect to each of a main scanning direction and a sub-scanning direction of the infrared light image and the corrected visible light image. apparatus.   The image processing device according to claim 1, and at least one imaging device that inputs an infrared light image and a visible light image obtained by imaging the image processing device. Including an imaging system. An image processing method executed by an image processing apparatus for processing an image captured by at least one imaging apparatus,
Obtaining an infrared light image and a visible light image as the image from the at least one imaging device;
Performing saturation correction on the obtained visible light image;
Using the brightness information of the acquired infrared light image and the saturation information and hue information of the visible light image subjected to saturation correction, the infrared light image and the saturation light corrected visible light image, And a step of generating a composite image by combining the infrared light image and the visible light image after correcting the positional shift. A program for causing a computer to execute an image processing method for processing an image captured by at least one imaging device,
Obtaining an infrared light image and a visible light image as the image from the at least one imaging device;
Performing saturation correction on the obtained visible light image;
Using the brightness information of the acquired infrared light image and the saturation information and hue information of the visible light image subjected to saturation correction, the infrared light image and the saturation light corrected visible light image, A program for causing the computer to execute a step of generating a composite image by correcting an infrared light image and a visible light image after correcting the positional shift.