JP6806356B2 - Image processing equipment, imaging system, image processing method and computer program - Google Patents

Description

The present invention relates to image processing.

As an image process for improving the image quality of a target image by a filtering process, a process using a bilateral filter is known. Bilateral filters can smooth an image while preserving the edges in the image. The bilateral filter includes a spatial weight based on the distance between the pixel of interest and a peripheral pixel, and a pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value. The difference between the pixel value of interest and the peripheral pixel value occurs in a place where the difference between the pixel values becomes large, such as a place having an edge. When the difference between the pixel value of interest and the peripheral pixel value is large, the pixel value weight becomes small. Edges are likely to be preserved in areas where the pixel value weight is small.

The pixel value weight may be obtained from a control image different from the target image. A filter in which the pixel value weight is obtained based on the control image is called a joint bilateral filter. The joint bilateral filter is sometimes called a cross bilateral filter. The joint bilateral filter is disclosed in, for example, Patent Document 1.

JP 2015-144423

If the target image has an edge that is not in the control image, the joint bilateral filter cannot sufficiently store that edge. At edges that are not in the control image, the pixel value weights based on the control image are not sufficiently small. Therefore, the edge in the target image is greatly affected by the peripheral pixel value and becomes blurred.

Therefore, improvement of the conventional joint bilateral filter is desired.

In one aspect of the present invention, the filter includes not only the first weight based on the control image but also the second weight based on the target image as the pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value. Since there is a second weight based on the target image as the pixel value weight, it is possible to suppress blurring of the edge of the target image.

It is a block diagram of an imaging system. It is explanatory drawing of the method of obtaining the weight in a trilateral filter. It is explanatory drawing of the method of obtaining the weight in a bilateral filter. It is an image which shows the processing result of bilateral filtering. It is a dither image of the image of FIG. 4A. It is an enlarged image of the image of FIG. 4A. It is a dither image of the image of FIG. 5A. It is an image which shows the processing result of bilateral filtering and trilateral filtering. It is a dither image of the image of FIG. 6A. It is an image which shows the processing result of trilateral filtering and local trilateral filtering. It is a dither image of the image of FIG. 7A.

[1. Outline of Embodiment]

(1) The image processing apparatus according to the embodiment executes a filtering process that applies a filter to a target image. The filter includes a spatial weight based on the distance between the pixel of interest and a peripheral pixel, and a pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value. The pixel value weights are the first weight based on the difference between the pixel value of interest and the peripheral pixel value in the control image obtained by imaging the same imaging object as the imaging object of the target image, and the pixel value of interest and the periphery in the target image. Includes a second weight based on the difference from the pixel value. Since the pixel value weight includes not only the first weight based on the control image but also the second weight based on the target image, it is possible to suppress blurring of the edge of the target image.

(2) The target image is an image having a lower resolution than the control image, and the filtering process includes a conversion process for converting the target image so that the number of pixels corresponds to the control image, and the second The weight is determined based on the target image that has undergone the conversion process, and the filter is preferably applied to the target image that has undergone the conversion process. In this case, the resolution of the target image can be increased.

(3) The control image is preferably a visible light image, and the target image is preferably an infrared image. In this case, blurring of the edges of the infrared image can be suppressed.

(4) The control image is a visible light image, the target image is an infrared image having a resolution lower than that of the visible light image, and the filtering process is such that the number of pixels corresponds to the visible light image. Further including a conversion process for converting the infrared image, the second weight is obtained based on the far-infrared image to which the conversion process has been performed, and the filter is applied to the infrared image to which the conversion process has been performed. Is preferable. In this case, the resolution of the infrared image can be increased while suppressing blurring of the edges of the infrared image.

(5) The first weight and the second weight are set based on the interrelationship between the control image and the target image in the local region to which the first weight and the second weight are applied. preferable. In this case, filtering becomes more appropriate.

(6) It is preferable that the first weight is set larger as the local interrelationship is higher, and the second weight is set larger as the local interrelationship is lower.

(7) The imaging system according to the embodiment includes an imaging unit that captures a target image and a control image, and an image processing device that executes a filtering process that applies the filter to the target image.

(8) The image processing method according to the embodiment includes executing a filtering process that applies the filter to the target image.

(9) The computer program according to the embodiment causes the computer to execute a filtering process for applying the filter to the target image.

[2. Details of the embodiment]

[2.1 Imaging system]

The imaging system 10 shown in FIG. 1 includes an imaging unit 20. The imaging unit 20 outputs a plurality of images. The plurality of images include a target image and a control image. The target image is an image that is the target of image processing. The control image is an image used to obtain the filter weight. The control image is an image obtained by capturing the same image pickup object as the target image. As the control image, for example, an image obtained by capturing the same imaging target as the target image by an imaging method different from the target image is used.

The imaging method different from the target image is, for example, an imaging method in which the imaging target appearing in the control image has some difference from the imaging target appearing in the target image. The imaging method different from the target image is, for example, an imaging method in which the image is detected by infrared rays if the target image is a visible light image whose image is detected by visible light. Another example of an imaging method different from the target image is, for example, magnetic resonance imaging (MRI) when the target image is an ultrasonic image.

The difference in the imaging method may be the difference in the imaging conditions. The difference in imaging conditions is, for example, the difference in the amount of light during imaging. The difference in the amount of light may be caused by, for example, the presence or absence of a flash or the difference in ambient light. The difference in imaging conditions may be, for example, a difference in the lens used for imaging. Differences in lenses cause differences in the images obtained by imaging.

The image pickup unit 20 has one or more image pickup devices. The imaging unit 20 having a plurality of imaging devices 21 and 23 is configured as, for example, an array camera. The plurality of imaging devices 21 and 23 are arranged close to each other and can image the same imaging object. The plurality of imaging devices include, for example, an imaging device 21 for capturing a control image and an imaging device 23 for capturing a target image.

The image pickup device 21 is, for example, a visible light camera 21. The visible light camera 21 is an imaging device based on visible light. The visible light image captured by the visible light camera 21 has an image similar to that seen by the naked eye. In the embodiment, the visible light image is used as a control image.

The image pickup device 23 is, for example, an infrared camera 23. The infrared camera 23 is an infrared imaging device. The infrared image captured by the infrared camera 23 is an image that visualizes infrared rays from an imaged object. Infrared images are sometimes called thermal images. In the embodiment, the infrared image is used as the target image.

Infrared rays are electromagnetic waves having a wavelength in the range of about 0.7 μ to 1 mm. Infrared has a longer wavelength than visible light. Infrared rays are classified into, for example, near infrared rays, mid infrared rays and far infrared rays. Far infrared rays are, for example, electromagnetic waves having a wavelength of 8 to 14 μm. The infrared camera 23 may capture any infrared ray. The infrared camera 23 of the embodiment is, for example, a far-infrared camera. A far-infrared camera can capture the strength of far-infrared rays.

The infrared camera 23 is often inferior in resolution to the visible light camera 21. As a result, infrared images often have lower resolutions than visible light images. In particular, infrared images captured by far-infrared cameras have low resolution. Visible light cameras, on the other hand, have high resolution and are relatively inexpensive due to the development of CMOS technology. Infrared cameras are often expensive, even at low resolutions, and those with sufficiently high resolution are very expensive. This tendency is particularly remarkable in far-infrared cameras.

In the present embodiment, the image captured by the visible light camera 21 which is relatively inexpensive even at high resolution is used as the control image, and the image captured by the far infrared camera 23 having a lower resolution than the visible light camera 21 is targeted. It is an image. By adopting the low-resolution far-infrared camera 23, it is possible to suppress the increase in price of the imaging system 10.

The imaging unit 20 does not have to have a plurality of imaging devices 21 and 23, and may have only one imaging device. In order to capture the target image and the control image with one imaging device, the above-mentioned imaging conditions may be different.

As shown in FIG. 1, the image pickup system 10 includes an image processing device 30. The image processing device 30 includes, for example, a computer. The computer includes a processor 50 and a storage device 40. The processor 50 executes a computer program stored in the storage device 40. The computer program contains code that causes the computer to perform image processing on the target image. The computer program of the embodiment may also include a code that causes the computer to execute a process of controlling imaging by the imaging unit 20.

The storage device 40 has a storage area 41 for a control image. The control image captured by the visible light camera 21 is stored in the storage area 41. The storage device 40 has a storage area 43 for the target image. The target image captured by the infrared camera 23 is stored in the storage area 43. The storage device 40 has a storage area 45 of the generated image generated by image processing on the target image.

The image processing performed by the processor 50 includes a filtering process 51. In the filtering process 51, the processor 50 applies a filter to the target image stored in the storage area 43 to generate a generated image. The processor 50 stores the generated image in the storage area 45.

The filter of the embodiment is configured to have a plurality of filter weights. The filter of the embodiment is generated based on the control image and the target image. Details of the filtering process 51 will be described later.

[2.2 bilateral filter]

The bilateral filter on which the filter of the embodiment is based will be described below. The bilateral filter has a spatial weight and a luminance value weight using a Gaussian function. Bilateral filters smooth the image while preserving the edges. The filtering process using the bilateral filter is expressed by, for example, the following equation.
Here, I is a target image. F is an image generated by filtering. p is a pixel of interest. q is a peripheral pixel. The peripheral pixel p is a pixel included in the peripheral region of the pixel of interest p. The peripheral pixel includes the central pixel. I (q) is a pixel value in the peripheral pixel q of the target image I. F (I (p)) is a pixel value in the pixel of interest p of the generated image F. N (p) indicates the range of the peripheral region of the pixel p. N (p) is the range of the product-sum calculation in the formula 1A.

The bilateral filter represented by the formula 1A has a spatial weight W _s based on the distance between the pixel of interest p and the peripheral pixel q. The spatial weight W _s is obtained by, for example, the following equation. The spatial weight W _s decreases as the distance between the pixel of interest p and the peripheral pixel q increases. The spatial weight W _s smoothes the image.
Here, σ _s is a parameter that determines the weight W _s and indicates the variance.

The bilateral filter represented by the formula 1A has a pixel value weight W _c based on the difference between the pixel value I (p) of interest and the peripheral pixel value I (p) in the target image I. The pixel value weight is obtained by, for example, the following equation. The pixel value weight W _c becomes smaller as the difference between the pixel value I (p) of interest and the peripheral pixel value I (q) becomes larger. The edge of the target image I is stored by the pixel value weight W _c .
Here, σ _c is a parameter that determines the weight W _c , and indicates the variance.

In filtering using a bilateral filter, the peripheral pixel value I (q) of the target image I is multiplied by the product of the spatial weight W _s and the pixel value weight W _c, and the sum of the plurality of peripheral pixels q is calculated. Desired.

W in the equation 1A is a normalization parameter, and is obtained by the product-sum calculation of W _s and W _c as in the following equation.

[2.3 joint bilateral filter]
A filter in which the product of weights W _s and W _c obtained based on a control image I different from the target image O is used for filtering the target image O is called a joint bilateral filter. The filtering process using the joint bilateral filter is expressed by, for example, the following equation.
Here, O is a target image. F is an image generated by filtering. p is a pixel of interest. q is a peripheral pixel. O (q) is a pixel value in the peripheral pixel q of the target image O. F (O (p)) is a pixel value in the pixel of interest p of the generated image F. N (p) indicates the range of the peripheral region of the pixel p. N (p) is the range of the product-sum calculation in Equation 1B.

In the formula 1B, the weights W _s and W _c are obtained based on the control image I, not the target image O. The other points of the formula 1B are the same as those of the formula 1A.

The joint bilateral filter is used to filter another target image O such as a distance image by using the weights W _s and W _c obtained based on a control image such as a visible light image.

When a joint bilateral filter is applied to a pair of a high resolution control image and a low resolution target image, the resolution of the target image can be increased to the resolution of the control image. Such high resolution processing is called upsampling. Upsampling is expressed by the following equation.
Here, T is a conversion process for obtaining the pixel correspondence between the high-resolution control image I and the low-resolution target image O. The conversion process is performed by, for example, pixel interpolation processing. The target image O that has undergone the conversion process has a number of pixels corresponding to the number of pixels of the control image I. T (p) is the pixel value of the pixel q in the target image O that has undergone the conversion process. The other points of the formula 1C are the same as those of the formula 1B.

[2.4 trilateral filter]
The filter of the embodiment has not only the weights W _s and W _c obtained from the control image I, but also the weights W _o obtained from the target image O. The weight W _o is a weight based on the difference between the pixel value O (p) of interest and the peripheral pixel value O (q) in the target image O. The weight W _c is obtained from the control image I, whereas the weight W _o is obtained from the target image O. Hereinafter, the weight W _c may be referred to as a first weight, and the weight W _o may be referred to as a second weight.

Hereinafter, the filter of the embodiment is referred to as a trilateral filter. The filtering process by the trilateral filter corresponds to the filtering process 51 shown in FIG. 1, and is represented by, for example, the following equation.
Here, O is a target image. F is an image generated by filtering. p is a pixel of interest. q is a peripheral pixel. T is a conversion process for obtaining the pixel correspondence between the high-resolution control image I and the low-resolution target image Q. T (p) is the pixel value of the pixel q in the target image O that has undergone the conversion process. O (T (q) is the pixel value in the pixel q of the target image O that has undergone the conversion process. F (O (p)) is the pixel value in the pixel p of the generated image F. N (p). Indicates the range of the peripheral region of the pixel p. N (p) is the range of the product-sum calculation in the equation 1D.

The filter represented by the formula 1D includes a conversion process T for a low-resolution target image. By the conversion process T, the number of pixels of the low-resolution target image O can be made to correspond to the high-resolution control image I. However, the trilateral filter does not have to include the conversion process T.

As shown in FIG. 2, the spatial weight W _s in the equation 1D is obtained by the equation 2 based on the control image I. The first weight W _c , which is the luminance value weight in the formula 1D, is obtained by the formula 3 based on the control image I. The spatial weight W _s may be obtained based on the target image O subjected to the conversion process T.

As shown in FIG. 2, the second weight W _o is the other brightness value weights in Formula 1D, based on the object image O, obtained by the following equation. The second weight _Wo becomes smaller as the difference between the attention pixel value O (T (p)) of the target image O and the peripheral pixel value O (T (q)) becomes larger. The second weight W _o, it is possible to prevent the edge of the object image O is blurred.
Here, σ _o is a parameter that determines the weight W _o , and indicates the variance.

The processor 50 can perform a process of setting a value of σ ₀ . The processor 50 can also perform a process of setting the values of σ _s and σ _c . The processor 50 can also perform a process of adjusting the balance between the sizes of the weight W _c and the weight W _o . By increasing the weight W _c, during filtering, will effectiveness is stronger control image I, by increasing the weight W _o, during filtering, effectiveness of the target image I becomes stronger. The balance adjustment between the weight W _c and the size of the weight W _o may be performed by, for example, adjusting the values of σ _c and σ _o , or by multiplying the weight by a coefficient for adjustment. good. The balance between the weight W _c and the weight W _o may be adjusted for each image, or may be locally performed for each pixel in the image or for each partial region in the image.

In the filtering process 51 using the trilateral filter, the product of the spatial weight W _s , the first weight W _c, and the second weight W _o is multiplied by the peripheral pixel value O (T (q)) of the target image O. Then, a product-sum calculation is performed to obtain the sum of the plurality of peripheral pixels q.

W in Equation 1D is a normalized parameter, and is obtained by multiply-accumulate calculation of W _s , W _c, and W _o as in the following equation. In the filtering process 51, W normalizes the product-sum calculation result of the spatial weight W _s , the first weight W _c, and the second weight W _o . The normalized value becomes the pixel value of pixel p in the generated image.

FIG. 3 shows how to obtain the weights W _s and W _c in the bilateral filter. As is apparent from comparison between FIGS. 2 and 3, the tri-lateral filter, the joint bilateral filter is obtained by adding the luminance value weight W _o, determined from the target image O. Luminance value weight W _o, determined from the target image O is to function to store edge of the object image O, it is possible to prevent the edge of the generated image is blurred.

The second weight W _o in formula 1D, in that a luminance value weights obtained from the target image, in common with the luminance value weight W _c in bilateral filter shown in Equation 1A. Also, space weighting W _s in Formula 1D also, those similar to the spatial weight W _s in bilateral filter shown in Equation 1A. Therefore, the trilateral filter represented by the formula 1D can be considered to be a fusion of the joint bilateral filter represented by the formula 1C and the bilateral filter represented by the formula 1A.

The joint bilateral filter cannot save the edge of the target image in the generated image when the target image has an edge that is not in the control image, and the generated image may be partially separated from the target image. .. However, tri-lateral filter, because it has a luminance value weight W _o which function to store the edge of the object image O, restrain the blurred no edge in the control image. As a result, it is possible to prevent the generated image from being significantly separated from the target image. Moreover, since the trilateral filter is based on the joint bilateral filter, it can be expected to function as a joint bilateral filter.

4A and 5A show an example of the filtering processing result by the joint bilateral filter. The control image I shown in FIG. 4A is a high-resolution color visible light image captured by the visible light camera 21. The object to be imaged is the roof of a building. The target image O shown in FIG. 4A is an infrared image obtained by capturing the same imaging target with the infrared camera 23. Infrared images have a lower resolution than visible light images.

The images I, O, and F shown in FIG. 5A are enlarged images of the images I, O, and F shown in FIG. 4A. Focusing on the region A of the control image I in FIG. 5A, in the control image I, the roof surface and the roof of the structure built on the roof surface have the same color, and there is no edge at the boundary between the two. On the other hand, focusing on the region B of the target image O, since the temperatures of the roof surface and the roof of the structure built on the roof surface are different, there is a difference in shade between the two as an infrared image. There is an edge at the boundary of.

Although the area A and the area B are the same area in the image pickup object, the edge does not exist in the area A of the control image I and the edge exists in the area B of the target image O due to the difference in the imaging method. When a joint bilateral filter having weights W _s and W _c obtained from the control image I is applied to the target image O, as shown in the region C of the generated image F in FIG. 5A, the region B of the target image O is covered. The existing edge is dragged by the control image I having no edge and becomes blurred.

FIG. 6A shows the images 102a and 102b generated by bilateral filtering and the images 103a and 103b generated by trilateral filtering. The generated image 102b and the generated image 103b are enlarged images of the generated image 102a and the generated image 103a, respectively. The edge is blurred in the region C of the generated image 102b by bilateral filtering, whereas the edge is preserved in the region D of the generated image 103b by trilateral filtering.

As described above, in the joint bilateral filter, when the low resolution infrared image is increased in resolution by using the high resolution visible light image, it may not be possible to achieve an appropriate high resolution, whereas the trilateral of the embodiment is used. Filtering can appropriately increase the resolution of low-resolution infrared images. When the trilateral filtering of the embodiment is used, a high-resolution infrared image can be obtained by using a relatively inexpensive low-resolution infrared camera 23, which is advantageous because it is not necessary to use a very expensive high-resolution infrared camera. is there.

The images in FIGS. 4B, 5B and 6B are dithered images obtained by subjecting the images in FIGS. 4A, 5A and 6A to dithering processing. .. Performing the dithering process is irrelevant to the present invention and its embodiments. At the time of publication of the present application by an authorized institution, even if the images in FIGS. 4A, 5A and 6A are deteriorated, a third party recognizes the images according to FIGS. 4B, 5B and 6B. It is intended to be able to. FIG. 7B, which will be described later, is also a dither image of FIG. 7A, and is based on the same intention.

By the way, the filtering process by the trilateral filter can be applied to the target image O having a vector value. Trilateral filtering to the target image O having a vector value is expressed by, for example, the following equation. The points not particularly described with respect to the formula 1E are the same as those of the formula 1D.

The second weight W _o in Formula 1E, based on the object image O, obtained by the following equation.

[2.4 local trilateral filter]
The tri-lateral filter shown in Formula 1E, to be done also filtered by the second weight W _o based on object image O, if the spatial resolution of the object image O is low, the limit occurs in the improvement of the spatial resolution. Therefore, if the balance between the sizes of the first weight W _c and the second weight W _o in the trilateral filter is locally changed according to the location (coordinates) on the image, for example, the spatial resolution in the generated image F can be increased. It can be improved to obtain a sharper image. A trilateral filter in which the balance between the sizes of the first weight W _c and the second weight W _o is changed according to the location on the image is called a local trilateral filter. The local trilateral filter may be applied to a target image having the same resolution as the control image.

The filtering process 51 using local tri lateral filter embodiments, the balance of the first weight W _c and the second weight W _o is set based on local correlation of the control image I and target image O To. The local interrelationship refers to the reciprocity between the two images I and O in the local region of the two images I and O. The local region is a range consisting of one or more pixels. As the local mutual relationship, for example, the amount of local mutual information between the control image I and the target image O is used.

The local mutual information μ _XY (p) in the pixels p of the two images Ix and the image Iy is defined by, for example, the following equation.
Here, N (p) is a neighborhood system of the pixel P, and _PXY ( _IX (q), I _Y (q)) is the simultaneous occurrence probability of the pixel values _IX (q) and _YY (q), P _X. ( _IX (q)) and P _Y (I _Y (q)) are the occurrence probabilities of the pixel values _IX (q) and _YY (q), respectively.

The processor 50, in the filtering process, for each pixel p, determine local mutual information mu _co of the control image I and target image O. In certain target pixel p, the greater the local mutual information mu _co of the control image I and target image O, in the vicinity of the target pixel p, high interconnectedness between two images I, O. If the interrelationship is high, there is no problem even if the effect of the control image I is strengthened. Therefore, if the local mutual information μ _co is large in a certain pixel of interest p, the processor 50 has the weights W _c and W _o of both weights W _c and W _{o in} the pixel of interest p so that the first weight W _c based on the control image I is larger. Set the size balance.

On the other hand, if the amount of local mutual information between the control image I and the target image O is small in a certain pixel of interest p, the interrelationship between the two images I and O is low in the vicinity of the pixel of interest p. If the interrelationship is low, the effectiveness of the control image I should be weakened and the effectiveness of the target image O should be strengthened. Therefore, if the local mutual information μ _co is small in a certain pixel of interest p, the processor 50 has the two weights W _c and W _o of the pixel of interest p so that the second weight W _o based on the target image O is larger. Set the size balance.

The balance adjustment of the magnitudes of the weights W _c and W _o can be performed, for example, by the processor 50 executing a process of adjusting the values of the parameters σ _c and σ _o as described above. The values of the parameters σ _c and σ _o may be adjusted for each pixel, or may be adjusted for each partial region including a plurality of pixels. Adjustment of the values of σ _c and σ _o is performed, for example, by using the parameters σ _c and σ _o as a function of interrelationship. More specifically, the values of σ _c and σ _o can be adjusted by setting the parameters σ _c and σ _o to the functions σ _c (μ _co ) and σ _o (μ _co ) of the mutual information amount μ _co. ..

Function σ _c (μ _co) is the larger the local mutual information mu _co, the value of the function σ _c (μ _co) decreases, the smaller the local mutual information mu _co, function σ _c (μ _co ) May be large. Function σ _o (μ _co) is the larger the local mutual information mu _co, increase the value of the function σ _o (μ _co) is, the smaller the local mutual information mu _co, function σ _o (μ _co) The value of becomes smaller.

In other words, as the local mutual information μ _co increases, the value of the function σ _c (μ _co ) decreases, and the value of the function σ _o (μ _co ) increases. As a result, the first weight W _c based on the control image I becomes larger. Further, as the local mutual information μ _co decreases, the value of the function σ _c (μ _co ) increases and the value of the function σ _o (μ _co ) decreases. Thus, more of the second weight W _o based on object image O becomes large.

An example of parameters σ _C and σ _O , which are functions of mutual information, is shown below. The calculation method of the parameters σ _C and σ _O , which are functions of mutual information, is not limited to the following. In the following, σ _rO , σ _gO , and σ _bO are parameters for each of the red, green, and blue channels r, g, and b of the color control image I, respectively. σ _rC , σ _gC , and σ _bC are parameters for each of the red, green, and blue channels r, g, and b of the color control image I, respectively.
_{Here, μ ro, μ go, μ} bo , respectively, represent the red color control image I, green, image _I r and blue _channels, I g, the local mutual information between _{I b,} respectively and the target image O There is.

As shown in Equations 8 to 10, the parameters σ _rO , σ _gO , and σ _bO are functions of μ _rO , μ _go , and μ _bO , respectively. Parameter sigma _O for the second weight _{W O} is the color channel r, g, the parameters found for b σ _rO, σ _gO, based on sigma _bO, determined as in equation 11.

As shown in Equations 8 to 10, the parameters σ _rC , σ _gC , and σ _bC are also functions of μ _rO , μ _go , and μ _bO , respectively. The first weight W _c is obtained by the following equation based on the parameters σ _rC , σ _gC , and σ _bC obtained for each of the color channels r, g, and b.

FIG. 7A shows the images 103a, 103b, 103c generated by trilateral filtering and the images 104a, 104b, 104c generated by local trilateral filtering. The generated image 103b and the generated image 104b are enlarged images of the generated images 103a and 104a, respectively. The generated images 103c and 104c are enlarged images of the generated image 103b and the generated image 104b, respectively.

In the region E of the image 104c generated by the local trilateral filtering, the edges are sharper and the spatial resolution is improved as compared with the region D of the image 103b generated by the trilateral filtering.

[3. Addendum]
The present invention is not limited to the above-described embodiment, and various modifications are possible.

10 Imaging system 20 Imaging unit 21 Visible light camera 23 Infrared camera 30 Image processing device 40 Storage device 41 Control image storage area 43 Target image storage area 45 Generated image storage area 50 Processor 51 Filtering process 102a, 102b Generated image 103a, 103b, 103c Generated image 104a, 104b, 104c Generated image O Target image I Control image

Claims (8)

An image processing device that executes filtering processing that applies a filter to a target image.
The filter
Spatial weight based on the distance between the pixel of interest and peripheral pixels,
The pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value,
Including
The pixel value weight is
The first weight based on the difference between the pixel value of interest and the peripheral pixel value in the control image obtained by imaging the same imaging object as the imaging object of the target image,
A second weight based on the difference between the pixel value of interest and the peripheral pixel value in the target image,
Only including,
An image processing device in which the first weight and the second weight are set based on the interrelationship between the control image and the target image in a local region to which the first weight and the second weight are applied . The target image is an image having a lower resolution than the control image.
The filtering process includes a conversion process of converting the target image so that the number of pixels corresponds to the control image.
The second weight is obtained based on the target image that has undergone the conversion process.
The image processing apparatus according to claim 1, wherein the filter is applied to the target image to which the conversion process has been performed. The control image is a visible light image and is
The image processing apparatus according to claim 1, wherein the target image is an infrared image. The control image is a visible light image and is
The target image is an infrared image having a resolution lower than that of the visible light image.
The filtering process further includes a conversion process for converting the infrared image so that the number of pixels corresponds to the visible light image.
The second weight, the conversion processing is determined based on Kiaka external image before was made,
The image processing apparatus according to claim 1, wherein the filter is applied to the infrared image that has undergone the conversion process. The first weight is set larger as the local interrelationship is higher.
The image processing apparatus according to any one of claims 1 to 4, wherein the second weight is set larger as the local interrelationship is lower. An imaging unit that captures the target image and control image,
An image processing device that executes a filtering process that applies a filter to the target image, and
With
The filter
Spatial weight based on the distance between the pixel of interest and peripheral pixels,
The pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value,
Including
The pixel value weight is
The first weight based on the difference between the pixel value of interest and the peripheral pixel value in the control image,
A second weight based on the difference between the pixel value of interest and the peripheral pixel value in the target image,
Only including,
The first weight and the second weight are set based on the interrelationship between the control image and the target image in the local region to which the first weight and the second weight are applied. system. An image processing method that includes performing a filtering process that applies a filter to a target image.
The filter
Spatial weight based on the distance between the pixel of interest and peripheral pixels,
The pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value,
Including
The pixel value weight is
The first weight based on the difference between the pixel value of interest and the peripheral pixel value in the control image obtained by imaging the same imaging object as the imaging object of the target image,
A second weight based on the difference between the pixel value of interest and the peripheral pixel value in the target image,
Only including,
The first weight and the second weight are set based on the interrelationship between the control image and the target image in the local region to which the first weight and the second weight are applied. Processing method. A computer program that causes a computer to perform a filtering process that applies a filter to a target image.
The filter
Spatial weight based on the distance between the pixel of interest and peripheral pixels,
The pixel value weight based on the difference between the pixel value of interest and the peripheral pixel value,
Including
The pixel value weight is
The first weight based on the difference between the pixel value of interest and the peripheral pixel value in the control image obtained by imaging the same imaging object as the imaging object of the target image,
A second weight based on the difference between the pixel value of interest and the peripheral pixel value in the target image,
Only including,
The first weight and the second weight are set based on the interrelationship between the control image and the target image in the local region to which the first weight and the second weight are applied. program.