JP6497579B2 - Image composition system, image composition method, image composition program - Google Patents

Description

  The present invention relates to an image processing technique, and more specifically, an image composition system, an image composition method, and an image composition that obtain a desired composite image using a plurality of images having different properties such as a plurality of images having different sensitivity characteristics with respect to wavelength bands. Regarding the program.

  Recently, various image composition methods have been developed. Also, there are various cameras that operate with various settings (for example, exposure amount and exposure time), and these cameras can acquire images with various sensitivity characteristics with respect to wavelength bands. Even with cameras having the same settings, substantially different images are acquired depending on whether or not the flash is used at the time of shooting.

  In addition, cameras using sensors suitable for imaging various target objects are widely used. For example, in order to monitor a person or the like, a monitoring camera using a visible light sensor is widely used. On the other hand, cameras using non-visible light sensors such as a near infrared camera and a far infrared camera are widely used for nighttime monitoring. In addition, near-UV cameras are also commercially available. Alternatively, devices for imaging wavelengths longer than the visible light wavelength region such as terahertz waves and radio waves are also commercially available.

  In general, cameras using different types of image sensors have different sensitivity characteristics with respect to the frequency of light, so that an image acquired by one of the image sensors tends to have a lower image quality than the other image. Also, imaging settings are often different. For example, when lighting is taken with a visible light camera, the pixel value near the light emitting part is saturated (white) in the image taken with the visible light camera (visible light image) because the exposure amount is too large in the pixels near the light emitting part. Skip). On the other hand, even with the same angle of view, an image taken with an infrared camera (infrared light image) can be imaged within the normal range without saturation of pixel values near the light emitting unit. By using a plurality of images having different sensitivity characteristics with respect to the wavelength band in this way, it is estimated that a useful composite image that eliminates “whiteout” can be obtained. Similarly, by using a plurality of images having different sensitivity characteristics with respect to the wavelength band, it is possible to estimate the elimination of “blackout”.

  For example, there are the following three documents related to image composition processing using a plurality of images having different sensitivity characteristics with respect to the wavelength band.

In Patent Document 1, a visible light image and an infrared light image are simultaneously captured at the same angle of view and with the same exposure time using the same imaging sensor, and the entire image is corrected after correcting the “whiteout” portion of the visible light image. A method for adjusting the gradation of the image is disclosed. In this method, the entire image is prepared while complementing the layer feature of the “whiteout” portion of the visible light image from the image features of the visible light image and the infrared light image simultaneously captured at the same angle of view and the same exposure time. As a result, a high dynamic range image (HDR image (High Dynamic Range Image)) having a “whiteout” portion of the visible light image within the range is generated.
In the processing operation in the specific embodiment, as shown in FIG. 9, a visible light image and an infrared light image are simultaneously captured by the imaging unit, and each image is output to the image processing unit. The image processing unit detects whether or not a saturated pixel exists in the visible light image. When the saturated pixel exists in the visible light image, the pixel output value of the visible light image around the saturated pixel and the infrared light A ratio (correction coefficient) with the pixel output value of the light image is obtained. A correction value is calculated by multiplying the pixel output value of the infrared light image in the saturated pixel by the correction coefficient. Then, the pixel value and the correction value of the saturated pixel of the visible light image are exchanged. Finally, the gradation conversion unit converts the gradation of the output image based on the correction pixel value.

  Further, Patent Document 2 discloses that the higher luminance is obtained by comparing the luminance of the visible light image and the infrared light image in the same region of the visible light image and the infrared light image with the same angle of view acquired from a plurality of imaging sensors. The technique used for the HDR image using the luminance information of the image is disclosed.

  In Patent Document 3, an infrared light receiving element group is arranged together with a visible light receiving element group in one image sensor so that the visible light and infrared light signals having the same angle of view are within the same region. An imaging device (camera) that enables luminance synthesis of visible light and infrared light in the field is disclosed.

  Non-Patent Documents 1 to 7 below are elemental technologies that can be used in the present invention and are prior arts in the field of the present invention.

JP 2006-180270 A JP 2010-166363 A JP 2013-255144 A

Rudin, L. I .; Osher, S .; Fatemi, E. (1992). "Nonlinear total variation based noise removal algorithms". Physica D 60: 259-268. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “BM3D Image Denoising with Shape-Adaptive Principal Component Analysis”, Proc. Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS'09), Saint-Malo , France, April 2009. S. Park, M. Park, and M. G. Kang, “Super-resolution image reconstruction, a technical overview,” IEEE Signal Process. Mag., Vol. 20, no. 5, pp. 21-36, May 2003. B. D. Lucas and T. Kanade (1981), “An iterative image registration technique with an application to stereo vision,” Proceedings of Imaging Understanding Workshop, pages 121-130 B. Zitova´, J. Flusser, “Image registration methods: a survey,” Image Vision Computing. 21 (11) (2003) 977-1000. Y. Boykov, O. Veksler and R. Zabih, "Fast approximate energy minimisation via graph cuts", IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1222-1239, (2001) Pearl, Judea. “Reverend Bayes on inference engines: A distributed hierarchical approach”. Proceedings of the Second National Conference on Artificial Intelligence. AAAI-82, (1982). P´erez, P., Gangnet, M. and Blake, A. “Poisson image editing,” ACM Transactions on Graphics vol. 22 (3), p. 313-318 (2003).

  The following points can be pointed out as examples of the prior art patent documents 1 to 3.

  Since the method disclosed in Patent Document 1 performs the image quality enhancement process only on pixels in which the pixel value of the visible light image is saturated, the image quality enhancement process is not affected except in areas where overexposure exists. For example, if a visible light image and an infrared light image are captured using an infrared flash in a dark place, the visible light image has a lot of noise and the infrared light image is clear, but the image quality improvement processing works effectively. Absent. Similarly, even when the image is taken in a place with fog or haze, even if the infrared light image is clear, there is no change in the transparency of the visible light image, and a whitish image is acquired.

  The methods disclosed in Patent Documents 2 and 3 are the same, and although image composition is performed using image features between a visible light image and an infrared light image, the area to be synthesized and priority of that area are used. Leave room for improvement in images (plus reflection). If the improvement point is pointed out simply, the area | region to synthesize | combine from a some image and the point that the size are artificially defined beforehand are mentioned. In addition, depending on the system, it may be a problem that the imaging unit is required to simultaneously acquire a visible light image and an infrared light image at the same angle of view.

  The inventors studied an image synthesis method using an image feature that can improve the visibility of each image from a plurality of images having different properties such as sensitivity characteristics with respect to a wavelength band. In this method, we studied an algorithm that can acquire a composite image based on the region of good visibility that each image has over other images.

  In view of the above consideration, the present invention has further improved visibility by deriving a good visibility region with respect to other images included in an image having each sensitivity characteristic from a plurality of images having different properties over the entire image. An image composition system, an image composition method, and an image composition program for generating a composite image are provided.

The image composition system according to the present invention includes a plurality of images having different properties, and each of the images representing the height of visibility for each image as index data for selecting a region having higher visibility than the other images. A selection index calculation unit that quantifies and acquires selection index data and inter-image selection index data that represents the magnitude of continuity of appearance between images, and refers to the index data acquired from the plurality of images A region selecting unit that selects a region having higher visibility used to generate a composite image from the plurality of images by integrally deriving a priority image and its region range; and And an image composition unit that generates a composite image by combining regions with higher visibility selected by the unit.

An image composition method by an information processing system according to the present invention acquires a plurality of images having different properties, and uses image data as index data for selecting a region having higher visibility than other images from each of the plurality of images. The in-image selection index data representing the level of visibility for each image, and the inter-image selection index data representing the magnitude of the continuity of the appearance between the images are quantified for each image, and quantified from the plurality of images With reference to the index data, a region with higher visibility used for generating a composite image is selected from the plurality of images by selecting the priority image and its region range by derivation processing. A composite image is generated by combining regions with higher visibility.

Image combining program according to the present invention, an information processing system, different from a plurality of images each natures, as index data for visibility to select a higher region than the other images, the visibility of each image height Selection index calculation means for quantifying and acquiring, for each image, the in-image selection index data representing the distance and the inter-image selection index data representing the magnitude of the continuity of the appearance between the images, and obtained from the plurality of images A region selection unit that refers to the index data and selects a region with higher visibility used to generate a composite image from the plurality of images by integrally deriving the priority image and its region range. And an image composition unit that generates a composite image by combining regions with higher visibility selected by the region selection unit.

  According to the present invention, from a plurality of images having different properties, a good visibility region with respect to other images included in each sensitivity characteristic image is derived over the entire image, and a composite image with further improved visibility is obtained. An image composition system, an image composition method, and an image composition program can be provided.

1 is a configuration diagram illustrating an image composition system according to a first embodiment. 5 is a flowchart illustrating an example of processing operation performed in the image composition system according to the first embodiment. It is a block diagram which shows the image display system containing an image composition system. It is a flowchart which shows the processing operation example of an image display system. It is explanatory drawing 1 shown by the description which simplified the image synthesis by this invention. FIG. 3 is an explanatory diagram 2 showing a simplified depiction of image composition according to the present invention. It is a block diagram which shows the image display system containing the image composition system of a modification. It is a block diagram which shows the image composition system which concerns on one Example. 1 is a configuration diagram illustrating an image composition system described in Patent Document 1. FIG.

Embodiments of the present invention will be described with reference to the drawings.
Hereinafter, as a plurality of images having different properties, a plurality of images having different sensitivity characteristics with respect to wavelength bands (an image acquired from a visible light sensor (hereinafter referred to as a visible light image) and an image acquired from a non-visible light sensor (hereinafter referred to as an image) A non-visible light image)) will be described as an example. The image used in the present invention is not limited to this. For example, an image used for image synthesis may be an image obtained by both of a plurality of different image sensors from a non-visible light sensor, or may be an image obtained by both from a visible light sensor. Moreover, the image acquired from the camera from which exposure amount differs, and the image from which the presence or absence of the flash at the time of imaging differ may be sufficient.
Another example of the image may be an image having different properties such as a visible light image and a depth image. In addition, the image composition system may use a visible light image and an image obtained by computerizing movement and a solid, such as an optical flow image or a stereo image. Alternatively, the image synthesis system may use, for example, a visible image and a simulation image synthesized by CG (Computer Graphics) or the like, or may use a visible image and an image simulating some physical quantity. In any case, the present invention may be anything as long as a plurality of images for the same object are input images.

  In addition, an embodiment in which an image group that has been imaged externally is used as an input image will be described. The image acquisition unit includes an optical element and an image sensor and generates an input image in the image composition system. Also good.

[First Embodiment]
FIG. 1 is a configuration diagram showing an image composition system 1 according to the first embodiment.

  The image composition system 1 according to the first embodiment includes a selection index calculation unit 10, a region selection unit 20, and an image composition unit 30.

  The image composition system 1 accepts images having different sensitivity characteristics for a plurality of wavelength bands as an input image group, and outputs the generated composite image as required. For example, the output form may be output to the display unit in real time, or may be stored as image data in a storage unit such as a semiconductor memory or an HDD (Hard disk drive). This storage unit may be provided in the image composition system 1 or an external storage device may be used.

The selection index calculation unit 10 acquires, for each image, a visibility index value in each image as index data for selecting a region with higher visibility from a plurality of images that are input images. The selection index calculation unit 10 may acquire, as index data, an index value that combines a visibility index in each image and an index for another image of each image for each image. .
In addition, the area | region with high visibility here may be an area | region considered important for the use use in a user.

Hereinafter, as an example, a case where an area is selected from an infrared image will be described.
In many scenes, a region desired to be selected as having high visibility such as a human body or a car tends to show a temperature above a certain level or a temperature within a specific range in an infrared image.

Therefore, for example, if a function having a large pixel value of the infrared image in a desired temperature range is set in the selection index calculation unit 10 so that an image synthesis system selects a region of a temperature above or within a temperature range from the image. Good. The area selection unit 20 described later selects a desired area according to an index value based on this function.
Similarly, a case where a desired area is selected from a depth image will be described as an example. In this example, the selection index calculation unit 10 is configured to output the index value with a function having a larger value as the pixel value of the depth image is larger, so that the pixel in the closer subject is more easily selected. Good.
Similarly, a case where an area is selected from an optical flow image will be described as an example. In this example, the selection index calculation unit 10 may be configured to output the index value with a function having a larger value as the optical flow value is larger.

That is, in order to select a region with high visibility (that is, a region that is important for the user), the user defines a function that represents a pixel value and high visibility for each type of input image. You may design operation | movement of the selection parameter | index calculation part 10 so that an area | region with high visibility may be selected based on a function.
Note that the above function for selecting a desired region may be manually defined by the user using, for example, a B-spline function, or such a function is calculated using techniques such as image recognition and machine learning. May be.

Hereinafter, a method for calculating a function for selecting a region with high visibility (that is, a region important for the user) by machine learning will be described in detail.
First, in order to collect data for learning, the user has a high visibility region (that is, important for the user) for a plurality of learning images different from the input image presented by the machine learning machine. Select the appropriate area. Next, the machine learning machine, for example, calculates a feature vector for each pixel from all learning images. Further, the classifier learns the relationship between each feature vector and a region with high visibility by using, for example, a support vector machine. The machine learning result may be acquired by building a machine learning machine on the same computer as the image composition system, or may be acquired via a machine learning function in another computer or a cloud environment. As described above, an area with high visibility (that is, an area important for the user) can be expressed by the discriminator.
Actually, when calculating a region with high visibility with respect to the input image, first, the selection index calculation unit 10 uses the same calculation method used during learning for each pixel of the input image. Is calculated. Next, the selection index calculation unit 10 uses the discriminator obtained by machine learning to acquire an index value indicating the size of whether or not each pixel is a highly visible region from each feature vector. .

  The acquired index value may be temporarily stored in an arbitrary storage area, or the area selection unit 20 may acquire it immediately. The input image is not limited to two images, and may be three or more images.

  In the following, a visibility index in each image is referred to as an intra-image selection index, and an index for each of the other images is referred to as an inter-image selection index.

  The area selection unit 20 refers to the index data calculated by the selection index calculation unit 10 and selects a higher-visibility area used to generate a composite image from a plurality of images that are input images as a priority image. And the area range are selected through derivation processing. In this area selection processing, composite pattern data indicating a region with higher visibility due to the relationship between an arbitrary image included in a plurality of images and other images is a function including an intra-image selection index and an inter-image selection index as variables. Find it.

  The image composition unit 30 combines a region with higher visibility from a plurality of images that are input images with a reference image in accordance with composite pattern data indicating a region with higher visibility selected by the region selection unit 20. Is generated. Further, as image synthesis, a luminance adjustment process based on the luminance of the reference image may be added to each partial image reflected in the reference image. Here, the amount of information processing can be reduced by using the reference image. Further, a composite image may be obtained by combining selected regions without using a reference image.

  In addition, in the image composition, the image composition unit 30 may identify each region range to be image composition and a priority image in the region, perform image composition for each region, and then compose the entire image. good. In this case, one or a plurality of regions are respectively synthesized, and as a result, a synthesized image having better visibility is obtained. In addition, for image synthesis for each region, it is also possible to select an image synthesis method according to the index data calculated by the selection index calculation unit 10. For example, based on the luminance ratio of the contact area between the reference image and the corresponding area of the priority image, the blend ratio of the priority image and the reference image is changed, or the color complementing method is selected based on the type of priority image or the size of the area Good.

Note that the user may select any of the input images as the reference image, or the image synthesis system 1 may automatically select the reference image.
As a reference image selection algorithm, for example, a method of selecting an image with higher visibility based on the overall visibility index in an image for each input image can be used. As a more specific example, as a reference image selection process, an image having a large area based on the area of an appropriate visibility (for example, within a predetermined luminance range) of the entire surface or a selected predetermined range of each image. May be selected as a reference image. In addition, an image obtained by a photographing method or an image having sensitivity characteristics with respect to a wavelength band, which is generally better, may be selected in accordance with a time zone, an imaging target, brightness at the time of photographing, and the like.

  As a result, it is possible to obtain a composite image in which an image area in which the visibility of the reference image can be enhanced in an image acquired with other sensitivity characteristics is reflected over the entire image so as to be visible. For example, in the case of a daytime landscape image, a visible light image is often selected as the reference image. In this case, an area portion having better visibility than the visible light image is extracted from the invisible light image, and the image is combined with the visible light image. In addition, when a non-visible light image is selected as the reference image, an area portion having better visibility than the non-visible light image is extracted from the visible light image and is combined with the non-visible light image.

In the above description, the example in which the selection index calculation unit 10, the region selection unit 20, and the image synthesis unit 30 operate in a single hierarchy has been described. However, the present invention need not limit the handling of images to a single hierarchy.
For example, one or a plurality of input images are accepted as images having a plurality of layers (image pyramids) having different frequency components, or the input image is decomposed into image data having a plurality of layers by preprocessing, You may employ | adopt the algorithm which performs a processing operation. In a more specific example, the selection index calculation unit 10 and the region selection unit 20 perform each process in parallel for each layer. Next, the image composition unit 30 may integrate the processing results in the respective layers to generate a final composite image.
In this way, by handling each input image divided into a plurality of layers having different frequency components, there is a region with high visibility in the high frequency component in the visible light image, and in the same region of the low frequency component in the far infrared image. When there is a region with high visibility, the region selection unit 20 selects a visible light image from a layer where the high frequency component is dominant, and selects a far infrared image from a layer where the low frequency component is dominant. Then, the image composition unit 30 synthesizes the image of the region using the image information of the high frequency component of the visible light image and the image information of the low frequency component of the far infrared image. As a result, it is possible to synthesize an image that retains both high visibility (importance for the user).

  With such a configuration, a good visibility region for the other images included in the images of each sensitivity characteristic is derived from a plurality of images having different sensitivity characteristics with respect to the wavelength band, and the visibility is further enhanced. A composite image can be obtained.

[Image processing flow]
Next, the image composition process will be described.
FIG. 2 shows an example of an image processing flow performed in the image composition system 1. This processing flow and the processing flow to be described later are examples, and if the order of processing is changed, the repetition method of each processing, the method applied to each processing, the number of images to be processed, etc. are appropriately changed and read as necessary. Good.

  First, the selection index calculation unit 10 quantifies the intra-image selection index and the inter-image selection index for each image from the input image group (a plurality of images having different sensitivity characteristics with respect to the wavelength band), thereby improving the visibility. Index data for selecting a high region is acquired (S101). In this processing example, in addition to the intra-image selection index, an index value representing the level of similarity based on pixel level characteristics between images is quantified as an inter-image selection index. An example of a function for obtaining an index value of a pixel level feature is a combination of a function including a pixel level feature including a luminance ratio, pixel continuity, and a gradient of the luminance value between pixels, and a selection index between images and an image. A selection index may be calculated.

  In the case of intra-image selection index data, an index having a larger value as the contrast increases may be digitized.

In the case of inter-image selection index data, by combining the following functions, an index value combining both connectivity between pixels, which is a pixel level feature, and a gradient of luminance values between pixels can be obtained.
1) As a function indicating connectivity between pixels, a function that increases an inter-image selection index in a region having a pixel value closer to an adjacent pixel for each of a plurality of images.
2) A function that decreases the inter-image selection index in a portion where the luminance gradients of a plurality of images are similar as a function indicating the gradient of the luminance value between pixels. Note that the selection index calculation unit 10 calculates index data from a plurality of images. Before obtaining the image, it is desirable to perform a normalization process on the input image with respect to any one of a reference image for determining visibility or a plurality of images. By performing the normalization process, the bias in visibility for each image caused by differences in imaging conditions, imaging elements, output forms, and the like is corrected.

  Further, the selection index calculation unit 10 performs an alignment process on the input image with respect to either the reference image for determining the angle of view or the plurality of images before acquiring the index data from the plurality of images. May be. By performing the alignment process, it is possible to reduce the amount of information processing in the subsequent stage, the deterioration of image composition, and the like.

Next, the region selection unit 20 refers to the acquired index data, and integrally derives a priority image and its region range from a plurality of images, and has a higher visibility used for the composite image. Is selected (S102).
The region is selected by referring to the index data showing the characteristics of the pixel level and using a function having as a variable the index value for each pixel in the image and the index value between the pixels of the image to be compared. By executing image processing for the entire image, the priority image and its area range are determined as composite pattern data. Further, the image composition processing method for each region may be described in the composition pattern data.

Next, the image composition unit 30 generates a composite image by combining the selected regions with higher visibility (S103). By using the synthesized pattern data, it is possible to identify the selected region with higher visibility. In addition, when performing image composition processing for each region, by using composition pattern data, identification (algorithm selection) of the image composition processing method for each region and common to a plurality of regions using the same image composition processing method. Centralization of arithmetic processing becomes easy.
For example, the image composition unit 30 determines, based on the continuity of the color component of the image acquired from the sensor in the visible light region and the appearance frequency of the color component, in each partial image to be reflected in the reference image before the composite image. The process of correcting the color component in the non-visible light region to the color component in the visible light region can be performed in a unified manner.

  By performing the above processing operation, a good visibility region with respect to other images included in the image of each sensitivity characteristic is derived from a plurality of images having different sensitivity characteristics with respect to the wavelength band over the entire image. A composite image with a higher image quality can be obtained.

  Next, a configuration example of the first embodiment will be described with reference to the drawings.

FIG. 3 is a configuration diagram showing an image display system including an image composition system.
Referring to FIG. 3, the image display system 100 includes a computer (central processing unit; processor; data processing unit; image synthesis system) 100 that operates by program control, an image input unit 101, and an image output unit 102. Yes.
A computer (central processing unit; processor; data processing device; image synthesis system) 100 includes a preprocessing unit 110, a selection index calculation unit 120, a region selection unit 130, and an image synthesis unit 140.
Each of these units generally operates as follows.

The image input unit 101 receives a plurality of images having different properties such as a visible light image acquired from a camera with a built-in visible light sensor and a non-visible light image acquired from a camera with a built-in non-visible light sensor. The Further, for example, an image before flash, an image during flash, an image after flash, or images acquired with different camera settings may be used as input images. The input data format may be any format. For example, multiple image data such as a bitmap image or a compressed image may be used as an input image, or a video signal such as a composite video signal may be received as an input signal and converted into image data on the computer 100 side.
The input image is recorded in a memory (not shown) of the computer 100 or the like.
In the following, description will be given for an example in which a visible light image and a near-infrared light image as a non-visible light image are each input as an input image. Note that an ultraviolet image, a radio wave image, or a terahertz image may be used as the invisible light image. Any image may be used as long as the captured frequency component is imaged.

  The preprocessing unit 110 performs preprocessing on both the visible light image and the invisible light image. For the preprocessing, noise removal processing, blur removal processing, sharpening processing, image enhancement processing, or the like may be appropriately executed. Details of these processes will be described below. The noise removal process is a process for removing sensor noise and compression noise included in the visible light image and the non-visible light image. As a method of removing these noises, for example, noise removal may be performed using a filter such as a Gaussian filter, a median filter, or a bilateral filter. For example, non-patent document 1 may be based on image reconstruction. Noise removal may be performed using a method. Alternatively, noise removal may be performed using a patch-based method as described in Non-Patent Document 2.

  The blur removal process performed by the preprocessing unit 110 is a process for removing blur included in the visible light image and the invisible light image. Examples of blur include out-of-focus blur and motion blur. As a method for performing the blur removal process, the blur removal process may be performed using, for example, the Lucy Richardson method.

  Further, the preprocessing unit 110 desirably performs a sharpening process, an image enhancement process, a super-resolution process, or the like in one or a plurality of combinations in order to provide better visibility to the user. As an example of the sharpening process, for example, the visibility may be improved by correcting the brightness and contrast for each of the visible light image and the non-visible light image. Further, as an example of the image enhancement process, for example, a process of enhancing a high frequency component in the image may be performed for each of the visible light image and the non-visible light image in order to improve the resolution. As an example of the super-resolution processing, for example, the resolution described in Non-Patent Document 3 may be used to improve the resolution of the visible light image and the non-visible light image. The preprocessing unit 110 stores a large number of these preprocessed images in a memory (not shown) or the like, and stores the images so that they can be appropriately read during subsequent processing.

  Next, the selection index calculation unit 120 performs the following processing on the visible light image and the non-visible light image pre-processed by the pre-processing unit 110: 1) In an image defined based on the visibility of each pixel Two selection indexes and 2) an inter-image selection index defined based on connectivity between pixels in one image are calculated as index data for selecting a region with high visibility from each image. . When there are a plurality of preprocessing settings, the selection index calculation unit 120 calculates index data for each image.

  In addition, the selection index calculation unit 120 preferably performs normalization processing and alignment processing to match the pixel value of the invisible light image with the pixel value of the visible light image before acquiring the index data.

  As an example of the normalization process, the selection index calculation unit 120 calculates the average and variance of the visible light image and the non-visible light image, and normalizes the pixel values of the non-visible light image so that the average and the variance of the two match. You just have to. Further, as the alignment process, a positional deviation existing between the visible light image and the invisible light image is corrected. As a technique for correcting the positional deviation, for example, a region-based method may be used as in Non-Patent Document 4, or a method using feature points as in Non-Patent Document 5 may be used. is there.

  At this time, when the number of input images is large, the amount of normalization processing and alignment processing can be reduced by determining a reference image for determining visibility and angle of view in advance. In addition, it is good also as a structure which implements a normalization process and a position alignment process with respect to all the combinations of an input image.

  Hereinafter, the selection index data acquired by the selection index calculation unit 120, that is, the intra-image selection index and the inter-image selection index will be described more specifically. In the following, a case where there is one visible light image and one invisible light image will be described, but the present invention is not limited to this. For example, there may be a plurality of non-visible light images / multiple settings, or a plurality of visible light images / multiple settings. In short, anything is acceptable as long as there are a plurality of images having different properties.

  First, the in-image selection index will be described. The intra-image selection index may be defined based on the gradient of the luminance value for each pixel, for example. That is, the in-image selection index for each pixel of the visible light image and the in-image selection index for each pixel of the invisible light image may be defined as larger amounts as the luminance gradient in the image increases.

  In the above, as a method of extracting the luminance gradient, differentiation in the x direction and the y direction may be used, or an intra-image selection index may be calculated for each pixel using a Laplacian filter. Alternatively, a filter for extracting a specific high frequency may be defined in advance, and an intra-image selection index may be calculated for each pixel based on the magnitude of the frequency component. Alternatively, the difference value between the maximum value and the minimum value of the pixel values in the peripheral area may be used as the in-image selection index. Alternatively, information entropy in a peripheral region centered on the pixel of interest may be calculated and used as an in-image selection index.

In the above description, the method for calculating the in-image selection index for each pixel based on the luminance gradient has been described. However, the present invention is not limited to this.
For example, in addition to the gradient of the luminance value for each pixel, the in-image selection index for each pixel of the visible light image and the pixel of the invisible light image based on the direction of the luminance value gradient and the magnitude of the luminance value The in-image selection index may be calculated. For example, when the direction of the gradient of the brightness value and the magnitude of the brightness value are significantly different from those of the surroundings, a function defined so that the selection index in the image is increased may be used. That is, a histogram is created for the direction and magnitude of the gradient of the luminance value for each region, and the image for each pixel of the visible light image is increased so that the in-image selection index of the pixel greatly deviating from the average value in this histogram is increased. The inner selection index and the intra-image selection index for each pixel of the invisible light image may be calculated.

  In the above, the method for calculating the in-image selection index for each pixel based on the luminance gradient, the magnitude of the luminance value, and the direction of the luminance gradient has been described. However, statistical analysis such as regression analysis or machine learning was used. A method may be used.

  For example, from a visible light image and a non-visible light image prepared separately in advance, the prior probability distribution of the image is calculated using regression analysis, etc., and an index such as information entropy is calculated using this probability distribution, and based on this value The in-image selection index may be calculated.

  Alternatively, rectangular patches are sampled from these separately prepared visible light images and non-visible light images, these are clustered by a method such as the K-average method, and the intra-image selection index is calculated using the clustered results. May be. For example, the similarity of the image between the representative clustering center of the clustered rectangular patch and the rectangular patch extracted from the visible light image and the non-visible light image that are actually synthesized is expressed as PSNR (Peak signal-to -noise ratio), SSIM (Structural similarity), etc., and the amount that the value increases as they are similar can be used as an in-image selection index.

  Alternatively, from visible and non-visible light images prepared in advance, blood vessels such as fingerprints, irises, veins, characters, human bodies, faces, vehicles, buildings, defects in products and buildings, plant vegetation, object surface The characteristics of each detection target are learned in advance with a detector that detects a specific pattern, a pathological abnormality such as a tumor in the human body, a specific cell or chromosome, etc., and these detectors can easily detect them. You may employ | adopt the parameter | index which becomes so large value as a selection index in an image.

  Or, with a detector prepared in advance, blood vessels such as fingerprints, irises, veins, characters, human body, face, vehicles, buildings, defects in products and buildings, plant vegetation, specific patterns on the object surface, tumors in the human body After detecting pathological abnormalities such as specific cells, chromosomes, etc., the larger the area in which humans or machines can recognize or identify these objects with high accuracy, the larger the value. A quality evaluation index may be designed separately, and this image quality evaluation index may be used as an in-image selection index.

  Alternatively, a visible light image and a non-visible light image to be synthesized, and a synthesized image most desirable for the user manually created from these images (hereinafter referred to as a correct synthesized image) are prepared in advance. The in-image selection index may be calculated from the invisible light image and the correct composite image by recursive learning.

  Alternatively, the in-image selection index is not only automatically determined from the input visible light image and invisible light image, but the user may manually provide the in-image selection index for each pixel.

  Alternatively, the in-image selection index may have a smaller value for an abnormal region such as a shadow portion in illumination or a portion where the subject is clearly blocked by another object. By acquiring the index in this way, it is possible to prevent a synthesized image from being synthesized by selecting a shadow portion or a portion where the subject is clearly blocked by another object.

For example, for each of a visible light image and an invisible light image, a sample of an image with a high image selection index and a low reliability image are collected in advance, and regression analysis is performed using these images. The in-image selection index may be calculated.
In the following, the in-image selection index for each pixel of the visible light image and the invisible light image at the pixel p is represented as E ^VIS (p) and E ^INVIS (p), respectively. These intra-image selection indices for each pixel are appropriately recorded in a memory (not shown). Alternatively, from a plurality of sample images, the prior probability distribution of the visible light image and the invisible light image is calculated on a patch basis by deep learning or the like, and the larger the prior probability distribution, the higher the in-image selection index, An in-image selection index may be calculated.

Alternatively, with the intention of automatically or semi-automatically recognizing the finally synthesized image using image recognition technology, using a technique such as deep learning, for example, so that this recognition result is high, An in-image selection index may be designed.
In the above description, the case where the input image is one visible light image and one invisible light image is described, but the present invention is not limited to this. For example, when there are a plurality of visible light images and non-visible light images, the selection index calculation unit 120 may calculate an intra-image selection index corresponding to E ^VIS (p) and E ^INVIS (p) for each image.

Next, an inter-image selection index that is defined based on the gradient of luminance values and the connectivity between pixels will be described. Based on the gradient of the luminance value between pixel p and pixel q in one image and the connectivity between pixels, the inter-image selection index defined by the similarity between one image and another image is V ( p, q), an image selection index may be defined as follows.
The index between images is: (a) For each visible light image and non-visible light image, the closer the pixel value to the adjacent pixel is, the larger the index value is; and (b) between the visible light image and the invisible light image. It is defined so that the index value becomes small in the portion where the luminance gradient is similar.
As described above, the selection index calculation unit 120 determines whether the similarity is high or low based on features (such as a luminance gradient or an equivalent luminance range) between pixels between arbitrary regions included between the images of the synthesis target candidates. May be calculated as a value and acquired as an inter-image selection index.

In the following, a vertical vector I ^{VIS in} which pixel values of a visible light image having N pixels are arranged in raster scan order, and a vertical vector I ^VIS in which pixel values of an invisible light image having N pixels are arranged in raster scan order. The vector I ^INVIS is expressed as follows.
At this time, with respect to (a) above, for each of the visible light image and the invisible light image, as the amount that the inter-image selection index increases in the region where the pixel value of the adjacent pixel is closer, for example, two pixels are p. , q, the inter-image selection index between the pixels p and q may be defined as follows.
On the other hand, for (b) above, as an inter-image selection index in which the inter-image selection index becomes small in a portion where the luminance gradient of the visible light image and the non-visible light image is similar, two pixels are similarly represented by p, When q is used, an inter-image selection index between the pixels p and q may be defined as follows. However, γ is a positive constant predetermined by the user. Θ _VIS and θ _INVIS are positive constants determined in advance by the user.
Note that ε and γ are positive constants given in advance by the user.

Next, based on the inter-image selection index V _connect (p, q) and V _grad (p, q) between these two types of pixels, the selection index between the pixels p and q is, for example, as follows: And appropriately recorded in a memory (not shown).

The area selection unit 130 uses the selection indexes E ^VIS (p) and E ^INVIS (p) for each pixel calculated for each image by the selection index calculation unit 120 and the selection index V (p, q) between the pixels. Then, an area selection process for generating a composite pattern that determines which image of the input image to be used when the composite image is generated by the image composition unit 140 described later is performed. In the derived composite pattern data, information in which the priority image of each area and its area range are combined using a visibility index for each pixel and an index for other images of each image appears.
In the following, a set of discrete labels X = (X ₁ , X ₂ ... X defined for each pixel (region), which of the visible light image and the invisible light image is used as the composite pattern data. _p ... X _N ) Here, the set X of discrete labels is referred to as a discrete label set and may be expressed as X = {X _p }. Note that the discrete label may be changed for each pixel, and each grid in which several pixels are collected may be used as a unit area.
In the discrete label set X = {X _p }, when the value of the label X _p at the pixel p 1 is 0 when the image is synthesized, the visible light image is preferentially taken into the synthesized image at the pixel p 1, and the value of X _p Is a variable that means that the invisible light image is preferentially taken into the composite image at the pixel p 1.
Heretofore, the case where the set X of discrete labels corresponds to two input images of the visible light image and the non-visible light image and takes values of 0 and 1, but the present invention is not limited to this. . For example, when there are a plurality of images input by the image input unit 101, the region selection unit 130 may define X as a set of discrete labels corresponding thereto. More specifically, when the image input unit 101 inputs one visible light image and four non-visible light images, these discrete labels are five discrete values of 0, 1, 2, 3, and 4. Can be defined as a variable that takes

As a method of selecting a region by the region selection unit 130, for example, if a region is selected by calculating a discrete label set that minimizes the function E ({X _p }) using the discrete label set as a variable, Good. As a calculation formula of the discrete label set, a calculation formula including an intra-image selection index and an inter-image selection index can be used as in the following formula (8). It is also possible to assign a weight variable to each of the first term on the right side and the second term on the right side, and to change the priority of the intra-image selection index or the priority of the inter-image selection index.
Here, g _p (X _p ) and h _pq (X _p, X _q ) are functions defined for each pixel and between the pixels, and the selection index for each pixel calculated by the selection index calculation unit 120 , A function configured using a selection index between pixels. Specifically, for example, g _p (X _p ) and h _pq (X _p, X _q ) may be defined as follows.
As a method for minimizing such a function, for example, a graph cut described in Non-Patent Document 6 may be used, or a probability propagation method described in Non-Patent Document 7 may be used. Next, the area selection unit 130 records X = {Xp} that minimizes the above function in a memory (not shown) as appropriate, and an arbitrary image included in the input image used to generate a composite image And synthetic pattern data indicating regions with higher visibility in relation to other images.

The image composition unit 140 is based on the discrete label set X = {Xp} representing which image calculated by the region selection unit 130 is used preferentially and the input image (visible light image and invisible light image). Generate a composite image.
As a method for generating a composite image, for example, alpha blending may be used. Specifically, when the pixel value of the composite image at the pixel p is Ip, the pixel value of the visible light image and the pixel value of the non-visible light image at the pixel p are used to express the image using the following equation: What is necessary is just to synthesize.
Here, α is a blend ratio, which is an amount that takes a value in the range of 0 to 1, and is predetermined by the user.
When the component information of the non-visible light image is only the luminance value, alpha blending is performed only on the luminance component, and the values of the visible light image may be used for other components such as a color difference component. Alternatively, the correction coefficient may be calculated from the ratio between the luminance value of the visible light image and the luminance value of the composite image, and the color component may be restored with a value obtained by multiplying the calculated correction coefficient. Alternatively, blending may be performed by changing α for each pixel. For example, as the ratio of the in-image selection indexes E ^VIS (p) and E ^INVIS (p) for each pixel is larger, α may be made larger.

Alternatively, the image synthesis unit 140 may synthesize an image using Poisson editing as described in Non-Patent Document 8, instead of synthesizing an image using alpha blending. That is, the image is synthesized by solving the following equation (12) using the variational method so that the luminance value of the pixel for which the discrete label set X is 1 becomes the luminance value of the visible light image, The luminance value may be restored.
Here, Ω is a region where the discrete label set X is 1, and represents a region where an image is preferentially synthesized using a visible light image.
As in the case of generating a composite image by alpha blending, when the component information of the non-visible light image is only the luminance value, alpha blending is performed only for the luminance component, and the visible light image is used for other components such as a color difference component. The value of may be used. Alternatively, as described below, the color information may be restored by solving Equation (13) while adding constraint conditions for components other than the luminance component (for example, UV components). That is, the image may be synthesized by solving Equation (14) using a variational method.
Here, λ _U and λ _V are weights predetermined by the user. Alternatively, in order to restore information other than the luminance value, the relationship between the luminance value and the color component may be learned from the visible light image used for restoration, and the image may be restored using this. For example, a conditional probability distribution function of the U component and the V component in the luminance value I may be created from the visible light image used for generating the composite image, and the color component may be restored using this. More specifically, when the conditional probability distribution function with U and V in the luminance component I is P (U | I) and P (V | I), I and U satisfy the following equations: , V may be obtained to restore the color information.
Here, ηU and ηV are weights predetermined by the user. Until now, as a method for obtaining P (U | I) and P (V | I), a conditional probability distribution function with U and V in the luminance component I is used to generate a composite image. Although the case where an optical image is used has been described, a plurality of learning images prepared in advance may be used. In addition, the method using the conditional probability distribution function for the pixel values I, U, and V of one pixel has been described so far, but the probability distribution function may be configured using a local region such as a patch. In addition, when restoring the color, the image composition unit 140 restores the color information using Expressions (13) and (14) only when the color information is inaccurate due to overexposure, blackout, and the like. In the region, the value of the visible light image may be used.
The composite image restored as described above is recorded in a memory (not shown) or an auxiliary storage device (not shown) by the image composition unit 140.

  In the present invention, the method of using Y of the YUV component as the luminance information has been described, but other color expression methods may be used. For example, the V component in the HSV color space may be handled as luminance information, and a composite image may be generated by the above method. Alternatively, the area to be processed in the YUV color space and the area to be processed in the HSV color space may be switched legally by, for example, threshold processing. In short, any method may be used as long as it is a method for separating brightness information corresponding to a luminance value and color information which is other information.

  In the present invention, the method of expressing the color of the composite image using Expression (13) or Expression (14) has been described, but the present invention is not limited to this. For example, only the luminance component is restored in advance using Expression (12), and then the color component is constrained so that the color components in the peripheral area of pixels having similar luminance components are the same color component. This may be expressed while imposing. Specifically, based on the luminance component restored by equation (12), an area having a luminance component pattern similar to the area where the color component is to be expressed is searched from the image where the luminance component is restored, and this search is performed. The color component may be restored by pasting the color component of the region.

  In the present invention, the method of expressing the color of the composite image using the formula (13) or (14) has been described, but the present invention is not limited to this. For example, the components of the wavelength bands of the visible image and the invisible image prepared in advance are represented by a small number of basis functions using principal component analysis, etc., and then only the luminance component is restored by Expression (12), and the restored luminance From the component values, the color component may be expressed as a linear combination of these few bases.

  Such color restoration may be performed on the entire composite image, or may be performed only on a shadow in the image or an area where an object is clearly occluded.

  The image output unit 102 outputs the composite image generated by the image composition unit 140. Further, the image output unit 102 may output not only the composite image but also the visible light image and the non-visible light image, each image with preprocessing added, or the discrete label set calculated by the region selection unit 120 at the same time. Good.

Next, the overall operation of the image display system will be described with reference to FIGS. FIG. 4 is a flowchart illustrating an operation example of the image display system.
First, when a visible light image and a non-visible light image are input to the image input unit 101, the image input unit 101 records both input images in a memory (not shown) (S111). The preprocessing unit 110 performs preprocessing such as noise removal on each of the visible light image and the non-visible light image (S112).

Next, the selection index calculation unit 120 performs normalization processing and alignment processing on each of the visible light image and the non-visible light image preprocessed by the preprocessing unit 110, and then selects the in-image selection index. Then, two of the inter-image selection indexes are calculated as selection index data (S113).
Next, the region selection unit 130 uses the selection index data for each pixel calculated by the selection index calculation unit 120 and the selection index data between pixels to select a region with higher visibility to be introduced into the composite image. (S104).

  Next, the image composition unit 140 generates a composite image using the one or more regions selected as having higher visibility derived from the region selection unit 130, the visible light image, and the invisible light image (S105). ).

  Finally, the composite image synthesized by the image composition unit 140 is output from the image output unit 102 (S106).

[Description of effects]
Next, the effect of this embodiment will be described.
The image composition system according to the present embodiment has an in-image selection index that defines a high visibility in an image for each image, so that it can be viewed from a plurality of images having different sensitivity characteristics with respect to an input wavelength band. In this configuration, a region having a higher characteristic than other images is selected in relation to the other images. Furthermore, the image composition system according to the present embodiment has an inter-image selection index that defines the degree of similarity of image features between images (between arbitrary regions), so that the selected high-visibility region is smooth. Therefore, it is possible to generate an image having no sense of incongruity as a whole image.
For this reason, even if it is other than a saturated region such as overexposure or blackout, if there is an equivalent region with higher visibility in another image, it is replaced. As a result, a composite image having excellent visibility as a whole can be obtained.

  Here, the image composition according to the present invention will be described with a simplified depiction using FIG. 5 and FIG.

In FIG. 1 of FIG. 5, a visible light image and a near-infrared light image (non-visible light image) are received as two input images. As shown in FIGS. 5A and 5B, the two input images have the following characteristics.
Visible light images have good and bad visibility of human faces due to the influence of background illumination. In addition, the visible light image has good visibility of the clothing area, and the clothing pattern can also be visually recognized. Moreover, the visibility which is good in a lighting fixture part is not acquired.

  On the other hand, near-infrared light images also have good and bad visibility of human faces. In addition, the near-infrared light image has a low reflectance of a paint or the like in the near-infrared region, so the visibility of the clothing region is poor.

By implementing the present invention, the selection index calculation unit 120 calculates a higher area where each selection index data can be viewed better than the other of the visible light image and the non-visible light image.
For this reason, the result (region selection result) selected by the region selection unit 130 is as follows. As shown in FIG. 5C, each region is obtained from different images in the face part, the clothing part, and the lighting part. Adopted.
The image composition unit 140 refers to the region selection result and the input image, and as a result, a composite image in which regions having good sensitivity characteristics with respect to the wavelength band and continuous regions are cut out integrally (FIG. 5). (D)) is synthesized.
As a result, an image having high visibility as a whole including individual areas such as a face part, a clothing part, and a lighting part can be obtained. In addition, position correction and hue correction between both images are also performed.

In FIG. 2 of FIG. 6, (a) a near-ultraviolet light image, (b) a visible light image, and (c) a near-infrared light image are accepted as three flower images having different input properties. As shown in the figure, each input image has its own characteristic.
In the near-ultraviolet light image (a), a beautiful pattern that cannot be visually recognized by the human eye is depicted at the base of the petal.
In the visible light image (b), outlines and hues that are visible with the naked eye are depicted.
In the near-infrared light image (c), the contours of the central part of the flower (male and female moth) and petals are depicted more clearly than the visible light image.
The three input images are received, and the selection index calculation unit 120 calculates the area where the selection index data of each image can be visually recognized better than the other images.
For this reason, the result (region selection result) selected by the region selection unit 130 reflects the visibility and continuity of each region from different images, as depicted in FIG. Adopted. The hatched area in FIG. 5D is a near ultraviolet light image area, the white area is a visible light image area, and the black area is a near infrared light image area.
The image composition unit 140 refers to the region selection result and the input image, and as a result, a composite image in which regions having good sensitivity characteristics with respect to the wavelength band and continuous regions are cut out integrally (FIG. 6). (E)) is synthesized.
As a result, an image having high visibility such as a flower shape or pattern as a whole is obtained. In addition, position correction and hue correction between both images are also performed.

[Modification]
As a modification of the present invention, as shown in FIG. 7, an image input unit 101, a preprocessing unit 110, a selection index calculation unit 120, a region selection unit 130, an image composition unit 140, an abnormal region detection unit 150, an image output unit. Processing may be performed with the configuration of 102.
Note that the image input unit 101, the image output unit 102, the preprocessing unit 110, the region selection unit 130, and the image composition unit 140 are the same as described above, and thus the description thereof is omitted.

Selection index calculating section 120 is a function that combines the equation represented by g _p in (9) (X _p), the abnormal region value Q _p of the abnormal region detection unit 150 described later and (X _p) d _p Let (X _p ) be a new selection index for each pixel. In this modification, when the selection index calculation unit 120 first calculates the selection index for each pixel, the abnormal area value Q _p (X _p ) is set to zero, and the selection index d _p (X _{p for} each pixel). ) May be calculated.
Note that the selection index d _p (X _p ) for each pixel may be represented by a function having the abnormal region value Q _p (X _p ) as a variable, as shown in the following formula (15). The following equation (15) is represented as a function of the sum of g _p (X _p ) represented by equation (9) and the abnormal region value Q _p (X _p ). For example, the selection index d _p (X _p ) may be expressed as a linear combination or multiplication of g _p (X _p ) and Q _p (X _p ).

The abnormal area detection unit 150 operates to detect an area (abnormal area) that is not preferable for the user in the image synthesized by the image synthesis unit 140. The unfavorable area for the user here is a shadow of an object that does not exist in the reference image or a moving object that does not exist in the reference image. Alternatively, when there is a slight misalignment between input images, artifacts caused by the misalignment are also unfavorable areas for the user. The abnormal area detection unit 150 calculates such an unfavorable area for the user between the image (composite image) generated by the image synthesis unit 140 and each image (input image) input by the image input unit 101. Detection is performed by calculating the correlation. For example, as the correlation calculated between the image generated by the image composition unit 140 and each image input by the image input unit 101, a luminance value or a normalized cross-correlation between images may be used.
It is desirable that the abnormal area detection unit 150 obtains the correlation between the input image and the composite image not only for the reference image but also for many input images. Further, a value may be given to the correlation result value for each image type. When the correlation between the composite image and only the reference image is taken, the abnormal area detection unit 150 operates so as not to be regarded as an abnormal area as long as the composite image and the reference image are similar. For this reason, by adding the processing by the abnormal region detection unit 150, the composite image becomes an image having a high similarity to the reference image. On the other hand, when the correlation is performed for the input image (group) other than the composite image and the reference image, the abnormal area detection unit 150 detects an abnormal area based on all the images (group) correlated with the composite image. Operate. For this reason, the process by the abnormal region detection unit 150 makes the final composite image an image having good visibility as a whole without being excessively similar to any one input image. .

More specifically, for example, the abnormal area detection unit 150 is defined by the following equation (15) in order to determine whether a certain pixel of the image synthesized by the image synthesis unit 140 is an abnormal area. The calculated value may be calculated as the abnormal region value Q _p (X _p ). When the number of input images is three or more, the φ variable may be increased as appropriate.
However, φvis and φinvis are normalized cross-correlations between the luminance value and the gradient of the luminance value between the image (composite image) generated by the image synthesis unit 140 and the visible light image or the non-visible light image, respectively. . Φth is a threshold value for determining whether or not the pixel is an abnormal region.

When Expression (15) is used, when the sum of the normalized correlation φvis between the synthesized image and the visible image and the normalized cross-correlation φinvis between the synthesized image and the invisible light image is equal to or less than φth, Q _p (X _p ) Becomes zero. For this reason, the selection index calculation unit 120 is not affected by the abnormal region in calculating the index value. As a result, the pixel p does not become an abnormal region.
On the other hand, when the sum of the normalized correlation φvis between the synthesized image and the visible image and the normalized cross-correlation φinvis between the synthesized image and the invisible light image is greater than or equal to φth, Q _p (X _p ) is a value greater than or equal to zero. . For this reason, the selection index calculation unit 120 calculates a value affected by the abnormal region when calculating the index value. As a result, the region selection unit 130 can more easily select a reference image in which no abnormal region exists.
Although the case where Q _p (X _p ) is a simple sum of φinvis and φvis has been described above, the present invention is not limited to this. For example, Q _p (X _p ) may be a linear combination of φinvis and φvis, or the relationship (function form) between Q _p (X _p ) and φvis of φinvis may be calculated by machine learning. Good.
Further, the feedback to the selection index calculation unit 120 by the abnormal region detection unit 150 can obtain an image having higher visibility by repeating an appropriate number of times. The number of feedbacks may be a predetermined number, or may be stopped when the change amount of the composite image converges to some extent.

  By having the above configuration, the image composition system can synthesize an image having high visibility while removing an abnormal region in the composite image.

  As described above, according to the present image composition system, a good visibility region with respect to other images included in an image having each sensitivity characteristic is derived from a plurality of images having different properties over the entire image, and the visual recognition is performed. It is possible to obtain a composite image with further improved visibility by reflecting the image area whose visibility is improved over the entire image so as to be visible.

  Each unit of the image processing system may be realized using a combination of hardware and software. In the form in which hardware and software are combined, as shown in FIG. 8, an image composition program is developed in RAM, and hardware such as a control unit (CPU: Central Processing Unit) is operated based on the program. , Function as all or a part of the preprocessing unit, the selection index calculation unit, the region selection unit, and the image synthesis unit. Further, this program may be recorded non-temporarily on a recording medium and distributed. The program recorded on the recording medium is read into a memory via a wired, wireless, or recording medium itself, and operates a control unit or the like. Examples of the recording medium include an optical disk, a magnetic disk, a semiconductor memory device, and a hard disk. Further, all or a part of each part may be formed by constructing a circuit on an LSI (Large Scale Integration).

  The specific configuration of the present invention is not limited to the above-described embodiment, and even if there is a change in a range that does not depart from the gist of the present invention, such as separation / merging of block configuration, replacement of procedures, combination of each embodiment, etc. It is included in this invention.

In addition, a part or all of the above-described embodiments can be described as follows. Note that the following supplementary notes do not limit the present invention.
[Appendix 1]
A selection index calculation unit that obtains index data for selecting a region with higher visibility from a plurality of images having different properties by quantifying the index value within the image of each image;
Referring to the index data acquired from the plurality of images, a process of deriving a priority image and its region range integrally from the plurality of images to be used for generating a composite image An area selection section to select and
An image composition system comprising: an image composition unit that generates a composite image by combining regions with higher visibility selected by the region selection unit.

[Appendix 2]
The selection index calculation unit receives a plurality of images having different sensitivity characteristics with respect to a wavelength band as a plurality of images having different properties, and each of the regions having higher visibility based on each wavelength band of the plurality of images, Quantified and acquired by the index value within the image and the index value between images,
The region selection unit refers to the index data of each image, derives a region with higher visibility from each image with respect to another image, and selects the priority image. The image composition system described in the appendix.

[Appendix 3]
The selection index calculation unit, as the index data for selecting a region having high visibility from a plurality of images having different sensitivity characteristics with respect to a wavelength band, in-image selection index data representing a high visibility for each image, The image composition system as described in the above supplementary note, wherein inter-image selection index data representing the magnitude of continuity of appearance between images is acquired for each image.

[Appendix 4]
The selection index calculation unit calculates a feature vector for each pixel of each of the plurality of images as the index data for selecting a region having high visibility from a plurality of images, and uses a discriminator obtained by machine learning. The image composition system according to the above supplementary note, wherein an index value indicating the size of whether or not each pixel is a region with high visibility is obtained from the feature vector of each pixel.

[Appendix 5]
The selection index calculation unit and the region selection unit, the images having a plurality of layers having different frequency components that are the plurality of images, each processing is performed in parallel for each layer,
The image composition system according to the above supplementary note, wherein the image composition unit generates the composite image by combining regions with higher visibility selected by the region selection unit in each layer.

[Appendix 6]
The selection index calculation unit performs a normalization process on a reference image for determining visibility or an image of the plurality of images for the plurality of images before acquiring the index data from the plurality of images. The image composition system as described in the above supplementary note.

[Appendix 7]
The selection index calculation unit performs an alignment process on a reference image for determining an angle of view or an image of the plurality of images for the plurality of images before acquiring the index data from the plurality of images. The image composition system as described in the above supplementary note.

[Appendix 8]
The image acquisition unit further acquires a plurality of images having different sensitivity characteristics with respect to the wavelength band,
The selection index calculation unit obtains, as the index data, an index value representing a level of similarity based on a feature of a pixel level between arbitrary regions included between images included in the plurality of images. And
The area selection unit refers to the index data calculated by the selection index calculation unit including the characteristics of the pixel level for each image, determines the priority image and its area range, the index value for each pixel, and the pixel interval The image composition system as described in the above supplementary note, wherein the region selection process is performed for each pixel by calculating the index value of ## EQU2 ## with an energy function having as a variable.

[Appendix 9]
The selection index calculation unit
As the inter-image selection index data,
As described in the above supplementary note, as a function indicating connectivity between pixels, for each of the plurality of images, an area selection index is obtained in a region having a pixel value closer to an adjacent pixel. Image composition system.

[Appendix 10]
The selection index calculation unit
As the inter-image selection index data,
The image according to the above-mentioned supplementary note, which is obtained using a function that reduces an inter-image selection index in a portion where the luminance gradients of the plurality of images are similar as a function indicating a gradient of luminance values between pixels. Synthesis system.

[Appendix 11]
The selection index calculation unit
The inter-image selection index data,
As a function indicating connectivity between pixels, for each of the plurality of images, a function that increases an inter-image selection index in a region closer to a pixel value with an adjacent pixel;
A function that indicates a gradient of luminance values between pixels is acquired using a function that combines a function that reduces an inter-image selection index in a portion where the luminance gradients of the plurality of images are similar to each other. The image composition system described in the above supplementary note.

[Appendix 12]
The image composition system according to the above supplementary note, wherein the selection index calculation unit acquires an index having a larger value as the contrast is larger as the intra-image selection index data.

[Appendix 13]
The image composition unit is configured to display the color components of the invisible light region included in the image to be synthesized based on the continuity of the color components of the image acquired from the sensor in the visible light region and the appearance frequency of the color components. The image composition system as set forth in the above supplementary note, wherein a composite image is generated after correcting the color component of the region.

[Appendix 14]
The image processing apparatus further includes an abnormal region detection unit that obtains an index value of an abnormal region with respect to the reference image included in the composite image based on a correlation between the composite image combined by the image combining unit and a reference image included in the plurality of images. And
The image composition system according to the above supplementary note, wherein the selection index calculation unit acquires the index data with a function having the index value of the abnormal region as a variable.

[Appendix 15]
An abnormal region detection unit for obtaining index values of abnormal regions for the plurality of images of the composite image based on the correlation between the composite image and the plurality of images combined by the image composition unit;
The image composition system according to the above supplementary note, wherein the selection index calculation unit acquires the index data with a function having the index value of the abnormal region as a variable.

[Appendix 16]
Acquire multiple images with different properties,
Index data for selecting a region with higher visibility from the plurality of images is quantified by an index value within the image of each image,
Referring to the index data acquired from the plurality of images, a process of deriving a priority image and its region range integrally from the plurality of images to be used for generating a composite image Select
An image synthesizing method by an information processing system, wherein a synthesized image is generated by combining selected regions with higher visibility.

[Appendix 17]
The information processing system includes:
Receiving a plurality of images having different sensitivity characteristics with respect to wavelength bands as a plurality of images having different properties, a region having higher visibility with reference to each wavelength band of the plurality of images, and an index value within the image of each image And numerically obtained with the index value between images,
The image composition according to the above supplementary note, wherein the priority image is selected by deriving a region having higher visibility with respect to another image from each image with reference to the index data of each image Method.

[Appendix 18]
In the information processing system, as the index data for selecting a region having high visibility from a plurality of images having different sensitivity characteristics with respect to the wavelength band, selection index data in the image representing high visibility for each image, and image The image composition method as described in the above supplementary note, wherein inter-image selection index data representing the magnitude of continuity of appearance between images is acquired for each image.

[Appendix 19]
The information processing system uses a discriminator obtained by machine learning by calculating a feature vector for each pixel of the plurality of images as the index data for selecting a region having high visibility from the plurality of images. The image composition method according to the above supplementary note, wherein an index value indicating the size of whether or not each pixel is a highly visible region is obtained from the feature vector of each pixel.

[Appendix 20]
The information processing system includes:
An image having a plurality of hierarchies with different frequency components being the plurality of images, and performing each processing in parallel for each hierarchy,
The image composition method according to the above supplementary note, wherein the composite image is generated by combining regions with higher visibility selected in each layer.

[Appendix 21]
Before the information processing system acquires the index data from the plurality of images, the information processing system performs a normalization process on the reference image for determining visibility or any of the plurality of images for the plurality of images. The image composition method as described in the above supplementary note.

[Appendix 22]
The information processing system performs an alignment process on a reference image for determining an angle of view or an image of any of the plurality of images before obtaining the index data from the plurality of images. The image composition method as described in the above supplementary note.

[Appendix 23]
The information processing system includes:
Generate multiple images with different sensitivity characteristics for wavelength bands,
As the index data, the index value representing the level of similarity based on the feature of the pixel level between the arbitrary areas included between the images included in the plurality of images is obtained by quantification,
The selection index calculation unit refers to the index data calculated for each image including the pixel level feature, and the priority image and its area range are changed as the variable for the index value for each pixel and the index value between pixels. The image composition method as described in the above supplementary note, wherein region selection processing is performed for each pixel by calculating with the energy function included in the image.

[Appendix 24]
The information processing system includes:
As the inter-image selection index data,
As described in the above supplementary note, as a function indicating connectivity between pixels, for each of the plurality of images, an area selection index is obtained in a region having a pixel value closer to an adjacent pixel. Image composition method.

[Appendix 25]
The information processing system includes:
As the inter-image selection index data,
The image according to the above-mentioned supplementary note, which is obtained using a function that reduces an inter-image selection index in a portion where the luminance gradients of the plurality of images are similar as a function indicating a gradient of luminance values between pixels. Synthesis method.

[Appendix 26]
The information processing system includes:
The inter-image selection index data,
As a function indicating connectivity between pixels, for each of the plurality of images, a function that increases an inter-image selection index in a region closer to a pixel value with an adjacent pixel;
A function that indicates a gradient of luminance values between pixels is acquired using a function that combines a function that reduces an inter-image selection index in a portion where the luminance gradients of the plurality of images are similar to each other. The image composition method according to the above supplementary note.

[Appendix 27]
The image synthesizing method according to the above supplementary note, wherein the information processing system acquires an index having a larger value as the contrast is larger as the intra-image selection index data.

[Appendix 28]
The information processing system converts the color components of the invisible light region included in the image to be synthesized into visible light based on the continuity of the color components of the image acquired from the sensor in the visible light region and the appearance frequency of the color components. The image composition method as described in the above supplementary note, wherein a composite image is generated after correcting the color component of the region.

[Appendix 29]
Obtaining an index value of an abnormal region with respect to the reference image of the composite image based on a correlation between the composite image and a reference image included in the plurality of images;
The image composition method according to the above supplementary note, wherein the index data is acquired by a function having the index value of the abnormal region as a variable.

[Appendix 30]
Obtaining an index value of an abnormal region for the plurality of images of the composite image based on a correlation between the composite image and the plurality of images;
The image composition method according to the above supplementary note, wherein the index data is acquired by a function having the index value of the abnormal region as a variable.

[Appendix 31]
The processor of the information processing system,
A selection index calculation unit that obtains index data for selecting a region with higher visibility from a plurality of images having different properties by quantifying the index value within the image of each image;
Referring to the index data acquired from the plurality of images, a process of deriving a priority image and its region range integrally from the plurality of images to be used for generating a composite image An area selection section to select and
An image composition program that operates as an image composition unit that generates a composite image by combining regions with higher visibility selected by the region selection unit.

[Appendix 32]
The selection index calculation unit receives a plurality of images having different sensitivity characteristics with respect to wavelength bands as a plurality of images having different properties, and each of the regions having higher visibility based on each wavelength band of the plurality of images, The image index value between images and the index value between images are numerically acquired,
The region selection unit refers to the index data of each image, derives a region having higher visibility from each image with respect to another image, and causes the priority image to be selected. The image composition program described in the appendix.

[Appendix 33]
In the selection index calculation unit, as the index data for selecting a region having high visibility from a plurality of images having different sensitivity characteristics with respect to a wavelength band, in-image selection index data representing high visibility for each image, The image composition program as described in the above supplementary note, wherein the image selection index data indicating the degree of continuity of appearance between images is acquired for each image.

[Appendix 34]
The selection index calculation unit calculates a feature vector for each pixel of each of the plurality of images as the index data for selecting a region with high visibility from a plurality of images, and an identifier obtained by machine learning. The image composition program as described in the above supplementary note, wherein an index value indicating the size of whether or not each pixel is a region with high visibility is obtained from a feature vector of each pixel.

[Appendix 35]
In the selection index calculation unit and the region selection unit, the image having a plurality of layers having different frequency components that are the plurality of images, each processing is performed in parallel for each layer,
The image composition program according to the above supplementary note, wherein the composite image is generated by combining the image composition unit with a region having higher visibility selected by the region selection unit in each layer.

[Appendix 36]
Before the index data is acquired from the plurality of images, the selection index calculation unit is configured to perform a normalization process on a reference image for determining visibility or any of the plurality of images for the plurality of images. The image composition program as described in the above supplementary note.

[Appendix 37]
Before the selection index calculation unit acquires the index data from the plurality of images, the selection index calculation unit is configured to perform an alignment process on a reference image for determining an angle of view or any of the plurality of images. The image composition program as described in the above supplementary note.

[Appendix 38]
The information processing system is further operated as an image acquisition unit that acquires a plurality of images having different sensitivity characteristics with respect to a wavelength band,
The selection index calculation unit obtains, as the index data, an index value representing a level of similarity based on a feature of a pixel level between arbitrary areas included between the images included in the plurality of images. Let
The area selection unit refers to the index data calculated by the selection index calculation unit including the characteristics of the pixel level for each image, the priority image and its area range, the index value for each pixel and the pixel The image composition program as described in the above supplementary note, wherein the region selection process is performed for each pixel by calculating the index value of ## EQU2 ## with an energy function having as a variable.

[Appendix 39]
In the selection index calculation unit,
The inter-image selection index data,
The image according to the above-described supplementary note, wherein as a function indicating connectivity between pixels, an image selection index is acquired with a function that increases in a region having a pixel value closer to an adjacent pixel for each of the plurality of images. Synthesis program.

[Appendix 40]
In the selection index calculation unit,
The inter-image selection index data,
The image composition program as described in the above supplementary note, wherein a function that indicates a gradient of a luminance value between pixels is acquired by a function that reduces an inter-image selection index in a portion where luminance gradients of the plurality of images are similar to each other .

[Appendix 41]
In the selection index calculation unit,
The inter-image selection index data,
As a function indicating connectivity between pixels, for each of the plurality of images, a function that increases an inter-image selection index in a region closer to a pixel value with an adjacent pixel;
The above-mentioned supplementary note, wherein as a function indicating a gradient of luminance values between pixels, a function in which a selection index between images decreases in a portion where luminance gradients of the plurality of images are similar is acquired as a combined function. The image composition program described in 1.

[Appendix 42]
The image composition program according to the above supplementary note, wherein the selection index calculation unit causes an index having a larger value as the contrast increases to be acquired as the in-image selection index data.

[Appendix 43]
Based on the continuity of the color components of the image acquired from the sensor in the visible light region and the appearance frequency of the color components, the image composition unit converts the color components of the invisible light region included in the image to be synthesized into visible light. The image composition program as described in the above supplementary note, wherein a composite image is generated after correcting the color component of the region.

[Appendix 44]
The processor;
Based on the correlation between the synthesized image synthesized by the image synthesizing unit and the reference images included in the plurality of images, the synthesized image is operated as an abnormal region detecting unit that obtains an index value of the abnormal region with respect to the reference image. ,
The image composition program according to the above supplementary note, wherein the selection index calculation unit causes the index data to be acquired by a function having an index value of the abnormal region as a variable.

[Appendix 45]
The processor;
Operating as an abnormal region detection unit for obtaining index values of abnormal regions for the plurality of images of the composite image based on the correlation between the composite image and the plurality of images combined by the image composition unit;
The image composition program according to the above supplementary note, wherein the selection index calculation unit causes the index data to be acquired by a function having an index value of the abnormal region as a variable.

[Appendix 46]
A computer-readable recording medium that non-temporarily records the image composition program described in the above supplementary note.

DESCRIPTION OF SYMBOLS 1 Image composition system 10 Selection parameter | index calculation part (selection parameter | index calculation means)
20 area selection section (area selection means)
30 Image composition section (image composition means)
100 Computer (Central processing unit; Processor; Data processing unit; Image composition system)
101 Image input unit 102 Image output unit 110 Pre-processing unit (pre-processing means)
120 selection index calculation unit (selection index calculation means)
130 Area selection unit (area selection means)
140 Image composition unit (image composition means)
150 Abnormal area detection unit (abnormal area detection means)

Claims (10)

As index data for selecting a region having higher visibility than other images from each of a plurality of images having different properties, intra-image selection index data indicating the level of visibility for each image, and appearance between images A selection index calculation unit that quantifies and acquires inter-image selection index data representing the magnitude of continuity for each image ;
Referring to the index data acquired from the plurality of images, a process of deriving a priority image and its region range integrally from the plurality of images to be used for generating a composite image An area selection section to select and
An image composition system comprising: an image composition unit that generates a composite image by combining regions with higher visibility selected by the region selection unit. The selection index calculation unit accepts a plurality of images having different sensitivity characteristics with respect to wavelength bands as the plurality of images having different properties, and the visibility is higher than other images based on each wavelength band of the plurality of images. The index data indicating a high region is obtained by quantifying the index value within the image and the index value between images of each image,
2. The image composition system according to claim 1, wherein the region selecting unit refers to the index data of each image and selects a region having higher visibility from each image with respect to another image. . A function representing the height of visibility is defined for each type of image in advance,
The selection index calculation unit quantifies and acquires the index data of each of the plurality of images having different properties using the respective functions that represent high visibility for each predefined image type.
The image synthesizing system according to claim 1 or 2.   The selection index calculation unit performs a normalization process on a reference image for determining visibility or an image of the plurality of images for the plurality of images before acquiring the index data from the plurality of images. The image composition system according to any one of claims 1 to 3, wherein   The selection index calculation unit performs an alignment process on a reference image for determining an angle of view or an image of the plurality of images for the plurality of images before acquiring the index data from the plurality of images. The image composition system according to any one of claims 1 to 4, wherein The image acquisition unit further acquires a plurality of images having different sensitivity characteristics with respect to the wavelength band,
The selection index calculation unit obtains, as the index data, an index value representing a level of similarity based on a feature of a pixel level between arbitrary regions included between images included in the plurality of images. And
The area selection unit refers to the index data calculated by the selection index calculation unit including the characteristics of the pixel level for each image, determines the priority image and its area range, the index value for each pixel, and the pixel interval 6. The image composition system according to claim 1, wherein the region selection process is performed for each pixel by calculating an index value of ## EQU2 ## with an energy function having as a variable. The selection index calculation unit
The inter-image selection index data,
As a function indicating connectivity between pixels, for each of the plurality of images, a function that increases an inter-image selection index in a region closer to a pixel value with an adjacent pixel;
A function that indicates a gradient of luminance values between pixels is acquired using a function that combines a function that reduces an inter-image selection index in a portion where the luminance gradients of the plurality of images are similar to each other. The image composition system according to any one of claims 1 to 6.   The image composition system according to any one of claims 1 to 6, wherein the selection index calculation unit acquires an index having a larger value as the contrast is larger as the in-image selection index data. Acquire multiple images with different properties,
As index data for selecting a region having higher visibility than each other image from each of the plurality of images, intra-image selection index data indicating the level of visibility for each image, and continuity of appearance between images And the inter-image selection index data representing the size of each image is digitized for each image ,
By referring to the index data obtained by digitizing from said plurality of images, derived from the plurality of images, a higher region visibility used to generate a composite image, first image and the area scope integrally Process and select,
An image synthesizing method by an information processing system, wherein a synthesized image is generated by combining selected regions with higher visibility. Information processing system
As index data for selecting a region having higher visibility than other images from each of a plurality of images having different properties, intra-image selection index data indicating the level of visibility for each image, and appearance between images A selection index calculation means for obtaining and numerically obtaining the inter-image selection index data representing the degree of continuity of each image ,
Referring to the index data acquired from the plurality of images, a process of deriving a priority image and its region range integrally from the plurality of images to be used for generating a composite image Area selection means to select,
An image composition program that operates as an image composition unit that generates a composite image by combining regions with higher visibility selected by the region selection unit.