WO2017038090A1

WO2017038090A1 - Image processing device, image processing method, and recording medium whereon image processing program is stored

Info

Publication number: WO2017038090A1
Application number: PCT/JP2016/003969
Authority: WO
Inventors: 融濡木; 友宏松木; 松本　道弘
Original assignee: 日本電気航空宇宙システム株式会社
Priority date: 2015-09-01
Filing date: 2016-08-31
Publication date: 2017-03-09
Also published as: JP6744317B2; JPWO2017038090A1

Abstract

The purpose of the present invention is to efficiently generate a pansharpened image which has an appropriate tonal range and whereupon tone mapping processing has been carried out. Provided is an image processing device, comprising a tonal range-adjusted image generating unit which generates a first tonal range-converted image by converting the tonal range of a first image having one or more instances of color information. The image processing device further comprises a composite image generating means which generates a composite image which is an image which has one or more instances of color information and a resolution which is at least higher than the first image, by adjusting the first image using characteristic information generated on the basis of the first image and a second image which has a higher resolution than the first image and relating to the tonal range of the second image, and the tonal range of the first tonal range-converted image.

Description

Image processing apparatus, image processing method, and recording medium storing image processing program

The present invention relates to a technique for generating high-resolution image data using a plurality of image data having different resolutions.

A technique for obtaining a high-resolution multispectral image from an image having a high resolution and an image having a low resolution is known. For example, as a technology that can improve the visibility of high-resolution images (satellite images, etc.), a high-resolution multispectral image is created using a panchromatic image that is a high-resolution image and a multispectral image that is a low-resolution image. Processing to perform (pan sharpening processing) is known.

In general, when high gradation image data (for example, satellite image) is output to a low gradation display system (for example, graphic board, display, printer, etc.), the display range is compressed. As a result, gradation information may be lost in the display result. On the other hand, there is known a technique for reducing information loss in a display result by performing tone conversion processing on high tone image data in advance and performing range conversion (for example, range compression).

Techniques related to the above-described pan-sharpening process or tone mapping process are disclosed in the following patent documents.

Patent Document 1 (International Publication No. 2015/037189) discloses a technique related to noise removal in an image generated by pan-sharpening processing. The technique disclosed in Patent Document 1 performs multiple resolution decomposition of a high-resolution image to a resolution corresponding to the low-resolution image. Such a technique reduces noise by correcting the decomposed image using a low-resolution image.

Patent Document 2 (Japanese Patent Application Laid-Open No. 2014-174817) discloses that when tone mapping processing is performed on a plurality of high-tone images partially overlapping each other, luminance correction of each image is performed based on the luminance value distribution of each image. Disclosed is a technique for determining coefficients used in the above.

Japanese Patent Laid-Open No. 2013-109759 discloses a technique related to pan-sharpening processing using dictionary learning. The technique disclosed in Patent Document 3 extracts a feature vector from a panchromatic image and a multispectral image, and decomposes the feature vector into a vector having no missing value and a vector having a missing value. . The technique disclosed in Patent Literature 3 predicts a missing value (color information in a high-resolution image) based on a dictionary learned using a vector having no missing value.

Patent Document 4 (Japanese Patent Application Laid-Open No. 2011-1000042) uses a panchromatic image captured from a satellite and a color (multispectral) image that is geometrically transformed so as to be superposed on the panchromatic image. Disclosed is a technique for executing the process.

Japanese Patent Laid-Open No. 2015-033582 discloses pan sharpening using sparse spectral data detected by photon counting detection and dense panchromatic data detected by an energy integrator in an X-ray computed tomography apparatus. A technique for executing processing is disclosed.

Patent Document 6 (U.S. Pat. No. 8,737,733) discloses a statistical technique in which the luminance value of a panchromatic image and the luminance value of a multispectral image are close to each other in order to improve the color of a pan-sharpened image. A technique for converting using a model is disclosed.

Also, Non-Patent Document 1 discloses a technique for reducing information loss by decomposing a high gradation image into a base layer and a detail layer and performing range compression in accordance with the depicted scene.

International Publication No. 2015/037189 JP 2014-174817 A JP 2013-109759 A JP 2011-1000042 Japanese Patent Laying-Open No. 2015-033582 U.S. Pat. No. 8,737,733

Tone mapping processing that locally controls the range compression rate so as to reduce local information loss in image data has a high calculation cost. That is, when such processing is applied to a high-resolution panchromatic image or pan-sharpened image, there is a problem that the processing load is high. When tone mapping and pan-sharpening are executed, a high-resolution image with appropriate gradation is generated based on the gradation (for example, luminance gradation) of the original multispectral image or panchromatic image. It is required to do. None of the above patent documents is a technology that can directly solve such problems.

The present invention has been made in view of the above circumstances. That is, the main object of the present invention is to provide a technology capable of efficiently generating a high-resolution image having an appropriate gradation using a plurality of images having different resolutions.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention has the following arrangement. That is, an image processing apparatus according to an aspect of the present invention includes a gradation adjustment image generation unit that generates a first gradation conversion image by converting gradations of a first image having one or more color information. , Feature information relating to the gradation of the second image generated based on the first image and a second image having a higher resolution than the first image, and the first gradation conversion By adjusting the first image using the gradation of the image, a composite image that generates one or more color information and generates a composite image that is at least a higher resolution than the first image A generating unit.

The image processing method according to one embodiment of the present invention generates a first gradation-converted image by converting gradations of pixels included in a first image having one or more color information, and Characteristic information about the gradation of the second image, the first gradation-converted image, and the second gradation image generated based on the first image and the second image having a higher resolution than the first image, The first image is used to generate a composite image that has one or more pieces of color information and is at least a higher resolution than the first image.

The object can also be achieved by an image processing apparatus having the above configuration or a computer program for realizing the image processing method by a computer, and a computer-readable recording medium in which the computer program is stored. Achieved.

According to the present invention, it is possible to efficiently generate a high-resolution image having an appropriate gradation using a plurality of images having different resolutions.

FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to the first embodiment of the present invention. FIG. 2 is an explanatory diagram schematically showing an image having the same resolution as the panchromatic image. FIG. 3 is a flowchart showing a specific example of the operation of the image processing apparatus according to the first embodiment of the present invention. FIG. 4 is a block diagram illustrating a functional configuration of an image processing apparatus according to the second embodiment of the present invention. FIG. 5 is a block diagram illustrating a functional configuration of an image processing apparatus according to a modification of the second embodiment of the present invention. FIG. 6 is a diagram illustrating a configuration of a hardware device capable of realizing the image processing apparatus according to each embodiment of the present invention. FIG. 7 is a diagram schematically illustrating an example of a cumulative histogram when the luminance of the multispectral image before and after tone mapping and the luminance of the panchromatic image are normalized.

Prior to the description of the embodiment of the present invention, the technical background related to the present invention will be described in more detail. In the following, description will be made using satellite image data taken from a satellite as an example of image data, but the present invention is not limited to this.

There is a technical field (for example, image interpretation) that recognizes and classifies an object depicted in an image based on the result of a human viewing an image (image data). In such a technical field, it may be required to generate an image with high visibility when displayed, rather than an image whose description is physically correct.

For example, as a technology aimed at improving (improving) the visibility in the spatial direction and wavelength direction of an image, pan-sharp that creates a high-resolution multispectral image from a high-resolution panchromatic image and a low-resolution multispectral image There is a process. Such pan-sharpening processing is used, for example, as processing for improving the visibility of a satellite image captured by an artificial satellite.

Here, the panchromatic image is, for example, image data composed of a single color (monochrome) with a high spatial resolution. The multispectral image is image data including a plurality of pieces of spectral information (for example, color information in the visible light region), although the spatial resolution is lower than that of the panchromatic image. The content captured in the panchromatic image and the content captured in the multispectral image may be the same or substantially the same. In addition, as long as it is possible to determine the correspondence between the content captured in one image and the content captured in the other image, the content captured in each image may not be the same. As a method for capturing (or generating) these images (for example, imaging using a multispectral sensor and a panchromatic sensor), a well-known technique can be employed.

On the other hand, when considering the gradation in the image, the image data may have a gradation higher than the gradation that can be output by the display system (for example, graphic board, display, printer, etc.). For example, generally, satellite images captured by artificial satellites often have gradations higher than the gradations that can be output by the display system. When such high gradation data is output to a low gradation display system, the display range is compressed (gradation is crushed), and information loss occurs. In relation to the display of such high gradation data, a tone mapping process is known that reduces the loss of information by converting (compressing) the range in the gradation direction.

However, a tone mapping process that performs different processing for each local area in an image, which is generally called a local tone mapping process, has a high calculation cost. When such tone mapping processing is applied to a high-resolution panchromatic image or pan-sharpened image, the processing load is high.

On the other hand, for example, it is conceivable to reduce the calculation cost by performing the tone sharpening process after performing the tone mapping process on the low-resolution multispectral image.

However, when such a method is adopted, the brightness distribution in the image is changed by the tone mapping process for the low-resolution multispectral image, so the result of the pan-sharpening process may be inappropriate. .

Generally, there is a correlation between the luminance (luminance distribution) of a panchromatic image and the luminance (luminance distribution) of a multispectral image. That is, it is assumed that a portion having high luminance in the panchromatic image has high luminance in the multispectral image and vice versa. On the other hand, when tone mapping processing is performed on a multispectral image, the luminance distribution of the multispectral image is compressed, for example, to a certain range centered on a specific luminance average. That is, the shape of the luminance distribution is different between the original multispectral image and the tone-mapped multispectral image. As a result, when a tone mapping process is performed on a multispectral image, the correlation between the luminance of the multispectral image and the luminance of the panchromatic image is lost.

In a certain type of pan-sharpening processing, there is a case where processing for associating the panchromatic image with the luminance of the multispectral image (luminance matching) is executed. As described above, since the correlation is lost between the luminance of the tone-mapped multispectral image and the luminance of the panchromatic image, appropriate luminance matching cannot be performed. As a result, when a low-resolution multispectral image on which tone mapping has been performed in advance and a panchromatic image are used, there is a possibility that a pan-sharpened image having an appropriate gradation may not be generated.

On the other hand, the technique according to the present invention exemplified using the following embodiments can reduce the amount of data handled in the tone mapping process by incorporating the tone mapping process into the pan sharpening process, for example. . Furthermore, the technology according to the present invention can realize high-precision luminance matching processing in, for example, pan-sharpening processing. The technology according to the present invention can efficiently create a pan-sharpened image subjected to tone mapping processing. Hereinafter, the present invention will be described in detail using the description of each embodiment.

<First Embodiment>
DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments capable of implementing the present invention will be described in detail with reference to the drawings. The configurations described in the following embodiments are exemplifications, and the technical scope of the present invention is not limited thereto.

Note that the image processing apparatus described below may be configured using a single apparatus (physical or virtual apparatus). Such an image processing apparatus may be realized using a plurality of apparatuses (physical or virtual apparatuses). That is, in this case, the image processing apparatus is realized as a system including a plurality of apparatuses. A plurality of devices constituting such an image processing device may be communicably connected via a communication network (communication line) that is wired, wireless, or an arbitrary combination thereof. The communication network may be a physical communication network or a virtual communication network.

[Constitution]
Next, the configuration of the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram illustrating a functional configuration of the image processing apparatus 100. The direction of the arrow in FIG. 1 is a display showing an example of the transfer direction of the image data, and does not limit the present embodiment.

The image processing apparatus 100 according to the present embodiment is communicably connected to an input apparatus 200 that receives a low-resolution multispectral image and a high-resolution panchromatic image. The image processing apparatus 100 is communicably connected to an output apparatus 300 that outputs a pan-sharpened image generated by the image processing apparatus 100. A communication method for connecting the image processing apparatus 100, the input apparatus 200, and the output apparatus 300 may be selected as appropriate.

Next, elements constituting the image processing apparatus 100, the input apparatus 200, and the output apparatus 300 will be described.

First, the image processing apparatus 100 includes an enlargement processing unit 101, a luminance extraction unit 102, a tone mapping processing unit 103, an enlargement processing unit 104, a local feature extraction unit 105, an output luminance calculation unit 106, and an image composition unit 107. . These components constituting the image processing apparatus 100 may be communicably connected using an arbitrary communication method.

The enlargement processing unit 101 receives a multispectral image (sometimes referred to as a “first image”) from the input device 200 described later. Such a multispectral image is, for example, a color image including a plurality of pieces of spectral information (color information) as described above. The multispectral image is an image having a lower spatial resolution than a panchromatic image (sometimes referred to as a “second image”) that the image processing apparatus 100 separately receives from the input device 200.

The enlargement processing unit 101 converts the resolution of the multispectral image received from the input device 200. Specifically, the enlargement processing unit 101 up-samples, for example, the resolution of the multispectral image so as to be the same resolution as the panchromatic image received from the input device 200. As a method for converting the resolution of an image (upsampling method), a known method may be employed.

The luminance extraction unit 102 extracts gradation information related to the image from the input multispectral image. For example, the luminance extraction unit 102 may extract gradation information related to the luminance of the input multispectral image. Hereinafter, the gradation information regarding the luminance of the image may be simply referred to as “luminance information”. The luminance extraction unit 102 may extract the luminance (luminance value) for each pixel constituting the multispectral image, for example, and use the set as luminance information. In this case, the luminance information may be, for example, a luminance image generated by extracting only the luminance of each pixel included in the input multispectral image. The luminance range of the multispectral image (for example, the number of quantization bits) and the luminance range of the panchromatic image may be the same or different. The luminance extraction unit 102 may extract other information that can express the light and shade of each pixel constituting the image, other than “luminance”, as the gradation information regarding the image.

The tone mapping processing unit 103 compresses the luminance range by executing tone mapping on the low-resolution (low spatial resolution) luminance information extracted from the multispectral image by the luminance extraction unit 102 (range). Compression process). For example, the tone mapping processing unit 103 may execute the range compression processing by converting (compressing) the luminance value included in the luminance information extracted by the luminance extracting unit 102 so as to be within a certain luminance range. . For example, the tone mapping processing unit 103 may perform range compression processing on the luminance image extracted from the multispectral image by the luminance extraction unit 102. Hereinafter, the luminance information (luminance image) on which the range compression processing has been performed may be referred to as a “first gradation converted image”.

The enlargement processing unit 104 converts the resolution of the luminance information (the first gradation conversion image) that has undergone range compression in the tone mapping processing unit 103. Similar to the enlargement processing unit 101, the enlargement processing unit 104 may upsample the luminance information so as to have the same resolution as the panchromatic image received from the input device 200.

The local feature extraction unit 105 generates local feature information based on information (luminance change information) representing a local luminance change in the panchromatic image input from the input device 200. For example, the luminance change information may be a value indicating the degree of change in luminance. Specifically, the local feature extraction unit 105 performs matching (luminance matching) between the luminance of the panchromatic image input from the input device 200 and the luminance of the multispectral image, for example. Then, the local feature extraction unit 105 converts local luminance change information in the panchromatic image into a luminance range of the multispectral image based on the result of the luminance matching. Such a converted panchromatic image may be referred to as a “second gradation converted image”. The local feature extraction unit 105 generates local feature information based on the second tone conversion image.

The output luminance calculation unit 106 generates high-resolution luminance information using the local features generated by the local feature extraction unit 105 and the luminance information (range compressed) that has been upsampled by the enlargement processing unit 104. . The output luminance calculation unit 106 calculates the high-resolution luminance information by multiplying, for example, the local feature generated by the local feature extraction unit 105 and the luminance information upsampled by the enlargement processing unit 104. May be. The high-resolution luminance information generated by the output luminance calculation unit 106 includes local features of the panchromatic image and global features in which the luminance range is compressed.

The image composition unit 107 generates a pan-sharpened image based on the multispectral image upsampled by the enlargement processing unit 101 and the high-resolution luminance information generated by the output luminance calculation unit 106. The image synthesis unit 107 synthesizes the pan-sharpened image by replacing the luminance information of the multispectral image upsampled by the enlargement processing unit 101 with the high-resolution luminance information generated by the output luminance calculation unit 106, for example. May be. In this case, the image composition unit 107 may set the color component (color information) of the pan-sharpened image using the color information of the multispectral image upsampled by the enlargement processing unit 101. Hereinafter, the pan-sharpened image may be referred to as a “composite image”. The specific method for executing the pan sharpening process is not limited to the above.

The input device 200 includes a multispectral image input unit 201 and a panchromatic image input unit 202. The input device 200 may be connected to an image generation device (not shown) that captures a multispectral image and a panchromatic image. The input device 200 may include such an image generation device. The input device 200 supplies (provides) the multispectral image and panchromatic image received as input to the image processing device 100.

The multispectral image input unit 201 receives, as an input, a multispectral image captured using, for example, a multispectral sensor, and supplies the multispectral image to the image processing apparatus 100. Such a multispectral image may be sent to the enlargement processing unit 101 and the luminance extraction unit 102 in the image processing apparatus 100, for example.

The panchromatic image input unit 202 receives, for example, a panchromatic image captured using a panchromatic sensor or the like, and supplies the panchromatic image to the image processing apparatus 100. Such a panchromatic image may be sent to the local feature extraction unit 105 in the image processing apparatus 100, for example.

The output device 300 receives the pan-sharpened image generated by the image processing device 100 and outputs it to a display system (display device or the like) not shown. The output device 300 may output the received pan-sharpened image to a display system such as a graphic processing device, various display devices, or a printer in an information processing device such as a computer. The output device 300 may include such a display system.

[Operation]
Next, the operation of the image processing apparatus 100 according to the present embodiment will be described in detail with reference to FIGS. FIG. 3 is a flowchart illustrating the operation of the image processing apparatus 100 according to this embodiment. The flowchart illustrated in FIG. 3 is one specific example, and the order of each process can be appropriately changed within a range that does not affect the process result. In addition, each process of the flowchart illustrated in FIG. 3 may be executed in parallel as long as the process result is not affected.

First, the low-resolution multispectral image and the high-resolution panchromatic image received by the input device 200 are provided to the image processing device 100 (step S3001). Hereinafter, such a multispectral image may be represented as “M” and a panchromatic image may be represented as “P”.

Next, the image processing apparatus 100 (particularly the enlargement processing unit 101) converts the resolution of the provided multispectral image (step S3002). The enlargement processing unit 101 may upsample the multispectral image to the same resolution as the panchromatic image by using, for example, a bilinear method. The enlargement processing unit 101 may appropriately use a method other than the bilinear method (for example, a bicubic interpolation method), and the upsampling method is not limited to a specific method. Hereinafter, the up-sampled multispectral image may be represented as “U (M)”. Note that the processing in step S3002 and the processing in steps S3003 to S3007 described below may be executed in parallel.

Next, the image processing apparatus 100 (particularly the luminance extraction unit 102) extracts luminance information from the multispectral image M (step S3003). The luminance extraction unit 102 may extract the luminance information “I _M ” by performing color space conversion processing on the multispectral image M, for example. The conversion of the color space may be, for example, a process of converting the color space of the multispectral image “M” into a color space such as HLS (Hue, Lightness, Saturation), YCbCr, HCS (Hyperspherical Color Space). . The method for converting the color space is not limited to a specific method. In addition, the luminance extraction unit 102 may extract the luminance information using a known method other than the color space conversion described above. The luminance information “I _M ” includes the luminance value of each pixel included in the multispectral image M and has a resolution equivalent to that of the multispectral image “M”.

Next, the image processing apparatus 100 (in particular, the tone mapping processing unit 103) executes tone mapping processing (step S3004). The tone mapping process will be specifically described below. The tone mapping processing unit 103 separates (decomposes) the luminance information “I _M ” of the multispectral image “M” extracted by the luminance extracting unit 102 into a global feature component and a local feature component. The global feature component represents, for example, a spatially smooth luminance change in the image, and the local feature represents, for example, a local luminance change around a certain target pixel. As one specific example of such global feature components, information representing spatial illumination light included in an image may be used. As one specific example of the local feature component, information representing the reflectance of an object included in the image may be used. In general, light incident on the human visual system is represented by the product of the illumination light and the reflectance of the object, and human perception (sight) is known to be sensitive to the reflectance of the object. Yes. Accordingly, for example, by compressing the illumination light component (range compression), it is possible to effectively compress the luminance range of the entire image while maintaining the reflectance component that affects the perception of the eye.

Specifically, the tone mapping processing unit 103 inputs the luminance information “I _M ” of the multispectral image as “I” in the following formula (1), and uses the “I _M ” as the global feature component “B ( I) "and the local feature component" T (I) ".

However, in the above formula (1), “x _ij (I)” is a pixel value (real number) in the pixel in the “i” row “j” column of the image (or luminance information) “I”. The pixel value may be a luminance value in the pixel. In Expression (1), the image “I” may be an arbitrary image (or luminance information, etc.), “B (I)” represents a global feature component of the image “I”, and “T (I) “Represents a local feature component of the image“ I ”.

Specifically, the tone mapping processing unit 103 applies the global feature component “B (I _M )” regarding “I _M ” by applying an edge preserving smoothing filter to the luminance information “I _M ”, for example. You may ask for it. Assuming that the illumination light component included in the image changes more slowly than the reflectance component from the object, it is possible to extract the illumination light component by using the edge preserving smoothing filter. is there. The edge preserving smoothing filter may be any filter that preserves edge information included in an image and can average (smooth) the image. As the edge-preserving smoothing filter, for example, a k-nearest neighbor averaging filter, a bilateral filter, a median filter, or a WLS filter (Weighted Led Squares Filter) may be employed. By using an edge-preserving smoothing filter, edges are preserved when the global feature components of the image are extracted (the edges are not dull), so that the occurrence of halo in tone mapping processing can be suppressed. is there.

Further, the tone mapping processing unit 103 may obtain the local feature component “T (I _M )” using, for example, the following equation (2). In Expression (2), “I” represents an arbitrary image (or luminance information) (that is, if “I” in Expression (2) is replaced with “I _M ”, local feature component “T” from Expression (2). (I _M ) "is obtained). From the expression (2), the local feature component “T (I _M )” is obtained from the ratio (brightness ratio) between the luminance of a certain pixel included in the luminance information “I _M ” and the global feature. From equation (2), it can be considered that the local feature component “T (I _M )” represents the degree of change in luminance value with respect to the global feature component “B (I _M )” in each pixel. . For example, when the global feature component “B (I _M )” represents the illumination light component in the multispectral image, the local feature component “T (I _M )” is considered to represent the reflectance from the object.

Next, as shown in the following formula (3), the tone mapping processing unit 103 applies each element (pixel) “x _ij (B (I _M ))” of the global feature component “B (I _M )”. Then, by applying the smooth range compression function “f”, the range (pixel value range) is compressed. Then, the tone mapping processing unit 103 performs the range-compressed pixel value of each element and the pixel value of each element “x _ij (T (I _M ))” of the local feature component previously separated by Expression (1). Multiply with.

The tone mapping processing unit 103 obtains luminance information “R (I _M )” (first gradation conversion image) of the multispectral image subjected to tone mapping as described above. In the present embodiment, as the compression function “f”, for example, a function represented by the following expression (4) is used. The compression function illustrated in Expression (4) is a compression function that takes into account human visual characteristics, and can visually compress equivalently regardless of the magnitude of the luminance value of each pixel. The tone mapping processing unit 103 calculates a compressed luminance value by substituting the luminance value of each pixel of the multispectral image into Equation (4).

Note that the tone mapping processing unit 103 may adopt a function other than Expression (4) as the compression function “f”. That is, an appropriate compression function “f” may be appropriately selected according to the purpose of tone mapping.

The image processing apparatus 100 (particularly, the enlargement processing unit 104) converts the resolution of the luminance information “R (I _M )” of the multispectral image that has undergone range compression in the tone mapping processing unit 103 (step S3005). Specifically, the enlargement processing unit 104 upsamples the luminance information “R (I _M )” to the same resolution as the panchromatic image “P”. A specific upsampling method may be the same as that of the enlargement processing unit 101. Hereinafter, the luminance information after upsampling is expressed as “U ^o R (I _M )”.

Further, the image processing apparatus 100 (particularly the local feature extraction unit 105) includes the luminance information “I _M ” extracted by the luminance extraction unit 102 and the panchromatic image “P” in a high-resolution image. Local feature information (feature amount) is generated (step S3006). Note that the processing in step S3006 described below may be executed in parallel with the processing in steps S3004 and S3005 described above.

The local feature extraction unit 105 generates local feature information by executing the following processing, for example. First, the local feature extraction unit 105 performs luminance matching (histogram matching) using a luminance histogram of the panchromatic image “P” and the luminance information “I _M ” of the multispectral image. Accordingly, the local feature extraction unit 105 performs panchromatic image “P ^” (“second gradation conversion image” in which the pixel value (luminance value) of each pixel is converted in accordance with the luminance range of “I _M ”. )). At this time, since the multispectral image “M” input to the local feature extraction unit 105 is not tone-mapped, the luminance distribution of the luminance information “I _M ” of the multispectral image and the panchromatic image “P”. It is not significantly different from the luminance distribution. That is, the correlation between the luminance distribution of the luminance information “I _M ” of the multispectral image and the luminance distribution of the panchromatic image “P” is maintained. Therefore, the local feature extraction unit 105 can execute the luminance matching with high accuracy.

Here, it is difficult to accurately perform luminance matching between the luminance information of the multispectral image after tone mapping and the panchromatic image “P”. This is because the luminance distribution of the tone-mapped multispectral image is different in shape from the luminance distribution of the original multispectral image, so that the luminance information of the panchromatic image “P” and the luminance information of the multispectral image “M”. This is because the correlation between the two is lost. As a specific processing method for luminance matching, a known method (for example, a method disclosed in Patent Document 6, a method using a cumulative distribution function, or the like) can be employed.

Next, the local feature extraction unit 105 calculates local feature information (local feature amount) “μ” included in the panchromatic image “P ^”. The local feature information “μ” may represent, for example, the degree of local luminance change in the panchromatic image “P ^” (for example, the degree of luminance change in the target pixel). Specifically, the local feature information μ is, for example, “P ^” when the result of smoothing the panchromatic image “P ^” is expressed as “S (P ^)”. The luminance ratio between the luminance of a certain pixel and the luminance of the corresponding pixel in "S (P ^)" may be used.

This will be specifically described with reference to FIG. 2 and the following formula (5). FIG. 2 is a diagram schematically showing an image “I” having a resolution equal to that of the panchromatic image “P”. The image “I” may be, for example, an image obtained by up-sampling the luminance information “I _M ” of the multispectral image to the resolution of the panchromatic image. In this case, an image “I” having a resolution equal to that of the panchromatic image “P” is obtained by replacing each pixel of luminance information “I _M ” of the low-resolution multispectral image with “C _L (I) × C _P (I)”. It is considered that the image is subdivided into "".

“C _P (I)” is a pixel spacing ratio between the image “I” and the panchromatic image “P” in the pixel direction (horizontal direction). “C _P (I)” is calculated by, for example, “((total number of pixels in the pixel direction of the panchromatic image“ P ”) / (total number of pixels in the pixel direction of the image“ I ”before being subdivided))” ”. Is done.

“C _L (I)” is a pixel spacing ratio between the image “I” and the panchromatic image “P” in the line direction (vertical direction). “C _L (I)” is calculated by, for example, “((total number of pixels in line direction of panchromatic image“ P ”) / (total number of pixels in line direction of image“ I ”before being subdivided))” ” Is done.

Hereinafter, in FIG. 2, the “m” row “n” column in the region corresponding to the pixel “i” row “j” column of the luminance information “I _M ” of the multispectral image for the image “I”. The pixel value is expressed as “y _mn ^ij (I)”.

The luminance _{^{"y mn ij (P ^)}} " in a certain pixel having a panchromatic image "P ^", the luminance of a pixel of the smoothed panchromatic image _{"S (P ^)""} y mn ij (S (P ^)) ”And the local feature“ μ _mn ^ij ”is expressed by the following equation (6). That is, the local feature extraction unit 105 may calculate the local feature “μ _mn ^ij ” (luminance ratio) using the equation (6).

Next, the image processing apparatus 100 (particularly, the output luminance calculation unit 106), based on the result of step S3005 and the result of step S3006, high-resolution luminance information (local information) is reflected ( Output brightness adjustment information) is generated (step S3007). Specifically, the output luminance calculation unit 106 uses the local feature amount “μ _mn ^ij ” obtained by the local feature extraction unit 105 as high-resolution luminance information “y _mn ^ij ” obtained by the enlargement processing unit 104. The result obtained by multiplying (U ^o R (I _M )) ”may be output luminance adjustment information. In this case, the output luminance adjustment information is expressed in a format of “μ _mn ^ij * y _mn ^ij (U ^o R (I _M ))”. As described above, the global feature of “U ^○ R (I _M )” is subjected to range compression by the tone mapping processing unit 103 with respect to the luminance “I _M ” of the multispectral image “M”. Is the up-sampled luminance information. Thus, luminance information (output luminance adjustment information) including local features derived from the panchromatic image “P” and global features derived from the multispectral image “M” with a compressed luminance range. Is obtained.

The image processing apparatus 100 (in particular, the image composition unit 107) performs panning based on the output luminance adjustment information and the upsampled multispectral image (“U (M)”) generated by the enlargement processing unit 101. A sharpened image is generated (step S3008).

Specifically, the image composition unit 107 performs color space conversion on the multispectral image “U (M)” upsampled by the enlargement processing unit 101, and separates luminance information and color information. The conversion of the color space may be the same as the process in step S3003, for example. Next, the image composition unit 107 converts the luminance information separated in the image “U (M)” into the output luminance adjustment information “μ _mn ^ij * y _mn ^ij (U ^o R ( I _M )) ”. The image combining unit 107 combines the replaced luminance information and the color information separated from U (M), and returns (converts) the combined image to the original color space again. As a result, the image composition unit 107 obtains a pan-sharpened image on which tone mapping processing has been performed. The image composition unit 107 provides the generated pan-sharpened image to the output device 300.

The output device 300 outputs the pan-sharpened image supplied from the image processing device 100 (step S3009).

Hereinafter, the brightness information (output brightness adjustment information) “μ _mn ^ij * y _mn ^ij (U ^o R (I _M ))” generated by the output brightness calculating unit 106 is generated using a well-known method. It will be explained that the result of tone mapping can be approximated with high accuracy. Incidentally, the luminance information of the pan-sharpened image generated using well-known methods, is labeled "I _Q", the "I _Q" the pixel value of each pixel of the result of tone mapping (luminance value), " y _mn ^ij (R (I _Q )) ”.

Equation (7) represents the relationship between the luminance information “I _Q ” of the pan-sharpened image generated using a known method and the luminance information “I _M ” of the multispectral image.

In Expression (7), “U (I _M )” represents the result of up-sampling the luminance information “I _M ” of the multispectral image. “Y _mn ^ij (U (I _M ))” represents a pixel value (luminance value) of a specific pixel in the result of up-sampling the luminance information “I _M ” of the multispectral image. “Μ _mn ^ij ” represents a local feature amount.

From the formula (7) and the formulas (1) to (3), after creating a pan-sharpened image, an image when tone mapping processing is performed on the pan-sharpened image "R (I _Q ) The brightness value of “” is expressed by the following equation (8).

The image processing apparatus 100 according to the present embodiment up-samples the global characteristic component “B (I _Q )” of the luminance information “I _Q ” of the pan-sharpened image and the luminance information “I _M ” of the multispectral image. U can be approximated using a "global features component ^{_{of" B ○ U (I M)}} "(I M) ( equation (9)). The reason will be described below.

As shown in Expression (7), the luminance information “I _Q ” is a local luminance change of a scale (range) smaller than one pixel of the original multispectral image M with respect to the luminance value of “U (I _M )”. It is calculated by multiplying the information “μ” (“μ _mn ^ij ”). The local luminance change information “μ” is a local feature amount obtained from the high-resolution panchromatic image P. The local luminance change information “μ” represents, for example, a change in luminance value in a range smaller than the range represented by one pixel constituting the (original) multispectral image M before upsampling. It is considered that the effect (influence) that the “μ” representing the luminance change in a narrow range has on the global characteristics of the pan-sharpened image is sufficiently small (for example, it may be ignored). As a specific example, a case is assumed in which a luminance component to which a smoothing filter is applied is used as a global feature. When the global feature of the brightness information “I _Q ” of the pan-sharpened image is calculated, the effect of the local brightness change information “μ” is equalized by smoothing, so that the effect on the global feature is reduced. Thus, the global feature component “B (I _Q )” obtained from the luminance information of the pan-sharpened image is the global feature component “B ^○ U for the image“ U (I _M ) ”obtained by up-sampling the multispectral image. (I _M ) "can be approximated.

Hereinafter, the luminance information _{"I M"} result of tone mapping after upsampling multispectral image "M" is denoted as ^{_{"R ○ U (I M)}} ". First, when the approximation of the equation (9) is applied to the equation (8), the following equation (10) is obtained in consideration of the equations (1) to (3).

In many cases, the brightness range of an image does not change significantly before and after upsampling an image. That is, even if the range compression by tone mapping is performed at any timing before and after the multi-spectral image “M” is up-sampled, there is often no significant difference in the result. From the above, when the luminance information “I _M ” of the multispectral image “M” is up-sampled after tone mapping, and expressed as “U ^o R (I _M )”, in this embodiment, the expression (11) The approximation shown is considered to hold.

From the above, when Expression (11) is reflected to Expression (10), the following Expression (12) is obtained.

As shown in Expression (12), the image processing apparatus 100 according to the present embodiment performs the tone mapping on the luminance information “I _M ” of the low-resolution multispectral image “M” (“R (I _M )”. ) Is used to approximate luminance information “y _mn ^ij (R (I _Q ))” obtained by performing tone mapping on a high-resolution pan-sharpened image. Thus, according to the image processing apparatus 100 of the present embodiment, it is possible to avoid the execution of tone mapping processing for a high-resolution image (or luminance information), and thus the calculation cost is reduced.

Further, as described above, the image processing apparatus 100 according to the present embodiment uses the multi-spectral image “M” that is not tone-mapped and the panchromatic image “P”, and uses the local feature information (local feature amount). ) Is extracted (step S3006). Since the multispectral image “M” is not tone-mapped, the luminance distribution between the multispectral image “M” and the panchromatic image “P” is not significantly different. That is, the image processing apparatus 100 can execute appropriate luminance matching using the luminance information of the panchromatic image “P” and the luminance information “I _M ” of the multispectral image “M”. Thereby, for example, the image processing apparatus 100 generates a pan-sharpened image having an appropriate gradation in which the gradation change (contrast) for each pixel derived from the high-gradation panchromatic image P is appropriately reflected. .

This will be described using a specific example shown in FIG. FIG. 7 is a diagram schematically illustrating an example of a cumulative histogram when the luminances of the multispectral image and the panchromatic image are normalized. The solid line shown in FIG. 7A represents the normalized luminance cumulative frequency for the multispectral image. The dotted line shown in FIG. 7B represents the normalized cumulative frequency of luminance for the tone-mapped multispectral image. A one-dot chain line shown in C of FIG. 7 represents the cumulative frequency of normalized luminance regarding the panchromatic image. G1 in FIG. 7 schematically represents a luminance ratio between specific luminances when a panchromatic image is subjected to luminance mapping with respect to a multispectral image. G2 in FIG. 7 schematically represents a luminance ratio between specific luminances when luminance mapping is performed on a panchromatic image with respect to a multi-spectral image that has been tone-mapped.

Generally, by tone mapping, the luminance of a multispectral image is compressed to a predetermined luminance range centered on a specific luminance average. As a result, the cumulative frequency of the luminance (normalized) of the tone-mapped multispectral image changes more rapidly than the cumulative frequency of the original multispectral image (normalized). For this reason, when G1 and G2 are compared, the luminance ratio represented by G2 is smaller than the luminance ratio represented by G1. This is considered to indicate that the local contrast in the image (for example, the degree of change in luminance at a certain pixel) is small. That is, if the luminance of the panchromatic image is matched with the luminance of the tone-mapped multispectral image, the local luminance contrast derived from the panchromatic image may not be appropriately reflected. Specifically, for example, there is a possibility that the local brightness contrast sensed when a human visually recognizes the image is lowered. On the other hand, the image processing apparatus 100 according to the present embodiment performs luminance matching using a multispectral image that has not been tone-mapped and a panchromatic image. As a result, the image processing apparatus 100 can generate a pan-sharpened image in which local brightness contrast derived from the panchromatic image is appropriately reflected.

As described above, the image processing apparatus 100 according to this embodiment reduces the amount of data handled in the tone mapping process by executing the tone mapping process on the low-resolution multispectral image “M”. As a result, the image processing apparatus 100 creates a pan-sharpened image having an appropriate gradation subjected to the tone mapping process at high speed.

In summary, the image processing apparatus 100 according to the present embodiment extracts local features derived from a panchromatic image and multispectral images extracted based on the result of luminance matching between the panchromatic image and the multispectral image. A pan-sharpening process is executed using the extracted global features. The image processing apparatus 100 uses the original (non-tone mapped) multispectral image in luminance matching for obtaining local features. The image processing apparatus 100 uses luminance information of a low-resolution multispectral image when compressing the luminance range of the global feature (global luminance change) by tone mapping. As a result, the image processing apparatus 100 can reduce the calculation cost (computation amount) as compared with the case where the tone mapping process is executed after the high-resolution pan-sharpened image is created. As described above, according to the image processing apparatus 100 of this embodiment, a high-resolution image (for example, panning) having an appropriate gradation using a plurality of images (for example, panchromatic image and multispectral image) having different resolutions. (Sharpened image) can be generated efficiently.

<Modification of First Embodiment>
The following modifications can be realized for the image processing apparatus 100 according to the first embodiment of the present invention described above.

The image processing apparatus 100 does not necessarily need to use the luminance ratio between the pixel value of a specific pixel and the pixel value of the smoothed image as exemplified in the above formula (6) as the local feature amount. The image processing apparatus 100 can represent the brightness value of a pan-sharpened image in the form of a product of local features derived from a high-resolution panchromatic image and global features derived from a low-resolution multispectral image. Any possible process may be selected.

For example, as one of the pan-sharpening processes, the multispectral image is appropriately color-separated (decomposed into a hue component and a luminance component), the luminance component is replaced with the luminance component of the panchromatic image, and then the color synthesis (hue component) And a luminance component are known. The image processing apparatus 100 may employ such pan-sharpening processing. In this case, for example, the image processing apparatus 100 may use the brightness of the up-sampled multispectral image instead of the brightness of the smoothed image as the denominator of Expression (6). In equation (7), the brightness of the up-sampled multispectral image is multiplied by the local feature “μ”, so that the brightness information of the panchromatic image is used as it is as the brightness information of the pan-sharpened image. .

Further, the image processing apparatus 100 may use a color space (for example, HCS method) in which the value of each band (spectral band) in the multispectral image is proportional to the luminance. Specifically, the image processing apparatus 100 may convert the color space of the multispectral image provided from the input apparatus 200 into the color space as described above. In this case, it is possible to simplify the color separation and composition processing in the image composition unit 10 7. By multiplying the luminance of the multispectral image up-sampled by the enlargement processing unit 101 by the output luminance adjustment information obtained by the output luminance calculating unit 27, it is possible to generate a pan-sharpened image at high speed. .

<Second Embodiment>
Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 4 is a block diagram illustrating a functional configuration of an image processing apparatus 400 according to the second embodiment of the present invention.

As illustrated in FIG. 4, the image processing apparatus 400 according to the present embodiment includes a gradation adjustment image generation unit (gradation adjustment image generation unit) 401 and a composite image generation unit (synthesis image generation unit) 402. The gradation adjustment image generation unit 401 and the composite image generation unit 402 may be communicably connected using an arbitrary communication method. In the following, components constituting the image processing apparatus 400 will be described.

The gradation adjustment image generation unit 401 generates a first gradation conversion image by converting the gradation of the first image having one or more color information. The first image may be, for example, a multispectral image. For example, the gradation adjustment image generation unit 401 may convert the luminance gradation of each pixel included in the first image (for example, a multispectral image). In this case, the gradation adjusted image generation unit 401 may perform range compression on the gradation of the first image, for example, by executing tone mapping processing on the first image. The gradation adjustment image generation unit 401 may convert the gradation of the first image and the resolution of the first image in which the gradation is converted. The tone adjustment image generation unit 401 as described above may correspond to, for example, the luminance extraction unit 102, the tone mapping processing unit 103, the enlargement processing unit 104, and the like in the first embodiment. It is not limited.

The composite image generation unit 402 uses the feature information regarding the gradation of the second image having a higher resolution than that of the first image and the gradation of the first gradation conversion image, and uses the first image. Adjust. Based on the result, the composite image generation unit 402 generates a composite image having one or more pieces of color information and having a resolution higher than that of the first image. The second image may be, for example, a panchromatic image having a higher resolution than the multispectral image. The feature information may be information representing a local feature in the second image generated based on the first image and the second image, for example. The feature information may correspond to, for example, local feature information (local feature amount) in the first embodiment. Further, the composite image may be the pan-sharpened image in the first embodiment.

Specifically, the composite image generation unit 402 converts, for example, the second gradation obtained by converting the gradation (for example, luminance gradation) of the second image in accordance with the gradation of the first image. Information representing local features in the converted image may be extracted as the feature information. The feature information may be information having the same resolution as that of the second image, for example. Further, the composite image generation unit 402 may convert the resolution of the first image, for example. Then, the composite image generation unit 402 uses, for example, the feature information and the first gradation conversion information to adjust the gradation of the first image in which the resolution is converted. The composite image may be generated. The composite image generation unit 402 may correspond to, for example, the local feature extraction unit 105, the output luminance calculation unit 106, and the image synthesis unit 107 in the first embodiment, but is not limited thereto. .

The image processing apparatus 400 configured as described above generates, for example, a first tone conversion image by executing tone conversion processing such as tone mapping processing on the first image. In this case, since the first image may be a low-resolution multispectral image, the calculation cost required for the gradation conversion process is relatively low.

Further, the image processing apparatus 400 adjusts the first image by using the feature information relating to the gradation of the second image and the gradation of the first gradation conversion image, thereby adjusting the composite image. Generate. The second image may be a panchromatic image, for example.

The image processing device 400 generates a composite image having an appropriate gradation based on the gradation (for example, luminance gradation) of the first image (for example, multispectral image) or the second image (for example, panchromatic image). Is possible. This is because the image processing apparatus 400, for example, based on the feature information included in the high-resolution second image and the original first image before tone conversion such as tone mapping is performed. It is because it produces | generates. This is because the image processing apparatus 400 adjusts the first image based on the feature information and the gradation of the first gradation conversion image. Further, since the image processing apparatus 400 does not need to execute gradation conversion processing that directly targets an image with a high resolution, the calculation cost required for generating a composite image can be reduced. That is, the image processing apparatus 400 can efficiently generate a composite image.

As described above, the image processing apparatus 400 according to the present embodiment uses a plurality of images with different resolutions (for example, the first image and the second image) to generate a high-resolution image (for example, an appropriate gradation) (for example, The above composite image) can be generated efficiently.

<Modification of Second Embodiment>
A modification of the second embodiment will be described. In the following description, the same reference numerals are assigned to the same configurations as those in the second embodiment, and the detailed description is omitted.

The image processing apparatus 400 according to the present modification may further include a feature information generation unit (feature information generation means) 403 as illustrated in FIG. Note that the gradation adjustment image generation unit 401 and the composite image generation unit 402 may be the same as those in the second embodiment.

The feature information generation unit 403 extracts the gradations of the first and second images, and converts the gradation of the second image according to the gradation range of the first image, thereby converting the first image and the second image. 2 gradation conversion images are generated. The feature information generation unit 403 extracts the degree of local gradation change in the generated second gradation conversion image as the feature information. The feature information generation unit 403 may correspond to, for example, the local feature extraction unit 105 in the first embodiment. In this modification, the composite image generation unit 402 may generate the composite image using the feature information generated by the feature information generation unit 403.

The feature information generation unit 403 in the present modification appropriately converts the gradation of the second image in accordance with the gradation range of the first image, for example, by executing a luminance matching process or the like. 2 gradation conversion images can be generated. This is because the first image has not been subjected to gradation conversion processing such as tone mapping, and therefore, for example, the correlation between the gradation of the first image and the gradation of the second image This is because it is maintained. In addition, the feature information generation unit 403 in the present modification extracts the degree of local gradation change in the second gradation conversion image as the feature information.

The composite image generation unit 402 obtains feature information representing local features of the high-resolution second tone-converted image and the first tone-converted image whose tone is converted (compressed) by tone mapping or the like. By using this, it is possible to generate high-resolution luminance information (for example, the output luminance adjustment information). Then, the composite image generation unit 402 generates the composite image based on the high-resolution luminance information and the first image. As described above, the image processing apparatus 400 according to the present modification can generate a composite image having an appropriate gradation based on the gradation (for example, the luminance gradation) of the multispectral image and the panchromatic image. In addition, since the image processing apparatus 400 in the present modification has the same configuration as that of the second embodiment, the same effect as that of the image processing apparatus 400 in the second embodiment can be obtained.

<Configuration of hardware and software program (computer program)>
Hereinafter, a hardware configuration capable of realizing each of the above-described embodiments will be described.

In the following description, the image processing apparatuses (100, 400) described in the above embodiments are collectively referred to simply as “image processing apparatus”. Each component of the image processing apparatus is simply referred to as “component of the image processing apparatus”.

The image processing apparatus described in each of the above embodiments may be configured by one or a plurality of dedicated hardware devices. In that case, each component shown in each of the above figures (FIGS. 1, 4, and 5) uses hardware (an integrated circuit or a storage device on which processing logic is mounted) that is partially or fully integrated. May be realized.

When the image processing apparatus is realized by dedicated hardware, the components of the image processing apparatus include, for example, a circuit mechanism that can provide each function (for example, an integrated circuit such as a SoC (System on a Chip)). May be implemented. In this case, the data held by the components of the image processing apparatus is, for example, a RAM (Random Access Memory) area integrated as SoC, a flash memory area, or a storage device (such as a semiconductor storage device) connected to the SoC. May be stored. In this case, a well-known communication bus may be employed as a communication line that connects each component of the image processing apparatus. Further, the communication line connecting each component is not limited to bus connection, and each component may be connected by peer-to-peer. When the image processing apparatus is configured by a plurality of hardware devices, each hardware device may be communicably connected by any communication means (wired, wireless, or a combination thereof).

Further, the above-described image processing apparatus or its constituent elements may be configured by general-purpose hardware as illustrated in FIG. 6 and various software programs (computer programs) executed by the hardware. In this case, the image processing apparatus may be configured by an arbitrary number of general-purpose hardware devices and software programs. That is, an individual hardware device may be assigned to each component constituting the image processing apparatus, and a plurality of components may be realized using one hardware device.

The arithmetic device 601 in FIG. 6 is an arithmetic processing device such as a general-purpose CPU (Central Processing Unit) or a microprocessor. The arithmetic device 601 may read various software programs stored in a nonvolatile storage device 603, which will be described later, into the storage device 602, and execute processing according to the software programs. For example, the components of the image processing apparatus in each of the above embodiments may be realized as a software program executed by the arithmetic device 601.

The storage device 602 is a memory device such as a RAM that can be referred to from the arithmetic device 601, and stores software programs, various data, and the like. Note that the storage device 602 may be a volatile memory device.

The non-volatile storage device 603 is a non-volatile storage device such as a magnetic disk drive or a semiconductor storage device using flash memory. The nonvolatile storage device 603 can store various software programs, data, and the like.

The network interface 606 is an interface device connected to a communication network. For example, a wired and wireless LAN (Local Area Network) connection interface device, a SAN connection interface device, or the like may be employed.

The drive device 604 is, for example, a device that processes reading and writing of data with respect to a recording medium 605 described later.

The recording medium 605 is an arbitrary recording medium capable of recording data, such as an optical disk, a magneto-optical disk, and a semiconductor flash memory.

The input / output interface 607 is a device that controls input / output with an external device.

The image processing apparatus (or its constituent elements) according to the present invention described with the above-described embodiments as an example can realize the functions described in the above-described embodiments with respect to, for example, the hardware apparatus illustrated in FIG. It may be realized by supplying a software program. More specifically, for example, the present invention may be realized by causing the arithmetic device 601 to execute a software program supplied to such a device. In this case, an operating system running on the hardware device, database management software, network software, middleware such as a virtual environment platform, etc. may execute part of each process.

Furthermore, the software program may be recorded on the recording medium 605. In this case, the software program may be configured to be stored in the nonvolatile storage device 603 through the drive device 604 as appropriate at the shipping stage or the operation stage of the image processing apparatus.

In the above case, the method of supplying various software programs to the hardware is installed in the apparatus using an appropriate jig in the manufacturing stage before shipment or the maintenance stage after shipment. A method may be adopted. As a method for supplying various software programs, a general procedure may be adopted at present, such as a method of downloading from the outside via a communication line such as the Internet.

In such a case, the present invention can be understood to be constituted by a code constituting the software program or a computer-readable recording medium on which the code is recorded. In this case, the recording medium is not limited to a medium independent of the hardware device, but includes a recording medium in which a software program transmitted via a LAN or the Internet is downloaded and stored or temporarily stored.

Further, the above-described image processing apparatus may be configured by a virtual environment obtained by virtualizing the hardware device illustrated in FIG. 6 and various software programs (computer programs) executed in the virtual environment. . In this case, the components of the hardware device illustrated in FIG. 6 are provided as virtual devices in the virtual environment. In this case as well, the present invention can be realized with the same configuration as when the hardware device illustrated in FIG. 6 is configured as a physical device.

In the above, this invention was demonstrated as an example applied to exemplary embodiment mentioned above. However, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to such embodiments. In such a case, new embodiments to which such changes or improvements are added can also be included in the technical scope of the present invention. Furthermore, the embodiments described above, or embodiments obtained by combining the new embodiments with such changes or improvements can also be included in the technical scope of the present invention. This is clear from the matters described in the claims.
This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2015-171875 for which it applied on September 1, 2015, and takes in those the indications of all here.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Enlargement process part 102 Luminance extraction part 103 Tone mapping process part 104 Enlargement process part 105 Local feature extraction part 106 Output luminance calculation part 107 Image composition part 200 Input device 201 Multispectral image input part 202 Panchromatic image input Unit 400 image processing apparatus 401 gradation adjustment image generation unit 402 composite image generation unit 403 feature information generation unit

Claims

Gradation-adjusted image generating means for generating a first gradation-converted image by converting the gradation of the first image having one or more color information;
Feature information related to the gradation of the second image generated based on the first image and a second image having a higher resolution than the first image, and the first gradation-converted image By adjusting the first image using the gradation of the image, a composite image generation that generates a composite image having at least one color information and at least a higher resolution than the first image is performed. Means,
An image processing apparatus comprising:
The composite image generation means includes the feature information generated based on a second gradation conversion image obtained by converting the gradation of the second image according to the gradation range of the first image, The image processing apparatus according to claim 1, wherein the composite image is generated by adjusting the first image using the gradation of the first gradation-converted image.
The gradation adjusted image generation means compresses the gradation of the first image to a specific gradation range by performing tone mapping processing on the first image, and the first image whose gradation has been compressed is compressed. The image processing apparatus according to claim 2, wherein the first gradation conversion image is generated by converting an image into a resolution of the second image.
The synthesized image generating means converts the resolution of the first image, which has not been subjected to the tone mapping process, into the resolution of the second image;
By adjusting the gradation of the first image whose resolution has been converted using the feature information relating to the gradation of the second image and the gradation of the first gradation-converted image, The image processing apparatus according to claim 3, wherein the composite image is generated.
The second gradation-converted image is obtained by extracting the gradation of the first and second images and converting the gradation of the second image in accordance with the gradation range of the first image. And a feature information generating means for extracting the degree of local gradation change in the generated second gradation conversion image as the feature information,
The image processing apparatus according to claim 4, wherein the composite image generation unit generates the composite image using the feature information generated by the feature information generation unit.
The feature information generation unit, for each pixel included in the second gradation conversion image, a global feature amount calculated based on the gradation of a plurality of pixels including the pixel, and the gradation of the pixel The image processing apparatus according to claim 5, wherein the feature information that is a local feature amount related to the pixel is generated on the basis of.
The feature information generation unit calculates a global feature amount calculated by smoothing the second gradation conversion image for each pixel included in the second gradation conversion image, and a gradation of the pixel. The image processing apparatus according to claim 5, wherein the feature information, which is a local feature amount related to the pixel, is generated based on.
The feature information generation unit is configured to change a gradation of a pixel included in the second gradation-converted image, a global feature amount representing a global characteristic of luminance in the second gradation-converted image, and the second Are decomposed into local feature amounts representing local feature amounts of luminance in the gradation-converted image, and based on the luminance of the pixel and the global feature amount, the local feature amount is used as the feature information. The image processing apparatus according to claim 5, which calculates the image processing apparatus.
The composite image generation means is based on the feature information calculated for each pixel included in the second gradation conversion image and the gradation for each pixel included in the first gradation conversion image. The image processing according to any one of claims 5 to 8, wherein a gradation for each pixel included in an image obtained by converting the first image into the resolution of the second image is adjusted using the calculated value. apparatus.
The composite image generating means is a value obtained by multiplying the feature information calculated for each pixel included in the second gradation conversion image by a gradation for each pixel included in the first gradation conversion image. 9. The composite image is generated by replacing a gradation for each pixel included in an image obtained by converting the first image into the resolution of the second image using the first image. An image processing apparatus according to 1.
Generating a first gradation-converted image by converting the gradation of pixels included in the first image having one or more color information;
Feature information related to the gradation of the second image generated based on the first image and a second image having a higher resolution than the first image, and the first gradation-converted image And an image processing method for generating a composite image having at least one color information and having a resolution higher than that of the first image.
Processing for generating a first gradation-converted image by converting the gradation of pixels included in the first image having one or more color information;
Feature information related to the gradation of the second image generated based on the first image and a second image having a higher resolution than the first image, and the first gradation-converted image And the first image, and causing the computer to execute a process of generating a composite image that has one or more color information and has at least a higher resolution than the first image. A recording medium that stores a program.
An image processing apparatus according to any one of claims 1 to 12,
A multispectral image and a panchromatic image having a resolution higher than that of the multispectral image are accepted as inputs, and the multispectral image and the first image are provided to the image processing apparatus, and the panchromatic image is provided to the second spectral image. An input device provided as an image to the image processing device;
And an output device that outputs the composite image generated by the image processing device.