JP7134532B1 - Image synthesis display device, method, and program - Google Patents

Abstract

An object of the present invention is to display an image useful for assisting image interpretation without depending on subjectivity. A setting unit (14) sets a representative value of pixel values determined based on a histogram of pixel values of an input image as a reference value, and sets an upper limit value to a plurality of pixel values between the maximum pixel value and the reference value+1. By setting a plurality of lower limit values between the minimum pixel value and the pixel value of the reference value -1, a plurality of ranges from the upper limit value to the lower limit value are set, and the conversion unit 16 converts the pixels of the input image Transformed images converted so that the values are within the set range are generated for each of the plurality of ranges, and the display control unit 20 repeats each of the generated multiple transformed images in order of range size. control to display. [Selection drawing] Fig. 3

Description

The present invention relates to an image synthesis display device, an image synthesis display method, and an image synthesis display program.

Conventionally, we target various digital images such as satellite images, drone images, medical images, microscope images, experimental measurement images, astronomical images, etc. with high resolution and multi-wavelength, and various texture features such as texture, roughness, dirt, etc. Analyzing so-called image features such as cracks, edges, line structures, etc. has been done. At this time, the contrast of the image is adjusted while adjusting the upper and lower limits of the pixel values on the histogram of the pixel values (luminance values) of the pixels in the image.

JP-A-2002-366103

In adjusting the contrast of an image, it is necessary to repeat the upper and lower limits described above until the desired image feature can be emphasized. In addition, there is a problem that the difference in image quality of the image and the determination of the enhancement result of the image feature depend on the subjectivity of the engineer.

The present invention has been made in view of the above points, and aims to provide an image synthesizing display device, method, and program capable of displaying an image useful for assisting image interpretation without depending on subjectivity. aim.

In order to achieve the above object, an image synthesis display device according to the present invention uses a representative value of pixel values determined based on a histogram of pixel values of an input image as a reference value, and sets an upper limit value from the maximum value of the pixel values. By setting a plurality of pixel values between the pixel values one from the reference value to the maximum value and setting a plurality of lower limit values from the minimum pixel value to the pixel value from the reference value to the minimum value, a setting unit for setting a plurality of ranges from the upper limit value to the lower limit value; and a display control unit configured to repeatedly display each of the plurality of converted images generated by the conversion unit in order of the size of the range. .

According to the image synthesizing display device of the present invention, the representative value of the pixel values determined by the setting unit based on the histogram of the pixel values of the input image is used as the reference value, and the upper limit value is set from the maximum value of the pixel values to the reference value. By setting a plurality of pixel values between one maximum value and setting a plurality of lower limit values between the minimum pixel value and a pixel value one minimum value from the reference value, the range from the upper limit value to the lower limit value A plurality of ranges are set, the conversion unit generates a converted image in which the pixel values of the input image are converted so as to be within the range set by the setting unit for each of the plurality of ranges, and the display control unit converts Control is performed so that each of the plurality of transformed images generated by the unit is repeatedly displayed in order of range size. As a result, it is possible to display an image that is useful for assisting image interpretation without depending on subjectivity. The above function of the image synthesizing and displaying apparatus according to the present invention is a function of automating contrast stretching, and is capable of coping with various contrast stretching processes and has expandability.

Further, the setting unit sets a value obtained by adding a value obtained by multiplying the standard deviation of the pixel values of the input image by a predetermined value to the reference value as the upper limit value, and sets the value obtained by multiplying the standard deviation by the predetermined value as the reference value. A plurality of ranges may be set by setting a plurality of values as the predetermined value in the range whose lower limit is a value obtained by subtracting from the value. Thereby, it is possible to easily set a plurality of ranges from the maximum range to the extremely narrow minimum range.

Also, the representative value may be an average value, a median value, a mode value, a first quartile, or a third quartile of the pixel values of the input image. This makes it possible to set an appropriate range according to the input image.

Further, when the input image is a 3-band image, the conversion unit generates the converted image for each of the plurality of ranges for each band image, and the image synthesizing display device constructs the 3-band image. Each band image assigned to the R-plane, the G-plane, and the B-plane is assigned to the R-plane, the G-plane, and the B-plane in the combination that minimizes the correlated color temperature. a synthesizing unit for generating, for each of the plurality of ranges, a synthesized image obtained by synthesizing the transformed images corresponding to the band images; , may be controlled to be repeatedly displayed in order of the size of the range. As a result, it is possible to generate a composite image with good coloring and high visibility. The above function of the image synthesizing and displaying apparatus according to the present invention is a function of automatically assigning RGB to each plane by introducing the correlated color temperature as an index at the time of assignment, and has practicality, novelty, and expandability.

Further, the synthesizing unit sets the average value of the pixel values of the converted image corresponding to each of the band images to the value of the RGB color system, and converts the value of the RGB color system into the value of the XYZ color system. , chromaticity coordinates are calculated from the values of the XYZ color system, the chromaticity coordinates are converted into constants for calculating the correlated color temperature, and the correlated colors are calculated based on a calculation formula using the constants. Temperature may be calculated.

Further, when the input image is a three-band image, the conversion unit generates the conversion image for each of the plurality of ranges for each band image, and the image synthesizing device generates the conversion image for each of the plurality of ranges, Among the band images, the band image having the largest average pixel value of the converted image corresponding to the band image is assigned to the R plane, the band image having the smallest average pixel value is assigned to the G plane, and the remaining band images are assigned to the G plane. a synthesizing unit that allocates the band image to a B plane and generates a synthesized image obtained by synthesizing the transformed image corresponding to each of the band images for each of the plurality of ranges; may be controlled so that each of the plurality of synthesized images generated by is repeatedly displayed in order of the size of the range. As a result, it is possible to generate a composite image with good coloring and high visibility. The above function of the image synthesizing display device according to the present invention is a function of automatic allocation to each plane of RGB, which introduces the average value of the pixel values of the converted image as an index at the time of allocation. have sex.

Further, when the input image is a multiband image having three or more bands, the image synthesizing and displaying apparatus performs principal component analysis on the multiband image, and performs a first principal component image, a second The apparatus may include an analysis unit that generates an image composed of a principal component image and a third principal component image as the 3-band image. This makes it possible to reduce information while retaining spectral information over substantially the entire multiband image. Therefore, various images can be applied as input images, and it has flexibility and expandability.

Further, the display control unit controls the image data written in the frame memory to display the input image on a display screen including a display area for displaying the input image and a display area for displaying the converted image. The generated converted image may be displayed. Further, the display control unit may display the converted image in real time while capturing an image displayed by an image display system that is already in operation as the input image. Thereby, the converted image or the synthesized image can be displayed in real time with respect to the display of the input image. In other words, it is a device that can operate in combination with an existing system that generates a composite video that emphasizes image features in real time while capturing still images or moving images displayed by an existing system, and is an image that can realize support for idea creation. It can provide a composite display device, and has practicality and expansibility.

Further, the display control unit controls a display area for displaying the input image, a display area for displaying the synthesized image, and a display area for displaying the R plane, the G plane, and the B plane when generating the synthesized image. A display screen including a selection area for selecting whether the allocation of the converted images of is determined by the synthesizing unit or is manually specified. Accordingly, it is possible to selectively display an automatically generated composite image and a composite image generated based on manual settings.

Further, in the image synthesis display method according to the present invention, the representative value of the pixel values determined by the setting unit based on the histogram of the pixel values of the input image is used as the reference value, and the upper limit value is set from the maximum value of the pixel values to the reference value. By setting a plurality of pixel values between the minimum value of pixel values and a pixel value that is one pixel value less than the reference value, the upper limit A plurality of ranges from a value to the lower limit value are set, and a conversion unit converts a converted image so that pixel values of the input image are within the range set by the setting unit, and converts the converted image into each of the plurality of ranges. is generated, and the display control unit controls to repeatedly display each of the plurality of converted images generated by the conversion unit in order of the size of the range.

Further, in the image composition display program according to the present invention, the computer uses a representative value of pixel values determined based on a histogram of pixel values of an input image as a reference value, and sets an upper limit value from the maximum value of the pixel values to the reference value. By setting a plurality of pixel values between the pixel values one above the maximum value and setting a plurality of lower limit values between the minimum pixel value and the pixel value one pixel value below the reference value, the upper limit value to the lower limit value, and generating a converted image in which pixel values of the input image are converted so as to be within the range set by the setting unit for each of the plurality of ranges. A program for functioning as a conversion unit and a display control unit that controls to repeatedly display each of the plurality of converted images generated by the conversion unit in order of the size of the range.

INDUSTRIAL APPLICABILITY According to the image synthesizing display device, method, and program according to the present invention, it is possible to display an image that is useful for supporting image interpretation without depending on subjectivity.

It is a block diagram showing a schematic configuration of the present embodiment. 2 is a block diagram showing the hardware configuration of an image synthesizing display device; FIG. 2 is a block diagram showing a functional configuration of an image synthesizing display device; FIG. FIG. 4 is a diagram showing an example of a multiband image and first to third principal component images; FIG. 10 is a diagram for explaining the generation of a stretch image; FIG. FIG. 10 is a diagram showing an example of a stretched image for each stretch range and a stretched image for each principal component image; It is a figure for demonstrating visibility based on a chromaticity diagram. It is a figure for demonstrating relative luminous efficiency. FIG. 10 is a diagram showing an example of a color composite image for each stretch range; It is a figure which shows an example of a display screen. 9 is a flowchart showing an example of image synthesis display processing; FIG. 10 is a diagram showing an example of a color composite image when a hyperspectral image is applied as an input image; FIG. 13 is a diagram showing an example of a color composite image and a correlated color temperature for all cases of combinations of assignments of RGB to each plane of the first to third principal component images in the example of FIG. 12; FIG. 10 is a diagram showing an example of a color composite image when applying a three-band RGB high-resolution image as an input image (with application of principal component analysis); FIG. 15 is a diagram showing an example of a color composite image and correlated color temperatures for all cases of combinations of assignments of RGB to each plane of the first to third principal component images in the example of FIG. 14; FIG. 10 is a diagram showing an example of a color composite image when a three-band RGB high-resolution image is applied as an input image (no application of principal component analysis); FIG. 10 is a diagram showing a comparison of color composite images obtained by applying and not applying principal component analysis to an RGB 3-band high-resolution image that is an input image; FIG. 10 is a diagram showing an example of a color composite image when applying an RGB 3-band image captured by a smartphone as an input image (with application of principal component analysis); FIG. 10 is a diagram showing an example of a color composite image when applying an RGB 3-band image captured by a smartphone as an input image (without application of principal component analysis); FIG. 10 is a diagram showing a comparison of color composite images obtained by applying and not applying principal component analysis to an RGB 3-band image captured by a smartphone, which is an input image; FIG. 10 is a diagram showing another example of a color composite image when applying an RGB 3-band image captured by a smartphone as an input image (with application of principal component analysis);

An example of an embodiment of the present invention will be described below with reference to the drawings.

As shown in FIG. 1, in the image synthesizing and displaying apparatus 10 according to the present embodiment, an image captured by a camera 22 and an image stored in an image database (DB) stored in an external device 24 can be transmitted via a network. entered through This image is hereinafter referred to as an input image. The input image is a monochrome image, a color image composed of three bands of R, G, and B, an ultra-high resolution image, a hyperspectral image, a microwave image radar image (polarization image), an ultraviolet image, a near-infrared image, Various images such as medical images can be applied. In this embodiment, a case where the input image is a multiband image with three or more bands will be described.

Next, the hardware configuration of the image synthesizing display device 10 will be described.

As shown in FIG. 2, the image synthesis display device 10 includes a CPU (Central Processing Unit) 32, a memory 34, a storage device 36, an input device 38, an output device 40, a recording medium reading device 42, and a communication I/F (Interface ) 44. Each component is communicatively connected to each other via a bus.

The storage device 36 stores an image composition display program for executing image composition display processing. The CPU 32 is a central processing unit that executes various programs and controls each configuration. That is, the CPU 32 reads a program from the storage device 36 and executes the program using the memory 34 as a work area. The CPU 32 performs control of each configuration and various arithmetic processing according to programs stored in the storage device 36 .

The memory 34 is composed of a RAM (Random Access Memory) and temporarily stores programs and data as a work area. The storage device 36 is composed of a ROM (Read Only Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and stores various programs including an operating system and various data.

The input device 38 is, for example, a device for performing various inputs such as a keyboard and a mouse. The output device 40 is, for example, a device for outputting various information, such as a display and a printer. A touch panel display may be used as the output device 40 to function as the input device 38 .

The recording medium reading device 42 reads data stored in various recording media such as CD-ROMs (Compact Disc Read Only Memory), Blu-ray discs, and USB (Universal Serial Bus) memories, and writes data to the recording media. I do. The communication I/F 44 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark), for example.

Next, the functional configuration of the image synthesis display device 10 according to this embodiment will be described.

As shown in FIG. 3, the image synthesis display device 10 includes an analysis unit 12, a setting unit 14, a conversion unit 16, a synthesis unit 18, and a display control unit 20 as functional configurations. Each functional configuration is realized by the CPU 32 reading out an image composition display program stored in the storage device 36, developing it in the memory 34, and executing it.

The analysis unit 12 performs principal component analysis on the multiband image, which is an input image, and generates a first principal component image, a second principal component image, and a third principal component image having the top three cumulative contribution rates. That is, the analysis unit 12 is a functional unit that orthogonalizes and contracts each band image for a multiband image having a high correlation between adjacent band images.

For example, as shown in the upper diagram of FIG. 4, the input image is a multiband image composed of band image 0 to band image p (p=24 in the example of FIG. 4), which is an image of a cracked concrete surface. Suppose there is The analysis unit 12 sets an explanatory variable represented by a vector having each pixel value of each band image as an element x _i (i=0, 1, . . . , p), and sets the explanatory variable x _i is calculated as the _first principal component z1. z ₁ is expressed as z ₁ =a ₁₀ x ₀ +a ₁₁ x ₁ + . . . +a _1p x _p using coefficient a _1i . In addition, the analysis unit 12 calculates, among the axes perpendicular to the axis corresponding to the _first principal component z1, the axis on which the explanatory variable _xi has the largest variance as the _second principal component z2. _{z 2} is expressed as _{z 2} =a _{20 x 0} +a ₂₁ x ₁ + . The analysis unit 12 repeats the calculation of the principal component in the same way, and uses the _coefficient a _mi to calculate the _{m - th principal component z m} Calculate The analysis unit 12 selects the principal components having the top three cumulative contribution ratios among the first principal component z ₁ , the second principal component z ₂ , . . . , and the _m -th principal component zm. The analysis unit 12 selects the coefficients a _{( 1 )0} , a ₍₁₎₁ , . is used to generate the first principal component image as shown in the lower diagram of FIG. Similarly, the analysis unit 12 generates a second principal component image based on the selected principal component z ₍₂₎ with the second highest cumulative contribution rate, and generates a second principal component image based on the selected principal component z ₍₃₎ with the third highest cumulative contribution rate. to generate a third principal component image.

In the example of FIG. 4, the three selected principal components have a cumulative contribution rate of 73.9%. I understand. Also, the first principal component image, the second principal component image, and the third principal component image have differences in the average value and the standard deviation (σ) of the pixel values, respectively. Further, an image representing the entire concrete surface is generated as the first principal component image, an image in which the influence of the halogen lamp at the time of photographing is emphasized is generated as the second principal component image, and an image in which the influence of cracks and roughness is emphasized. is generated as the third principal component image.

The setting unit 14 sets the representative value of the pixel values determined based on the histogram of the pixel values of the principal component images generated by the analysis unit 12 as the reference value, and sets the upper limit value to the pixel value +1 from the maximum pixel value. A plurality of values are set, and a plurality of lower limit values are set between the minimum pixel value and the pixel value of the reference value -1. Thereby, the setting unit 14 sets a plurality of ranges from the upper limit value to the lower limit value as stretch ranges for adjusting the contrast of pixel values. The representative value may be the average value, the median value, the mode value, the first quartile, the third quartile, etc. of the pixel values of each principal component image.

More specifically, for example, the setting unit 14 sets the representative value as the average value, sets the value obtained by multiplying the standard deviation of the pixel values of each principal component image by a predetermined value, and adds the value to the average value as the upper limit value, and sets the standard deviation as the upper limit value. A plurality of stretch ranges are set by setting a plurality of predetermined values in a stretch range whose lower limit is a value obtained by subtracting a value obtained by multiplying a predetermined value by the average value. That is, the setting unit 14 sets the upper limit using the following formula (1) and sets the lower limit using the following formula (2).
Upper limit value = average pixel value of principal component image + nσ (1)
Lower limit = average pixel value of principal component image - nσ (2)
where σ is the standard deviation. Also, n is a predetermined value, and may be set to n=0.25, 0.5, 0.75, 1, 1.25, . . .

Also, the setting unit 14 sets the maximum and minimum values of the upper limit and the lower limit as shown below.
Maximum upper limit = 255 Minimum upper limit = Average + 1
Minimum lower limit value = 0 Maximum lower limit value = Average - 1
In this case, as the stretch range to be set, 0 (lower limit) to 255 (upper limit) is set as the widest stretch range, and the average value -1 (lower limit) to average value + 1 (upper limit) is the narrowest stretch Set as a range.

The transformation unit 16 generates a stretched image by transforming the pixel values of the principal component image so as to become the stretched range set by the setting unit 14 for each of the plurality of stretched ranges. Specifically, as shown in FIG. 5, the conversion unit 16 sets the pixel value corresponding to the set lower limit value to 0, the pixel value corresponding to the upper limit value to 255, and sets the pixel value to 0 in the principal component image before conversion. A stretched image is generated by linearly transforming pixel values between .about.255. As shown in the upper diagram of FIG. 6, the conversion unit 16 generates a plurality of stretched images corresponding to each set stretch range for each principal component image. In addition to linear conversion, conversion using a higher-order function, logarithmic function, sine or cosine function may be applied to the conversion of pixel values.

The synthesizing unit 18 assigns the three principal component images to the R plane, the G plane, and the B plane with the combination that minimizes the correlated color temperature for each combination when the three principal component images are assigned to the R plane, the G plane, and the B plane, respectively. A color synthesized image obtained by synthesizing the stretched images corresponding to the extracted principal component images is generated for each of the plurality of stretching ranges. Specifically, the synthesizing unit 18 sets the average value of the pixel values of the stretched image to values in the RGB color system, converts the values in the RGB color system to values in the XYZ color system, and converts the values in the XYZ color system. Calculate the chromaticity coordinates from the value of Then, the synthesizing unit 18 converts the chromaticity coordinates into constants for calculating the correlated color temperature, and calculates the correlated color temperature based on a calculation formula using the constants.

More specifically, the synthesizing unit 18 converts the RGB color system into the XYZ color system according to the following formulas (3) to (5). Note that the values of R, G, and B use the average pixel values of the stretched images for a plurality of stretching ranges corresponding to the principal component images assigned to the R plane, G plane, and B plane, respectively.
X=2.7689×R+1.7515×G+1.1301×B (3)
Y=R+4.5907×G+0.0601×B (4)
Z=0.0565×G+5.5943×B (5)

Then, the synthesizing unit 18 obtains chromaticity coordinates x and y using the following formulas (6) and (7), and converts the chromaticity coordinates x and y into a constant n using the following formula (8).
x=X/(X+Y+Z) (6)
y=Y/(X+Y+Z) (7)
n=((x-0.332))/((y-0.1858)) (8)

Further, the synthesizing unit 18 substitutes the constant n into the following equation (9) to calculate the correlated color temperature T _corr [K].
T _corr =−437×n ³ +3601×n ²
-6861 x n + 5514.31 (9)

Here, there are 6 cases (= ₃ P ₃ ) of combinations when the stretched images corresponding to the first to third principal component images are assigned to each of the RGB planes. Therefore, the synthesizer 18 calculates the correlated color temperature T _corr [K] for each of these six cases. Then, the synthesizing unit 18 allocates the three principal component images to each of the RGB planes in the case where the correlated color temperature T _corr is the minimum. Then, the synthesizing unit 18 synthesizes a stretched image corresponding to each principal component image for each stretch range. As a result, a color composite image is generated for each stretch range.

Here, the reason why the combinations that minimize the correlated color temperature are assigned to the RGB planes of the first to third principal component images will be described. In general, when generating a color composite image, a stretch range in which image features are emphasized and allocation to RGB planes are determined by trial while performing contrast stretching and observing the RGB color development state. . In other words, the condition is that the visibility of the colors on the color composite image is improved, which is related to the hue. Regarding this point, the following can be read from the chromaticity diagram shown in FIG.
(1) Six colors of green, yellowish green, yellow, orange, vermilion, and red can be visually recognized from the top, from green to red.
(2) From green to blue, only about four colors of green, pale blue, blue, and purple can be visually recognized from the bottom.

That is, by assigning the principal component image to each of the RGB planes so that the value of the correlated color temperature T _corr is small and the color is reddish, the number of colors of the image features on the color composite image increases, and the image Features can be emphasized, and visibility is also improved. The grounds for this can be explained from the viewpoint of relative luminous efficiency. Specifically, in the case of visible light, the color differs depending on the wavelength, and the brightness perceived by the eyes differs depending on the wavelength even for electromagnetic waves (light) with the same energy. For example, we feel yellow and green light bright, and red and blue light dark. Humans feel the brightest is yellow-green electromagnetic waves (light) with a wavelength of "near 555 nm". As shown in FIG. 8, the brightness corresponding to a wavelength of 555 nm is defined as "1 (maximum sensitivity)", and the ratio (relative value) of sensitivity for each wavelength to the maximum sensitivity is defined as "standard relative luminosity (photopic vision)". To tell. Also, in dark places, the maximum sensitivity shifts to blue-green (scotopic vision). As can be seen from FIG. 8, assigning the main component image to each of the RGB planes so as to develop colors in the “green-yellow-yellow-red” system increases the relative luminosity.

Also, in the example of FIG. 6, in the allocation with the minimum correlated color temperature value, the first principal component image was allocated to the R plane, and the third principal component image was allocated to the G plane. Further, among the three principal component images, the average value (136) of the stretched image corresponding to the first principal component image is the largest, and the average value (126) of the stretched image corresponding to the third principal component image is the smallest. there were. On the color wheel, reds and greens are in a complementary color positional relationship, and are a combination that can enhance contrast. That is, the fact that the principal component images showing the maximum and minimum average values of the stretched images corresponding to the principal component images are assigned to the R plane and the G plane respectively means that the assignment of RGB to each plane by the correlated color temperature is confirms that it is appropriate.

Therefore, the synthesizing unit 18 allocates to the R plane the principal component image having the largest average pixel value of the plurality of stretched images corresponding to the principal component image among the three principal component images, and the average pixel value is A color composite image may be generated by assigning the smallest principal component image to the G plane and assigning the remaining principal component images to the B plane and synthesizing them for each of the plurality of stretching ranges.

In the example of FIG. 6, the first principal component image representing "information on the entire surface" is assigned to red (R plane), and the third principal component image representing "roughness information" is assigned to green (G plane) for synthesis. By doing so, the roughness is emphasized to 5 to 6 colors due to the relationship between the number of colors and the complementary colors of the intermediate colors, and the saturation of each other increases, making each other's colors stand out more vividly, creating a color composite image with clear contrast. is generated.

The display control unit 20 controls so that each of the plurality of stretched images generated by the conversion unit 16 is repeatedly displayed on the display device in the order of the size of the stretching range. For example, the display control unit 20 displays a plurality of stretched images corresponding to one of the principal component images as shown in the upper diagram of FIG. 6 while sequentially switching them at predetermined time intervals. As a result, a plurality of stretched images are reproduced as moving images (hereinafter referred to as "stretched moving images").

Further, the display control unit 20 controls to repeatedly display each of the plurality of color composite images generated by the composite unit 18 on the display device in order of the size of the stretch range. For example, the display control unit 20 displays a plurality of color composite images generated for each stretch range, as shown in FIG. 9, while sequentially switching them at predetermined time intervals. As a result, a plurality of color composite images are reproduced as a moving image (hereinafter referred to as "color composite moving image"). In FIG. 9, the color composite image is represented in grayscale, but in the color composite image, the stretched images corresponding to the first to third principal component images are actually assigned to the respective RGB planes. This is a composite color image. The same applies to each of the following figures.

In addition, the display control unit 20 can accept various settings from the user, such as whether or not to apply principal component analysis, the stretch range, and allocation to the RGB planes. , an image based on the received settings may be displayed instead of the image generated in . In order to realize this, the display control section 20 may display a display screen 50 as shown in FIG. 10, for example.

In the example of FIG. 10 , the display screen 50 includes an input image display area 52 and a video playback area 54 . The display control unit 20 displays the input image based on the image data written in the frame memory for the input image display area 52 in the input image display area 52 . The display control unit 20 also reproduces, in the moving image reproduction area 54, a stretched moving image or a color composite moving image generated from the image data of the input image written in the frame memory. As a result, even when an input image is enlarged or reduced, a corresponding moving image can be displayed in real time. In addition, the display control unit 20 captures in real time an input image to be displayed in the input image display area 52 from the frame memory of the image display system that is already in operation, and displays a moving image generated from the input image in the moving image reproduction area 54 in real time. to play. As a result, it is possible to apply this to various input images and display moving images for interpreting image features in real time without relying on display software.

Input images include various inspection and measurement videos by drone, ultra-high resolution images (e.g., 0.08mm/pixel), ultraviolet, visible, near-infrared, infrared, microwave, and other wavelength bands. images, drone videos, Google Earth images, street view videos, mobile mapping videos, hyperspectral cube videos, microscope images, meteorological images (typhoons, cloud images), astronomical images, archaeological field images, various satellite images, CT images , MRI images, ultrasound images, 3D moving images, images captured by smartphones, and the like can be applied. When a moving image is input, the moving image is paused at a scene whose image feature is desired to be interpreted, and the frame image corresponding to the frame at the time of the pause is used as the input image.

The display screen 50 also includes a check box 56 for selecting whether to play the stretched video or the color composite video in the video playback area 54 . When the stretched moving image is selected in the check box 56 , the display control unit 20 reproduces the stretched moving image in the moving image reproduction area 54 . On the other hand, when the color composite moving image is selected in the check box 56 , the display control unit 20 reproduces the color composite moving image in the moving image reproduction area 54 . Note that if the input image is a single-band image, the check box 56 may be configured so that the color composite moving image cannot be selected.

Display screen 50 also includes a check box 58 for selecting whether to apply principal component analysis to the input image. When the input image is an RGB 3-band image, each band image can be assigned to each RGB plane without performing principal component analysis when generating a color composite image. In the example of FIG. 10, in the check box 58, "RBG 3 bands" represents a case where the original RGB are assigned to each plane without applying the principal component analysis. The display control unit 20 notifies the analysis unit 12 of the selection result of the check box 58 . As a result, when applying principal component analysis is selected in the check box 58, the analysis unit 12 executes the principal component analysis of the input image. Note that if the input image is a multiband image with four or more bands, the check box 58 may be made selectable only when principal component analysis is applied. Also, if the input image is a single-band image, the check box 58 may be hidden.

The display screen 50 also includes a reproduction speed selection area 60 for selecting the reproduction speed of the moving image reproduced in the moving image reproduction area 54 . The display control unit 20 reproduces the moving image at the reproduction speed selected in the reproduction speed selection area 60 in the moving image reproduction area 54 . The display screen 50 also includes a range selection area 62 . Range selection area 62 includes a pause button and a selection box for selecting a stretch range. When the pause button is selected, the display control unit 20 pauses the moving image being reproduced in the moving image reproduction area 54, and displays the stretched image or color composite image corresponding to the stretch range selected in the selection box as the moving image. Displayed in the playback area 54 . The selection box can switch the stretching range sequentially in order of range width. As a result, the stretched moving image or the color composited image can be displayed in a frame-by-frame manner, and the stretched image or the color composited image corresponding to the stretched range for which the image feature is desired to be confirmed can be easily specified.

The display screen 50 also includes a setting area 64 for setting the assignment of each band image or principal component image to the RGB planes. The setting area 64 includes a check box for selecting whether to apply allocation by the synthesizing unit 18 (“recommended allocation” in FIG. 10) or to manually allocate. The setting area 64 also includes a designation box for designating which principal component image or band image is to be allocated to each of the RGB planes in the case of manual allocation. The designation box is composed of, for example, a pull-down menu or the like, and when principal component analysis is performed on the input image, the first to third principal component images are displayed as options, and principal component analysis is not performed. In this case, RGB band images are displayed as options. The display control unit 20 notifies the synthesizing unit 18 of the information set in the setting area 64 . Accordingly, when the recommended allocation is set, the composition unit 18 performs allocation based on the above-described correlated color temperature to generate a color composite image. On the other hand, when the allocation is manually set, the synthesizing unit 18 performs allocation according to the setting to generate a color composite image.

Display screen 50 also includes an execute button 66 and a save button 68 . When the execution button 66 is selected, the display control unit 20 reproduces the moving image in the moving image reproduction area 54 according to the settings selected for each area. When the save button 68 is selected, the display control unit 20 saves the moving image being reproduced in a predetermined storage area. In this case, the display control unit 20 may save the moving image reproduced at the set reproduction speed, or may save a plurality of stretched images or a plurality of color composite images constituting the moving image. In addition, the display control unit 20 may store information set in each area together with the moving image. Furthermore, when the save button 68 is selected while the moving image is paused, the display control unit 20 causes the display control unit 20 to save a stretched image or a color composite image, which is a still image corresponding to the paused frame. can be

Next, the operation of the image synthesizing display device 10 according to this embodiment will be described.

FIG. 11 is a flowchart showing the flow of image composition display processing executed by the CPU 32 of the image composition display device 10. As shown in FIG. The CPU 32 reads out the image composition display program from the storage device 36, develops it in the memory 34, and executes it, whereby the CPU 32 functions as each functional configuration of the image composition display device 10, and the image composition display processing shown in FIG. 11 is executed. be done. Here, a case where a multiband image of three or more bands is used as an input image will be described as an example.

First, in step S10, the analysis unit 12 acquires a multiband input image. Next, in step S12, the analysis unit 12 performs principal component analysis using each band image of the input image as an explanatory variable, selects the principal components with the highest cumulative contribution rates, and selects the coefficients of the selected principal components and The pixel values of each band image are used to generate first to third principal component images.

Next, in step S14, the setting unit 14 sets, for example, the maximum upper limit value = 255, the minimum upper limit value = the average pixel value of the main component image + 1, the minimum lower limit value = 0, the lower limit value A plurality of stretch ranges are set, which range from the lower limit value to the upper limit value where maximum value=average value−1. Then, the converting unit 16 converts the pixel values of the principal component image so as to correspond to the set stretch range to generate a stretched image for each of the plurality of stretch ranges.

Next, in step S16, the synthesizing unit 18 calculates the average value of the pixel values of the plurality of stretched images corresponding to each principal component image, and further calculates the average value for each principal component image. Then, the synthesizing unit 18 calculates the correlated color temperature for each combination when the three principal component images are assigned to the R plane, the G plane, and the B plane, respectively, using the calculated average value of the pixel values of the stretched image. calculate. Then, the synthesizing unit 18 synthesizes the stretched images corresponding to the principal component images assigned to the R plane, the G plane, and the B plane in a combination that minimizes the correlated color temperature, and creates a color synthesized image in a plurality of stretch ranges. Generate for each.

Next, in step S18, the display control unit 20 controls the display device to repeatedly display each of the plurality of generated color composite images in the order of the size of the stretch range while switching at predetermined time intervals. By doing so, a color composite moving image is reproduced. Then, the image synthesis display processing ends.

As described above, the image synthesis display apparatus according to the present embodiment uses the representative value of pixel values determined based on the histogram of the pixel values of the input image as the reference value, and the upper limit value from the maximum pixel value to the reference value. A plurality of stretch ranges from the upper limit value to the lower limit value are set by setting a plurality of values between +1 and setting a plurality of lower limit values between the minimum pixel value and the reference value −1. Further, the image synthesizing and displaying device generates a stretched image in which the pixel values of the input image are converted so as to be within the set stretched range for each of the plurality of stretched ranges. Further, the image synthesizing display device controls to repeatedly display each of the plurality of generated stretched images in the order of stretch range width. As a result, it is possible to display an image that is useful for assisting image interpretation without depending on subjectivity. In addition, since a stretched image with an extremely narrow stretch range is also generated as described above, it is possible to confirm the enhanced image features.

Further, when the input image is a 3-band image, the image synthesis display apparatus according to the present embodiment generates a stretch image corresponding to a plurality of stretch ranges for each band image, and then divides each band image into each RGB plane. A color synthesized image that is assigned and synthesized is generated for each stretch range. For this assignment, the image synthesis display device assigns so that the correlated color temperature is minimized. As a result, image features can be emphasized, and a highly visible color composite image can be generated. Further, the image enhancement result of the stretched image with the extremely narrow stretch range as described above is also reflected in the color composite image. In addition, since manual contrast stretching processing is completely unnecessary, the efficiency of color composite image generation is greatly improved. In addition, by generating a moving image of multiple stretched images and color composite images as described above, it is possible to dynamically visualize changes in image quality in the process of contrast stretching and the entire process of image feature enhancement, resulting in a high reading support effect. can get.

As sensors are developed with the aim of improving spatial resolution and wavelength resolution (multi-band), the use of hyperspectral data is also attracting attention in recent years. Since hyperspectral data has a high correlation between adjacent band images, it is common to generate a color composite image using observed images corresponding to normal R, G, and B bands. However, in this process, spectral information possessed by hyperspectral data is not utilized. Although it is possible to select arbitrary 3-band hyperspectral data and perform color synthesis, the hue and saturation on the color synthesized image differ due to differences in the allocation of RGB to each plane, resulting in image feature enhancement. In some cases, no effect is obtained. When the input image is a multi-band image with three or more bands, the image synthesis display device according to the present embodiment performs principal component analysis to determine the first to third principal components based on the top three cumulative contribution ratio principal components. Generate a component image and treat it as the 3-band image above. As a result, the above problem can be solved, and a color composite moving image can be generated while maintaining spectral information covering almost the entire hyperspectral data.

Also, the image synthesis display device according to the present embodiment can apply all kinds of images as input images. As a result, the image synthesizing display device can be incorporated into various existing devices such as sensors, cameras, and image processing and analysis systems. Image processing function.

In the above-described embodiment, a multiband image obtained by photographing a concrete surface including cracks is mainly used as an input image. It is also applicable to analysis of CT images and the like. Examples of color composite images generated for various input images are shown below.

FIG. 12 is an example of a color composite image generated in this embodiment when a hyperspectral image of a concrete surface including cracks is applied as an input image. Although the image on the right side of FIG. 12 is represented in grayscale, in reality, the rough surface is expressed in red, and the cracks are emphasized in green so as to cover the rough surface, resulting in a color composite image. , it is possible to confirm a high interpretation support effect for the image features. In addition, by reproducing color composite images for a plurality of stretch ranges as moving images, it is possible to dynamically visually recognize changes in image quality and the entire process of image feature enhancement during the contrast stretching process.

FIG. 13 shows an example of color composite images and correlated color temperatures for all cases of combinations of assignments of RGB to each plane of the first to third principal component images in the example of FIG. In this example, the average of the pixel values of the stretched image corresponding to each principal component image is 136 (large) for the first principal component image, 127 (medium) for the second principal component image, The third principal component image was 126 (small). As shown in FIG. 13, the correlated color temperature is the minimum when the first principal component image with the largest average pixel value is assigned to the R plane and the third principal component image with the smallest average pixel value is assigned to the G plane. It can be confirmed that the allocation with good coloring and high visibility can be automatically performed.

FIG. 14 is a color composite image generated in this embodiment when applying a high-resolution RGB 3-band image of a concrete surface including cracks as an input image and applying principal component analysis to the input image. An example. Similar to the example of FIG. 12, the rough surface is expressed in red, and the cracks are emphasized in green or blue to cover the rough surface, resulting in a color composite image. You can check the support effect. Also, FIG. 15 shows an example of a color composite image and correlated color temperature for all cases of combinations of assignment of RGB to each plane of the first to third principal component images in the example of FIG. In the example of FIG. 15, the maximum average value of pixel values is the first principal component, the middle value is the second principal component, and the minimum value is the third principal component. "1" means that there is no significant difference. Therefore, the case in which the first principal component image with the largest average pixel value is assigned to the R plane and the third principal component image with the smallest average pixel value is assigned to the G plane are different from the case in which the correlated color temperature is the smallest. However, in the former case as well, the correlated color temperature is the second lowest after the case where the correlated color temperature is the lowest, and similar to the example of FIG. I can confirm that it can be done automatically.

FIG. 16 is a color composite image generated in this embodiment when a high-resolution RGB 3-band image of a concrete surface including cracks is applied as an input image, and principal component analysis is not applied to the input image. An example. In this example, a color composite image is generated by assigning the R-band image (red image) of the input image to the R-plane, the G-band image (green image) to the G-plane, and the B-band image (blue image) to the B-plane. ing. Also, FIG. 17 shows a comparison between a color composite image to which principal component analysis is applied and a color composite image to which principal component analysis is not applied for this input image. All color composite images are images in which the image features are emphasized, but in particular, when principal component analysis is applied, classification is performed according to the image features simply by performing color synthesis processing. A color composite image in which the image features are emphasized is obtained.

FIG. 18 is an example of a color composite image generated in this embodiment when applying an RGB 3-band image of the cheek skin surface photographed with a smartphone as an input image and applying principal component analysis to the input image. be. Although the right figure in FIG. 18 is represented in grayscale, it is actually a color composite image in which the redness and dullness of the skin are emphasized in green to yellow. be able to. Also, FIG. 19 is an example of a color composite image generated when principal component analysis is not applied to a similar input image. In this example, a color composite image is generated by assigning the R-band image (red image) of the input image to the R-plane, the G-band image (green image) to the G-plane, and the B-band image (blue image) to the B-plane. ing. Also, FIG. 20 shows a comparison between a color composite image to which principal component analysis is applied and a color composite image to which principal component analysis is not applied for this input image. All color composite images are images in which the image features are emphasized, but in particular, when principal component analysis is applied, classification is performed according to the image features simply by performing color synthesis processing. A color composite image in which the image features are emphasized is obtained. In this way, the technology disclosed can be applied to an input image captured by a smartphone, so skin diagnosis can be performed using a simple system, for example, in beauty and medical fields.

FIG. 21 is an example of a color composite image generated in this embodiment when applying an RGB 3-band image of a palm photographed with a smartphone as an input image and applying principal component analysis to the input image. Although the image on the right side of FIG. 21 is represented in gray scale, it is actually a color composite image because the color of the palm and the palm print are emphasized, and it is possible to confirm a high interpretation support effect for the image features. . As a result, processing such as palmprint authentication can be realized with a simple system.

In the above description, the case where the input image is a 3-band image or a multi-band image has been described, but the technology disclosed herein can also apply a monochrome image (single-band image) such as an X-ray photograph as the input image. . In this case, by reproducing the stretched moving image, it is possible to dynamically visually recognize the change in image quality and the entire process of image feature enhancement during the contrast stretching process without performing manual contrast stretching.

Moreover, various processors other than the CPU may execute the image synthesis display processing executed by the CPU reading the software (program) in the above embodiment. In this case, the processor is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit) for executing specific processing. A dedicated electric circuit or the like, which is a processor having a specially designed circuit configuration, is exemplified. In addition, the image synthesis display processing may be executed by one of these various processors, or a combination of two or more processors of the same or different type (for example, multiple FPGAs, and a CPU and FPGA combination). combination, etc.). More specifically, the hardware structure of these various processors is an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in the above-described embodiment, the image composition display program is pre-stored (installed) in the storage device, but the present invention is not limited to this. The program may be provided in a form recorded on a recording medium such as a CD-ROM, a DVD-ROM (Digital Versatile Disc Read Only Memory), and a USB memory. Also, the program may be downloaded from an external device via a network.

10 image synthesis display device 12 analysis unit 14 setting unit 16 conversion unit 18 synthesis unit 20 display control unit 22 camera 24 external device 32 CPU
34 memory 36 storage device 38 input device 40 output device 42 recording medium reading device 50 display screen 52 input image display area 54 video playback area 56 check box 58 check box 60 playback speed selection area 62 range selection area 64 setting area 66 execution button 68 save button

Claims (12)

A representative value of pixel values determined based on a histogram of pixel values of an input image is set as a reference value, and a plurality of upper limit values are set between the maximum pixel value and a pixel value one pixel value above the reference value. a setting unit for setting a plurality of ranges from the upper limit value to the lower limit value by setting a plurality of lower limit values between the minimum pixel value and the pixel value one from the reference value to the minimum value;
a converting unit for generating a converted image for each of the plurality of ranges, in which the pixel values of the input image are converted so as to be within the range set by the setting unit;
a display control unit that controls to repeatedly display each of the plurality of converted images generated by the conversion unit in order of size of the range;
an image compositing display device comprising: The setting unit sets a value obtained by adding a value obtained by multiplying the standard deviation of the pixel values of the input image by a predetermined value to the reference value as an upper limit value, and sets a value obtained by multiplying the standard deviation by the predetermined value from the reference value. 2. The image synthesizing display device according to claim 1, wherein a plurality of said ranges are set by setting a plurality of values as said predetermined value in said range whose lower limit is the subtracted value. 3. The image composition according to claim 1, wherein said representative value is an average value, a median value, a mode value, a first quartile, or a third quartile of pixel values of said input image. display device. wherein, when the input image is a three-band image, the conversion unit generates the converted image for each of the plurality of ranges for each band image;
For each combination when each band image constituting the three-band image is assigned to the R plane, the G plane, and the B plane, the combination that minimizes the correlated color temperature is the R plane, the G plane, and the B plane. a synthesizing unit that generates a synthesized image obtained by synthesizing the transformed images corresponding to the band images assigned to the planes for each of the plurality of ranges;
4. The display control unit according to any one of claims 1 to 3, wherein each of the plurality of synthesized images generated by the synthesizing unit is controlled to be repeatedly displayed in order of size of the range. image synthesis display device. The synthesizing unit uses an average value of pixel values of the converted image corresponding to each band image as a value in an RGB color system, converts the value in the RGB color system into a value in an XYZ color system, and Chromaticity coordinates are calculated from values in the XYZ color system, the chromaticity coordinates are converted into constants for calculating the correlated color temperature, and the correlated color temperature is calculated based on a calculation formula using the constants. 5. The image compositing display device of claim 4, wherein: wherein, when the input image is a three-band image, the conversion unit generates the converted image for each of the plurality of ranges for each band image;
Among the band images constituting the three band images, the band image having the largest average pixel value of the converted image corresponding to the band image is assigned to the R plane, and the band image having the smallest average pixel value is assigned to the R plane. is assigned to the G plane, the remaining band images are assigned to the B plane, and a synthesized image obtained by synthesizing the transformed images corresponding to the respective band images is generated for each of the plurality of ranges,
4. The display control unit according to any one of claims 1 to 3, wherein each of the plurality of synthesized images generated by the synthesizing unit is controlled to be repeatedly displayed in order of size of the range. image synthesis display device. When the input image is a multiband image with three or more bands, principal component analysis is performed using a vector whose elements are each pixel value of each band image constituting the multiband image as an explanatory variable representing each band image. The first principal component image, 7. The image synthesizing display device according to any one of claims 4 to 6, further comprising an analysis unit that generates each of the second principal component image and the third principal component image as the 3-band image. The display control unit generates from image data written in a frame memory for displaying the input image on a display screen including a display area for displaying the input image and a display area for displaying the converted image. 8. The image synthesizing display device according to any one of claims 1 to 7, which displays the converted image. 9. The image composition according to any one of claims 1 to 8, wherein the display control unit displays the converted image in real time while fetching an image displayed by an image display system that is already in operation as the input image. display device. The display control unit includes a display area for displaying the input image, a display area for displaying the synthesized image, and the display area for displaying the R plane, the G plane, and the B plane when generating the synthesized image. 8. A display screen including a selection area for selecting whether the allocation of the converted images is determined by the synthesizing unit or manually specified. 1. The image synthesizing display device according to claim 1. A representative value of pixel values determined by a setting unit based on a histogram of pixel values of an input image is set as a reference value, and an upper limit value is set between the maximum pixel value and the pixel value one pixel value from the reference value to the maximum value. , and a plurality of lower limit values are set between the minimum pixel value and a pixel value that is one pixel value lower than the reference value, thereby setting a plurality of ranges from the upper limit value to the lower limit value. ,
generating a converted image for each of the plurality of ranges, wherein the conversion unit converts the pixel values of the input image so as to be within the range set by the setting unit;
An image synthesis display method, wherein a display control unit controls to repeatedly display each of the plurality of transformed images generated by the transforming unit in order of size of the range. the computer,
A representative value of pixel values determined based on a histogram of pixel values of an input image is set as a reference value, and a plurality of upper limit values are set between the maximum pixel value and a pixel value one pixel value above the reference value. a setting unit for setting a plurality of ranges from the upper limit value to the lower limit value by setting a plurality of lower limit values between the minimum pixel value and the pixel value one from the reference value to the minimum value;
a converting unit for generating a converted image for each of the plurality of ranges, in which the pixel values of the input image are converted to the range set by the setting unit;
An image synthesizing display program for functioning as a display control unit that controls to repeatedly display each of the plurality of transformed images generated by the transforming unit in order of size of the range.

Images