JP2011205275A - Image processor, image processing program, imager, and image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of reproducing an output image in consideration of an influence by a scene of an image or recognition of an object.SOLUTION: The image processor includes an acquisition portion, an adjustment portion, an input portion, and a reproduction portion. The acquisition portion acquires features of an object imaged in an input image. The adjustment portion adjusts the degree of adaptation for an observation environment of the input image and an observation environment of an output image based on the acquired features. The input portion inputs information about the observation environment of the input image and the observation environment of the output image. The reproduction portion reproduces the appearance of the output image based on the degree of adaptation and the information.

Description

  The present invention relates to an image processing device, an image processing program, an imaging device, and an image display device.

  There has been proposed an image processing method for performing color matching processing on input image data in consideration of illumination brightness and white point in an observation environment (see, for example, Patent Document 1).

JP 2001-143065 A

  By the way, in the image processing method described in Patent Document 1 described above, when the appearance of an image is reproduced, only the visual influence such as the color of the image is considered, and the influence of recognition of the scene and subject of the image is taken into consideration. It wasn't.

  An object of the present invention is to provide an image processing device, an image processing program, an imaging device, and an image display device that can reproduce the appearance of an image in consideration of the influence of recognition of an image scene and a subject.

  An image processing apparatus according to the present invention includes an acquisition unit that acquires a feature of a subject in an input image, and an adjustment unit that adjusts the degree of adaptation to the observation environment of the input image and the observation environment of the output image based on the feature. And an input unit that inputs information about the observation environment of the input image and the output image, and a reproduction unit that reproduces the appearance of the output image based on the degree of adaptation and the information.

  An image processing program according to the present invention is an image processing program that causes a computer to function as an image processing device that reproduces the appearance of an image, the step of acquiring a feature of a subject in an input image, Based on the step of adjusting the observation environment of the input image and the degree of adaptation of the output image to the observation environment, the step of inputting information on the observation environment of the input image and the output image, the degree of adaptation and the information And reproducing the appearance of the output image.

  An imaging apparatus according to the present invention is an imaging apparatus that captures an image of a subject, and includes the image processing apparatus according to the present invention. The image display device of the present invention is an image display device that displays an image, and includes the image processing device of the present invention.

  According to the image processing device, the image processing program, the imaging device, and the image display device of the present invention, it is possible to satisfactorily reproduce the appearance of the image in consideration of the influence of the recognition of the scene and subject of the image.

1 is a diagram schematically illustrating a schematic configuration of an image processing apparatus according to an embodiment. 1 is a block diagram showing a system configuration of an image processing apparatus according to an embodiment. 5 is a flowchart for explaining image processing executed on input image data in the image processing apparatus according to the embodiment.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. The image processing apparatus is an image such as a personal computer (PC), an imaging apparatus such as a digital camera, an image display apparatus for displaying an image based on the image data on a liquid crystal screen, etc. A device that can perform processing. In this embodiment, a PC will be described as an example of the image processing apparatus.

  FIG. 1 is a diagram schematically showing a schematic configuration of a PC as an image processing apparatus according to this embodiment. The PC 1 is connected to another computer 4 via a digital camera 2, a recording medium 3 constituted by a CD-ROM and the like, and the Internet and other electric communication lines 5. Therefore, the PC 1 is configured to be able to acquire various image data from the digital camera 2, the recording medium 3, another computer 4, and the like and to display an image based on the acquired image data on the display 20. FIG. 2 is a block diagram showing a system configuration of the PC 1 as the image processing apparatus according to this embodiment. As shown in FIG. 2, a recording medium 3, a USB port 14, a network terminal 16, a memory 18, a display 6, and an operation unit 22 are connected to the CPU 10 that comprehensively controls each unit of the PC 1.

  The USB port 12 is for connecting a USB device. In this embodiment, the digital camera 2 is connected to the PC 1 via the USB port 12. The network terminal 14 is used to connect to a network such as the Internet. In this embodiment, another computer 4 is connected to the PC 1 via the network terminal 14. The memory 18 stores a control program for causing the CPU 10 to execute. The display 6 displays an output image based on output image data obtained by performing image processing by the CPU 10 on input image data input from the digital camera 2, the recording medium 3, another computer 4, or the like. The operation unit 22 includes an operation member operated by a user, such as a keyboard and a pointing device.

  In the PC 1 according to this embodiment, image processing is performed on input image data input from the digital camera 2, the recording medium 3, another computer 4, and the like. An image processing program executed by the PC 1 for image processing is downloaded from the recording medium 3, another computer 4 via the Internet or other electric communication line, etc., stored in the memory 18, and installed in the PC 1. . Hereinafter, with reference to the flowchart shown in FIG. 3, the image processing executed on the input image data in the PC 1 according to this embodiment will be described.

  First, the CPU 10 acquires input image data (step S10). In this embodiment, it is assumed that image data of a photographed image photographed by the digital camera 2 is acquired as input image data. The input image data acquired in step S10 is composed of a color space RGB unique to the spectral sensitivity of the camera.

Next, the CPU 10 acquires information related to the observation environment of the input image (step S11). Specifically, as information related to the observation environment of the input image, information related to illumination of the shooting scene when the input image is captured by the digital camera 2 is acquired. That is, the illumination white point (X _W , Y _W , Z _W ), luminance Y _W0 , and ambient luminance Y _SW0 of the shooting scene are acquired. For example, a user (observer) measures a white point and luminance using a spectral radiance meter or the like, and reads a value obtained by inputting a measurement result from a keyboard. Alternatively, the result of the photometric function used for automatically adjusting the exposure (AE) by calculating the illumination white point (X _W , Y _W , Z _W ) at the time of shooting using the auto white balance function provided in the digital camera 2 The illumination brightness _YW0 at the time of photographing can be calculated by analyzing the above, and the ambient brightness _YSW0 can be calculated using the result of photometry with the photometry range of the photometry function expanded. Alternatively, information added to the input image data at the time of shooting is acquired from the digital camera 2 or is input by the user via the operation unit 22, and the illumination white point (X _W , Y _W , Z _W ) is determined from the added information. , Luminance Y _W0 , and ambient luminance Y _SW0 can also be acquired.

Next, CPU10 acquires the information regarding the observation environment at the time of observing an output image (step S12). In this embodiment, the user observes the output image by visually recognizing the output image displayed on the display 6. Therefore, for example, the CPU 10 displays an output image on the display 6 under the standard observation environment. Information regarding the sRGB standard observation environment (white point = D65, luminance Y _device0 ′ = 80Cd / m2, ambient luminance Y _SW0 ′ = 4 .1Cd / m2). When the output image is observed under an observation environment different from the sRGB standard observation environment, or when the output image is observed with a printed matter printed out, the CPU 10 is first input by the user via the operation unit 22. Ambient illumination white point (X _SW ′, Y _SW ′, Z _SW ′), ambient illumination luminance Y _SW0 ′, white point of the device used for output image observation (X _device ′, Y _device ′, Z _device ′), and The brightness Y _device0 ′ of the device is read. Then, a mixture of the ambient illumination white point and the device white point is used as an illumination white point (X _W ′, Y _W ′, Z _W ′) and ambient illumination brightness in the observation environment when observing the output image. Can be used as the luminance Y _W0 ′.

  Next, the CPU 10 acquires the characteristics of the input image (step S13). In this embodiment, the shooting scene of the input image and main subject information are acquired as features of the input image. Specifically, information relating to a photographic scene such as the sea and portrait recognized by the scene recognition function provided in the digital camera 2 (for example, added to the input image data) is acquired from the digital camera 2. Also, main subject information (for example, added to input image data) when a person recognized by the face recognition function provided in the digital camera 2 is a main subject can be acquired from the digital camera 2. . Further, the shooting scene and main subject information input by the photographer at the time of shooting may be acquired from the digital camera 2, and the shooting scene and main subject information input by the user via the operation unit 22 may be acquired. .

  Next, the CPU 10 adjusts the degree α of adaptation to the observation environment of the input image and the observation environment of the output image based on the shooting scene and main subject information of the input image acquired in step S13 (0.0 ≦ α ≦ 1.0). ) Is adjusted (step S14). Specifically, information on subjects having a characteristic color such as sea, portrait (skin color), banana, and the like, and the degree of adaptation α corresponding to each subject are registered in advance in a table format and stored in the memory 18 or the like. Then, it is determined whether or not there is a subject with a characteristic color in the input image. If there is a subject with a characteristic color, it matches the subject from the registered subject information. The information of the subject is read and the adaptation degree α registered in advance is set. That is, when there is a subject with a characteristic color, the adaptation degree α is increased (for example, α = 0.9 is set).

  In addition, the adaptation degree α can be adjusted based on at least one of the size, position, and degree of focus of the subject with a characteristic color on the shooting screen. For example, when the size of the subject with respect to the shooting screen is large, the degree of adaptation α is increased compared to the case where the size is small. Similarly, when the position of the subject with respect to the shooting screen is the center, the degree of adaptation α is increased as compared to the case of the edge, and when the degree of focus of the subject is high, the degree of adaptation α is compared with the case of low. increase.

  When there are a plurality of subjects with characteristic colors, the degree of adaptation of each registered subject is read, the average value of each degree of adaptation is calculated, and the calculated average value is set as the degree of adaptation α. . In addition, according to at least one of the size, position, and degree of focus of a plurality of subjects on the shooting screen, weighting of the degree of adaptation of each subject is performed, a weighted average value of each degree of adaptation is calculated, and the calculated weighting The average value can also be set as the adaptation degree α. In addition, when there are a plurality of subjects with distinctive colors, the degree of adaptation corresponding to each registered subject is read, and the degree of adaptation corresponding to one subject is gradually changed to the degree of adaptation corresponding to another subject. It is also possible to set the adaptation degree α so as to change. In addition, if the lighting conditions for illuminating the subject and the lighting conditions for illuminating the background are different, the degree of adaptation should be set so as to gradually change from the degree of adaptation corresponding to the subject to the degree of adaptation corresponding to the background. You can also. On the other hand, when there is no subject with a characteristic color, the adaptation degree α is set to 0.0.

Next, the CPU 10 performs chromatic adaptation conversion processing based on the adaptation degree α adjusted in step S14 (step S15). Specifically, the adaptation factor D indicating the degree of adaptation of the input image to the observation environment and the adaptation factor D indicating the degree of adaptation of the output image to the observation environment while considering the degree of adaptation α adjusted in step S14. 'Is calculated from the illumination brightness and ambient brightness in each observation environment (see Equations 1 and 2).

  Here, LA and LA ′ indicate the adaptation luminance, and the adaptation luminances LA and LA ′ are the luminances that are adapted in the observation environment of the input image and the output image, and are about 1/5 of the normal illumination luminance. F and F ′ are parameters determined according to the ratio of the illumination luminance and the ambient luminance in the observation environment of the input image and the output image.

Since the input image is described in the device-dependent color space RGB, color conversion is performed into the CIE 1931 XYZ color space, which is a device-independent color space. In this embodiment, since the input image is a captured image captured by the digital camera 2, it is described in a specific color space RGB determined by the spectral sensitivity of the camera. Therefore, color conversion is performed to a device-independent XYZ color space using a predetermined camera RGB → XYZ matrix M _{RGB → XYZ} . Alternatively, when the input image is encoded in a standard color space (for example, sRGB), the standard color conversion matrix M _{RGB → XYZ} is used to perform color conversion into an XYZ color space that does not depend on the device. Further, color conversion is _performed from XYZ to cone response LMS using matrix _MCAT02 . Using the illumination white points (X _W , Y _W , Z _W ), (X _W ′, Y _W ′, Z _W ′) and the adaptation factors D, D ′ in the observation environment of the input image and the output image in the LMS space Kries color adaptation conversion is performed, and LMS is converted to XYZ, and XYZ is converted back to RGB of the output device. That is, color conversion as shown in the following equation (see Equations 3 to 7) is performed for each pixel.

However,

In the above description, (X _W ′, Y _W ′, Z _W ′) is used as the illumination white point in the output image observation environment. However, the device white point (X _device ′, Y at the time of output image observation) is used. _{If the device} ', Z _device ') and the ambient illumination white point (X _SW ', Y _SW ', Z _SW ') are different, the illumination in the viewing environment of the output image as a mixture of the device white point and the ambient illumination white point The white point (X _W ′, Y _W ′, Z _W ′) and the illumination brightness Y _wo ′ are determined. Specifically, using the device luminance Y _device0 ′, the ambient illumination luminance Y _SW0 ′, and the ratio R _mix of the influence of the device light source used for image observation and the ambient illumination light source, the following equation (see Equations 9 to 11) is given. Calculate as shown.

  Next, the CPU 10 reproduces the appearance of the output image (step S16). In this embodiment, the output image is displayed on the display 6.

  According to the PC 1 as the image processing apparatus according to the present embodiment, the features of the input image, that is, the shooting scene and main subject information are acquired. In addition, the degree of adaptation of the output image to the observation environment is increased. Therefore, it is possible to satisfactorily reproduce the output image in consideration of the effect of the input image shooting scene and subject recognition.

  Conventionally, the color appearance is different under different observation environments such as the white point of illumination, brightness, and ambient brightness, and CIECAM97s and CIECAM02 are examples of color appearance models that take into account such an observation environment. It was. For example, tristimulus values XYZ, adaptation luminance La, and the like are calculated as the monitor viewing conditions as values depending on the monitor white and monitor luminance, and are applied to the color appearance model. The higher the luminance, the more the white point of the observation environment adapts to the white point of the observation environment, so that the white point of the observation environment appears closer to “white”. In a color appearance model such as CIECAM02, an adaptation factor indicating the degree of adaptation to the white point of the observation environment, reflecting such visual characteristics, is set as illumination luminance and ambient brightness in the observation environment. It was calculated by the function of

  That is, in the above-described conventional color appearance model, only the visual influence (such as chromatic adaptation) is taken into consideration, so that the appearance close to the appearance of the color patch is reproduced. However, humans originally view images, actual scenes, etc. as a result of visual processing of signals acquired by vision, and the effects of recognition of scenes and subjects when viewing such images, actual scenes, etc. In view of this, there is a case where an appearance close to the appearance of an image or an actual scene cannot be reproduced. In other words, information on the colors of these scenes and objects, such as the sea, sky, human skin, and banana color, is input to the human brain in advance, and these recognitions are taken into consideration in the conventional color appearance model. However, in this embodiment, since these recognitions are taken into consideration, it is possible to reproduce an appearance close to the appearance of an image or an actual scene.

  In the above-described embodiment, the degree α of adaptation to the observation environment of the input image and the observation environment of the output image is adjusted based on the shooting scene of the input image and main subject information. The degree of adaptation α may be adjusted according to the appearance to the user. In this case, a function is provided that allows the user to change the adjusted adaptation degree α. When the adaptation degree α is changed by the user, the CPU 10 stores information related to the change in the memory 18. Then, the CPU 10 adjusts the degree of adaptation α in consideration of the number of changes in the past, the difference between before and after the change, and the like. For example, for a user who feels that the output image looks too red for an input image taken under red illumination, the adaptation level α is increased and the output image is taken against an input image taken under red illumination. If the user feels red is not enough, the degree of adaptation α is lowered.

  In the above-described embodiment, the case where the input image is a captured image captured by the digital camera 2 has been described as an example. However, the present invention can be applied to a case where the input image is an image other than the captured image. Can be applied. For example, when the appearance of the input image on the monitor is reproduced to the appearance of the output image by printing, that is, when the appearance of the input image at the time of observing the input image is reproduced as the appearance of the output image at the time of observing the output image. The invention can be applied. In this case, in step S10 shown in FIG. 3, an image at the time of observing the input image is obtained instead of the photographed image, and at step S11, an observation environment at the time of observing the input image is obtained instead of the observation environment at the time of photographing. To do.

  In the above-described embodiment, the PC is described as an example of the image processing apparatus of the present invention. However, the present invention can also be applied to an imaging apparatus such as a digital camera. In this case, the CPU provided in the digital camera acquires the observation environment at the time of observing the output image specified by the user, the acquired observation environment at the time of observing the output image, the captured image captured by the digital camera, and the captured image An output image is created based on the observation environment specified by the shooting scene, and the created output image is stored in a memory or the like provided in the digital camera. When the user does not designate an observation environment for observing the output image, a standard observation environment such as an sRGB observation environment is used.

  In addition, when the digital camera has an auto white balance function, the present invention can be applied when white balance is adjusted in the digital camera. Specifically, the characteristics of the shooting scene and subject to be photographed are acquired, the adaptation degree α is adjusted according to the acquired characteristics, and the adjusted adaptation degree α is reflected in the gain value of the white balance. By reflecting the adaptation degree α on the gain value of the white balance, it is possible to obtain a photographed image that takes into consideration the influence of recognition of the photographed scene and subject such as a photographed scene and subject having a characteristic color.

  In the above-described embodiment, the von Kries chromatic adaptation conversion process has been described as an example of the chromatic adaptation conversion process in the color appearance model. The invention can be applied.

  In the above-described embodiment, an image processing program for causing the PC 1 to execute image processing is read from the memory 18 or the like and executed. In other words, the image processing program used in the above-described embodiment is an image processing program that causes the PC 1 to function as an image processing device that reproduces the appearance of an image. Adjusting the degree of adaptation to the observation environment of the input image and the observation environment of the output image based on the characteristics of the input image, inputting information regarding the observation environment of the input image and information regarding the observation environment of the output image, Reproducing the appearance of the output image based on the adjusted degree of adaptation of the input image to the observation environment and the output image to the observation environment, information on the input environment of the input image and information on the observation environment of the output image; Is running. Therefore, according to the image processing program used in the above-described embodiment, it is possible to satisfactorily reproduce the output image in consideration of the influence of the recognition of the scene and subject of the input image.

  Further, the present invention is not limited to the contents described in the above embodiment. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Personal computer (PC), 2 ... Digital camera, 3 ... Recording medium, 4 ... Other computer, 6 ... Display, 10 ... CPU.

Claims (9)

An acquisition unit that acquires the characteristics of the subject in the input image;
An adjustment unit that adjusts the degree of adaptation to the observation environment of the input image and the observation environment of the output image based on the characteristics;
An input unit for inputting information regarding an observation environment of the input image and the output image;
A reproduction unit that reproduces the appearance of the output image based on the degree of adaptation and the information;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the adjustment unit increases the degree of adaptation.   The image processing according to claim 1, wherein the adjustment unit adjusts the degree of adaptation based on at least one of a size, a position, and a focus degree of a main subject in the input image. apparatus. A changing unit capable of changing the degree of adaptation;
The image processing apparatus according to claim 1, wherein the adjustment unit adjusts the degree of adaptation based on a past change result by the change unit. The acquisition unit acquires a plurality of the features,
The image processing apparatus according to any one of claims 1 to 4, wherein the adjustment unit adjusts an average adaptation degree of each adaptation degree based on each feature. The acquisition unit acquires the first feature and the second feature as the feature,
The adjustment unit adjusts the adaptation degree so as to gradually change from a first adaptation degree based on the first feature to a second adaptation degree based on the second feature. The image processing apparatus according to claim 4. An image processing program that causes a computer to function as an image processing device that reproduces the appearance of an image,
Obtaining the characteristics of the subject in the input image;
Adjusting the degree of adaptation to the observation environment of the input image and the observation environment of the output image based on the characteristics;
Inputting information relating to the observation environment of the input image and the output image;
Reproducing the appearance of the output image based on the adaptation degree and the information;
An image processing program comprising: An imaging device for imaging a subject,
An imaging apparatus comprising the image processing apparatus according to any one of claims 1 to 6. An image display device for displaying an image,
An image display device comprising the image processing device according to any one of claims 1 to 6.

Images