JP2010045455A - Image processor, image processing method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce operation addition in image processing and to shorten a processing time by efficiently making correction of moving image data corresponding to an observation environment. <P>SOLUTION: This image processor is provided with: a dividing means for dividing moving image data into a plurality of scenes; a calculating means for calculating a feature amount about the image quality of a scene; a classifying means for classifying the plurality of scenes into reference point scenes and non-reference point scenes; a selecting means for selecting one or more reference scenes from a scene set comprising the reference point scenes and a series of non-reference point scenes following the reference point scenes when they exist; a generating means for generating correction data that is applied to the scene set on the basis of a feature amount of one or more reference scenes; and a converting means for performing conversion processing of all scenes included in the scene set with correction data about the scene set. The classifying means classifies a scene the scene time duration of which is shorter than an observer's adaptation time to the scene as a non-reference point scene. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program.

  In reproduction of moving image data, in order to reproduce high-quality images on a display device, it is common to perform various image correction processes depending on the environment in which the user observes the display device. The moving image data changes in brightness and vividness when switching scenes or during a series of scenes. Therefore, in order to achieve high image quality of moving images, it is necessary to perform an optimal correction process for each scene according to the observation environment.

  In the prior art, scene switching or scene change is detected by analyzing a frame image of a moving image. Then, at the timing of the detected scene change or scene change, an optimal correction amount is obtained for each scene, and correction processing is performed on the moving image data.

  On the other hand, a human being who observes a moving image undergoes a physiological phenomenon of adaptation to changes in scene brightness and color. The time required for adaptation is studied and quantified as in Non-Patent Document 1.

  In the correction for the moving image data, the correction amount is calculated on the condition that the vision is adapted to the observation environment light and the displayed scene. Therefore, when switching to the next scene while adaptation to the scene change is not completed, the observer observes the scene that has been subjected to the correction process without being adapted to the scene. And at the start of the next scene, it tries to adapt again.

The technique described in Patent Document 1 counts the number of frames, and forcibly updates the correction amount when a predetermined number of frames is exceeded.
JP 2002-262303 A MD Fairchild, L. Reniff, “Time Course of Chromatic Adaptation for Color-Appearance Judgments,” (1995)

  If correction is performed without considering human adaptation time as in the above technology, correction is performed even for scenes that are not necessary, which increases the computational load and processing time of image processing, resulting in inefficiency. Is.

  Even in the technique described in Patent Document 1, since the correction amount is updated when the number of frames exceeds the count condition, the correction is performed even when the same scene is continuous and the correction amount need not be updated, Inefficient.

  In view of the above, an object of the present invention is to efficiently correct moving image data, reduce calculation addition in image processing, and shorten processing time.

In view of the above problems, an image processing apparatus according to the present invention is provided.
A dividing means for dividing moving image data into a plurality of scenes;
Calculating means for calculating a feature amount relating to the image quality of the scene;
Classifying means for classifying the plurality of scenes into a base point scene and a non-base point scene based on the duration of each scene;
Selecting means for selecting one or more reference scenes from a scene set comprising a base scene and a series of non-base scenes that, if present, follow the base scene;
Generating means for generating correction data to be applied to the scene set based on the feature amount of the one or more reference scenes;
Conversion means for performing conversion processing on all scenes included in the scene set using the correction data for the scene set;
The classifying unit classifies a scene whose duration time is shorter than an observer's adaptation time for the scene as the non-base point scene.

  According to the present invention, it is possible to efficiently correct moving image data, and it is possible to reduce the addition of computation and shorten the processing time in image processing.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

<First Embodiment>
A configuration of the image processing apparatus 100 according to the present embodiment will be described with reference to FIGS. 1 to 3. FIG. 1 is an example of a hardware block diagram of an image processing apparatus 100 according to the present embodiment.

  The image processing apparatus 100 includes a CPU 101, ROM 102, RAM 103, hard disk 104, operation unit 105, input device 106, and output device 107.

  The CPU 101 controls the entire image processing apparatus 100. The ROM 102 and the hard disk 104 store information data, a control program, an OS (Operating System), a color matching processing module, a device driver, and the like. The RAM 103 is used as a work area and a temporary save area for various control programs and input data.

  The operation unit 105 is used by the user for setting the image processing apparatus 100 and inputting moving image data, and includes, for example, a keyboard, a mouse, a display, and the like.

  The input device 106 includes a video camera including a CCD sensor, and acquires color space information in an observation environment for moving image data. The output device 107 includes a display and outputs moving image data.

  FIG. 2 is an example of a functional block diagram of the image processing apparatus 100 according to the present embodiment. The image processing apparatus 100 includes a storage unit 201, an environment information acquisition unit 202, a color conversion unit 203, a calculation unit 204, a profile generation unit 205, and an image conversion unit 206.

  The storage unit 201 stores moving image data to be processed (hereinafter referred to as input moving image data), other data such as a correction profile used in the image processing apparatus 100, and the like. The ROM 102, the RAM 103, and the hard disk 104 function as the storage unit 201, and are used properly according to the use of data.

  The environment information acquisition unit 202 acquires information regarding the observation environment where the image processing apparatus 100 is placed. The information related to the observation environment includes the illuminance and color temperature of the light source in the ambient light, the adaptation coefficient of the ambient light with respect to the white color of the display device, and the like.

  The color conversion unit 203 performs color conversion processing for converting the input moving image data into the color space coordinate system. Information relating to the observation environment necessary for the color conversion process is appropriately acquired by the environment information acquisition unit 202. The color conversion method by the color conversion unit 203 includes a method of converting input moving image data into a JCh coordinate system by CIECAM02, a method of conversion into an HSV coordinate system, and the like, and these methods are used as necessary.

  Based on the color space coordinate system converted by the color conversion unit 203, the calculation unit 204 calculates a feature amount and a variation amount in each frame included in the input moving image data. The feature amount is a value that characterizes the image quality of moving image data, and is a value obtained by, for example, white balance, brightness (contrast), exposure, sharpness, saturation, etc., or a combination thereof. In order to grasp the average behavior of moving image data, in the present embodiment, an average value related to feature quantities of several frames before and after the target frame is handled as a feature quantity in the target frame. The fluctuation amount is a difference between the feature amount in the target frame and the feature amount in the immediately preceding frame. As the reproduction contents of the moving image data change greatly, the value of the variation amount of the feature amount between frames also increases.

  The profile generation unit 205 generates a correction profile corresponding to the observation environment of the image processing apparatus 100 based on the feature amount of the moving image data in order to realize high-quality moving image reproduction of the input moving image data.

  The image conversion unit 206 corrects the input moving image data using the correction profile. The corrected moving image data (hereinafter referred to as output moving image data) may be reproduced on the output device 107 as it is or may be stored in the storage unit 201.

  Next, details of the profile generation unit 205 will be described with reference to FIG. FIG. 3 is an example of a detailed block diagram of the profile generation unit 205. The profile generation unit 205 includes a division unit 301, a classification unit 302, a selection unit 303, and a generation unit 304.

  The dividing unit 301 divides input moving image data into a plurality of scenes. A scene is a part of moving image data and is composed of one or more continuous frames. The scene division by the dividing unit 301 may be any of the conventional techniques. For example, it may be divided according to the results of brightness detection, saturation detection, exposure detection, motion detection, etc. for each frame, or may be divided using each method in combination.

  An example of the division will be described with reference to FIG. FIG. 4 is a diagram for explaining an example in which scene division is performed using the fluctuation amount. In the graph 400, the horizontal axis represents the reproduction time of the frames constituting the moving image data, and the vertical axis represents the variation amount in each frame. In the graph 400, a frame in which the fluctuation amount starts to increase rapidly is defined as the start of a new scene. As a result, the input moving image data is divided into four scenes from a scene 401 to a scene 404. As shown in the graph 400, the variation amount varies with the progress of the scene.

  The feature amount of each scene varies from scene to scene. Here, the feature amount of the scene is a value calculated based on the feature amount of each frame constituting the scene, and is, for example, an average value of the feature amount of each frame. Therefore, in principle, it is desirable to perform appropriate correction for each scene in order to improve the quality of moving image data. However, the correction amount is created assuming a state after the observer has adapted to the scene. Therefore, if the duration of the scene is compared with the time until the observer completes adaptation to the scene (hereinafter referred to as adaptation time), and the duration is shorter, the correction is performed. Can not be effective. Therefore, correction is not performed for a scene whose duration is shorter than the adaptation time, and the correction result of the scene before that scene is used as it is. Thereby, unnecessary processing can be omitted.

  Hereinafter, a scene to be corrected is referred to as a base scene, and a scene that is not corrected is referred to as a non-base scene. Also, a set of scenes composed of a base point scene and a series of non-base point scenes that follow this base point scene, if present, is called a scene set. Since the original correction is not performed for the non-base scene, the moving image data is reproduced based on the same correction amount as the correction performed in the base scene included in the same scene set.

  The classification unit 302 illustrated in FIG. 3 classifies each scene into a base scene and a non-base scene based on the adaptation time.

  The classification of the base scene and the non-base scene will be described with reference to FIG. FIG. 5 is a diagram for explaining an example of an operation for performing classification based on a comparison between the duration of the scene and the adaptation time. In the graph 500, the horizontal axis represents the reproduction time of the frames constituting the moving image data, and the vertical axis represents the variation amount in each frame. The adaptation time 511 to the adaptation time 514 represent the time required for the observer to adapt to each scene. The adaptation of the observer starts from the change of the scene corresponding to the start point of the arrow, and the adaptation is completed at the end point of the arrow.

  The dividing unit 301 divides the moving image data into four scenes from the scene 501 to the scene 504. First, the first scene 501 of the moving image data is corrected regardless of the duration of the duration. Therefore, the classification unit 302 classifies the scene 501 into the base scene. Next, the duration of the scene 502 and the adaptation time 512 are compared. Since the adaptation time 512 is longer, even if the correction for the scene 502 is performed, the process proceeds to the next scene 503 before the adaptation of the observer is completed. Therefore, the classification unit 302 classifies the scene 502 as a non-base scene. Subsequently, when the duration of the scene 503 is compared with the adaptation time 513, the adaptation time 513 is shorter. In this case, by correcting the scene 503, the observer can observe an optimal scene after adaptation. Therefore, the classification unit 302 classifies the scene 503 into the base scene.

  When the duration of the first scene 501 is shorter than the adaptation time 511, the scene 501 may be classified as a non-base scene and reproduced using a default correction amount.

  At this time, the scene 501 that is the base scene and the scene 502 that is the subsequent non-base scene constitute one scene set 521. In addition, since the scenes 503 and 504 that are the base point scenes do not have the following non-base point scenes, they constitute a single scene set 523 and 524, respectively. Correction data is created for each scene set defined in this way.

  Returning to FIG. 3, the description of the profile generation unit 205 will be continued. The selection unit 303 selects one or more scenes used for generating correction data for each scene set from each scene set. The selected scene is called a reference scene.

  In FIG. 5, the scene set 521 includes a scene 501 and a scene 502. Since the scene 502 has a large variation value, the feature amount of the scene 501 and the feature amount of the scene 502 are different. Therefore, in order to perform correction based on the same correction amount for these scenes, correction data for realizing high image quality is created for two scenes having different characteristics at the same time. Therefore, both scenes are selected as reference scenes, and correction data is generated based on these scenes. For a scene set 523 that includes only one scene, the scene is selected as a reference scene.

  Returning to FIG. 3, the description of the profile generation unit 205 will be continued. The generation unit 304 generates correction data for each scene set based on the feature amount of the reference scene. As described above, since the feature amount of the scene is calculated based on the observation environment, correction data corresponding to the observation environment is generated. When there are a plurality of reference scenes, the correction data is calculated for each scene, and further, the correction data is combined to determine correction data for the scene set. Here, the correction data indicates the amount of correction related to saturation correction, white balance correction, brightness correction, exposure correction, sharpness correction, and color correction (memory color reproduction, etc.) in order to achieve high image quality in moving image reproduction. Described data. Further, a correction profile in which correction data is associated with each scene set is created.

  The configuration of the correction profile will be described with reference to FIG. FIG. 6 is a diagram for explaining an example of the configuration of the correction profile 600.

  In the correction profile 600, the scene ID 601 is set with information for specifying a scene to be corrected. For example, the scene ID or the reproduction time of the first frame of the scene is used. Note that the scene set to the scene ID 601 is a base scene. In the various parameters 602, the correction amount of each correction data applied in the scene indicated by each scene ID 601 is set.

  The image conversion unit 206 refers to the correction profile stored in the storage unit 201 and performs correction based on the correction amount of each parameter on the input moving image data at the start of the base scene.

  Finally, the operation of the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 7 is a flowchart for explaining an example of the operation of the image processing apparatus 100. The processing shown in this flowchart is processed by the CPU 101 executing a computer program written in the ROM 102.

  In step S701, the image processing apparatus 100 acquires input moving image data.

  In step S <b> 702, the observer sets an adaptation time used by the classification unit 302. The adaptation time may be stored in advance in the storage unit 201 and the value may be used as a default value.

  In step S703, the dividing unit 301 divides the input moving image data into a plurality of scenes.

  In step S704, the classification unit 302 classifies each scene into a base scene and a non-base scene.

  In step S705, the selection unit 303 selects a reference scene from each scene set.

  In step S706, the generation unit 304 generates a correction profile.

  In step S707, the image conversion unit 206 corrects the input moving image data based on the correction profile, and converts it into output moving image data.

  As described above, according to the present embodiment, by correcting the moving image data based on the adaptation time according to human perception, it is possible to reduce the calculation load and the processing time in the correction processing, and the moving image correction processing To improve efficiency.

<Second Embodiment>
In the first embodiment, all scenes having a duration longer than the adaptation time are classified as base scenes. However, for a scene with less variation in the feature amount compared to the immediately preceding scene, it is possible to simplify the processing by using the same correction amount as the previous scene.

  FIG. 8 is a diagram for explaining an example of an operation for classifying scenes using threshold values. Similar to the first embodiment, the input moving image data is divided into four scenes from a scene 801 to a scene 804. The scene 801 is classified as a base scene, and the scene 802 is classified as a non-base scene.

  Scene 803 has a longer duration than adaptation time 813. However, since the fluctuation amount in the scene 803 is a small value, even if the same correction amount as that of the correction data up to the immediately preceding scene 802 is used, effective correction can be performed. Therefore, even if the scene has a duration longer than the adaptation time, the classification unit 302 classifies the scene whose fluctuation amount is lower than the threshold value 830 as a non-base scene. As a result, scenes 801 to 803 constitute one scene set 821 and are corrected using the same correction amount.

  In addition, since the difference in feature quantity between scenes included in the scene set 821 is small, the selection unit 303 may select any one scene as a reference scene. For example, a scene 801 that is a base scene can be selected as a reference scene. As a matter of course, as in the first embodiment, all scenes included in the scene set may be selected as reference scenes.

  As described above, a more efficient correction process can be provided.

<Third Embodiment>
In the above embodiment, all scenes having a duration shorter than the adaptation time are classified as non-base scenes. However, the feature amount may change extremely compared to the immediately preceding scene. For example, there may be a sudden difference in color temperature in white balance, or a sharp change in brightness such as switching from a bright image to a dark image. With respect to such a scene, even if the adaptation to the scene is not completed, the observer may feel uncomfortable if correction is not performed. In the present embodiment, appropriate correction is performed even in such a case. FIG. 9 is a diagram for explaining an example of an operation of classifying scenes using threshold values in the present embodiment. In the scene 902, the duration time is shorter than the adaptation time 912, but the value of the fluctuation amount is extremely large. Therefore, if the same correction amount as that of the scene 901 is used, the correction is not sufficient. Therefore, a scene having a fluctuation amount exceeding the threshold 930 is classified as a base point scene regardless of the duration of the duration.

  As described above, it is possible to provide a moving image reproduction with no sense of incongruity to the observer while improving the efficiency of the correction process.

<Fourth Embodiment>
In the embodiments described above, a common scene set is defined for various types of correction data, and correction data is generated for each scene set. However, a scene set may be defined for each type of correction data. For example, when the scene is switched, the white balance may change greatly, but the lightness may hardly change. In such a case, the classification unit 302 classifies the switched scene as the base scene with respect to the white balance, and classifies the scene as a non-base scene with respect to the brightness.

  An example of the correction profile 1000 in this embodiment is shown in FIG. The white balance correction 1001 and the brightness correction 1002 perform correction in different scenes.

  Furthermore, the adaptation time may be set individually for each type of correction data. For example, the adaptation time may be different between the change in white balance and the change in brightness. Also, in the change in brightness, a difference in tendency of brightness change may be determined, and an adaptation time corresponding to light adaptation and dark adaptation may be set. By setting the adaptation time individually for each type of correction data, correction can be performed more efficiently.

<Other embodiments>
Note that the present invention may be applied to a system including a plurality of devices (for example, a host computer, an interface device, a video camera, a monitor, etc.). Further, the present invention may be applied to a device composed of a single device (for example, a digital video camera with a liquid crystal monitor, a liquid crystal monitor and a digital camera device with a moving image shooting function, a mobile phone with a camera, etc.).

  The object of the present invention can also be achieved by supplying, to a system, a storage medium that records the code of a computer program that realizes the functions described above, and the system reads and executes the code of the computer program. As a storage medium for supplying the program code, for example, a floppy disk (registered trademark), a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. I can do it. In this case, the computer program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the computer program code constitutes the present invention. In addition, the operating system (OS) running on the computer performs part or all of the actual processing based on the code instruction of the program, and the above-described functions are realized by the processing. .

  Furthermore, you may implement | achieve with the following forms. That is, the computer program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Then, based on the instruction of the code of the computer program, the above-described functions are realized by the CPU or the like provided in the function expansion card or function expansion unit performing part or all of the actual processing.

When the present invention is applied to the above storage medium, the computer program code corresponding to the flowchart described above is stored in the storage medium.

It is an example of a hardware block diagram of the image processing apparatus 100 in the embodiment of the present invention. It is an example of the functional block diagram of the image processing apparatus 100 in the embodiment of the present invention. It is an example of the detailed block diagram of the profile production | generation part 205 in embodiment of this invention. It is a figure explaining an example which performs a scene division | segmentation using the variation | change_quantity in embodiment of this invention. It is a figure explaining an example of the operation | movement which classify | categorizes based on the comparison with the continuation time of a scene, and the adaptation time in the 1st Embodiment of this invention. It is a figure explaining an example of composition of amendment profile 600 in a 1st embodiment of the present invention. 6 is a flowchart illustrating an example of an operation of the image processing apparatus 100 according to the embodiment of the present invention. It is a figure explaining an example of operation which classifies a scene using a threshold in a 2nd embodiment of the present invention. It is a figure explaining an example of the operation | movement which classifies a scene using a threshold value in the 3rd Embodiment of this invention. It is a figure explaining an example of composition of amendment profile 1000 in a 4th embodiment of the present invention.

Explanation of symbols

100 Image processing apparatus 101 CPU
102 ROM
103 RAM
104 Hard Disk 105 Operation Unit 106 Input Device 107 Output Device

Claims (12)

A dividing means for dividing moving image data into a plurality of scenes;
Calculating means for calculating a feature amount relating to the image quality of the scene;
Classifying means for classifying the plurality of scenes into a base point scene and a non-base point scene based on the duration of each scene;
Selecting means for selecting one or more reference scenes from a scene set comprising a base scene and a series of non-base scenes that, if present, follow the base scene;
Generating means for generating correction data to be applied to the scene set based on the feature amount of the one or more reference scenes;
Conversion means for performing conversion processing on all scenes included in the scene set using the correction data for the scene set;
The image classification apparatus classifies a scene whose duration time is shorter than an observer's adaptation time for the scene as the non-base scene. The calculation means further calculates a variation amount of a feature amount between frames constituting each scene,
The image processing apparatus according to claim 1, wherein the classification unit further classifies a scene in which the variation amount is always below a threshold as a non-base scene. The calculation means further calculates a variation amount of a feature amount between frames constituting each scene,
The image processing apparatus according to claim 1, wherein the classification unit further classifies a scene having the variation amount exceeding a threshold as a base scene.   The image processing apparatus according to claim 1, wherein the classification unit further classifies the first scene of the moving image data as a base scene.   The image processing apparatus according to claim 1, wherein the selection unit selects a base scene included in the scene set as the reference scene of the scene set.   The image processing apparatus according to claim 1, wherein the selection unit selects all scenes included in the scene set as the reference scenes of the scene set.   The image processing apparatus according to claim 2, wherein the selection unit selects a scene having a variation amount exceeding the threshold as the reference scene.   The image processing apparatus according to claim 1, wherein the classification unit classifies a scene for each type of the correction data.   The image processing apparatus according to claim 8, wherein the classification unit uses an adaptation time corresponding to the type of the correction data for each type of division.   The image processing apparatus according to claim 1, wherein the feature amount is determined based on at least one of white balance, brightness, exposure, sharpness, and saturation. A dividing step of dividing the moving image data into a plurality of scenes;
A calculating step in which a calculating unit calculates a feature amount related to the image quality of the scene;
A classifying step for classifying the plurality of scenes into a base scene and a non-base scene based on the duration of each scene;
A selection step in which the selection means selects one or more reference scenes from a scene set composed of a base scene and a series of non-base scenes that follow the base scene if present;
A generating step of generating correction data to be applied to the scene set based on the feature amount of the one or more reference scenes;
A conversion step of performing conversion processing on all scenes included in the scene set with the correction data for the scene set;
An image processing method, wherein in the classification step, a scene whose duration is shorter than an observer's adaptation time for the scene is classified as the non-base scene.   A computer program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 10.

Images