JP2011223054A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology to perform control of advanced image processing while suppressing recording capacity in an image processing apparatus at the time of recording.SOLUTION: An image processing apparatus comprises: an image analyzing section for analyzing an input image and calculating a sequential feature amount indicating a feature of an image for each frame; a scene detecting section for dividing the input image into a plurality of scenes; a scene feature analyzing section for calculating a scene feature amount indicating a feature of images of the entire scene from the sequential feature amount of each frame included in the scene; a control amount generating section for generating a control amount for use in an image correction in all the frames included in the scene based on the scene feature amount; an accumulating section for accumulating the input image and the control amount; and an image processing section for performing image processing for the scene of the accumulated input image with use of the control amount corresponding to the scene to be output as a reproduced image.

Description

  The present invention relates to an image processing apparatus and an image processing method.

  2. Description of the Related Art Conventionally, there has been known a technique of recording image feature amounts as metadata in image-quality image processing in an image processing apparatus that accumulates input moving image images in a storage device unit and later reproduces the images. In this method, image analysis is performed when a still image or a moving image is accumulated, and the brightness or color tendency of the image for each frame is digitized as a feature amount. Then, the feature amount is converted into metadata and recorded together, and image processing is performed based on the metadata during image reproduction (Patent Documents 1 and 2).

  Further, as a general trend in recent years, as the contents of image processing are advanced, the feature amount of the image to be referred to is detailed and the amount of information is increasing. For example, as information to be referred to for correcting the dynamic range of the image signal, the brightness histogram of the input signal is used instead of the brightness of the entire image. As a specific example, an image processing method is known in which a luminance correction curve is generated by accumulating luminance histograms and adaptively correcting the luminance (Patent Document 3).

Further, an image processing method for performing image enhancement processing according to the spatial frequency characteristics of the processing target image is known (Patent Document 4).
In addition, the MPEG-7 standard is defined as a standard for converting image characteristics into metadata and utilizing it for image retrieval and the like.

JP 2003-324756 A JP 2008-084213 A Japanese Unexamined Patent Publication No. 03-126377 JP 2009-1111727 A

However, when the detailed feature amount is converted into metadata by the above-described technique, the generated metadata becomes a huge amount of data. For example, when a luminance histogram of 256 categories obtained by classifying the luminance component of each pixel constituting the image into 256 levels of brightness and totaling the frequency is converted into metadata by a notation similar to the MPEG-7 standard, the metadata shown in FIG. It can be expressed as follows. In this representation, about 9000 characters are required as metadata of the luminance histogram for each frame. If this is accumulated for two hours corresponding to one movie, the storage capacity increases as follows, which causes an increase in the cost of the apparatus.
9000 characters / frame × 60 frames / s × 60 seconds × 60 minutes × 2 hours
= 3888000000 characters ≒ 3.6GB

  Here, only the luminance histogram was converted to metadata. However, in order to further enhance picture creation control, if other various features are also converted to metadata at the same time, the metadata capacity will increase and the cost of the device will increase. It becomes a factor of the rise.

The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique for controlling recording capacity at the time of recording in an image processing apparatus and performing advanced image processing control. To do.

  In order to solve the above-described problems, the present invention employs the following configuration. That is, an image analysis unit that analyzes an input image to calculate a sequential feature amount that represents a feature of the image for each frame, a scene detection unit that divides the input image into a plurality of scenes, and a frame that is included in the scene. A scene feature analysis unit that calculates a scene feature amount representing the image feature of the entire scene from the sequential feature amount, and a control amount that is used for correcting images of all frames included in the scene based on the scene feature amount A control amount generation unit, a storage unit that stores the input image and the control amount, and a scene of the stored input image are subjected to image processing using a control amount corresponding to the scene, and output as a reproduced image An image processing apparatus having an image processing unit.

  The present invention also employs the following configuration. That is, an image processing method executed by an image processing apparatus, comprising: an image analysis step in which an input image is analyzed to calculate a sequential feature amount representing image features for each frame; and the input image is divided into a plurality of scenes. Included in the scene based on the scene feature amount, a scene feature analysis step for calculating a scene feature amount representing a feature of the image of the entire scene from a sequential feature amount of each frame included in the scene, A control amount generating step for generating a control amount used for correcting the image of all frames, a storage step for storing the input image and the control amount, and a control corresponding to the scene in the stored input image scene And an image processing step of performing image processing using the amount and outputting as a reproduced image.

  According to the present invention, it is possible to suppress the storage capacity at the time of recording in the image processing apparatus and perform advanced image processing.

1 is an overall configuration diagram of an image processing apparatus according to Embodiment 1. FIG. The detailed block diagram of the image analysis part 5 and the scene feature analysis part 7. FIG. 7 is a flowchart for explaining the operation of the scene feature analysis unit 7; The conceptual diagram of the operation | movement along the time of the scene feature analysis part 7. FIG. The block diagram of the scene picture creation analysis part 8. FIG. The conceptual diagram of each signal handled by the scene picture making analysis part 8. FIG. The conceptual diagram of a brightness correction curve. An example of generated metadata R30. The block diagram of the picture making control part 6. FIG. The block diagram of the selection part 9. FIG. The conceptual diagram of operation | movement of the correction curve restoration part 95. FIG. The conceptual diagram of the operation | movement along the time of the image processing apparatus at the time of reproduction | regeneration. FIG. 3 is a diagram illustrating a subject to be photographed in the first embodiment. FIG. 4 is a diagram illustrating an image photographed in the first embodiment. FIG. 6 is a diagram illustrating a luminance histogram of an image shot in Example 1. The figure which shows the brightness | luminance correction curve in the case of performing optimization sequentially on the image | photographed image. The figure which shows the image in the case of performing optimization sequentially on the image | photographed image. FIG. 6 is a diagram showing a peak hold histogram R72 in the first embodiment. FIG. 4 is a diagram illustrating a scene-optimized brightness correction curve according to the first embodiment. FIG. 3 is a diagram illustrating a scene optimized image according to the first embodiment. FIG. 10 is a diagram illustrating an image from which an effective image processing result is obtained according to the second embodiment. FIG. 10 is a diagram illustrating an image from which an effective image processing result is obtained according to the third embodiment. FIG. 10 is an overall configuration diagram of an image processing apparatus according to a fourth embodiment. The figure which shows the example which made the luminance histogram metadata.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. The following image processing apparatus is used to record an input image and output it on demand. The image processing device can be a single device, a broadcast receiving device such as a television or STB (Set Top Box),
It can also be incorporated into a display device. The present invention can also be applied to a camcorder that performs shooting and image storage. Each component of the image processing apparatus may be configured by a dedicated circuit or may be configured as a program that operates on the CPU. Further, the present invention can be realized as an image processing method by causing a program or the like to execute each process performed by the image processing apparatus.

<Example 1>
FIG. 1 shows an overall configuration diagram of an apparatus according to the first embodiment of the present invention. Hereinafter, the operation when recording an image in the image processing apparatus according to the first embodiment will be described with reference to FIG.
The time code assigning unit 2 assigns a time code to an input image signal R1 input from an image input means (not shown) and outputs an input image signal R2 with time code. The image selector 3 selects the input image signal R2 with time code and outputs the selected image signal R3. The image analysis unit 5 analyzes the selected image signal R3 for each frame and sequentially outputs a feature amount R5 for each frame. Details of the operation in which the image analysis unit 5 sequentially outputs the feature amount R5 will be described later.

  The scene feature analyzing unit 7 sequentially detects cuts between successive frame cuts from the feature amount R5, classifies a plurality of scenes, and outputs a scene feature amount R7 that is an image feature of the entire scene. Details of the operation of the scene feature analysis unit 7 will be described later. The scene picture creation analysis unit 8 inputs a scene feature amount R7 for each scene, and outputs a scene control scenario R8 that is a picture creation control amount for each scene. Details of the processing of the scene picture creation analysis unit 8 will be described later. The metadata generation unit 30 converts the scene control scenario R8 into metadata and outputs the generated metadata R30. The image recording unit 31 associates the selected video signal R3 with the generated metadata R30, converts it into a stream for storage, and outputs it as a recording signal R31. The storage device unit 32 records the recording signal R31 on the hard disk device. The storage device unit corresponds to the storage unit of the present invention.

  The time code generator 18 resets the time code at the start of recording, and generates a time code that counts up on a frame basis until the recording stops thereafter. The generated time code is distributed to each block as a time code R18 (not shown). The control unit 19 controls the operation of each block of the apparatus through a control line (not shown).

Next, details of the image analysis unit 5 and the scene feature analysis unit 7 during image recording will be described. A detailed block diagram of the image analysis unit 5 and the scene feature analysis unit 7 is shown in FIG. The APL calculation unit 51 calculates an APL (Average Picture Level) of the selected image signal R3 for each frame and outputs an APL value R51. The luminance histogram calculation unit 51 calculates a luminance histogram of the selected image signal R3 for each frame and outputs a luminance histogram R52. The sequential feature amount R5 is a structure including an APL value R51 and a luminance histogram R52 for each frame.

The scene change detection unit 71 monitors the APL value R51, determines that a scene change has occurred when the amount of change between frames exceeds a threshold value, and outputs a scene change detection signal R71. As a result, scene detection is performed, and a plurality of scenes each having a start point and a duration are segmented. A peak hold histogram generator 72 (MaxHist generator) receives a luminance histogram R52 and a scene change detection signal R71 for each frame. Based on these, the maximum value of the frequency of each category of the luminance histogram R52 between scene changes is calculated, and a peak hold histogram R72 is output. Details of the method of generating the peak hold histogram R72 will be described later. The cumulative histogram generation unit 73 calculates the cumulative frequency of each category of the luminance histogram R52 between scene changes based on the luminance histogram R52 for each frame and the scene change detection signal R71, and generates a cumulative histogram R73. Details of the method of generating the cumulative histogram R73 will be described later. The scene histogram generator 74 generates a scene feature amount R7 based on the peak hold histogram R72 and the cumulative histogram R73. The scene change detection unit corresponds to the scene detection unit of the present invention.

Next, the detailed operation of the scene feature analysis unit 7 will be described. A flow chart showing the operation of the scene feature analysis unit 7 is shown in FIG. The control of the operation is performed by the control unit 19 through a control line (not shown). A conceptual diagram of the operation is shown in FIG.
The four arrays used in the following description are defined as follows.
InH [x] ... Scene histogram R7 (scene feature)
MaxH [x] ... peak hold histogram R72
TtlH [x] ... Cumulative histogram R73
AveH [x] ... Average histogram (not shown)
In the above array, x is the number of the category of luminance to be counted and takes a range of 0 to 255. 0 is the darkest category and 255 is the brightest category. For example, the frequency of the category with the lowest luminance in the scene histogram R7 (scene feature amount) is represented as InH [0].

The scene feature analysis unit 7 is activated for each frame, and the flow of FIG. 3 starts.
In step S701, a sequential feature amount R5 including an APL value R51 and a luminance histogram R52 is input.
In step S702, the scene change detection unit 71 determines whether or not there is a scene change. The value of the previous frame of the APL value R51 is compared with the value of the current frame. If the absolute value of the difference is equal to or greater than the threshold, it is determined that there is a scene change, and the process proceeds to step S705. If there is no scene change, the process proceeds to step S703.

In step S703, the peak hold histogram generation unit 72 performs a peak hold histogram generation calculation. The following calculation is performed on the held peak hold histogram R72 (MaxH [x]).
MaxH [x] = max (MaxH [x], InH [x])

In step S <b> 704, the cumulative histogram generation unit 73 performs a cumulative histogram generation calculation. The following calculation is performed on the stored cumulative histogram R73 (TtlH [x]).
TtlH [x] = TtlH [x] + InH [x]
If there is no scene change, the scene feature analysis process ends in step S704.

In step S705, the scene histogram generation unit 74 calculates the number of frames (Duration) from the previous scene change to the current frame. Subtract the saved TimePoint from the time code of the current frame to make Duration. For example, at time Tb in the conceptual diagram (FIG. 4), Duration is as follows.
Duration = Tb-Ta = Da

In step S706, the scene histogram generation unit 74 generates an average histogram AveH [x] (not shown). From the accumulated histogram R73 (TtlH [x]) held and the number of duration frames calculated in step S705, the following calculation is performed.
AveH [x] = TtlH [x] / Duration

In step S707, the scene histogram generator 74 generates a sum of absolute histogram differences SAHD (sum of absolute histogram differences) (not shown). The following calculation is performed from the peak hold histogram R72 (MaxH [x]) and the average histogram AveH [x].

In step S708, it is determined whether to output the scene feature amount R7. Comparing the difference absolute value sum SAHD with a predetermined threshold, and if the difference absolute value sum SAHD is larger, it is determined that there is a large variation in the luminance distribution in the scene and the scene optimal control is performed. The process advances to step S709 to output the scene feature amount R7. If the difference absolute value sum SAHD is smaller, it is determined that the scene corresponds to the sequential adaptive control, and the process proceeds to step S710 without outputting the scene feature amount.
In step S709, the scene histogram generation unit 74 outputs the scene feature amount R7. The peak hold histogram R72 is output as the scene feature amount R7. This value is used as a feature of the image of the entire scene.
In step S710, the current time code is recorded as a new TimePoint in a register (not shown) in the scene histogram generator 74.

In step S711, the peak hold histogram R72 held in the peak hold histogram generator 72 is cleared.
In step S712, the cumulative histogram R73 held in the cumulative histogram generator 73 is cleared.
With the above control, the scene feature analysis unit 7 generates a scene feature amount R7.

Referring to the conceptual diagram of FIG. 4, the point of time code Tb when the scene change is detected is an example in which a scene feature amount R7 for the Shot-A video section represented by TimePoint = Ta and Duration = Da is generated. is there. Further, at the time code Tc, the scene feature R7 is not output for the Shot-B video section because the change in the feature R5 is not large.

  Next, details of a method for generating the scene control scenario R8 in the scene picture creation analysis unit 8 will be described. A block diagram of the scene picture creation analysis unit 8 is shown in FIG.

The histogram normalization unit 81 normalizes the histogram of the scene feature amount R7 and outputs a normalized histogram R81. A conceptual diagram of the input scene feature R7 is shown in FIG.
A conceptual diagram of the normalized histogram R81 is shown in FIG.
The histogram integration unit 82 cumulatively adds the normalized histogram R81 in order from the lower category (FIG. 6 (c)). Then, a brightness correction curve R82 for tone correction is generated from the cumulative addition histogram (FIG. 6 (d)). Since the brightness correction curve is generated and the control amount is generated in this way, the scene picture creation analysis unit corresponds to the control amount generation unit of the present invention.
The standard curve storage unit 83 stores a standard brightness correction curve set in a ROM or the like for each picture making mode that is a policy of image correction of the image processing apparatus. These standard luminance correction curves correspond to the predetermined control amount of the present invention. In this embodiment, it is assumed that there are two sets of picture making modes, and two sets of standard luminance correction curves R83 shown in FIGS. 7A and 7B are stored. The horizontal axis represents the gradation value of the input image, and the vertical axis represents the gradation value of the output image. FIG. 7A shows a picture making mode in which the number of bits of the output image is evenly assigned to each gradation of the input image. On the other hand, FIG. 7B shows a picture making mode in which a large number of bits are assigned to the intermediate gradation of the input image.

The representative point difference extraction unit 84 extracts a difference between the generated luminance correction curve R82 and the standard luminance correction curve R83 to generate a scene control scenario R8. For each set of the standard luminance correction curve R83, a difference at a representative point unique to each set is extracted. Since the gradation to be emphasized may be different for each picture-making mode, the representative point for difference extraction is unique for each mode. A conceptual diagram of the difference extraction is shown in FIGS. 7 (c) and 7 (d). A scene control scenario R8 is obtained by associating the obtained difference with the time information of the target scene. That is, the scene control scenario R8 is a structure including the following ten elements.
TimePoint of the target scene, Duration of the target scene
Position of representative point 1 of mode (a), difference of representative point 1 of mode (a) Position of representative point 2 of mode (a), difference of representative point 2 of mode (a) Representative point 1 of mode (b) Position, difference of representative point 1 of mode (b) position of representative point 2 of mode (b), difference of representative point 2 of mode (b)

With the above operation, it is possible to record an image in the apparatus of the first embodiment. The scene control scenario corresponds to the difference information of the present invention. The scene picture creation analysis unit corresponds to the difference information generation unit of the present invention.
If the scene feature amount R7 is not output, the scene control scenario R8 is also not output.
FIG. 8 shows generated metadata R30 obtained by converting the scene control scenario R8 into metadata by the metadata generating unit 30.

Next, an operation for reproducing an image in the image processing apparatus according to the first embodiment will be described.
The image reproduction unit 33 acquires the reproduction signal R32 corresponding to the image to be reproduced from the storage device unit 32, separates it into a metadata and image stream, and outputs the reproduction metadata R33 and the reproduction image signal R13. When the scenario request signal R99 from the control selection unit 9 is set to Enable, the metadata analysis unit 34 analyzes the reproduction metadata R33 and outputs a reproduction scene control scenario R34 associated with the time code.

  The image selection unit 3 selects the reproduction image signal R13 and outputs a selection image signal R3. The image analysis unit 5 analyzes the selected image signal R3 for each frame, and sequentially outputs the feature amount R5 for each frame, as in recording. The picture making control unit 6 that has received the sequential feature quantity R5 outputs a sequential control quantity R6 for performing the adaptive picture making by controlling the image processing unit 4 for each frame. Details of the processing of the picture making control unit 6 will be described later. The control selection unit 9 selects either the sequential control amount R6 or the reproduction scene control scenario R34 and outputs the selected control amount R9. Details of the processing of the control selection unit 9 will be described later. The image processing unit 4 performs image processing on the selected image signal R3 according to the selection control amount R9 and outputs a display image signal R4. Details of the image processing in the image processing unit 4 will be described later. The image display unit 10 displays an image according to the display video signal R4.

  Each of the above processes is performed by a control process that refers to a time code. The time code generator 18 acquires a time code from the reproduced image signal R13, and distributes the acquired time code to each block in the system as a time code R18 (not shown). The control unit 19 controls the operation of each block of the system through a control line (not shown).

Next, details of the sequential adaptive picture making in the picture making control unit 6 will be described. A block diagram of the picture making control unit 6 is shown in FIG. The picture making control unit 6 uses only the luminance histogram R52 in the sequential feature amount R5.
The sequential normalization unit 61 normalizes the luminance histogram R52 and outputs a sequential normalized histogram R61. Details of the normalization processing are the same as those of the histogram normalization unit 81.
The sequential histogram integration unit 62 generates a sequential control amount R6 that is a luminance correction curve for performing gradation correction by cumulatively adding the sequential normalized histogram R61 in order from the lower category. Details of the generation method are the same as those of the histogram integration unit 82.

Next, details of the operation of the selection control unit 9 will be described. A block diagram of the selector 9 is shown in FIG.
The time code comparison unit 91 compares the time code R18 and the section time code R93, and outputs a curve selection signal R91 and a scenario update signal R92. When the time code R18 of the image to be currently processed is NTC, the time point of the section time code R93 is STP, and the duration of the section time code R93 is SDR, it is determined whether the following equation holds.
STP ≤ NTC <STP + SDR
If established, the curve selection signal R91 is set to Enable. If not established, the curve selection signal R91 is set to Disable.

The time code comparison unit 93 also determines whether the following equation is established.
NTC ≧ STP + SDR
If established, the scenario update signal R92 is set to Enable. If not established, the scenario update signal R92 is set to Disable.

The scenario queue 93 is a queue for temporarily storing playback scene control scenarios. As the output of the queue, the section time code R93 including the TimePoint and Duration of the time section targeted by the scenario at the head of the queue and the scenario control value R94 that is the control value corresponding to the time section are always output. Further, when the scenario update signal R92 becomes Enable, one head of the queue is discarded. When the queue is empty, the scenario request signal R99 is enabled and the reproduction scene control scenario R34 is acquired from the metadata analysis unit 34 and stored at the rear end of the queue.

The correction curve restoration unit 95 restores the characteristics of the luminance correction curve based on the scenario control value R94, and outputs a corrected luminance correction curve R95 in which the inverse γ conversion curve is corrected. Details of the processing will be described below. A reference inverse γ conversion curve is selected according to the characteristics and processing mode of the entire processing system (FIG. 11A). Next, the two representative point positions Pa and Pb included in the scenario control value R94 are referred to (FIG. 11B). Next, the corrected representative points Na and Nb are calculated by adding the representative point differences Va and Vb from the reference curve at the representative point position (FIG. 11C). Next, a spline curve connecting the end point W (white point) from the origin B (black point) through the corrected representative points Na and Nb is calculated as a corrected luminance correction curve R95 (FIG. 11 (d)).

  The curve selection unit 96 selects either the corrected luminance correction curve R95 or the sequential control amount R6 based on the curve selection signal R91 and outputs it as the selected control amount R9. As a result, when a control scenario exists for an image section, image processing according to the control scenario is performed, and when there is no control scenario, adaptive image processing is sequentially performed.

  Next, details of image processing in the image processing unit 4 will be described. The image processing unit 4 is a lookup table composed of a RAM. The selection control amount R9 is connected to the RAM write port, and the content of the selection control amount R9 is written in the blanking period between frames of the selection image signal R3. The selection image signal R3 is connected to the read address port of the RAM, and refers to the selection control amount R9 written in the table. A value converted by referring to the table is output from the read data port of the RAM, and is output as a display image signal R4.

FIG. 12 shows a conceptual diagram of the operation of the system at the time of reproduction described above.
Prior to image reproduction, the scenario queue 93 is empty, so that the scenario request signal R92 is generated as many times as necessary to fill the queue.
Consider the selected image signal R3 indicated by the time code from TimePoint Ta to Duration Da. At this time, in the Shot-A video section, the condition matches in the comparison with the section time code R93 output from the scenario queue 93, so the curve selection signal R91 becomes Enable. Based on the curve selection signal R91, Va based on the reproduction scene control scenario R34 is selected as the selection control amount R9.

When the shot-A video section ends, a scenario update signal R92 is generated and the scenario Va at the head of the scenario queue 93 is discarded. Further, subsequently, a scenario request signal R99 is generated due to the occurrence of an empty queue, and the next control scenario Vd is replenished to the queue.
In the video section in which the selected image signal R3 is Shot-B, the condition does not match in the comparison with the section time code R93, so the curve selection signal R91 is disabled. Based on the curve selection signal R91, VX based on the sequential control amount R6 is selected as the selection control amount R9 in the Shot-B video section.
Thereafter, Vc based on the playback scene control scenario R34 is selected in the same manner in the Shot-C video section.

  With the configuration and control described above, it is possible to simultaneously record the scenario control amount converted to metadata at the time of image recording and read it at the time of reproduction to perform image processing. In the image processing at the time of reproduction with this configuration, metadata corresponding to the image section to be reproduced is read in advance, and an optimized image processing scenario is prepared in advance. As a result, it is possible to apply the picture optimization control optimized for the section from the beginning of the section without prefetching the image itself.

For example, when the camera pans from left to right in a dim room as shown in FIG. 13, image signals as shown in FIGS. 14 (a) to (c) are obtained. The luminance histograms R52 at the respective moments are obtained as shown in FIGS. 15 (a) to 15 (c).
When a conventionally known brightness correction curve generation is performed on this image signal, correction curves as shown in FIGS. 16A to 16C are obtained. As shown in FIGS. 17 (a) to 17 (c), the image subjected to this correction sometimes becomes unsightly because the brightness greatly fluctuates within the same scene.

  However, according to the apparatus configuration of the present embodiment, it is possible to recognize FIGS. 14 (a) to 14 (c) as a single image section and obtain an optimum correction curve for all the frames in the scene. Specifically, a peak hold histogram R72 as shown in FIG. 18 is obtained, and a luminance correction curve as shown in FIG. 19 is obtained as the corresponding scene control scenario R8 in FIGS. Since a single correction curve is applied to the section image from which the scene control scenario R8 is obtained, the luminance fluctuation of the entire video does not occur during panning, and FIGS. 20 (a) to (c). A stable image processing result as shown in FIG.

  As described above, it has been shown that the image processing result higher than that of the conventionally known image processing method can be obtained by the configuration and control of the first embodiment to which the present invention is applied. At this time, the scene control scenario R8 is output in the scene where the scene feature amount R7 is output, and the generated metadata R30 based on the scene control scenario R8 is recorded during recording. As a result, the metadata is reduced as compared with the case where the metadata is recorded for each frame.

  Further, even the system in which trick play such as fast-forwarding or rewinding may occur during image reproduction can be handled by the configuration of this embodiment.

The control selection unit 9 discards all control scenarios held in the scenario queue 93 by notification from the control unit 19 when trick play occurs.
The image playback unit 33 stops reading metadata during trick play. When trick play ends and the normal playback mode is set, the metadata read pointer is returned to the beginning of the metadata file, and the metadata is read sequentially from the beginning.

In this state, the following conditions are continuously satisfied in the determination by the time code comparison unit 91.
NTC ≧ STP + SDR
Therefore, the metadata is skipped continuously until it catches up with the time when the image is being reproduced. Since the amount of metadata data is very small compared to the image stream, it is possible to synchronize the read pointers in such a short time that the user cannot recognize even if rereading from the beginning of the file.

  Alternatively, the time code and the metadata read pointer may be synchronized using some search technique.

  By performing control in this way, this embodiment can be applied to a system in which trick play may occur.

  Also, by replacing specific examples of the components of the image processing apparatus shown in the present embodiment, various image processing effects can be obtained.

  For example, the section for dividing the scene control scenario can be classified according to the use of the system by replacing the detection method of the scene change detection unit 71. For example, as a method for detecting a break, a change in the luminance histogram R52 for each frame may be detected, or may be detected from a change in audio data accompanying the image. In addition to the shot change, the category to be detected may be a program change or a CM detection. In the case of simplifying the system, it is also possible to classify at a fixed time using a timer.

  In addition to the example shown in the present embodiment, the scene feature amount R7 calculated by the scene feature analysis unit 7 can be calculated using an average value in the scene or excluding singular values. .

  Further, in this embodiment, after generating the brightness correction curve R82 from the histogram of the scene feature amount R7, the difference from the standard brightness correction curve is extracted at the representative point to create a scene control scenario R8, which is converted into metadata and stored. ing. However, the luminance correction curve before being converted to metadata can be stored together with the input image and used for correction processing during image reproduction. Alternatively, scene feature amounts can be accumulated, and a correction process can be performed by obtaining a luminance correction curve during reproduction. Even in this case, the storage capacity can be reduced as compared with the case where a histogram is created and stored for each frame.

  In addition to the example shown in the present embodiment, the luminance correction curve R82 calculated by the scene picture creation analysis unit 8 can use a method such as pattern classification based on the shape of a histogram.

  Further, the scene control scenario R8 calculated by the scene picture creation analysis unit 8 can use the constituent numerical value of the histogram itself in addition to the example of conversion into the secondary format as shown in the present embodiment.

  In addition, the generated metadata R30 generated by the metadata generating unit 30 may be in a unique text format or in a binary format in addition to the example shown in the present embodiment.

<Example 2>
In the first embodiment, the sequential feature amount R5 detected by the image analysis unit 5 for image processing is a luminance histogram. However, the present invention may be applied to a system that performs image processing using other feature amounts. It is possible to apply.
In the second embodiment, an apparatus that performs image processing using a color histogram will be described. The overall configuration diagram of the system of the present embodiment is the same as the overall configuration diagram of the first embodiment shown in FIG. 1, and the functions of some components are different.

  In this embodiment, the image analysis unit 5 performs color analysis of the selected image signal R3 and generates a saturation histogram. The image processing unit 4 changes the saturation of the selected image signal R3 according to the selection control amount R9. In addition, if the overall saturation of the analyzed image is low, the picture making control unit 6 outputs a sequential control amount R6 that performs control to increase the saturation to such an extent that color saturation does not occur.

Further, the scene feature analysis unit 7 generates a peak hold histogram of the saturation histogram for the section image in the same manner as in the first embodiment. In addition, the scene picture creation analysis unit 8 calculates the saturation control amount optimized for the saturation peak hold histogram as the control scenario R8.
Other components perform the same operation and control as in the first embodiment.

FIG. 21 shows an example in which the system of this embodiment is applied to an indoor scene. First, an interior with a low saturation as shown in FIG. 21 (a) is placed. Subsequently, as shown in FIG. 21 (b), a person wearing brightly colored clothes is displayed on the screen. I walked inside. Assume the scene.

In this case, in conventionally known image processing by sequential control, control is performed so as to increase the saturation of the entire screen at the time of FIG. Further, at the time of FIG. 21 (b), improvement in saturation is conservative so that the color of the clothes of the person is not saturated. This control changes the color tone of the background even in a single scene, resulting in a strange image processing. In addition, there is known a technique for suppressing a sudden change in control by giving a time constant to sequential control. In this case, control for improving saturation optimized at the time of FIG. This also applies to 21 (b). As a result, the color of the clothes of the person is saturated, and the image quality may be lowered.

However, according to the apparatus configuration of the second embodiment to which the present invention is applied, it is possible to perform control that optimizes FIGS. 21A to 21B as a continuous single scene. As a result, it is possible to control the saturation to be applied to the subsequent images from the time point shown in FIG.

  With the configuration described above, it is possible to perform advanced image processing that applies a control scenario optimized for a section image even for color saturation. Since the metadata at that time is generated based on the scene control scenario, the amount can be reduced as compared with the case where the metadata is generated for each frame.

<Example 3>
In the embodiment described above, the sequential feature amount R5 detected by the image analysis unit 5 for image processing is some sort of histogram, but it may be a system that performs image processing using other feature amounts. The invention can be applied.
In the third embodiment, an apparatus that performs image processing using a spatial frequency power spectrum will be described. The overall configuration diagram of the system of the present embodiment is the same as the overall configuration diagram of the first embodiment shown in FIG. 1, and the functions of some components are different.

In this embodiment, the image analysis unit 5 performs FFT analysis on the selected image signal R3 to generate a spatial frequency power spectrum. The image processing unit 4 changes the sharpness of the selected image signal R3 according to the selection control amount R9. Further, if the overall sharpness of the analyzed image is low, the picture making control unit 6 outputs a sequential control amount R6 for performing control to increase the sharpness so that no ringing occurs at the edge of the image.

In addition, the scene feature analysis unit 7 generates a peak hold spectrum of the spatial frequency power spectrum for the section image by a method almost the same as that of the first embodiment. In addition, the scene picture creation analysis unit 8 calculates a control amount of sharpness optimized as a peak hold spectrum of the spatial frequency power spectrum as a control scenario R8.
Other components perform the same operation and control as in the first embodiment.

FIG. 22 shows an example in which the system of this embodiment is applied to an outdoor scene. First, as shown in FIG. 22 (a), a landscape photographed with a soft focus as a whole is displayed, and subsequently, FIG. 22 (b).
As shown in, a large number of character telops scroll in the screen. Assume the scene.

In this case, in conventionally known image processing by sequential control, control is performed so as to increase the sharpness of the entire screen at the time of FIG. Further, at the time of FIG. 22 (b), sharpness improvement is conservative so as not to cause ringing in the character telop. This control changes the sharpness of the background even in a single scene, resulting in a strange image processing. Further, there is known a technique for suppressing a sudden change in control by giving a time constant to sequential control. In this case, control for improving the sharpness optimized at the time of FIG. This also applies to 22 (b). As a result, ringing does not occur around the character telop, and the image quality may be deteriorated.

However, according to the apparatus configuration of the third embodiment to which the present invention is applied, it is possible to perform control that optimizes FIGS. 22A to 22B as a continuous single scene. As a result, it is possible to control the sharpness that can be applied to subsequent images from the time point shown in FIG.

  With the configuration described above, it is possible to perform advanced image processing that applies a control scenario optimized for a section image with respect to image sharpness. Since the metadata at that time is generated based on the scene control scenario, the amount can be reduced as compared with the case where the metadata is generated for each frame.

<Example 4>
In the present embodiment, an example will be described in which the present invention is applied to a camcorder that uses an imaging device as an input unit and that captures and stores captured images. Also in the present embodiment, the present invention can be applied with a configuration substantially similar to that of the first embodiment.

  FIG. 23 shows an overall configuration diagram of an apparatus according to the fourth embodiment. The imaging unit 21 outputs a captured image R21 captured by the imaging element in the recording mode. The external output unit 22 converts the display image signal R4 into a video signal conforming to a standard such as HDMI, and outputs an external output signal R22.

  Other configurations and controls are the same as those in the first embodiment. That is, the image processing apparatus detects a scene change at the time of recording, and stores the scene control scenario as metadata. Thereby, the amount of metadata can be reduced. In playback, advanced image processing using a scene control scenario is possible. If it does in this way, it will become possible to apply the present invention also to a camcorder.

  4: image processing unit, 5: image analysis unit, 7: scene feature analysis unit, 10: image display unit, 32: storage device unit, 33: image reproduction unit

Claims (7)

An image analysis unit for analyzing the input image and calculating a sequential feature amount representing the feature of the image for each frame;
A scene detector for dividing the input image into a plurality of scenes;
A scene feature analysis unit that calculates a scene feature amount representing a feature of an image of the entire scene from a sequential feature amount of each frame included in the scene;
A control amount generating unit that generates a control amount to be used for correcting images of all frames included in the scene based on the scene feature amount;
An accumulation unit for accumulating the input image and the control amount;
An image processing apparatus comprising: an image processing unit configured to perform image processing on a scene of the accumulated input image using a control amount corresponding to the scene and output the image as a reproduction image. A difference information generation unit that generates difference information that is information representing a difference between a predetermined control amount when performing standard correction on the image of the frame and the control amount generated by the control amount generation unit;
The storage unit stores the difference information together with the input image instead of the control amount,
2. The image processing unit according to claim 1, wherein the image processing unit performs image processing by obtaining a control amount used in correction of a frame image from the difference information and a predetermined control amount when performing standard correction. Image processing device. The scene feature analysis unit determines whether a variation in luminance distribution of each frame included in the scene is larger than a predetermined value, calculates the scene feature amount only for a scene determined to be large,
The image processing unit performs image processing by obtaining a control amount based on a sequential feature amount of each frame in a scene in which the scene feature amount is not calculated. An image processing apparatus according to 1. The sequential feature amount is a luminance histogram calculated from a frame image,
The scene feature amount is a peak hold histogram in which the luminance histogram of each frame included in the scene is compared and the highest frequency in the scene is selected for each gradation value.
The control amount generated by the control amount generation unit is generated based on a histogram obtained by accumulating the peak hold histogram from the lower gradation value, and a luminance correction curve that associates the luminance of the input image with the luminance of the reproduced image. And
The image processing apparatus according to claim 1, wherein the image processing unit performs image processing using the luminance correction curve. The sequential feature amount is a luminance histogram calculated from a frame image,
The scene feature amount is a peak hold histogram in which the luminance histogram of each frame included in the scene is compared and the highest frequency in the scene is selected for each gradation value.
The control amount generated by the control amount generation unit is generated based on a histogram obtained by accumulating the peak hold histogram from the lower gradation value, and a luminance correction curve that associates the luminance of the input image with the luminance of the reproduced image. And
The predetermined control amount is a luminance correction curve when performing standard correction,
The difference information generation unit generates the difference information based on a difference in a predetermined gradation value between the luminance correction curve generated by the control amount generation unit and the luminance correction curve when the standard correction is performed. Is what
The image processing unit performs image processing by generating a luminance correction curve used for correcting a frame image from the difference information and a luminance correction curve when performing standard correction. The image processing apparatus described. The image processing unit performs image processing in a picture making mode selected from a plurality of picture making modes, and selects a luminance correction curve for performing the standard correction according to the picture making mode. The image processing apparatus according to claim 5. An image processing method executed by an image processing apparatus,
An image analysis step of analyzing the input image and calculating a sequential feature amount representing the feature of the image for each frame;
A scene detection step of dividing the input image into a plurality of scenes;
A scene feature analysis step of calculating a scene feature amount representing a feature of an image of the entire scene from a sequential feature amount of each frame included in the scene;
A control amount generating step for generating a control amount to be used for correcting images of all frames included in the scene based on the scene feature amount;
An accumulation step for accumulating the input image and the control amount;
An image processing method comprising: an image processing step of performing image processing on the accumulated scene of the input image using a control amount corresponding to the scene and outputting as a reproduced image.

Images