JP2010004261A - Image processing apparatus, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of quickly displaying a complicated area. <P>SOLUTION: The image processing apparatus includes: a display screen on which an image before encoding is displayed; a complicated area detection part for detecting a complicated pixel area from the image before encoding; a complicated area display part for displaying the complicated pixel area detected by the complicated area detection part, on the display screen; an operation input part for a user to designate an arbitrary area in the image as a designated area; and an image processing part for subjecting the designated area designated through the operation input part, to prescribed image processing. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method.

  In recent years, with the rapid development of video processing technology, imaging devices that can easily shoot high-quality digital video have become popular. Since such digital video data of high-quality video has an enormous amount of data, various devices have been devised to record it economically. Among them, as an encoding technique for compressing data capacity of high-quality video data while maintaining high image quality, for example, a compression encoding technique standardized by MPEG (Moving Picture Experts Group) or VCEG (Video Coding Experts Group) Is well known.

  Such an imaging apparatus is usually equipped with an imaging device that photoelectrically converts light incident through an optical system and outputs an electrical signal, and a digital video signal is generated from the electrical signal. Furthermore, for the above reason, the digital video signal is encoded by a predetermined encoding method by an encoder mounted on the imaging apparatus. At this time, the encoder executes encoding processing based on, for example, a compression encoding method standardized by the above-described MPEG or the like. The encoded digital video signal is recorded on a recording medium connected to the imaging device.

  However, compression encoding methods such as MPEG may cause a reduction in compression rate or compression noise when the pre-encoded image is complex or the image contains a subject that moves rapidly. In many cases, video quality after encoding is degraded. In addition, if the bit amount allocated per unit time is fixed, a large amount of code is allocated to an image including a complex pixel area, and the video quality as a whole may be degraded. Therefore, it is required to reduce an image including a complicated pixel region and a subject that moves rapidly.

  In response to such a request, Patent Document 1 below discloses a technique for displaying on a screen a coding rate that changes according to the complexity of an image, the intensity of movement of a subject, and the like. Furthermore, the document 1 discloses a technique for displaying a warning on a screen when a predetermined coding rate is exceeded. Patent Document 2 below discloses a technique in which a high importance level is set in an area that a user considers important in an image, and encoding is performed with a code amount corresponding to the importance level. Further, Patent Document 3 below discloses a technique for performing edge detection and complex area detection of an image, and automatically assigning a large amount of code to the edge area and complex area obtained as a result of the detection. It is disclosed.

JP 2003-250114 A JP-A-8-336135 JP-A-9-98421

  However, even if the coding rate is displayed on the screen as in Patent Document 1, it is difficult for the user to easily determine which part of the subject is a complex area. Therefore, it is difficult to adjust the angle or the like so as to avoid a complicated area unless it is a user having a skilled photographing technique. Further, when the user designates an important area in advance as in the above-mentioned Patent Document 2, a sufficient effect cannot be obtained in a shooting scene in which the complex area changes from moment to moment. Furthermore, setting the importance level is a heavy burden on the user.

  In addition, when encoding is performed by automatically allocating a code amount as described in Patent Document 3, a large amount of code amount is not always allocated to a region that is actually considered important by the user. The image quality of the area may be lowered. Further, in the technique of the same document 3, even if there is a complex area that can be avoided by adjusting the angle of the user, information on the complex area is not presented to the user. Therefore, when the imaging is continued while including the complex area and the encoding process is executed, a larger amount of code is consumed than an image in which the complex area is avoided.

  That is, even if the above techniques are used alone or in combination, it is difficult to present a complex pixel region included in the captured image to the user in real time. For this reason, the image quality of a region important for the user is deteriorated, or a complex region that can be avoided by a user operation is included in the image, and the code amount is relatively increased. Furthermore, if the configuration is such that information obtained after the encoding process is presented to the user as in Patent Document 1 described above, the encoding process takes time, and it becomes difficult for the user to take complex area avoidance actions in real time.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to present a complicated pixel area to a user earlier and to provide a predetermined pixel area designated by the user. It is an object of the present invention to provide a new and improved image processing apparatus and an image processing method capable of performing image processing.

  In order to solve the above problems, according to an aspect of the present invention, a display screen on which an image before encoding is displayed, a complex area detection unit that detects a complex pixel area from the image before encoding, and the complex A complex region display unit for displaying a complex pixel region detected by the region detection unit on the display screen, an operation input unit for a user to designate an arbitrary region in the image as a designated region, and the operation input unit. There is provided an image processing apparatus including an image processing unit that performs predetermined image processing on a designated area designated via the image processing unit.

  As described above, the image processing apparatus has a display screen on which an image before encoding is displayed, and the complex area detection unit can detect a complex pixel area from the image before encoding. it can. In the image processing apparatus, the complex area display unit displays the complex pixel area detected by the complex area detection unit on the display screen. The image processing apparatus includes an operation input unit for a user to specify an arbitrary area in the image as a specified area, and is further specified by the image processing unit via the operation input unit. It is possible to perform predetermined image processing on the designated area.

  Further, the image processing unit is an encoder that encodes the image, and a relatively large code amount or a relatively small code is specified in a designated area designated via the operation input unit when the image is coded. It may be configured to allocate and encode quantities.

  The image processing unit may be configured to perform a low-pass filter, a color reduction process, or a depth-of-field reduction process on a designated area designated via the operation input unit.

  The image processing apparatus may further include a designated area display unit that displays an outer edge of the designated area designated via the operation input unit on the display screen. In this case, the designated area display unit enlarges the size of the designated area according to the length of time that the operation input unit is operated by the user.

  The designated area display unit enlarges the designated area so that the outer edge of the designated area is aligned with the outer edge of the complex pixel area detected by the complex area detecting unit when the size of the designated area is enlarged. It may be configured as follows.

  The complex area detecting unit detects a pixel area having a large discrete cosine transform coefficient, wavelet coefficient, motion vector length, flatness, or intra AC detected from the pre-encoded image as the complex pixel area. It may be configured to.

  The image processing apparatus may further include an imaging unit that forms an image of a subject from light incident through the optical system. In this case, the imaging unit displays the image of the subject on the display screen, and inputs the image of the subject to the complex region detection unit as the image before encoding.

  In order to solve the above-described problem, according to another aspect of the present invention, a complex region detection step in which a complex pixel region is detected from an image before encoding, an image before encoding is displayed on a display screen, A complex region display step in which the complex pixel region detected in the complex region detection step is displayed on the display screen; and when an arbitrary region in the image is designated by the user as the designated region, at least the designated region There is provided an image processing method including an image processing step in which predetermined image processing is performed.

  As described above, according to the present invention, a complicated pixel region can be presented to the user earlier, and predetermined image processing can be performed on the pixel region designated by the user.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Structure of this article]
First, prior to describing the embodiment of the present invention in detail, the purpose of the embodiment will be briefly described. In this, the outline | summary of the technique which concerns on the same embodiment for achieving the said objective is demonstrated. Next, a specific configuration example of the apparatus and method according to the embodiment will be shown, and the functional configuration of the apparatus and the flow of processing according to the method will be described.

[the purpose]
As already mentioned, when an encoding method such as MPEG is used, if each video frame contains complex pixel areas, or if the motion between video frames is intense, the compression rate will decrease or compression noise will occur. Resulting in. Also, in this method, the amount of bits allocated per unit time is fixed, and when the upper limit is exceeded, the image quality is extremely deteriorated. For this reason, it is indispensable for improving the video quality to remove or reduce the complex area of the image constituting the video frame at the time of shooting, or to reduce the subject that moves rapidly as much as possible.

  If complex pixel areas and pixel areas with high motion are identified in real time and these pixel areas can be avoided during shooting, the encoded video quality can be improved even with the same bit rate setting. Furthermore, if these pixel areas are reduced, the encoded video data size can be reduced. That is, it is possible to generate high-quality and small-sized video data. An embodiment of the present invention to be described later is mainly intended to realize generation of such video data.

  In recent years, with the spread of high-speed communication networks, video content such as video distribution and video posting has been frequently exchanged even at the individual level. For example, moving image data is exchanged in real time via the Internet, such as moving image chat using a Web camera. The moving image content exchanged in such a case is desired to have a high image quality at a low bit rate. For example, in the case of a video chat or the like, the important part of the exchanged video is the central portion where the face is shown. In this case, if the surrounding area includes a complex area or a subject that moves rapidly is captured, the compression rate decreases and the video quality deteriorates.

  Similarly, when producing video content such as dramas and movies, for example, if the background area is not so important and the complex area is included, or if a subject with intense movement is reflected, the video quality will deteriorate. End up. In addition, if a video with a large amount of information is included in the background, the viewer will recognize it as a so-called noisy video. For example, when the leaves of trees reflected in the background are swayed by the wind, the pixel area of that portion becomes a complex pixel area and a pixel area where motion detection is difficult. Moreover, since it becomes a noisy image | video, the impression which a viewer receives from this video content may deteriorate.

  There are cases where such complicated video portions or video portions with intense movement can be easily avoided by adjusting the camera angle or narrowing the angle of view during shooting. In the first place, if such a video portion can be avoided at the time of shooting, the video quality after encoding will not deteriorate due to these video portions. However, unless it is a highly skilled user, it is difficult to accurately determine a complex video portion or a video portion with a lot of movement. Also, even complex video parts or video parts with high motion may be considered important by the user, and if an attempt is made to automatically determine and correct complex areas etc., the encoded video will be displayed by the user. There is a possibility of deviation from the intention.

  From such a viewpoint, the embodiment of the present invention to be described later proposes a technique for detecting a complex pixel region included in each video frame and a severe motion between the video frames and presenting the detection result to the user. In the embodiment, a technique is proposed in which the user can grasp the complex area in real time at the time of shooting and present the complex area to the user at a higher speed so that the complex area can be avoided. By using these technologies, it is possible to avoid in real time complex areas that the user has determined to be unimportant during shooting, reducing the amount of video data after encoding, and reducing the video quality of important parts. Can be improved. Hereinafter, a configuration example of the apparatus and method according to the embodiment will be described in detail.

<Embodiment>
Hereinafter, an embodiment of the present invention will be described. The present embodiment relates to a technique for detecting a complex pixel area from an image before encoding and displaying it on a display screen, and performing predetermined image processing on a designated area designated by a user based on the display content. Since a complex pixel area is detected from the image before encoding, the complex pixel area can be displayed in real time. By referring to the information displayed in this way, the user can avoid complicated pixel areas and the like while interactively adjusting camera work and composition.

[Functional Configuration of Imaging Device 100]
First, the functional configuration of the imaging apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram illustrating a functional configuration example of the imaging apparatus 100 according to the present embodiment. The technology of the imaging apparatus 100 is applied to, for example, a personal computer to which a camcorder or a Web camera is connected, a mobile phone with a camera, or the like. Note that the imaging apparatus 100 relates to a part of the techniques according to the present embodiment for displaying a complex pixel region at high speed.

  As shown in FIG. 1, the imaging apparatus 100 mainly includes an optical system 102, an image sensor 104, and an image processing block 130. The image sensor 104 is an example of an imaging unit. The image processing block 130 is an example of an image processing apparatus. The function of the image processing block 130 can be realized by the hardware configuration shown in FIG.

(Optical system 102, image sensor 104)
The optical system 102 is formed by one or a plurality of lenses for forming an image of a subject when the reflected light is incident on the subject. The light incident through the optical system 102 is input to the image sensor 104 as an optical signal.

  The image sensor 104 is formed of a photoelectric conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), for example. The image sensor 104 photoelectrically converts an optical signal input via the optical system 102 and outputs an electrical signal. The electrical signal output from the image sensor 104 is input to the image processing block 130 (image processing unit 132) as a digital image signal. When a moving image is captured, the subject image is continuously input to the image sensor 104, and digital image signals in units of pages (frame or field units) are continuously input to the image processing unit 132.

(Image processing block 130)
The image processing block 130 mainly includes an image processing unit 132, an encoder 134, an image storage unit 136, an image output control unit 138, and a display panel 140. The image output control unit 138 is an example of a complex region detection unit, a complex region display unit, and a designated region display unit. The display panel 140 is an example of a display screen.

(Image processing unit 132)
First, when a digital image signal is input from the image sensor 104, the image processing unit 132 performs image processing such as white balance adjustment and γ correction on the digital image signal. In addition, the image processing unit 132 applies a low-pass filter, red-eye correction, and the like to the digital image signal according to user settings. The digital image signal subjected to image processing by the image processing unit 132 is input to the encoder 134 and the image output control unit 138.

(Encoder 134)
When a digital image signal is input from the image processing unit 132, the encoder 134 performs an encoding process on the digital image signal by a predetermined encoding method. Examples of the predetermined encoding method include MPEG-2, MPEG-4, H.264, and the like. A method standardized by H.264 / AVC or the like is used. Of course, the encoding method applicable to this embodiment is not limited to these examples.

  Here, as an example, a functional configuration of the encoder 134 when using an encoding method defined as MPEG-2 will be briefly described.

  In MPEG-2, a motion compensation interframe differential encoding technique is used. First, the encoder 134 detects a motion vector from continuously input digital image signals. At this time, the encoder 134 performs motion detection using a block matching method or the like. Next, the encoder 134 differentially encodes the digital image signal while performing motion compensation on the digital image signal using the detected motion vector.

  Thereafter, the encoder 134 performs DCT on difference information that cannot be corrected by motion compensation. Further, the encoder 134 quantizes the difference information and the like subjected to DCT, and encodes it using a variable length code such as a Huffman code. In this way, the digital image signal is encoded by the encoder 134. The encoded digital image signal is recorded in the image storage unit 136 as an encoding result.

(Image output control unit 138)
Meanwhile, the digital image signal output from the image processing unit 132 is also input to the image output control unit 138 independently of the encoder 134. The image output control unit 138 calculates a code such as a DCT coefficient from the digital image signal before encoding, and evaluates the complexity of each pixel area included in the digital image signal using this code. The evaluation result by the image output control unit 138 is displayed on the display panel 140. At this time, the image output control unit 138 displays the digital image signal together with the evaluation result. This display method will be described in detail later.

  Here, a detailed functional configuration of the image output control unit 138 will be described with reference to FIG. FIG. 2 is an explanatory diagram showing a detailed functional configuration of the image output control unit 138.

  As shown in FIG. 2, the image output control unit 138 mainly includes an image signal evaluation unit 152, an evaluation result visualization unit 154, and an image superposition control unit 156. Note that digital image signals are input to the image signal evaluation unit 152 and the image superimposition control unit 156.

(Image signal evaluation unit 152)
The image signal evaluation unit 152 evaluates a spatial image characteristic or a temporal image characteristic of each pixel area included in the digital image signal. That is, the image signal evaluation unit 152 evaluates the complexity of each pixel area of the digital image signal, the intensity of movement between the digital image signals, and the like. As the pixel region, a pixel block composed of one pixel or a plurality of pixels is used as one unit.

  For example, the image signal evaluation unit 152 performs DCT (Discrete Cosine Transform) on the digital image signal to calculate a DCT coefficient. Here, for convenience of explanation, the configuration in which the DCT coefficient is calculated by the image signal evaluation unit 152 has been described. However, for example, parameters such as a wavelet coefficient, flatness, or intra AC are calculated instead of the DCT coefficient. It may be configured as follows. Further, the image signal evaluation unit 152 may be configured to detect a motion vector using a plurality of digital image signals. These settings are preferably determined so that the encoder 134 matches the corresponding encoding scheme.

  By the way, the above flatness is data representing the spatial flatness of an image. The flatness becomes smaller as the design of the image is spatially complex, and becomes larger as it is flatter. On the other hand, the intra AC is obtained by calculating the difference of each pixel value from the average value of the macroblock and accumulating it in units of images. Therefore, the intra AC substantially corresponds to the generated code amount in the intra-frame encoding process. Therefore, the intra AC can be used as an index indicating the complexity of the picture pattern and the size of the compressed data amount.

  Next, the image signal evaluation unit 152 evaluates the digital image signal using these parameters. When evaluating the spatial image characteristics of each pixel region, the image signal evaluation unit 152 uses parameters calculated by frequency analysis such as DCT coefficients and wavelet coefficients. In this case, when the parameter corresponding to the high frequency component of a certain pixel region has a large value, the image signal evaluation unit 152 evaluates that the pixel region is a complex pixel region. Conversely, when the parameter corresponding to the high-frequency component of a certain pixel area has a small value, the image signal evaluation unit 152 evaluates that the pixel area is not a complex area.

  On the other hand, when evaluating temporal image characteristics of each pixel region, the image signal evaluation unit 152 uses a motion vector obtained by motion detection between digital image signals. In this case, the image signal evaluation unit 152 refers to the magnitude of a motion vector indicating movement of a certain pixel area as a parameter between digital image signals, and when the parameter has a large value, the pixel area has a strong motion. Evaluated to be a pixel area. On the other hand, when the magnitude of the motion vector of a certain pixel area has a small value between digital image signals, the image signal evaluation unit 152 evaluates that the pixel area is not a pixel area with intense motion.

  In addition, the image signal evaluation unit 152 may calculate the size of the motion vector for each pixel area included in the digital image signal, and calculate an average value in the entire image or an area of a predetermined size, for example. With this configuration, it is possible to evaluate the intensity of movement in the entire image or a region of a predetermined size, and it is possible to capture the intensity of movement in each video scene. Since the technique of this embodiment is a technique for presenting a complex area or the like to the user, it is important that the user can easily recognize such an area.

  The evaluation result obtained by the image signal evaluation unit 152 in this way is input to the evaluation result visualization unit 154.

(Evaluation result visualization unit 154)
The evaluation result visualization unit 154 reconfigures the evaluation result obtained by the image signal evaluation unit 152 into a predetermined display format.

  For example, the evaluation result visualization unit 154 assigns a predetermined color or pattern to the complex area of the digital image signal based on the evaluation result. At this time, the evaluation result visualization unit 154 assigns a dark hue or a fine pattern to a pixel region having a large complexity or intense motion, for example. Further, the evaluation result visualization unit 154 specifies a region where a parameter indicating complexity, intensity of movement, or the like exceeds a predetermined value, and sets the outer edge of the region to be drawn or drawn with a predetermined color.

  Further, the evaluation result visualization unit 154 graphs the evaluation result of the entire digital image signal based on the size of the parameter value indicating complexity, intensity of movement, or the like, or displays the parameter value as a numerical value. Or set. Furthermore, the evaluation result visualization unit 154 may be configured to limit an area where a complex area or the like is displayed so as to exclude a predetermined preset area. In this case, a predetermined color or pattern indicating a complex area or the like is not displayed in the preset area.

  It is preferable that the preset area is configured to be arbitrarily set by the user. Even if the area that the user considers important is complicated or moves violently, it is preferable to process by assigning a larger amount of code. Therefore, it is not necessary to display a complex area or the like in an area that the user considers important. As described above, since the complex area or the like is not displayed in the preset area, the user only has to pay attention to the displayed complex area or the like, so that troublesomeness is reduced and the user can concentrate on the shooting operation.

  The display format information assigned or set by the evaluation result visualization unit 154 in this manner is input to the image superimposition control unit 156 as a visualization result.

(Image superposition control unit 156)
The image superposition control unit 156 displays the digital image signal on the display panel 140 and superimposes the evaluation result obtained by the image signal evaluation unit 152 on the digital image signal. At this time, the image superposition control unit 156 displays the evaluation result on the display panel 140 according to the display format of the visualization result set by the evaluation result visualization unit 154. For example, when displaying a pattern indicating the outer edge of the complex area corresponding to the evaluation result, the image superimposing control unit 156 superimposes and synthesizes the digital image signal and the pattern while adjusting the positional relationship with each other. Then, the superimposed and synthesized image is displayed on the display panel 140.

  The detailed functional configuration of the image output control unit 138 has been described above. As described above, the image output control unit 138 has a function of evaluating the complexity or the like of the digital image signal and displaying the evaluation result on the display panel 140. In the above description, regarding the evaluation method, various methods using DCT coefficients, motion vectors, and the like can be applied, and a specific evaluation method when using these methods has been described. Of these, a more detailed and specific description of the method using the DCT coefficient is desired with reference to FIG.

(Evaluation method using DCT coefficient)
Here, regarding the evaluation method using the DCT coefficient, a detailed functional configuration of the image signal evaluation unit 152 will be described with reference to FIG. In this, the calculation method of a DCT coefficient is demonstrated easily, referring FIGS. 4-6. Further, a specific example of visualization of the evaluation result by the evaluation result visualization unit 154 will be described with reference to FIG. Then, with reference to FIG. 8, the result of superimposing and combining the evaluation results by the image superimposing control unit 156 will be described.

  FIG. 3 is an explanatory diagram showing a detailed functional configuration of the image signal evaluation unit 152. 4 to 6 are explanatory diagrams for explaining a DCT coefficient calculation method. FIG. 7 is an explanatory diagram showing an example of a specific evaluation result visualization performed by the evaluation result visualization unit 154. FIG. 8 is an explanatory diagram illustrating an example of the result of superimposing and combining the evaluation results by the image superimposing control unit 156.

  As shown in FIG. 3, the image signal evaluation unit 152 includes an image region division unit 172, a frequency analysis unit 174, and an analysis result digitization unit 176.

  First, a digital image signal is input to the image region dividing unit 172. The image area dividing unit 172 divides the digital image signal into pixel areas of a predetermined size as shown in FIG. In the example of FIG. 4, the digital image signal is divided into pixel areas of 8 vertical pixels × 8 horizontal pixels. Of course, the predetermined size is not limited to 8 vertical pixels × 8 horizontal pixels, and may be a unit of one pixel or a pixel region of 9 or more vertical and horizontal pixels. That is, the predetermined size can be set arbitrarily.

  Information on the pixel area divided by the image area dividing unit 172 is input to the frequency analyzing unit 174. The frequency analysis unit 174 performs DCT on each pixel region. First, as shown in FIG. 5, the frequency analysis unit 174 performs discrete cosine transformation on the pixel value of each pixel included in each pixel region to calculate a frequency versus intensity (DCT coefficient) distribution in pixel region (block) units. .

  Although DCT is used in this example, the same processing is performed even when another frequency analysis method is used. By these processes, a spatial frequency characteristic for each divided pixel region is obtained. As already described, the spatial frequency characteristics indicated by parameters such as DCT coefficients indicate the complexity of the pixel region.

  The DCT coefficient obtained by the frequency analysis unit 174 is input to the analysis result digitizing unit 176. The analysis result digitizing unit 176 classifies the DCT coefficients for each pixel region into a predetermined number of stages and converts the DCT coefficients into numerical values representing the degree of complexity. For example, the analysis result digitizing unit 176 divides the DCT coefficient into five stages of evaluation values based on four preset threshold values.

  At this time, the analysis result digitizing unit 176 determines which of the sections divided by the four thresholds is the total intensity value of the DCT coefficients equal to or higher than the predetermined frequency. Based on the determination result, as shown in FIG. 6, the evaluation value assigned to each section is associated with the pixel area to be evaluated. The evaluation value obtained by the analysis result digitizing unit 176 in this way is input to the evaluation result visualization unit 154 as an evaluation result.

  As shown in the example of FIG. 7, the evaluation result visualization unit 154 determines the shade of the pattern displayed in each pixel area according to the evaluation value. For example, a pixel region having a large evaluation value is displayed with a dark pattern. Conversely, a pixel region having a small evaluation value is displayed in a light pattern.

  In addition to this, as a method of visualization expression, for example, it is conceivable to add colors and numbers, or add icons to display. A method of providing an arbitrary threshold value and displaying only a pixel region having an evaluation value equal to or higher than the threshold value is also conceivable. Further, a method of grouping a plurality of pixel areas having similar evaluation values and displaying the boundary line of each group is also conceivable. A method of displaying a histogram or various graphs corresponding to the evaluation value is also conceivable.

  Further, a method of displaying a combination of these visualization methods is also conceivable. Furthermore, a method of predicting an encoded image or a portion where compression noise is generated from an evaluation result and displaying a predicted image may be considered. As described above, the information calculated by the evaluation result visualization unit 154 is display information of the evaluation result. That is, the evaluation result visualization unit 154 generates a display layer on which a pattern indicating the evaluation result is displayed. The visualization result is input to the image superposition control unit 156.

  As shown in FIG. 8, the image superimposing control unit 156 superimposes and combines the original digital image signal and the evaluation result display layer generated by the evaluation result visualizing unit 154. At this time, the image superimposition control unit 156 performs superposition and synthesis after adjusting the position as necessary so that the original digital image signal and the evaluation result display layer can be spatially consistent. Then, an image obtained by superposition and synthesis is displayed on the display panel 140.

  As shown in FIG. 8, in the superimposed image, a dark pattern is displayed in a complex pixel area of the original image, and a light pattern is displayed in an uncomplicated area. As described above, the complex area is expressed by the shading of the pattern, so that the user can easily determine the complex area. When a display form such as a histogram or graph is set by the evaluation result visualization unit 154, the histogram or graph is displayed so as to be superimposed on the original image.

  In the above, a series of flow was demonstrated regarding the evaluation method using a DCT coefficient. Here, the case where DCT coefficients are used has been described as an example, but a series of these processes is the same except for the calculation process of DCT coefficients even when wavelet coefficients or other frequency analysis methods are used. In the case where the magnitude of the motion vector is used as the evaluation value, the same method is used in calculation of the evaluation value, visualization, superimposition processing, and the like. As the evaluation value, code data such as a DCT coefficient is not used as it is, but for example, an encoding difficulty level, a compression noise generation rate, or the like may be used.

(Display panel 140)
Refer to FIG. 1 again. The digital image signal on which the evaluation result is superimposed as described above is displayed on the display panel 140. The display panel 140 may be a display monitor built in the imaging apparatus 100 or a display monitor connected to the imaging apparatus 100 as an external device. Further, the display panel 140 may have an input device function such as a touch panel. In this case, the display panel 140 may be a touch panel with a pressure detection function having a function of detecting a pressing force when the user presses the screen. Further, the display panel 140 may have a function of three-dimensionally detecting the position on the screen that the user touches or approaches without directly touching the screen.

  The functional configuration of the imaging device 100 according to the present embodiment has been described above. As described above, since the image output control unit 138 is configured independently of the encoder 134, it is possible to evaluate a complex area and display the evaluation result regardless of the processing time and the encoding result for the encoding process. Become. As a result, the processing related to the display of the complex area and the like is executed at high speed, and the complex area and the like can be presented to the user in real time.

  In the above description, the imaging device 100 has been described as a single device, but the present embodiment is not limited to this. For example, the image processing block 130 may be realized by an externally connected information processing apparatus. The image storage unit 136 may be realized by an externally connected storage device or an external storage device connected via a communication device (not shown).

  As described above, the functions of the image processing block 130 configuring the imaging apparatus 100 can be realized by the hardware configuration illustrated in FIG. For example, the functions of the image processing unit 132, the encoder 134, and the image output control unit 138 are realized by the CPU 902 based on a program recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. The function of the image storage unit 136 is realized by the RAM 906, the storage unit 920, or the removable recording medium 928. Further, the function of the display panel 140 is realized by the output unit 918. However, when the display panel 140 has a function as input means such as a touch panel, it corresponds to the input unit 916.

[Functional Configuration of Imaging Device 200]
Next, a functional configuration of the imaging apparatus 200 according to the present embodiment will be described with reference to FIG. FIG. 9 is an explanatory diagram illustrating a functional configuration example of the imaging apparatus 200 according to the present embodiment. The technology related to the imaging apparatus 200 is applied to, for example, a personal computer to which a camcorder or a Web camera is connected, a mobile phone with a camera, or the like. Note that the imaging apparatus 200 relates to a technique for improving the image quality of an encoding result by using a digital image signal evaluation result for bit allocation prediction at the time of encoding, among the techniques according to the present embodiment.

  As shown in FIG. 9, the imaging apparatus 200 mainly includes an optical system 202, an image sensor 204, and an image processing block 230. The image sensor 204 is an example of an imaging unit. The image processing block 230 is an example of an image processing device. The function of the image processing block 230 can be realized by the hardware configuration shown in FIG.

(Optical system 202, image sensor 204)
The optical system 202 is formed by one or a plurality of lenses for forming an image of a subject when the reflected light is incident on the subject. Light incident through the optical system 202 is input to the image sensor 204 as an optical signal.

  The image sensor 204 is formed by a photoelectric conversion element such as a CCD or a CMOS, for example. The image sensor 204 photoelectrically converts an optical signal input via the optical system 202 and outputs an electrical signal. The electrical signal output from the image sensor 204 is input to the image processing block 230 (image processing unit 232) as a digital image signal. Note that when moving images are captured, subject images are continuously input to the image sensor 204, and digital image signals in units of pages are continuously input to the image processing unit 232.

(Image processing block 230)
The image processing block 230 mainly includes an image processing unit 232, an encoder 234, an image storage unit 236, an image output control unit 238, an assigned bit change unit 239, and a display panel 240. The image output control unit 238 is an example of a complex region detection unit, a complex region display unit, and a designated region display unit. The display panel 240 is an example of a display screen.

(Image processing unit 232)
First, when a digital image signal is input from the image sensor 204, the image processing unit 232 performs image processing such as white balance adjustment and γ correction on the digital image signal. The image processing unit 232 performs a low-pass filter, red-eye correction, and the like on the digital image signal according to user settings. The digital image signal that has been subjected to image processing by the image processing unit 232 is input to the encoder 234 and the image output control unit 238.

(Encoder 234)
When a digital image signal is input from the image processing unit 232, the encoder 234 performs an encoding process on the digital image signal by a predetermined encoding method. Examples of the predetermined encoding method include MPEG-2, MPEG-4, H.264, and the like. A method standardized by H.264 / AVC or the like is used. Of course, the encoding method applicable to this embodiment is not limited to these examples. The digital image signal encoded by the encoder 234 is recorded in the image storage unit 236 as an encoding result.

(Image output control unit 238)
The digital image signal output from the image processing unit 232 is also input to the image output control unit 238 provided independently of the encoder 234. The image output control unit 238 calculates a code such as a DCT coefficient from the digital image signal before encoding, and evaluates the complexity of each pixel area included in the digital image signal using this code. However, instead of the DCT coefficient, for example, a parameter such as a wavelet coefficient, flatness, or intra AC may be calculated, and the evaluation result may be calculated using the parameter. In addition, the image output control unit 238 may be configured to detect a motion vector using a plurality of digital image signals and calculate an evaluation result based on the magnitude of the motion vector. The evaluation result calculated by the image output control unit 238 is input to the allocation bit changing unit 239.

(Assignment bit changing unit 239)
The assigned bit changing unit 239 changes the assigned bit amount of the pixel area designated by the user, and inputs the changed assigned bit amount to the encoder 234. In the encoder 234, the designated area is encoded with the allocated bit amount changed by the allocated bit changing unit 239.

  For example, when the user designates an important pixel area, the assigned bit changing unit 239 increases the assigned bit amount of the designated area. With this configuration, when the bit amount allocated to the entire digital image signal is fixed, a high-quality encoded image that more suits the user's preference is generated.

  Conversely, when the user designates a complex area, the assigned bit changing unit 239 reduces the amount of assigned bits in the designated area. With such a configuration, when the bit amount to be assigned to the entire digital image signal is fixed, a large bit amount is assigned to the pixel region other than the complex region, and the image quality of the digital image signal can be improved. Of course, whether to increase or decrease the allocated bit amount can be appropriately changed according to the application and purpose.

  Here, the detailed functional configuration of the allocation bit changing unit 239 will be described with reference to FIG. 10, and a series of processing flows by the functional configuration will be described. FIG. 10 is an explanatory diagram showing a detailed functional configuration of the allocation bit changing unit 239.

  As shown in FIG. 10, the allocated bit changing unit 239 includes an area selecting unit 252, an allocated bit amount changing unit 254, and an operation result output unit 256.

  First, a digital image signal on which an evaluation result is superimposed is input from the image output control unit 238 to the region selection unit 252. The superimposed digital image signal is also input to the display panel 240 and displayed. The user designates a desired pixel region with reference to a composite image of the evaluation result displayed on the display panel 240 and the digital image signal. At this time, the user designates, for example, a pixel region displayed as a complex region or a region with high movement based on the evaluation result. When the pixel area is designated by the user, the area selecting unit 252 inputs information on the designated area to the allocated bit amount changing unit 254.

  The allocated bit amount changing unit 254 sets the allocated bit amount of the designated area and inputs it to the operation result output unit 256. At this time, the bit amount allocated to the designated area is designated by the user, or is decreased or increased by a predetermined amount. When designated by the user, a bit amount to be assigned is set based on the number of times the touch panel is pressed, the pressing force, or a numerical value input by a keyboard or the like. Of course, the present embodiment is not limited to this, and the bit amount desired by the user may be allocated by an arbitrary method.

  The allocated bit amount of the designated area set by the allocated bit amount changing unit 254 is input to the encoder 234 via the operation result output unit 256. At this time, the position information of the designated area is also input to the encoder 234 together with the allocated bit amount of the designated area. As a result, the encoder 234 can perform the encoding process with the assigned bit amount designated for the designated area.

(Display panel 240)
By the way, the display panel 240 may be a display monitor or a finder mounted on the imaging apparatus 200 or a display monitor connected to the imaging apparatus 200 as an external device. Further, the display panel 240 may have an input device function such as a touch panel. In this case, the display panel 240 may be a touch panel with a pressure detection function having a function of detecting a pressing force when the user presses the screen. In addition, the display panel 240 may have a function of three-dimensionally detecting the position on the screen that the user has touched or approached without directly touching the screen.

  The functional configuration of the imaging apparatus 200 has been described above. As described above, the imaging apparatus 200 includes the allocated bit changing unit 239, so that the user can allocate an arbitrary bit amount to the designated area. Therefore, when the user designates a complex area or the like displayed on the display panel 240 and reduces the allocated bit amount, a larger bit amount is allocated in addition to the designated area, and the image quality is improved. Conversely, by specifying a pixel area that is considered to be important by the user and assigning a large amount of bits, the user can select the important pixel area without increasing the data amount of the entire image. Image quality can be improved.

  As described above, the function of the image processing block 230 constituting the imaging device 200 can be realized by the hardware configuration shown in FIG. For example, the functions of the image processing unit 232, the encoder 234, the image output control unit 238, and the assigned bit change unit 239 are realized by the CPU 902 based on a program recorded in the ROM 904, RAM 906, storage unit 920, or removable recording medium 928. The

  The function of the image storage unit 236 is realized by the RAM 906, the storage unit 920, or the removable recording medium 928. Further, the function of the display panel 240 is realized by the output unit 918. However, when the display panel 240 has a function as input means such as a touch panel, the display panel 240 also corresponds to the input unit 916. Also, user input to the assigned bit changing unit 239 is realized by the function of the input unit 916.

[Functional Configuration of Image Processing Device 330]
Next, a functional configuration of the image processing apparatus 330 according to the present embodiment will be described with reference to FIG. FIG. 11 is an explanatory diagram illustrating a functional configuration example of the image processing apparatus 330 according to the present embodiment. The image processing device 330 is obtained by applying the technique of the imaging device 100 to an encoding result evaluation process.

  As shown in FIG. 11, the image processing device 330 mainly includes an image processing unit 332, an encoder 334, an image storage unit 336, a decoder 337, an image output control unit 338, and a display panel 340. The The functions of the image processing apparatus 330 can be realized by the hardware configuration shown in FIG. The image processing unit 332 is an example of a complex area detection unit. The image output control unit 338 is an example of a complex area display unit and a designated area display unit. The display panel 340 is an example of a display screen.

(Image processing unit 332)
First, when a digital image signal is input, the image processing unit 332 performs image processing such as white balance adjustment and γ correction on the digital image signal. In addition, the image processing unit 332 performs a low-pass filter, red-eye correction, and the like on the digital image signal according to user settings. The digital image signal that has been subjected to image processing by the image processing unit 332 is input to the encoder 334.

(Encoder 334)
When a digital image signal is input from the image processing unit 332, the encoder 334 performs an encoding process on the digital image signal by a predetermined encoding method. Examples of the predetermined encoding method include MPEG-2, MPEG-4, H.264, and the like. A method standardized by H.264 / AVC or the like is used. Of course, the encoding method applicable to this embodiment is not limited to these examples. The digital image signal encoded by the encoder 334 is input to the decoder 337 as an encoding result and recorded in the image storage unit 336. Further, the encoding information used for the encoding process is input to the image output control unit 338.

(Decoder 337)
The decoder 337 decodes the encoded digital image signal sequence input from the encoder 334 and restores the pre-encoded digital image signal sequence again. At this time, the decoder 337 decodes the digital image signal based on the same encoding method as the encoder 334. The digital image signal decoded by the decoder 337 is input to the image output control unit 338.

(Image output control unit 338)
The image output control unit 338 evaluates the complexity of each pixel region using a code such as a DCT coefficient obtained from the digital image signal input from the decoder 337. However, instead of the DCT coefficient, for example, the evaluation result may be calculated based on parameters such as a wavelet coefficient, flatness, or intra AC. In addition, the image output control unit 338 may be configured to detect a motion vector using a plurality of digital image signals and calculate an evaluation result based on the magnitude of the motion vector.

  That is, the digital image signal sequence encoded by the encoder 334 is evaluated. At this time, the image output control unit 338 generates evaluation information in consideration of the encoded information input from the encoder 334. The evaluation information generated by the image output control unit 338 is superimposed on the digital image signal and displayed on the display panel 340.

(Display panel 340)
The display panel 340 may be a display monitor mounted on the image processing apparatus 330 or a display monitor connected to the image processing apparatus 330 as an external device. Further, the display panel 340 may have an input device function such as a touch panel. In this case, the display panel 340 may be a touch panel with a pressure detection function having a function of detecting a pressing force when the user presses the screen. In addition, the display panel 340 may have a function of three-dimensionally detecting the position on the screen that the user has touched or approached without directly touching the screen.

  The functional configuration of the image processing apparatus 330 has been described above. As described above, the image processing apparatus 330 can evaluate the encoded digital image signal series output from the encoder 334 and can superimpose the evaluation result and the digital image signal to present them to the user. Also, evaluation information is generated taking into account the encoded information used for encoding, and the user can obtain an evaluation result in consideration of the encoded information.

  As described above, the functions of the image processing apparatus 330 can be realized by the hardware configuration shown in FIG. For example, the functions of the image processing unit 332, the encoder 334, the decoder 337, and the image output control unit 338 are realized by the CPU 902 based on a program recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. The function of the image storage unit 336 is realized by the RAM 906, the storage unit 920, or the removable recording medium 928. Further, the function of the display panel 340 is realized by the output unit 918. However, when the display panel 340 has a function as input means such as a touch panel, the display panel 340 also corresponds to the input unit 916.

  The basic technology according to the present embodiment has been described above by taking the imaging devices 100 and 200 and the image processing device 330 as examples. Hereinafter, as a specific apparatus configuration incorporating these techniques, the imaging apparatus 400 is taken as an example, and a more specific technical description will be given. In this, a series of processing flows including user operations will be described, and a display configuration on the screen will be described with a specific example.

[Functional Configuration of Imaging Device 400]
First, the functional configuration of the imaging apparatus 400 according to the present embodiment will be described with reference to FIG. FIG. 12 is an explanatory diagram illustrating a functional configuration example of the imaging apparatus 400 according to the present embodiment.

  As shown in FIG. 12, the imaging apparatus 400 mainly includes an image sensor 402, an image processing block 404, a control unit 406, an operation unit 408, a display panel 410, and a recording medium 412. The image sensor 402 is an example of an imaging unit. The image processing block 404 is an example of an image processing apparatus. The function of the image processing block 404 can be realized by the hardware configuration shown in FIG. The display panel 410 is an example of a display screen.

(Image sensor 402)
The image sensor 402 is formed by a photoelectric conversion element such as a CCD or a CMOS, for example. The image sensor 402 photoelectrically converts an optical signal input via the optical system and outputs an electrical signal. The electrical signal output from the image sensor 402 is input to the image processing block 404 (image processing unit 430) as a digital image signal. When a moving image is captured, the subject image is continuously input to the image sensor 402, and the digital image signal for each page is continuously input to the image processing unit 430.

(Control unit 406, operation unit 408)
The control unit 406 is means for controlling the image processing block 404 in accordance with a user operation by the operation unit 408. As the operation unit 408, for example, an operation unit such as a keyboard, a mouse, a remote controller, a button, a touch panel, a touch pad, an optical sensor, or a cursor is used. However, in the case of a touch panel, the function of the operation unit 408 is realized by combining with the display panel 410.

(Display panel 410)
The display panel 410 may be a display monitor or finder mounted on the imaging apparatus 400 or a display monitor connected to the imaging apparatus 400 as an external device. The display panel 410 may have an input device function such as a touch panel. In this case, the display panel 410 may be a touch panel with a pressure detection function having a function of detecting a pressing force when the user presses the screen. Further, the display panel 410 may have a function of three-dimensionally detecting the position on the screen that the user touches or approaches without directly touching the screen.

(Recording medium 412)
The recording medium 412 is configured by, for example, a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor recording medium. Digital image signals are mainly recorded on the recording medium 412. The recording medium 412 may record various setting information, meta information related to digital image signals, and the like. The recording medium 412 is controlled to read or write data by a media control circuit 442 included in the image processing block 404.

(Image processing block 404)
The image processing block 404 mainly includes an image processing unit 430, an image output control unit 436, an encoder / decoder 438, a buffer memory 440, and a media control circuit 442. The image processing unit 430 includes a preprocessing circuit 432 and a camera signal processing circuit 434. The image output control unit 436 is an example of a complex region detection unit, a complex region display unit, and a designated region display unit. The image output control unit 436 and the encoder / decoder 438 are examples of an image processing unit.

(Pre-processing circuit 432)
The pre-processing circuit 432 includes, for example, a CDS (Credited Double Sampling) circuit, an AGC (Automatic Gain Control) circuit, an A / D (Analog to Digital) converter, and the like. The CDS circuit is a circuit that removes reset noise included in the image signal input from the image sensor 402. The AGC circuit is a circuit that controls amplification of an image signal to a predetermined level. The A / D converter is a circuit that converts an analog image signal into a digital image signal. Digital image signals generated by these circuits are input to the camera signal processing circuit 434.

(Camera signal processing circuit 434)
The camera signal processing circuit 434 includes, for example, a feeder back clamp circuit, a white clip circuit, a base clip circuit, a white balance circuit, and a γ correction circuit.

  The feeder back clamp circuit is a circuit for fixing the black level OB (optical black) of the digital image signal to a constant reference value. The white clip circuit is a limiter circuit that prevents modulation of a predetermined amplitude or more from being applied in order to prevent overmodulation of the image signal. The base clipping circuit is a circuit for clipping low level noise. The white balance circuit is a circuit for adjusting the white balance. The γ correction circuit is a circuit for performing predetermined γ correction on the image signal. The digital image signal processed by these circuits is input to the image output control unit 436 and the encoder / decoder 438.

(Image output control unit 436)
The image output control unit 436 calculates a code such as a DCT coefficient from the digital image signal before encoding, and evaluates the complexity of each pixel area included in the digital image signal using this code. However, instead of the DCT coefficient, for example, a parameter such as a wavelet coefficient, flatness, or intra AC may be calculated, and the evaluation result may be calculated using the parameter. The image output control unit 436 may be configured to detect a motion vector using a plurality of digital image signals and calculate an evaluation result based on the magnitude of the motion vector. The image output control unit 436 superimposes the calculated evaluation result on the digital image signal and displays it on the display panel 410.

  Thereafter, when the pixel area is designated by the user via the operation unit 408 and the control unit 406, the image output control unit 436 changes the setting information of the bit amount allocated to the pixel area designated by the user. Then, the image output control unit 436 inputs the position information of the image area designated by the user and the information on the changed bit amount allocated to the designated area to the encoder / decoder 438. For example, when the user designates a complex area, the image output control unit 436 updates the setting information so that the allocated bit amount of the designated area is reduced. The updated setting information and the position information of the designated area are input to the encoder / decoder 438.

  Further, the image output control unit 436 is configured to perform predetermined image processing such as a low-pass filter on the designated area instead of changing the allocated bit amount of the designated area when the pixel area is designated by the user. It may be. As the predetermined image processing, in addition to the low-pass filter, for example, color reduction processing, depth-of-field reduction processing, or the like is used. Of course, the predetermined image processing applicable to the present embodiment is not limited to this, and may be appropriately changed according to the purpose.

(Encoder / Decoder 438)
The encoder / decoder 438 performs an encoding process on the digital image signal input from the camera signal processing circuit 434 using a predetermined encoding method. At this time, the encoder / decoder 438 performs the encoding process based on the position information of the designated area input from the image output control unit 436 and the allocated bit amount of the designated area. That is, the encoder / decoder 438 performs an encoding process on the designated area with the bit amount set by the image output control unit 436.

  The predetermined encoding method includes, for example, MPEG-2, MPEG-4, H.264, and the like. A method standardized by H.264 / AVC or the like is used. Of course, the encoding method applicable to this embodiment is not limited to these examples. For example, when an encoding method standardized by MPEG-2 is used, a motion compensation interframe difference encoding method or the like is used as described above. In such a case, the encoder / decoder 438 uses the buffer memory 440 as a frame memory. The digital image signal thus encoded by the encoder / decoder 438 is recorded on the recording medium 412 via the media control circuit 442.

  The encoder / decoder 438 may be configured to store the encoded digital image signal in the buffer memory 440, decode the digital image signal, and input the decoded digital image signal to the image output control unit 436. Such a configuration is used when calculating evaluation information for the encoded digital image signal as in the image processing apparatus 330 described above. In this case, evaluation information for the encoded digital image signal is superimposed on the digital image signal and displayed on the display panel 410.

  The functional configuration of the imaging apparatus 400 according to the present embodiment has been described above. As described above, the imaging apparatus 400 detects a pixel area including complexity or intensity of motion from the digital image signal before being encoded by the encoder / decoder 438, and superimposes the pixel area on the digital image signal. There is one feature in the configuration to display.

  With such a configuration, the complexity area and the like can be presented to the user at high speed. As a result, it becomes possible for the user to designate a complicated pixel region or the like in real time. In addition, the user can easily avoid complicated areas from the imaging range by adjusting the angle and the angle of view. As a result, since the complex area included in the digital image signal is reduced and the amount of bits allocated to other portions is increased, an effect of relatively improving the image quality after encoding can be obtained.

  Furthermore, when a pixel area is designated by the user based on information such as the presented complex area, the imaging device 400 is configured to perform predetermined image processing on the designated area. For example, the imaging apparatus 400 applies a low pass filter to the designated area or performs an encoding process by reducing the allocated bit amount of the designated area. By adopting such a configuration, it is possible to reduce the amount of code allocated to an area that the user considers unimportant, and even if the bit amount allocated per unit time is fixed, Video quality can be improved.

  Note that the function of the image processing block 404 constituting the imaging apparatus 400 can be realized by the hardware configuration shown in FIG. For example, the functions of the encoder / decoder 438 and the image output control unit 436 are realized by the CPU 902 based on a program recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. The function of the operation unit 408 is realized by the input unit 916.

  The function of the buffer memory 440 is realized by the RAM 906, the storage unit 920, or the removable recording medium 928. Further, the recording medium 412 is realized by the storage unit 920 or the removable recording medium 928. Further, the function of the display panel 410 is realized by the output unit 918. However, when the display panel 410 has a function as input means such as a touch panel, it corresponds to the input unit 916.

[Processing flow including user operations]
Here, the flow of image processing according to the present embodiment including the operation of the imaging apparatus 400 and user operations will be described.

  First, an overall flow of image processing according to the present embodiment will be described with reference to FIGS. 13 and 14. Next, the flow of the composition setting operation will be described with reference to FIG. Next, the flow of an optical system setting operation will be described with reference to FIG. Next, the flow of the image processing setting operation will be described with reference to FIG. Next, the flow of an encoding setting operation will be described with reference to FIG.

(Overall flow)
First, referring to FIG. As shown in FIG. 13, the image complexity is visualized by the imaging device 400 (SU10). As described above, the imaging apparatus 400 can visualize one or both of the spatial complexity and the temporal complexity of the digital image signal.

  For example, a pixel region having a large high frequency component coefficient obtained by frequency analysis of a digital image signal is visualized in a predetermined display format as a complex region having spatial complexity. In addition, the motion vector of the pixel area is calculated by motion detection, and the pixel area having a large motion vector is visualized in a predetermined display format as a complex area having temporal complexity.

  Next, the complexity is adjusted by the user and the imaging device 400 at the stage of image creation (SU20). Examples of elements that can be adjusted include composition, video processing, and encoding settings.

  Examples of composition adjustment elements include camera work, focus, iris, and zoom. Composition adjustment is mainly performed by the user. However, the user adjusts the composition while referring to the complex area displayed on the display panel 410 of the imaging apparatus 400. The imaging apparatus 400 updates the display of the complex area in real time when the composition is changed by the user. Therefore, the user can interactively adjust the composition while referring to the display panel 410. Such composition adjustment is realized by the technique of the imaging apparatus 400 capable of displaying a complex region at high speed.

  Examples of adjustment elements for image processing include an optical filter and a digital filter. Adjustment of image processing is mainly performed by the user. However, in the case of a digital filter, for example, it is realized in a format in which the imaging apparatus 400 performs a filtering process on a pixel region designated by the user. Even in such a case, the user can interactively adjust the video processing while referring to the display panel 410. The adjustment of the image processing is also realized by the technology of the imaging apparatus 400 that can display a complex area at high speed.

  As an adjustment element of the encoding setting, for example, there is a weighting adjustment for adjusting the bit amount allocated to the pixel area. The adjustment of the encoding setting is mainly performed by the user. However, it is realized in a format in which the imaging device 400 weights the allocated bit amount for the pixel region designated by the user. Also in such a case, the user can specify a pixel area to be weighted interactively while referring to the display panel 410. The adjustment of the encoding setting is also realized by the technology of the imaging apparatus 400 that can display a complex area at high speed.

  After the above adjustment is performed, the imaging device 400 performs an encoding process on the digital image signal (SU30). At this time, as compared with the case where the encoding process is performed on the image before performing the above adjustment, a larger bit amount is allocated to the area which is regarded as important by the user. High image quality.

  For example, when the composition is adjusted and the complex area of the imaging range is reduced, if the amount of bits allocated to the entire image is the same, the amount of bits taken in the complex area can be relatively small, and therefore, The image quality is improved. Further, in the case of a configuration in which a large amount of bits is allocated to the designated area, naturally, the designated area that is important to the user is improved in image quality, and an encoded image suitable for the user's preference is obtained.

  Reference is now made to FIG. FIG. 14 is an explanatory diagram showing an overall processing flow focused on user operations.

  As shown in FIG. 14, first, display setting of a complex area is performed by the user (S102). For example, a display method, a threshold value, an area designation, etc. are set. As a display method of the complex area, for example, whether to display with a pattern on another window (FIG. 19), display with a pattern on the display screen (FIG. 23), or display with a color (FIG. 25, FIG. 25). FIG. 26) is set.

  The threshold value is a threshold value used when converting DCT coefficients or the like obtained from the digital image signal before encoding into evaluation information. The area designation setting is a setting for the user to designate an area. For example, when the display panel 410 is touched by the user, the setting is whether to display a frame (FIG. 21) or display with a pattern (FIG. 22) in the touched pixel area of a predetermined size.

  Next, the user determines whether or not the desired display setting has been achieved (S104). If the desired display setting is obtained, the process proceeds to step S106. On the other hand, if the desired display setting is not obtained, the process returns to step S102 to change the display setting.

  In step S106, the user determines what processing is to be performed (S106). When setting the composition, the process proceeds to step S108. For example, the camera position or orientation is adjusted, the lens used is changed, or the zoom magnification is adjusted. When setting the optical system, the process proceeds to step S110, and parameters such as focus, iris, and optical filter are adjusted. When performing image processing setting, the process proceeds to step S112, and adjustment such as LPF (Low-Pass Filter) is performed.

  Next, the user determines whether there is a complex area to be avoided (S114). If there is a complex area to be avoided, the process proceeds to step S106, and the setting is changed to avoid the complex area. If there is no complicated area to avoid, the process proceeds to step S116.

  In this step, even if a complex area exists, if it is considered that the user should not avoid the complex area, it is important that the complex area is left without being avoided. For example, even if there are a plurality of complex areas having the same complexity, only those who think that the user is important can be left. It should be noted that such selective processing that reflects user preferences cannot be realized in an apparatus that automatically determines a complex area and performs image processing.

  Next, in step S116, it is determined whether or not the display setting of the complex area is changed by the user (S116). When changing the display setting of the complex area, the process proceeds to step S102. On the other hand, if the display setting of the complex area is not changed, the process proceeds to step S118. In step S118, encoding setting is performed by the user (S118). For example, the user sets a weighting amount of the bit amount. Next, recording of a digital image signal is started (S120), and a series of setting processes is completed.

(Composition setting operation flow (S108))
Here, the composition setting operation executed in step S108 will be described with reference to FIG. FIG. 15 is an explanatory diagram showing the flow of the composition setting process.

  As shown in FIG. 15, first, it is determined whether or not the composition is changed by the user (S132). When the composition is changed, the process proceeds to step S134. On the other hand, when the composition is not changed, a series of processes related to composition setting is ended. In step S134, the composition is set by the user (S134). At this time, the user changes the composition so as to avoid the complex area. Next, the user determines whether or not the desired composition has been achieved (S136). If it is a desired composition, a series of processes related to composition setting is terminated. On the other hand, if the desired composition is not obtained, the process returns to step S132.

  In the above-described step S136, the user refers to the complex area displayed on the display panel 410 of the imaging apparatus 400 and determines whether or not the composition is a desired one. At the time of this determination processing, the composition setting performed in step S134 needs to be reflected and the complex area of the display panel 410 needs to be updated. The composition change is repeatedly performed with repeated fine adjustments. For this reason, if it takes a long time for the composition setting change to be reflected, the burden on the user will increase, and it will be difficult to avoid complex areas because the subject will move or the lighting will change. there is a possibility. Considering these points, it can be said that the complex area high-speed display function of the imaging device 400 is indispensable for the composition setting process shown in FIG.

(Flow of optical system setting operation (S110))
Here, the optical system setting operation executed in step S110 will be described with reference to FIG. FIG. 16 is an explanatory diagram showing the flow of the optical system setting process.

  As shown in FIG. 16, first, it is determined whether or not the user changes the optical system (S152). When the optical system is changed, the process proceeds to step S154. On the other hand, when the optical system is not changed, a series of processes related to the optical system setting is ended. In step S154, the user sets the optical system (S154). At this time, the user adjusts the optical system so as to avoid the complicated area. Next, the user determines whether or not a desired image has been obtained (S156). If the image is a desired image, a series of processing relating to the optical system setting is terminated. On the other hand, if it is not a desired image, the process returns to step S152.

  In the above-described step S156, the user refers to the complex area displayed on the display panel 410 of the imaging apparatus 400 and determines whether the image is a desired image. At the time of this determination process, the setting of the optical system performed in step S154 is reflected, and the complex area of the display panel 410 needs to be updated. The change of the optical system is repeatedly performed with repeated fine adjustments. For this reason, if it takes a long time to reflect the change in the settings of the optical system, the burden on the user will increase, and it will be difficult to avoid complicated areas due to movement of the subject and changes in illumination. There is a possibility. Considering these points, it can be said that the high-speed display function of the complex area of the imaging apparatus 400 is indispensable for the optical system setting process shown in FIG.

(Flow of image processing setting operation (S112))
Here, the image processing setting operation executed in step S112 will be described with reference to FIG. FIG. 17 is an explanatory diagram showing the flow of the image processing setting operation.

  As shown in FIG. 17, first, the user determines whether or not to change the image processing setting (S172). When the image processing setting is changed, the process proceeds to step S174. On the other hand, when the image processing setting is not changed, the series of processing related to the image processing setting is ended. In step S174, it is determined whether or not the user designates an image processing area (S174). When an image processing area is designated, the process proceeds to step S176. On the other hand, if no image processing area is designated, the process proceeds to step S178.

  In step S176, an image processing area is designated by the user (S176). For example, an area on the touch panel (display panel 410) is pressed by the user and designated as an image processing area. Next, image processing is set by the user (S178). At this time, the type of the image processing method applied to the designated area is set by the user (S178). Next, the user determines whether or not a desired image has been obtained (S180). If the image is a desired image, a series of processes related to the image processing setting is terminated. On the other hand, if it is not a desired image, the process returns to step S172.

(Flow of encoding setting operation (S118))
Here, the encoding setting operation executed in step S118 will be described with reference to FIG. FIG. 18 is an explanatory diagram showing the flow of the encoding setting operation.

  As shown in FIG. 18, first, the user determines whether or not to change the encoding setting (S192). When the encoding setting is changed, the process proceeds to step S194. On the other hand, when the encoding setting is not changed, a series of processes related to the encoding setting is terminated. In step S194, it is determined whether or not the user designates a weighted area (S194). If a weighted area is designated, the process proceeds to step S196. On the other hand, if no weighting area is designated, the process proceeds to step S198.

  In step S196, the user designates a weighting area (image processing area) (S196). For example, an area on the touch panel (display panel 410) is pressed by the user and designated as an image processing area. Next, encoding is set by the user (S198). At this time, the bit amount (weight amount) allocated to the designated area is set by the user (S198), and a series of processing relating to the encoding setting is ended.

  The series of processes related to user operations has been described above. As described above, by using the function of the imaging apparatus 400 according to the present embodiment, it is possible to effectively avoid the complex area by the user operation as described above. In other words, the display of the complex area is updated in real time according to the user's operation, and the user performs the avoidance action with reference to the displayed complex area, so that the encoding efficiency of the video data is increased and the video quality is improved. It is.

[Display structure of complex area]
Here, a display configuration of a complex area or the like displayed on the display panel 410 of the imaging apparatus 400 will be described with reference to FIGS. Display control according to this display configuration is realized mainly by the function of the image output control unit 436.

(Configuration to display complex areas in a separate window)
First, refer to FIG. FIG. 19 illustrates a state in which an image and another window are superimposed and displayed on the display screen of the display panel 410. The image displayed on the display screen shows a state in which a plurality of children are playing a ball.

  This image includes a plurality of trees in the background. In the case of such an image, since the sky and the ground are relatively flat symbols, it is assumed that the complexity is small. On the other hand, the trees in the background include a relatively fine structure, so it is presumed that the complexity is large. Also, it can be inferred that the child's arm and the ball being thrown are relatively fast moving parts. Furthermore, the leaves of the trees are thought to be complex and intensely moving when the strong wind blows and shakes.

  If such an image is encoded by an MPEG method or the like, a large amount of code is assigned to the background trees. For this reason, compression noise occurs in the foreground children's pattern, which should be important for the photographer, and the image quality deteriorates significantly. On the other hand, in the case of a device that automatically detects a complex area and partially reduces the bit rate, the child's arm and ball part are also drawn at a low bit rate, and the video quality of the area important for the photographer is reduced. There is a high possibility that it will decline. On the other hand, when the photographer removes a part of the trees from the photographing range, the video quality after encoding can be easily improved.

  However, it is difficult for many photographers to grasp such a complex area during photographing. Therefore, a complex area is displayed as shown in FIG. 19 so that any photographer can easily grasp the complex area. In the example of FIG. 19, a separate window is displayed, and a pattern indicating a complex region is displayed in the separate window so as to be superimposed on the image. In this example, the complex area is displayed by hatching, but the present invention is not limited to this. In this example, the pattern is shaded according to the degree of complexity. Therefore, the photographer can improve the video quality after encoding by adjusting the angle, the angle of view, and the like so as to avoid a pattern portion having a high density.

(Configuration to display the touch panel area in a separate window)
Reference is now made to FIG. In FIG. 20, an image similar to the example of FIG. 19 is displayed. In the case of the imaging apparatus 400, when a photographer designates an arbitrary pixel area, it is possible to perform image processing on the designated area or change an allocated bit amount. As the specification method, for example, a method of specifying by touching the display screen with a finger or the like is preferable.

  In the example of FIG. 20, a state where a complex area corresponding to a portion of trees included in the background is touched and selected as a designated area is shown. In this example, the outer edge of the designated area is displayed in a separate window with a thick line. As described above, by selectively designating the complex area, it is possible to perform image processing only on the complex area desired by the photographer.

(Configuration to display the touch panel area on the main window (outer edge display))
Further, as shown in FIG. 21, the outer edge of the designated area may be displayed so as to be superimposed on the through image displayed on the display screen. Thus, by displaying the complex area in a separate window and displaying the designated area on the main screen, the photographer can easily recognize how far the outer edge of the designated area is.

  Further, an area touched by the photographer may be set on the main screen, and the separate window may be limited to display only the complex area. With this configuration, the photographer can easily select a complex area. The designated area is not limited to a complex area, and may be an arbitrary pixel area. However, it may be possible to achieve higher operability when the complex area is selected as the designated area by touching the displayed complex area. Convenience will be improved by allowing such display settings to be arbitrarily performed by the user.

(Configuration to display the touch panel area in the main window (object display))
Reference is now made to FIG. FIG. 22 shows a configuration in which a pixel area specified by touching is displayed as an object having a predetermined shape. In the example of FIG. 22, a configuration in which a circular designated area is displayed on the touched portion is drawn. In this example, the diameter of the circle is increased according to the touched time length. That is, the photographer can designate a large pixel area by one touch by touching for a long time.

  The shape of the object may be circular as shown in FIG. 22, but may be set to an arbitrary shape such as a substantially polygon such as a triangle or a quadrangle, an ellipse, a star, or a saddle. Furthermore, the object may be configured to expand uniformly according to the length of the touch, but may be configured to expand along the outer edge of the complex region, for example. By enlarging the object so that it does not protrude from the outer edge of the complex area, the photographer can easily select only a part of the complex area. Moreover, when the complex area is displayed hierarchically, the designated area may be configured to expand along the outer edge of the complex area in each hierarchy.

  In the above example, the object is configured to spread according to the length of the touch, but the present embodiment is not limited to this. For example, the object may be configured to expand according to the pressing force at the time of touching, or may be configured to expand according to the shape of the locus drawn by sliding the touched finger or the like. Also good. Further, the enlargement ratio may be changed according to the number of touched fingers and the contact area.

(Refine complex area (color))
Reference is now made to FIG. FIG. 23 shows a display configuration in which an image and a complex area are superimposed and displayed on the main screen. Further, in FIG. 23, a color designation bar for narrowing down the complex area by color is superimposed and displayed. The color designation bar includes a color designation slide bar and a luminance designation slide bar.

  When a color is specified by the color designation slide bar, only the color area that matches or is close to the specified color is displayed in the complex area displayed on the display screen, and the designation of other complex areas is canceled. The Similarly, when the brightness is specified by the brightness specifying slide bar, only the area of the brightness that matches or is close to the specified brightness is displayed in the complex area displayed on the display screen. Is released. That is, only the complex area that matches the combination of the designated color designated by the color designation bar and the designated luminance is displayed, and the designation of other complex areas is cancelled.

  The complex area included in the background and the complex area included in the foreground often have different colors and brightness. In addition, the same type of subject often includes similar colors. For this reason, the narrow-down display based on color information is effective when it is desired to designate the same type of subject with complexity as a complex area. That is, the narrowed display based on the color information is performed so that the photographer can easily specify a desired complex area.

(Specify preset area)
Reference is now made to FIG. FIG. 24 shows a display configuration in which a preset area where a complex area is not displayed is set. For a pixel area where the photographer has no plan to remove from the shooting range and does not want to reduce the image quality, a preset area may be set in advance, and a pattern indicating a complex area may not be displayed in that area. Good. By using such a display configuration, the photographer may accidentally remove important areas from the shooting range in order to avoid complex areas, or set designated areas that reduce the allocated bit amount. I can prevent it. In addition, it is possible to avoid the trouble of displaying a pattern indicating a complex area in an area that does not have complexity for the photographer, and the photographer can concentrate on photographing.

(Display of complex area by color)
Reference is now made to FIG. The example of FIG. 25 is an example in the case where a complex area is expressed by color, shading, or the like. Although it is displayed as a mesh due to restrictions on expression, for example, the mesh portion can be expressed so as to be painted with a conspicuous color such as blue or red. Expression by color is advantageous in that the degree of freedom of expression is higher than expression by pattern. In the case of a color, since the color and brightness can be adjusted, the color can be changed or the brightness can be changed according to the degree of complexity. An example is shown in FIG. When the shade of the color is changed according to the evaluation value indicating the complexity, it is expressed as shown in FIG. Here, the expression is expressed by the density of the mesh, but by assigning different colors to the respective stages, it is possible to make the expression easy for the photographer to recognize the degree of complexity.

(Display of complex area and adjustment menu)
Reference is now made to FIG. FIG. 27 is a display menu screen for the imaging apparatus 400 to prompt the user to select an adjustment method when a complex area is detected. As described above, the imaging apparatus 400 can perform image processing on the complex area and change the allocated bit amount. Therefore, when a complex area is detected, the imaging apparatus 400 selects what kind of image processing or the like should be performed on the complex area or the area designated by the user. When selected by the user, the imaging apparatus 400 executes image processing or the like based on the selected method.

(Display of hints related to user imaging operations)
Reference is now made to FIG. FIG. 28 is a screen on which the imaging apparatus 400 presents an adjustment method for avoiding the complex area to the user when the complex area is detected.

  Again, avoidance of complex areas can be realized by angle adjustment or the like without performing image processing. When a large number of complex areas are included in the imaging range, by excluding most of the complex areas from the imaging range, the amount of video data after encoding is greatly reduced, and a great improvement in video quality can be obtained. However, many photographers cannot easily recognize a complex area and cannot instantly determine how to avoid the complex area. Therefore, as shown in FIG. 28, the imaging apparatus 400 presents a method for avoiding the complex area to the user.

  At this time, the imaging apparatus 400 instructs the user to shift the angle to the right when there are many complex areas on the left side of the screen. At this time, the imaging apparatus 400 may be configured to present a method of moving the angle so as to avoid a part including a complex area with a high degree of complexity from the imaging range. Of course, not only the angle but also one or a plurality of hints related to composition adjustment for avoiding complicated areas such as focus and zoom may be presented to the user. By adopting such a configuration, even if the user does not recognize the complex area accurately or cannot immediately determine the avoidance action, the user can easily avoid the complex area and obtain a high-quality image. Can be taken.

(Preset menu display)
Reference is now made to FIG. FIG. 29 is an explanatory diagram showing a menu screen for displaying a preset menu to the user. Examples of the preset menu include “video chat”, “Web upload”, “athletic meet”, “nature / landscape”, and “drama shooting”. When these preset menus are selected, comments are displayed as shown in FIG. For example, in the case of “Web upload”, a comment such as “Display a complex area near the outer frame of the screen. Lower the bit rate of the complex area” is displayed. Furthermore, in the case of user designation, a comment such as “an area for displaying complexity and a threshold value can be freely set. Processing to a complex area can also be freely performed” is displayed.

(Parameter adjustment screen)
Reference is now made to FIG. FIG. 30 shows a display configuration for dynamically adjusting the parameters when an image processing method is selected by the user. As shown in FIG. 30, an adjustment bar is displayed on the display screen, and by operating this adjustment bar, for example, parameters of the LPF can be changed. When this parameter is changed, LPF is applied to the complex area based on the changed parameter and displayed. Of course, the adjustment parameter is not limited to the LPF, and may be a depth of field, a degree of color reduction, an allocated bit amount, or the like. As described above, the video after the image processing can be confirmed and the parameters can be adjusted interactively, so that the effect of the image processing on the complex area can be easily confirmed.

[Hardware configuration (information processing equipment)]
The function of each component included in the above-described device can be realized, for example, by an information processing device having a hardware configuration illustrated in FIG. 31 using a computer program for realizing the above-described function. FIG. 31 is an explanatory diagram showing a hardware configuration of an information processing apparatus capable of realizing the functions of each component of the apparatus.

  The form of this information processing apparatus is arbitrary. For example, a personal computer, a mobile phone, a personal information system such as a personal handy phone system (PHS), a personal digital assistant (PDA), a game machine, or various information appliances This includes forms.

  As shown in FIG. 31, this information processing apparatus mainly includes a CPU (Central Processing Unit) 902, a ROM (Read Only Memory) 904, a RAM (Random Access Memory) 906, a host bus 908, and a bridge 910. An external bus 912, an interface 914, an input unit 916, an output unit 918, a storage unit 920, a drive 922, a connection port 924, and a communication unit 926.

  The CPU 902 functions as, for example, an arithmetic processing unit or a control unit, and controls the overall operation of each component or a part thereof based on various programs recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. . The ROM 904 stores, for example, a program read by the CPU 902 and data used for calculation. The RAM 906 temporarily or permanently stores, for example, a program that is read into the CPU 902 and various parameters that change as appropriate when the program is executed. These components are connected to each other by, for example, a host bus 908 capable of high-speed data transmission. The host bus 908 is connected to an external bus 912 having a relatively low data transmission speed via a bridge 910, for example.

  The input unit 916 is an operation unit such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. The input unit 916 may be remote control means (so-called remote controller) capable of transmitting a control signal using infrared rays or other radio waves. Note that the input unit 916 includes an input control circuit for transmitting information input using the above-described operation means to the CPU 902 as an input signal.

  The output unit 918 is, for example, a display device such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an ELD (Electro-Luminescence Display), an audio device such as a speaker, a headphone, or the like. It is a device capable of visually or audibly notifying acquired information to a user, such as a mobile phone or a facsimile.

  The storage unit 920 is a device for storing various data, and includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. Is done.

  The drive 922 is a device that reads information recorded on a removable recording medium 928 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 928. The removable recording medium 928 is, for example, a DVD medium, a Blu-ray medium, an HD DVD medium, a compact flash (registered trademark) (CF; CompactFlash), a memory stick, or an SD memory card (Secure Digital memory card). Of course, the removable recording medium 928 may be, for example, an IC card (Integrated Circuit Card) on which a non-contact IC chip is mounted, an electronic device, or the like.

  The connection port 924 is a port for connecting an external connection device 930 such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. is there. The external connection device 930 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

  The communication unit 926 is a communication device for connecting to the network 932. For example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB), optical communication A router, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The network 932 connected to the communication unit 926 is configured by a wired or wireless network, such as the Internet, home LAN, infrared communication, visible light communication, broadcast, or satellite communication.

[Use Case]
Finally, the use case is described.

  When used for “video chat”, the user only performs operations such as changing the direction and zooming. In addition, the user thinks that only what he / she wants to show his / her face and the other party needs to be beautifully photographed. In some cases, in order to increase anonymity, the image quality of only the face portion may be reduced. In addition, there is a demand for minimizing the video size. “Video chat” is intended for such users. For these users, the orientation and layout of the camera can be changed while looking at the complexity display, so that the user can make adjustments in real time so that it can be easily seen by the other party. The same applies to “Web upload” in that a bit rate is assigned only to an important part with a minimum amount of data.

  When used for "athletic meet" photography, there is a lot of human movement and there are many trees and greens, so compression noise is likely to occur. In addition, the photographer wants to shoot beautifully around his child. Furthermore, since the subject moving in a crowded place is photographed, the movement of the camera is large. In such a situation, where to set the shooting position is very important. In such a case, it is possible to confirm in advance a shooting position that includes a scene where a child is running in a race or the like but does not include a complex background area. In addition, it is possible to avoid the complex area by checking the angle including the complex area in advance and adjusting the zoom to the child when the angle is reached.

  When used for “drama shooting”, there is a demand for reducing the bit rate as much as possible. In addition, if there is a complicated part in the background, it will be a so-called noisy picture for the viewer, and there is a demand to avoid such a situation. In order to meet these demands, when shooting a drama or a movie, a search for a shooting location called location hunting is performed. At that time, if the imaging apparatus 400 is used, it is possible to easily search for a shooting location in consideration of complexity. In addition, if a threshold value for complexity display is set, it is possible to determine a shooting location according to the above request in consideration of disturbances such as wind.

  With respect to dealing with sudden scene changes and the like, the technology of the imaging apparatus 400 can be suitably applied to imaging equipment used for live broadcasting. This is because location hunting is not possible in live broadcasting, and it is necessary to respond in real time to sudden scene changes while shooting. If the technique of the imaging apparatus 400 is applied in such a case, it becomes possible to respond to a scene change interactively.

[Summary]
In the case of digital photography, you can reduce the bit rate of the same video if you make a picture that does not include a complicated background. However, if the composition is automatically changed, the video will become strange. In addition, the photographer does not know the area where the bit rate is often divided. Even if the same subject is shot, it is possible to avoid a complicated image area that does not need to be included in the video by moving the composition slightly up and down and left and right. As a result, the bit rate can be lowered. Furthermore, it is possible to eliminate the stress that the photographer shoots while taking care of the data amount.

  In order to realize such a situation, real-time property is important for displaying a complex area. It is also important to obtain an interactive operational feeling. In order to suppress the overall code amount, when a code amount threshold value to be assigned to each video frame is determined, it is very large that an area where a large amount of code amount should be assigned in one frame can be set interactively. It is a merit. Also, it can be suitably used to avoid a situation in which a certain scene is clearly reflected, but the quality of the image is degraded when a person enters or a complicated area enters.

  The effects of this embodiment will be enumerated. First, the user can change / adjust camera work, composition, camera settings, focus, zoom, iris, lens used, optical filter, depth of field, etc. while viewing the displayed complex area. . As a result, it is possible to obtain a guideline for capturing an image desired by the user, such as an image with high image quality, low capacity, low noise, and noisy. In addition, even in video content that requires shooting at a low bit rate that is assumed to be exchanged via the Internet, a complex or intensely moving region that is present in an unimportant part of an image is easily found. There is a merit that

  The same applies to video content that requires higher image quality. In these video contents, it is possible to reduce image quality degradation in important areas by removing complicated or intensely moving areas from the shooting area, shifting the focus, or reducing the amount of allocated bits. In addition, it is possible to grasp how much noise is present and how much image quality is obtained before reproducing the recorded moving image.

  Further, as an important effect, for example, there is an effect that the image quality can be improved by performing a simple operation such as slightly shifting the direction in which the user points the imaging apparatus. However, in order to obtain such an effect, in this embodiment, a complicated region and a region with high movement are displayed on the original image so as to indicate which region of the image and which subject is complex, It is possible to identify whether the is intense. As a result, the user can easily specify the direction in which the imaging device is slightly shifted.

  However, such processing requires real-time properties, but in the case of this embodiment, there is little time delay in the evaluation results of the movement of the subject, the complexity displayed on the monitor, and the intensity of the movement. Even when a moving image is captured, the imaging direction can be adjusted without a problem. The technology according to the present embodiment can be used not only as a method for recording moving images but also as an apparatus for measuring the complexity of an image and the intensity of movement. For example, it can be applied to the evaluation of cityscapes and artworks. In the present embodiment, since a complicated algorithm is not used, the technology of the present embodiment can be mounted without much effort on an existing imaging apparatus.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  For example, when the complex area is detected, the imaging apparatus 400 may present the existence of the complex area to the user by making a speaker sound or turning on a light source such as an LED.

It is explanatory drawing which shows the structural example of the imaging device which concerns on one Embodiment of this invention. It is explanatory drawing which shows the structural example of the image output control part which concerns on this embodiment. It is explanatory drawing which shows the structural example of the image signal evaluation part which concerns on this embodiment. It is explanatory drawing for demonstrating the detection process of a complexity area | region. It is explanatory drawing for demonstrating the detection process of a complexity area | region. It is explanatory drawing for demonstrating the detection process of a complexity area | region. It is explanatory drawing for demonstrating the detection process of a complexity area | region. It is explanatory drawing for demonstrating the detection process of a complexity area | region. It is explanatory drawing which shows the structural example of the imaging device which concerns on this embodiment. It is explanatory drawing which shows the structural example of the allocation bit change part which concerns on this embodiment. It is explanatory drawing which shows the structural example of the imaging device which concerns on this embodiment. It is explanatory drawing which shows an example of the apparatus structure of the imaging device which concerns on this embodiment. It is explanatory drawing which shows the whole flow of the image processing method which concerns on this embodiment. It is explanatory drawing which shows the whole flow of the user operation which concerns on this embodiment. It is explanatory drawing which shows the flow of the composition setting operation which concerns on this embodiment. It is explanatory drawing which shows the flow of the setting operation of the optical system which concerns on this embodiment. It is explanatory drawing which shows the flow of image processing setting operation which concerns on this embodiment. It is explanatory drawing which shows the flow of the encoding setting operation which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the complex area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the designation | designated area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the designation | designated area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the designation | designated area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the complex area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the preset area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the complex area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the complex area | region which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the adjustment menu which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the operation hint which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the preset menu which concerns on this embodiment. It is explanatory drawing which shows the example of a display structure of the parameter adjustment screen which concerns on this embodiment. It is explanatory drawing which shows the hardware structural example which concerns on this embodiment.

Explanation of symbols

100, 200 Imaging device 102, 202 Optical system 104, 204 Image sensor 130, 230 Image processing block 132, 232 Image processing unit 134, 234 Encoder 136, 236 Image storage unit 138, 238 Image output control unit 140, 240 Display panel 152 Image signal evaluation unit 154 Evaluation result visualization unit 156 Image superposition control unit 172 Image region division unit 174 Frequency analysis unit 176 Analysis result digitization unit 239 Allocation bit change unit 252 Area selection unit 254 Allocation bit amount change unit 256 Operation result output unit 330 Image processing device 332 Image processing unit 334 Encoder 336 Image storage unit 337 Decoder 338 Image output control unit 340 Display panel 400 Imaging device 402 Image sensor 404 Image processing block 406 Control unit 408 Operation unit 410 Display panel 412 Recording media 430 Image processing unit 432 Pre-processing circuit 434 Camera signal processing circuit 436 Image output control unit 438 Encoder / decoder 440 Buffer memory 442 Media control circuit

Claims (8)

A display screen that displays the unencoded image,
A complex region detection unit for detecting a complex pixel region from the image before encoding;
A complex area display unit that displays a complex pixel area detected by the complex area detection unit on the display screen;
An operation input unit for a user to designate an arbitrary area in the image as a designated area;
An image processing unit that performs predetermined image processing on a designated area designated via the operation input unit;
An image processing apparatus comprising:   The image processing unit is an encoder that encodes the image, and a relatively large code amount or a relatively small code amount is added to a designated area designated through the operation input unit when the image is coded. The image processing apparatus according to claim 1, wherein the image processing apparatus performs allocation and encoding.   The image processing apparatus according to claim 1, wherein the image processing unit performs a low-pass filter, a color reduction process, or a depth-of-field reduction process on a designated area designated via the operation input unit. A designated area display unit for displaying an outer edge of the designated area designated via the operation input unit on the display screen;
The image processing apparatus according to claim 1, wherein the designated area display unit enlarges the size of the designated area according to a length of time that the operation input unit is operated by a user.   The designated area display unit enlarges the designated area so that an outer edge of the designated area is along an outer edge of the complex pixel area detected by the complex area detecting unit when the size of the designated area is enlarged. Item 5. The image processing apparatus according to Item 4.   The complex area detection unit detects a pixel area having a large discrete cosine transform coefficient, wavelet coefficient, motion vector length, flatness, or intra AC detected from the pre-encoded image as the complex pixel area. The image processing apparatus according to claim 1. An imaging unit that forms an image of a subject from light incident through the optical system;
The image processing apparatus according to claim 1, wherein the imaging unit displays the image of the subject on the display screen, and inputs the image of the subject to the complex region detection unit as the image before the encoding. A complex region detection step in which a complex pixel region is detected from an image before encoding;
A complex region display step in which an image before encoding is displayed on the display screen, and the complex pixel region detected in the complex region detection step is displayed on the display screen;
An image processing step in which a predetermined image processing is performed on at least the designated area when an arbitrary area in the image is designated by the user as the designated area;
Including an image processing method.

Images