JP6083162B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, a super-resolution process for generating a high-resolution image based on a plurality of low-resolution images having a positional shift is known (for example, see Patent Document 1).

  In the super-resolution processing, for example, a single high-resolution still image can be generated by combining a plurality of low-resolution still images captured by an electronic camera.

JP 2009-55131 A

  However, when super-resolution processing is performed using video data that has been subjected to compression processing, if block noise that is compression distortion is included in the video data, for example, super-resolution processing is performed using a plurality of frames. There is a problem in that the accuracy of image alignment required at the time is lowered.

  In view of the above circumstances, an object of the present invention is to provide means capable of performing super-resolution processing with high accuracy even for moving image data subjected to compression processing.

An image processing apparatus according to a first invention includes a selection unit, a determination unit, and a noise removal unit. The selection unit reads moving image data compressed using inter-frame prediction, and selects a plurality of frames to be processed. The determination unit analyzes information on the frame selected by the selection unit, and determines whether noise is to be removed for each frame. The noise removal unit performs the noise removal process for each frame according to the determination result of the determination unit. The determination unit is configured to determine not to remove noise with respect to the first frame used in inter-frame prediction and removal of noise with respect to the first second frame highly compressed than the frame Judge .

In a second aspect based on the first aspect, the image processing apparatus further includes a super-resolution processing section. The super-resolution processing unit generates a higher resolution image than the frame by performing super-resolution processing on the plurality of frames to be processed including the frame on which the removal processing has been performed.

In a third aspect based on the first aspect, the determination unit determines whether or not to perform the removal process based on at least one of a compression rate and a compression format for each frame.

In a fourth aspect based on the first aspect, the noise removing unit performs a deblocking filter process for removing block noise as the noise.

In a fifth aspect based on the first aspect , the determination unit performs the removal process on the I frame, the B frame, and the P frame used for the inter-frame prediction according to a preset compression rate threshold value. It is determined whether or not.

In a sixth aspect based on the second aspect , the determination unit excludes a frame having a resolution unsuitable for the super-resolution processing from the processing target.

An imaging apparatus according to a seventh aspect includes a compression processing unit that compresses moving image data generated by continuously capturing subject images, and the image processing apparatus according to any one of the first to sixth aspects. .

An image processing program according to an eighth aspect causes a computer to execute a selection step, a determination step, and a noise removal step. The selection step reads moving image data compressed using inter-frame prediction, and selects a plurality of frames to be processed. In the determination step, information on the frame selected in the selection step is analyzed to determine whether noise is to be removed for each frame. In the noise removal step, the noise removal processing is performed for each frame according to the determination result of the determination step. The determination step, the determination not to remove noise with respect to the first frame used in inter-frame prediction and removal of noise with respect to the first second frame highly compressed than the frame Judge .

  The present invention can provide means for performing super-resolution processing with high accuracy even for moving image data that has been subjected to compression processing.

The block diagram which shows the structural example of the electronic camera 1 Diagram explaining the outline of the super-resolution processing mode A flowchart showing the operation of the electronic camera 1 under the super-resolution processing mode. Flowchart of determination processing subroutine Super-resolution processing subroutine flowchart The flowchart which shows the modification of operation | movement of the electronic camera 1 under super-resolution processing mode. Block diagram showing a configuration example of the computer 100

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the first embodiment, an embodiment relating to an imaging apparatus, an image processing apparatus, and an image processing program according to the present invention will be described.

  FIG. 1 is a block diagram illustrating a configuration example of the electronic camera 1. Here, the electronic camera 1 that is an embodiment of the imaging apparatus according to the present invention includes the function of the image processing apparatus according to the present invention and the image processing program according to the present invention that is processed inside the electronic camera 1.

  Further, the electronic camera 1 has a moving image shooting function, and has a super-resolution processing mode in which super-resolution processing can be performed with high accuracy even for moving image data subjected to compression processing.

  The electronic camera 1 includes a photographing optical system 10, an image sensor 11, a signal processing unit 12, a RAM (Random Access Memory) 13, a flash memory 14 and a recording interface unit (hereinafter referred to as “recording I / F unit”). ) 15, a display interface unit (hereinafter referred to as “display I / F unit”) 16, a display monitor 17, an operation unit 18, a release button 19, a moving image shooting button 20, and a CPU (Central Processing Unit) 21. And a bus 22.

  Among these, the signal processing unit 12, the RAM 13, the flash memory 14, the recording I / F unit 15, the display I / F unit 16, and the CPU 21 are connected to each other via a bus 22.

  The photographing optical system 10 includes, for example, a plurality of lens groups including a zoom lens that adjusts the focal length and a focus lens that adjusts the imaging position on the imaging surface of the imaging element 11. A lens driving unit (not shown) adjusts the lens position of the zoom lens and the focus lens in the photographic optical system 10 in the optical axis direction in accordance with an instruction from the CPU 21. For the sake of simplicity, FIG. 1 shows the photographing optical system 10 as a single lens.

  The image sensor 11 captures a subject image and generates an image (analog image signal). The analog image signal output from the image sensor 11 is input to the signal processing unit 12. Note that three types of color filters of R (red), G (green), and B (blue) are arranged on the imaging surface of the imaging device 11 in, for example, a Bayer array. Further, the charge accumulation time of the image sensor 11 and the reading of the image signal are controlled by a timing generator (not shown). The image sensor 11 is a CCD (Charge Coupled Device) type or CMOS (Complementary Metal-Oxide Semiconductor) type color image sensor.

  The signal processing unit 12 includes an analog front end (AFE) circuit, an A / D conversion unit, and a digital front end (DFE) circuit.

  The AFE circuit performs analog signal processing on the image signal output from the image sensor 11. The A / D converter converts an analog image signal into a digital image signal. The DFE circuit performs digital signal processing on the image signal after A / D conversion. The image signal output from the image sensor 11 is temporarily recorded in the RAM 13 as image data via the bus 22. The RAM 13 is a volatile memory, and has a buffer memory area for temporarily recording image data.

  The flash memory 14 is a rewritable nonvolatile semiconductor memory. The flash memory 14 stores a program for controlling the electronic camera 1.

  The recording I / F unit 15 is formed with a connector (not shown) for connecting a detachable recording medium M1. Then, the recording I / F unit 15 accesses the recording medium M1 connected to the connector and performs recording processing of still image data and moving image data. The recording medium M1 is, for example, a card-type nonvolatile memory card. FIG. 1 shows the recording medium M1 after being connected to the connector. The recording medium M1 may be used as a computer-readable recording medium that records the image processing program according to the embodiment of the present invention.

  The display I / F unit 16 provides an interface for displaying an image or the like on the display screen of the display monitor 17. The display monitor 17 is a display device (liquid crystal monitor) using a liquid crystal display medium, for example. The display monitor 17 displays, for example, a still image, a moving image, or an operation menu of the electronic camera 1. The operation unit 18 has a plurality of buttons for receiving a photographer's operation (user input), and receives an instruction input for operating the electronic camera 1.

  The release button 19 is half-pressed (starts shooting preparation operations such as autofocus (AF) and automatic exposure (AE) before shooting) and fully pressed (starts shooting operations to acquire a still image for recording). ) And an instruction input.

  The moving image shooting button 20 receives an instruction input for starting moving image shooting (moving image recording) by a pressing operation, and receives an instruction input for ending moving image shooting by a second pressing operation.

  The CPU 21 has a function of an ASIC (Image Engine: Application Specific Integrated Circuit), and is a microprocessor that performs various calculations and overall control of the electronic camera 1. CPU21 controls each part of the electronic camera 1, etc. by executing said program. Further, the CPU 21 reads the image data recorded in the RAM 13 and performs various image processing.

  Further, the CPU 21 stores, for example, the image processing program according to the embodiment of the present invention from the recording medium M1 in the flash memory 14. Thereby, CPU21 is provided with the selection part 21a, the determination part 21b, the noise removal part 21c, and the super-resolution process part 21d as one aspect | mode of the image processing apparatus which concerns on this invention. Further, the CPU 21 includes a compression processing unit 21e, a decryption processing unit 21f, and a recording processing unit 21g.

  In this embodiment, in order to make the explanation easy to understand, the selection unit 21a, the determination unit 21b, the noise removal unit 21c, and the super-resolution processing unit 21d are included in the CPU 21 in FIG. A separate dedicated hardware circuit may be used. Furthermore, in the present embodiment, the compression processing unit 21e, the decoding processing unit 21f, and the recording processing unit 21g may be included in a dedicated hardware circuit.

  Further, the CPU 21 executes the processing of the flowcharts shown in FIGS. 3 to 6 (hereinafter referred to as “flow processing”) according to the image processing program.

  The selection unit 21a reads the compressed moving image data and selects a plurality of frames to be processed. Here, the compressed moving image data is, for example, moving image data that has been compressed using inter-frame prediction. Inter-frame prediction is used for encoding processing such as MPEG (Moving Picture Experts Group) format in order to increase compression efficiency.

  In moving image data compressed using interframe prediction, unlike RAW data in the case of non-compression, for example, an I frame (Intra- coded frame), P frame (Predicted Frame) generated by predicting from the past frame in time (forward prediction), and B frame generated by predicting from the past and future in time (Bi- directional Predicted Frame).

  Since the data compressed using the inter-frame prediction is likely to generate block noise when decoded, it is suitable as data to which the super-resolution processing mode of the present embodiment is applied. The method of selecting a frame will be described later with reference to FIG.

  The determination unit 21b analyzes information on the frame selected by the selection unit 21a, and determines whether or not to remove block noise for each frame. Specifically, the determination unit 21b determines whether or not to perform block noise removal processing based on at least one of the compression rate and the compression format for each frame. For example, the determination unit 21b determines whether or not to perform block noise removal processing on an I frame, B frame, and P frame used for inter-frame prediction, according to a preset compression rate threshold. Thereby, the determination unit 21b can determine whether or not to perform a block noise removal process for each frame type.

  Here, for example, in the MPEG format encoding process, a discrete cosine transformation is performed on the image signal for each pixel block of a predetermined unit. In this encoding process, after performing discrete cosine transform, a predetermined quantization process is performed to reduce the amount of information.

  At this time, in this encoding process, the compression rate can be increased according to the degree of the quantization process. However, in order to discard some information, mosaic block noise is generated according to the compression rate.

  That is, in the MPEG format encoding process, a quantization process or the like is performed for each pixel block (for example, 8 × 8 pixels) in a predetermined unit. Therefore, block boundaries (block noise) are generated when decoding is performed according to the compression rate. Will occur. Even in other encoding processes, block noise occurs according to the compression rate when decoding.

  Therefore, in the present embodiment, a threshold is provided for the compression rate, and for example, the determination unit 21b performs determination to perform block noise removal processing for B frames and P frames that are more highly compressed than I frames. Accordingly, the determination unit 21b sets a block noise higher than the I frame as a block noise removal process target. Such a determination is made because the B frame and the P frame are more likely to generate block noise when decoded than the I frame.

  The determination unit 21b may set a threshold value for the file size of the frame. That is, the determination unit 21b determines that the frame is a highly compressed frame when the file size of the frame is equal to or smaller than the threshold value. Thereby, the determination part 21b can perform the determination which performs the removal process of a block noise about B frame and P frame higher compression than I frame.

  Further, the determination unit 21b excludes a frame having a resolution unsuitable for the super-resolution processing from the processing target. This determination is performed because the frame does not contribute to the super-resolution processing. Note that the frame having a resolution unsuitable for the super-resolution processing includes, for example, a low-resolution frame caused by blur or the like, a frame having a high compression rate as a frame used for the super-resolution processing, and the like.

  The noise removal unit 21c performs block noise removal processing for each frame according to the determination result of the determination unit 21b. Specifically, the noise removing unit 21c performs a known deblocking filter process (hereinafter referred to as “DF process”) for smoothing the block boundary in order to remove block noise. Thereby, the noise removal part 21c can smooth only a block boundary (block noise), and can suppress generation | occurrence | production of block noise.

  That is, the noise removal unit 21c performs DF processing on one to all frames according to the determination result of the determination unit 21b among the frames used for the super-resolution processing.

  Here, the location where block noise occurs (distorted boundary) can be estimated according to the compression format, and block noise is more likely to occur as the compression setting is higher. In this embodiment, block noise is generated. Apply DF processing under easy-to-use conditions. If the deblocking filter is applied to an unnecessary part, there is a possibility that the image quality may be deteriorated. Therefore, in the present embodiment, for example, in the case of an I frame, it is not necessary to apply DF processing. Thereby, the noise removal part 21c can perform DF processing, suppressing image quality degradation.

  The super-resolution processing unit 21d performs super-resolution processing on a plurality of frames to be processed including frames subjected to the removal processing, and generates a high-resolution still image. The details of the super-resolution processing will be described later with reference to FIG.

  The compression processing unit 21e performs compression processing on still image data or moving image data. The compression processing unit 21e performs compression processing on still image data in, for example, JPEG (Joint Photographic Experts Group) format. The compression processing unit 21e performs compression processing on the moving image data in, for example, MPEG format.

  Note that the compression format of moving image data employed in the present embodiment is not limited to the MPEG (for example, MPEG-1 to 4) format, and is, for example, a compression encoding method for moving image data recommended by the ITU (International Telecommunication Union). H. is one. H.264 or H.264 It may be 265.

  Alternatively, the compression format of moving image data employed in the present embodiment may be Motion-JPEG. In the case of Motion-JPEG, the compression rate is reduced by the amount of compression between frames, but since JPEG compression is performed for each frame, there is a possibility that block noise occurs depending on the compression rate. Therefore, the determination unit 21b may perform DF processing on all frames, for example. That is, the determination unit 21b can determine whether or not to perform block noise removal processing based on the compression rate and the compression format. Therefore, the compression format of the moving image data employed in the present embodiment is not particularly limited. In this embodiment, the I frame, the B frame, and the P frame may be an aspect of the compression format.

  The decoding processing unit 21f performs decoding processing by decompressing the compressed still image data and moving image data. The recording processing unit 21g records the still image generated by the super-resolution processing in the flash memory 14 or the recording medium M1.

  Next, the operation of the electronic camera 1 will be described.

  FIG. 2 is a diagram for explaining the outline of the super-resolution processing mode. FIG. 3 is a flowchart showing the operation of the electronic camera 1 under the super-resolution processing mode. The CPU 21 executes the processing of the flow shown in FIG. 3 based on the image processing program.

  In the processing of the flow shown in FIG. 3, moving image data that has been subjected to compression processing is a processing target. For example, the moving image data is captured by the electronic camera 1 and recorded on the recording medium M1. In the present embodiment, as an example, the plurality of frames to be processed are moving image data composed of frames in which the same subject is captured continuously in time with substantially the same angle of view.

  Here, the CPU 21 receives an instruction input for the super-resolution processing mode via the operation unit 18. Then, the CPU 21 shifts to the super-resolution processing mode, receives the moving image reproduction instruction input from the user via the operation unit 18, reads out the moving image data from the recording medium M 1, decodes it, and then displays it on the display monitor. 17 causes the moving image to be reproduced.

  Then, the user can specify an image to be subjected to super-resolution processing for a desired one scene while watching the reproduced moving image. At this time, the CPU 21 stops the operation in a desired one scene by receiving an instruction input for stopping the moving image reproduction from the user via the operation unit 18. Then, CPU21 starts the process of the flow shown below.

  Step S101: The selection unit 21a of the CPU 21 performs a frame selection process. Specifically, the selection unit 21a refers to the moving image data in order to perform a super-resolution process in step S105, which will be described later, and a frame that is the source of the image specified by the user and a plurality of frames before and after the frame. Select.

  FIG. 2A is a diagram for explaining frame selection processing. In FIG. 2A, for convenience of explanation, numbering (1 to 12) is performed from the nth frame during moving image reproduction. Also, I, B, and P in the figure each indicate whether the corresponding numbered frame is an I frame, a B frame, or a P frame.

  As an example, when the frame that is the basis of the image specified by the user (the image displayed on the display monitor 17) is the frame 3 in FIG. 2A, the selection unit 21 selects the compressed moving image data. (For example, frames 1 to 5) are read from the recording medium M1 to the RAM 13 and selected as the frames 1 to 5 to be processed. And CPU21 transfers to the process of step S102.

  Step S102: The determination unit 21b of the CPU 21 performs determination processing for the frame selected by the selection unit 21b. Specifically, the determination unit 21b analyzes frame information and determines whether to remove block noise for each frame. The frame information includes a compression rate and a compression format. Here, the determination unit 21b executes the flow of the determination process subroutine shown in FIG. FIG. 2B illustrates the determination process. When the flow of the subroutine of the determination process ends, the CPU 21 proceeds to the process of step S103.

  Step S103: The decoding processing unit 21f of the CPU 21 performs frame decoding processing. As an example, the decoding processing unit 21 f performs a decoding process based on the MPEG format on the frames 1 to 5 to be processed that are temporarily recorded in the RAM 13. And CPU21 transfers to the process of step S104.

  Step S104: The noise removal unit 21c of the CPU 21 performs DF processing as necessary. Specifically, the noise removing unit 21c refers to a flag to be described later and performs DF processing on the frame set by the determining unit 21b to perform DF processing.

  FIG. 2C illustrates a frame for performing DF processing. The upper frame represents the state before DF processing, and the frame in which block noise exists symbolically represents the block noise in a square shape. Therefore, actually, a plurality of block noises exist in a mosaic pattern.

  Further, the lower frame represents the state after the DF processing, and represents a state in which block noise is removed.

  That is, the noise removal unit 21c performs DF processing on the frames 2 to 5. And CPU21 transfers to the process of step S105.

  Step S105: The super-resolution processing unit 21d of the CPU 21 performs super-resolution processing. Specifically, the super-resolution processing unit 21d executes the process of the subroutine of the super-resolution process shown in FIG. FIG. 2D schematically shows that a high-resolution still image is generated by the super-resolution processing. When the processing of the super resolution processing subroutine flow ends, the CPU 21 proceeds to the processing of step S106.

  Step S106: The recording processor 21g of the CPU 21 performs a recording process. Specifically, a new still image generated by performing the super-resolution processing is recorded in the flash memory 14 or the recording medium M1. Then, the CPU 21 ends the processing of the flow shown in FIG.

  Next, the flow of the determination process subroutine will be described.

  FIG. 4 is a flowchart of a subroutine for determination processing.

  Step S201: The determination unit 21b performs analysis processing on the nth frame. Specifically, the determination unit 21b sequentially reads out the frames to be processed selected by the selection unit 21 and performs analysis processing. Depending on the analysis result, the CPU 21 proceeds to one of steps S202, S203, and S204.

  Here, with reference to FIG. 2B as an example, the determination unit 21b reads the first frame. In this case, since the first frame is an I frame, the determination unit 21b determines not to perform DF processing. Therefore, the CPU 21 proceeds to step S202.

  Step S202: The determination unit 21b sets the DF processing flag to OFF. Here, when the flag is off, it means that the noise removing unit 21c does not perform DF processing for a frame for which the flag is set to off. Then, the CPU 21 proceeds to step S205.

  Step S203: The determination unit 21b sets the DF processing flag to ON. Here, when the flag is on, it means that the noise removing unit 21c performs DF processing on the frame for which the flag is set to on. Then, the CPU 21 proceeds to step S205.

  Step S204: The determination unit 21b performs a process of excluding the nth frame from the processing target. That is, the determination unit 21b excludes a frame having a resolution inappropriate for the super-resolution processing from the processing target. Then, the CPU 21 proceeds to step S205.

  Step S205: The determination unit 21b determines whether there is a next frame. If there is no next frame (step S205: No), the CPU 21 returns to the process of step S103 shown in FIG. On the other hand, when there is a next frame (step S205: Yes), the CPU 21 proceeds to the process of step S206.

  Step S206: The determination unit 21b sets the next frame number (n = n + 1). And CPU21 returns to the process of step S201, and repeats the process after step S201 continuously. As an example, as shown in FIG. 2B, since the second to fourth frames are B frames, the determination unit 21b sets the DF processing flag to ON (see step S203). As an example, as shown in FIG. 2B, since the fifth frame is a P frame, the determination unit 21b sets the DF processing flag to ON (see step S203).

  However, in the case of FIG. 2B, the process of step S204 was not performed. However, as described above, in the case of a frame having a resolution inappropriate for the super-resolution process, the process proceeds to the process of step S204. The n-th frame is removed from the processing target.

  Next, the flow of the super-resolution processing subroutine will be described.

  FIG. 5 is a flowchart of a subroutine for super-resolution processing.

  Step S301: The super-resolution processing unit 21d performs alignment processing. Specifically, the super-resolution processor 21d performs alignment processing for a plurality of frames after the processing in step S104. At this time, the super-resolution processing unit 21d uses a low-resolution frame serving as a reference for alignment as a reference image.

  As an example, the super-resolution processing unit 21d uses the frame 3 after the DF processing illustrated in FIG. 2C as a reference image. Then, the super-resolution processing unit 21d performs alignment processing on the reference image using low-resolution images (frame 1, frames 2, 4, and 5 after DF processing) that are to be aligned. . Note that the alignment processing is not particularly limited as long as it is alignment used for super-resolution processing. The super-resolution processing unit 21d may apply a known region-based matching method, for example, as the alignment processing.

  Here, if block noise exists in the reference image or the low-resolution image to be aligned, the block noise becomes an object of alignment between frames, and the accuracy of alignment between subjects is significantly reduced. However, in this embodiment, since block noise is removed in advance, the alignment process can be performed with high accuracy. And CPU21 transfers to the process of step S302.

  Step S302: The super-resolution processor 21d performs high-resolution still image composition processing. Specifically, the super-resolution processing unit 21d generates a high-resolution still image by synthesizing the low-resolution image after the alignment process. The super-resolution processing unit 21d is not limited to a specific super-resolution process. For example, the well-known ML (Maximum-likelihood) method, MAP (Maximum A Posterior) method, or POCS (Projection Onto Convex Sets) method is used. The super-resolution processing can be performed using a method such as the above. And CPU21 returns to the process of step S106 shown in FIG.

  As described above, the electronic camera 1 according to the present embodiment generates a high-resolution still image on which super-resolution processing has been performed by the CPU 21 performing the processing shown in FIGS. 3 to 5 according to the image processing program. can do.

  Therefore, according to the present embodiment, it is possible to provide means capable of performing super-resolution processing with high accuracy even for moving image data that has been subjected to compression processing.

  Next, a modification of the first embodiment will be described.

  FIG. 6 is a flowchart showing a modification of the operation of the electronic camera 1 under the super-resolution processing mode. In the flowchart shown in FIG. 3, after the determination unit 21b performs frame determination processing (see step S102), the decoding processing unit 21f performs frame decoding (see step S103).

  On the other hand, in the modification of the first embodiment, as a difference in the flowchart shown in FIG. 3, after the decoding processing unit 21f decodes the frame (see step S402), the determination unit 21b performs the frame determination processing. The configuration is used (see step S403).

  Also in the modified example of the first embodiment, the CPU 21 receives an instruction input for stopping the moving image reproduction from the user via the operation unit 18, and stops it in a desired one scene. Then, the CPU 21 starts processing of the flow shown in FIG. Note that the processing of the flow shown in FIG. 6 will be described in a simplified manner because it is only partly different from the processing of the flow shown in FIG.

  Step S401: The selection unit 21a performs a frame selection process.

  Step S402: The decoding processor 21f performs a frame decoding process.

  Step S403: The determination unit 21b performs determination processing on the frame selected by the selection unit 21b. In this case, since each frame has already been decoded, the determination unit 21b performs DF processing based on the information of the frame before decoding (any one of the I, B, and P frames). It may be determined whether or not to perform.

  Step S404: The noise removing unit 21c performs DF processing as necessary.

  Step S405: The super-resolution processing unit 21d performs super-resolution processing.

  Step S406: The recording processing unit 21g performs a recording process. Specifically, a new still image generated by performing the super-resolution processing is recorded on the recording medium M1. Then, the CPU 21 ends the processing of the flow shown in FIG.

  As described above, according to the modification of the first embodiment, the electronic camera 1 performs super-resolution processing by the CPU 21 performing the processing of the flow shown in FIGS. 6, 4 and 5 according to the image processing program. The applied high-resolution still image can be generated.

In the super-resolution processing mode, one high-resolution still image is generated from a plurality of frames in accordance with user input. Here, as another super-resolution processing mode, for example, the CPU 21 automatically performs the processing of the flow shown in FIG. 3 or 6 to select a plurality of frames to be processed and perform super-resolution. It is good also as a structure which produces | generates the still image which performed the process. Alternatively, the CPU 21 automatically repeats the processing of the flow shown in FIG. 3 or FIG. 6, selects a plurality of frames to be processed, and generates a plurality of still images subjected to super-resolution processing in time series. It is also good.
(Second Embodiment)
Next, a second embodiment will be described. In the second embodiment, the image processing program according to the present invention is used by being incorporated in a computer.

  FIG. 7 is a block diagram illustrating a configuration example of the computer 100. The computer 100 includes a control unit 2, a recording I / F unit 3, a RAM 4, a recording device 5, an input / output interface unit (hereinafter referred to as “input / output I / F unit”) 6, and a bus 7. .

  Here, the control unit 2, the recording I / F unit 3, the RAM 4, the recording device 5, and the input / output I / F unit 6 are connected to each other via a bus 7.

  The recording I / F unit 3 can be detachably connected to the nonvolatile storage medium M1. On this recording medium M1, an image processing program and compressed moving image data photographed by the electronic camera 1 are recorded.

  The RAM 4 temporarily records the calculation result of the program. The RAM 4 is composed of, for example, a volatile SDRAM (Synchronous Dynamic Random Access Memory).

  The recording device 5 is composed of a recording medium such as a hard disk or a nonvolatile semiconductor memory. The recording device 5 stores an image processing program that is an embodiment of the present invention.

  The input / output I / F unit 6 is connected to an input device 8 (a pointing device such as a keyboard and a mouse) and a display unit (a liquid crystal monitor or the like) 9. The input / output I / F unit 6 accepts various inputs from the input device 8 and outputs display data to the display unit 9.

  The control unit 2 is a processor that controls the computer 100. The control unit 2 controls each unit of the computer 100 by executing an OS (operation system) stored in advance in the recording device 5.

  In addition, the control unit 2 stores the image processing program according to the embodiment of the present invention in the recording device 5 from the recording medium M1. This image processing program can be applied to the computer 100 as well as the electronic camera 1 described above. By storing this image processing program in the recording device 5, the control unit 2, as one aspect of the image processing device according to the present invention, includes a selection unit 2 a, a determination unit 2 b, a noise removal unit 2 c, and a super solution. And an image processing unit 2d. Thus, the computer 100 may be an embodiment of the image processing apparatus according to the present invention.

  The control unit 2 includes a compression processing unit 2e, a decoding processing unit 2f, and a recording processing unit 2g. Each of these units includes a selection unit 21a, a determination unit 21b, a noise removal unit 21c, a super-resolution processing unit 21d, a compression processing unit 21e, a decoding processing unit 21f, and a recording processing unit 21g of the electronic camera 1 of the first embodiment. Since it has the same function, description is abbreviate | omitted.

  Next, the operation of the computer 100 will be described. Here, the control unit 2 receives an instruction input for starting the image processing program via the input device 8. Then, the control unit 2 receives a moving image reproduction instruction input from the user via the input device 8, reads out the moving image data from the recording medium M 1, decrypts the moving image data, and causes the display unit 9 to reproduce the moving image. As in the first embodiment, the user can specify an image to be super-resolved with respect to a desired one scene while viewing the reproduced moving image. At this time, the control unit 2 receives an instruction input for stopping the moving image reproduction from the user via the input device 8, thereby stopping in a desired one scene. Then, the control part 2 starts the process of the flow shown in FIG. 3 or FIG. 6 according to the image processing program. In the computer 100, the control unit 2 performs the processing of the flow shown in FIG. 3 to FIG. 5 or FIG. 4 to FIG. Can be performed.

  As described above, the computer 100 according to the present embodiment has been subjected to the super-resolution processing by the control unit 2 performing the processing shown in FIGS. 3 to 5 or FIGS. 4 to 6 according to the image processing program. A high-resolution still image can be generated.

  The image processing apparatus according to the present invention may be incorporated in the computer 100 as a dedicated hardware circuit that performs the same processing as the processing of the flow shown in FIGS. 3 to 5 or 4 to 6.

(Supplementary items of the embodiment)
(1) In the above-described embodiment (first embodiment), a lens-integrated compact type electronic camera is exemplified, but an imaging apparatus to which the present invention can be applied is not limited to this type of electronic camera. That is, the imaging apparatus to which the present invention is applicable includes, for example, a single-lens reflex type electronic camera. The imaging apparatus to which the present invention is applicable includes an interchangeable lens type electronic camera that does not require a quick return mirror.

  The imaging apparatus to which the present invention can be applied also includes a mobile phone having a camera function and a personal digital assistant (PDA).

  (2) The moving image data in the above embodiment may include a group of images taken continuously.

  (3) In the above embodiment, DF processing is adopted to remove block noise, but other noise reduction filters may be adopted.

  For example, the noise removal unit 21c may use an average value filter (Gaussian filter or the like), a bilateral filter (Bilateral Filter), or the like. Even if these filters are used, the block noise can be smoothed. Further, the noise removing unit 21c may employ a combination of these plural filters selectively.

  (4) In the above embodiment (first embodiment), the determination unit 21b determines whether to perform block noise removal processing based on at least one of the compression rate and the compression format for each frame. did. Here, the determination unit 21b may further include means for detecting block noise to determine whether or not to perform block noise removal processing. As a means for detecting block noise, a known technique may be applied. For example, block noise may be detected from a level difference of pixel values at a boundary between blocks obtained by dividing the frame into a plurality of blocks. And the determination part 21b determines by performing a DF process to the location which detected block noise by applying the means to detect block noise. And the noise removal part 21c performs DF process to the location which detected the block noise.

  Thereby, the super-resolution processing unit 2d can perform the super-resolution processing with high accuracy even for the moving image data subjected to the compression processing. Similarly, in the second embodiment, the determination unit 2b may further include a means for detecting block noise to determine whether or not to perform block noise removal processing.

DESCRIPTION OF SYMBOLS 1 ... Electronic camera, 2 ... Control part, 21 ... CPU, 21a ... Selection part, 2b, 21b ... Determination part, 2c, 21c ... Noise removal part, 2d, 21d ..Super-resolution processing unit, 100 ... computer

Claims (8)

A selection unit that reads out video data compressed using inter-frame prediction and selects a plurality of frames to be processed;
Analyzing the information of the frame selected by the selection unit, and determining whether to remove noise for each frame;
A noise removal unit that performs the noise removal process for each frame according to the determination result of the determination unit,
The determination unit is configured to determine not to remove noise with respect to the first frame used in inter-frame prediction and removal of noise with respect to the first second frame highly compressed than the frame An image processing apparatus for determining . The image processing apparatus according to claim 1.
Further, an image including a super-resolution processing unit that generates a higher-resolution image than the frame by performing super-resolution processing on the plurality of frames to be processed including the frame subjected to the removal processing. Processing equipment. The image processing apparatus according to claim 1.
The determination unit is an image processing apparatus that determines whether or not to perform the removal process based on at least one of a compression rate and a compression format for each frame. The image processing apparatus according to claim 1.
The noise removal unit is an image processing apparatus that performs deblocking filter processing to remove block noise as the noise. The image processing apparatus according to claim 1.
The determination unit is an image processing apparatus that determines whether or not to perform the removal process according to a preset compression rate threshold for an I frame, a B frame, and a P frame used for the inter-frame prediction. The image processing apparatus according to claim 2,
The determination unit is an image processing apparatus that removes a frame having a resolution unsuitable for the super-resolution processing from the processing target. A compression processing unit that compresses moving image data generated by continuously capturing subject images;
The image processing apparatus according to any one of claims 1 to 6,
An imaging apparatus comprising: A selection step of reading the video data compressed using inter-frame prediction and selecting a plurality of frames to be processed;
Analyzing the information of the frame selected in the selection step, and determining whether to remove noise for each frame;
According to the determination result of the determination step, the computer performs a noise removal step of performing the noise removal processing for each frame,
The determination step, the determination not to remove noise with respect to the first frame used in inter-frame prediction and removal of noise with respect to the first second frame highly compressed than the frame An image processing program for determination .

Images