JP6552351B2 - Image processing apparatus, imaging apparatus, image processing method, program, and storage medium - Google Patents

Description

  The present invention relates to an imaging device capable of capturing a moving image.

  In general, a photographing apparatus such as a camera performs various image correction processes such as blur correction, distortion correction, chromatic aberration correction, and contour correction on a photographed image signal to generate a recorded image. Since this image correction process is a process executed for each pixel, it takes a lot of processing time. Patent Document 1 discloses an image motion correction device that limits image correction processing.

  In addition, an imaging device using a face recognition technique or the like is known for focusing on a main subject. Patent Document 2 discloses an image processing apparatus that improves the tracking of a recognized subject.

  Patent Document 3 has a plurality of light receiving elements corresponding to a plurality of microlenses, receives a light flux from an optical system with a plurality of light receiving elements, and outputs a plurality of light reception signals, and shifts an image plane by the optical system. An image combining apparatus is disclosed that generates a plurality of image signals with different focus based on an amount.

  In Patent Document 4, as a technique for recording an image with different shooting conditions as a moving image, a moving image stream is generated separately in the near focus state and the far focus state, and a repeat image is inserted to adjust the time axis for easy viewing. An imaging device for generating a moving image is disclosed.

Japanese Patent Laid-Open No. 11-183951 JP 2013-13037 A JP 2011-97645 A JP 2009-1001916 A

  However, as in the prior art, in an apparatus that captures a plurality of frame images with different focus at the same timing and records a plurality of moving images for each focus, various image correction processing is performed on the plurality of frame images at a predetermined timing. Need to be done within. However, if the same image correction processing is performed on all the captured frame images, a large processing time is required, and it becomes difficult to generate an image to be encoded within the processing frame rate of a moving image. Further, since the image correction processing is adaptively performed on the basis of the photographed image, the frame rate based on the worst time required for the correction processing becomes the maximum frame rate of the moving image.

  Therefore, the present invention provides an image processing device, an imaging device, an image processing method, and a program that can acquire a plurality of moving images at a high frame rate by changing the image correction processing of the plurality of frame images according to the focus state of the main subject. And a storage medium.

  An image processing apparatus according to an aspect of the present invention includes a signal processing unit that generates a plurality of frame images that are captured every predetermined period and that have different shooting conditions, and a prioritization that prioritizes the plurality of frame images. And an image correction unit that performs correction processing on the plurality of frame images, and the image correction unit changes the correction processing on the plurality of frame images in accordance with the priority.

  An imaging apparatus according to another aspect of the present invention includes an imaging unit that outputs a plurality of image data having different imaging conditions at predetermined intervals, and a signal processing unit that generates a plurality of frame images based on the plurality of image data. , Prioritizing means for prioritizing the plurality of frame images, and image correction means for performing correction processing on the plurality of frame images, wherein the image correction means is responsive to the priority order. The correction processing for the plurality of frame images is changed.

  An image processing method according to another aspect of the present invention includes a step of generating a plurality of frame images captured at predetermined intervals and having different shooting conditions, a step of assigning priorities to the plurality of frame images, Performing correction processing on a plurality of frame images, and performing the correction processing includes changing the correction processing on the plurality of frame images according to the priority.

  According to another aspect of the present invention, there is provided a program that generates a plurality of frame images that are captured every predetermined period and that have different shooting conditions, a step that prioritizes the plurality of frame images, A program that causes a computer to execute a correction process on a frame image, wherein the correction process changes the correction process on the plurality of frame images according to the priority order. Including.

  A storage medium according to another aspect of the present invention stores the program.

  Other objects and features of the invention are described in the following embodiments.

  According to the present invention, an image processing device, an imaging device, an image processing method capable of acquiring a plurality of moving images at a high frame rate by changing the image correction processing of a plurality of frame images according to the focus state of the main subject, A program and a storage medium can be provided.

It is a block diagram of an imaging device in each embodiment. It is a data flow figure of the recording operation by the imaging device in each embodiment. It is explanatory drawing of a production | generation of the moving image stream by the imaging device in each embodiment. FIG. 7 is a timing chart showing the relationship between the imaging cycle of the imaging apparatus and the image correction process period in each embodiment. In Embodiment 1, it is explanatory drawing of determination of the priority of the image correction process by the focus state of the main subject. 6 is a timing diagram of image correction processing according to priority in Embodiment 1. FIG. 7 is a flowchart showing an operation of image correction processing according to priority in the first embodiment. In Embodiment 2, it is a timing diagram of the image correction process according to a priority. In Embodiment 2, it is a flow chart which shows operation of image amendment processing according to a priority.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

First Embodiment
First, with reference to FIG. 1, an imaging apparatus according to a first embodiment of the present invention will be described. FIG. 1 is a block diagram of an imaging apparatus 100 according to this embodiment.

  In FIG. 1, reference numeral 200 denotes a recording medium such as a memory card which can be detached from the imaging apparatus 100. Reference numeral 101 denotes a DRAM (Dynamic Random Access Memory) as a main memory of the imaging apparatus 100. The DRAM 101 temporarily holds image data, encoded image data read from the recording medium 200, and the like. Reference numeral 102 denotes a data bus for accessing the DRAM 101. Reference numeral 103 denotes a lens unit including a lens for a photographer to photograph a subject (form a subject image as an optical image). In the present embodiment, the imaging apparatus 100 includes the lens unit 103 (imaging optical system) (that is, the imaging apparatus body and the lens unit are integrally configured), but is not limited thereto. . The present embodiment can also be applied to an imaging system including an imaging device body and a lens unit that can be attached to and detached from the imaging device body.

  An imaging unit (imaging element) 104 includes a microlens array configured by arraying a plurality of microlenses and a light receiving element array including a plurality of light receiving elements provided corresponding to the plurality of microlenses. . The imaging unit 104 receives the light flux from the lens unit 103 (optical image formed through the lens unit 103) by the plurality of light receiving elements through the corresponding microlens, and outputs a plurality of light reception signals. The imaging unit 104 generates a plurality of image signals with different focus based on the amount of image plane displacement by the imaging optical system (lens unit 103) when the received light signal is acquired. The image signal generated by the imaging unit 104 is temporarily held in the DRAM 101 as digital data (image data). In particular, in the present embodiment, the imaging unit 104 outputs a plurality of image data having different shooting conditions for each predetermined period.

  The signal processing unit 105 develops the image signal acquired by the imaging unit 104 and generates a plurality of frame images with different focus. A plurality of frame images generated by the signal processing means 105 are temporarily held in the DRAM 101. The subject detection unit 106 (face detection unit) detects a subject region (face region) from the frame image generated by the signal processing unit 105. The focusing determination unit 107 digitizes the high frequency component of the image of the subject area (face area) detected by the subject detection unit 106, and determines the focusing state. The image classification unit 108 classifies the plurality of frame images generated by the signal processing unit 105 into a plurality of image groups in accordance with the photographing condition (photographing focus condition). The moving image encoding unit 109 generates a multi-channel moving image stream using each of the plurality of image groups classified by the image classification unit 108.

  The panel unit 110 reads frame image data held in the DRAM 101. Then, the panel unit 110 displays a frame image on a display panel such as a liquid crystal element (Liquid Crystal Display) or an organic EL element (Organic Electro-Luminescence). A Media I / F 111 is an interface (I / F) with the recording medium 200 mounted in the imaging apparatus 100, reads image data from the DRAM 101, and records the read image data in the recording medium 200. The control bus 112 is used by the CPU 113 to issue a control instruction to each block of the imaging device 100.

The CPU 113 (control device) controls the entire imaging device 100. The program bus 114 connects the CPU 113 to a read only memory (ROM) 115, a random access memory (RAM) 116, and an operation key 117. The ROM 115 stores a program executed by the CPU 113.
Also, the following programs are stored in the ROM 115 and can be executed by the CPU 113. The RAM 116 is a work memory of the CPU 113. The operation key 117 is an operation unit such as a button or an arrow key for the user to issue an operation instruction of the imaging apparatus 100.

  The prioritizing means 118 prioritizes the importance of each frame image based on the focus determination result by the focus determination means 107. The image correcting unit 119 performs image correction processing according to the importance of the frame image by the prioritizing unit 118. Further, in the present embodiment, the ROM 115 stores a correction information generation program for generating correction information that associates the correction processing content with the frame image for each frame image corrected by the image correction unit 119. Further, the ROM 115 stores a correction information addition program for adding the correction information at a predetermined position of the moving picture stream of a plurality of channels generated by the moving picture coding unit 109.

  Next, with reference to FIG. 2, the recording operation by the imaging device 100 in the present embodiment will be described. FIG. 2 is a data flow diagram of the recording operation by the imaging device 100. As shown in FIG. In the present embodiment, the imaging apparatus 100 performs an imaging process, a correction / classification process, an encoding process, and a recording process, and the respective processes are performed in parallel. In FIG. 2, the photographed frame image buffer, the encoded image buffer and the additional information buffer of each group, and the stream buffer correspond to the memory area allocated to the DRAM 101.

  The imaging process is performed using the imaging unit 104 and the signal processing unit 105, and generates a plurality of captured frame images at each predetermined imaging timing. The correction / classification processing is performed by the object detection unit 106 and the focus determination unit 107, and the priority setting unit 118 calculates the correction level for each frame based on the focus determination result of each frame image. The image correction unit 119 performs image correction on the frame image stored in the shooting frame image buffer according to the calculated correction level, and the corrected frame image is set according to shooting conditions such as a focus distance. It transfers to the encoding image buffer of Groups 1-5. The image correction unit 119 transfers correction information accompanying the transferred frame image to the additional information buffer.

  The encoding process is performed using the moving picture encoding unit 109. When a frame image to be encoded is stored in the group 1 to 5 encoded image buffer, the moving image encoding unit 109 performs encoding processing so as to be a moving image stream for each group, and to the stream buffer corresponding to each group Output. Note that whether or not a frame image to be encoded is stored in each encoded image buffer is detected by the CPU 113 or the moving image encoding unit 109. In addition, with regard to the moving picture stream to be encoded by the moving picture encoding unit 109, Group 1 is encoded as a base view (main stream) in encoding Multiview Video Coding (MVC) of 3D video. Also, the groups 2 to 5 are encoded as non-base views (extent streams).

  The recording process is performed using the Media I / F 111. When a recordable moving picture stream is stored in the stream buffers of Groups 1 to 5, the Media I / F 111 performs a writing process on the recording medium 200 so as to be a file for each group. Note that the CPU 113 or the Media I / F 111 detects whether or not a recordable moving picture stream is stored in the stream buffer.

  As described above, imaging data is recorded as a moving image stream on the recording medium 200 by imaging processing, correction / classification processing, encoding processing, and recording processing via each data buffer.

  Next, generation of a moving image stream by the imaging apparatus 100 will be described with reference to FIG. FIG. 3 is an explanatory diagram of generation of a moving image stream in the imaging apparatus 100. In FIG. 3, the frame image is generated for each piece of focus information, and the vertical axis indicates the focus distance, and the horizontal axis indicates the imaging time. In the present embodiment, frame images corresponding to five focus distances F1 to F5 are captured at imaging timings T1 to T6 for each imaging cycle T, but the present invention is not limited to this.

  In the present embodiment, the frame images generated at each photographing timing are frame images (T1-F1 to T6-F5) of the focus distances F1 to F5 at the respective photographing timings T1 to T6. Further, the in-focus position of the main subject changes as in frame images T1-F3, T2-F2, T3-F1, T4-F2, T5-F3, and T6-F4. The in-focus determination unit 107 determines the in-focus state of the main subject based on the amount of high-frequency components of the main subject area for frame images generated at the same shooting timing. Further, the image classification unit 108 classifies each frame image into Group 1 to Group 5 in order of the image with the largest number of high frequency components in the main subject region to the smallest number of images.

  Under this condition, the generated video stream is a video stream for each of Groups 1 to 5. The video stream of Group 1 is a video stream (main video generated using the frame images T1-F3, T2-F2, T3-F1, T4-F2, T5-F3, and T6-F4 that are the focus positions of the main subject. Stream). The Group 2 moving image stream is a moving image stream (extent moving image stream) generated using the frame images T1-F2, T2-F1, T3-F2, T4-F1, T5-F2, and T6-F3. Similarly, the group 3 moving image stream is a moving image stream (extent moving image stream) generated using the frame images T1-F4, T2-F3, T3-F3, T4-F3, T5-F4, and T6-F5. Further, the group 4 moving image stream is a moving image stream (extent moving image stream) generated using the frame images T1-F1, T2-F4, T3-F4, T4-F4, T5-F5, and T6-F2. The group 5 moving image stream is a moving image stream (extent moving image stream) generated using the frame images T1-F5, T2-F5, T3-F5, T4-F5, T5-F1, and T6-F1. As described above, the frame images in the generated five moving image streams are streams for each focus state of the main subject. The movie stream of Group 1 is a movie stream in which the main subject is always in focus.

  Next, the relationship between the imaging period T by the imaging apparatus 100 and the image correction processing period H will be described with reference to FIG. FIG. 4 is a timing chart showing the relationship between the imaging cycle T by the imaging device 100 and the image correction processing period H (period of chromatic aberration correction, shake correction, distortion correction processing). Here, for all the frame images (T1-F1, T1-F2,..., T6-F5) generated at each photographing timing described with reference to FIG. A case will be described in which processing is performed in the order of chromatic aberration correction, blur correction, and distortion correction. However, the present embodiment is not limited to this.

  As shown in FIG. 4, with regard to the chromatic aberration correction processing, processing is performed on all of the frame images T1-F1, T1-F2, T1-F3, T1-F4, and T1-F5 captured at the imaging cycle T. In this case, the process can be completed within the imaging cycle T. On the other hand, the blur correction process and the distortion correction require a processing time longer than the imaging cycle T when processing is performed on all frame images. As a result, when the correction process is performed on the frame image captured in the imaging cycle T, the correction processing period H in which the correction process is completed is longer than the imaging cycle T. For this reason, the time for recording a moving image is a time that can be stored in the captured frame image buffer of the DRAM 101. Moreover, in order to realize long-time moving image shooting under these conditions, the imaging cycle T may be set to the correction processing period H or more, but the moving image frame rate is lowered.

  Next, with reference to FIG. 5 and FIG. 6, the determination of the priority order of the image correction process according to the focus state of the main subject and the timing of the image correction process according to the priority order will be described. FIG. 5 is an explanatory diagram of determination of the priority order of image correction processing according to the focus state of the main subject. FIG. 6 is a timing chart of the image correction process according to the priority order of the image correction process.

  In FIG. 5, as in FIG. 3, the frame image is generated for each piece of focus information, the vertical axis indicates the focus distance, and the horizontal axis indicates the imaging time. In the present embodiment, frame images corresponding to five focus distances F1 to F5 are captured at imaging timings T1 to T6 for each imaging cycle T, but the present invention is not limited to this. Further, in FIG. 5, as in FIG. 3, frame images generated at each imaging timing are frame images (T1 to F1 to T6 to F5) of focus distances F1 to F5 at imaging timings T1 to T6 respectively. Further, the in-focus position of the main subject changes as in frame images T1-F3, T2-F2, T3-F1, T4-F2, T5-F3, and T6-F4. The in-focus determination unit 107 determines the in-focus state of the main subject based on the amount of high-frequency components of the main subject area for frame images generated at the same shooting timing. Then, prioritizing means 118 (weighting means) prioritizes the image correction process in order of the image with the largest number of high frequency components in the main subject region to the smallest image.

  When prioritization of correction processing is performed under these conditions, each frame image is prioritized as follows. That is, the main (Group 1) moving picture stream is configured by the frame images T1-F3, T2-F2, T3-F1, T4-F2, T5-F3 and T6-F4 having the highest priority. Subsequently, an extent (Group2) moving image stream is composed of the frame images T1-F2, T2-F1, T3-F2, T4-F1, T5-F2, and T6-F3 with high priority. Subsequently, an extent (Group 3) moving image stream is configured by the frame images T1-F4, T2-F3, T3-F3, T4-F3, T5-F4, and T6-F5 with the highest priority. Subsequently, an extent (Group4) moving image stream is composed of the frame images T1-F1, T2-F4, T3-F4, T4-F4, T5-F5, and T6-F2 having a high priority. Further, the frame image T1-F5, T2-F5, T3-F5, T4-F5, T5-F1, and T6-F1 having the lowest priority form an extent (Group5) moving image stream.

  The image correction unit 119 performs an image correction process according to the priority determined by the priority setting unit 118. In the present embodiment, the image correction unit 119 performs an image correction process according to the correction priority as follows. That is, the image correction unit 119 performs, as image correction processing, the frame image (Group 1) having the highest priority (for priority 1) and the frame image (Group 2) having the next highest priority (for priority 2). Chromatic aberration correction, blur correction, and distortion correction are performed. The image correction unit 119 performs chromatic aberration correction and blur correction as image correction processing on the frame image (Group 3) having the next highest priority (priority 3). Further, the image correction unit 119 performs the chromatic aberration as an image correction process on the next highest priority (for priority 4) frame image (Group 4) and the lowest priority (for priority 5) frame image (Group 3). Make corrections.

  When the correction process is performed under this condition, the image correction process timing as shown in FIG. 6 is obtained. That is, the correction processing period H for the frame image captured in the period of the imaging cycle T can be shortened by not performing the correction process that requires a processing time for the frame having a low correction priority. As described above, in the present embodiment, for the frame image captured during the imaging cycle T, the priority of the image correction processing according to the focus state of the main subject is determined, and the image correction processing is changed according to the priority. To do. Thereby, the correction processing period H can be made shorter than the period of the imaging cycle T, and the imaging frame rate can be improved.

  Next, the operation of the imaging apparatus 100 will be described with reference to FIG. 7 regarding the image correction processing according to the priority order according to the focus state of the main subject described with reference to FIGS. 5 and 6. FIG. 7 is a flowchart illustrating the operation of the imaging apparatus 100 regarding the image correction processing according to the priority order according to the focus state of the main subject. Each step in FIG. 7 is mainly performed by each part (the object detection means 106, the focus determination means 107, the priority assigning means 118, the image correction means 119) of the imaging device 100 based on an instruction of the CPU 113 of the imaging device 100. To be executed.

  First, in step S701, the subject detection unit 106 detects a main subject area using a plurality (N) of frame images captured during the imaging cycle T. Then, the in-focus determination unit 107 determines a frame image (main-object in-focus frame image) in which the in-focus state of the main object area detected by the object detection unit 106 is the best.

  Subsequently, in step S702, the priority assigning unit 118 determines the priority of the image correction processing for the N frame images captured based on the main subject in-focus frame image determined in step S701. . That is, the prioritizing means 118 determines the priority of each frame image according to the distance from the main subject focused frame image. In step S703, the CPU 113 (image correction unit 119) initializes the value n of the image correction processing counter that counts the number of frame images subjected to the image correction processing to one. The value n of the image correction processing counter is associated with the captured N frame images on a one-to-one basis.

  Subsequently, in step S704, the image correction unit 119 determines whether the value n of the image correction processing counter is equal to or less than the number N of captured frame images. If the value N of the image correction processing counter is larger than the number N of frame images in step S704, the CPU 113 (image correction means 119) determines that the image correction processing for all the captured frame images is completed, End the flow On the other hand, if the value n of the image correction processing counter is equal to or smaller than the number N of frame images in step S704, the process proceeds to step S705. In step S 705, the image correction unit 119 determines whether the frame image associated with the value n of the image correction processing counter is a frame image with priority 2 or higher (that is, priority 1 or priority 2). Do.

  If it is determined in step S705 that the frame image is a frame image of priority 2 or higher (ie, priority 1 or priority 2), the flow proceeds to step S706. In step S706, the image correction unit 119 performs chromatic aberration correction, blur correction, and distortion correction as image correction processing. Then, in step S 707, the CPU 113 (image correction means 119) increments the value n of the image correction processing counter by one, and returns to step S 704. On the other hand, if it is determined in step S705 that the frame image is less than priority 2 (ie, priority is 3 to 5), the flow proceeds to step S708. In step S708, the image correction unit 119 determines whether or not the frame image associated with the value n of the image correction processing counter is a frame image of priority 3.

  If it is determined in step S708 that the frame image is a priority 3 frame image, the process advances to step S709. In step S709, the image correction unit 119 performs chromatic aberration correction and blur correction as image correction processing. Then, in step S 707, the CPU 113 (image correction means 119) increments the value n of the image correction processing counter by one, and returns to step S 704. On the other hand, if the frame image is not a priority 3 frame image (that is, a priority 4 or priority 5 frame image) in step S708, chromatic aberration correction is performed as an image correction process. Then, in step S 707, the CPU 113 (image correction means 119) increments the value n of the image correction processing counter by one, and returns to step S 704.

  As described above, in an imaging device that generates a plurality of frame images having different shooting conditions at the same shooting timing, the frame images are prioritized according to the focus state of the main subject at each shooting timing, and the priority order is set accordingly. Perform image correction processing. That is, sufficient image correction processing is performed on a focused frame image of a relatively important main subject, and image correction processing is reduced on a frame image having a relatively low degree of importance. As a result, it is possible to improve the frame rate in a plurality of moving image stream shootings, or to avoid a decrease in the frame rate.

Second Embodiment
Next, a second embodiment of the present invention will be described with reference to FIGS. 8 and 9. The present embodiment is different from the first embodiment in that the correction processing in the correction processing period H2 is changed based on the previous correction processing period H1 when performing the image correction processing according to the priority. The basic configuration and the basic operation of the imaging device in the present embodiment are the same as the configuration and the operation of the imaging device 100 of the first embodiment described with reference to FIG. To do.

  FIG. 8 is a timing chart of the image correction process according to the priority order of the image correction process depending on the focus state of the main subject. In FIG. 8, the priority order of the correction process and the type of correction process for the frame image captured at the imaging cycle T (shooting timings T1 and T2) are as follows. That is, the image correction unit 119 performs color aberration correction, blurring correction, and the like as image correction processing on the frame image with the highest priority (of priority 1) and the frame image of the next highest priority (with priority 2). And distortion correction is performed. Further, the image correction unit 119 performs chromatic aberration correction and shake correction as an image correction process on the next highest priority frame image (of priority 3). The image correcting unit 119 performs chromatic aberration correction as an image correction process on the frame image having the next highest priority (priority 4) and the frame image having the lowest priority (priority 5).

  When the correction process is performed under this condition, the image correction process timing as shown in FIG. 8 is obtained. The correction processing period H1 for the frame image captured during the period of the imaging timing T1 is longer than the imaging period T (imaging timing T1). In this case, in the present embodiment, the priority order and type of correction processing for the frame image captured in the next imaging period T (imaging timing T2) are changed. For example, the priority order and the type of correction processing after the change at the photographing timing T2 are as follows. That is, the image correction unit 119 performs chromatic aberration correction, shake correction, and distortion correction as image correction processing on the frame image with the highest priority (priority 1). The image correcting unit 119 performs chromatic aberration correction and image correction processing on the frame image having the next highest priority (priority 2) and the frame image having the second highest priority (priority 3). Perform image stabilization. The image correcting unit 119 performs chromatic aberration correction as an image correction process on the frame image having the next highest priority (priority 4) and the frame image having the lowest priority (priority 5).

  As described above, when the type of correction processing for the priority is changed, the image correction processing period H2 for a frame image captured during the imaging timing T2 becomes equal to or less than the imaging cycle T (imaging timing T2). As a result, it is possible to secure an image correction processing period H3 for a frame image captured at the next imaging cycle T (imaging timing T3).

  Next, with reference to FIG. 9, the operation of the imaging apparatus 100 will be described with regard to the image correction processing according to the priority according to the in-focus state of the main subject described with reference to FIG. FIG. 9 is a flowchart showing an operation of the imaging apparatus 100 regarding the image correction processing according to the priority according to the in-focus state of the main subject. Each step in FIG. 9 is mainly executed by the image correction unit 119 based on a command from the CPU 113 of the imaging apparatus 100. Further, the flow of FIG. 9 is executed for each shooting cycle T, that is, for each shooting timing T1, T2,..., Tn, Tn + 1,. In the image correction process corresponding to the priority order, the image correction process of the first embodiment described with reference to FIG. 7 is set as an initial value (standard image correction process).

  First, in step S901, the image correcting unit 119 performs image correction processing (standard image correction processing or first image processing) according to the priority order described with reference to FIG. 7 on the frame image corresponding to the set priority order. 1 image correction process). With respect to the correction processing period H executed at this time, the correction processing periods H corresponding to the photographing timings T1, T2,... Tn, Tn + 1,... Are H1, H2,. I assume.

  Subsequently, in step S902, the image correction unit 119 determines whether the correction processing period Hn corresponding to the photographing timing Tn (preceding photographing timing) executed in step S901 is equal to or more than the photographing cycle T. If the correction processing period Tn corresponding to the shooting timing Tn is equal to or longer than the shooting cycle T in step S902, the process proceeds to step S903. In step S 903, the image correction unit 119 changes (the content or type of) the correction processing according to the priority so that the image correction period Hn + 1 corresponding to the next shooting timing Tn + 1 becomes short. At this time, as described with reference to, for example, FIG. 8, the image correction unit 119 changes the type of the image correction process corresponding to each priority. That is, as the image correction process for the frame image of priority 2, only the chromatic aberration correction and the shake correction are performed (that is, the distortion correction is not performed). In the present embodiment, the image correction process at this time is referred to as a second image correction process.

  In the present embodiment, when the image correction process corresponding to the currently set priority order is the second image correction process, the content of the third image correction process is changed as the image correction process to be set next. Can. For example, when the currently set image correction process is the second image correction process, the third image correction process is set as follows. That is, the image correction unit 119 performs chromatic aberration correction and shake correction as an image correction process on the frame image with the highest priority (priority 1). The image correction unit 119 performs chromatic aberration correction as an image correction process on the frame image having the next highest priority (priority 2) and the frame image having the second highest priority (priority 3). Do. Further, the image correction unit 119 does not perform the image correction process on the next highest priority frame image (of priority 4) and the lowest priority frame image (of priority 5).

  As described above, in the present embodiment, when the immediately preceding image correction process is equal to or longer than the imaging cycle T, the content and type of the next image correction process are reduced (so that the performance of the image correction process is reduced) Set the next image correction process. For example, if the immediately preceding image correction process is a standard image correction process (first image correction process), the second image correction process is set as an image correction process to be executed during the next image correction process period. Similarly, when the immediately preceding image correction process is the second image correction process, the third image correction process is set as the image correction process to be executed during the next image correction process period. When the immediately preceding image correction process is the third image correction process, the image correction process is not performed during the next image correction process period. Further, when the image correction process is not performed in the immediately preceding image correction process period, the image correction process is not performed in the next image correction process period.

  On the other hand, if the immediately preceding image correction process is smaller than the shooting period T in step S902, the process proceeds to step S904. In step S904, the image correction unit 119 changes the content or type of the image correction process according to the priority order so that the performance of the image correction process at the next shooting timing Tn + 1 is improved. For example, when the immediately preceding image correction process is the standard image correction process (first image correction process), the standard image correction process (first image correction process) is performed as the image correction process executed in the next image correction process period. Set When the immediately preceding image correction process is the second image correction process, the standard image correction process (first image correction process) is set as the image correction process to be executed during the next image correction process period. Further, if the immediately preceding image correction process is the third image correction process, the second image correction process is set as the image correction process to be executed during the next image correction process period. When the image correction process is not performed in the immediately preceding image correction process period, the third image correction process is set as the image correction process to be executed in the next image correction process period.

  In the present embodiment, the image correction process for each frame image is dynamically changed, but the present invention is not limited to this. For example, the contents of the image correction processing for each frame image may be added to the moving picture stream and recorded in accordance with a correction information addition processing program executed by the CPU 113.

  As described above, in the present embodiment, when the image correction processing period for the frame image captured at the immediately preceding shooting timing is a period equal to or longer than the shooting cycle, the priority order at the next shooting timing and the content and type of the image correction processing are set. Dynamically change. As a result, it is possible to effectively avoid the frame memory overflow in shooting of a plurality of moving picture streams, and it is possible to stabilize the shooting frame rate.

  As described above, in each embodiment, the image processing apparatus (the imaging apparatus 100) includes the signal processing unit 105, the priority assigning unit 118, and the image correction unit 119. The signal processing means 105 generates a plurality of frame images which are photographed at predetermined intervals (photographing cycle T, photographing timing) and whose photographing conditions are different from each other. The prioritizing means 118 assigns priorities (calculates scores) for a plurality of frame images. The image correction unit 119 performs correction processing on a plurality of frame images. Further, the image correction unit 119 changes the correction processing for a plurality of frame images according to the priority.

  Preferably, the prioritizing means 118 assigns priorities according to the shooting conditions for a plurality of frame images. Preferably, the image correction unit 119 performs a first correction process of the first processing load on a first frame image having a first priority among the plurality of frame images. In addition, the image correction unit 119 has a second processing load smaller than the first processing load on a second frame image having a second priority lower than the first priority among the plurality of frame images. A second correction process is performed. Here, the processing load of the correction processing is a load corresponding to the number and strength of the correction processing, and the time required for the correction processing changes according to the processing load. In addition, preferably, the image processing apparatus includes a moving image encoding unit 109 that generates moving image data (moving image stream) of a plurality of channels based on a plurality of frame images for each predetermined period.

  Preferably, the image processing apparatus includes a subject detection unit 106 that detects a subject region from a plurality of frame images, and a focus determination unit 107 that determines the in-focus state of the subject region. Prioritizing means 118 assigns priorities according to the in-focus state. More preferably, the priority assigning unit 118 sets, as the first priority, the priority of the first frame image in which the in-focus state is the first in-focus state among the plurality of frame images. The prioritizing means 118 also sets the second frame image in which the in-focus state is in the second in-focus state with less high-frequency components than the first in-focus state to the second priority order lower than the first priority order. I assume.

  Preferably, as the correction processing, the image correction unit 119 performs shake correction processing on a plurality of frame images, and changes the strength of the shake correction processing according to the priority. For example, as the distance from the focus position of the frame image in which the main subject is in focus is increased, the strength (level) of the correction processing is decreased, or the search range during the blur correction processing is narrowed. Preferably, the image correction unit 119 performs distortion correction processing on a plurality of frame images as correction processing, and changes the strength of the distortion correction processing according to priority. At this time, the distortion correction processing may be performed only for the frame images having a predetermined priority order with reference to the frame image in focus on the main subject. Preferably, the image correction unit 119 performs chromatic aberration correction processing on a plurality of frame images, and changes the intensity of the chromatic aberration correction processing according to the priority. At this time, chromatic aberration correction may not be performed on a frame image having a low high-frequency component (that is, almost blurred).

  Preferably, the image correction unit 119 changes the correction processing method according to the priority order according to the processing time required for the correction processing. More preferably, the image correcting unit captures images at a second imaging timing after the first imaging timing based on a processing time of correction processing for a plurality of frame images captured at the first imaging timing as a predetermined period. A correction processing method for a plurality of frame images is determined. More preferably, the first shooting timing is a shooting timing immediately before the second shooting timing. Preferably, the image processing apparatus includes correction information generation means (CPU 113, ROM 115, correction information generation program) for generating correction information related to correction processing for each frame image. The image processing apparatus further includes correction information adding means (CPU 113, ROM 115, information addition) for adding correction information related to correction processing to moving image data (moving image stream) of a plurality of channels generated based on a plurality of frame images for each predetermined period. Program). By adding correction information related to the correction process to the moving image stream, it is possible to correct an uncorrected image during reproduction.

  Preferably, the signal processing unit 105 generates a plurality of frame images having different focus positions (focus distance or focus distance) as imaging conditions. More preferably, the priority assigning unit 118 sets a frame image closest to the main subject and the in-focus position among the plurality of frame images as the highest priority frame image. Further, the image correction means 119 makes the strength of the correction processing for the top priority frame image stronger than the strength of the correction processing for the other frame images excluding the top priority frame image.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  According to each embodiment, a plurality of frame images having different shooting conditions (focusing conditions) in a predetermined period are prioritized according to the focusing condition of the main subject, and each frame image is imaged according to the priority order. Change the correction process. As a result, a plurality of moving images having a higher frame rate can be recorded simultaneously. That is, according to each embodiment, an image processing device, an imaging device, and an image processing capable of acquiring a plurality of moving images at a high frame rate by changing the image correction processing of the plurality of frame images according to the focus state of the main subject. A method, program, and storage medium can be provided.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

100 Imaging device (image processing device)
105 Signal processing means 118 Prioritizing means 119 Image correction means

Claims (18)

Signal processing means for generating a plurality of frame images which are photographed at predetermined intervals and which have different photographing conditions;
Priority ordering means for prioritizing the plurality of frame images;
Image correction means for performing correction processing on the plurality of frame images,
The image processing apparatus, wherein the image correction unit changes the correction processing on the plurality of frame images in accordance with the priority.   The image processing apparatus according to claim 1, wherein the prioritizing unit prioritizes the plurality of frame images according to the photographing condition. The image correction unit is configured to, of the plurality of frame images,
Performing a first correction process of a first processing load on a first frame image in which the priority is a first priority;
A second correction process of a second processing load smaller than the first processing load is performed on a second frame image having a second priority which is lower than the first priority. The image processing apparatus according to claim 1 or 2.   The image processing apparatus according to any one of claims 1 to 3, further comprising: moving image encoding means for generating moving image data of a plurality of channels based on the plurality of frame images for each predetermined period. Subject detection means for detecting a subject region from the plurality of frame images;
Focusing determination means for determining a focusing state of the subject area;
The image processing apparatus according to any one of claims 1 to 4, wherein the priority assigning unit assigns the priority according to the in-focus state. The prioritizing unit is configured to, of the plurality of frame images,
The priority of the first frame image in which the in-focus state is the first in-focus state is set as a first priority,
Setting the priority of the second frame image in which the in-focus state is a second in-focus state in which the high-frequency component is smaller than that of the first in-focus state to a second priority lower than the first priority; The image processing apparatus according to claim 5, characterized in that The image correcting means includes
As the correction processing, shake correction processing is performed on the plurality of frame images;
The image processing apparatus according to any one of claims 1 to 6, wherein the strength of the shake correction process is changed according to the priority. The image correcting means includes
As the correction processing, distortion correction processing for the plurality of frame images is performed,
The image processing apparatus according to any one of claims 1 to 7, wherein the strength of the distortion correction process is changed according to the priority. The image correction means performs chromatic aberration correction processing on the plurality of frame images,
The image processing apparatus according to any one of claims 1 to 8, wherein the intensity of the chromatic aberration correction process is changed according to the priority.   The image processing apparatus according to claim 1, wherein the image correction unit changes a correction processing method according to the priority order according to a processing time required for the correction processing. .   The image correcting unit is photographed at a second photographing timing after the first photographing timing based on a processing time of the correction processing for a plurality of frame images photographed at the first photographing timing as the predetermined period. The image processing apparatus according to claim 10, wherein the correction processing method for a plurality of frame images is determined. Correction information generating means for generating correction information related to the correction processing for each frame image;
4. A correction information adding unit for adding correction information related to the correction process to moving image data of a plurality of channels generated based on the plurality of frame images for each predetermined period. 11. The image processing device according to any one of 11.   The image processing apparatus according to any one of claims 1 to 12, wherein the signal processing unit generates the plurality of frame images having different in-focus positions as the imaging conditions. Among the plurality of frame images, the prioritizing means sets a frame image closest to the main subject and the in-focus position as a top priority frame image.
14. The image processing method according to claim 13, wherein the image correction means makes the strength of the correction processing for the top priority frame image stronger than the strength of the correction processing for other frame images excluding the top priority frame image. The image processing apparatus described. Imaging means for outputting a plurality of image data having different shooting conditions for each predetermined period;
Signal processing means for generating a plurality of frame images based on the plurality of image data;
Priority ordering means for prioritizing the plurality of frame images;
Image correction means for performing correction processing on the plurality of frame images,
The image pickup apparatus, wherein the image correction unit changes the correction processing on the plurality of frame images in accordance with the priority. Generating a plurality of frame images captured at predetermined intervals and having different imaging conditions;
Prioritizing the plurality of frame images;
Performing a correction process on the plurality of frame images;
The image processing method according to claim 1, wherein the step of performing the correction process includes the step of changing the correction process on the plurality of frame images according to the priority order. Generating a plurality of frame images captured at predetermined intervals and having different imaging conditions;
Prioritizing the plurality of frame images;
A program for causing a computer to execute a correction process on the plurality of frame images.
The program according to claim 1, wherein the step of performing the correction process includes the step of changing the correction process on the plurality of frame images according to the priority.   A storage medium storing the program according to claim 17.