JP2016082474A - Image processing system, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system which can improve visibility of a moving image reproduced in search reproduction.SOLUTION: An image processing system includes: image data generation means for generating image data in response to a signal from an imaging element including a unit pixel consisting of a plurality of pixels allocated to a plurality of microlenses; frame image generation means for generating a frame image for reproducing a moving image in response to data of a prescribed region, which is read from the unit pixel in the image data; and change means for changing a reading region in the unit pixel in accordance with a reproduction mode on reproduction of the moving image in the frame image generation means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus using light field processing technology, and more particularly to an image processing apparatus that generates an image whose focal length can be changed after shooting.

  In recent years, a camera called a light field camera has been proposed as a camera that can change the focus position after photographing. This camera acquires light ray information including angle information indicating an incident angle in addition to position information indicating a position where a light ray from a subject is incident on the imaging surface. This ray information is called light field data (hereinafter referred to as LF data).

  In Non-Patent Document 1, a large number of microlenses are arranged between the main lens and the image sensor, and the incident light beam is dispersed by the microlens and each dispersed light beam is received by the image sensor. An imaging apparatus that acquires LF data is disclosed. Non-Patent Document 1 discloses a refocus technique for generating an image at a focus position different from that at the time of photographing by calculating image data on a virtual imaging plane using LF data.

  Patent Document 1 discloses an imaging device using a two-dimensional imaging element in which one microlens and a plurality of divided photoelectric conversion units are formed for one pixel. The divided photoelectric conversion units are configured to receive light that has passed through different pupil partial regions of the exit pupil of the photographing lens through one microlens. A plurality of parallax images corresponding to the divided pupil partial regions can be generated by the photoelectric conversion signals from each of the divided photoelectric conversion units. The focus position can be changed after shooting by performing a composition process on a portion to be focused using the plurality of obtained parallax images.

US Pat. No. 4,410,804

Ren Ng et al. “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  The light field processing technique as described above can also be applied to moving image shooting. In video format playback, search playback such as fast forward and rewind, which is not possible in still image format playback, can be performed. The light field image may be refocused by the user, and a refocus image with a shallow depth of field is often used as the light field image. For this reason, when a moving image is searched and reproduced with an image having a shallow depth of field, there is a problem that a user cannot easily find a desired scene because of poor visibility.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an image processing apparatus capable of improving the visibility of a moving image reproduced by search reproduction.

  In order to achieve the above object, according to the image processing apparatus of the present invention, image data is obtained based on a signal from an image sensor including a unit pixel composed of a plurality of pixels assigned to each of a plurality of microlenses. In the image data generating means for generating, the frame image generating means for generating a frame image for moving image reproduction based on the data of the predetermined area read from the unit pixel in the image data, and the frame image generating means And changing means for changing a reading area in the unit pixel in accordance with a playback mode related to playback of a moving image.

  According to the present invention, it is possible to improve the visibility of a moving image to be reproduced during search reproduction.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. (A) is a figure which shows the positional relationship of a main lens, a micro lens, and an image pick-up element, (b) is a figure which shows the correspondence of a micro lens and the pixel contained in an image pick-up element. It is a schematic diagram which shows an optical path. It is the schematic diagram modeled about the main lens and the image pick-up element. It is the schematic diagram which made the simple model about the main lens surface, the image pick-up element surface, and the refocus surface. It is a figure explaining the read-out area | region of LF data at the time of search reproduction | regeneration. It is a flowchart which shows the image processing which concerns on the 1st Embodiment of this invention. It is a figure explaining the production | generation of the frame image in refocus reproduction | regeneration mode and search reproduction | regeneration mode. It is a block diagram which shows the structure of the image processing apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the image processing which concerns on the 2nd Embodiment of this invention. It is a figure explaining the frame addition of LF data at the time of search reproduction.

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the drawings.
[First Embodiment]
An image processing apparatus according to a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment.

  The image processing apparatus 100 includes a main lens 101 as a photographing lens, a microlens array 102, an image sensor 103, an LF data input unit 104, a video output unit 109, a display panel 110, and a refocus processing unit 114. The image processing apparatus 100 further includes a memory 116, a media I / F 117, a recording medium 118, a controller 119, an operation unit 120, a memory bus 121, and a search image processing unit 122.

The main lens 101 is an imaging optical system composed of a single lens or a plurality of lenses, and light of a subject incident from the optical axis Z direction (left side in FIG. 1) is emitted to the microlens array 102.
The microlens array 102 is composed of a plurality of microlenses (not shown), and is disposed between the main lens 101 and the image sensor 103. The light incident from the main lens 101 is dispersed by the microlens array 102 according to the incident angle and is emitted to the image sensor 103. Specifically, the micro lens array 102 is disposed in the vicinity of the focal position of the main lens 101. Light that has passed through different pupil regions of the main lens 101 is divided and emitted for each pupil region.

  The imaging element 103 is a CCD sensor or a CMOS sensor in which a plurality of pixels (photoelectric conversion elements) that optically convert light incident from the microlens array 102 into electrical signals are arranged in a matrix. The image sensor 103 outputs an electrical signal generated by photoelectric conversion as an image signal to the LF data input unit 104 serving as an image data generation unit.

  The LF data input unit 104 converts the image signal input from the image sensor 103 into a digital signal and performs development processing for converting the image signal into a predetermined image file format such as JPEG for each frame of the image frame. As LF data. The LF data input unit 104 stores the converted LF data in an LF data area serving as a buffer area in the memory 116. The development process is a process for changing or adjusting the image quality of the digital image data using parameters relating to the digital image data such as color correction, resolution, and image compression.

  The video output unit 109 reads the refocus video from the video signal area of the memory 116 and outputs it to the display panel 110 and a video output terminal (not shown). The display panel 110 displays the video signal input from the video output unit 109 as a video. On the display panel 110, a refocus image being shot is displayed during shooting, and an image of LF data corresponding to the playback mode is displayed during moving image playback.

  The refocus processing unit 114 as a frame image generation unit performs refocus processing by reading out LF data stored in the LF data area of the memory 116 by the media I / F 117 in the refocus playback mode as the normal playback mode. The video data generated by the refocus processing is stored in the video signal area of the memory 116.

  The refocus playback mode is a playback mode for the user to view a moving image. In the refocus playback mode, playback is typically performed at a playback speed comparable to that during shooting. The search reproduction mode is a reproduction mode including an operation for a user to search for a desired scene in a moving image, such as fast forward and rewind. In the search playback mode, moving image playback is performed at a higher speed than the moving image playback speed in the refocus playback mode. High-speed playback in the search playback mode is typically performed by thinning out a part of images constituting a moving image.

  The memory 116 includes a dynamic RAM that can be randomly accessed at high speed, and is used as a work memory. The memory 116 is used as, for example, an LF data area or a video signal area. The memory 116 includes a ROM, and various programs and data for operating the controller 119 are stored in the ROM.

  The media I / F 117 is an interface that controls reading and writing of data with respect to the recording medium 118. At the time of shooting, the media I / F 117 reads out the LF data stored in the LF data area of the memory 116 by the LF data input unit 104, that is, the LF data being shot by the LF data input unit 104 in response to a command signal from the controller 119, Store. During normal playback, the media I / F 117 reads all LF data recorded from the recording medium 118 in response to a command signal from the controller 119 and stores it in the LF data area of the memory 116. At the time of search reproduction, the media I / F 117 reads the LF data recorded on the recording medium 118 in response to a command signal from the controller 119 every predetermined frame (for example, every 4 frames for 4 × speed reproduction), and the memory 116 Stored in the LF data area.

  The recording medium 118 is a recording medium such as a hard disk drive or a nonvolatile semiconductor memory (for example, a flash memory). The recording medium 118 records data in a file system format such as FAT (File Allocation Table).

  The controller 119 serving as a changing unit is a controller that controls the entire system of the image processing apparatus 100, and is a so-called CPU (Central Processing Unit). The controller 119 controls the operation mode of the image processing apparatus 100 to a shooting mode for shooting, a refocus playback mode for normal playback, a search playback mode for high speed playback, and the like based on a user operation on the operation unit 120.

  The operation unit 120 is an operation unit operated by a user. For example, a shutter button, a recording button for capturing a moving image, a playback button for reproducing the captured moving image, buttons such as a setting button for performing various settings, and the surface of the display panel 110 The touch panel sensor etc. which are arrange | positioned. The user can change the shooting mode / refocus playback mode / search playback mode by operating the operation unit 120. When the user operates the operation unit 120, various operation signals are transmitted from the operation unit 120 to the controller 119.

  The LF data input unit 104, the video output unit 109, the refocus processing unit 114, the memory 116, the media I / F 117, the controller 119, and the search image processing unit 122 are connected to the memory bus 121. The LF data input unit 104, video output unit 109, refocus processing unit 114, media I / F 117, controller 119, and search image processing unit 122 write / read data to / from the memory 116 via the memory bus 121. To do. The memory bus 121 adjusts memory access requests from each component and controls to read / write data from / to the memory 116 in a time division manner.

  When the operation mode of the image processing apparatus 100 is the high-speed playback mode, the search image processing unit 122 serving as a frame image generation unit selects pixels in a predetermined area from the LF data stored in the LF data area of the memory 116. Read out to generate a frame image. The video data generated by the search reproduction process is stored in the video signal area of the memory 116.

  Next, the operations of the main lens 101, the microlens array 102, the image sensor 103, and the LF data input unit 104 at the time of shooting will be described with reference to FIGS. 2 (a), 2 (b), 3, and 4. explain.

  FIG. 2A is a schematic diagram showing the positional relationship between the main lens 101, the microlens array 102, and the image sensor 103. As shown in FIG. 2A, the microlens array 102 is located between the main lens 101 and the image sensor 103, and has a plurality of microlenses 102a arranged in a matrix. Note that the size and the number of the microlenses 102a in the present embodiment are not limited to those illustrated.

  FIG. 2B is a diagram illustrating a correspondence relationship between the microlens 102 a and the pixel 103 a included in the imaging element 103 as viewed from the optical axis Z direction. As shown in FIG. 2B, n × m pixels 103a in the image sensor 103 are arranged so as to correspond to the microlens 102a. The number of divided light rays is determined by the number of pixels 103a corresponding to one microlens 102a (ie, n × m). FIG. 2B shows a state in which 5 × 5 pixels 103a are arranged for one microlens 102a as an example. Hereinafter, the n × m pixels 103a are referred to as unit pixels.

  FIG. 3 is a schematic diagram showing an optical path to the main lens 101, the micro lens 102a, and the image sensor 103. As shown in FIG. The light emitted from the point 307 of the subject 305 enters the main lens 101 as a light beam. The incident light beam is condensed by the main lens 101 and projected onto the imaging surface 308 on the surface of the micro lens 102a. The projected light is dispersed by the microlens 102 a according to the incident angle of the light and received by the pixels 321 to 325 of the image sensor 103. Light received by the pixels 321 to 325 is photoelectrically converted and output as an imaging signal.

  The light received by the pixel 321 is a light flux from the optical path 311. Similarly, light received by the pixel 322 is an optical path 312, light received by the pixel 323 is optical path 313, light received by the pixel 324 is optical path 314, and light received by the pixel 325 is a light flux from the optical path 315. is there. That is, the light received by the pixels 321 to 325 is light emitted from the same point 307, but is received by different pixels depending on the incident angle of the light beam.

  The imaging signal output from the imaging element 103 is input to the LF data input unit 104. The LF data input unit 104 digitizes the imaging signal, performs development processing to convert it into LF data, and stores it in the LF data area of the memory 116. Thereafter, the LF data stored in the memory 116 is recorded on the recording medium 118 by the controller 119.

  Next, signal processing during refocus reproduction will be described. The controller 119 reads the data of all the pixels for one frame of the LF data stored in the recording medium 118 via the media I / F 117 and stores it in the LF data area in the memory 116. The refocus processing unit 114 performs refocus processing based on the LF data stored in the LF data area of the memory 116, and stores the frame image data in the frame image data area of the memory 116. The refocus process will be described later.

  The above operation is continuously performed, and frame image data corresponding to the number of frames that can be processed as moving image data is stored in the frame image data area of the memory 116. The video output unit 109 reads out the frame image data from the frame image data area of the memory 116 and generates a moving image, thereby generating a moving image signal. The generated moving image signal is output to the display panel 110, and a moving image is displayed on the display panel 110.

  Next, generation of a refocus image performed in the refocus processing unit 114 will be described. The generation of the refocus image is specifically performed as follows based on the “Light Field Photography” method.

First, the function L _F (x, y, u, v) representing the ray information will be described. FIG. 4 is a schematic diagram showing a simplified positional relationship between the main lens 101 and the image sensor 103. The coordinate axes of the main lens 101 are u and v, and the coordinate axes of the image sensor 103 are x and y. Let F be the distance between the main lens 101 and the image sensor 103. At this time, the light beam L passing through the main lens 101 and the image sensor 103 is defined by a four-dimensional function L _F (x, y, u, v).

Next, a calculation method in refocus image generation will be described. FIG. 5 is a schematic diagram illustrating the main lens surface, the refocus surface, and the imaging surface of the microlens in a simplified manner. FIG. 5 shows the positional relationship between the lens surface 501, the refocus surface 502 determined by the refocus coefficient α, and the imaging surface 503 of the microlens. In FIG. 5, the position of the pupil division area of the main lens surface 501 is coordinates (u, v), the pixel position on the refocus plane is coordinates (x ′, y ′), and the position of the microlens imaging surface is coordinates (x , Y), the distance from the main lens surface 501 to the refocus surface 502 is F ′. The light beam L enters from the point u on the main lens surface 501, passes through the point x ′ on the refocus surface 502, and reaches the point x on the imaging surface 503. Here, since the straight ray L passes through the point x ′ on the refocus plane 502 and the point x on the imaging surface 503 of the microlens, x = u + (x′−u) / α. . Similarly, a light reception signal on the imaging surface 503 of the microlens of a light ray that enters from the coordinates (u, v) of the main lens surface 501 and passes through the coordinates (x ′, y ′) of the refocus surface 502 is a light beam L. The function L _F (x, y, u, v) that represents can be expressed as shown in Equation (1).

Further, the refocus image E _{F ′} (x ′, y ′) obtained at the coordinates (x ′, y ′) of the refocus surface 502 is obtained by converting L _F (x, y, u, v) to the main lens surface 501. Since it is integrated with respect to the u-axis and the v-axis, it can be expressed as equation (2). That is, the refocus image E _{F ′} (x ′, y ′) is obtained by integrating the function L _F (x, y, u, v) with respect to the pupil division region of the main lens 101. The refocus image is generated by the calculation process of Expression (2). Note that the integration range of the coordinates (u, v) of the main lens surface 501 that is the integration coefficient of Expression (2) is the entire main lens surface 501.

  The refocus processing unit 114 receives a preset integration range of the coordinates (u, v) of the main lens surface 501 by using the LF data read from the LF data area of the memory 116, and refocuses from the equation (2). Calculate the focus image. In the first refocus process, the refocus coefficient α is set so that the center position of the image becomes the in-focus position, for example. In the second and subsequent refocus processing, a refocus coefficient α is set at a location selected by the user, and a refocus image is calculated from Equation (2).

  Next, signal processing during search reproduction will be described. FIG. 6 is a diagram for explaining a read area of LF data at the time of search reproduction. The controller 119 reads out a part of one frame of LF data stored in the recording medium 118 via the media I / F 117 and stores it in the LF data area in the memory 116. The area of the LF data to be read out may be only LF data corresponding to the central pixel of each microlens 102.

  For example, as shown in FIG. 6, the controller 119 reads LF data corresponding to the pixel 601 indicated by the white frame, and does not read LF data corresponding to the pixel 602 indicated by the hatched line. As shown in FIG. 6, the controller 119 reads out data of 5 pixels at the center of 25 pixels (5 × 5) constituting the unit pixel arranged for one microlens 102 a, A frame image with a deep depth of field is created. Among the unit pixels arranged for each microlens 102a, a plurality of pixels at the center have a focus state of light rays that have passed through the optical axis of the microlens 102a rather than pixels corresponding to the peripheral edge of the microlens 102a. It is good. The controller 119 restricts reading of pixel data corresponding to the peripheral portion of the microlens 102a, and reads data of a pixel at the center of the unit pixel corresponding to the optical axis of the microlens 102a. The search image processing unit 122 generates a frame image having a deep depth of field based on the read pixel data. Reading out the data of the pixel at the center of the unit pixel corresponds to, for example, reducing the lens aperture (reducing the lens aperture). The search image processing unit 122 stores the generated frame image data in the frame image data area of the memory 116.

  When the frame image data of the number of frames that can be processed as moving image data is stored in the frame image data area of the memory 116, the video output unit 109 reads out the frame image data from the frame image data area of the memory 116 and performs moving image processing. Generate a signal. The generated moving image signal is output to the display panel 110, and a moving image is displayed on the display panel 110.

  Next, frame images generated in the refocus playback mode and search playback mode will be described. FIG. 7 is a flowchart illustrating image processing according to the present embodiment, and FIG. 8 is a diagram illustrating frame image generation in the refocus playback mode and the search playback mode. The flowchart shown in FIG. 7 is performed when the operation unit 120 is operated by the user. FIG. 8 shows a case where the operation mode transitions from the refocus playback mode 805 to the 4 × speed search playback mode 806 and then transitions to the refocus playback mode 807. In FIG. 8, as an example, the pixel image of the image sensor 103 is shown only in a region corresponding to two microlenses, but it goes without saying that the image sensor 103 is applied to the entire image sensor 103. Further, LF data 811 to 825 shown in FIG. 8 is LF data stored in the recording medium 118. Note that the processing shown in FIG. 7 is performed until reproduction of a moving image is stopped by a user operation.

  In step S701, the controller 119 determines the type of playback mode for playing back the moving image selected by the user. If the playback mode is the refocus playback mode (step S701: refocus playback mode), the controller 119 determines that the playback is refocus playback and proceeds to step S702.

  In step S702, the refocus processing unit 114 reads LF data and performs refocus processing. Specifically, the controller 119 reads out data of all pixels as a reading area in the LF data stored in the recording medium 118 via the medium I / F 117 and stores the data in the LF data area of the memory 116. The refocus processing unit 114 reads the LF data stored in the LF data area of the memory 116, and calculates a frame image that is a refocus image using Expression (2).

  As illustrated in FIG. 8, in the refocus playback mode 805, the refocus processing unit 114 performs a refocus process on all the pixels of the LF data 811 to generate a frame image 851. Similarly, refocus processing is performed using the next LF data 812 to generate a frame image 852. Thereby, a frame image with a deep depth of field can be obtained. In FIG. 8, the readout area read by the controller 119 is shown in white, and the pixel area not read out by the controller 119 is shown by diagonal lines.

  On the other hand, in step S701, if the playback mode is the search playback mode (step S701: search playback mode), the controller 119 determines that it is search playback and proceeds to step S703.

  In step S703, the controller 119 reads out pixel data of a predetermined area of the LF data. Specifically, the controller 119 reads out data in a predetermined area from each of the unit pixels for every predetermined number of frames in the LF data stored in the recording medium 118. The controller 119 skips the LF data included in the predetermined number of frames along the time axis, for example, reads out data of a predetermined pixel area of a unit pixel as a reading area from the LF data for the last one frame. And stored in the LF data area in the memory 116. The reading area read by the controller 119 is a pixel area corresponding to the center of the microlens 102a.

  In step S <b> 704, the search image processing unit 122 generates a frame image from the LF data stored in the LF data area of the memory 116. The search image processing unit 122 reads out the pixel data of the LF data stored in the memory 116 in step S703, and generates a frame image having a deep depth of field. The number of frame images generated by the controller 119 is the number of frames corresponding to the playback speed set in the search playback mode.

  As shown in FIG. 8, for example, when the search reproduction mode 806 is used to search and reproduce a moving image at a quadruple speed, the controller 119 has three LF data 813 to 815 among the LF data 813 to 816 corresponding to four imaging frames. LF data for the frame is not read. Then, the controller 119 reads out the LF data 816 of the fourth frame. At this time, in the LF data 816, the controller 119 reads out five pixels at the center out of 25 pixels (5 × 5) as unit pixels as a reading area. In each unit pixel, the readout area read by the controller 119 is the same area. In step S703, the search image processing unit 122 reads pixel data from the LF data 816, and generates a frame image 853 having a deep depth of field. A frame image 853 having a deep depth of field can be generated by reading out data of five pixels at the center of the microlens 102 in a favorable focus state and adding the images of the respective pixels.

  The controller 119 performs the same operation during the search playback mode 806, does not read out the LF data 817 to 819 corresponding to the imaging frame in the time axis direction, and in the LF data 820, the five pixels serving as the center of the microlens 102a. Read the data. The search image processing unit 122 generates a frame image 854 having a deep depth of field from the pixel data read from the LF data 820. As described above, in the search reproduction mode 806, the processing time for generating a frame image is shortened by thinning out a predetermined number of LF data compared to the refocus reproduction mode 805, so that smooth moving image reproduction can be performed. it can.

  In step S <b> 705, the refocus processing unit 114 or the search image processing unit 122 stores the generated frame image data in the frame image area in the memory 116. In the subsequent step S706, the controller 119 determines whether a predetermined number of frame image data stored in the frame image area in the memory 116 has been stored. When the predetermined number of frame image data is stored (step S706: Yes), the controller 119 determines that the number of frame image data necessary for moving image processing is stored, and proceeds to step S707. On the other hand, if the number of frame image data stored in the frame image data area is less than the predetermined number (step S706: No), the controller 119 has not yet stored the number of frame image data necessary for moving image processing. Determine and return to step S701.

  In step S707, the video output unit 109 reads frame image data from the frame image data area of the memory 116, performs moving image processing, and generates a moving image signal. In subsequent step S708, the video output unit 109 outputs the generated moving image signal to the display panel 110. On the display panel 110, the generated video image is displayed.

  As described above, during search playback, a frame image is generated based on the pixel corresponding to the center of the microlens 102 in a favorable focus state, thereby displaying a moving image with a deep depth of field. Can obtain an image with good visibility.

[Second Embodiment]
An image processing apparatus according to the second embodiment of the present invention will be described. This embodiment is different from the first embodiment in that a frame pixel addition unit 901 is added to the image processing apparatus, and the other configurations are common. Therefore, the description about a common location is abbreviate | omitted.

  FIG. 9 is a block diagram showing a configuration of an image processing apparatus according to the second embodiment of the present invention. The image processing apparatus 100A includes a main lens 101, a micro lens 102, an image sensor 103, an LF data input unit 104, a video output unit 109, a display panel 110, and a refocus processing unit 114. The image processing apparatus 100A further includes a memory 116, a media I / F 117, a recording medium 118, a controller 119, an operation unit 120, a memory bus 121, a search image processing unit 122, and a frame pixel addition unit 901.

  In the search reproduction mode, the frame pixel adding unit 901 adds continuous frame image data stored in the frame image data area of the memory 116 to generate new frame image data.

  Next, frame images generated in the refocus playback mode and search playback mode will be described. FIG. 10 is a flowchart illustrating image processing according to the present embodiment, and FIG. 11 is a diagram illustrating frame image generation in the refocus playback mode and the search playback mode. The flowchart shown in FIG. 10 is performed when the operation unit 120 is operated by the user. FIG. 11 illustrates a case where the moving image playback mode transitions from the refocus playback mode 1105 to the 4 × speed search playback mode 1106 and then transitions to the refocus playback mode 1107. In FIG. 11, as an example, the pixel image of the image sensor 103 is shown only in a region corresponding to two microlenses, but it goes without saying that the image sensor 103 is applied to the entire image sensor 103. Further, LF data 1111 to 1125 shown in FIG. 11 is LF data stored in the recording medium 118.

  In step S1001, the controller 119 determines the type of playback mode for playing back the moving image selected by the user. When the playback mode is the refocus playback mode (step S1001: refocus playback mode), the controller 119 determines that the playback is normal playback and proceeds to step S1002.

  In step S1002, the refocus processing unit 114 reads LF data and performs refocus processing. Specifically, the controller 119 reads out the data of all pixels of the LF data as a reading area in the LF data stored in the recording medium 118 via the media I / F 117 and stores it in the LF data area of the memory 116. The refocus processing unit 114 reads the LF data stored in the LF data area of the memory 116, and calculates a frame image that is a refocus image using Expression (2). As shown in FIG. 11, the refocus processing unit 114 reads the LF data 1111 stored in the LF data area of the memory 116, and generates a frame image 1151 that is a refocus image. Similarly, the controller 119 reads the data of all the pixels of the LF data 1112, and the refocus processing unit 114 generates a frame image 1152 based on the read data. In FIG. 11, a read area that is read by the controller 119 and transferred to the memory 116 is indicated in white, and a pixel area that is not read by the controller 119 is indicated by hatching.

  On the other hand, in step S1001, if the playback mode is the search playback mode (step S1001: search playback mode), the controller 119 determines that the playback is at high speed, and proceeds to step S1003.

  In step S1003, the controller 119 reads a predetermined area of the unit pixel as a reading area from each of the predetermined number of LF data. Specifically, the controller 119 reads out data of a predetermined pixel area from each of the unit pixels for each predetermined number of frames for the LF data stored in the recording medium 118. The controller 119 skips the LF data included in the predetermined number of frames along the time axis, reads out data of predetermined pixel areas in the LF data of two or more frames adjacent on the time axis, and Stored in the LF data area. The pixel read out by the controller 119 is a pixel area corresponding to the central portion of the microlens 102a.

  In step S1004, the search image processing unit 122 generates a frame image from the pixel area data read in step S1003. Specifically, the search image processing unit 122 generates a frame image with a deep depth of field from the pixel data included in the read area read from the LF data area for each read frame. The search image processing unit 122 generates a frame image corresponding to the read frame number.

  In step S1005, the frame pixel addition unit 901 simply adds the plurality of frame image data generated in step S1004 to generate one frame image. The imaging times of a plurality of continuous LF data read in step S1003 are close in time. By simply adding a plurality of frame image data generated from the plurality of LF data, random noise included in the LF data is reduced. Therefore, in the search reproduction mode 1106, the S / N that decreases as the number of frames from which LF data is read can be improved.

  As shown in FIG. 11, for example, when a moving image is reproduced at 4 × speed in the search reproduction mode 906, the controller 119 does not read out the LF data 1113 and 1114 arranged in time series. The controller 119 creates one frame image for every four imaging frames. For example, for the LF data 1113 to 1116, the controller 119 reads the LF data 1115 and 1116 without reading the LF data 1113 and 1114 from the recording medium 118. At that time, the LF data 1115 and 1116 read out only 5 pixels at the center of 25 pixels (5 × 5) which are constituent pixels of the micro lens 102a. The search image processing unit 122 generates a frame image having a deep depth of field from the read five pixels. Since the LF data 1115 and 1116 are continuous data, for example, the imaging time is closer in time than the LF data 1113 and 1114, the frame pixel adding unit 901 simply adds the two frame image data to obtain the frame image 1153. Is generated. Similar processing is performed during the period of the search reproduction mode 1106, and the frame pixel addition unit 901 generates a frame image 1154 from the LF data 1117 to 1120.

  In step S <b> 1006, the refocus processing unit 114 or the frame pixel addition unit 901 stores the generated frame image data in the frame image area in the memory 116. In step S1007, the controller 119 determines whether a predetermined number of pieces of frame image data stored in the frame image area in the memory 116 have been stored. When the predetermined number of frame image data is stored (step S1007: Yes), the controller 119 determines that the number of frame image data necessary for moving image processing is stored, and proceeds to step S1008. On the other hand, when the number of frame image data stored in the frame image data area is less than the predetermined number (step S1007: No), the controller 119 determines that the number of frame image data necessary for moving image processing is not stored. Then, the process returns to step S1001.

  In step S1008, the video output unit 109 reads the frame image data from the frame image data area of the memory 116, performs the moving image processing, and generates a moving image signal. In subsequent step S1009, the video output unit 109 outputs the generated video signal to the display panel 110, and the video of the video is displayed on the display panel 110.

  As described above, at the time of search reproduction, a frame image is generated based on the LF data of the pixel region corresponding to the central portion of the microlens 102a, and the frame images whose shooting times are close are added. As a result, a frame image having a deep depth of field and an improved S / N is generated, so that a moving image with a deep depth of field can be obtained, and the user can obtain an image with good visibility even during search playback. It is done.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100, 100A Image processing device 114 Refocus processing unit 119 Controller 122 Search image processing unit

Claims (8)

Image data generating means for generating image data based on a signal from an image sensor including a unit pixel consisting of a plurality of pixels assigned to each of a plurality of microlenses;
Frame image generation means for generating a frame image for moving image reproduction based on data of a predetermined area read from the unit pixel in the image data;
In the frame image generating means, a changing means for changing a readout area in the unit pixel according to a playback mode related to playback of a moving image;
An image processing apparatus comprising: The playback mode includes a normal playback mode and a search playback mode,
The image processing apparatus according to claim 1, wherein a moving image reproduction speed in the search reproduction mode is higher than a moving image reproduction speed in the normal reproduction mode.   3. The frame image generation unit, when the playback mode is the search playback mode, restricts reading of data of a pixel corresponding to a peripheral portion of the microlens in the unit pixel. The image processing apparatus described.   3. The frame image generation unit, when the playback mode is the search playback mode, generates the frame image by adding image data of a plurality of frames that are adjacent on a time axis. The image processing apparatus according to 3.   5. The image according to claim 2, wherein, when the playback mode is the search playback mode, the frame image generation unit generates a frame image for each predetermined number of frames. 6. Processing equipment. An image data generation step for generating image data based on a signal from an imaging device including a unit pixel composed of a plurality of pixels assigned to each of the plurality of microlenses;
A frame image generation step of generating a frame image for moving image reproduction based on data of a predetermined area read from the unit pixel in the image data;
In the frame image generation means, a change step of changing a readout area in the unit pixel according to a playback mode related to playback of a moving image;
An image processing method comprising: A program to be executed by a computer of an image processing apparatus that reproduces a movie,
Image data generating means for generating image data based on a signal from an imaging device including a unit pixel composed of a plurality of pixels assigned to each of a plurality of microlenses;
Frame image generation means for generating a frame image for moving image reproduction based on data of a predetermined area read from the unit pixel in the image data;
In the frame image generation means, changing means for changing a readout area in the unit pixel according to a playback mode related to playback of a moving image,
Program to function as.   A computer-readable storage medium storing the program according to claim 7.

Images