JP2016082475A - Image processing system, and control method and program of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system which can improve visibility of a moving image reproduced in search reproduction.SOLUTION: An image processing system includes:refocus image generation means for generating a refocus image where a focus position is changed by using image data including information showing a direction and intensity of a light beam which is made incident from a subject; and a subject depth setting means for setting subject depth of the refocus image used of reproducing a moving image in accordance with a type of a reproduction mode of the moving image.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing apparatus capable of generating an image whose focus position has been changed after shooting, a control method thereof, and a program.

  In recent years, a camera called a light field camera has been proposed as a camera that can change the focus position after photographing. This camera acquires light ray information including angle information indicating an incident angle in addition to position information indicating a position where a light ray from a subject is incident on the imaging surface. This ray information is called light field data (hereinafter referred to as LF data).

  In Non-Patent Document 1, a large number of microlenses are arranged between a main lens and an image sensor, and the incident light beam is dispersed by the microlens, and each dispersed light beam is received by the image sensor. An imaging device that acquires LF data is disclosed. Non-Patent Document 1 discloses a refocus technique for generating an image at a focus position different from that at the time of photographing by calculating image data on a virtual imaging plane using LF data.

  Patent Document 1 discloses an imaging device using a two-dimensional imaging element in which one microlens and a plurality of divided photoelectric conversion units are formed for one pixel. The divided photoelectric conversion units are configured to receive light that has passed through different pupil partial regions of the exit pupil of the photographing lens through one microlens. A plurality of parallax images corresponding to the divided pupil partial areas can be generated by the photoelectric conversion signals from the respective divided photoelectric conversion units. By using the plurality of obtained parallax images and combining the portions to be focused, the focus position can be changed after shooting.

US Pat. No. 4,410,804

Ren Ng et al. “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  The light field processing technique as described above can also be applied to moving image shooting. In reproduction of the moving image format, search reproduction such as fast forward and rewind, which is not possible in reproduction of the still image format, can be performed. The light field image may be refocused by the user, and a refocus image with a shallow depth of field is often used as the light field image. For this reason, when a moving image is searched and reproduced with an image having a shallow depth of field, there is a problem that a user cannot easily find a desired scene because of poor visibility.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an image processing apparatus capable of improving the visibility of a moving image reproduced by search reproduction.

  In order to achieve the above object, an image processing apparatus according to an aspect of the present invention provides a refocus image in which a focus position is changed using image data including information indicating the direction and intensity of a light beam incident from a subject. And a depth-of-field setting means for setting a depth of field of a refocused image used for moving image reproduction according to the type of moving image reproduction mode. To do.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus which can improve the visibility of the moving image reproduced | regenerated by search reproduction | regeneration can be provided.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. (A) is a figure which shows the positional relationship of a main lens, a microlens, and an image pick-up element, (b) is a figure which shows the correspondence of a microlens and a pixel. It is a figure which shows the optical path of the light radiate | emitted from a to-be-photographed object and injecting into an image pick-up element. It is the model which showed the positional relationship of a main lens and an image pick-up element. It is the model which showed the positional relationship of the main lens surface, a refocus surface, and the imaging surface of a micro lens. It is a figure which shows the more detailed structure of the refocus process part which concerns on the 1st Embodiment of this invention. It is a flowchart explaining the process in the refocus processing part which concerns on the 1st Embodiment of this invention. It is a block diagram showing the more detailed structure of the refocus process part which concerns on the 2nd Embodiment of this invention. It is a flowchart explaining the process in the refocus processing part which concerns on the 2nd Embodiment of this invention. It is a figure which shows the frame structure in the case of producing the image for search reproduction | regeneration for every 4 frames.

  Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the drawings.

(First embodiment)
An image processing apparatus according to a first embodiment of the present invention will be described. FIG. 1 is a block diagram illustrating the configuration of the first embodiment.

  The image processing apparatus 100 includes a main lens 101, a microlens array 102, an image sensor 103, an LF data input unit 104, a video output unit 109, a display panel 110, a refocus processing unit 114, a memory 116, a media I / F 117, and a recording medium. 118. The image processing apparatus 100 further includes a controller 119, an operation unit 120, and a memory bus 121.

  The main lens 101 is an imaging optical system composed of a single lens or a plurality of lenses, and the light of the subject incident from the front surface (left side in FIG. 1) is emitted to the microlens array 102. The microlens array 102 includes a plurality of microlenses (not shown), and is disposed between the main lens 101 and the image sensor 103. The light incident from the main lens 101 is dispersed by the microlens array 102 according to the incident angle and emitted to the image sensor 103. More specifically, the microlens array 102 is disposed in the vicinity of the focal position of the main lens 101. Light that has passed through different pupil regions of the main lens 101 is divided for each pupil region and emitted.

  The imaging element 103 is a CCD sensor or a CMOS sensor in which a plurality of pixels (photoelectric conversion elements) that photoelectrically convert light incident from the microlens array 102 and convert it into an electrical signal are arranged in a matrix. The imaging element 103 outputs an electrical signal generated by photoelectric conversion to the LF data input unit 104 as an imaging signal.

  The LF data input unit 104 converts the image signal input from the image sensor 103 into a digital signal, and performs development processing for converting the image signal into a predetermined image file format such as JPEG for each frame of the image frame. Convert to The LF data input unit 104 stores the converted LF data in an LF data area serving as a buffer area in the memory 116. The development process is a process for changing or adjusting the image quality of the digital image data using parameters relating to the digital image data such as color correction, resolution, and image compression.

  The video output unit 109 reads the refocus video from the video signal area of the memory 116 and outputs it to the display panel 110 and a video output terminal (not shown). The display panel 110 displays the video signal input from the video output unit 109 as a video. The display panel 110 displays a refocused image being shot during shooting, and displays an image of LF data corresponding to the type of playback mode when playing back a moving image.

  The refocus processing unit 114 performs refocus processing by reading out the LF data stored in the LF data area of the memory 116 by the media I / F 117 in the normal reproduction (refocus reproduction) mode and the search reproduction mode that is high-speed reproduction. . The video data generated by the refocus processing is stored in the video signal area of the memory 116.

  The normal playback mode is a playback mode for the user to view a moving image. In the normal playback mode, playback is typically performed at a playback speed comparable to that during shooting. The search reproduction mode is a reproduction mode including an operation for a user to search for a desired scene in a moving image, such as fast forward and rewind. In the search playback mode, playback is performed at a higher speed than in the normal playback mode. The high-speed playback in the search playback mode is typically performed by thinning out a part of images constituting a moving image, but may be performed by increasing the number of frames displayed per unit time.

  The memory 116 is a dynamic RAM (Random Access Memory) that can be randomly accessed at high speed, and is used as a work memory. The memory 116 is used as, for example, an LF data area or a video signal area. Various programs are stored in the memory 116, and the various programs are executed by the controller 119.

  The media I / F 117 is an interface that controls reading and writing of data with respect to the recording medium 118. At the time of shooting, the media I / F 117 reads out the LF data stored in the LF data area of the memory 116 by the LF data input unit 104, that is, the LF data being shot, and stores it in the recording medium 118. During normal playback, the media I / F 117 reads all the LF data recorded from the recording medium 118 and stores it in the LF data area of the memory 116. At the time of search reproduction, the LF data recorded from the recording medium 118 is read every predetermined frame (for example, every 4 frames for 4 × speed reproduction) and stored in the LF data area of the memory 116.

  The recording medium 118 is a recording medium such as a hard disk drive or a nonvolatile semiconductor memory (for example, a flash memory). The recording medium 118 records data in a file system format such as FAT (File Allocation Table).

  The controller 119 is a controller that controls the entire system of the image processing apparatus 100, and is a so-called CPU (Central Processing Unit). The controller 119 controls the operation mode of the image processing apparatus 100 to an imaging mode for photographing, a normal playback mode, a search playback mode that is high-speed playback, and the like based on a user (not shown) operation on the operation unit 120.

  The operation unit 120 is an operation unit operated by a user. For example, a shutter button, a recording button for capturing a moving image, a playback button for reproducing the captured moving image, buttons such as a setting button for performing various settings, and the surface of the display panel 110 The touch panel sensor etc. which are arrange | positioned. The user can change the shooting mode / normal playback mode / search playback mode by operating the operation unit 120. When the user operates the operation unit 120, various operation signals are transmitted from the operation unit 120 to the controller 119.

  The LF data input unit 104, the video output unit 109, the refocus processing unit 114, the memory 116, the media I / F 117, and the controller 119 are connected to the memory bus 121. The LF data input unit 104, the video output unit 109, the refocus processing unit 114, the media I / F 117, and the controller 119 write / read data to / from the memory 116 via the memory bus 121. The memory bus 121 adjusts the memory access request from each component and controls to read / write data from / to the memory 116 in a time division manner.

  Next, operations of the main lens 101, the microlens array 102, the image sensor 103, and the LF data input unit 104 at the time of photographing will be described in detail with reference to FIGS. 2 (a), 2 (b), 3 and 4. Explained.

  FIG. 2A is a schematic diagram showing the positional relationship between the main lens 101, the microlens array 102, and the image sensor 103. As shown in FIG. 2A, the microlens array 102 is positioned between the main lens 101 and the image sensor 103, and has a plurality of microlenses 102a arranged in a matrix. Note that the size and the number of the microlenses 102a in the present embodiment are not limited to those illustrated.

  FIG. 2B is a diagram illustrating a correspondence relationship between the microlens 102 a and the pixel 103 a included in the image sensor 103. As shown in FIG. 2B, n × m pixels 103a in the image sensor 103 are arranged so as to correspond to one microlens 102a. The number of divisions of light rays is determined by the number of pixels 103a corresponding to one microlens 102a (that is, n × m). FIG. 2B shows a state in which 5 × 5 pixels 103a are arranged for one microlens 102a as an example.

  FIG. 3 is a diagram illustrating an optical path of incident light to the main lens 101, the microlens 102, and the image sensor 103. A light beam emitted from a point 307, which is one point of the subject 305, enters the main lens 101, is condensed by the main lens 101, and is projected onto the imaging surface 308 on the surface of the microlens 302. The projected light is dispersed in the microlens 302 according to the incident angle of the light, and is received by the pixels 321 to 325 of the image sensor 103. Light received by the pixels 321 to 325 is photoelectrically converted and output as an imaging signal.

  The light received by the pixel 321 is a light flux that has passed through the optical path 311. Similarly, the light received by the pixels 322 to 325 is a light flux that passes through the optical paths 312 to 315, respectively. That is, the light received by the pixels 321 to 325 is light emitted from the same point, but is received by different pixels depending on the incident angle of the light beam.

  The imaging signal output from the imaging element 103 is input to the LF data input unit 104. The LF data input unit 104 digitizes the imaging signal, performs development processing to convert it into LF data, and stores it in the LF data area of the memory 116. Thereafter, the LF data stored in the memory 116 is recorded on the recording medium 118 by the controller 119.

  Next, signal processing during normal reproduction will be described. The controller 119 reads all data for one frame of the LF data stored in the recording medium 118 via the media I / F 117 and stores it in the LF data area in the memory 116. The refocus processing unit 114 performs refocus processing based on the LF data stored in the LF data area of the memory 116, and stores the frame image data in the frame image data area of the memory 116. The refocus process will be described later.

  When frame image data is created in the frame image data area of the memory 116 by performing the above operation continuously, the video output unit 109 reads the frame image data from the frame image data area of the memory 116 and generates a moving image. To generate a moving image signal. The generated moving image signal is output to the display panel 110, and a moving image is displayed on the display panel 110.

  Next, details of generation of a refocus image performed in the refocus processing unit 114 will be described. The generation of the refocus image is specifically performed as follows based on the “Light Field Photography” method.

First, the function L _F (x, y, u, v) representing the ray information will be described. FIG. 4 is a schematic diagram showing the positional relationship between the main lens 101 and the image sensor 103 in a simplified manner. The coordinate axes of the main lens 101 are u and v, the coordinate axes of the image sensor 103 are x and y, and the distance between the main lens 101 and the image sensor 103 is F. At this time, the light ray L passing through the main lens 101 and the image sensor 103 is defined by a four-dimensional function L _F (x, y, u, v).

Next, a calculation method in refocus image generation will be described. FIG. 5 is a diagram showing a positional relationship between the main lens surface 501, the refocus surface 502 determined by the refocus coefficient α, and the imaging surface 503 of the microlens. A light ray L that enters from the point u on the lens surface 501 and passes through the point x ′ on the refocus surface 502 reaches the point x on the imaging surface 503 of the microlens. Here, since the straight ray L passes through the point x ′ on the refocus plane 502 and the point x on the imaging surface 503 of the microlens, x = u + (x′−u) / α. . Similarly, the light reception signal on the imaging surface of the microlens of the light beam that enters from the coordinates (u, v) of the main lens surface 501 and passes the coordinates (x ′, y ′) of the refocus surface 502 It represents the function _{L F (x, y, u} , v) to can be expressed as equation (1) using.

Further, the refocus image EF _′ (x ′, y ′) obtained on the refocus surface is obtained by integrating the function L _F (x, y, u, v) with respect to the u axis and the v axis of the main lens surface. Since it is a thing, it can represent like Formula (2). The refocus image is generated by the calculation process of Expression (2).

As described above, the refocused image E _{F ′} (x ', Y') can be calculated. Since the refocus plane can be set arbitrarily, according to this method, the focus position can be changed to an arbitrary position after shooting.

  FIG. 6 is a diagram showing a more detailed configuration of the refocus processing unit 114. The refocus processing unit 114 includes a depth of field setting unit 601 and a refocus image generation unit 602. FIG. 7 is a flowchart for explaining processing in the refocus processing unit 114. Processing performed by the refocus processing unit 114 will be described with reference to FIGS. 6 and 7.

  When the user inputs a normal reproduction or search reproduction instruction to the operation unit 120, the controller 119 transmits a signal indicating a reproduction mode such as a normal reproduction operation signal or a search reproduction operation signal to the depth of field setting unit 601. In step S701, the depth-of-field setting unit 601 receives signals indicating these playback modes. In step S702, the depth-of-field setting unit 601 determines the reproduction mode based on the received signal, and the coordinates (u, v) of the main lens surface 501 that are the coefficients of Expression (2) according to the reproduction mode. Determine the integration range.

  Narrowing the integration range of the coordinates (u, v) of the main lens surface 501 is equivalent to reducing the lens aperture (reducing the lens aperture). Conversely, increasing the integration range of the coordinates (u, v) of the main lens surface 501 corresponds to increasing the lens aperture (expanding the lens aperture). Thereby, the depth of field of the refocus image can be changed.

  When the operation signal input from the controller 119 is a normal reproduction operation signal, the integration range of the coordinates (u, v) of the main lens surface 501 is the entire main lens surface 501 (the process proceeds to step S703). When the operation signal input from the controller 119 is a search reproduction operation signal, the integration range of the coordinates (u, v) of the main lens surface 501 is narrowed according to the double speed value of the search reproduction (the process proceeds to step S704). That is, the integration range of the coordinates (u, v) of the main lens surface 501 is set to be narrower in the search reproduction than in the normal reproduction. The depth-of-field setting unit 601 transmits the integration range set in this way to the refocus image generation unit 602.

  The refocus image generation unit 602 calculates the LF data read from the LF data area of the memory 116 and the integration range of the coordinates (u, v) of the main lens surface 501 received from the depth of field setting unit 601. Then, the refocus image is calculated by the equation (2). If the integration range is the entire main lens surface 501, the focus image generation unit 602 generates a refocus image with a shallow depth of field in step S703. When the integration range is narrowed, in step S704, the refocus image generation unit 602 generates a refocus image with a deep depth of field.

  The refocus image generation unit 602 stores the generated refocus image in the frame image data area of the memory 116 (step S705). The video output unit 109 reads out this image from the memory 116 to generate a moving image and displays it on the display panel 110.

  The refocus coefficient α varies depending on the position to be focused in the image. For example, in the first refocus process, the refocus coefficient α can be set so that the center position of the image is the in-focus position. In the second and subsequent refocus processing, a refocus coefficient α is set at a position arbitrarily selected by the user, and a refocus image can be created from Expression (2).

  As described above, the image processing apparatus 100 according to the present embodiment creates and displays an image with a shallow depth of field during normal playback, and creates and displays an image with a deep depth of field during search playback. Let Thereby, the visibility of the moving image reproduced by the search reproduction can be improved, and the user can easily select a desired scene.

  In the above description, after receiving a signal indicating search reproduction (that is, during search reproduction), a refocus image having a deep depth of field to be displayed by search reproduction is generated. It may be done before playback. That is, it can be performed in an arbitrary period before reproduction, such as between shooting and recording, between recording and reproduction. In this case, a refocus image having a deep depth of field created in advance is recorded on the recording medium 118 via the media I / F 117. At the time of search reproduction, in steps S704 and S705 in FIG. 7, a refocus image having a deep depth of field is read from the recording medium 118 to the memory 116 without being newly generated.

(Second Embodiment)
An image processing apparatus according to the second embodiment of the present invention will be described. The configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the first embodiment shown in FIG. 1 except for the configuration and processing of the refocus processing unit. Hereinafter, the details of the configuration of the refocus processing unit of the present embodiment will be described.

  FIG. 8 is a block diagram showing a more detailed configuration of the refocus processing unit. The refocus processing unit 114 according to the second embodiment includes a refocus image generation unit 801, a focus evaluation value calculation unit 802, and a refocus coefficient setting unit 803.

  When normal reproduction is performed, a refocus image is generated by the refocus image generation unit 801 as in the first embodiment. As an example, when the user designates the position of the image using the operation unit 120, the refocus coefficient α is set from the coordinates (x ′, y ′) of the refocus plane 502 at that position. By applying the focus coefficient α to the equation (2), a refocus image is generated. In this case, the integration range of the coordinates (u, v) of the main lens surface 501 in the equation (2) is the entire main lens surface 501.

  When search reproduction is performed, a refocus image divided into predetermined blocks is generated by the refocus image generation unit 801, the focus evaluation value calculation unit 802, and the refocus coefficient setting unit 803 in FIG. The predetermined block is obtained by dividing the pixels of the entire screen after generation of the refocus image by a predetermined shape and size. For example, when the screen is divided into eight equal parts in the row direction and the column direction, 8 × 8 = There are 64 blocks. A refocus image is generated for each block. A plurality of refocus images are generated for each block using a plurality of refocus coefficients. Thereafter, a focus evaluation value indicating the degree of focus of each generated image is calculated, and a refocus image having the highest focus evaluation value is selected. As a result, it is possible to select the most focused image for each block.

  By performing the above processing on all the blocks, it is possible to obtain a refocus image in which each block is focused (that is, an image in which the focus position is optimized at a different position for each block). By combining these blocks, an in-focus image can be obtained over the entire screen. In other words, an image having a deep depth of field can be obtained. Note that, unlike the case of the first embodiment, in this embodiment, the integral range of the coordinates (u, v) of the main lens surface 501 in the equation (2) is also used for the entire main lens surface in search reproduction. An image having a deep depth of field can be obtained.

  FIG. 9 is a flowchart showing search reproduction image generation processing in the refocus processing unit 114. Processing performed by the refocus processing unit 114 will be described with reference to FIGS. 8 and 9.

  In step S901, the refocus coefficient setting unit 803 sets the refocus coefficient α to a default value. In step S902, the refocus image generation unit 801 generates a refocus image of each block using the refocus coefficient α. The refocus coefficient α used in step S902 is the default value set in step S901 if it is the first time, and the value set in step S907 described later if it is the second time or later. In step 903, the focus evaluation value calculation unit 802 calculates the focus evaluation value of each block from the refocus image of each block generated in step 902.

  In step 904, it is determined whether the focus evaluation value in each block obtained in step 903 is higher than the highest focus evaluation value of the refocus image generated using a different refocus coefficient α in the past. to decide. If there is a block determined to be high (in the case of YES), the process proceeds to step 905. When there is no block determined to be high (in the case of NO), the process proceeds to step S906.

  In step 905, the refocus image of the block for which it has been determined in step S 904 that a focus evaluation value higher than the past maximum value has been obtained is output to the memory 116.

  In step S906, it is determined whether the focus evaluation value of the refocus image of each block obtained so far satisfies a predetermined criterion. When the criterion is satisfied (in the case of YES), the search reproduction image generation process is terminated. If the standard is not satisfied (in the case of NO), the refocus process is insufficient, and the process proceeds to step S907.

  In step S907, after changing the refocus coefficient α of each block, the process proceeds to step S902. The refocus coefficient after this change can be determined to increase or decrease by a predetermined value from the previous value, for example. Further, the changed value may be determined with reference to the relationship between the past refocus coefficient α and the focus evaluation value. In this way, the focus evaluation value of each block can be improved by looping from step S902 to step S907.

  The search reproduction image generation process in the refocus processing unit 114 can be performed at an arbitrary timing as long as it is after shooting and before reproduction. For example, it can be performed immediately after shooting, such as when the image processing apparatus 100 is turned on. Further, it may be performed in the background of the photographing process. In addition, the search reproduction image may be generated for all the frames, or a part of the frames may be thinned out by generating the images for each predetermined frame.

  FIG. 10 shows a frame configuration when a search reproduction image is created every four frames. Of the acquired video data, search reproduction images are generated for search reproduction image generation frames 1001 to 1005 arranged every four frames. When the created search reproduction image is reproduced as it is at the same processing speed, search reproduction is performed at a quadruple speed.

  A method for switching between a normal reproduction refocus image and a search reproduction image in accordance with the reproduction mode will be described regarding the reading of an image from the recording medium 118 performed by the media I / F 117. A time code is added as header information to the refocus image created by the refocus processing unit 114 and then recorded on the recording medium 118. When switching from normal reproduction to search reproduction, the media I / F 117 reads from the recording medium 118 a search reproduction image to which the same value as the value of the time code added to the header of the refocus image for normal reproduction is added. Select and output. In addition, when switching from search reproduction to normal reproduction, the media I / F 117 records a refocus image for normal reproduction to which a time code having the same value as the time code value of the header of the search reproduction image is added. Select from media 118 and output. As a result, the replay image for normal reproduction and the image for search reproduction can be switched.

  In the first embodiment, when generating an image with a deep depth of field, a method of narrowing the integration range of the coordinates (u, v) of the main lens surface 501 in Expression (2) is used. This is similar to reducing the amount of light incident on the lens. For this reason, it is necessary to increase the ISO sensitivity in a low-illumination shooting scene, and there are cases where the noise of the search reproduction image increases (S / N deteriorates). On the other hand, in the second embodiment, by combining a plurality of images having different focus positions, an image having a deep depth of field is generated without narrowing the integration range. Therefore, the above-described noise generation factors are unlikely to occur, and a search reproduction image with good image quality can be obtained even in a low-illumination shooting scene.

  Further, in the present embodiment, as in the first embodiment, an image with a deep depth of field can be created and displayed during search reproduction. Thereby, the visibility of the moving image reproduced by the search reproduction can be improved, and the user can easily select a desired scene.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

114 Refocus processing unit 601 Depth of field setting unit (depth of field setting means)
602 Refocus image generation unit (refocus image generation means)

Claims (10)

Refocus image generation means for generating a refocus image in which the focus position is changed using image data including information indicating the direction and intensity of light rays incident from the subject;
An image processing apparatus comprising: a depth-of-field setting unit configured to set a depth of field of a refocus image used for reproducing the moving image according to a type of a reproduction mode related to the reproduction of the moving image. The playback mode includes a normal playback mode and a search playback mode,
The image processing apparatus according to claim 1, wherein the search reproduction mode is a reproduction mode in which reproduction is performed at a higher speed than the normal reproduction mode. The depth of field of the refocus image used for reproduction in the search reproduction mode is deeper than the depth of field of the refocus image used for reproduction in the normal reproduction mode. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the depth-of-field setting unit sets the depth of field according to a reproduction speed in the search reproduction mode.   The image processing apparatus according to claim 2, wherein the refocus image generation unit generates a refocus image used in the search reproduction mode by combining a plurality of refocus images having different focus positions.   The image processing apparatus according to claim 5, wherein the refocus image generation unit generates a refocus image used in the search reproduction mode every predetermined number of frames.   The refocus image generation means creates an image used in the search reproduction mode and records it on a recording medium during a period from the photographing of the image data to reproduction in the search reproduction mode. The image processing apparatus according to claim 2. Generating a refocused image with the focus position changed using image data including information indicating the direction and intensity of light rays incident from the subject;
And a step of setting a depth of field of a refocused image used for reproducing the moving image according to the type of the moving image reproducing mode. Depending on the type of refocus image generation means for generating a refocus image whose focus position has been changed using image data including information indicating the direction and intensity of light rays incident from the subject, and the type of playback mode of the video, A program for controlling an image processing apparatus, which causes a computer to function as a depth of field setting means for setting a depth of field of a refocused image used for reproducing the moving image.   A computer-readable storage medium storing the program according to claim 9.

Images