JP2015139019A - Device and program for image composition - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and a program for image composition, generating a composed image having no incongruity, at a position outside the range of the depth of field of a refocus image, when setting a focal position and a point of sight after imaging a subject.SOLUTION: The image composition device includes: a composed image generator unit 114a for inserting second image data into a predetermined position in the depth direction of first image data whose focal position can be changed in the depth direction; and a change unit 114b for changing the focal position of the composite image data, according to the insertion position of the second image data in the depth direction of the first image data.

Description

  The present invention relates to an image composition device and an image composition program.

  2. Description of the Related Art An imaging device that can set a focus position, a viewpoint, and the like after shooting a subject is known. There is a light field camera as this type of imaging apparatus. In this light field camera, the imaging unit acquires a large number of light rays incident on the camera as a light field (light space). Then, the light field camera generates a refocus image in which the in-focus position and the like are changed in the depth direction by performing image processing on the acquired light field information. Patent Document 1 describes an image composition device that generates a composite image by inserting a two-dimensional image at an arbitrary position in the depth direction in a refocus image captured by a light field camera.

JP 2010-68018 A

  However, in the above-described image composition device, an uncomfortable composite image is generated such that a focused two-dimensional image is inserted at a position outside the range of the depth of field of the refocus image. There is a fear.

  An object of an aspect of the present invention is to generate a composite image without a sense of incongruity.

  According to the first aspect of the present invention, the composite image generation unit that generates the composite image data by inserting the second image data at a predetermined position in the depth direction in the first image data in which the focus position can be changed in the depth direction. An image synthesizing apparatus is provided that includes a changing unit that changes the in-focus position of the synthesized image data according to the insertion position of the second image data in the depth direction in the first image data.

  According to the second aspect of the present invention, the second image data in which the in-focus position can be changed in the depth direction is inserted into the predetermined position in the depth direction in the first image data in which the in-focus position can be changed in the depth direction. A composite image generation unit that generates composite image data, and a combination of composite image data so that a relationship between a distance from the imaging device to the subject of the first image data matches a distance from the imaging device to the subject of the second image data. An image synthesizing device is provided that includes a changing unit that changes a focal position.

  According to the third aspect of the present invention, the composite image that generates the composite image data by inserting the second image data at a predetermined position in the depth direction in the first image data in which the in-focus position can be changed in the depth direction. There is provided an image composition program that executes a generation process and a change process for changing a focus position of the composite image data in accordance with the insertion position of the second image data in the depth direction in the first image data.

  According to the fourth aspect of the present invention, the second image data in which the in-focus position can be changed in the depth direction and the second image data in which the in-focus position can be changed in the depth direction are stored in the computer. The composite image generation processing for generating the composite image data by inserting the composite image so that the relationship between the distance from the imaging device to the subject of the first image data and the distance from the imaging device to the subject of the second image data is matched. An image synthesizing program for executing a change process for changing a focus position of data is provided.

  According to the aspect of the present invention, it is possible to generate a composite image without a sense of incongruity.

It is a block diagram which shows the structure of an image composition apparatus. It is sectional drawing which shows schematic structure of the optical system part in the light field camera which is an example of an imaging device. It is a figure explaining the pixel arrangement | sequence and incident area of an image pick-up element. It is a block diagram which shows the structure of a light field camera. It is a figure explaining the reconstruction of an image. 6 is a flowchart for explaining image processing executed by the image processing unit shown in FIG. 4. It is a figure which shows the example of a 1st image, a 2nd image, and a synthesized image. It is a flowchart for demonstrating the image composition process which the arithmetic processing part of 1st Embodiment performs. It is a figure which shows the example of a display of the bird's-eye view on the graphical user interface of 1st Embodiment. It is a figure which shows the example of a display of the bird's-eye view on the graphical user interface of 1st Embodiment. It is a flowchart for demonstrating the image composition process which the arithmetic processing part of 2nd Embodiment performs. It is a figure which shows the example of a display of the bird's-eye view on the graphical user interface of 2nd Embodiment. It is a figure which shows the example of a display of the bird's-eye view on the graphical user interface of 2nd Embodiment. It is a block diagram which shows the structure of an image composition system. It is a block diagram which shows the structure which the imaging device and the electronic device isolate | separated.

  Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to this. Further, in the drawings, in order to describe the embodiment, the scale may be appropriately changed and expressed, for example, by partially enlarging or emphasizing.

<First Embodiment>
FIG. 1 is a block diagram illustrating a configuration of the image composition apparatus 101. As illustrated in FIG. 1, the image composition device 101 is connected to a light field camera 1, a display device (display unit) 102, a keyboard 103, and a mouse 104 which are examples of an imaging device. The image composition device 101 inserts the second image data at an arbitrary position in the depth direction in the first image data photographed by the light field camera 1 to generate composite image data. Further, the image composition device 101 combines the composite image data so that the distance from the light field camera 1 to the subject in the first image data matches the distance from the light field camera 1 to the subject in the second image data. The focus position and the depth of field range are changed.

  In the present embodiment, the first image data is image data of a refocus image whose focus position can be changed in the depth direction, and the second image data is a two-dimensional image whose focus position cannot be changed in the depth direction. Image data. Note that the direction from the light field camera 1 toward the subject is referred to as the back side direction, and the opposite side direction is referred to as the near side direction. The direction on the back side and the direction on the near side are referred to as the depth direction. The depth of field refers to the range of the focal length of a subject that appears to be in focus.

  The image composition device 101 is realized by a personal computer, for example. As shown in FIG. 1, the image composition apparatus 101 includes an arithmetic processing unit 110 and a storage unit 120. The arithmetic processing unit 110 controls the synthesis of a plurality of image data. The storage unit 120 includes first image data captured by the light field camera 1, second image data, and various information necessary for image composition processing by the arithmetic processing unit 110 (for example, the size and position of a subject in the image data). Position data, distance data indicating the distance from the light field camera 1 to the subject at the time of shooting), and the like are stored.

  As shown in FIG. 1, the arithmetic processing unit 110 includes a display control unit 111, an image input unit 112, an operation control unit 113, and an image composition unit 114. The display control unit 111 is a graphical user interface (hereinafter referred to as GUI) that is operated by the user when the user synthesizes image data on the display screen 500 of the display device 102 such as a liquid crystal display. The various images (see FIGS. 10 and 11) are displayed.

  The image input unit 112 is connected to the light field camera 1 to input the first image data stored in the light field camera 1 and the position data and distance data of the subject in the first image data. . The image input unit 112 stores the first image data, position data, and distance data input from the light field camera 1 in an image data storage area in the storage unit 120. Further, the image input unit 112 is connected to a normal digital camera (imaging device) (not shown) to input the second image data stored in the digital camera. The image input unit 112 stores the second image data input from the digital camera in an image data storage area in the storage unit 120. The image input unit 112 acquires image data and the like (first image data, second image data, position data, and distance data) through a communication network (not shown) such as the Internet, and stores the acquired image data and the like. It is also possible to store in the image data storage area 120.

  The operation control unit 113 is connected to a keyboard 103 and a mouse 104 that constitute an operation unit that can be operated by a user. The operation control unit 113 inputs information according to the operation of the keyboard 103 and the mouse 104 by the user, and outputs the information to the display control unit 111 and the image composition unit 114.

  As shown in FIG. 1, the image composition unit 114 includes an image generation unit (composite image generation unit) 114a, a change unit 114b, and a storage control unit 114c. The image generation unit 114a performs a process of generating the composite image data by inserting the second image data at a predetermined position in the depth direction in the first image data. In addition, the image generation unit 114a performs processing for processing one or both of the first image data and the second image data in the composite image data as necessary.

  The changing unit 114b performs a process of changing the in-focus position of the composite image data according to the insertion position of the second image data in the depth direction in the first image data. The changing unit 114b performs a process of changing the range of the depth of field of the composite image data according to the insertion position of the second image data in the depth direction in the first image data. The storage control unit 114 c performs processing for storing the composite image data in the storage unit 120.

  The display control unit 111, the image input unit 112, the operation control unit 113, and the image composition unit 114 (the image generation unit 114a, the change unit 114b, and the storage control unit 114c) in the arithmetic processing unit 110 are configured with a CPU ( This corresponds to processing or control executed by a control device such as a Central Processing Unit) according to a control program (image synthesis program) stored in the storage unit 120.

  The storage unit 120 has an image data storage area for storing image data (first image data, second image data, composite image data), position data, and distance data. In addition, the storage unit 120 stores various types of information necessary for display such as a GUI by the display control unit 111. Further, as described above, the storage unit 120 has a control program (image composition) for executing the processing of the display control unit 111, the image input unit 112, the operation control unit 113, and the image composition unit 114 in the arithmetic processing unit 110. Program).

  Next, the configuration of the light field camera 1 shown in FIG. 1 will be described. FIG. 2 is a cross-sectional view illustrating a schematic configuration of an optical system portion in a light field camera 1 which is an example of an imaging apparatus. In FIG. 2, the right direction (optical axis direction) of the paper surface is the X axis, the upward direction of the paper surface orthogonal to the X axis is the Z axis, and the direction from the back of the paper surface orthogonal to the X axis and the Z axis is the direction from the back. Is the Y axis. FIG. 2 is a cross-sectional view of the optical system portion of the light field camera 1 taken along a plane including the optical axis.

  The light field camera 1 of the present embodiment acquires information on a light field (light space) that is a light ray set in a three-dimensional space, and generates an image based on the acquired information on the light field. In the following description, the light field information is referred to as light ray information. A typical function of the light field camera 1 is a refocus function for generating an image whose focal length is changed by post-processing after shooting. Therefore, the light field camera 1 is also called a refocus camera.

  The light field camera 1 of the present embodiment includes a photographing optical system (imaging optical system) 11 and a microlens array 12 as its optical system part. FIG. 2 also shows the image sensor 20 that is a configuration other than the optical system portion in the light field camera 1.

  The photographing optical system 11 is a single focus lens composed of a plurality of lens groups. The photographing optical system 11 forms an image of a light beam from the subject in the vicinity of its focal plane (imaging plane). Further, the photographing optical system 11 includes an aperture stop (not shown) that controls an aperture value (F value) during photographing. Since the aperture value can be changed by reconstructing the image after shooting, the aperture diameter of the aperture stop may be fixed. As shown in FIG. 2, a microlens array 12 and an image sensor 20 are sequentially arranged near the focal plane of the photographing optical system 11.

  The microlens array 12 has a configuration in which a plurality of microlenses (positive lenses) having positive power are two-dimensionally arranged. In the microlens array 12 shown in FIG. 2, only three microlenses are arranged in the Z-axis direction, but actually, a large number of microlenses are arranged in the Z-axis direction and the Y-axis direction.

The image pickup device 20 is disposed on the rear side of the microlens array 12 (opposite side of the photographing optical system 11) with a slight gap. The image sensor 20 receives light beams that have passed through the microlenses of the microlens array 12 by a plurality of photoelectric conversion elements, and photoelectrically converts the light beams received by the photoelectric conversion elements to generate a plurality of pixel signals. The plurality of photoelectric conversion elements are arranged in a matrix, and each photoelectric conversion element forms a pixel. In the present embodiment, the image sensor 20 is a CCD (Charge Coupled Device) or CMOS (Complementary).
It consists of a two-dimensional solid-state image sensor such as a metal-oxide semiconductor).

  In the example shown in the partially enlarged view of FIG. 2, a plurality of pixels are arranged on the rear side of each of the plurality of microlenses 12a, 12b, and 12c. Specifically, three pixels 21a, 21b, 21c are arranged on the rear side of the micro lens 12a, three pixels 22a, 22b, 22c are arranged on the rear side of the micro lens 12b, and on the rear side of the micro lens 12c. Three pixels 23a, 23b, and 23c are arranged.

  Three pixels 21a, 21b, and 21c are pixels 21 corresponding to the microlens 12a, three pixels 22a, 22b, and 22c are pixels 22 corresponding to the microlens 12b, and three pixels 23a, 23b, and 23c are microscopic. This is a pixel 23 corresponding to the lens 12c. In the partially enlarged view of FIG. 2, three pixels are shown as the pixels corresponding to the microlenses 12a, 12b, and 12c, but in reality, a large number of pixels are arranged in the Z-axis direction and the Y-axis direction ( (See FIG. 3 described later).

  Next, acquisition of a light field by the light field camera 1 will be briefly described with reference to FIG. In FIG. 2, it is assumed that the subject at the position A focuses on the position of the microlens array 12 by the photographing optical system 11. The light beam from the point A1 on the subject passes through the partial region 11a on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21c of the image sensor 20. Further, another light ray from the point A1 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21b of the imaging device 20. Further, another light ray from the point A1 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21a of the imaging device 20.

  Similarly, the light beam from the point A2 on the subject passes through the partial area 11a on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22c of the image sensor 20. Further, another ray from the point A2 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22b of the image sensor 20. Further, another light ray from the point A2 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22a of the imaging device 20.

  Further, the light beam from the point A3 on the subject passes through the partial area 11a on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23c of the image sensor 20. Further, another light ray from the point A3 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23b of the image sensor 20. Further, another light ray from the point A3 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23a of the image sensor 20. As described above, the light beam from the subject at the position A is received by the pixels of the image sensor 20 located on the rear side of the microlenses 12a, 12b, and 12c, whereby the brightness and the traveling direction are acquired as light beam information. The

  Of the three pixels corresponding to the microlenses 12a, 12b, and 12c, the image generated by the pixel signals of the lower pixels 21c, 22c, and 23c is an image of light rays that have passed through the partial region 11a. In addition, the image generated by the pixel signals of the middle pixels 21b, 22b, and 23b is an image of light rays that have passed through the partial region 11b. Further, the image generated by the pixel signals of the upper pixels 21a, 22a, and 23a is an image of light rays that have passed through the partial region 11c. For example, when the number of pixels corresponding to each microlens is N, N reconstructed images configured by arranging pixel values of pixels located at the same position with respect to the microlens are configured to have the photographic optical system 11 N stereo image groups acquired by dividing into partial regions are formed.

  FIG. 3 is a diagram for explaining the pixel array and the incident region of the image sensor 20. 3 shows a state where the image sensor 20 is observed from the X-axis direction of FIG. In the image sensor 20 shown in FIG. 3, a region where pixels are arranged is referred to as a pixel region 200. In the pixel region 200, for example, 10 million or more pixels are arranged in a matrix. The circular area in the pixel area 200 is an area where light rays that have passed through one microlens are incident. Accordingly, a plurality of circular regions are formed for each of the plurality of microlenses. These regions are referred to as incident regions 201, 202. For example, 586 × 440 microlenses are arranged. However, such an arrangement number is an example, and a larger or smaller number may be used. In the example shown in FIG. 3, the diameter of one microlens is 13 pixels. However, the number of pixels is not limited to this.

  As described above, the light field camera 1 extracts and reconstructs pixel values of pixels at the same position in the incident area (the same position in the Y-axis direction and the Z-axis direction in the incident area) from all the microlenses. . The reconstructed image group obtained in this way corresponds to an image group in which a subject is observed from different partial areas of the photographing optical system 11. Therefore, these reconstructed image groups have parallax according to the position of the partial region and the distance to the subject. For example, as shown in FIG. 3, a reconstructed image composed of pixel values of pixels of each microlens located at the same position as the pixel 201a in the incident area 201 and each micro located at the same position as the pixel 201b in the incident area 201 It has parallax with the reconstructed image composed of pixel values of the lens pixels.

  FIG. 4 is a block diagram showing a configuration of the light field camera 1. The light field camera 1 illustrated in FIG. 4 includes an imaging unit 10, an image processing unit 30, a work memory 40, a display unit 50, an operation unit 55, a recording unit 60, and a system control unit 70. The imaging unit 10 includes a photographing optical system 11, a microlens array 12, a drive control unit 13, an imaging element 20, and a drive unit 21. Since the photographing optical system 11 and the microlens array 12 have been described with reference to FIG. 2, description thereof is omitted in FIG.

  For example, a Bayer array color filter array is disposed on the imaging surface of the imaging element 20. The image sensor 20 makes the light beam that has passed through the photographing optical system 11 and the microlens array 12 enter. The image sensor 20 photoelectrically converts an incident light beam to generate a pixel signal of each pixel of an analog signal. In the imaging unit 10, an amplifier (not shown) amplifies the pixel signal of each pixel of the analog signal with a predetermined gain factor (amplification factor). Further, an A / D converter (not shown) converts the pixel signal of each pixel of the analog signal amplified by the amplifier into the pixel signal of each pixel of the digital signal. Then, the imaging unit 10 outputs RAW data including pixel signals of each pixel of the digital signal to the image processing unit 30. The light ray information acquired by the light field camera 1 is specified by RAW data output from the imaging unit 10.

  The drive control unit 13 performs aperture stop drive control based on control information transmitted from the system control unit 70 in order to adjust the aperture diameter of the aperture stop included in the photographing optical system 11. The drive unit 21 is a control circuit that controls driving of the image sensor 20 under imaging conditions set based on an instruction signal from the system control unit 70. Examples of imaging conditions include exposure time (shutter speed), frame rate, gain, and the like.

  Here, the exposure time refers to the time from when the photoelectric conversion element of the image sensor 20 starts to accumulate after the start of charge accumulation according to incident light. The frame rate is a value representing the number of frames processed (displayed or recorded) per unit time in a moving image. The unit of the frame rate is represented by fps (Frames Per Second). The gain refers to a gain factor (amplification factor) of an amplifier that amplifies the pixel signal. By changing this gain, the ISO sensitivity can be changed. This ISO sensitivity is a photographic film standard established by ISO and represents how much light the photographic film can record. However, generally, ISO sensitivity is also used when expressing the sensitivity of the image sensor 20. In this case, the ISO sensitivity is a value representing the ability of the image sensor 20 to capture light.

  The image processing unit 30 performs various kinds of image processing on the RAW data including the pixel signal of each pixel, and generates image data in a predetermined file format (for example, JPEG format). The image data generated by the image processing unit 30 includes image data of still images, moving images, and live view images. The live view image is an image displayed on the display unit 50 by sequentially outputting the image data generated by the image processing unit 30 to the display unit 50. The live view image is used for the user to confirm the image of the subject imaged by the imaging unit 10. The live view image is also called a through image or a preview image.

  The image processing unit 30 uses the work memory 40 as a work space to generate image data set at an arbitrary focal length by performing predetermined image processing on RAW data composed of pixel signals of each pixel. Specifically, the image processing unit 30 generates image data set at an arbitrary focal length by performing the following image processing.

  FIG. 5 is a diagram illustrating image reconstruction. As shown in FIG. 5, the image processing unit 30 has pixel values (values indicated by pixel signals) of the pixels 201a, 202a... At the same position with respect to the respective incident areas 201, 202. To extract. The image processing unit 30 rearranges the extracted pixel values to generate reconstructed image data 400a. When the number of pixels corresponding to each of the incident areas 201, 202... Is N, N reconstructed image data are generated.

  The image processing unit 30 translates the N reconstructed image data groups by a predetermined amount in accordance with a predetermined focal length. Then, the image processing unit 30 adds and averages the N reconstructed image data groups after the parallel movement. Thereby, image data set to a predetermined focal length is generated. Since the image of the subject existing at a depth corresponding to the amount of movement is aligned, a sharp contour image is formed. An image of a subject that does not exist at a depth corresponding to the amount of movement is blurred. The image processing described above is an example, and the image processing unit 30 may generate image data set at an arbitrary focal length by executing image processing different from the image processing described above. For example, the image processing unit 30 calculates a light beam acquired by each pixel of the image sensor 20 by performing a predetermined calculation. Then, the image processing unit 30 generates image data set to a predetermined focal length by projecting the calculated light beam onto a virtual image plane set to a predetermined focal length.

  In the present embodiment, the image processing unit 30 has a focal position (focal point) such as image data focused on the foremost subject, image data focused on the farthest subject, and image data focused on a subject at an intermediate position. A plurality of image data having different distances are generated.

  The image processing unit 30 performs various types of image processing on image data set at an arbitrary focus using the work memory 40 as a work space. For example, the image processing unit 30 generates RGB image data by performing color signal processing (tone correction) on a signal obtained by the Bayer array. The image processing unit 30 performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the RGB image data. Further, the image processing unit 30 performs a process of compressing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary.

  The image processing unit 30 detects one or more subjects from the image data. In the present embodiment, the image processing unit 30 detects the subject by specifying the boundary of the subject region based on the contrast and color change between the bright and dark portions in the image data. Note that the image processing unit 30 may detect a subject of a human subject using a known face detection function. Further, the image processing unit 30 may detect a moving subject by comparing a plurality of pieces of image data obtained in time series for a moving subject (moving subject). In addition to the face detection, the image processing unit 30 may detect a human body included in the image data as a subject as described in, for example, Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940). .

  Further, the image processing unit 30 detects (calculates) a distance (focal length) to each detected subject. For example, the image processing unit 30 detects the parallax of each subject in a plurality of reconstructed image data (for example, reconstructed image data composed of pixel values of pixels at two specific positions in the incident region), and uses the detected parallax. Based on this, the distance to each subject is calculated.

  The image processing unit 30 outputs the generated image data to the recording unit 60. In addition, the image processing unit 30 outputs the generated image data to the display unit 50. In addition, the image processing unit 30 outputs position data indicating the detected size of the subject and the position in the image data in units of pixels to the recording unit 60. In addition, the image processing unit 30 outputs distance data indicating the detected distance to each subject to the recording unit 60.

  The work memory 40 temporarily stores RAW data, image data, and the like when image processing by the image processing unit 30 is performed. The display unit 50 displays an image captured by the imaging unit 10 and various information. The display unit 50 includes a display screen 51 configured by, for example, a liquid crystal display panel. The image displayed on the display unit 50 includes a still image, a moving image, and a live view image.

  The operation unit 55 includes a release switch (a switch that is pressed when shooting a still image), a moving image switch (a switch that is pressed when shooting a moving image), various operation switches, and the like that are operated by the user. The operation unit 55 outputs a signal corresponding to the operation by the user to the system control unit 70. The recording unit 60 has a card slot into which a storage medium such as a memory card can be mounted. The recording unit 60 stores the image data and various data generated by the image processing unit 30 in a recording medium mounted in the card slot. The recording unit 60 has an internal memory. The recording unit 60 can also record the image data and various data generated by the image processing unit 30 in the internal memory.

  The system control unit 70 controls the overall processing and operation of the light field camera 1. The system control unit 70 has a CPU (Central Processing Unit). The system control unit 70 performs control to output the image data generated by the image processing unit 30 to the display unit 50 and display an image (live view image, still image, moving image) on the display screen 51 of the display unit 50. . Further, the system control unit 70 performs control to read out the image data recorded in the recording unit 60 and output the image data to the display unit 50 and display an image on the display screen 51 of the display unit 50.

  In addition, the system control unit 70 instructs the predetermined imaging conditions in order to perform imaging with predetermined imaging conditions (exposure time, frame rate, gain, etc.) on the imaging surface (pixel area 200) of the imaging device 20. A signal is output to the drive unit 21. The system control unit 70 also instructs the image processing unit 30 to execute image processing with predetermined control parameters (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate). The instruction signal is output to the image processing unit 30. Further, the system control unit 70 controls the recording unit 60 to record the image data by causing the recording unit 60 to output the image data generated by the image processing unit 30. Further, the system control unit 70 causes the recording unit 60 to output the position data and distance data of each subject detected by the image processing unit 30, and records the image data, the position data, and the distance data in association with each subject. Control for recording in the unit 60 is performed. In addition to the above control, the system control unit 70 also performs control such as adjustment of the aperture diameter of the aperture stop provided in the photographing optical system 11.

  Next, image processing executed by the image processing unit 30 of the light field camera 1 will be described. FIG. 6 is a flowchart for explaining image processing executed by the image processing unit 30 shown in FIG. In the processing shown in FIG. 6, the image processing unit 30 acquires RAW data output from the imaging unit 10 (step S1). Then, the image processing unit 30 generates image data of a predetermined in-focus position (focal length, focal position) based on the acquired RAW data (step S2).

  Specifically, as described above, the image processing unit 30 extracts pixel values of pixels at the same position with respect to each incident region in the pixel region 200, and reconstructs the extracted pixel values to obtain a plurality of pixels. A reconstructed image data group is generated. Then, the image processing unit 30 translates the plurality of reconstructed image data groups according to a predetermined focus position, and adds and averages the plurality of reconstructed image data groups after the parallel movement. As a result, image data set at a predetermined in-focus position is generated. In step S2, the image processing unit 30 generates a plurality of image data having different in-focus positions. The image data generated in step S2 includes still image data, moving image data, and live view image data.

  Next, the image processing unit 30 performs color signal processing (color tone correction), white balance adjustment, sharpness adjustment, gamma correction, gradation adjustment, and the like on the plurality of image data having different focus positions generated in step S2. Various image processing is executed (step S3).

  The image processing unit 30 detects one or more subjects from the generated image data (step S4). The image processing unit 30 outputs position data indicating the detected size and position of one or more subjects to the system control unit 70. Further, the image processing unit 30 detects the distance (focal length) to one or more subjects detected in step S4 (step S5). The image processing unit 30 outputs distance data indicating the detected distance to one or more subjects to the system control unit 70.

  Thereafter, the image processing unit 30 records a plurality of pieces of image data having different in-focus positions in the recording unit 60 (step S6). At this time, the system control unit 70 records the subject position data and distance data output from the image processing unit 30 in the recording unit 60 in association with each subject. The image processing unit 30 may record the RAW data in the recording unit 60 in association with the image data or the like.

  Note that a plurality of image data (first image data in the present embodiment), position data, and distance data recorded in the recording unit 60 with different in-focus positions are connected to the image synthesis apparatus 101 by the light field camera 1. Thus, the image data is stored in the image data storage area of the storage unit 120 by the image input unit 112.

  Next, image composition by the arithmetic processing unit 110 according to the first embodiment will be described. FIG. 7 is a diagram illustrating examples of the first image 601, the second image 602, and the composite image 603. In the present embodiment, as illustrated in FIG. 7, image synthesis by the arithmetic processing unit 110 will be described by taking as an example a case where a user inserts the second image 602 into the first image 601 to create the synthesized image 603. Here, the first image 601 is a refocus image in which the focus position can be changed in the depth direction. The second image 602 is a two-dimensional image in which the focus position cannot be changed in the depth direction. The composite image 603 is a refocus image that can change the in-focus position in the depth direction.

  In the first image 601 shown in FIG. 7, the person O1 is displayed on the left side. In the first image 601, the tree O2 is displayed on the right side and the house O3 is displayed near the center. In the first image 601, the tree O2 and the house O3 are located outside the range of the depth of field, so these images are blurred. In the example shown in FIG. 7, the second image 602 is an image of a person cut out from a certain two-dimensional image. In the example shown in FIG. 7, in the composite image 603, the second image 602 (person) is arranged in front of the house O3 in the first image 601. The second image 602 is an image focused on a person.

  FIG. 8 is a flowchart for explaining image composition processing executed by the arithmetic processing unit 110 according to the first embodiment. In the processing shown in FIG. 8, when a control program (for example, an image composition program such as application software for performing image composition) is activated in response to the operation of the keyboard 103 or the mouse 104 by the user, the display control unit 111 The GUI image is read from the storage unit 120 and displayed on the display screen 500 of the display device 102 (step S11).

  When the first image data (for example, the image data of the first image 601 shown in FIG. 7) is selected by the user's operation of the keyboard 103 or the mouse 104, the image generation unit 114a stores the image data in the storage unit 120. The first image data stored in the region and the position data and distance data of the subject in the first image data associated with the first image data are read and acquired (step S12). When the second image data (for example, the second image 602 shown in FIG. 7) is selected by the user operating the keyboard 103 or the mouse 104, the image generation unit 114a stores the image data in the image data storage area of the storage unit 120. The acquired second image data is read and acquired (step S12).

  As shown in FIG. 7, when the second image data inserted into the first image data is a part of the two-dimensional whole image data, the image generation unit 114a is set by the user. A process of cutting out the second image data from the entire image data is performed according to the operation of the keyboard 103 or the mouse 104.

  Based on the first image data, position data, and distance data acquired in step S12, the display control unit 111 determines the size and position of each of the subjects O1 to O3 in the first image data, and the light field camera 1 at the time of shooting. To the subjects O1 to O3. The display control unit 111 then determines the light field camera 1 and the subjects O1 to O3 based on the recognized sizes of the subjects O1 to O3, the positions in the first image data, and the distances to the subjects O1 to O3. Is displayed on the GUI (step S13).

  9 and 10 are diagrams illustrating display examples of an overhead view 300 on the graphical user interface (GUI) of the first embodiment. As shown in FIGS. 9 and 10, the display screen 500 of the display device 102 has, as a GUI, a distance relationship between the light field camera 1 and each of the subjects O1 to O3, a focus position of each of the subjects O1 to O3, and An overhead view 300 representing the range of the depth of field of one image data (the synthesized image data after the synthesis of the second image data. The same applies in FIGS. 9 and 10 below) is displayed. In the overhead view 300 shown in FIGS. 9 and 10, the light field camera 1 is displayed at the camera position d0, the person O1 is displayed at the focus position d1, the tree O2 is displayed at the focus position d2, and the focus position d3. The house O3 is displayed on the screen.

  The focus position of the first image data shown in FIG. 9 is the focus position d1 of the person O1. Further, the range of the depth of field of the first image data shown in FIG. 9 is a range before and after the focus position d1 of the person O1 in the depth direction. That is, the range of the depth of field of the first image data shown in FIG. 9 is a range from the distance d1a from the light field camera 1 to the distance d1b from the light field camera 1. The range of the focus position and depth of field of the first image data shown in FIG. 9 is automatically set by the image processing unit 30 of the light field camera 1. Further, the focus position of the first image data shown in FIG. 10 is also the focus position d1 of the person O1. However, the range of the depth of field of the first image data shown in FIG. 10 is a range from the distance d1a from the light field camera 1 to the distance d1c from the light field camera 1. That is, the depth of field of the first image data shown in FIG. 10 is wider than the depth of field of the first image data shown in FIG.

  Further, as shown in FIGS. 9 and 10, a small display area 501 is provided at the upper right of the display screen 500 of the display device 102. In this display area 501, a composite image 603 (or the first image 601 before composition) is displayed. In this way, by displaying the composite image 603 in the display area 501, the user can visually recognize the composite image 603 being created using the GUI.

  Next, the image generation unit 114a inserts the second image data at a predetermined position in the depth direction of the first image data in accordance with the operation of the mouse 104 by the user, and generates composite image data (step S14). . In the example shown in FIGS. 9 and 10, the user performs an arbitrary position on the overhead view 300 (a position at a distance d11 from the light field camera 1 in FIG. Then, the second image 602 is inserted at a distance d12 from the light field camera 1. The image generation unit 114a generates composite image data corresponding to the insertion position of the second image data in the depth direction of the first image data. Then, the image generation unit 114a displays a composite image 603 based on the generated composite image data in the display area 501.

  The changing unit 114b changes one or both of the in-focus position and the depth of field of the composite image data according to the insertion position of the second image data (step S15). Specifically, when a focus position or a range of depth of field is designated by the user's operation of the mouse 104 or the like, the changing unit 114b has a plurality of first positions having focus positions close to the designated distance. Get image data. The image data at the in-focus position designated by the user is intermediate image data of the plurality of first image data acquired by the changing unit 114b. For this reason, the changing unit 114b performs weighted average processing that adds a weight according to the distance to the acquired plurality of first image data and then averages the image data at the in-focus position designated by the user. Is generated. In addition, the changing unit 114b generates image data in the range of the depth of field specified by the user by combining the image data at a plurality of different in-focus positions.

  In the example shown in FIG. 9, the second image 602 is inserted at a position of a distance d11 within the range of the depth of field in the first image data (within the range from the distance d1a to the distance d1b). In this case, since the second image 602 is located within a range where it appears to be in focus, the composite image 603 is an image that does not feel uncomfortable for the user. On the other hand, in the example shown in FIG. 10, the second image 602 is inserted at a position of a distance d12 outside the range of the initial depth of field in the first image data. In this case, since the second image 602 is located within the out-of-focus range, the composite image 603 is an image that is uncomfortable for the user. Therefore, for example, when the user operates the mouse 104, the range of the depth of field is set such that the insertion position of the second image 602 (the position at the distance d12) is included in the range of the depth of field of the first image data. Is changed to a range from the distance d1a to the distance d1c.

  Note that the changing unit 114b may automatically change one or both of the in-focus position and the depth of field of the composite image data in accordance with the insertion position of the second image data. In this case, the changing unit 114b changes, for example, the depth of field of the composite image data to a range including the insertion position of the second image data so that the composite image 603 becomes an uncomfortable image.

  Next, the image generation unit 114a processes (that is, edits) one or both of the first image data and the second image data as necessary (step S16). For example, the image generation unit 114a determines the size of the subjects O1 to O3 included in the first image 601 and the second image in accordance with the operation of the mouse 104 or the like by the user so that the composite image 603 becomes an uncomfortable image. The size of 602 is changed. When the second image data is located outside the range of the depth of field of the composite image data, the image generation unit 114a displays the second image 602 according to the operation of the mouse 104 or the like by the user. The resolution of the second image data is changed so that it appears blurred. Note that the image generation unit 114a determines the subjects O1 to O3 and the second image included in the first image 601 according to the relationship between the focus position and depth of field of the composite image data and the insertion position of the second image data. The size of 602 may be automatically changed, and the resolution of the second image data may be automatically changed.

  Thereafter, the storage control unit 114c stores the composite image data in the image data storage area of the storage unit 120 in accordance with the operation of the user's mouse 104 or the like (step S17).

  As described above, in the first embodiment, a composite image that generates composite image data by inserting the second image data at a predetermined position in the depth direction in the first image data in which the focus position can be changed in the depth direction. A generation unit 114a and a changing unit 114b that changes the in-focus position of the composite image data according to the insertion position of the second image data in the depth direction in the first image data. According to such a configuration, it is possible to generate a composite image 603 that does not give the user a feeling of strangeness. For example, it is possible to avoid the generation of a composite image having a sense of incongruity such that the focused second image data is inserted at a position that is not the focus position in the depth direction in the first image data.

  In the first embodiment, the changing unit 114b changes the in-focus position of the composite image data to the insertion position of the second image data. According to such a configuration, it is possible to generate composite image data in which the second image data is inserted at the in-focus position.

  In the first embodiment, the changing unit 114b changes the range of the depth of field of the composite image data according to the insertion position of the second image data in the depth direction in the first image data. According to such a configuration, it is possible to generate composite image data without a sense of incongruity, and the width of the position in the depth direction where the second image data can be inserted is widened. In the first embodiment, the changing unit 114b reliably generates composite image data without a sense of incongruity when the range of the depth of field of the composite image data is changed to a range including the insertion position of the second image data. can do. Further, in the first embodiment, the composite image generation unit 114a edits the second image data according to the insertion position of the second image data in the depth direction in the first image data, so that there is a further uncomfortable composition. Image data can be generated.

  In the first embodiment, a diagram 300 representing the distance relationship between the imaging device 1 and the subject, the in-focus position, and the range of the depth of field in one or both of the first image data and the second image data is displayed. The display control part 111 displayed on the part 102 is provided. According to such a configuration, the distance relationship between the imaging device 1 and the subject, the in-focus position, and the range of the depth of field can be easily recognized by the user. Therefore, it is possible to easily perform the operation of generating the composite image data without a sense of incongruity by the user.

Second Embodiment
In the first embodiment described above, the image composition unit 114 generates the composite image data by inserting the second image data, which is two-dimensional image data whose focus position cannot be changed, into the first image data. Met. On the other hand, in the second embodiment, composite image data is generated by inserting second image data, which is refocus image data whose focus position can be changed, into the first image data.

  FIG. 11 is a flowchart for explaining image composition processing executed by the arithmetic processing unit 110 according to the second embodiment. In the processing shown in FIG. 11, when a control program is activated in response to a user's operation of the keyboard 103 or the mouse 104, the display control unit 111 reads a GUI image from the storage unit 120 and displays on the display device 102. It is displayed on the screen 500 (step S21).

  When the first image data (for example, the image data of the first image 601 shown in FIG. 7) is selected by the user's operation of the keyboard 103 or the mouse 104, the image generation unit 114a stores the image data in the storage unit 120. The first image data stored in the region and the position data and distance data of the subject in the first image data associated with the first image data are read and acquired (step S22). When the second image data is selected by the user's operation of the keyboard 103 or the mouse 104, the image generation unit 114a includes the second image data stored in the image data storage area of the storage unit 120 and the second image data. The subject position data and distance data in the second image data associated with the two image data are read out and acquired (step S22).

  Based on the first image data, position data, and distance data acquired in step S22, the display control unit 111 determines the size and position of each of the subjects O1 to O3 in the first image data, and the light field camera 1 at the time of shooting. To the subjects O1 to O3. The display control unit 111 also determines the size and position of the subject O11 in the second image data and the light field camera 1 at the time of shooting based on the second image data, the position data, and the distance data acquired in step S22. The distance to the subject O11 is recognized. The display control unit 111 then recognizes the size of each of the subjects O1 to O3, the position in the first image data, the distance to each of the subjects O1 to O3, the size of the subject O11, the position in the second image data, and Based on the distance to the subject O11, an overhead view 301 when the light field camera 1 and each of the subjects O1 to O3 are looked down and when the light field camera 1 and the subject O11 are looked down is displayed on the GUI (step). S23).

  12 and 13 are diagrams illustrating display examples of an overhead view 301 on the graphical user interface (GUI) of the second embodiment. As shown in FIGS. 12 and 13, the display screen 500 of the display device 102 has, as a GUI, a distance relationship between the light field camera 1 and each of the subjects O1 to O3, a focus position of each of the subjects O1 to O3, and An overhead view of the first image representing the range of the depth of field of one image data (the synthesized image data after the synthesis of the second image data. The same applies in FIGS. 12 and 13 below) is displayed. . In addition, the display screen 500 of the display device 102 includes, as a GUI, a second relationship representing a distance relationship between the light field camera 1 and the subject O11, a focus position of the subject O11, and a range of the depth of field of the second image data. An overhead view of the image is displayed. As described above, the overhead view 301 includes the overhead view of the first image and the overhead view of the second image.

  In the overhead view of the first image shown in FIGS. 12 and 13, the light field camera 1 is displayed at the camera position d0, the person O1 is displayed at the focus position d1, and the tree O2 is displayed at the focus position d2. House O3 is displayed at the focal position d3. In the overhead view of the second image shown in FIGS. 12 and 13, the light field camera 1 is displayed at the camera position d0, and the person O11 is displayed at the in-focus position d21.

  The focus position of the first image data shown in FIG. 12 is the focus position d1 of the person O1. Further, the range of the depth of field of the first image data shown in FIG. 12 is a range (a range from the distance d1a to the distance d1b) in the depth direction of the in-focus position d1 of the person O1. Further, the focus position of the second image data shown in FIG. 12 is the focus position d21 of the person O11. Further, the range of the depth of field of the second image data shown in FIG. 12 is a range in the depth direction of the in-focus position d21 of the person O11 (range from the distance d21a to the distance d21b).

  Further, the focus positions of the first image data and the second image data shown in FIG. 13 are both the focus position d21 of the person O11. Moreover, the range of the depth of field of the first image data and the second image data shown in FIG. 13 is the range in the depth direction of the in-focus position d21 of the person O11 (the range from the distance d21a to the distance d21b). ).

  As shown in FIGS. 12 and 13, a small display area 501 is provided on the upper right of the display screen 500 of the display device 102. In this display area 501, a composite image 604 (or the first image 601 before composition) is displayed.

  Next, the changing unit 114b determines the insertion position of the second image data, and changes the range to the range of the in-focus position and the depth of field of the composite image data according to the determined insertion position. Then, the changing unit 114b changes the focus position and the depth of field range of the first image data in accordance with the changed focus position and the depth of field range of the combined image data, and also changes the second image. The range of the data focus position and the depth of field is changed (step S24). For example, the changing unit 114b automatically adjusts one or both of the focus position and the depth of field range of the composite image data in accordance with one or both of the focus position and the depth of field range of the second image data. Thus, one or both of the in-focus position and the depth of field range of the first image data is changed in accordance with the in-focus position and the depth of field range of the composite image data. Note that the changing unit 114b may determine the insertion position of the second image data in accordance with the operation of the mouse 104 by the user.

  The changing unit 114b adjusts the focus position and the depth of field range of the composite image data in accordance with one or both of the focus position and the depth of field range of the first image data or the second image data. One or both of them is automatically changed, and one of the focus position and depth of field range of the second image data or the first image data is adjusted according to the focus position and depth of field range of the composite image data. Or both may be changed.

  Next, the image generator 114a inserts the second image data at a predetermined position in the depth direction of the first image data to generate composite image data (step S25). As shown in FIGS. 12 and 13, when the second image data inserted into the first image data is a part of the whole image data (for example, image data of the person O11), image generation is performed. The unit 114a performs a process of cutting out the second image data from the entire image data according to the operation of the keyboard 103 and the mouse 104 by the user.

  In the example illustrated in FIG. 13, for example, when the user performs a drag-and-drop operation of the mouse 104, the relationship between the distance to the subjects O1 to O3 of the first image data and the distance to the subject O11 of the second image data is matched. In this manner, the second image (image of the person O11) is inserted at a predetermined position on the overhead view 301 (a position at a distance d21 from the light field camera 1 in FIG. 13). The image generation unit 114a generates composite image data corresponding to the insertion position of the second image data in the depth direction of the first image data. Then, the image generation unit 114a displays a composite image 604 based on the generated composite image data in the display area 501.

  Next, the image generation unit 114a processes (that is, edits) one or both of the first image data and the second image data as necessary (step S26). For example, the image generation unit 114a changes the size of the subjects O1 to O3 included in the first image 601 and the size of the subject O11 included in the second image according to the operation of the mouse 104 or the like by the user. Note that the image generation unit 114a may automatically change the size of the subjects O1 to O3 included in the first image 601 and the size of the subject O11 included in the second image.

  Thereafter, the storage control unit 114c stores the composite image data in the image data storage area of the storage unit 120 in accordance with the user's operation of the mouse 104 or the like (step S27).

  As described above, in the second embodiment, the second image data in which the in-focus position can be changed in the depth direction and the second image data in which the in-focus position can be changed in the depth direction are used. The composite image generation unit 114a that inserts and generates composite image data matches the distance from the imaging device 1 to the subject of the first image data and the distance from the imaging device 1 to the subject of the second image data. A changing unit 114b for changing the in-focus position of the composite image data. According to such a configuration, it is possible to reliably generate composite image data that does not cause a sense of incongruity.

  In the second embodiment, the changing unit 114b includes the first image data and the first image data in accordance with the in-focus position and the depth of field of the composite image data before the composite image generation unit 114a generates the composite image data. The focus position and depth of field of the two image data are changed. According to such a configuration, the work burden at the time of image composition by the user is reduced. In the second embodiment, the changing unit 114b changes the in-focus position and the depth of field range of the composite image data to the in-focus position and the depth of field range of the first image data or the second image data. Even in this case, it is possible to reliably generate composite image data that does not cause a sense of incongruity.

  Note that the changing unit 114b may change the in-focus position and the range of the depth of field after combining the first image data and the second image data. Even if the focus position and the depth of field range of the first image data, the second image data, and the composite image data are changed by the user's operation of the mouse 104 or the like, the change unit 114b automatically It may be changed.

  In the second embodiment described above, the changing unit 114b matches the distance to the subject of the first image data and the distance to the subject of the second image data based on the absolute distance from the light field camera 1. The focus position and the depth of field range were changed. However, the changing unit 114b, based on the relative distance from the light field camera 1, adjusts the focus position and the field of view so that the distance to the subject of the first image data matches the distance to the subject of the second image data. The depth range may be changed. In this case, the image generation unit 114a changes (processes) the size and resolution of the subject so that the composite image does not feel uncomfortable.

  In addition, since the size of the far subject is small and the resolution is low, inserting image data of such a subject at a short distance in the depth direction of the first image data causes a sense of incongruity due to the low resolution image. Therefore, in the first embodiment and the second embodiment described above, the display control unit 111 performs the second image in the first image data according to the resolution of the subject of the first image data and the resolution of the subject of the second image data. An overhead view showing a range in which data can be inserted is displayed on the display unit 102. According to such a configuration, it is possible to clearly indicate to the user a range in which a composite image that does not cause a sense of incongruity can be generated, and the operability for the user is improved.

<Image composition system>
In each of the above-described embodiments (the first embodiment and the second embodiment) and the modifications, the system is a stand-alone system. However, the present invention is not limited to such a system. For example, an image display system SYS connected through a communication network such as a client server system will be described. In the following description, components that are the same as or equivalent to those in the first to third embodiments are given the same reference numerals, and descriptions thereof are omitted or simplified.

  FIG. 14 is a block diagram showing the configuration of the image composition system SYS. As illustrated in FIG. 14, the image display system SYS includes a display control unit 111, an image input unit 112, an operation control unit 113, and a communication unit 115 in the arithmetic processing unit 110B of the client terminal 101B. The display control unit 111, the image input unit 112, and the operation control unit 113 correspond to the display control unit 111, the image input unit 112, and the operation control unit 113 illustrated in FIG. Further, the storage unit 120B illustrated in FIG. 14 corresponds to the storage unit 120 illustrated in FIG.

  As illustrated in FIG. 14, the arithmetic processing unit 110 </ b> B of the client terminal 101 </ b> B includes a communication unit 115 that transmits and receives data to and from the server (image composition device) 106 via the communication network 180. The arithmetic processing unit 160 of the server 106 includes a communication unit 161 that transmits / receives data to / from the client terminal 101 </ b> B via the communication network 180. In addition, the image composition unit 162 of the arithmetic processing unit 160 accumulates image data (first image data and second image data) transmitted from the client terminal 101B in the storage unit 170 in response to a request from the client terminal 101B. . Then, the image composition unit 162 executes image composition control based on the image data, similarly to the image composition unit 114 shown in FIG.

  The configuration of the communication unit 115 and the configuration of the communication unit 161 and the image composition unit 162 in the arithmetic processing unit 160 are executed by a control device such as a CPU according to a control program stored in the storage units 120B and 170. It is realized by.

  The user operates the keyboard 103 and mouse 104 connected to the client terminal 101 </ b> B to read out image data stored in the light field camera 1 and the storage unit 120 </ b> B, and transmits the read image data to the server 106. Then, the user operates the keyboard 103 and the mouse 104 to request the server 6 to display a GUI. In response to a GUI display request, the image composition unit 162 in the server 106 transmits image data of an overhead view on the GUI (image data on which a subject or the like is arranged) to the client terminal 101B. In the client terminal 101 </ b> B, when the communication unit 115 receives the overhead view image data on the GUI transmitted from the server 106, the display control unit 111 displays the received overhead view image data on the display screen 500 of the display device 102. .

  Further, the user operates the mouse 104 or the like to insert the second image data at a predetermined insertion position in the first image data based on the overhead view image displayed on the display screen 500, and to focus. Change the range of position and depth of field. The communication unit 115 transmits operation information operated by the user to the server 106. The image composition unit 162 of the server 106 inserts the second image data at a predetermined insertion position in the first image data according to the operation information from the client terminal 101B, and sets the range of the in-focus position and the depth of field. change. Then, the image composition unit 162 transmits the image data of the overhead view on the GUI after such processing to the client terminal 101B. In the client terminal 101 </ b> B, when the communication unit 115 receives the overhead view image data on the GUI transmitted from the server 106, the display control unit 111 displays the received overhead view image data on the display screen 500 of the display device 102. .

  By repeating such processing, the image composition unit 162 generates a composite image desired by the user. Then, the image composition unit 162 transmits the generated composite image data to the client terminal 101B based on the composite image data transmission request from the client terminal 101B.

  According to such a configuration, it is not necessary for the client terminal 101B to store a large amount of data, and costs can be reduced, for example, data management can be unified.

  In the image display system SYS shown in FIG. 14, the display control unit 111 of the client terminal 101B performs display control, but the present invention is not limited to this. For example, the client terminal 101B may be provided with a browser, and display content of an overhead view as shown in FIG.

<Modification of imaging device>
The light field camera 1 in the first embodiment described above may have a configuration in which the imaging device 1A and the electronic device 1B are separated.

  FIG. 15 is a block diagram illustrating a configuration in which the imaging apparatus and the electronic device are separated. In the configuration shown in FIG. 15, the imaging device 1 </ b> A is a device that captures an image of a subject and has a refocus function. The imaging apparatus 1A includes an imaging unit 10, an image processing unit 30, a work memory 40, an operation unit 55, a recording unit 60, and a first system control unit 70A. In the imaging apparatus 1A, the configurations of the imaging unit 10, the image processing unit 30, the work memory 40, the operation unit 55, and the recording unit 60 are the same as the configurations illustrated in FIG. Accordingly, the same components are denoted by the same reference numerals, and redundant description is omitted.

  The electronic device 1B is a device that displays images (still images, moving images, live view images). The electronic device 1B includes a display unit 50 and a second system control unit 70B. The configuration of the display unit 50 in the electronic device 1B is the same as the configuration illustrated in FIG. Accordingly, the same components are denoted by the same reference numerals, and redundant description is omitted.

  The first system control unit 70A includes a first communication unit 75A. The second system control unit 70B has a second communication unit 75B. The first communication unit 75A and the second communication unit 75B transmit and receive signals to each other in a wired or wireless manner. In such a configuration, the first system control unit 70A receives the second image data (the image data image-processed by the image processing unit 30 and the image data recorded in the recording unit 60) via the first communication unit 75A. It transmits to the communication part 75B. The second system control unit 70B causes the display unit 50 to display the image data received by the second communication unit 75B.

  Note that the imaging apparatus 1A includes, for example, a digital camera, a smartphone, a mobile phone, and a personal computer having an imaging function and a communication function, and the electronic device 1B includes, for example, a smartphone, a mobile phone, and a portable personal computer that have a communication function. Consists of a portable terminal such as a computer.

  According to the configuration as described above, in addition to the effects described in the first embodiment and the second embodiment, an image (refocused image) captured by the imaging device 1A using a mobile terminal such as a smartphone is an electronic device. It can be displayed on the display unit 50 of 1B. Note that the image data captured by the imaging device 1A may be transmitted to the image composition device 101 via the system control unit 70A, or may be transmitted to the image composition device 101 via the system control unit 70B.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various modifications or improvements can be added to the above-described embodiment without departing from the spirit of the present invention. In addition, one or more of the requirements described in the above embodiments may be omitted. Such modifications, improvements, and omitted forms are also included in the technical scope of the present invention. In addition, the configurations of the above-described embodiments and modifications can be applied in appropriate combinations.

  For example, in each of the above-described embodiments, the light field camera 1 is cited as an example of the imaging apparatus. However, the present invention is not limited to this. For example, a camera array in which a plurality of cameras are arranged may be used. Even in such a camera array, a plurality of images having parallax can be acquired.

  Further, in each of the above-described embodiments, the image composition unit 114 changes the focus position and the range of the depth of field using the refocus image captured by the light field camera 1. However, the image composition unit 114 can change the in-focus position (focal position) and the range of depth of field using a plurality of images captured by a digital camera (imaging device) having a focus bracket function. It is. That is, the first image data in the first embodiment described above, and the first image data and the second image data in the second embodiment described above are captured by the digital camera (imaging device) while changing the focus position. Image data generated based on a plurality of image data with different in-focus positions is included. Here, the focus bracket refers to taking a plurality of images with different in-focus positions by changing the in-focus position while continuously shooting. The image composition unit 114 selects a plurality of image data with different in-focus positions according to the in-focus position and depth of field specified by the user, and combines the selected image data so as to achieve in-focus. The range of position and depth of field can be changed.

  In each of the above-described embodiments, the image processing unit 30 of the light field camera 1 generates a plurality of image data having different in-focus positions based on the RAW data, detects the subject, and detects the distance to the subject. It was. However, the image composition unit 114 acquires RAW data from the light field camera 1, generates a plurality of image data with different in-focus positions based on the acquired RAW data, detects a subject, and detects a distance to the subject. May be. Further, in each of the above-described embodiments, the number of second image data inserted into the first image data is not limited to one and may be plural.

  Further, in each of the above-described embodiments, the image composition unit 114 performs a weighted average process that averages a plurality of image data (first image data, second image data) after adding a weight according to the distance. The range of focus position and depth of field was changed. However, the image composition unit 114 may change the range of the focus position and the depth of field by generating image data of a predetermined focus position and depth of field based on the RAW data.

  Further, the image processing unit 30 and the system control unit 70 illustrated in FIG. 4 may be configured integrally. In this case, the system control unit having one CPU executes the processing based on the control program, thereby taking on the functions of the image processing unit 30 and the system control unit 70. In the configuration shown in FIG. 15, the image processing unit 30 and the first system control unit 70A may be configured integrally. In this case, the system control unit having one CPU performs the processing based on the control program, thereby performing the function of the image processing unit 30 and the function of the first system control unit 70A.

  In the first embodiment described above, the color filter array is a Bayer array, but an array other than this array may be used.

  Further, the light field camera 1 according to each of the above-described embodiments may include a zoom function for performing zooming adjustment automatically or in accordance with an operation of the operation unit 55 or the like by a user. In this case, the drive control unit 13 performs drive control of the photographing optical system 11 (for example, a zooming lens) based on control information transmitted from the system control unit 70 in order to perform zooming adjustment. The light field camera 1 can change the focal position after photographing the subject, but may include an autofocus function that automatically adjusts the focal position before photographing the subject. In this case, the drive control unit 13 performs drive control of the photographing optical system 11 (for example, a focusing lens) based on control information transmitted from the system control unit 70 in order to adjust the focus of the photographing optical system 11. .

  In the second embodiment described above, the image composition unit 114 generates the composite image data by inserting the second image data into the first image data. However, the first image data is inserted into the second image data. Thus, the composite image data may be generated. Further, the image composition unit 114 may change the in-focus position and the depth of field according to the operation of the user's mouse 104 or the like or automatically.

  DESCRIPTION OF SYMBOLS 1 ... Light field camera (imaging apparatus), 1A ... Imaging apparatus, 1B ... Electronic device, 101 ... Image composition apparatus, 101B ... Client terminal, 102 ... Display apparatus (display part), 106 ... Server (image composition apparatus), 110 , 110B ... arithmetic processing unit, 111 ... display control unit, 112 ... image input unit, 113 ... operation control unit, 114 ... image composition unit, 114a ... image generation unit (composite image generation unit), 114b ... change unit, 114c ... Storage control unit, 115 ... communication unit, 120, 120B ... storage unit, 160 ... arithmetic processing unit, 161 ... communication unit, 162 ... image composition unit, 170 ... storage unit

Claims (14)

A composite image generation unit that generates composite image data by inserting the second image data at a predetermined position in the depth direction in the first image data in which the focus position can be changed in the depth direction;
An image synthesizing apparatus comprising: a changing unit that changes a focus position of the synthesized image data in accordance with an insertion position of the second image data in the depth direction in the first image data.   The image synthesizing apparatus according to claim 1, wherein the changing unit changes a focus position of the composite image data to an insertion position of the second image data.   The said change part changes the range of the depth of field of the said synthesized image data according to the insertion position of the said 2nd image data of the said depth direction in the said 1st image data. Image composition device.   The image synthesizing apparatus according to claim 3, wherein the changing unit changes a range of a depth of field of the synthesized image data to a range including an insertion position of the second image data.   5. The first image data according to claim 1, wherein the first image data includes image data generated based on a plurality of pieces of image data having different in-focus positions, which are captured by the imaging device while changing the in-focus positions. The image composition apparatus described. Composite image generation for generating composite image data by inserting second image data whose focus position can be changed in the depth direction into a predetermined position in the depth direction in the first image data whose focus position can be changed in the depth direction And
A changing unit that changes a focus position of the composite image data so that a relationship between a distance from the imaging device to the subject of the first image data matches a distance from the imaging device to the subject of the second image data; An image composition apparatus comprising:   The changing unit is configured to adjust the first image data and the second image data in accordance with a focus position and a depth of field of the composite image data before the composite image generation unit generates the composite image data. The image composition apparatus according to claim 6, wherein the focus position and the depth of field are changed.   The change unit changes a range of a focus position and a depth of field of the composite image data to a range of a focus position and a depth of field of the first image data or the second image data. The image composition device according to claim 7.   The first image data and the second image data include image data generated based on a plurality of pieces of image data having different in-focus positions, which are captured by the imaging device while changing the in-focus positions. The image composition device according to claim 8.   A display showing a distance relationship between the imaging device and the subject, the in-focus position, and the range of the depth of field in one or both of the first image data and the second image data is displayed on the display unit. The image composition device according to any one of claims 1 to 9, further comprising a display control unit.   The display control unit displays a diagram representing an insertable range of the second image data in the first image data according to the resolution of the subject of the first image data and the resolution of the subject of the second image data. The image composition device according to claim 10, which is displayed on a part.   The composite image generation unit according to any one of claims 1 to 11, wherein the second image data is edited according to an insertion position of the second image data in the depth direction in the first image data. Image composition device. On the computer,
A composite image generation process for generating composite image data by inserting the second image data at a predetermined position in the depth direction in the first image data in which the focus position can be changed in the depth direction;
An image composition program for executing a change process for changing a focus position of the composite image data according to an insertion position of the second image data in the depth direction in the first image data. On the computer,
Composite image generation for generating composite image data by inserting second image data whose focus position can be changed in the depth direction into a predetermined position in the depth direction in the first image data whose focus position can be changed in the depth direction Processing,
A change process for changing the in-focus position of the composite image data so that the relationship between the distance from the imaging device to the subject of the first image data and the distance from the imaging device to the subject of the second image data match. An image composition program for executing.