JP6566091B2 - Image generation device and image search device - Google Patents

Description

  The present invention relates to an image generation device, an image search device, an image generation program, and an image search program.

  As an apparatus for searching for an image, for example, there is one described in Patent Document 1. In this apparatus, the CPU extracts a partial image such as a person's face from the image, and calculates numerical data obtained by quantifying the positions and positional relationships of the eyes and nose in the extracted partial image. Then, the CPU performs an image search by comparing the numerical data with numerical data corresponding to the search information. Further, the CPU narrows down the partial images to be compared based on the focus position (focus position) at the time of image capture.

JP 2005-80216 A

  However, the above-described apparatus is not suitable for searching for an image (referred to as a refocus image) whose focus position can be changed in the depth direction, and there is still room for improvement.

  An object of an aspect of the present invention is to perform an image search suitable for a refocus image whose focus position can be changed in the depth direction.

According to an aspect of the present invention, a search unit that searches for image data that can generate an image including a specific subject from image data that can generate an image with a changed focal position, and image data that is searched by the search unit An image generation apparatus is provided that includes a generation unit that generates an image focused on a predetermined position within a predetermined range including a specific subject in the depth direction.
According to an aspect of the present invention, a search unit that searches for image data that can generate an image including a specific subject from image data that can generate an image with a changed focal position, and image data that is searched by the search unit And a display unit that displays the generated image, and an image search device is provided in which the image displayed on the display unit is an image including a specific subject.
According to the first aspect of the present invention, a detection unit that detects a subject included in image data whose focus position can be changed in the depth direction and detects the position of the detected subject in the depth direction, and a subject to be searched A designation unit for designating accompanying information about the subject, a search unit for searching for a subject corresponding to the accompanying information designated by the designation unit, and a predetermined in-focus position according to the position in the depth direction of the subject searched by the search unit An image generation apparatus including an image generation unit that generates image data is provided.

  According to the second aspect of the present invention, a detection unit that detects a subject included in one or a plurality of image data whose focus position can be changed in the depth direction, and detects a position of the detected subject in the depth direction; An image search apparatus is provided that includes a specifying unit that specifies a position in the depth direction of a subject to be searched, and a search unit that searches for image data including a subject corresponding to the position specified by the specifying unit.

  According to the third aspect of the present invention, the computer detects the subject included in the image data whose focus position can be changed in the depth direction, and detects the position of the detected subject in the depth direction, and the search A specification process for specifying accompanying information related to the target subject, a search process for searching for a subject corresponding to the accompanying information specified in the specification process, and a predetermined match according to the position in the depth direction of the subject searched by the search process. An image generation program for executing image generation processing for generating image data at a focal position is provided.

  According to the fourth aspect of the present invention, the computer detects a subject included in one or a plurality of image data whose focus position can be changed in the depth direction, and detects a position in the depth direction of the detected subject. An image search program is provided that executes a process, a specifying process that specifies a position in the depth direction of a subject to be searched, and a searching process that searches for image data including a subject corresponding to the position specified in the specifying process. The

  According to the aspect of the present invention, it is possible to perform an image search suitable for a refocus image whose focus position can be changed in the depth direction.

It is a block diagram which shows the structure of an image search / generation apparatus (image search apparatus). It is sectional drawing which shows schematic structure of the optical system part in the light field camera which is an example of an imaging device. It is a figure explaining the pixel arrangement | sequence and incident area of an image pick-up element. It is a block diagram which shows the structure of a light field camera. It is a figure explaining the reconstruction of an image. It is a flowchart for demonstrating the image processing which the calculating part shown in FIG. 1 performs. It is a figure which shows the distance relationship between a light field camera and each to-be-photographed object. It is a figure which shows the example of the EXIF information added to image data. It is a figure which shows the example of a display of the image memorize | stored in the memory | storage part. It is a flowchart for demonstrating the image search and production | generation process which an arithmetic processing part performs. It is a figure which shows the example of a display of the image search screen of 1st Embodiment. It is a figure which shows the example of a display of the image search screen of 1st Embodiment. It is a figure which shows the example of a display of the image search screen of 1st Embodiment. It is a figure which shows the example of a display of the image search screen of 2nd Embodiment. It is a figure which shows the example of a display of the image which has parallax. It is a block diagram which shows the structure of an image search and production system.

  Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to this. Further, in the drawings, in order to describe the embodiment, the scale may be appropriately changed and expressed, for example, a part of the drawing may be enlarged or emphasized.

<First Embodiment>
FIG. 1 is a block diagram showing the configuration of the image search / generation apparatus 101. As illustrated in FIG. 1, an image search / generation device (image search device, image generation device) 101 is connected to a light field camera 1, a display device 102, a keyboard 103, and a mouse 104, which are examples of an imaging device. . Here, the light field camera 1 acquires a large number of light rays incident on the camera by the imaging unit as a light field (light space). Then, the light field camera 1 generates a refocused image in which the in-focus position is changed in the depth direction by performing image processing on the acquired light field information. In the following description, information on a light field that is a set of rays in a three-dimensional space is referred to as ray information or RAW data.

  The image search / generation apparatus 101 searches for a subject included in the image data captured by the light field camera 1 based on information specified by the user, and focuses on the position in the depth direction of the searched subject. Generate data. Note that the direction from the light field camera 1 toward the subject is referred to as the back side direction, and the opposite side direction is referred to as the near side direction. The direction on the back side and the direction on the near side are referred to as the depth direction.

  The image search / generation apparatus 101 is realized by a personal computer, for example. As illustrated in FIG. 1, the image search / generation apparatus 101 includes an arithmetic processing unit 110 and a storage unit 120. The arithmetic processing unit 110 controls the search and generation of image data. The storage unit 120 stores RAW data acquired by the light field camera 1, image data generated from the RAW data (including accompanying information added to the image data), and the like. In the present embodiment, the accompanying information added to the image data is EXIF information (Exchangeable Image file Format). The EXIF information includes, for example, shooting scene information (portrait, landscape, etc.) added to each image data, subject type data indicating the type (type) of the subject added to each subject in the image data, and image Subject identification data for identifying individual subjects added for each subject in the data, information on the focus position of the subject added for each subject (that is, a distance indicating the distance from the light field camera 1 to the subject) Data) (see FIG. 8 described later). The subject identification data includes feature points of the subject, names of subjects, and the like.

  The feature point of the subject refers to characteristic information of the subject for identifying whether the subject is the same subject or a different subject. For example, when the subject is a person, information such as the shape of the face of the person, the shape of the eyes, the positional relationship between the eyes and the nose is used as the feature point, and when the subject is a building, information such as the color and shape of the building Is a feature point.

  As shown in FIG. 1, the arithmetic processing unit 110 includes a display control unit 111, an image input unit 112, an operation control unit 113, and a calculation unit 114. The display control unit 111 performs control to display an image search screen (see FIGS. 11 to 13) for the user to search for image data on the display screen 500 of the display device 102 such as a liquid crystal display. . The image search screen is a graphical user interface (GUI) used when a user searches for an image. Further, the display control unit 111 detects a plurality of subjects included in the plurality of image data, and cuts out the detected image data of the plurality of subjects. Then, the display control unit 111 displays an image based on the extracted image data of the plurality of subjects on the first display region (first display region 501 for displaying the subject list) of the display screen 500 of the display device 102; see FIG. ) Is displayed.

  In addition, the display control unit 111 displays images based on the image data of the plurality of subjects displayed in the first display area on the display device 102 by the user's operation of the mouse 104 or the like (for example, drag and drop operation). Control to move to the second display area of the screen 500 (second display area 502 for displaying the subject to be searched; see FIG. 11) is performed. In addition, the display control unit 111 performs control to change the display position in the depth direction of the image based on the image data of a plurality of subjects by an operation (for example, drag and drop operation) of the mouse 104 by the user.

  The image input unit 112 is connected to the light field camera 1 to input RAW data and image data (including accompanying information) stored in the light field camera 1. The image input unit 112 stores RAW data and image data (including accompanying information) input from the light field camera 1 in an image data storage area in the storage unit 120. The image input unit 112 acquires RAW data (RAW data, image data) through a communication network (not shown) such as the Internet, and stores the acquired RAW data in the image data storage area in the storage unit 120. Is also possible.

  The operation control unit 113 is connected to a keyboard 103 and a mouse 104 that constitute an operation unit that can be operated by a user. The operation control unit 113 inputs information according to the operation of the keyboard 103 and the mouse 104 by the user, and outputs the information to the display control unit 111 and the calculation unit 114.

  As shown in FIG. 1, the calculation unit 114 includes an image generation unit 114a, a detection unit 114b, a designation unit 114c, and a search unit 114d. When the RAW data is acquired from the light field camera 1 (see step S1 in FIG. 6), the image generation unit 114a generates a plurality of image data having different in-focus positions based on the RAW data. In addition, the image generation unit 114a performs various image processing such as color signal processing (tone correction), white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on a plurality of image data with different in-focus positions. . Further, the image generation unit 114a searches based on the RAW data when the search unit 114d described later searches for image data including a subject (subject to be searched) corresponding to the search condition (see step S19 in FIG. 10). The image data focused on the position in the depth direction of the subject searched by the unit 114d is generated.

  When the detection unit 114b acquires RAW data from the light field camera 1 (see step S1 in FIG. 6), the detection unit 114b includes one or more subjects (hereinafter referred to as “subject”) in the image data generated based on the RAW data by the image generation unit 114a. , A subject to be registered). In addition, the detection unit 114b extracts the feature points of the detected subject to be registered. Then, the detection unit 114b sequentially detects the feature points of the registered subject to be registered and the feature points of a plurality of subjects included in one or a plurality of image data stored in the storage unit 120, and detects them. It is determined whether image data including the same subject as the subject to be registered exists in the storage unit 120.

  When the detection unit 114b determines that image data including the same subject as the registration target subject does not exist in the storage unit 120, the detection unit 114b adds new subject type data and subject identification to the EXIF information of the image data including the registration target subject. Add data for each subject. On the other hand, when the detection unit 114b determines that image data including the same subject as the subject to be registered exists in the storage unit 120, the image in the storage unit 120 is included in the EXIF information of the image data including the subject to be registered. The same data as the subject type data and subject identification data of the EXIF information added to the data is added for each subject. Note that the detection unit 114b also adds shooting scene information for each image data to the EXIF information of the image data including the subject to be registered.

  In addition, the detection unit 114b detects the position in the depth direction of the subject to be registered (the focus position of the subject). The detection unit 114b adds distance data (in-focus position) indicating the position of the registration target subject in the depth direction to the EXIF information of the image data including the registration target subject for each subject. The calculation unit 114 stores the image data to which the EXIF information is added together with the RAW data in the storage unit 120.

  The designation unit 114c designates a search condition for image data when the user searches for image data. Specifically, the operation control unit 113 outputs information input in accordance with the operation of the keyboard 103 and the mouse 104 to the user for searching for image data. The designation unit 114c inputs information from the operation control unit 113 and designates the input information as a search condition for image data. The search condition is information on at least one of the above-described subject type data (person, house, tree, mountain, etc.), subject identification data (subject feature point, subject name), and subject depth direction position. It is.

  The search unit 114d searches the image data stored in the storage unit 120 for image data including a subject corresponding to the search condition specified by the specifying unit 114c. Note that the display control unit 111 displays a list of images that are image search results of the search unit 114 d on the display screen 500 of the display device 102.

  The configuration of the display control unit 111, the image input unit 112, the operation control unit 113, and the calculation unit 114 (the image generation unit 114a, the detection unit 114b, the specification unit 114c, and the search unit 114d) in the calculation processing unit 110 is as follows. This corresponds to processing or control executed by a control device such as a CPU (Central Processing Unit) in accordance with a control program (image generation program, image search program) stored in the storage unit 120.

  In the storage unit 120, an image data storage area for storing RAW data and image data (accompanying information) is formed. In addition, the storage unit 120 stores various information necessary for display such as an image search screen by the display control unit 111. Further, as described above, the storage unit 120 has a control program (image generation program) for executing the processing of the display control unit 111, the image input unit 112, the operation control unit 113, and the calculation unit 114 in the calculation processing unit 110. , Image search program) and the like are also stored.

  Next, the configuration of the light field camera 1 shown in FIG. 1 will be described. FIG. 2 is a cross-sectional view illustrating a schematic configuration of an optical system portion in a light field camera 1 which is an example of an imaging apparatus. In FIG. 2, the right direction (optical axis direction) of the paper surface is the X axis, the upward direction of the paper surface orthogonal to the X axis is the Z axis, and the direction from the back of the paper surface orthogonal to the X axis and the Z axis is the direction from the back. Is the Y axis. FIG. 2 is a cross-sectional view when the optical system portion in the light field camera 1 is cut along a plane including the optical axis.

  A typical function of the light field camera 1 is a refocus function for generating an image whose focal length is changed by post-processing after shooting. Therefore, the light field camera 1 is also called a refocus camera.

  The light field camera 1 of the present embodiment includes a photographing optical system (imaging optical system) 11 and a microlens array 12 as its optical system part. FIG. 2 also shows the image sensor 20 that is a configuration other than the optical system portion in the light field camera 1.

  The photographing optical system 11 is a single focus lens composed of a plurality of lens groups. The photographing optical system 11 forms an image of a light beam from the subject in the vicinity of its focal plane (imaging plane). Further, the photographing optical system 11 includes an aperture stop (not shown) that controls an aperture value (F value) during photographing. Since the aperture value can be changed by reconstructing the image after shooting, the aperture diameter of the aperture stop may be fixed. As shown in FIG. 2, a microlens array 12 and an image sensor 20 are sequentially arranged near the focal plane of the photographing optical system 11.

  The microlens array 12 has a configuration in which a plurality of microlenses (positive lenses) having positive power are two-dimensionally arranged. In the microlens array 12 shown in FIG. 2, only three microlenses are arranged in the Z-axis direction, but actually, a large number of microlenses are arranged in the Z-axis direction and the Y-axis direction.

  The image pickup device 20 is disposed on the rear side of the microlens array 12 (opposite side of the photographing optical system 11) with a slight gap. The image sensor 20 receives light beams that have passed through the microlenses of the microlens array 12 by a plurality of photoelectric conversion elements, and photoelectrically converts the light beams received by the photoelectric conversion elements to generate a plurality of pixel signals. The plurality of photoelectric conversion elements are arranged in a matrix, and each photoelectric conversion element forms a pixel. In the present embodiment, the image sensor 20 is constituted by a two-dimensional solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor).

  In the example shown in the partially enlarged view of FIG. 2, a plurality of pixels are arranged on the rear side of each of the plurality of microlenses 12a, 12b, and 12c. Specifically, three pixels 21a, 21b, 21c are arranged on the rear side of the micro lens 12a, three pixels 22a, 22b, 22c are arranged on the rear side of the micro lens 12b, and on the rear side of the micro lens 12c. Three pixels 23a, 23b, and 23c are arranged.

  Three pixels 21a, 21b, and 21c are pixels 21 corresponding to the microlens 12a, three pixels 22a, 22b, and 22c are pixels 22 corresponding to the microlens 12b, and three pixels 23a, 23b, and 23c are microscopic. This is a pixel 23 corresponding to the lens 12c. In the partially enlarged view of FIG. 2, three pixels are shown as the pixels corresponding to the microlenses 12a, 12b, and 12c, but in reality, a large number of pixels are arranged in the Z-axis direction and the Y-axis direction ( (See FIG. 3 described later).

  Next, acquisition of a light field by the light field camera 1 will be briefly described with reference to FIG. In FIG. 2, it is assumed that the subject at the position A focuses on the position of the microlens array 12 by the photographing optical system 11. The light beam from the point A1 on the subject passes through the partial region 11a on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21c of the image sensor 20. Further, another light ray from the point A1 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21b of the imaging device 20. Further, another light ray from the point A1 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12a, and is received by the pixel 21a of the imaging device 20.

  Similarly, the light beam from the point A2 on the subject passes through the partial area 11a on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22c of the image sensor 20. Further, another ray from the point A2 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22b of the image sensor 20. Further, another light ray from the point A2 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12b, and is received by the pixel 22a of the imaging device 20.

  Further, the light beam from the point A3 on the subject passes through the partial area 11a on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23c of the image sensor 20. Further, another light ray from the point A3 on the subject passes through the partial region 11b on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23b of the image sensor 20. Further, another light ray from the point A3 on the subject passes through the partial region 11c on the pupil of the photographing optical system 11, passes through the microlens 12c, and is received by the pixel 23a of the image sensor 20. As described above, the light beam from the subject at the position A is received by the pixels of the image sensor 20 located on the rear side of the microlenses 12a, 12b, and 12c, whereby the brightness and the traveling direction are acquired as light beam information. The

  Of the three pixels corresponding to the microlenses 12a, 12b, and 12c, the image generated by the pixel signals of the lower pixels 21c, 22c, and 23c is an image of light rays that have passed through the partial region 11a. In addition, the image generated by the pixel signals of the middle pixels 21b, 22b, and 23b is an image of light rays that have passed through the partial region 11b. Further, the image generated by the pixel signals of the upper pixels 21a, 22a, and 23a is an image of light rays that have passed through the partial region 11c. For example, when the number of pixels corresponding to each microlens is N, N reconstructed images configured by arranging pixel values of pixels located at the same position with respect to the microlens are configured to capture the photographing optical system 11 as N. N stereo image groups acquired by dividing into partial regions are formed.

  FIG. 3 is a diagram for explaining the pixel array and the incident region of the image sensor 20. 3 shows a state where the image sensor 20 is observed from the X-axis direction of FIG. In the image sensor 20 shown in FIG. 3, a region where pixels are arranged is referred to as a pixel region 200. In the pixel region 200, for example, 10 million or more pixels are arranged in a matrix. The circular area in the pixel area 200 is an area where light rays that have passed through one microlens are incident. Accordingly, a plurality of circular regions are formed for each of the plurality of microlenses. These regions are referred to as incident regions 201, 202. For example, 586 × 440 microlenses are arranged. However, such an arrangement number is an example, and a larger or smaller number may be used. In the example shown in FIG. 3, the diameter of one microlens is 13 pixels. However, the number of pixels is not limited to this.

  As described above, the light field camera 1 extracts and reconstructs pixel values of pixels at the same position in the incident area (the same position in the Y-axis direction and the Z-axis direction in the incident area) from all the microlenses. . The reconstructed image group obtained in this way corresponds to an image group in which a subject is observed from different partial areas of the photographing optical system 11. Therefore, these reconstructed image groups have parallax according to the position of the partial region and the distance to the subject. For example, as shown in FIG. 3, a reconstructed image composed of pixel values of pixels of each microlens located at the same position as the pixel 201a in the incident area 201 and each micro located at the same position as the pixel 201b in the incident area 201 It has parallax with the reconstructed image composed of pixel values of the lens pixels.

  FIG. 4 is a block diagram showing a configuration of the light field camera 1. The light field camera 1 illustrated in FIG. 4 includes an imaging unit 10, an image processing unit 30, a work memory 40, a display unit 50, an operation unit 55, a recording unit 60, and a system control unit 70. The imaging unit 10 includes a photographing optical system 11, a microlens array 12, a drive control unit 13, an imaging element 20, and a drive unit 21. Since the photographing optical system 11 and the microlens array 12 have been described with reference to FIG. 2, description thereof is omitted in FIG.

  For example, a Bayer array color filter array is disposed on the imaging surface of the imaging element 20. The image sensor 20 makes the light beam that has passed through the photographing optical system 11 and the microlens array 12 enter. The image sensor 20 photoelectrically converts an incident light beam to generate a pixel signal of each pixel of an analog signal. In the imaging unit 10, an amplifier (not shown) amplifies the pixel signal of each pixel of the analog signal with a predetermined gain factor (amplification factor). Further, an A / D converter (not shown) converts the pixel signal of each pixel of the analog signal amplified by the amplifier into the pixel signal of each pixel of the digital signal. Then, the imaging unit 10 outputs RAW data including pixel signals of each pixel of the digital signal to the image processing unit 30. The light ray information acquired by the light field camera 1 is specified by RAW data output from the imaging unit 10.

  The drive control unit 13 performs aperture stop drive control based on control information transmitted from the system control unit 70 in order to adjust the aperture diameter of the aperture stop included in the photographing optical system 11. The drive unit 21 is a control circuit that controls driving of the image sensor 20 under imaging conditions set based on an instruction signal from the system control unit 70. Examples of imaging conditions include exposure time (shutter speed), frame rate, gain, and the like.

  Here, the exposure time refers to the time from when the photoelectric conversion element of the image sensor 20 starts to accumulate after the start of charge accumulation according to incident light. The frame rate is a value representing the number of frames processed (displayed or recorded) per unit time in a moving image. The unit of the frame rate is represented by fps (Frames Per Second). The gain refers to a gain factor (amplification factor) of an amplifier that amplifies the pixel signal. By changing this gain, the ISO sensitivity can be changed. This ISO sensitivity is a photographic film standard established by ISO and represents how much light the photographic film can record. However, generally, ISO sensitivity is also used when expressing the sensitivity of the image sensor 20. In this case, the ISO sensitivity is a value representing the ability of the image sensor 20 to capture light.

  The image processing unit 30 performs various kinds of image processing on the RAW data including the pixel signal of each pixel, and generates image data in a predetermined file format (for example, JPEG format). The image data generated by the image processing unit 30 includes image data of still images, moving images, and live view images. The live view image is an image displayed on the display unit 50 by sequentially outputting the image data generated by the image processing unit 30 to the display unit 50. The live view image is used for the user to confirm the image of the subject imaged by the imaging unit 10. The live view image is also called a through image or a preview image.

  The image processing unit 30 uses the work memory 40 as a work space to generate image data set at an arbitrary focal length by performing predetermined image processing on RAW data composed of pixel signals of each pixel. Specifically, the image processing unit 30 generates image data set at an arbitrary focal length by performing the following image processing.

  FIG. 5 is a diagram illustrating image reconstruction. As shown in FIG. 5, the image processing unit 30 has pixel values (values indicated by pixel signals) of the pixels 201a, 202a... At the same position with respect to the respective incident areas 201, 202. To extract. The image processing unit 30 rearranges the extracted pixel values to generate reconstructed image data 400a. When the number of pixels corresponding to each of the incident areas 201, 202... Is N, N reconstructed image data are generated.

  The image processing unit 30 translates the N reconstructed image data groups by a predetermined amount in accordance with a predetermined focal length. Then, the image processing unit 30 adds and averages the N reconstructed image data groups after the parallel movement. Thereby, image data set to a predetermined focal length is generated. Since the image of the subject existing at a depth corresponding to the amount of movement is aligned, a sharp contour image is formed. An image of a subject that does not exist at a depth corresponding to the amount of movement is blurred. The image processing described above is an example, and the image processing unit 30 may generate image data set at an arbitrary focal length by executing image processing different from the image processing described above. For example, the image processing unit 30 calculates a light beam acquired by each pixel of the image sensor 20 by performing a predetermined calculation. Then, the image processing unit 30 generates image data set to a predetermined focal length by projecting the calculated light beam onto a virtual image plane set to a predetermined focal length.

  In the present embodiment, the image processing unit 30 has a focal position (focal point) such as image data focused on the foremost subject, image data focused on the farthest subject, and image data focused on a subject at an intermediate position. A plurality of image data having different distances are generated.

  The image processing unit 30 performs various types of image processing on image data set at an arbitrary focus using the work memory 40 as a work space. For example, the image processing unit 30 generates RGB image data by performing color signal processing (tone correction) on a signal obtained by the Bayer array. The image processing unit 30 performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the RGB image data. Further, the image processing unit 30 performs a process of compressing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary. The image processing unit 30 outputs the generated image data to the recording unit 60. In addition, the image processing unit 30 outputs the generated image data to the display unit 50.

  The work memory 40 temporarily stores RAW data, image data, and the like when image processing by the image processing unit 30 is performed. The display unit 50 displays an image captured by the imaging unit 10 and various information. The display unit 50 includes a display screen 51 configured by, for example, a liquid crystal display panel. The image displayed on the display unit 50 includes a still image, a moving image, and a live view image.

  The operation unit 55 includes a release switch (a switch that is pressed when shooting a still image), a moving image switch (a switch that is pressed when shooting a moving image), various operation switches, and the like that are operated by the user. The operation unit 55 outputs a signal corresponding to the operation by the user to the system control unit 70. The recording unit 60 has a card slot into which a storage medium such as a memory card can be mounted. The recording unit 60 stores the image data and various data generated by the image processing unit 30 in a recording medium mounted in the card slot. The recording unit 60 has an internal memory. The recording unit 60 can also record the image data and various data generated by the image processing unit 30 in the internal memory.

  The system control unit 70 controls the overall processing and operation of the light field camera 1. The system control unit 70 has a CPU (Central Processing Unit). The system control unit 70 performs control to output the image data generated by the image processing unit 30 to the display unit 50 and display an image (live view image, still image, moving image) on the display screen 51 of the display unit 50. . Further, the system control unit 70 performs control to read out the image data recorded in the recording unit 60 and output the image data to the display unit 50 and display an image on the display screen 51 of the display unit 50.

  In addition, the system control unit 70 instructs the predetermined imaging conditions in order to perform imaging with predetermined imaging conditions (exposure time, frame rate, gain, etc.) on the imaging surface (pixel area 200) of the imaging device 20. A signal is output to the drive unit 21. The system control unit 70 also instructs the image processing unit 30 to execute image processing with predetermined control parameters (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate). The instruction signal is output to the image processing unit 30. Further, the system control unit 70 performs control to output the RAW data to the recording unit 60 and cause the recording unit 60 to record the RAW data. Further, the system control unit 70 controls the recording unit 60 to record the image data by causing the recording unit 60 to output the image data generated by the image processing unit 30. In addition to the above control, the system control unit 70 also performs control such as adjustment of the aperture diameter of the aperture stop provided in the photographing optical system 11.

  Next, image processing executed by the calculation unit 114 of the calculation processing unit 110 in the image search / generation apparatus 101 will be described. FIG. 6 is a flowchart for explaining the image processing executed by the calculation unit 114 shown in FIG. When the light field camera 1 is connected to the image search / generation device 101, the RAW data recorded in the recording unit 60 of the light field camera 1 is output to the light field camera 1 via the image input unit 112. It is stored in the image data storage area of the storage unit 120.

  In the process shown in FIG. 6, the image generation unit 114a of the calculation unit 114 acquires the RAW data stored in the image data storage area of the storage unit 120 (step S1). Then, the image generation unit 114a generates image data of a predetermined in-focus position (focal length, focal position) based on the acquired RAW data (Step S2). Here, the image generation unit 114a generates image data at a predetermined in-focus position using a method similar to the method in which the image processing unit 30 generates image data based on the RAW data (see FIG. 5).

  Specifically, as described above, the image generation unit 114a extracts the pixel value of the pixel located at the same position with respect to each incident region in the pixel region 200 as described in FIG. Are reconstructed to generate a plurality of reconstructed image data groups. Then, the image generation unit 114a translates the plurality of reconstructed image data groups according to a predetermined focus position, and adds and averages the plurality of reconstructed image data groups after the parallel movement. As a result, image data set at a predetermined in-focus position is generated. In step S2, the image generation unit 114a generates pan-focus image data focused on all subjects. The image data generated in step S2 includes still image data, moving image data, and live view image data.

  FIG. 7 is a diagram illustrating a distance relationship between the light field camera 1 and each of the subjects O1 to O4. In the example shown in FIG. 7, the light field camera 1 is located at the camera position d0. In addition, the person O1 who is the subject is located at the in-focus position d1. In addition, the person O2 as the subject is located at the focusing position d2, the tree O3 as the subject is located at the focusing position d3, and the house O4 as the subject is located at the focusing position d4. In FIG. 7, the direction from the light field camera 1 toward the subjects O1 to O4 (the direction of the arrow) is referred to as the back side direction, and the opposite direction is referred to as the near side direction. The direction on the back side and the direction on the near side are referred to as the depth direction. In step S2, the image generation unit 114a generates, for example, four image data at the in-focus positions d1 to d4.

  Next, the image generation unit 114a performs various processes such as color signal processing (color tone correction), white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the image data at the predetermined in-focus position generated in step S2. Image processing is executed (step S3). The various image processes executed by the image generation unit 114a are the same as the various image processes executed by the image processing unit 30 described above.

  The detecting unit 114b detects one or more subjects from the generated image data (for example, pan focus image data) (step S4). In the present embodiment, the detection unit 114b detects a subject of a human subject using a known face detection function. In addition, for a subject other than a person, the detection unit 114b detects the subject by specifying the boundary of the subject region based on the contrast and color change between the bright and dark portions in the image data. Note that the detection unit 114b may detect a moving subject by comparing a plurality of pieces of image data obtained in time series for a moving subject (moving subject). In addition to the face detection, the detection unit 114b may detect a human body included in the image data as a subject as described in, for example, Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940).

  In addition, the detection unit 114b extracts the feature points of the detected subject to be registered. Then, the detection unit 114b sequentially detects the feature points of the registered subject to be registered and the feature points of a plurality of subjects included in one or a plurality of image data stored in the storage unit 120, and detects them. It is determined whether image data including the same subject as the registered subject is present in the storage unit 120 (step S5). Then, the detection unit 114b adds EXIF information to the image data for each subject based on the determination result.

  FIG. 8 is a diagram illustrating an example of EXIF information added to image data. The EXIF information shown in FIG. 8 is image data (for example, JPEG format image data) generated based on RAW data when the light field camera 1 captures the distance-related subjects O1 to O4 shown in FIG. EXIF information. In the example illustrated in FIG. 8, when image data including the same subject as the registration target subjects O1 and O2 exists in the storage unit 120, the detection unit 114b performs EXIF of image data including the registration target subjects O1 and O2. Information is the same as the subject type data (“person” in FIG. 8) and subject identification data (“X, aa”, “Y, bb” in FIG. 8) of the EXIF information added to the image data in the storage unit 120. Data is added to each of the subjects O1 and O2.

  Here, “X” in FIG. 8 indicates the name of the person O1 (for example, the name or nickname of the person O1), and “aa” indicates the feature point information of the person O1. “Y” indicates the name of the person O2 (for example, the name or nickname of the person O2), and “bb” indicates information on the characteristic points of the person O1.

  In the example shown in FIG. 8, when the same subject as the registration target subjects O3 and O4 does not exist in the storage unit 120, the detection unit 114b uses the EXIF information of the image data including the registration target subjects O3 and O4. New subject type data (“tree” and “house” in FIG. 8) and subject identification data (“cc” and “dd” in FIG. 8) are added for each subject. Here, “cc” in FIG. 8 indicates feature point information of the tree O3. “Dd” indicates information on the feature point of the house O4. In FIG. 8, no name is added to the tree O3 and the house O4, but the analysis unit 114b may add a name to the tree O3 and the house O4 automatically or according to an operation by the user.

  Note that the detection unit 114b also adds shooting scene information for each image data to the EXIF information of the image data including the subject to be registered. The analysis unit 114b determines a photographic scene using a known scene recognition technique, and registers the determined photographic scene as EXIF information in the image data. In the example illustrated in FIG. 8, “portrait” is registered as information on the shooting scene.

  Next, the detection unit 114b detects the distance (focus position, focal length) to the subject to be registered (step S6). For example, the detection unit 114b detects the parallax of each subject in a plurality of reconstructed image data (for example, reconstructed image data including pixel values of pixels at two specific positions in the incident region), and based on the detected parallax To calculate the distance to each subject.

  In the example shown in FIG. 8, the detection unit 114b detects the distance from the light field camera 1 to the person O1 as d1, detects the distance from the light field camera 1 to the person O2 as d2, and detects the distance from the light field camera 1 to the tree. The distance to O3 is detected as d3, and the distance from the light field camera 1 to the house O4 is detected as d4. The detection unit 114b adds distance data indicating the distance to each of the subjects O1 to O4 to the EXIF information of the image data including the subjects O1 to O4 for each of the subjects O1 to O4. Thereafter, the arithmetic unit 114 stores the image data to which the EXIF information is added together with the RAW data in the storage unit 120 (step S7). Note that the RAW data and the image data are associated with each other by the calculation unit 114 and stored in the storage unit 120.

  Next, image search / generation processing by the arithmetic processing unit 110 according to the first embodiment will be described. FIG. 9 is a diagram illustrating a display example of images 601 to 603 stored in the storage unit 120. In the image 601 shown in FIG. 9A, a person O1, a person O2, a tree O3, and a house O4 are shown. Note that the image 601 is an image when the light field camera 1 captures the distance-related subjects O1 to O4 shown in FIG. Therefore, the EXIF information of the image 601 has the contents shown in FIG. In the image 602 shown in FIG. 9B, a person O1, a person O2, and a mountain O5 are shown. As shown in FIG. 9 (2), in the image 602, the person O2 is located on the most front side, the person O1 is located on the back side of the person O2, and the mountain O5 is located on the farthest side. ing. In the image 603 shown in FIG. 9 (3), a tree O3 and a mountain O5 are shown. As shown in FIG. 9 (3), in the image 603, the tree O3 is located on the near side, and the mountain O5 is located on the far side. Note that the images 601 and 602 shown in FIGS. 9A and 9B have the “portrait” shooting scene, and the image 603 shown in FIG. 9C has the “landscape” shooting scene.

  FIG. 10 is a flowchart for explaining image search / generation processing executed by the arithmetic processing unit 110. In the processing shown in FIG. 10, when a control program (for example, application software for performing image search / generation) is started in response to a user's operation of the keyboard 103 or the mouse 104, the display control unit 111 performs image search. The screen is read from the storage unit 120 and displayed on the display screen 500 of the display device 102 (step S11).

  11 to 13 are diagrams illustrating display examples of the image search screen according to the first embodiment. In step S <b> 11, the display control unit 111 displays an image search screen as shown in FIG. 11 (1) on the display screen 500 of the display device 102.

  Next, the display control unit 111 reads out and acquires image data (image data to which EXIF information is added) from the storage unit 120, and the calculation unit 114 is RAW data and image data (image data to which EXIF information is added). ) Is read from the storage unit 120 and acquired (step S12). In the example illustrated in FIG. 9, image data corresponding to the images 601 to 603 is read by the display control unit 111, and RAW data and image data corresponding to the images 601 to 603 are read by the calculation unit 114.

  The display control unit 111 detects a plurality of subjects included in the plurality of image data, and uses the detected image data of the plurality of subjects (for example, subject image data to which subject type data, subject identification data, and distance data are added). cut. Then, the display control unit 111 displays a list of images based on the extracted image data of the plurality of subjects in the first display area 501 of the display screen 500 of the display device 102 (step S13). In the example illustrated in FIG. 11A, the display control unit 111 detects a plurality of subjects O1 to O5 included in the plurality of images 601 to 603, and cuts out image data of the detected subjects O1 to O5. Then, the display control unit 111 displays a list of images based on the extracted image data of the subjects O1 to O5 in the first display area 501. At this time, the display control unit 111 identifies the same subject that exists separately in the plurality of image data based on subject identification data (feature points, names) added to each subject. Then, the display control unit 111 displays only one image in the first display area 501 for the same subject.

  When the user designates the subject to be searched, the user performs a drag-and-drop operation with the mouse 104 on the subject image to search from the subject images listed in the first display area 501. The target subject image is selected, and the selected subject image is moved to the second display area 502. The second display area 502 is a display area for displaying a subject to be searched.

  In response to such an operation of the mouse 104 by the user, the designation unit 114c of the calculation unit 114 identifies the subject moved to the second display area 502 as a subject to be searched. Specifically, the designation unit 114 c recognizes that the subject has been moved to the second display area 502 based on information from the operation control unit 113. The designation unit 114c identifies the subject based on subject identification data added to the image data of the subject moved to the second display area 502. Then, the designation unit 114c designates (sets) subject identification data (feature point, name) of the identified subject as a subject to be searched.

  In addition, the display control unit 111 moves and displays the subject image displayed in the first display area 501 in the second display area 502 in accordance with the operation of the mouse 104 by the user. In the example shown in FIG. 11B, the image of the person O1 and the image of the person O2 are moved from the first display area 501 to the second display area 502. In this case, an image including the images of the person O1 and the person O2 is designated as the image search condition.

  The display control unit 111 determines whether or not a subject to be searched has been designated (that is, whether or not the subject image has been moved to the second display area 502) (step S14). If the display control unit 111 determines that the subject to be searched is not designated, the display control unit 111 ends the process. On the other hand, if the display control unit 111 determines that the subject to be searched has been specified, based on the information from the operation control unit 113, has the detail specification button 503 arranged at the upper right of the image search screen been pressed? Whether or not (clicked with the mouse 104) is determined (step S15).

  If the display control unit 111 determines that the detail designation button 503 has been pressed, the first display area 501 is deleted, the second display area 502 is expanded, and the search target is further displayed, as shown in FIG. Is displayed in the second display area 502 when the image of the subject is viewed from obliquely above (step S16). In the example shown in FIG. 12 (3), images of the person O1 and the person O2 are displayed side by side in the vicinity of the center position in the depth direction on the overhead view 300.

  The user can specify the relative position in the depth direction of a plurality of search target subjects (whether a certain subject is nearer or deeper than the other subjects in the depth direction) as an image search condition. Specifically, the user performs the drag-and-drop operation of the mouse 104 on the subject image to set the position in the depth direction of one or both of the plurality of search target subjects on the overhead view 300. Move. In accordance with such an operation of the mouse 104 by the user, the display control unit 111 changes the display position in the depth direction of the subject to be searched on the overhead view 300 (step S17).

  In the example shown in FIG. 12 (4), the image of the person O1 is moved to a position closer to the near side in the depth direction than the image of the person O2 by the operation of the mouse 104 by the user. As a result, it is specified that the image search condition includes the images of the person O1 and the person O2, and that the image of the person O1 is positioned in front of the image of the person O2. The designation unit 114c also designates (sets) the relative position in the depth direction of a plurality of search target subjects as a search condition.

  When the user searches for an image that matches the image search condition after specifying the image search condition, the user presses the search button 504 arranged in the lower right of the image search screen (clicks with the mouse 104). To do). The search unit 114d determines whether or not the search button 504 has been pressed (whether or not the mouse 104 has been clicked) based on the information from the operation control unit 113 (step S18). When the user wants to return the position in the depth direction of the plurality of search target subjects to the original position, “restore” is arranged at the lower right of the image search screen shown in FIG. A button 505 (a button arranged next to the search button 504) is clicked with the mouse 104.

  If the search unit 114d determines that the search button 504 has been pressed, the search unit 114d performs an image search (step S19). That is, the search unit 114d searches the image data stored in the storage unit 120 for image data that meets the search condition specified by the specifying unit 114c. In addition, the image generation unit 114a generates image data in the range of the in-focus position and the depth of field corresponding to the search condition (step S20). Note that the depth of field refers to the range of the focal length of a subject that appears to be in focus. Specifically, when the search condition includes the person O1 and the person O2, the image generation unit 114a sets the in-focus position to the in-focus position d1 of the person O1, for example, as illustrated in FIG. The position dA is an intermediate position between the in-focus position d2 of the person O2, and the range of the depth of field includes the in-focus position d1 of the person O1 and the in-focus position d2 of the person O2 (from the position d11 to the position d12). Range) is generated. As a result, the subject that the user wants to check can be displayed immediately.

  Then, as shown in FIG. 13 (6), the display control unit 111 displays the image of the image search result by the search unit 114d (the image based on the image data matching the search condition) for displaying the image search result. 3 is displayed in the display area 506 (step S21). In the example shown in FIG. 13 (6), the image 601 shown in FIG. 9 (1) is displayed as a result of the image search. The image 601 includes images of the person O1 and the person O2, and the image of the person O1 is located on the near side of the image of the person O2.

  Thereafter, the display control unit 111 determines whether an image as an image search result has been selected by the operation of the mouse 104 by the user (step S22). If it is determined that an image has been selected, the display control unit 111 displays the selected image on the display screen 500 (step S23).

  In addition, the image generation unit 114a does not perform the process of generating the image data of the in-focus position and the depth of field according to the search condition in step S20, and the display control unit 111 displays the image at the predetermined in-focus position. Thumbnails may be displayed. Then, when an image is selected by the user in step S22, the image generation unit 114a generates image data of the in-focus position and the depth of field according to the search condition, and the display control unit 111 displays the image. An image based on the data may be displayed. The image displayed at the predetermined focus position displayed as a thumbnail may be, for example, an image based on the image data (pan focus image data) acquired in step S12, or the subject (a plurality of subjects specified by the search condition). When a subject is designated, it may be an image focused on one of a plurality of subjects).

  In addition, the image generation unit 114a, as image data of the in-focus position and the depth of field according to the search condition, includes image data in which the positions of all the subjects specified as the search condition are included within the range of the depth of field. Was generated. However, the image generation unit 114a may generate image data in which the position of any one or a plurality of subjects specified as the search condition is included within the range of the depth of field. The image generation unit 114a may generate image data focused on the foremost subject among the plurality of subjects specified as the search conditions.

  The image generation unit 114a may generate a pan-focus image as the image data of the in-focus position and the depth of field according to the search condition. In addition, when the search unit 114d cannot search for image data that meets the search condition, it may display that there is no search result, or may conform to a condition similar to the search condition. The obtained image data may be displayed as an image search result. For example, when a plurality of subjects are designated as search conditions, image data including one of them may be displayed as an image search result.

  As described above, in the first embodiment, the detection unit 114b that detects the subject included in the image data whose focus position can be changed in the depth direction and detects the position of the detected subject in the depth direction; A designation unit 114c that designates accompanying information (for example, EXIF information) related to the subject to be searched, a search unit 114d that searches for a subject corresponding to the accompanying information designated by the designation unit 114c, and a subject that has been searched by the search unit 114d An image generation unit 114a that generates image data at a predetermined in-focus position according to the position in the depth direction. According to such a configuration, it is possible to perform an image search suitable for an image whose focus position can be changed in the depth direction, and image data at the focus position corresponding to the position in the depth direction of the subject to be searched. Can be generated. Thus, the user can immediately confirm the subject to be searched when the search result image is displayed.

  In the first embodiment, the designation unit 114c can designate a position in the depth direction as accompanying information, and the search unit 114d searches for a subject corresponding to the position designated by the designation unit 114c. According to such a configuration, it is possible to search for an image captured by the user paying attention to the subject to be searched. For example, since the main subject that attracts the most attention is often located at the forefront, an image in which the subject to be searched is the main subject can be searched.

  In the first embodiment, the designation unit 114c can designate the relative positions in the depth direction of a plurality of subjects, and the search unit 114d searches for a plurality of subjects corresponding to the relative positions designated by the designation unit 114c. Then, the image generation unit 114a generates image data focused on the position in the depth direction of at least one of the plurality of subjects searched by the detection unit 114d. According to such a configuration, it is possible to perform an image search paying attention to the positional relationship between a plurality of subjects, and to generate image data reflecting the search results.

  In the first embodiment, the designation unit 114c can designate the relative positions in the depth direction of a plurality of subjects, and the search unit 114d searches for a plurality of subjects corresponding to the relative positions designated by the designation unit 114c. Then, the image generation unit 114a generates image data of the depth of field including the positions in the depth direction of the plurality of subjects searched by the detection unit 114d. According to such a configuration, it is possible to perform an image search paying attention to the positional relationship between a plurality of subjects, and to generate image data reflecting the search results.

  In the first embodiment, the detection unit 114b detects a subject included in one or a plurality of image data whose focus position can be changed in the depth direction, and detects the position of the detected subject in the depth direction. A designation unit 114c that designates the position of the target subject in the depth direction and a search unit 114d that retrieves image data including the subject corresponding to the position designated by the designation unit 114c are provided. According to such a configuration, it is possible to perform an image search suitable for an image whose focus position can be changed in the depth direction. In addition, it is possible to search for an image captured by the user paying attention to the subject to be searched.

  In the first embodiment, a plurality of subjects detected by the detection unit 114b are displayed, and a display for changing the display positions of the plurality of subjects according to the relative positions of the plurality of subjects designated by the designation unit 114c. A control unit 111 is provided. According to such a configuration, the relative positions of a plurality of subjects can be clearly indicated to the user, and the user can specify a search condition while confirming the relative positions of the plurality of subjects. As a result, the operability for the user when specifying the search condition is improved.

Second Embodiment
In the first embodiment described above, the search target and the positional relationship between the search target subjects are specified as search conditions. On the other hand, in the second embodiment, a search condition is designated by character input.

  FIG. 14 is a diagram illustrating a display example of an image search screen according to the second embodiment. As shown in FIG. 14, the image search screen of the second embodiment is provided with a character input field 510 for inputting characters as search conditions. In the example shown in FIG. 14, a character “person” is input in the character input field 510.

  When “person” is input in the character input field 510, the search unit 114d searches for image data whose subject type data is “person” in the image data stored in the storage unit 120 in step S19. Then, the display control unit 111 displays the image search result by the search unit 114d on the display screen 500 (step S21). In the example shown in FIG. 14, images 601 and 602 including at least one of a person O1 and a person O2 are displayed.

  In the character input field 510, for example, when “X” that is the name of the person O1 is input, the search unit 114d searches the image data including the person O1.

  As described above, the search unit 114d can perform an image search based on EXIF information (accompanying information) even by inputting characters. Note that the image search of the second embodiment and the image search of the first embodiment described above can be applied in combination.

  FIG. 15 is a diagram illustrating a display example of an image having parallax. In the example shown in FIG. 15, in the image 604, the person O1 is displayed on the left side. A bar O6 is displayed on the right side of the person O1. Further, the person O2 is displayed obliquely behind the bar O6. The tree O3 is displayed on the right side, and the house O4 is displayed near the center.

  As described above, the reconstructed image data group generated by the image generation unit 114a has parallax with respect to the subjects O1 to O4 and O6. The parallax increases as the subject is closer to the light field camera 1 and decreases as the subject is farther from the light field camera 1. Therefore, the parallax for the person O1 is the largest and the parallax for the house O4 is the smallest. In FIG. 15, the parallax in the person O1 and the stick O6 is also shown. The position of the person O1 ′ in certain reconstructed image data differs from the position of the person O1 ″ in other reconstructed image data due to parallax. Also, the position of the bar O6 ′ in certain reconstructed image data and other positions The position of the bar O6 ″ in the reconstructed image data differs depending on the parallax. Although not shown in FIG. 15, the person O2, the tree O3, and the house O4 also have parallax between a plurality of reconstructed image data.

  Based on the image data having such parallax, the image generation unit 114a can delete a subject that is located in front of the subject to be searched and hides the subject to be searched. That is, the image generation unit 114a deletes a subject (for example, violent O6) that hides a search target subject (for example, the person O2), and complements the image data of the deleted subject area with image data having parallax. According to such a configuration, the subject to be searched can be further paid attention.

<Image search / generation system>
In each of the above-described embodiments (the first embodiment and the second embodiment) and the modifications, the system is a stand-alone system. However, the present invention is not limited to such a system. For example, an image display system SYS connected through a communication network such as a client server system will be described. In the following description, components that are the same as or equivalent to those in the first to second embodiments are assigned the same reference numerals, and descriptions thereof are omitted or simplified.

  FIG. 16 is a block diagram showing the configuration of the image composition system SYS. As shown in FIG. 16, the image display system SYS includes a display control unit 111, an image input unit 112, an operation control unit 113, and a communication unit 115 in the arithmetic processing unit 110B of the client terminal 101B. The display control unit 111, the image input unit 112, and the operation control unit 113 correspond to the display control unit 111, the image input unit 112, and the operation control unit 113 illustrated in FIG. Further, the storage unit 120B illustrated in FIG. 16 corresponds to the storage unit 120 illustrated in FIG.

  As illustrated in FIG. 16, the arithmetic processing unit 110 </ b> B of the client terminal 101 </ b> B includes a communication unit 115 that transmits / receives data to / from the server (image search device, image generation device) 106 via the communication network 180. The arithmetic processing unit 160 of the server 106 includes a communication unit 161 that transmits / receives data to / from the client terminal 101 </ b> B via the communication network 180. In addition, the arithmetic unit 162 of the arithmetic processing unit 160 accumulates RAW data and image data transmitted from the client terminal 101B in the storage unit 170 in response to a request from the client terminal 101B. Then, the calculation unit 162 executes image search and image generation control based on the RAW data and the image data, similarly to the calculation unit 114 illustrated in FIG.

  Note that the configuration of the communication unit 115 and the configuration of the communication unit 161 and the calculation unit 162 in the calculation processing unit 160 are performed by a control device such as a CPU executing a process according to a control program stored in the storage units 120B and 170. Realized.

  According to such a configuration, it is not necessary for the client terminal 101B to store a large amount of data, and costs can be reduced, for example, data management can be unified. In addition, the processing load on the client terminal 101B can be reduced.

  In the image display system SYS illustrated in FIG. 16, the display control unit 111 of the client terminal 101B performs display control, but the present invention is not limited to this. For example, the client terminal 101B may include a browser, and display content of an image search screen as shown in FIG. 11 or the like on the display device 102 using the browser.

<Modification>
In the configuration described above, the arithmetic processing units 110 and 160 search for image data including a subject corresponding to the search condition based on the EXIF information added to the plurality of image data stored in the storage units 120 and 170. The image data corresponding to the search condition is generated and the search result image is displayed. However, the present invention is not limited to such a configuration, and the arithmetic processing units 110 and 160 search for image data including a subject corresponding to the search condition based on information stored in a database provided in the storage units 120 and 170. Generation of image data corresponding to the search condition and display of the search result image may be performed.

  In this case, the database includes image identification data (image ID) for identifying each of a plurality of image data, information on a shooting scene added to each image data, and images of one or a plurality of subjects in each image data. Data, subject type data indicating the type of one or more subjects in each image data, subject identification data (feature points, names, etc.) for identifying one or more subjects in each image data, each image The distance data (in-focus position information) of one or more subjects in the data is registered in association with each other.

  The arithmetic processing units 110 and 160 (the image generating unit 114a and the processing unit 162 corresponding to the image generating unit 114a) acquire the RAW data (step S1), and then focus on the acquired RAW data. A plurality of image data having different positions (for example, image data in JPEG format) is generated (steps S2 and S3). The calculation processing units 110 and 160 (the processing unit in the calculation unit 162 corresponding to the detection unit 114b and the detection unit 114b) are used for one or more registration targets based on the plurality of image data generated in steps S2 and S3. A subject is detected (step S4). As described above, since the arithmetic processing units 110 and 160 detect the subject based on the plurality of image data having different in-focus positions, the subject can be detected using the image data in which the subject is not blurred. Detection accuracy is improved.

  The arithmetic processing units 110 and 160 cut out image data of one or more registration target subjects detected in step S4. The arithmetic processing units 110 and 160 extract feature points of one or more registration target subjects. The arithmetic processing units 110 and 160 sequentially collate the extracted feature points of one or a plurality of subjects to be registered with the feature points of a plurality of subjects stored in the database in the storage unit 120, and It is determined whether or not the same subject as the subject is registered in the database.

  When it is determined that the same subject as the subject to be registered is not registered in the database, the arithmetic processing units 110 and 160 store new subject type data and subject identification data for each subject in association with the image identification data. Register with. At this time, the arithmetic processing units 110 and 160 register the extracted image data of the subject to be registered in association with the image identification data for each subject, and also register information on the shooting scene.

  On the other hand, when it is determined that the same subject as the subject to be registered is registered in the database, the arithmetic processing units 110 and 160 associate the subject type data and subject already registered in the database in association with the image identification data. The same data as the identification data is registered in the database for each subject. Also at this time, the arithmetic processing units 110 and 160 register the image data of the subject to be registered that is cut out in association with the image identification data, and also register information on the shooting scene.

  Further, the arithmetic processing units 110 and 160 detect the position in the depth direction (distance to the subject) of one or a plurality of subjects to be registered (step S5). The detection unit 114b registers distance data indicating distances to one or a plurality of registration target subjects in the database for each subject in association with the image identification data. Then, the arithmetic processing units 110 and 160 store the RAW data and the image data in the image data storage area of the storage units 120 and 170 in association with the image identification data (step S7).

  The arithmetic processing units 110 and 160 (the display control unit 111 and the processing unit 160 in the arithmetic unit 160 corresponding to the display control unit 111) display an image search screen (step S11), and RAW data and images from the storage units 120 and 170 are displayed. After reading and acquiring the data (step S12), the image data of a plurality of subjects registered in the database is read and displayed in a list in the first display area 501 (step S13). The arithmetic processing units 110 and 160 (the display control unit 111, the processing unit in the arithmetic unit 160 corresponding to the display control unit 111, the designation unit 114c, and the processing unit in the arithmetic processing unit 160 corresponding to the designation unit 114c) are: Steps S14 to S17 are executed.

  If it is determined that the search button 504 has been pressed (YES in step S18), the arithmetic processing units 110 and 160 collate the information registered in the database in the storage unit 120 with the information specified as the search condition. Then, image data that meets the search condition is searched from the image data stored in the image data storage area in the storage unit 120 (step S19). Specifically, the arithmetic processing units 110 and 160 search the database for data that matches the subject type data specified as the search condition (subject type data, subject identification data, distance data), and specify it as the search condition. The image identification data associated with the data matching the subject type data and the like is selected. Then, the arithmetic processing units 110 and 160 search the image data in the image data storage area corresponding to the image identification data as image data that meets the search condition. Thereafter, the arithmetic processing units 110 and 160 execute the processes of steps S20 to S23.

  According to such a configuration, since the arithmetic processing units 110 and 160 can search for an image and generate an image according to the search condition based on information registered in the database in advance, the image search and the image Can reduce the processing load when generating the image, and also reduce the processing time when searching for the image and generating the image.

  In the above-described modification, when the subject to be searched is determined in advance such as a family (for example, father, mother, son, daughter) or the like, the arithmetic processing units 110 and 160 may subject subjects of a plurality of subjects in advance. Information (subject type data and subject identification data (feature points, names, etc.)) may be registered in a database. In addition, the arithmetic processing units 110 and 160 specify only subjects registered in the database in advance as search subjects (search candidates), and perform image search based only on subject information of subjects registered in the database in advance. May be performed. In addition, the arithmetic processing units 110 and 160 may display only subjects previously registered in the database on the image search screen. In addition, the arithmetic processing units 110 and 160 previously register subject information (subject type data and subject identification data (feature points, names, etc.)) of a plurality of subjects in a database, and acquire subject names from the subject identification data. Then, the name of the subject may be registered in the EXIF information.

  The database is not limited to being provided in the storage unit 170 of the server 106, and may be provided in the storage unit 120B of the client terminal 101B. In this case, the arithmetic processing unit 160 accesses the database in the storage unit 120B of the client terminal 101B and performs processing such as image generation and image search.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various modifications or improvements can be added to the above-described embodiment without departing from the spirit of the present invention. In addition, one or more of the requirements described in the above embodiments may be omitted. Such modifications, improvements, and omitted forms are also included in the technical scope of the present invention. In addition, the configurations of the above-described embodiments and modifications can be applied in appropriate combinations.

  For example, in each of the above-described embodiments, the light field camera 1 is cited as an example of the imaging apparatus. However, the present invention is not limited to this. For example, a camera array in which a plurality of cameras are arranged may be used. Even in such a camera array, a plurality of images having parallax can be acquired.

  In each of the embodiments described above, the calculation unit 114 (and the calculation processing unit 160) changes the focus position and the range of the depth of field using the refocus image captured by the light field camera 1. . However, the calculation unit 114 (and the calculation processing unit 160) uses a plurality of images captured by a digital camera (imaging device) having a focus bracket function, and uses a focus position (focus position) and a range of depth of field. It is possible to change. That is, the “image data whose focus position can be changed in the depth direction” in the first embodiment described above includes a plurality of images with different focus positions that are captured by the digital camera (imaging device) while changing the focus position. Image data generated based on the data is included. Here, the focus bracket refers to taking a plurality of images with different in-focus positions by changing the in-focus position while continuously shooting. The calculation unit 114 (and the calculation processing unit 160) appropriately selects a plurality of image data with different focus positions according to the focus position and depth of field specified by the user, and synthesizes the selected image data. By doing so, the range of the focus position and the depth of field can be changed.

  In each embodiment described above, the detection unit 114b (and the arithmetic processing unit 160) detects a subject from the image data, detects the distance of the detected subject, and adds EXIF information to the image data for each subject. It was. However, the image processing unit 30 of the light field camera 1 may perform these processes, and the calculation unit 114 may acquire the image data to which the EXIF information is added from the light field camera 1 together with the RAW data.

  In each of the above-described embodiments, the image generation unit 114a (and the arithmetic processing unit 160) generates image data in a range of a predetermined focus position and depth of field based on the RAW data. The unit 114a (and the arithmetic processing unit 160) performs a weighted average process for averaging a plurality of pieces of image data after applying a weight according to the distance, thereby obtaining an image in a range of a predetermined focus position and a depth of field. Data may be generated.

  In addition, the image generation unit 114a (and the arithmetic processing unit 160) deletes the subject that hides the subject to be searched, but the display control unit 111 (and the arithmetic processing unit 160) deletes the image from which the subject has been deleted. You may make it display alternately the image which is not. In addition, the display control unit 111 (and the arithmetic processing unit 160) may alternately display a subject to be searched and a subject to hide the search target. In this case, it is possible to pay more attention to the subject to be searched. In addition, the subject to be searched can be easily confirmed.

  Further, the image processing unit 30 and the system control unit 70 illustrated in FIG. 4 may be configured integrally. In this case, the system control unit having one CPU executes the processing based on the control program, thereby taking on the functions of the image processing unit 30 and the system control unit 70.

  In the first embodiment described above, the color filter array is a Bayer array, but an array other than this array may be used.

  Further, the display device 102 has displayed various images including the image search screen in two dimensions, but may display them in three dimensions (3D display). In particular, when the overhead view 300 or the like is displayed three-dimensionally, the positional relationship of the subject is easily recognized.

  Further, as a method for managing the accompanying information (for example, see FIG. 8) of the image, the following method can be considered. (1) As in the case of the first embodiment described above, the arithmetic processing units 110 and 160 add accompanying information to one piece of image data (pan focus image data). (2) This is a method of adding accompanying information (for example, the same accompanying information) to each of the plurality of image data when the arithmetic processing units 110 and 160 generate a plurality of image data having different in-focus positions in step S2. In this case, group identification information for managing a group of a plurality of image data is added to each of the plurality of image data. (3) This is a method in which the arithmetic processing units 110 and 160 add accompanying information to the RAW data. (4) The arithmetic processing units 110 and 160 provide a data file (or data folder) different from the image data or RAW data, add accompanying information to the data file, and associate the data file with the image data or RAW data. is there. In this case, the data file may be a text file.

  Further, when the designation unit 114c (and the arithmetic processing unit 160) designates a plurality of conditions as search conditions, these conditions are AND conditions, but may be OR conditions. The designation unit 114c (and the arithmetic processing unit 160) designates the relative position of a plurality of subjects as a search condition, but the absolute position of one or more subjects (for example, a subject 100 m ahead of the light field camera 1). ) May be specified.

  In step S20, the image generation unit 114a (and the arithmetic processing unit 160) determines that the search condition is the image data of the focus position that matches the search condition when the focus positions of the plurality of subjects corresponding to the search condition are different. Image data focused on each of a plurality of subjects corresponding to may be generated. In this case, the image generation unit 114a (and the arithmetic processing unit 160) may divide the image area into a plurality of subject areas and generate image data focused on the subject for each of the divided areas. . In step S20, when the focus positions of a plurality of subjects corresponding to the search condition are different, the image generation unit 114a (and the arithmetic processing unit 160) performs pan-focus as image data of the focus position that matches the search condition. Image data may be generated. In step S20, when the position of the subject in the depth direction is specified as the search condition, the image generation unit 114a (and the arithmetic processing unit 160) is specified as the search condition for the searched image data. When image data focused on the subject is generated and the position of the subject in the depth direction is not specified as a search condition, pan-focus image data may be generated for the searched image data. In step S20, the image generation unit 114a (and the arithmetic processing unit 160) may generate a pan focus image when there is a subject different from the plurality of subjects corresponding to the search condition in the image data. .

  In step S20, the image generation unit 114a (and the arithmetic processing unit 160) includes in the image data a subject that is different from the plurality of subjects corresponding to the search condition, and the subject of the subject that is not the search target. When the position in the depth direction is within the range of the in-focus position of multiple subjects corresponding to the search condition, the range of depth of field is the range including the positions of multiple subjects corresponding to the search condition, and the search target Image data in which an outside subject is deleted may be generated.

In the third display area 506 for displaying the image search result, a “move to the focal position” button and a “move to the focal position” button are provided as focal position switching buttons. Then, when the user clicks the “move to the focal position” button, the image generation unit 114a (and the arithmetic processing unit 160) changes the focal position of the image data as the image search result to the near side. The image generation unit 114a (and the calculation processing unit 160) may change the focus position of the image data as the image search result to the back side by clicking the “move to the back of the focus position” button. At this time, the switching of the focal position may be performed in units of subjects. In addition, when a plurality of subjects are designated as subjects to be searched, the image generation unit 114a (and the arithmetic processing unit 160) changes the focus position in the image data each time the focus position switching button is clicked by the user. It may be configured to sequentially switch to the search target position.

  Note that “image data whose focus position can be changed in the depth direction” includes RAW data (ray information) and one or a plurality of images generated from the RAW data by the image generation unit 114a (calculation processing unit 160). Data (for example, pan focus image data, a plurality of image data with different in-focus positions) are included.

DESCRIPTION OF SYMBOLS 1 ... Light field camera (imaging device), 101 ... Image search / generation device (image generation device, image search device), 101B ... Client terminal, 102 ... Display device (display unit), 106 ... Server (image generation device, image) Retrieval device), 110, 110B ... arithmetic processing unit, 111 ... display control unit, 112 ... image input unit, 113 ... operation control unit, 114 ... calculation unit, 114a ... image generation unit, 114b ... detection unit, 114c ... designation unit , 114d ... search unit, 115 ... communication unit, 120, 120B ... storage unit, 160 ... calculation processing unit, 161 ... communication unit, 162 ... calculation unit, 170 ... storage unit

Claims (7)

A search unit for searching for the image data capable of generating the image including a specific subject from among the image data capable of generating an image with a changed focal position;
A generating unit configured to generate the image focused on a predetermined position within a predetermined range including the specific subject in the depth direction based on the image data searched by the search unit;
An image generation apparatus comprising:   The image generation apparatus according to claim 1, wherein the predetermined range is a depth of field.   The image generation apparatus according to claim 1, wherein the specific subject is a plurality of subjects.   The image generating apparatus according to claim 3, wherein each of the plurality of subjects is present at each position in the depth direction.   The image generation device according to claim 4, wherein the first subject included in the plurality of subjects is present in the depth direction as compared to a second subject different from the first subject.   The said image generation part produces | generates the said image of the depth of field which has focused on these several subjects based on the said image data searched by the said search part. The image generation apparatus according to one item. A search unit for searching for the image data capable of generating the image including a specific subject from among the image data capable of generating an image with a changed focal position;
A display unit that displays the generated image based on the image data searched by the search unit;
The image search device, wherein the image displayed on the display unit is the image including the specific subject.