JP6735566B2 - Image management apparatus, image reproduction apparatus and control method thereof, program, and storage medium - Google Patents

Description

The present invention relates to a technique for generating a refocus image in which an arbitrary focusing surface is focused from light field data.

In recent years, an imaging device called a light field camera, which can acquire not only brightness information of a light beam emitted from a point in a scene but also light space information (light field data) of the incident direction of the light beam, has been attracting attention.

For example, in Non-Patent Document 1, a microlens array is arranged between a photographing lens and an image pickup device, one microlens is made to correspond to a plurality of pixels of the image pickup device, and a plurality of light beams passing through the microlens are arranged. A method of acquiring each incident direction by a pixel is described.

Then, a normal captured image is generated from the light field data acquired (captured) as described above. Furthermore, if a method called "Light Field Photography" is applied, a refocus image in which an arbitrary virtual image plane (refocus plane) is focused after image capturing can be reconstructed.

Ren. Ng, 7 others, "Light Field Photography with a Hand-Held Plenoptic Camera", Stanford Tech Report CTSR 2005-02.

However, in order to reproduce the light field data as an image, an image reproduction device having a function of decoding (decoding) the light field data encoded by a predetermined encoding method to visualize the image is required. In this case, by acquiring the decoded refocused image from the image supply device such as a server via the network, it is possible to reproduce even the image reproducing device having no light field data decoding function. However, for the refocused image acquired via the network, the user cannot select an image reconstructed on any refocused surface, and thus an image focused on the subject desired by the user is acquired and reproduced. It may not be possible.

The present invention has been made in view of the above problems, and an object thereof is to obtain an image reconstructed by an arbitrary refocus surface even in a device that does not have a decoding function of light field data and focus on a desired subject. It is to realize a technology capable of reproducing a matched image.

To solve the above problems and to achieve the object, an image management apparatus of the present invention includes an acquisition means for acquiring image data including the La site field data, reconstructing the light field data at a plurality of different refocus plane a reconstruction means for generating a plurality of refocus images, generating means for generating a plurality of thumbnail images corresponding to each of the plurality of refocus images, from a first device having a display means via the communication means when receiving an acquisition request of the image corresponding to the light field data, said first transmission means for transmitting the plurality of thumbnail images in the apparatus, possess a plurality of different refocus plane, the light It includes a refocusing surface focused on each subject included in the field data and a refocusing surface focused on infinity .

According to the present invention, even an apparatus having no light field data decoding function can acquire an image reconstructed on an arbitrary refocus surface and reproduce an image in focus on a desired subject. ..

The system block diagram of this embodiment. FIG. 3 is a block diagram showing the configuration of the image recording apparatus of the present embodiment. The block diagram which shows the structure of the image management apparatus of this embodiment. The block diagram which shows the structure of the image reproduction apparatus of this embodiment. 3 is a flowchart showing the operation of the image recording apparatus of this embodiment. 6 is a flowchart showing the operation of the image management apparatus of this embodiment. 6 is a flowchart showing the operation of the image reproduction device of the present embodiment. The figure which illustrates the management information of the content in the image management apparatus of this embodiment. The figure which illustrates the thumbnail screen in the image reproducing apparatus of this embodiment. The figure which illustrates the refocus surface selection screen in the image reproduction apparatus of this embodiment.

Hereinafter, modes for carrying out the present invention will be described in detail. The embodiments described below are examples for realizing the present invention, and should be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions. However, the present invention is not limited to the embodiment. In addition, a part of each embodiment described below may be appropriately combined and configured.

<System Configuration> First, the system configuration of this embodiment will be described with reference to FIG.

In FIG. 1, the system according to the present exemplary embodiment includes an image recording apparatus 100 that generates and records data such as images (moving images and still images) and audio, and an image that manages contents such as images (moving images and still images) and audio. The management device 200 and the image reproduction device 300 that reproduces contents such as images (moving images, still images) and audio are communicably connected via a network 400 such as the Internet.

The image management device 200 receives contents such as images and sounds including light field data which is three-dimensional information (light space information) of the subject space from the image recording device 100 and manages the contents. The image management apparatus 200 transmits the content held by the image management apparatus 200 in response to the image acquisition request received from the image reproduction apparatus 300. The image recording apparatus 100 has a function of transferring content to the image management apparatus 200 via the network 400, a function of acquiring the content held by the image management apparatus 200 via the network 400, and displaying the content.

The image reproduction device 300 is connected to the image management device 200 via the network 400 and receives the content distributed from the image management device 200.

The image recording apparatus 100 captures an image and records a sound, generates content data such as an image and a sound and records the content data in a memory, and manages the content recorded in the memory in a predetermined unit via a network 400. It is possible to transfer to the device 200. The image management device 200 manages contents to be distributed to the image reproduction device 300. The image reproduction device 300 receives the content distributed from the image management device 200, and displays and reproduces the content on the display unit.

The image management apparatus 200 has a file server function, a database function, a print server function, a communication server function (HTTPD: HyperText Transfer Protocol Demon), and the like. The image management apparatus 200 is composed of, for example, a high performance personal computer, and each function is realized as application software executed on an operating system.

<Image Recording Device> The configuration and function of the image recording device 100 of the present embodiment will be described with reference to FIG.

In FIG. 2, the image recording apparatus 100 of the present embodiment is, for example, a light field camera capable of capturing (acquiring) light field data. The lens unit 101 includes a fixed lens group for condensing, a variable power lens group, a diaphragm, and a correction lens group having both a function of correcting an image-forming position moved by movement of the variable power lens group and a function of performing focus adjustment. Equipped with. The subject image light incident from the lens unit 101 is imaged on the image pickup surface of the image pickup element 104 via the microlens array 127. The image pickup device 104 is a photoelectric conversion device such as a CCD or a CMOS that converts light into an electric signal. The A/D conversion unit 105 performs a process of converting an analog image signal output from the image sensor 104 into a digital signal. Hereinafter, the respective units 101 to 105 are collectively referred to as the camera unit 123. Although not shown, a variable power lens group, an actuator such as a diaphragm, a sensor for camera shake correction (for example, an angular velocity sensor), and a correction optical system (shift lens etc.) are mounted.

The camera unit 123 is an image sensor module that captures an image of a subject, and is a light that includes not only the brightness information of the light beam emitted from a point in the scene that is incident via the lens unit 101 but also the information of the incident direction of the light beam. Get field data. In the camera unit 123, the microlens array 127 is arranged on the optical axis between the lens unit 101 and the image sensor 104. The microlens array 127 includes a plurality of microlenses 130.

The camera control unit 118 controls each unit of the camera unit 123 according to the control of the main control unit 110 described later. The camera control unit 118 also sends information obtained from the camera unit 123, such as focus information and camera shake information, to the main control unit 110.

The main control unit 110 includes a microcomputer such as a CPU and MPU, and controls the entire digital camera. The main control unit 110 is used to access a non-volatile memory (ROM) that stores various control programs of the digital camera, a volatile memory (RAM) that serves as a work area, data exchange with other hardware, and access control registers. It has at least an external bus and a timer for measuring time. Each functional block is connected to the bus 103, and functions as a transmission path for passing data under the control of the main control unit 110.

The signal processing unit 109 performs various kinds of signal processing on the image signal converted into digital data by the A/D conversion unit 105.

The encoding unit 102 includes a first encoder 102a that generates light field data and a second encoder 102b that generates standard image data such as a JPEG image under the control of the main control unit 110. The first encoder 102a encodes (compresses and encodes) the image signal output from the signal processing unit 109 by a predetermined encoding method to generate light field data. The second encoder 102b reconstructs the light field data on a predetermined refocus surface and generates standard image data such as JPEG encoded by a standard encoding method such as JPEG.

The recording/reproducing unit 121 has an interface with the encoding unit 102, the memory 106, the recording medium 120, and the decoding unit 108, and controls data transfer under the control of the main control unit 110.

The memory 106 can be used as a work area for each functional block, and can also be used as a buffer for temporarily storing the image data output from the camera unit 123 and encoded by the encoding unit 102 via the signal processing unit 109. To be done.

The recording medium 120 is a hard disk, a memory card, or the like that stores data such as a video (moving image or still image) captured by the camera unit 123 and audio recorded at the time of capturing. The recording/reproducing unit 121 has an interface with the main control unit 110 and the memory 106 connected to the bus 103 and an interface with the recording medium 120, and is for accessing the control register of the recording medium 120 from the main control unit 110 and the like. Conversion of. Upon receiving a recording start instruction from the user, the recording/reproducing unit 121 reads the data buffered in the memory 106 and records it on the recording medium 120. The recording/reproducing unit 121 also has a so-called direct memory access (DMA) function of automatically transferring read or write data by designating the start address and data amount of the memory 106 and the write start address of the recording medium 120. ..

The decoding unit 108 includes a first decoder 108a for decoding (decompressing) the light field data encoded by the encoding unit 102 into reproducible data, and similarly encoded standard image data into reproducible data. And a second decoder 108b for decoding. The first decoder 108a generates refocus image data by reconfiguring and decoding the refocus surface of the light field data so that the predetermined focus surface set by the main control unit 110 is in focus. .. The second decoder 108b sequentially reads standard image data from the address of the memory 106 designated by the main control unit 110, and, for example, ITU-R BT. It is converted into a digital video signal such as 656 (CCIR656) and output. Further, the decoding unit 108 can perform a data size reduction process or the like as necessary when reproducing the image data.

The image output unit 117 is connected to an external device and outputs the digital video signal decoded by the decoding unit 108 to an external device such as an external monitor.

The subject detection unit 124 detects a subject included in the image data output from the camera unit 123. The subject detection unit 124 detects, for example, a face area of a person included in the image data. This face detection process is performed by a known algorithm. For example, in the face detection process, a known feature point extraction process is used to extract feature points such as eye, nose, and mouth end points and face contour points from the image from the data of the through image or the captured image. The face area and the face size of the subject are detected based on the feature points.

The subject recognition unit 125 generates authentication information indicating the characteristics of the subject to be authenticated based on the detection result of the subject detection unit 124. The subject recognition unit 125, for example, generates face authentication information indicating the features of the face based on the result of the face detection process. For example, in the face recognition processing, face authentication data is generated from the positions of the detected feature points of the face, the size of the face part obtained from the feature points, the relative distance between the feature points, and the like.

The depth detector 126 detects distance information to the subject detected by the subject detector 124. The depth detection processing is detected using, for example, a phase difference type focus detection sensor.

An OSD (on-screen display) unit 113 superimposes information such as a menu screen for performing various settings of the camera, a title, and time on the video and outputs it.

The operation unit 112 is a switch member that can be operated by a user and is physically displaced. Like the touch panel 115, the operation unit 112 is controlled by the main control unit 110 based on an input signal corresponding to a user operation. The operation unit 112 is provided with a push button type switch for instructing the start of capturing a still image and the start or end of recording a moving image, a slide type or rotary type switch for changing the operation mode and various parameters, and the like.

When reconstructing the light field data, the focus changing unit 119 changes the setting of the refocusing surface (focusing surface) to the subject or the like designated by the user via the operation unit 112 or the like. The user can generate an image in focus on a different subject by performing an operation of changing the refocus surface on the image reconstructed with the default refocus surface.

The transmission/reception unit 107 transmits the image data including the light field data to the image management device 200.

The main control unit 110 is equipped with software that handles a predetermined file system, and data is written to and read from the recording medium 120 according to this file system. The non-volatile memory 111 is an electrically rewritable EEPROM.

An EVF (Electronic View Finder) 114 uses a display unit 116 such as a liquid crystal panel as a finder (small window through which a subject is viewed) of a camera.

The touch panel 115 is integrally configured with the screen of the display unit 116. The display unit 116 displays thumbnails and virtual buttons, and the touch panel 115 receives user operations on the screen.

The screen control unit 122 determines which virtual button has been pressed based on the output from the touch panel 115, and controls the virtual button output to the display unit 116.

<Image Management Device> The configuration and functions of the image management device 200 of this embodiment will be described with reference to FIG.

In FIG. 3, a transmission/reception unit 201 transmits/receives data between the image recording device 100 and the image reproduction device 300. The light field (lf) data decoding unit 202 is a functional module that receives the light field data from the image recording apparatus 100 and reconstructs and decodes the light field data stored in the data storage unit 207 by changing the refocus surface. The subject detection unit 203 is a functional module that detects a specific subject (person's face or the like) from the decoded refocus image. The subject detection unit 203 changes the refocus surface of the decoded light field data and performs subject detection processing at a predetermined cycle (for example, every 1/30 seconds). The subject distance detection unit 204 is a functional module that detects depth information of a specific subject detected by the subject detection unit 203. The depth information is position information of a refocusing surface (focusing surface) that is most in focus for each of one or more subjects detected by the subject detecting unit 203. The subject distance detection unit 204 detects and stores the position information of the most focused refocus surface of each subject in the light field data by using, for example, the contrast AF method.

The data storage unit 207 is a functional module that stores content such as images and sound including light field data acquired from the image recording device 100 and other external devices. The thumbnail generation unit 206 is a functional module that generates a thumbnail (reference image) of image data including the light field data stored in the data storage unit 207. The image decoding unit 205 is a functional module that decodes image data other than the light field data stored in the data storage unit 207.

The image management apparatus 200 according to the present embodiment includes not only the above-mentioned functional modules but also a CPU, a ROM, a RAM, a storage device such as a hard disk and hardware such as a network interface (not shown), and executes a dedicated software program. As a result, the operation of this embodiment described later is realized.

<Content Management Information> The content management information managed by the image management apparatus 200 of this embodiment will be described with reference to FIG.

In FIG. 8, File Name is the file name of the content managed by the image management apparatus 200. The file name of the image data including the light field data acquired from the image recording device 100 is managed. File Type indicates the type of file. For example, "0" is set for JPEG image data, and "1" is set for light field data. Thumb Name is a file name of a thumbnail generated from image data including light field data managed by the image management apparatus 200. The image management apparatus 200 manages image data including light field data and contents such as thumbnails generated from them using the management information of FIG.

<Image Reproducing Device> The configuration and functions of the image reproducing device 300 of the present embodiment will be described with reference to FIG.

In FIG. 4, a transmission/reception unit 301 is a functional module for transmitting/receiving data to/from the image management apparatus 200. The image reproduction device 300 transmits a thumbnail acquisition request and an image acquisition request described later to the image management device 200 via the transmission/reception unit 301, and receives image data including thumbnails and light field data from the image management device 200. The display unit 302 is a liquid crystal panel or the like that displays thumbnails and other image data acquired from the image management apparatus 200. The operation unit 303 is a switch member or a touch panel that is physically displaceable by the user, and the user can, for example, perform an operation for setting various functions of the power switch or the display unit 302 or a display via the operation unit 303. It is possible to perform an operation of selecting an image to be reproduced from the thumbnails displayed in the list in the section 302. The command generation unit 305 is a functional module that generates an image acquisition request command selected by the user via the operation unit 303. The image acquisition request command generated by the command generation unit 305 is transmitted to the image management apparatus 200 via the transmission/reception unit 301.

The image reproducing apparatus 300 according to the present embodiment includes not only the functional modules described above, but also a storage device such as a CPU, a ROM, a RAM, a hard disk and the like, and hardware such as a network interface, and executes a dedicated software program. As a result, the operation of this embodiment described later is realized.

<Photographing Operation of Image Recording Apparatus> Next, with reference to a flowchart of FIG. 5, a photographing operation of light field data by the image recording apparatus 100 of the present embodiment will be described.

The process of FIG. 5 is realized by loading the program stored in the non-volatile memory 111 into the memory 106 and executing it by the main control unit 110. Further, the processing of FIG. 5 is started when the operation mode of the image recording apparatus 100 is set to the shooting mode.

In step S501, the main control unit 110 sets the operation mode of the image recording apparatus 100 to the shooting mode, and executes the shooting preparation operation.

In S502, the main control unit 110 waits for a shooting start instruction to be input from the operation unit 112 or the touch panel 115, and proceeds to S503 when the shooting start instruction is detected.

In step S<b>503, the main control unit 110 controls the camera unit 123 by the camera control unit 118 to capture an image, and the signal processing unit 109 and the encoding unit 102 generate light field data.

In S<b>504, the main control unit 110 determines whether a shooting end instruction is input from the operation unit 112 or the touch panel 115, continues shooting an image in S<b>503 until the shooting end instruction is detected, and the shooting end instruction is detected. And it progresses to S505.

In S505, the main control unit 110 transmits the light field data generated in S503 to the image management apparatus 200 via the transmission/reception unit 107.

<Photographing Operation of Image Management Apparatus> Next, the operation of the image management apparatus 200 of the present embodiment will be described with reference to the flowchart of FIG.

Note that the processing of FIG. 6 is realized by loading the program stored in the ROM of the image management apparatus 200 into the memory and executing it by the CPU.

In step S<b>601, the CPU of the image management apparatus 200 (hereinafter, the image management apparatus 200) determines whether image data has been received from the image recording apparatus 100 via the transmission/reception unit 201. As a result of the determination, if the image data is not received, the process proceeds to S605, and if the image data is received, the image data is stored in the data storage unit 207 and the process proceeds to S602.

In step S602, the image management apparatus 200 decodes the image data acquired in step S601, and the subject detection unit 203 detects a subject included in the image.

In S603, in the image management apparatus 200, the subject distance detection unit 204 detects depth information for one or more subjects detected by the subject detection unit 203 and stores the depth information.

In S604, the image management apparatus 200 updates the management information managed by the image management apparatus 200. The thumbnail generation unit 206 also generates a thumbnail of the image data received from the image recording device 100. In this case, the thumbnail corresponding to the image other than the light field data is decoded by the image decoding unit 205, and the thumbnail corresponding to the light field data is based on the refocus plane preset by the lf data decoding unit 202. And decode the light field data. In this case, the refocusing surface may be a default focusing surface set by the image recording apparatus 100, or a focusing surface arbitrarily set by the image management apparatus 200.

In step S605, the image management apparatus 200 waits for a thumbnail acquisition request from the image reproduction apparatus 300 via the transmission/reception unit 201, and when the thumbnail acquisition request is received, the processing proceeds to step S606.

In S606, the image management apparatus 200 causes the thumbnail generation unit 206 to generate a thumbnail of the image data stored in the data storage unit 207 in response to the thumbnail acquisition request. Further, the image management device 200 transmits the thumbnail generated by the thumbnail generation unit 206 to the image reproduction device 300 via the transmission/reception unit 201. FIG. 9 illustrates thumbnails generated by the thumbnail generation unit 206 and transmitted to the image reproduction device 300. In FIG. 9, six types of thumbnails are displayed on the display unit 302, and an extension indicating a file format and a file name (file name of original image data) are additionally displayed on each thumbnail. Thumbnail 001. generated from image data other than light field data. jpg and 003. ~005. The jpg is an image decoded by the image decoding unit 205 and reduced to the thumbnail data size by the thumbnail generating unit 206.
Thumbnail 002. generated from light field data lf and 006. lf is an image that is reconstructed and decoded on the default refocus plane by the lf data decoding unit 202, and reduced to the thumbnail data size by the thumbnail generation unit 206. Although the file format is displayed as text in FIG. 9, it may be displayed as an identifiable index such as an icon.

In step S607, the image management apparatus 200 waits for an image acquisition request from the image reproduction apparatus 300 for the thumbnail transmitted in step S606. If the image management apparatus 200 receives the image acquisition request, the processing proceeds to step S608. As in the example of FIG. 9, thumbnails 006. If “lf” is selected, the command generation unit 305 selects the thumbnail 006.lf. If the light field data 006. An lf acquisition request command is generated and transmitted to the image management apparatus 200 via the transmission/reception unit 301.

In step S608, the image management apparatus 200 determines whether the image data specified by the image acquisition request is light field data. If the image data is not light field data, the process advances to step S609, and if it is light field data, step S610. Proceed to. The determination as to whether or not it is light field data is made by referring to the management information (FIG. 8) managed by the image management apparatus 200. For example, the thumbnail 006. Since File type has 1 as File Type, it is determined to be light field data.

In step S<b>609, when the image acquisition request specifies image data other than the light field data, the image management apparatus 200 transmits the image data stored in the data storage unit 207 to the image reproduction apparatus 300 via the transmission/reception unit 201. Send to. In addition, when the light field data is specified by the image acquisition request (S610 to S612), the image management apparatus 200 causes the lf data decoding unit 202 to specify the thumbnail for which the light field data is specified by the image acquisition request, as described later. The refocused image data reconstructed and decoded on the refocus plane corresponding to is generated, and is transmitted to the image reproduction device 300 via the transmission/reception unit 201.

In step S610, the image management apparatus 200 causes the lf data decoding unit 202, the subject detection unit 203, and the subject distance detection unit 204 to perform refocus plane selection information generation processing. The lf data decoding unit 202 generates refocus image data by reconstructing and decoding the light field data by changing the refocus surface. The subject detection unit 203 detects a specific subject from refocus image data reconstructed on different refocus planes. The subject distance detection unit 204 detects depth information of a specific subject detected by the subject detection unit 203. Refocused image data is generated by reconstructing and decoding the thus detected object on the refocusing surface that is most in focus. When a plurality of subjects are detected, the subject distance detection unit 204 detects the most focused refocus surface for each subject, and refocus image data reconstructed and decoded by the refocus surface is generated. To be done. Further, refocus image data focused on infinity such as a mountain is also generated. In this way, the refocus image data at the farthest position on the refocus surface focused on infinity is obtained from the light field data of the refocus surface having the largest focal length. In this way, refocus plane selection information in focus for each subject is generated from refocus image data having different refocus planes. The refocus plane selection information is an image obtained by reducing the refocus image data of different refocus planes to the thumbnail data size by the thumbnail generation unit 206.

FIG. 10 illustrates the refocus plane selection information generated from the refocus image data. Of the refocus surface selection information (thumbnail in this example) of FIG. 10, the thumbnail 006_101. jpg corresponds to the image reconstructed on the default refocus plane, and is the thumbnail 006.jpg of FIG. corresponds to lf. In addition, the thumbnail 006_102. jpg corresponds to the image reconstructed on the refocusing surface that is in focus on the subject A of the light field data. Thumbnail 006_102. In the refocus image corresponding to jpg, the subject detection unit 203 detects the subject A from the refocus image, and the subject distance detection unit 204 detects position information of the refocus surface (focus surface) in focus on the subject A. Then, it is generated by controlling the lf data decoding unit 202 to perform reconstruction based on the detected refocus surface. Further, the thumbnail generation unit 206 reduces the generated refocus image to a thumbnail size to generate refocus surface selection information. Thumbnail 006_103. jpg corresponds to the image reconstructed on the refocusing surface in focus on the subject B of the light field data. Thumbnail 006_104. jpg corresponds to the image reconstructed on the refocus surface in focus on the subject C of the light field data. Thumbnail 006_104. Since jpg corresponds to an image in focus on a landscape such as a mountain, it is generated by setting the refocus surface to the farthest position and reducing the generated refocus image.

In S611, the image management apparatus 200 transmits the refocus plane selection information generated in S610 to the image reproduction apparatus 300.

In step S612, the image management apparatus 200 waits for a refocus image acquisition request from the image reproduction apparatus 300 with respect to the refocus surface selection information transmitted in step S611, and when the refocus image acquisition request is received, the processing proceeds to step S609. move on. As in the example of FIG. 10, the thumbnail 006_103. When jpg is selected, the thumbnail 006_103. A refocus image acquisition request command (focusing on the subject B of the light field data 006.lf) corresponding to jpg is generated and transmitted to the image management apparatus 200 via the transmission/reception unit 301.

<Operation of Image Reproducing Device> Next, the operation of the image reproducing device 300 of the present embodiment will be described with reference to the flowchart of FIG.

The process of FIG. 7 is realized by loading a program stored in the ROM of the image reproducing device 300 into the memory and executing the program by the CPU. Further, the processing of FIG. 7 is started when the image reproducing apparatus 300 is powered on and the operation mode of the image reproducing apparatus 300 is set to the reproducing mode.

In step S<b>701, the CPU of the image reproduction device 300 (hereinafter, image reproduction device 300) sets the operation mode of the image reproduction device 300 to the reproduction mode.

In step S<b>702, the image reproduction device 300 transmits a thumbnail acquisition request for the image data stored in the data storage unit 207 of the image management device 200, and receives thumbnails as shown in FIG. 9. The thumbnails acquired from the image management apparatus 200 are displayed in a selectable list on the display unit 302 as shown in FIG.

In step S703, the image reproduction apparatus 300 selects an image that the user wants to reproduce via the operation unit 303 from the thumbnails acquired from the image management apparatus 200 in step S702. In the example of FIG. 9, the user selects the thumbnail 006. It shows a state in which lf is selected.

In step S<b>704, the image reproduction device 300 transmits an image acquisition request corresponding to the thumbnail selected by the user via the operation unit 303 in step S<b>703 to the image management device 200 via the transmission/reception unit 301. The image acquisition request is generated by the command generation unit 305 and transmitted from the transmission/reception unit 301 to the image management device 200.

In S705, the image reproduction device 300 determines whether or not the refocus plane selection information is received from the image management device 200 as a response to the image acquisition request transmitted in S704. As a result of the determination, when the refocus plane selection information is not received, the process proceeds to S708, the image data other than the light field data is received from the image management apparatus 200, displayed on the display unit 302, and the process ends. When the refocus plane selection information is received, the refocus plane selection information is displayed in a selectable list on the display unit 302 as shown in FIG. 10, and the process proceeds to S706.

In step S<b>706, the image reproduction device 300 selects the image that the user wants to reproduce via the operation unit 303 from the refocus surface selection information acquired from the image management device 200 in step S<b>705. In the example of FIG. 10, the user selects thumbnail 006_103. It shows a state in which jpg is selected.

In step S<b>707, the image reproduction device 300 transmits the refocus image acquisition request corresponding to the refocus surface selection information selected in step S<b>706 to the image management device 200.

In step S<b>708, the image reproducing apparatus 300 receives the refocus image data reconstructed and decoded by the refocus surface corresponding to the refocus surface selection information selected in step S<b>706 from the image management apparatus 200 and displays it on the display unit 302. Then, the process ends.

As described above, according to the present embodiment, the image management apparatus 200 generates the refocus plane selection information reconstructed by the arbitrary refocus plane from the light field data and provides the refocus plane selection information to the image playback apparatus 300. At 300, decoded refocus image data reconstructed on a desired refocus plane can be received. As a result, even if the image reproducing apparatus 300 does not have a light field data decoding function, the user can obtain an image reconstructed with an arbitrary refocus plane, and reproduce an image in focus on a desired subject. You will be able to.

The image recording apparatus 100 according to the present embodiment is not limited to a digital camera, and may be, for example, an information processing apparatus such as a personal computer or a tablet that can be connected to a network, a mobile electronic device such as a mobile phone or a smartphone such as a mobile phone, a surveillance camera. It can be applied to medical equipment. Further, the image management apparatus 200 of the present embodiment is not limited to a server connectable to a wide area communication network such as the Internet, but may be a server computer connected via a wired or wireless LAN (Local Area Network), for example. Is also applicable. Further, the image reproducing device 300 of the present embodiment is not limited to a television receiver, a recording device, and the like, and may be, for example, an information processing device such as a personal computer or a tablet that can be connected to a network, a mobile phone or a mobile phone such as a smartphone. It is also applicable to electronic devices and the like.

[Other Embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100... Image recording device, 200... Image management device, 300... Image reproducing device, 400... Network

Claims (14)

Obtaining means for obtaining the image data including the La site field data,
Reconstructing means for generating a plurality of refocused images by reconstructing the light field data with different refocusing surfaces,
Generating means for generating a plurality of thumbnail images corresponding to each of the plurality of refocus images;
When receiving an acquisition request of the image corresponding to the light field data from a first device having a display means via the communication means, transmitting means for transmitting the plurality of thumbnail images on the first device, have a,
The image management device according to claim 1, wherein the plurality of different refocusing surfaces include a refocusing surface focused on each subject included in the light field data and a refocusing surface focused on infinity . Subject detecting means for detecting a subject from the light field data,
A subject distance detecting means for detecting distance information of the subject detected by the subject detecting means,
The image management apparatus according to claim 1, wherein the reconstruction unit determines the plurality of refocus surfaces based on the distance information. Image data including the light field data, the type of the image data file name and file, image management apparatus according to claim 1 or 2, characterized in that it comprises a file name of the thumbnail image corresponding to the light field data .. The image management apparatus according to claim 2 , wherein the subject detection unit detects a specific subject based on a refocus image reconstructed on a different refocus plane by the reconstruction unit. The image management apparatus according to claim 4 , wherein the specific subject is a face of a person. The subject distance detecting means detects, for one or a plurality of subjects detected by the subject detecting means, position information of a refocusing surface which is most in focus for each subject,
The image management apparatus according to claim 2 , wherein the reconstructing unit generates a refocus image reconstructed on a refocusing surface that is most in focus for each subject detected by the subject detecting unit. The communication means is capable of communicating with the first device via a network,
7. The image management device according to claim 1, wherein the first device is an imaging device capable of imaging the light field data. Before Symbol image management apparatus according to any one of claims 1 to 7 the first device is characterized in that a device which does not include a function that forms reconfigure the light field data at predetermined refocus plane. First transmitting means for transmitting a request for acquiring a plurality of thumbnail images corresponding to a plurality of image data held by the image management device;
A first selection unit capable of selecting an image from the plurality of thumbnail images acquired from the image management device;
Second transmitting means for transmitting an image data acquisition request corresponding to the selected thumbnail image;
When the selected thumbnail image corresponds to light field data , a plurality of thumbnails respectively corresponding to a plurality of refocus images obtained by reconstructing the light field data from the image management device with a plurality of different refocus surfaces. First receiving means for receiving an image ,
Second selection means capable of selecting an image from a plurality of thumbnail images acquired from the image management device;
Second receiving means for receiving a refocus image corresponding to the thumbnail image selected by the second selecting means from the image management device ,
The different refocusing planes, a refocus plane in focus in each subject included in the light field data, images reproduced you; and a infinity focused on the refocus plane apparatus. The image reproducing device according to claim 9 , wherein the image reproducing device does not have a decoding function for reproducing the light field data. Obtaining means for obtaining the image data including the La site field data,
A control method for an image management apparatus, comprising: a reconstructing unit that reconstructs the light field data with different refocusing surfaces to generate a plurality of refocused images.
A generation step of generating a plurality of thumbnail images corresponding to each of the plurality of refocus images,
When receiving an acquisition request of the image corresponding to the light field data from a first device having a display means via the communication means, a transmission step of transmitting the plurality of thumbnail images on the first device, have a,
The control method, wherein the plurality of different refocusing surfaces include a refocusing surface focused on each subject included in the light field data and a refocusing surface focused on infinity . A method of controlling an image management device, comprising:
A first transmitting step of transmitting a request for acquiring a plurality of thumbnail images corresponding to a plurality of image data held by the image management device;
A first selecting step of selecting an image from the plurality of thumbnail images acquired from the image management device;
A second transmitting step of transmitting an image data acquisition request corresponding to the selected thumbnail image;
When the selected thumbnail image corresponds to light field data, a plurality of thumbnails respectively corresponding to a plurality of refocus images obtained by reconstructing the light field data from the image management device with a plurality of different refocus surfaces. A first receiving step of receiving an image,
A second selecting step of selecting an image from a plurality of thumbnail images acquired from the image management device;
A second receiving step of receiving a refocus image corresponding to the thumbnail image selected by the second selecting step from the image management device,
The control method, wherein the plurality of different refocusing surfaces include a refocusing surface focused on each subject included in the light field data and a refocusing surface focused on infinity . Program for causing a computer to function as each unit of the image reproducing apparatus according to image management apparatus or claim 9 or 10 according to any one of claims 1 to 8. The computer readable computer that stores a program to function as each means of the image reproducing apparatus according to image management apparatus or claim 9 or 10 according to any one of claims 1 to 8 Storage medium.