JP2013255172A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten a time from a user operation to creation and display of image data in a camera capable of refocusing as focus adjustment after imaging.SOLUTION: The order of a target to be focused is determined on the basis of distance information indicating the depth of an image indicated by multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions, and in accordance with the determined order, composite image data focused on the target is sequentially generated from the multi-viewpoint image data.

Description

  The present invention relates to an image processing apparatus and an image processing method for processing image data based on information acquired from a plurality of viewpoint positions.

  Currently, as a method for confirming a captured image after photographing in an image capturing apparatus, a method for displaying and confirming a captured image in a so-called reproduction mode is known.

  In recent years, an imaging apparatus using a technique called “Light Field Photography” has been proposed (Non-Patent Document 1, Patent Document 1). This imaging device is composed of an imaging lens, a microlens array, an imaging device, and an image processing unit, and the captured image data obtained from the imaging device is used in addition to the light intensity distribution on the light receiving surface. It also includes information on the direction of travel. In the image processing unit, observation images from a plurality of viewpoints and directions can be reconstructed.

  One of the reconstruction is focus adjustment processing after shooting (hereinafter referred to as refocusing) (for example, Patent Document 2), and an imaging device capable of refocusing after shooting (hereinafter referred to as a light field camera) has been developed. Has been.

JP 2010-183316 A JP 2011-22796 A

R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan: "Light Field Photography with a Hand-Held Plenoptic Camera", Stanford Tech Report CTSR 2005-02 (2005)

  However, in the current image processing apparatus, when generating refocus image data at the time of image display, a refocus image is generated according to a user operation. In the generation of refocus image data, in the method of Patent Document 2, it is necessary to generate intermediate viewpoint image data by estimation processing, and this generation requires enormous calculation costs. For this reason, since it takes time to display an image after a user operation, a waiting time has occurred.

  The image processing apparatus according to the present invention includes: a determination unit that determines the order of objects to be focused based on distance information indicating the depth of an image indicated by multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions; And generating means for sequentially generating composite image data focused on the object from the multi-viewpoint image data in accordance with the order determined by the determining means.

  According to the present invention, since refocus image data can be generated in advance using the distribution of distances in an image, there is an effect of improving user convenience with a simple configuration.

It is a block diagram which shows an example of the hardware constitutions of the whole multiview camera which concerns on Example 1 of this invention. It is a flowchart which shows an example of the refocus process at the time of the reproduction | regeneration which concerns on Example 1 of this invention. It is a flowchart which shows an example of the parameter selection process which concerns on Example 1 of this invention. It is a flowchart which shows an example of the prediction method of the refocus calculation area | region which concerns on Example 1 of this invention. It is a figure which shows an example of the refocus log | history regarding the past person which concerns on Example 1 of this invention. It is a figure which shows an example of the display of the refocus image which concerns on Example 1 of this invention. It is a figure which shows an example of the temporary image format which preserve | saves the refocus image data based on 1st Example of this invention. It is a flowchart which shows an example of operation | movement of the reproduction | regeneration refocus mode which concerns on Example 2 of this invention. It is a figure which shows an example of user I / F which changes the reproduction | regeneration refocus mode which concerns on Example 2 of this invention. It is a figure which shows an example of the operation | movement at the time of the reproduction | regeneration refocus mode change which concerns on Example 2 of this invention. It is a figure which shows an example of operation | movement in the reproduction | regeneration refocus mode at the time of the image switching which concerns on Example 2 of this invention. It is a figure which shows an example of the system configuration | structure which concerns on Example 1 of this invention. It is a figure which shows an example of a structure of the cloud server which concerns on Example 1 of this invention. It is a figure which shows an example of the image data format which preserve | saves the multi-viewpoint image which concerns on Example 1 of this invention. 1 is a block diagram showing an example of a hardware configuration according to Embodiment 1 of the present invention. It is a figure which shows an example of the pixel number distribution (histogram) for every distance which concerns on Example 3 of this invention. It is a figure which shows an example of the coefficient which multiplies pixel number distribution for every distance which concerns on Example 3 of this invention. It is a figure which shows an example of pixel number distribution (histogram) for every distance after the coefficient multiplication which concerns on Example 3 of this invention. It is a flowchart which shows an example of the operation | movement which concerns on the refocus image generation order determination which concerns on Example 3 of this invention. It is a flowchart which shows an example of the operation | movement which concerns on the data generation for the refocus image generation order determination which concerns on Example 4 of this invention. It is a figure which shows an example of the pixel number distribution (histogram) for every distance which concerns on Example 4 of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating an example of a hardware configuration of the image processing apparatus according to the first embodiment of the present invention. Hereinafter, FIG. 1 will be described in detail. The multi-lens camera imaging unit 101 is an aggregate of a plurality of mutually independent cameras having an optical system and an image sensor that are independent from each other. The multi-view camera imaging unit 101 includes an image sensor control unit and the like, and outputs a set of output data from a plurality of image sensors as multi-viewpoint image data.

  The RAM 102 is a memory used for temporary storage of multi-viewpoint image data captured by the multi-eye camera imaging unit 101, temporary storage of generated refocus image data, and other temporary data during calculation. The flash ROM 107 is a non-volatile memory, and accumulates and stores history relating to images (types and positions) and operations selected by the user in the past. The external memory 109 is an external memory such as an SD card. The external memory 109 is a non-volatile memory, and image data is saved even when the power is turned off. In the external memory 109, multi-viewpoint image data captured by the multi-view camera imaging unit 101 is stored as an image file in accordance with, for example, the image data format shown in FIG. Details of the image data format will be described later. The external memory 109 is also used as an area for temporarily storing image data being processed as a temporary image file.

  The memory control unit 110 includes a so-called bus system and a control unit of a memory and a device connected to the bus system, and controls reading and writing of data with respect to each memory of the RAM 102, the Flash ROM 107, and the external memory 109, for example.

  A user I / F (interface) 106 is an I / F (interface) for operating a device, selecting an image or region that a user desires to display refocused, or selecting a mode related to displaying a captured image. ). Specifically, it corresponds to a user i / F such as a touch panel installed on the display 105, a shutter button, and an operation dial.

  The overall control unit (CPU) 108 performs overall control of the device and calculation control such as selecting the most frequent area from the history information stored in the flash ROM 107 and instructing the refocus calculation unit 103.

  The refocus calculation unit 103 generates refocus image data from the multi-viewpoint image data in accordance with an instruction from the overall control unit 108. Here, an outline of generating refocus image data from multi-viewpoint image data will be described. In light field photography, the direction and intensity of light passing through each position (light field, hereinafter referred to as “LF”) is calculated from multiple viewpoint image data for a plurality of positions in space. Then, using the obtained LF information, an image when passing through the virtual optical system and forming an image on the virtual sensor is calculated. By appropriately setting such virtual optical systems and sensors, it is possible to generate refocus image data that is composite image data focused on a predetermined target. Note that the refocus processing is not the main focus of the present embodiment, and a method other than the method described above may be used. The generated refocus image data is stored in the RAM 102, the external memory 109, or the like. Details will be described later.

  The graphic processor 104 has a function of displaying the refocus image data generated by the refocus calculation unit 103 on the display 105. The network I / F 111 connects to the outside of the network and transfers data to / from an external device on the network.

The image analysis unit 112 detects the area of the subject in the image data and assigns an identification code to each area. The image analysis unit 112 stores the detected subject area information and identification code in the RAM 102. The subject in the present embodiment includes not only a person or animal in the foreground but also a landscape portion included in the background. The detection of the subject can be performed, for example, by utilizing some prominent features (two eyes, mouth, nose, etc.) and the inherent geometric positional relationship between the features. Alternatively, the detection of the subject can be performed by using a symmetrical feature of the face, a feature of the face color, template matching, a neural network, or the like. For the detection of the background, for example, the face of a person or animal in the image is detected by the method described above, and the position and size are calculated. Based on this calculation result, the image can be divided into a subject region including a person in the image and a background region other than that. For example, assuming that the subject area is a rectangle, the format of the area information is indicated by upper left and lower right coordinates. In addition, the subject identification target is three types, that is, a person (or face), an animal, and a landscape. However, the subject identification target is not limited to this, and finer classification is also possible. Note that the subject detection and identification method is not the main focus of the present embodiment, and methods other than those described above may be used. The subject detection and identification function in the image analysis unit 112 is performed immediately after the image is captured, and the result is stored in a nonvolatile memory such as the flash ROM 107 or the external memory 109, for example. Therefore, in the description of the present embodiment, it is assumed that the area information of the subject and the identification code are already stored.
The multi-viewpoint image data captured by the multiview camera imaging unit 101 is recorded in the external memory 109 as an image file according to the image data format of FIG.
FIG. 14 is a diagram illustrating an image data format used when multi-viewpoint image data is stored as an image file in the present embodiment. The image file shown in FIG. 14 includes information 1401 indicating image width, information 1402 indicating image height, refocus position information 1403, multi-view image header information 1410, and multi-view image data 1404.

  The refocus position information 1403 includes a position information presence flag 1405. The position information presence flag 1405 is information indicating whether or not refocus position information exists. This position information presence flag is, for example, 0 (false) in the initial state (there is no history). In the case of 1 (true), the values of the subsequent position information 1406 to 1409 can be validated. That is, when the position information presence flag is true, information indicating the refocused position is included. Top_left_x 1406 is information indicating the upper left X coordinate of the refocus area. Top_left_y 1407 is information indicating the upper left Y coordinate of the refocus area. Bottom_right_x 1408 is information indicating the lower right X coordinate of the refocus area. Bottom_right_y 1409 is information indicating the lower right Y coordinate of the refocus area.

  The position information presence flag 1405 is sufficient as 1-bit information indicating whether the image file is added or not, but is not limited to 1 bit. A multivalue indicating the format of the refocus position information format may be taken. For example, it is possible to adopt the following configuration. When the position information presence flag 1405 is 0, there is no position information. When the position information presence flag 1405 is 1, the position information is expressed as rectangular information as described above. When the position information presence flag 1405 is 2, the position information is expressed by using the center coordinates and radius of a circle indicating the focus target as position information.

  The multi-viewpoint image header information 1410 includes information necessary for performing the refocus processing and auxiliary imaging information. For example, the camera model name, lens configuration, imaging date, shutter speed, and exposure program. Further, the multi-view image header information 1410 includes position information of each image data in the multi-view image data used for the refocus processing. In the present embodiment, a configuration example in which the position information 1406 to 1409 exists regardless of whether the position information presence flag is true or false is shown. However, the present invention is not limited to this, and when the position information presence flag is false, the position information 1406 to 1409 entries themselves may not be included in the image data. Further, the refocus position information can be configured to indicate an arbitrary shape, and is not limited to the rectangular area described in the present embodiment.

  Note that the multi-viewpoint image data 1404 may be RAW image data obtained from an image sensor, or after development subjected to development processing such as demosaicing processing, white balance processing, gamma processing, and noise reduction processing. It may be image data. Further, in this embodiment, the example using the multi-view camera imaging unit 101 has been described. However, the multi-viewpoint image data may be data from a plurality of viewpoint positions, for example, using image data obtained using a microlens array. May be.

  FIG. 2 is a flowchart from the transition to the playback mode to the display of the first refocus image. The reproduction mode is an operation mode in which control for displaying an image file stored in the external memory 109 on the display 105 is performed. The flow in FIG. 2 is executed by the overall control unit (CPU) 108 controlling the refocus calculation unit 103, the graphic processor 104, and the like.

  A program for the overall control unit 108 to execute the control shown in FIG. 2 is stored in, for example, the Flash ROM 107 or the like. Further, the refocusing program described above is also stored in the same manner. The image file in the flow shown in FIG. 2 is based on the assumption that the multi-viewpoint image data captured by the multi-view camera imaging unit 101 is already recorded in the external memory 109 together with the header information as shown in FIG. Yes.

In step S201, the overall control unit 108 shifts the operation mode to the reproduction mode. The following three patterns can be considered as the case of moving to the processing in step S201.
1. 1. It is being activated in a mode other than the playback mode and then switched to the playback mode. 2. Playback mode already at startup When another image is specified in playback mode

  In step S <b> 202, the overall control unit 108 acquires an image file from the external memory 109. In step S203, the overall control unit 108 acquires the playback refocus mode and history information, and the process proceeds to S204. The reproduction refocus mode is a mode in which, for example, a human mode, a landscape mode, an animal mode, or the like is selected when displaying image data that has been subjected to refocus calculation (hereinafter referred to as refocus image data). Details of the playback refocus mode will be described in the second embodiment. The history information is information obtained by accumulating the types and positions of image data selected by the user in the past and the history regarding operations. Details of the history information will be described later. In step S204, the header information of the image file read by the overall control unit 108 is analyzed. The header information corresponds to 1401 to 1403 in FIG. In this embodiment, the refocus position information 1403 is particularly analyzed.

  In step S205, the overall control unit 108 determines whether there is a recommended parameter in the header information analyzed in step S204. The recommended parameter is a parameter indicating an image recommended for refocusing an image file, a display method, or the like. If the recommended parameter exists, the process proceeds to step S206. If the recommended parameter does not exist, the process proceeds to step S207. In the present embodiment, the position information presence flag 1405 shown in FIG. 14 is treated as a 1-bit flag, and when the flag value is 1, the overall control unit 108 determines that a recommended parameter exists and proceeds to step S206. Transition. If the value of the flag is 0, the overall control unit 108 determines that there is no recommended parameter, and proceeds to step S207.

  In step S206, the overall control unit 108 controls the refocus calculation unit 103 using the recommended parameters obtained by analyzing the refocus position information 1403 of the image file read in step S202, and the refocus image data is obtained. Generate.

  Specifically, the refocus calculation unit 103 uses, from the multi-viewpoint image data 1404 included in the reproduction file, a position where the average of the position information 1406 and 1408 is the X coordinate and the average of the position information 1407 and 1409 is the Y coordinate. Generate image data in focus. The refocus image data generated by the refocus calculation unit 103 is displayed on the display 105 via the graphic processor 104. The generated refocus image data is stored in the external memory 109 or the RAM 102 as a temporary image file in accordance with the temporary image file format shown in FIG. FIG. 7 is a diagram illustrating an example of a temporary image file format of an image file including refocus image data generated by the refocus calculation unit 103. The image file according to the temporary image file format includes refocus position information 702 and refocus image data 705. The refocus position information 702 includes information indicating a position where the focus is achieved when the refocus processing of the refocus image data 705 is performed. The position information 706 to 709 can be the same information as the position information 1406 to 1409 in FIG. The temporary image file shown in FIG. 7 is generated for each refocus image data, for example.

  If the process proceeds to step S206, the recommended parameter exists. Therefore, the temporary image file in which the information included in the recommended parameter is included in the refocus position information indicated by reference numeral 702 in FIG. 7 is stored. Note that the storage location of the temporary image file is not limited to the external memory 109 and the RAM 102, and may be stored in an arbitrary storage device on the cloud via the network I / F 111.

  Next, a process when there is no recommended parameter in the header information in step S205 will be described. The process in step S207 is a process when there is no recommended parameter in the image file read from the memory in step S202, that is, when the display is the first image file in the playback mode. In this embodiment, refocus image data is generated based on the history information and stored as a temporary image file. Also, history information is updated. The process of generating refocus image data based on the history information and the process of updating the history information will be described with reference to FIG.

  In step S208, the graphic processor 104 displays the refocused image generated in step S207 on the display 105. FIG. 6 is a diagram illustrating an example when the refocus image data generated in step S206 or step S207 is displayed. FIG. 6A shows an example of an image focused on a landscape (tree).

  In step S209, the graphic processor 104 additionally displays a parameter selection UI screen on the display 105. An example in which a parameter selection UI screen is added is shown in FIG. In the example of FIG. 6B, it is assumed that the user designates a point to be focused on the touch panel. That is, the graphic processor 104 displays a UI screen for allowing the user to input a position (parameter) to be focused. The parameter may represent two-dimensional position information in the image, or may represent a spatial distance (depth) in the multi-viewpoint image.

  In step S210, the overall control unit 108 updates the refocus position information 1403 of the image file read in step S202. That is, when the recommended parameter does not exist in the header information, the refocus position information 1403 is updated with the position information of the area for which the refocus position is determined in step S207. When the recommended parameter exists in the header information, the refocus position information 1403 of the image file read in step S202 does not need to be updated. The above is the operation when the playback mode is activated, that is, the processing related to the refocus image displayed first when a certain image file is read.

  Next, the operation after displaying the first refocus image data (display of the second and subsequent refocus image data, the end of the operation, etc.) will be described with reference to FIG. That is, the processing after displaying the refocus image that is initially displayed when the image file described in the flow of FIG. 2 is read will be described with reference to FIG. The entire flow shown in FIG. 3 is controlled by the overall control unit 108 as in the flow of FIG.

When the parameter selection UI shown in FIG. 6B is displayed in step S209 of FIG. 2, the user can select a parameter using the displayed UI. In step S301, the overall control unit 108 proceeds to step S304 when the user selects a parameter using the UI. In this embodiment, the user selects a parameter by designating one point on the screen. In step S304, the parameter selected by the user in step S301 is acquired. In step S304, the overall control unit 108 determines whether or not there is refocus image data including the parameter designated (selected) by the user in step S301 from the temporary image file group stored in the format of FIG. Determine whether. That is, it is determined whether or not there is a temporary image file including refocus position information 702 including parameters specified by the user. If there is a temporary image file including refocus image data that matches the parameter specified by the user in step S304, the overall control unit 108 advances the process to step S309. In step S309, information (position, display history information) of the temporary image including the refocus image is updated, and the process proceeds to step S306. Details will be described later.
In step S306, the graphic processor 104 displays a temporary image including refocus image data that conforms to the parameter specified by the user on the predetermined display 105.

  On the other hand, if there is no temporary image file including refocus image data matching the parameter specified by the user in step S304, the overall control unit 108 moves the process to step S305. In step S305, based on the parameter specified by the user in step S301, the refocus calculation unit 103 generates refocus image data focused on the area indicated by the parameter. In step S305, the generated refocus image data is stored as a temporary image file shown in FIG.

  When the process of step S305 is completed, the overall control unit 108 moves the process to step S309.

  In step S309, in step S305, the user updates information (position and display history information) related to the refocus image generated according to the parameter specified in step S301, and the process proceeds to step S306. Details will be described later.

In step S306, for example, the refocus image data generated or selected according to the parameter specified in step S301 is read from the RAM 102 or the like and displayed on the display 105.
Here, the information update in step S309 will be described in detail.

As described above, the image analysis unit 112 detects the area of the subject in the captured multi-viewpoint image data, and an identification code is assigned to each area. The area information and the identification code are stored in the FlashROM 107 or the external It is stored in a nonvolatile memory such as the memory 109. Here, the stored area information is compared with the specified parameter, and if the specified parameter is included in the area information, the area information is updated as position information. That is, the area information classified as the subject and the position of the subject is specified is updated as the refocus position information 702 of the temporary image file. On the other hand, the stored area information is compared with the designated parameter, and if the designated parameter is not included in the area information, the designated parameter is updated as the refocus position information 702 of the temporary image file. .
Next, history information will be described with reference to FIG. In FIG. 5, the identification code is an identification code identified by the image analysis unit 112. The frequency refers to the frequency at which the user instructs the display of the focused image for each area indicated by the identification code (person, landscape, etc.) in the playback mode. This frequency may be a frequency for each image file, or may be a frequency specified by a user in the image processing apparatus regardless of the image file. FIG. 5A shows an example in which 501 is an identification code = scenery, 502 is an identification code = person, and 503 is an identification code = animal. Here, the frequency ranking for each identification code is as follows.
1st place: Landscape (501)
2nd place: People (502)
3rd place: Animals (503)

FIG. 5B shows an example in which the identification code is further subdivided for the identification code = person. The distinction between the daughter, son, and father can be determined by, for example, calculating the feature vector of each face image and comparing it with the feature vector of the region determined to be a person. Here, the frequency ranking for each identification code is as follows.
1st place: User's daughter (505)
2nd place: User's son (506)
3rd place: User's father (507)
4th: Unregistered person (508)
In the present embodiment, the history information indicates frequency information of each identification code.

  Next, update of history information will be described. As described above, when the designated parameter is included in the area information, the frequency information of the identification code associated with the area is incremented. On the other hand, when the designated parameter is not included in the area information, the useful information is only the coordinate information, and the history information is not updated.

  Next, a process when the user does not select a parameter in step S301 will be described. In a state where the already displayed image is continuously displayed, the process proceeds to step S302.

In step S302, for all the region information extracted by the image analysis unit 112, the overall control unit 108 determines whether refocus image data in which each region information is focused is generated. Specifically, it is determined whether or not a temporary image file having refocus position information 702 including each area information exists for all area information.
If it is determined in step S302 that no refocus image has been generated for all region information, the process proceeds to S303. On the other hand, if it is determined in step S302 that the generation of the refocus image has been completed for all the area information, the process proceeds to S307.

  If it is determined that no refocus image has been generated for all region information, refocus image data is generated based on the history information in step S303. Details will be described later. Next, after the generated refocus image is displayed in S306, the process proceeds to step S307.

In S307, the overall control unit 108 determines whether to end or continue the playback mode. The playback mode end conditions in S307 are, for example, the following three conditions.
1. 1. When switching to a mode other than the playback mode 2. When the power is turned off. When another image is designated in the playback mode, if it is determined in step S307 that the playback mode has ended, the process proceeds to step S308, and various processes necessary to end the playback mode are executed, and the playback mode ends. On the other hand, if it is determined in step S307 that the playback mode is to be continued, the process returns to step S301, and the operations from step S301 are repeated.

  Note that parameter selection is performed by the user pointing to one point on the screen, but it can also be specified as a closed region. For example, the presence / absence of history information can be determined by comparing the image including the point specified by the user and the image included in the specified closed region with the history of the flash ROM (history storage unit) 107.

  Next, details of the processing in S207 of FIG. 2 and S303 of FIG. 3 will be described using the flowchart of FIG. The process shown in the flowchart of FIG. 4 is executed by the overall control unit (CPU) 108 controlling the refocus calculation unit 103, the graphic processor 104, and the like.

As described above, the image analysis unit 112 detects the area of the subject in the captured multi-viewpoint image data, and an identification code is assigned to each area, and the area information and the identification code are stored in the RAM 102 or externally. It is stored in a nonvolatile memory such as the memory 109. The history information is frequency information shown in FIG. That is, in the past history (frequency), it is assumed that a landscape image is ranked first in frequency, a person image ranked second, and an animal image ranked third. Further, it is assumed that the person's image has the first frequency as the daughter, the second as the son, the third as the father, and the fourth as an unregistered person.
In step S401, the overall control unit 108 identifies an area where refocus image data has not been generated. First, the overall control unit 108 acquires all the pairs of identification codes and region information of the image analysis result regarding the target image data (that is, the image data indicated by the multi-viewpoint image data of the image file read in S202). In addition, position information of refocus image data included in all temporary image files is acquired. Next, the refocus position information 702 included in the temporary image file is compared with the area information from the acquired information. Then, the area information is excluded from the area information, and the area information for which no corresponding position information exists is extracted. This area information corresponds to an area where no refocus image data is generated.

  Next, in step S402, the overall control unit 108 selects, from among the identification codes associated with the area information extracted in step S401, the one that matches the identification code with the highest frequency of history information. If the frequency of the identification code corresponding to the landscape is the highest as a result of selection, the process proceeds to step S403. In step S403, it is determined whether the area corresponding to the landscape image is in the area information extracted in S401. If there is a landscape image, the process proceeds to step S404.

In step S404, area information whose identification code is landscape is extracted, and when there is only one area, that area is determined as an area for generating next refocus image data. Further, when there are a plurality of regions having identification codes corresponding to the landscape, for example, the order can be determined by using the X coordinate and Y coordinate at the upper left of the region information. Specifically, the refocus calculation order can be determined in ascending order of the X coordinate value. If there are a plurality of the same X coordinates, the refocus calculation order can be determined in ascending order of the Y coordinates. Here, the region information with the highest rank is determined as the region for generating the next refocus image.
Next, the process proceeds to step S405. In step S405, the refocus calculation unit 103 generates refocus image data in which the area information of the rank is focused based on the rank determined in step S404. Specifically, image data in which a focus is set to a position where the average of the upper left and lower right coordinates is taken as the X coordinate and the Y coordinate is generated from the multi-viewpoint image data.

  Next, the process proceeds to step S406. In step S406, along with the refocus image data generated in step S405, a temporary image file having the focused area information as the refocus position information is stored in the temporary image file format shown in FIG.

  Next, the process proceeds to step S407. In step S407, the history information is updated. Specifically, the frequency corresponding to the selected identification code is incremented. Then, the process ends.

  Next, returning to step S402, if the frequency of the identification code corresponding to the person is the highest as a result of selection, the process proceeds to step S409. In step S409, it is determined whether there is a human image. If there is no human image, the process returns to step S402. If there is a human image, the process proceeds to step S410. In step S410, it is determined whether there is a registrant identification code as shown in FIG. If there is a registrant identification code, the process proceeds to step S411. In step S411, it is determined whether or not there is a registrant whose existence has been confirmed in step S410 in the history information. If there is a registrant in the history information, the process proceeds to step S412. In step S412, the order of refocus image generation is determined according to the frequency of history information. In FIG. 5B, since the first rank is the daughter, the second rank is the son, and the third rank is the father, for example, if there is a daughter in the identification code, the daughter is the highest rank.

  If it is determined in step S411 that there is no registrant in the history information, the process proceeds to step S413. In step S410, if there is no registrant identification code, the process proceeds to step S414. Note that steps S413 and S414 perform the same processing as step S404.

  Next, returning to step S402, if the frequency of the identification code corresponding to the animal is the highest as a result of selection, the process proceeds to step S415. If it is determined in step S415 that there is an animal image, the process proceeds to step S416. In addition, about the process of step S416, the process similar to S404 can be performed.

  With the configuration as described above, it is possible to perform generation of refocus image data before instructing the user by using the past history. That is, when an image file including multi-viewpoint image data is first read and the recommended parameter is not included in the header information of the file, refocus image data is generated without user instruction using history information. be able to. Further, by using the history information, it is possible to generate refocus image data according to the frequency of the subject that the user has focused on in the past. By generating refocus image data in advance without user instruction, when the user specifies the desired focus area, the user can display the desired refocus image without waiting for the refocus calculation time. There is a possible effect.

  In addition, it is now possible to predict the next refocus image while displaying the refocus image and perform the refocus calculation, so that the use efficiency of calculation resources is improved and there is no hardware upgrade. The same effect as an improvement in processing speed can be expected.

  Furthermore, each time the refocus calculation is repeated, the user's preference can be determined more clearly, so that the prediction accuracy of the refocus area is improved every time it is used, and the switching speed of the refocus image can be expected to be increased.

  In this embodiment, parameter information for specifying a focus position last specified by the user is embedded in an image file including multi-viewpoint image data. As a result, when the image file is read and displayed next time, the image that was last displayed by the user can be displayed without resetting the parameters. As a result, it is possible to save the user from setting parameters and to improve the operability of the apparatus incorporating the present technology.

  In this embodiment, for the sake of simplicity, the example of FIG. 4 has been described with three types of options (people, scenery, and animals) as an example, but identifiable items may be added to the options. . Further, in the present embodiment, the image of a person has been described as three registrants and unregistered persons such as a daughter, son, father, and unregistered person, but the number of registrants may be four or more. Absent. Although the registered person is also a family, the present invention is not limited to this.

  Further, in this embodiment, an example in which refocus image data is sequentially generated based on history information when a recommended parameter does not exist in the header information of the image file has been described. However, even when the recommended parameter exists in the header information of the image file, the refocus image data can be generated based on the history information as in the process of step S209 in FIG. 2 (that is, the process shown in FIG. 3). Please note that.

  Example 2 describes a process for changing the playback refocus mode. The reproduction refocus mode is a reproduction mode in which a subject to be refocused is set, such as a landscape mode and a person mode.

  A configuration for realizing the function of changing the playback refocus mode will be described with reference to the hardware block diagram of FIG. The change of the reproduction refocus mode is instructed via the user I / F unit 106.

  The overall control unit 108 notifies the refocus calculation unit 103 of the mode for the image to be reproduced at the time of transition to the reproduction mode. Similarly, when the user I / F unit 106 switches the image, the overall control unit 108 notifies the refocus calculation unit 103 of the mode for the image to be reproduced. Further, since the target to be recognized as a subject changes according to the playback refocus mode, when the playback refocus mode is changed, the subject recognition information as the target is notified to the image analysis unit 112.

  In response to the notification from the overall control unit 108, the refocus calculation unit 103 reads multi-viewpoint image data from the external memory 109 and generates pan focus image data. Further, based on the analysis result of the image analysis unit 112, a refocus calculation corresponding to the reproduction refocus mode is performed, and refocus image data corresponding to the reproduction refocus mode is generated.

  The image analysis unit 112 analyzes the presence / absence of a recognition target in the reproduction refocus mode designated by the user from the pan focus image data generated by the refocus calculation unit 103. When there is a recognition target in the reproduction refocus mode, the area information is notified to the overall control unit 108.

  The graphic processor 104 displays the pan focus image or the refocus image generated by the refocus calculation unit 103 on the display 105 according to the analysis result.

Hereinafter, the operation flow of the present embodiment will be described in detail.
FIG. 8 shows an operation flowchart in the change of the reproduction refocus mode. In step S801, when there is a user instruction to shift to the playback mode in the user I / F 106 operation, the mode shifts to the playback mode and proceeds to step S802. In step S802, an image file for reproduction is read and the process proceeds to step S803. In step S803, the refocus calculation unit 103 generates pan-focus image data from the image file acquired in step S802, and proceeds to step S804.

  In step S804, the current user-specified playback refocus mode is acquired and stored, and the process proceeds to step S805. Once the playback refocus mode is set, its contents are saved. When the display is switched to an image of another imaging scene, another image is displayed in the stored reproduction refocus mode. In step S805, the image analysis unit 112 performs recognition processing for the recognition target in the reproduction refocus mode designated by the user I / F 106, and the process proceeds to step S806.

  In step S806, if the designated recognition target exists as a result of the recognition process, the process proceeds to step S807. If the designated recognition target does not exist, the process proceeds to step S809.

  In step S807, refocus image data suitable for the reproduction refocus mode is generated based on the analysis result, and the process proceeds to step S808. In step S808, the generated refocus image data is displayed on the display, and the process proceeds to step S810.

  If it is determined in step S806 that the recognition target does not exist in the image, in step S809, the pan focus image is displayed on the display, and the process proceeds to step S810.

  If the playback refocus mode is changed by a user operation in step S810, the process proceeds to step S804, the playback refocus mode setting is changed, and refocus image data corresponding to the new playback refocus mode is generated and displayed. To do. If the playback refocus mode is not changed, there is no need to change the display on the display 105, and the process proceeds to step S811 while keeping the image as it is.

  In step S811, if there is an instruction to change the display to another image due to a user operation, the process proceeds to step S802, and the image file of the other image is read. Thereafter, the refocus image data corresponding to the reproduction refocus mode stored in step S804 is generated and displayed on the display 105. If there is no instruction to change to another image, it is not necessary to change the display 105, and the process ends.

  FIG. 9 shows an example of a user I / F for changing the playback refocus mode. The graphic processor 104 displays the pop-up menu shown in FIG. 9A on the display 105, and each mode such as “person mode” and “landscape mode” is selected by the user. As a result, the overall control unit 108 can switch the playback refocus mode. FIG. 9B shows a dial installed in the camera. When the user turns this playback dial, the overall control unit 108 can switch the playback refocus mode. For example, when the dial is set to “Pan focus mode”, a pan focus image is displayed, and when the dial is set to “People mode”, a refocused image is displayed to a person. In the present embodiment, the pop-up display of the combo box in FIG. 9A and the camera dial in FIG. 9B are shown as examples. However, the user I / F for changing the playback refocus mode is not limited to these. An I / F for displaying a virtual dial on the display and selecting the playback refocus mode may be used, or an I / F such as providing a slider to select the playback refocus mode may be used. Further, the mode may be selected by recognizing the voice from the user.

  FIG. 10 is a diagram showing an example of the refocusing operation by changing the playback refocusing mode in step S810. First, FIG. 10A shows an example in which a pan focus image is displayed in step S809, for example, and all objects are in focus.

  FIG. 10B shows a case where the playback refocus mode is changed when the pan focus image of FIG. 10A is displayed on the display, and a combo box is displayed. In addition, the user selects “person mode” as the playback refocus mode.

  FIG. 10C shows an image after the refocus processing when the reproduction refocus mode is set to “person mode” in FIG. 10B, and the refocused image is displayed only for the person.

  FIG. 10D shows an example when the “landscape mode” is selected, and an image refocused only on a landscape such as a mountain is displayed. For example, when the dial setting shown in FIG. 9B is set to “landscape mode”, this image is displayed by default when the playback mode is entered.

  FIG. 10E shows an example when the “animal mode” is selected, and a refocused image is displayed on the lower right animal.

  FIG. 10F shows an example when “person + landscape mode” is selected, and a refocused image is displayed for both the person and the landscape.

  FIG. 11 shows a display example in the case of switching to another image by a user operation during the playback mode in step S811. First, in FIG. 11A, the playback refocus mode is set to “person mode”, and an image refocused by a person is displayed. When the image data to be displayed on the display is switched during reproduction of the image, the reproduction refocus mode is stored in step S804, and therefore the image data displayed after the switching is also refocused in the “human mode”. Is displayed. FIG. 11B shows a refocus image after switching, and the image is displayed after being refocused by a person even after switching. Unless the user intentionally changes the playback refocus mode, the playback refocus mode is inherited even if the image is switched.

  The description of the second embodiment is based on the hardware configuration shown in FIG. 1, but this embodiment applies the hardware configuration shown in FIG. 1 to a system configuration including the network shown in FIG. 12, as shown in FIG. You can also That is, the image processing of multi-viewpoint image data captured by the multi-view camera 1201 can be realized by components such as the cloud server 1204 and the display terminals 1205 to 1207, for example. FIG. 15 shows a block diagram of the display terminals 1205 to 1207. In the block diagram shown in FIG. 15, processing equivalent to that described in FIG. 1 can be performed except that multi-viewpoint image data is acquired via the network I / F. FIG. 13 shows an example of a detailed hardware configuration block diagram of the cloud server 1204. In the cloud server 1204, a function corresponding to the image analysis unit 112 in FIG. 1 may be performed by the overall control unit 1307, or from the multi-view camera 1201 and the display terminals 1205-1207 via the network communication unit 1301. The analysis result may be received.

  Example 3 describes a method of using a depth map created after imaging in generating refocus image data. The depth map is data representing distance information in which the distance from the imaging surface to the subject is compiled into map information for each pixel. The depth map can be calculated from, for example, arrangement information of a plurality of imaging units in the multi-view camera imaging unit and a plurality of images obtained from the plurality of imaging units. For example, the distance to each subject can be calculated by performing triangulation using the position of the feature point between two images, the position of the corresponding camera, and the angle of view of the corresponding camera. Note that the arrangement information of the imaging unit can be acquired from, for example, multi-view image header information 1410 stored in the data format illustrated in FIG. The depth map is calculated under the control of the overall control unit 108. The depth map can be calculated by various methods in addition to the above example.

  In this embodiment, an example in which the generation order of refocus image data is determined using a depth map will be described. The difference from the first embodiment is step S207 in FIG. 2 and step S303 in FIG. 3, and the other main operation flow is the same as in FIG. 2 and FIG. The hardware configuration block diagram in the present embodiment is also the same as FIG. In this embodiment, since the generation order of the refocus image data is determined based on the depth map, the processing based on the history information described in the first embodiment can be omitted.

  First, when the overall control unit 108 creates a depth map after capturing multi-viewpoint image data or reading an image file, the depth map is converted into a format indicating the number of pixels for each distance. A diagram showing the format after conversion is shown in FIG. This figure shows a pixel number distribution (histogram) for each distance in the screen, and a portion with a large number of pixels means that the ratio of the subject existing at the distance to the screen is high. In the example of FIG. 16, the top four peaks a, b, c, and d are shown, and the number of pixels is in the order of c → a → d → b.

  Next, in the data in the format shown in FIG. 16, the overall control unit 108 multiplies each pixel by a coefficient corresponding to the distance from the screen center (weighting process). The coefficient is the highest at the center of the screen and decreases with the distance from the center. FIG. 17 is a diagram showing the properties of the coefficients.

  FIG. 18 shows the result of multiplying the pixel number distribution data (histogram) for each distance shown in FIG. 16 by the coefficient shown in FIG. By this multiplication, for example, when the background is a wall with a constant distance, it is possible to prevent the background region distribution from increasing and the generation order from increasing.

  In the example of FIG. 18, the ranks of the top three peaks are a → c → b. The generation order of the refocus image data is determined using the multiplication result. The above-described processing and depth map creation processing are performed when the overall control unit 108 in FIG. 1 reads out and executes programs stored in the memory RAM 102, the Flash ROM 107, the external memory 109, and the like.

  Hereinafter, a method for determining the refocus image data generation order in the present embodiment will be described in detail.

  FIG. 19 shows a determination operation flow. This operation is also performed when the overall control unit 108 in FIG. 1 reads and executes a program stored in the RAM 102, the Flash ROM 107, the external memory 109, or the like.

  In step S1901, the overall control unit 108 reads the multiplication result data. In step S1902, the overall control unit 108 detects a peak from the data read in step S1901. For example, a distance having the number of pixels equal to or greater than a certain threshold can be detected as a peak. In addition to the above, various methods can be used for the peak detection method in the histogram. In step S1903, the overall control unit 108 sorts the detected peaks in order of the number of pixels. In step S1904, the overall control unit 108 searches for peaks in the sort order, and excludes lower rank peaks than those existing in a predetermined adjacent range with respect to a certain rank peak. Thereby, it is possible to prevent the peaks from being concentrated at a very short distance.

  In the example of FIG. 18, as already described, the ranking of the top three peaks is a → c → b. Therefore, the refocus image data having the focal lengths A, C, and B, which are distances corresponding to the respective peaks, are in the first to third generation order. According to the refocus image data generation order determined above, in step S207 of FIG. 2, refocus image data with the generation order of 1st is generated. In step S303 in FIG. 3, refocus image data is generated in descending order of generation order.

  With the configuration as described above, it is possible to generate a refocus image from the distribution for each distance occupied in the screen without depending on the past operation history or the like and without requiring a recognition function in the screen. Therefore, there is an effect of improving user convenience with a simple configuration.

  It should be noted that the refocus image data generation order determination method used in the present embodiment can be implemented in combination with other determination methods such as the first embodiment and the second embodiment. Further, in this embodiment, the multi-camera imaging unit that acquires a plurality of images and the display that displays the generated image are described in the configuration provided in one device, but the configuration in which these are provided in an external device. It may be. That is, this embodiment can be applied to a system configuration including the network shown in FIG. 12 in place of the hardware configuration shown in FIG. 1 as shown in FIG. That is, the image processing of multi-viewpoint image data captured by the multi-view camera 1201 can be realized by components such as the cloud server 1204 and the display terminals 1205 to 1207, for example. FIG. 15 shows a block diagram of the display terminals 1205 to 1207. In the block diagram shown in FIG. 15, processing equivalent to that described in FIG. 1 can be performed except that multi-viewpoint image data is acquired via the network I / F. FIG. 13 shows an example of a detailed hardware configuration block diagram of the cloud server 1204. In the cloud server 1204, a function corresponding to the image analysis unit 112 in FIG. 1 may be performed by the overall control unit 1307, or from the multi-view camera 1201 and the display terminals 1205-1207 via the network communication unit 1301. The analysis result may be received.

  In the third embodiment, as an example of a method for determining the generation order of refocus image data, an example in which calculation is performed using only depth map information has been described. In the fourth embodiment, an example in which the refocus image generation order is determined by applying the subject region information output as the analysis result of the image analysis unit 112 to the depth map information will be described.

  That is, in this embodiment, it is assumed that the analysis result of the image analysis unit 112 has already been output, and the identification code and area coordinate information are stored in any of the RAM 102, the Flash ROM 107, and the external memory 109. It is assumed that the depth map is also stored in any of the above memories.

  Hereinafter, according to the operation flow shown in FIG. 20, data generation for determining the image generation order in the present embodiment, which is a difference from the third embodiment, will be described. This operation is performed when the overall control unit 108 in FIG. 1 reads and executes a program stored in the RAM 102, the Flash ROM 107, the external memory 109, or the like.

  First, in step S2001, the overall control unit 108 reads an image analysis result from the RAM 102. In step S2002, the overall control unit 108 determines the number of extracted subjects from the read result, and sets the number of subjects. In step S2003, the overall control unit 108 calculates the center of gravity position from the area information for each subject. In step S2004, the overall control unit 108 sets an initial value zero to the subject number x.

  In step S2005, the overall control unit 108 reads the coordinate information of the subject number x = 0, refers to the depth map for the coordinates where the subject of the subject number x = 0 exists, and accumulates the number of pixels for each distance. At this time, referring to the gravity center position of each subject calculated in step S2003, multiplication is performed by multiplying by a coefficient corresponding to the distance from the screen center position to the gravity center position. The coefficient can be a coefficient that is highest at the center of the screen as in FIG. 17 in the third embodiment, and decreases according to the distance from the center.

  In step S2006, the overall control unit 108 determines whether or not the integration of all the pixels has been completed for a certain subject. If the integration has not been completed, the process returns to step S2005 to continue the integration. If completed, the process proceeds to step S2007. In step S2007, the overall control unit 108 determines whether or not integration has been completed for all subjects. If completed, the process ends. If not, the process proceeds to step S2008. In step S2008, the subject number x is incremented by 1, and the process returns to step S2005 to continue the integration. The integration at this time is not for each subject but for all the pixels related to all subjects.

  FIG. 21 shows an example of the pixel number distribution (histogram) for each distance in the above operation. FIG. 21 shows an example in which six subjects from a to f are extracted. Each closed curved surface area indicated by a to f represents the number of pixels of each subject, and the entire envelope is a histogram obtained by integrating the number of pixels of all subjects. The generation order determination described later is performed using an envelope.

  FIG. 21A is a histogram before being multiplied by a coefficient corresponding to the distance from the screen center position to the center of gravity position, and FIG. 21B is a histogram after being multiplied by the coefficient. The top three peaks in FIG. 21A have a distance B → C → A, but in FIG. 21B, F → E → D.

  The determination method of the refocus image data generation order and the image generation according to the order can be the same as in the third embodiment. According to the configuration as described above, the following effects can be expected in addition to the effects described in the third embodiment. That is, since the refocus image generation order is determined based on the subject recognition result, it is possible to exclude a background that has not been recognized as a subject.

  Note that the refocus image data generation order determination method used in this embodiment can be combined with other determination methods such as the first embodiment and the second embodiment. For example, when the human mode is designated, there are an example in which the method of the present embodiment is applied only to a person among subjects, and an example in which the coefficient is changed not only by the distance from the screen center position but also by the mode. Can be mentioned. In addition, using the history information described in the first embodiment, the technique of this embodiment may be applied only to subjects whose operation frequency is equal to or higher than a predetermined threshold.

  The description of the fourth embodiment is based on the hardware configuration shown in FIG. 1, but in these embodiments, as shown in FIG. 12, the hardware configuration shown in FIG. It can also be applied to a system configuration including a network. In this case, the image processing described in the fourth embodiment can be realized in the components such as the cloud server 1204 and the display terminals 1205 to 1207, for example, image processing of multi-viewpoint image data captured by the multi-view camera 1201. Is possible. FIG. 15 shows a block diagram of the display terminals 1205 to 1207. In the block diagram shown in FIG. 15, processing equivalent to that described in FIG. 1 can be performed except that multi-viewpoint image data is acquired via the network I / F. FIG. 13 shows an example of a detailed hardware configuration block diagram of the cloud server 1204. In the cloud server 1204, a function corresponding to the image analysis unit 112 in FIG. 1 may be performed by the overall control unit 1307, or from the multi-view camera 1201 and the display terminals 1205-1207 via the network communication unit 1301. The analysis result may be received.

<Other examples>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (9)

Determining means for determining the order of objects to be focused based on distance information indicating the depth of an image indicated by multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions;
An image processing apparatus comprising: generation means for sequentially generating composite image data focused on the object from the multi-viewpoint image data according to the order determined by the determination means.   2. The image according to claim 1, wherein the determination unit calculates the number of pixels for each distance from a depth map of each pixel constituting the multi-viewpoint image, and determines the order based on the calculation result. Processing equipment.   The said determination means determines the said order | rank based on the result of further multiplying the coefficient according to the distance from the screen center of each pixel with respect to the calculated pixel number for every distance. The image processing apparatus described. Recognizing means for recognizing an object included in the image indicated by the multi-viewpoint image data;
The image processing apparatus according to claim 1, wherein the determination unit determines the order for each object recognized by the recognition unit.   The determining means calculates the centroid position of the object recognized by the recognizing means, and determines the rank based on a result of multiplying a coefficient corresponding to the distance from the screen center position of each object to the centroid position. The image processing apparatus according to claim 4. From the multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions, the composite image data focused on the object corresponding to the depth information indicating the depth of the multi-viewpoint image indicated by the multi-viewpoint image data is received from the user. An image processing apparatus comprising display means for displaying without inputting parameters. A determination step of determining the order of objects to be focused based on distance information indicating the depth of an image indicated by multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions;
An image processing method comprising: a generation step of sequentially generating composite image data focused on the object from the multi-viewpoint image data according to the order determined in the determination step. From the multi-viewpoint image data obtained by imaging from a plurality of viewpoint positions, the composite image data focused on the object corresponding to the depth information indicating the depth of the multi-viewpoint image indicated by the multi-viewpoint image data is received from the user. An image processing method comprising a display step of displaying without inputting parameters.   A program for causing a computer to function as the image processing apparatus according to claim 1.

Images