JP6082484B2 - Information processing apparatus, processing method thereof, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, a processing method thereof, and a program.

  2. Description of the Related Art Conventionally, there are known techniques for obtaining an arbitrary focus distance and depth of field, and generating images of a plurality of viewpoints based on images obtained by photographing a subject from many viewpoints. For example, in Non-Patent Document 1, a plurality of images with different viewpoints are acquired by arranging a plurality of cameras in a grid on a two-dimensional plane (hereinafter, such a multi-viewpoint image is referred to as an image group). . Then, by giving the viewpoint, the focus distance, and the aperture diameter as parameters, an image is generated from the image group by calculation processing (hereinafter, an image generated by giving parameters is referred to as a completed image). If no parameter is specified, image generation is performed using default values. On the other hand, Patent Document 1 discloses a method of automatically setting parameters for generating an image that allows a user to easily search for an image group later.

JP 2011-22796 A

Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Marc Levoy, Mark Horowitz, "High Performance Imaging Using Large Camera Arrays", USA, Proceedings of ACM SIGGRAPH 2005, 765-776 page.

  However, the default values of parameters and images generated from parameter values set to search for images are good for users, such as giving them to someone, viewing with friends and family, and organizing them as records. The completed image is not always given a parameter suitable for a certain purpose of use.

  If an image with parameters suitable for the purpose of use cannot be obtained, it is necessary for the user to adjust parameters such as the viewpoint, focus distance, and depth, which is troublesome. Further, depending on the skill of the user, there may be a case where it is not possible to grasp what kind of image can be generated by parameter designation, and the parameter designation operation itself may be difficult.

  Therefore, the present invention has been made in view of the above problems, and an object thereof is to provide a technique for generating an image suitable for a user from a multi-viewpoint image group.

  In order to solve the above problems, an information processing apparatus according to an aspect of the present invention includes the following arrangement. That is, an information processing apparatus that generates a composite image focused on a subject of interest from an image taken from a multi-viewpoint, the image holding means for holding the image taken from a multi-viewpoint, and a user of the composite image Information on the subject, information on the subject, subject information holding means for holding relationship information indicating a relationship between a user of the composite image and the subject, and information on the subject with respect to the multi-viewpointed image. A recognition unit for recognizing a subject based on the image, a determination unit for determining a subject of interest from the recognized subject based on the relationship information, and a composite image focused on the subject of interest was captured from the multi-viewpoint. Generating means for generating from an image.

  According to the present invention, an image suitable for a user can be generated from a multi-viewpoint image group.

The figure which shows an example of a structure of the information processing system which arranged and comprised the information processing apparatus concerning one embodiment of this invention. The figure which shows an example of a structure of an imaging part. 2 is a diagram illustrating an example of a functional configuration of the information processing apparatus 30. FIG. The figure which shows the outline | summary of related information. The figure which shows the outline | summary of the image obtained by the process of the to-be-recognized part 43. FIG. 4 is a flowchart showing an example of a processing flow of the information processing apparatus 30. 4 is a flowchart showing an example of a processing flow of the information processing apparatus 30. FIG. 9 is a diagram illustrating an example of a functional configuration of an information processing apparatus 30 according to the second embodiment. FIG. 9 is a diagram illustrating an example of a functional configuration of an information processing apparatus 30 according to a third embodiment. FIG. 9 is a diagram illustrating an example of a functional configuration of an information processing apparatus 30 according to a third embodiment. The figure which shows the outline | summary of a modification.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a diagram showing an example of a configuration of an information processing system configured by arranging an information processing apparatus according to an embodiment of the present invention.

  The information processing system includes an imaging device 10 and an information processing device 30, and these devices are communicably connected via a communication means such as a USB (Universal Serial Bus).

  The information processing apparatus 30 registers and manages various images (for example, images captured by the imaging apparatus 10). For example, the information processing apparatus 30 includes a computer. The computer includes main control means such as a CPU (Central Processing Unit), and storage means such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). In addition, the computer may also include input / output means such as a mouse and a display, communication means such as a network card, and the like. These components are connected by a bus or the like, and are controlled by the main control unit executing a program stored in the storage unit.

  The imaging apparatus 10 employs a multi-view system including a plurality of imaging units. Here, the imaging apparatus 10 includes a plurality of imaging units 101 to 109, a ROM 201, a RAM 202, a CPU 203, an operation unit 205, a display control unit 207, a display unit 206, and an imaging unit control unit 208. It is provided and configured. In addition to this, the imaging apparatus 10 is also provided with a digital signal processing unit 209, an encoder unit 210, an external memory control unit 211, and an image output unit 212. Note that these components provided in the imaging apparatus 10 are communicably connected via a bus 204 or the like.

  A CPU (Central Processing Unit) 203 performs various input / output controls and data processing, and a RAM (Random Access Memory) 202 temporarily stores calculation results and data. A ROM (Read Only Memory) 201 stores a program and data for executing control according to the present embodiment. Each configuration in the imaging apparatus 10 is realized, for example, by the CPU 203 executing a program stored in the ROM 201 using the RAM 202 as a work area.

  The operation unit 205 is realized by, for example, a button or a mode dial, and inputs various instructions from the user into the apparatus. The display unit 206 is realized by, for example, a liquid crystal display or the like, and displays various types of information (for example, photographed images and characters) for the user. The display control unit 207 controls display of various information on the display unit 206. Note that the operation unit 205 and the display unit 206 may be integrally configured and realized as a touch screen.

  The imaging unit control unit 208 controls the imaging system. For example, focus adjustment, shutter opening / closing, aperture adjustment, and the like are performed. This control is performed based on an instruction from the CPU 203.

  The digital signal processing unit 209 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 204. The encoder unit 210 converts the digital data into a file format such as JPEG or MPEG. The external memory control unit 211 controls communication with an external memory inserted in the imaging device 10. Examples of the external memory include information processing apparatuses (for example, PC (Personal Computer)) and media (for example, hard disk, memory card, CF card, SD card, USB memory) and the like.

  The imaging units 101 to 109 acquire a color image. The plurality of imaging units 101 to 109 are equally arranged on a square lattice, for example, as illustrated in FIG. When the user presses the shooting button 21, the imaging units 101 to 109 receive light information of the subject at the sensor (imaging device). Then, A / D conversion is performed on the received light signal inside the imaging apparatus 10, thereby acquiring a plurality of color images (digital data). With such a multi-lens imaging device 10, it is possible to obtain a color image group obtained by imaging the same subject from a plurality of viewpoint positions.

  Here, the configuration of the imaging units 101 to 109 will be further described. Since the imaging units 101 to 109 basically have the same configuration, the configuration of the imaging unit 101 will be described as a representative. As shown in FIG. 2B, the imaging unit 101 includes a plurality of lenses 301 to 303, a diaphragm 304, a shutter 305, an optical low-pass filter 306, an iR cut filter 307, a color filter 308, a sensor 309, and an A / D conversion unit. 310 is comprised.

  The plurality of lenses 301 to 303 includes, for example, a zoom lens 301, a focus lens 302, and a shake correction lens 303. The sensor 309 is realized by, for example, an imaging device such as a CMOS (Complimentary Metal Oxide Semiconductor) or a CCD (Charge-Coupled Devices), and detects light passing through each of the above-described configurations. The A / D conversion unit 310 converts the light detected by the sensor 309 into a digital value and supplies it to the bus 204 as digital data.

  The image output unit 212 outputs the color image group acquired by the imaging units 101 to 109 or the color image group output from the digital signal processing unit 209 in a state where it can be identified as an image group obtained by photographing the same subject. .

  The above is an explanation of an example of a functional configuration of the imaging apparatus 10. Note that the configuration of the imaging apparatus 10 described above is merely an example, and is not limited to the above configuration, and the configuration may be appropriately added or omitted as necessary.

  For example, in the above description, the case where the number of imaging units is nine is described as an example, but the number of imaging units is not limited to nine. That is, the imaging device only needs to include a plurality of imaging units, and any number of imaging units may be provided. Moreover, although the case where nine imaging units are equally arranged on a square lattice has been described as an example, the present invention is not limited to this, and the arrangement of imaging units may be arbitrarily determined. For example, they may be arranged radially or linearly, or may be arranged at random. In addition, some or all of the images captured by the imaging units 101 to 109 may be monochrome images. In that case, the configuration of the color filter 308 shown in FIG.

  Next, an example of a functional configuration of the information processing apparatus 30 illustrated in FIG. 1 will be described with reference to FIG.

  The information processing apparatus 30 includes an input unit 31, an output unit 32, a database 33, an image processing unit 34, and an image registration unit 35 as its functional configuration.

  The input unit 31 inputs, for example, an instruction from a user via a mouse or a keyboard into the apparatus, and the output unit 32 outputs various information to, for example, a liquid crystal display.

  The database 33 is implemented by, for example, a RAM or an external storage device, and functions as an image management unit that manages images and information related to images. More specifically, an image, attribute information about the image, user information of a user who uses the information processing apparatus 30, and the like are managed.

  The image registration unit 35 registers an image in the database 33. The image registration unit 35 is provided with a subject extraction unit 36. The subject extraction unit 36 performs face authentication processing on the image to be registered, thereby identifying the subject and extracting it. The face authentication process may be performed using a technique based on a general face feature. The face feature of a person to be identified is learned in advance, and a partial region in an image having a feature similar to the face feature is obtained. Identify that person. When registering an image, information on the subject specified by the face recognition of the subject extraction unit 36 is registered in the database 33 as subject information.

  The image processing unit 34 performs various types of image processing on the (multi-viewpoint) image group registered in the database 33. The image processing unit 34 includes an image acquisition unit 40, a relationship information generation unit 41, a relationship information management unit 42, a subject recognition unit 43, a target object determination unit 44, and a completed image generation unit 45. .

  The image acquisition unit 40 acquires an image group from the database 33. The relationship information generation unit 41 generates relationship information between subjects from the image group acquired by the image acquisition unit 40. When generating this relationship information, for example, the subject is recognized from the entire image group managed by the user, and the appearance frequency for each subject, the frequency of the subject that co-occurs in the same image (co-occurrence frequency), and the like are tabulated. To do. With the appearance frequency for each subject, for example, the ease of appearance of the subject in the image group managed by the user can be known, and this can be used to generate the relationship information between the user and the subject. In addition, since the co-occurrence frequency of the subjects shows the tendency of being photographed together, the relationship information between the subjects can be generated using this. Instead of using the appearance frequency as it is, the ratio of the appearance frequency to the number of image groups may be obtained.

  Note that the relationship information may be only information indicating a relationship that is simply related, but may include attribute information of the relationship. For example, the information such as the appearance frequency and the co-occurrence frequency of the subject as described above is information indicating the strength of the relationship, and may be included in the relationship information as attribute information.

  Of course, the attribute information is not limited to this. For example, it is assumed that the user can specify a logical image group for the information processing apparatus 30 (for example, it can be specified to collectively manage images taken in one trip). In this case, information such as the appearance frequency in the logical unit may be aggregated, and information indicating that each unit is distinguished may be included as the related attribute information.

  In addition, in the case where role information such as the existence of each other regarding the relationship between subjects is known, it may be included in the relationship information as relationship attribute information. For example, if the subjects D and M are in a parent-child relationship, the information may be included as relationship attribute information. Such role information may be acquired from any information. For example, you may acquire from the information which the user registered with respect to the information processing apparatus 30. FIG.

  For example, as shown in FIG. 4, the relationship information generated in this way can be expressed by a network structure in which each user or subject is a node. In this case, the node U indicates the user, and the nodes A, B, F, and G indicate the relationship that the appearance frequency is equal to or higher than a preset threshold in the image managed by the user. On the other hand, since the appearance frequency in the image group managed by the user is not sufficient, the node C has no relationship with the user. However, since the co-occurrence frequency of the node C and the nodes A and B is equal to or higher than a preset threshold value, the node C indicates that there is a relationship of being photographed simultaneously with the nodes A and B.

  The data format of the relationship information may be any format that can express the structure as described in FIG. 4 and may be any format. If there is a possibility that each relationship includes attribute information, the relationship information may be generated in an XML format in which structured information can be described. Further, for example, description may be made in XML description specifications (RDF / XML) using RDF (Resource Description Framework). RDF is standardized by the World Wide Web Consortium (W3C) as a format for describing resources such as people and things and their relationships.

  The relationship information management unit 42 manages the relationship information generated by the relationship information generation unit 41. Subject information management is performed by a configuration including the relationship information management unit 42, the database 33, and the like. The relationship information management unit 42 manages new relationship information by merging with existing relationship information. For example, if the relationship between the person A and the new person X is detected, the new relationship, its attribute information, and the fact that the target is X are added to the description part of the relationship related to the existing A.

  In addition, the relation information management unit 42 searches for relation information that matches the condition in response to a query including the subject or the relation and its attribute information as a key, and responds with the result. As a query, for example, a query that instructs a search for a relationship related to the person A and a person who is the relationship destination. For this search, for example, a query language such as SPARQL (SPARQL Protocol and RDF Query Language), which is a query description language for searching for data in RDF / XML format standardized by W3C, may be used. As for the relationship information managed and the data format of the relationship information to be responded, any data format can be used as long as it can describe structured information such as the RDF / XML format, as in the relationship information generation unit 41. .

  The subject recognition unit 43 performs face recognition processing while performing focus adjustment processing on the image group acquired by the image acquisition unit 40. As a result, an image of the focused subject is acquired. More specifically, focus adjustment processing is performed on each image, and an image focused on a target (subject) at each focus distance is generated. Note that the focus adjustment process may be performed, for example, by setting a parameter for changing (increasing) the focus distance at an arbitrary interval set in advance.

  Here, an outline of an image obtained by the processing of the subject recognition unit 43 will be described with reference to FIGS. 5 (a) to 5 (c).

  A to D51 indicate subjects. Here, the case where the imaging device 10 is present in the downward direction in the figure is shown, and the relative position from the imaging device 10 is shown. Reference numeral 52 denotes a focus distance set at a certain interval, and a complete image (an image generated by giving a parameter) using these focus distances as parameters is generated.

  Reference numeral 53 indicates a focus distance. With the focus distance in FIG. 5A, a completed image in which at least the subject A is in focus is obtained. Similarly, the subjects B and D are in focus at the focus distance in FIG. 5B, and the subject C is in focus at the focus distance in FIG.

  In the subject recognition unit 43, face recognition processing is performed on each image obtained in this way, and as shown in FIG. 5 (d), identification can be performed with a likelihood higher than a preset threshold value. Generate a list of people.

  The target object determination unit 44 determines a target subject (target subject) from among the subjects specified by the subject recognition unit 43. For example, when obtaining a completed image suitable for a user who has registered an image group, the target object determination unit 44 acquires relationship information including the user from the relationship information management unit 42. Then, a person included in the related information is selected from the persons specified by the subject recognition unit 43, and the person is set as a target of attention.

  At this time, it is necessary to associate the person (subject) specified by the target object determination unit 44 with the person (subject) managed by the relationship information management unit 42. Therefore, the target object determination unit 44 holds, for example, the correspondence between the subject specified by the subject extraction unit 36 and the subject specified by the subject recognition unit 43. Note that any method may be used as long as the subjects can be associated with each other. For example, by sharing the face recognition processing in the target object determination unit 44 and the subject extraction unit 36, the identification results of both may be shared.

  The completed image generation unit 45 generates an image in which focus adjustment has been performed on the target object determined by the target object determination unit 44. In this embodiment, the focus distance information and the face position when the subject person is identified by the subject recognition unit 43 are retained. Then, the focus distance at the face position is determined around the focus distance.

  Specifically, an image centered on the focus distance is generated, and the focus match is determined using a general-purpose autofocus method such as a contrast method for obtaining focus from the image. At this time, the image with the best focus is set as a completed image.

  If there are a plurality of target objects, a completed image for the target object found is generated. Of course, the present invention is not limited to such a method. For example, if frequency information is obtained as the attribute information of the relationship managed by the relationship information management unit 42, the subjects are sorted in descending order of frequency, and the higher-order subjects are sorted. Of these, a predetermined number of completed images may be generated.

  The above is an example of the configuration of the information processing apparatus 30. Note that the configuration of the information processing apparatus 30 illustrated in FIG. 3 is merely an example, and this configuration may be changed as appropriate. For example, a part or all of the configuration realized in the image processing unit 34 may be provided in an external device (for example, a cloud format). In that case, the information processing apparatus 30 performs various processes in response to processing results from the external apparatus.

  Next, an example of a processing flow of the information processing apparatus 30 illustrated in FIG. 1 will be described with reference to FIGS. 6 and 7. First, the flow of processing when acquiring the relationship information will be described with reference to FIG.

  The information processing apparatus 30 acquires an image group from the database 33 in the image acquisition unit 40 (S101). In the present embodiment, it is assumed that a logical group is defined in advance for the information processing apparatus 30 by the user, and management is performed so that images belong to the group. Therefore, in the process of S101, an image group is acquired for each logical group.

  Subsequently, in the image processing unit 34, the information processing apparatus 30 acquires subject information of the image group from the database 33 (S102). As described above, the subject information of the subject specified by the face recognition process of the subject extraction unit 36 is registered in the database 33 when the image is registered. Therefore, in the process of S102, the subject information is acquired from the database 33.

  In the image processing unit 34, the information processing apparatus 30 acquires information (user information) of a user who manages the image group acquired in S101 (S103). Note that the user here refers to, for example, a user who uses the information processing apparatus 30. In the process of S103, user information may be acquired based on, for example, login information. Of course, the structure which can input the user information used as object separately may be sufficient.

  When the user information is acquired, the information processing apparatus 30 aggregates the subject information acquired in the process of S102 in the relationship information generating unit 41 and generates the relationship information with the user acquired in the process of S103 (S104). . Then, the relationship information between the subjects is generated from the frequency with which the subject acquired in the process of S103 co-occurs in the same image (S105). Thereafter, the information processing apparatus 30 registers and manages the generated relation information in the relation information management unit 42 (S106).

  Next, the flow of processing when a completed image is generated (automatically) from an image group will be described with reference to FIG. The process shown in FIG. 7 is performed at the time of image registration, for example, and is subsequently executed after the process described with reference to FIG. Of course, it may be carried out by a user instruction or at an arbitrary timing (automatically).

  The information processing apparatus 30 acquires an image group in the image acquisition unit 40 (S201). Then, the subject recognition unit 43 processes the image group acquired in the process of S201, and recognizes the subject from the image group (exhaustively) while performing focus adjustment (S202).

  Subsequently, in the image processing unit 34, the information processing apparatus 30 acquires information (user information) of a user who is processing an image group (S203). The user here refers to, for example, a user who uses the information processing apparatus 30. In the process of S203, for example, user information may be acquired based on login information or the like.

  After the acquisition of the user information, the information processing apparatus 30 specifies the subject related to the user acquired in the process of S203 from the subjects recognized in the process of S202 in the target object determination unit 44, The target object is determined (S204).

  Finally, the information processing apparatus 30 acquires (generates) a completed image in which the focus adjustment is performed on the target of interest determined in the process of S204 in the completed image generation unit 45 (S205).

  As described above, according to the present embodiment, when an image group is registered, a completed image in which focus adjustment has been performed on the target object (an image on which focus adjustment has been performed on the target object) is generated. That is, based on a plurality of images (various focus distances), an image in which a subject closely related to the user is focused is generated. Therefore, an image suitable for the user is automatically generated.

  Thereby, the user's effort at the time of generating a completed image by specifying parameters or the like can be suppressed. For example, when an image related to an acquaintance from a plurality of image groups is sent, the user does not have to perform a complicated operation of generating a completed image for each image, so that the convenience for the user is improved.

(Embodiment 2)
Next, Embodiment 2 will be described. In the first embodiment described above, the case in which the target object determination unit 44 finds the target object based on the presence or absence of the relationship between the subjects has been described as an example, but the present invention is not limited to this. As an example, in the second embodiment, a case will be described in which a subject of interest is determined in accordance with a context such as who uses an image for what.

  As illustrated in FIG. 8, the information processing apparatus 30 according to the second embodiment is newly provided with a context acquisition unit 46 that acquires a context in the image processing unit 34.

  The context acquisition unit 46 acquires, for example, the context obtained in the process of operating the output function for the user. Then, in the target object determination unit 44 according to the second embodiment, the target object is determined based on the result (weighted result) of evaluating the relation information described above according to the context. In addition, the weighting according to a context shall be preset, for example. Examples of the context include presents for albums to grandparents, creation of a slide show of growth records of married daughters, travel photos viewed by accompanying people, and other information for assisting general browsing. That is, the context is set for each group of predetermined image groups.

  Here, the weighting is set in advance for each context based on a certain subject or a related attribute. For example, when presenting an album to grandparents, it is desirable to select an image centered on the grandchild. Therefore, if roll information is set as the relationship information, the relationship is weighted from the image group so that the subject corresponding to the grandchild as seen from the grandparents becomes the target of attention.

  In addition, when creating a slide show of a growth record of a married daughter, weighting is performed so as to include a subject corresponding to the daughter as viewed from the user. At this time, if a relationship such as the distribution of co-occurrence frequency at a certain time is managed as a relationship attribute, weighting is performed so that the daughter and the subject who often co-occurs at that time also become the subject of interest at the same time. May be performed.

  In addition, when traveling photographs are gathered and viewed by some of the accompanying persons, weighting is performed so as to include a person who is present on the spot as a target of interest among subjects related to the user's acquaintance. Or, for general browsing, importance is placed on relationships that can be regarded as important regardless of context. The relationship that can be regarded as important indicates, for example, a close blood relationship for the user or a high co-occurrence relationship at a close date and time as viewed from the time when the completed image is generated.

  Also, if the image group matches the schedule of a known annual event (for example, Mother's Day, Children's Day, family or acquaintance's birthday), the relationship with the person who will play a leading role in the event The rule may be weighted. For example, in the case of an image taken on Mother's Day, the relationship is weighted so that the mother or a subject that often co-occurs with the mother becomes the target of attention. Similarly, when the viewing date coincides with a known annual event, the subject that is the center of the event may be weighted so that it becomes the subject of interest.

  Note that such weighting does not need to be used exclusively. In particular, the content described as a general browsing rule is not dependent on a specific context, so always consider it. It doesn't matter.

  As described above, according to the second embodiment, a more suitable image can be obtained in context-dependent processing (album generation, slide show generation, etc.). In addition, since a suitable completed image can be automatically obtained, automatic generation processing by an application or the like can be realized without any user intervention.

(Embodiment 3)
Next, Embodiment 3 will be described. In the first embodiment described above, the case where the relationship information is generated from the image group acquired by the image acquisition unit 40 has been described. However, the present invention is not limited to this. For example, information such as a user's relationship with a user or a subject registered in a related information management service such as a social networking service (hereinafter referred to as SNS) may be used as the related information. In other words, the relationship information management service may be used for generation and management of relationship information between users and subjects.

  In this case, the configuration of the information processing apparatus 30 is, for example, the configuration illustrated in FIGS. 9 and 10. That is, a cooperation unit 47 is newly provided in the image processing unit 34. When the information processing apparatus 30 is connected to an external service, the cooperation unit 47 accesses the external service via a network interface (not shown) and cooperates. Thereby, a part of function (FIG. 9) or all the functions (FIG. 10) in the relationship information generation part 41 and the relationship information management part 42 are realizable using the said relationship information management service.

  Here, since it is necessary to use the relationship information on the relationship information management service, the cooperation unit 47 has at least a function of logging in to the relationship information management service as a user and a function of acquiring information about the relationship information of the user. Have. At this time, it is necessary to associate the relationship information on the relationship information management service with the relationship information handled in the information processing apparatus 30. For this association, a table for associating both identifiers to be managed may be created and managed by the linkage unit 47.

  Further, information corresponding to this table may be input and set in advance by the user. For example, when registering a face image as a recognition target in the information processing apparatus 30, an identifier such as a user name on the related information management service may be set as the registration target identifier. Even if it is not possible to input the association between the two, or when no input is made, the association may be performed using the user name and the user attribute managed by both. In this case, for example, it may be determined whether or not the association is performed based on whether or not the same information is input to the same attribute.

  In the related information management service, there may be a case where related information cannot be generated from an image. In such a case, the processing result by the relationship information generation unit 41 may be added to the relationship information in the relationship information management service via the cooperation unit 47. In this case, the cooperation unit 47 has a function of editing the relationship information on the relationship information management service. Then, the relationship information of a certain user generated by the relationship information generation unit 41 is added / updated as the relationship information of the user on the relationship information management service with which the user is associated.

  As described above, according to the third embodiment, when the creation and management of related information is performed by an external service such as SNS, these are linked. As a result, the management of the related information can be performed in a unified manner without being distributed. In addition, the cost of registration and maintenance can be reduced and the misregistration of mutual registration contents can be reduced as compared with the case where such relationship information is managed separately.

  The above is an example of a typical embodiment of the present invention, but the present invention is not limited to the embodiment described above and shown in the drawings, and can be appropriately modified and implemented without departing from the scope of the present invention. . Hereinafter, some modifications will be described.

(Modification 1)
In the above description, a case has been described in which the subject extraction unit 36 and the subject recognition unit 43 perform face recognition processing to specify a person from a face image, but the present invention is not limited to this. Recognition processing may be performed using a pet or object as a subject. For example, in the example described in the second embodiment, in the travel record, if the landmark of the destination can be recognized, the relationship between the landmark and the person is weighted by weighting the relationship between the landmark and the person who co-occurs. May be the target of attention.

(Modification 2)
In the above description, when there are a plurality of objects of interest, a case where a completed image is generated so that the plurality of objects of interest are in focus has been described. However, the present invention is not limited to this. For example, the focus distance of each target object is aggregated, the midpoint of the distance is set as the focus distance parameter, and the aperture diameter parameter is narrowed so that the depth width from the focus position is deep. Also good. By doing in this way, it is possible to generate a completed image in which the entire target object is in focus.

  This process will be described more specifically with reference to FIG. Here, A and B indicate a subject selected as a target of interest, and reference numeral 10 indicates an imaging device for capturing a multi-viewpoint image. In this case, the focus position is set to an intermediate point between A and B as indicated by reference numeral 61, the aperture diameter parameter is adjusted, and a complete image is generated so that the depth width 62 covers A and B.

  Note that the relationship information between the target objects managed by the relationship information management unit 42 may be acquired, and the process shown in FIG. 11A may be performed when there is a relationship between the target objects. That is, a set of target objects may be generated from a plurality of target objects based on the relationship information, and a completed image may be generated for each set.

  At this time, if information that can evaluate the strength of the relationship, such as the co-occurrence frequency between each target object, can be obtained as the attribute information of the relationship, the subjects having the co-occurring relationship are prioritized. Thus, a complete image may be generated in which the entire subject is in focus.

  In addition, when the relationship weighting information as described in the second exemplary embodiment is obtained, a completed image may be generated in which the entire subject having an important relationship along the context is focused.

  As described above, according to the second modification, a complete image is generated in which a plurality of subjects are all in focus. Thereby, also when utilizing such an image, the effort which a user produces | generates an image can be suppressed.

(Modification 3)
Furthermore, the present invention is not limited to the processing of the second modification. For example, a technique (conventional technique) that obtains a curved surface including the coordinate positions of each of a plurality of target objects and performs focus adjustment along the curved surface may be used. I do not care.

  This process will be described with reference to FIG. 11B. When there are four objects of interest from A to D, a curved virtual focal plane 70 is set, and focus adjustment is performed in accordance with this focal plane. A completed image is generated. Note that, as in the second modification, it is of course possible to determine the combination of the target objects by looking at the relationship information between the target objects.

  As described above, according to the third modification, even if there is variation in the distance distribution of each target object from the imaging device, it is possible to generate a completed image with a shallow depth focused on the entire target object.

(Modification 4)
For example, the recognition target in the subject extraction unit 36 may be character string information of a text object in the image. In this case, the character string is recognized, it is determined that the text information is related to the subject in the image, and when the character string and the subject are related, focus adjustment is performed so that both are in focus. A completed image is generated.

  For example, it is assumed that a signboard is shown in the image, the letters “entrance ceremony” are included, and the daughter is shown. In this situation, by referring to the daughter's date of birth information, image shooting date / time information, and basic knowledge information about the `` entrance ceremony '' You may make the image which focused on both the signs.

(Modification 5)
In the above-described embodiment, the case where an image group captured by the imaging apparatus 10 is input to the information processing apparatus 30 has been described, but the present invention is not limited to this. That is, an image group that can be subjected to a focus adjustment process may be input to the information processing apparatus 30. For example, it is known that an equivalent image group can be obtained even when a plurality of lenses and a single sensor are configured, and the image group obtained in this way may be input to the information processing apparatus 30.

(Modification 6)
In the embodiment described above, the focus adjustment process in the completed image generation unit 45 has been described by taking as an example the case where the target object is focused. However, the focus adjustment process for other parts, for example, the background part Is not specifically mentioned.

  Explaining this point, the range other than the target of interest may be a blurred image as much as possible, or an image in which other portions are focused as much as possible. For example, either may be determined depending on the context. That is, how to adjust the focus of the area other than the target of interest according to the context obtained by the configuration including the context acquisition unit 46 (see FIG. 8) may be managed together with the relationship evaluation rule. . And according to the content, the completion image generation part 45 produces | generates a completion image.

  Alternatively, as a more general example, if the subject of interest is a person, a face of a certain size is occupied in the image, and there is no other recognizable subject, an image such as a portrait is displayed. There is a high possibility that the user has taken an image. In such a case, an image with a blurred background is generated to emphasize the face portion. A completed image may be generated by associating such a typical rule with a focus adjustment method for a portion other than the target object. In this case, it is only necessary to set in advance in the completed image generation unit 45 information on the correspondence between the rule and the focus adjustment method of the part other than the target object.

(Modification 7)
In the above-described embodiment, the case where the processing target is a still image has been described as an example, but the present invention is not limited to this. For example, a moving image in which each frame has information on a multi-viewpoint image may be used. In that case, the above-described processing may be performed for one frame in the moving image.

  However, in order to browse as a moving image, it is not preferable that the subject of interest varies minutely for each frame. In many cases, a moving image captures a scene rather than capturing a momentary scene such as a photograph. Therefore, it is desirable to determine a target of interest for the entire scene and perform focus adjustment on the target of interest having a sense of unity throughout the scene.

  In this case, each component in the image processing unit 34 described above performs processing on the entire frame image within the time representing the scene. That is, the subject of the whole frame in a scene is totaled, a relationship is produced | generated from there, or the attention object is determined.

  In addition, the completed image as a moving image is determined by a general tracking process that tracks the determined subject of interest, and an area for performing focus adjustment in each frame is determined. A completed image is generated.

  Note that the subject of interest may be temporarily hidden behind an obstacle or out of the frame, resulting in a frame that cannot be tracked. So that the focus distance does not fluctuate greatly, the subject area and focus distance in the untracked frame are interpolated from the subject area and focus distance in the frame that can be tracked before and after. May be.

(Other embodiments)
It should be noted that the present invention can also take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to a device composed of a single device. good.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

30 Information Processing Device, 31 Input Unit, 32 Output Unit, 33 Database, 34 Image Processing Unit, 35 Image Registration Unit, 36 Subject Extraction Unit

Claims (13)

An information processing apparatus that generates a composite image focused on a subject of interest from an image taken from multiple viewpoints,
Image holding means for holding images taken from multiple viewpoints;
Input means for inputting a user of the composite image;
Subject information holding means for holding information about the subject and relationship information indicating a relationship between the user of the composite image and the subject;
Recognizing means for recognizing a subject based on information on the subject with respect to the multi-viewpoint image;
Determining means for determining a subject of interest from the recognized subjects based on the relationship information;
Generating means for generating a composite image focused on the subject of interest from the multi-viewpoint image;
An information processing apparatus comprising: Image holding means for holding images taken from multiple viewpoints;
Subject information holding means for holding information about the subject and relationship information indicating a relationship between a photographer of the image taken from the multi-viewpoint and the subject;
Recognizing means for recognizing a subject based on information on the subject with respect to the multi-viewpoint image;
Determining means for determining a subject of interest from the recognized subjects based on the relationship information;
Generating means for generating a composite image focused on the subject of interest from the multi-viewpoint image;
An information processing apparatus comprising: An information processing apparatus that generates a composite image focused on a subject of interest from an image taken from multiple viewpoints,
Image holding means for holding images taken from multiple viewpoints;
Subject information holding means for holding information about a subject and relationship information indicating a relationship between a user of the information processing apparatus and the subject;
Recognizing means for recognizing a subject based on information on the subject with respect to the multi-viewpoint image;
Determining means for determining a subject of interest from the recognized subjects based on the relationship information;
Generating means for generating a composite image focused on the subject of interest from the multi-viewpoint image;
An information processing apparatus comprising: Information relating to the subject, an information processing apparatus according to any one of claims 1 to 3, characterized in that the characteristic information learned in advance the subject. The relationship information includes the appearance frequency of the subject in the multi-viewpoint image,
Said generating means, wherein a composite image focused on the subject of the noted from the multi-viewpoint captured image, in any one of claims 1 to 4, wherein the generating sequentially the occurrence frequency is high Information processing device. When the determining means determines a plurality of subjects as the subject of interest,
The generating means sets a focus distance based on a focus distance of each of the subjects of interest and an aperture diameter parameter that covers a plurality of subjects of interest with a depth width, and combines the plurality of subjects of interest in focus. the information processing apparatus according to any one of claims 1 to 5, characterized in that to generate the image from the multi-view pictures. When the information processing apparatus is connected to an external service that manages information indicating the relationship between the subjects, the relationship information managed by the external service and the relationship information managed by the subject information holding unit are linked. the information processing apparatus according to any one of claims 1 to 6, characterized by further comprising a linkage means for. A processing method of an information processing device that generates a composite image focused on a subject of interest from an image taken from multiple viewpoints,
An image holding means for holding an image taken from multiple viewpoints;
An input means for inputting a user of the composite image;
Subject information holding means, the step of holding the information about the object, and a relation information showing the relation between the object and the user of the composite image,
A step of recognizing a subject based on information on the subject with respect to the multi-viewpoint image;
A step of determining a subject of interest from the recognized subjects based on the relationship information;
A generating unit generating a composite image focused on the subject of interest from the multi-viewpoint captured image;
The processing method characterized by including. A processing method for an information processing apparatus,
An image holding means for holding an image taken from multiple viewpoints;
Subject information holding means for holding information about the subject and relationship information indicating a relationship between a photographer of the image taken from the multi-viewpoint and the subject;
A step of recognizing a subject based on information on the subject with respect to the multi-viewpoint image;
A step of determining a subject of interest from the recognized subjects based on the relationship information;
A generating unit generating a composite image focused on the subject of interest from the multi-viewpoint captured image;
A processing method comprising: A processing method of an information processing device that generates a composite image focused on a subject of interest from an image taken from multiple viewpoints,
An image holding means for holding an image taken from multiple viewpoints;
Subject information holding means for holding information about a subject and relationship information indicating a relationship between a user of the information processing apparatus and the subject;
A step of recognizing a subject based on information on the subject with respect to the multi-viewpoint image;
A step of determining a subject of interest from the recognized subjects based on the relationship information;
A generating unit generating a composite image focused on the subject of interest from the multi-viewpoint captured image;
A processing method comprising: In a computer provided in an information processing device that generates a composite image focused on a subject of interest from an image taken from multiple viewpoints,
An image holding process for holding images taken from multiple viewpoints;
And as the input engineering to enter a user of the composite image,
A subject information holding step for holding information about a subject and relationship information indicating a relationship between a user of the composite image and the subject;
A recognition step for recognizing a subject based on information about the subject with respect to the multi-viewpoint image;
A determining step of determining a subject of interest from the recognized subjects based on the relationship information;
Generating a composite image focused on the subject of interest from the multi-viewpoint image;
A program for running In the computer equipped with the information processing device,
An image holding process for holding images taken from multiple viewpoints;
A subject information holding step for holding information about a subject and relationship information indicating a relationship between a photographer of the image taken from the multi-viewpoint and the subject;
A recognition step for recognizing a subject based on information about the subject with respect to the multi-viewpoint image;
A determining step of determining a subject of interest from the recognized subjects based on the relationship information;
Generating a composite image focused on the subject of interest from the multi-viewpoint image;
A program for running In a computer provided in an information processing device that generates a composite image focused on a subject of interest from an image taken from multiple viewpoints,
An image holding process for holding images taken from multiple viewpoints;
A subject information holding step for holding information about a subject and relationship information indicating a relationship between a user of the information processing apparatus and the subject;
A recognition step for recognizing a subject based on information about the subject with respect to the multi-viewpoint image;
A determining step of determining a subject of interest from the recognized subjects based on the relationship information;
Generating a composite image focused on the subject of interest from the multi-viewpoint image;
A program for running

Images