JP2017173900A - Information processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device that improves accuracy of image search.SOLUTION: A tagging device 10 according to the present invention includes: a communication unit 11 for obtaining multiple pieces of image management information indicating a time, a place, or an event related to an image; and a combination unit 14 for deriving context indicating an action of a user related to the image by combining the multiple pieces of image management information obtained by the communication unit 11.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus related to image management.

  2. Description of the Related Art Conventionally, a tagging system that associates date and time with an image is known (see, for example, Patent Document 1). By using such a tagging system, it becomes easier to search for images.

Special table 2015-501982 gazette

  Here, the tagging system as described above associates a date and a place with an image individually. In such a tagging system, it is difficult to say that the accuracy of image search is sufficiently secured.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an information processing apparatus that improves the accuracy of image search.

  An information processing apparatus according to an aspect of the present invention combines an acquisition unit that acquires a plurality of image management information indicating a time, place, or event relating to an image, and a plurality of image management information acquired by the acquisition unit, A combination unit for deriving a user context indicating the user's action related to the image.

  In this information processing apparatus, a plurality of pieces of image management information, that is, information indicating when, where, and what are combined is combined to derive a user context indicating the user's action related to the image. By using such a user context for image search, for example, compared to the case where image search is performed by associating information such as date and place with an image alone, it is more consistent with the image. Image retrieval can be performed on the basis of information. As a result, the accuracy of the image search can be improved. In addition, even with the same recognition target, different user contexts are estimated depending on the location or situation where an image is acquired (for example, imaged), so that the accuracy of image search can be improved.

  In addition, the information processing apparatus further includes a group creation unit that creates an image group in which a plurality of images in which the position information of the point where the image is recorded and the date and time information where the image is recorded are within a predetermined range are grouped, For each image group, the combination unit may derive a group context indicating the user's behavior based on the image management information of the images included in the image group.

  By deriving a group context for each group in consideration of the position and date and time, images related to a common event (event) can be easily obtained as the same search result. When the user context is derived only for one image unit, for example, when lunch is taken during the event, it is different because the position etc. is slightly different even though it is an image of a common event. It becomes a problem that the search result becomes (outputs in pieces). Further, when one user context is derived in a long span such as one day, there is a problem that images of different events are output as the same search result. By grouping in consideration of this point, the position, and the date and time, for example, only images of common events can be set as search results. In addition, for example, even when there is a misrecognition or estimation error in the user context of a single image, some errors can be ignored by considering the user context of a plurality of images, so the accuracy of image search is improved. It can be improved further.

  The information processing apparatus further includes a candidate estimation unit configured to estimate, as a context candidate, image management information that satisfies a predetermined condition regarding the accuracy of the image management information among the plurality of image management information acquired by the acquisition unit. The combination unit may derive the user context and the group context by combining the image management information that is determined as the context candidate by the candidate estimation unit among the plurality of image management information.

  Rather than randomly selecting image management information to be combined, it is possible to further improve the accuracy of image search by using information having high accuracy as image management information.

  The plurality of image management information includes a plurality of image recognition results regarding the image, and the candidate estimation unit includes an image recognition estimation unit that estimates an image candidate that is a context candidate based on the image recognition result regarding the image. The image recognition estimation unit estimates information indicating an object whose score indicating the degree of similarity is equal to or higher than a predetermined threshold in the image recognition result regarding the image as an image candidate for deriving a user context, and within the same image group The largest number of image candidates may be estimated as image candidates for deriving the group context of the image group.

  By using information having a high score indicating the degree of similarity as an image candidate, a user context can be derived by combining information having high accuracy as image management information, and the accuracy of image search can be further improved. That is, for example, it is possible to prevent a user context from being derived from an image (photograph) that has been blurred during imaging as an image candidate. Further, by setting the most image candidates in the same image group as image candidates for deriving the group context, the group context can be derived using the representative image candidates in the image group. The accuracy of the image search can be further improved. In addition, in the estimation for each image group, it is possible to consider an image having a low score for a single image by determining image candidates as described above, instead of simply judging by a score.

  The plurality of pieces of image management information include a plurality of pieces of POI information related to images, and the candidate estimation unit includes a POI estimation unit that estimates POI candidates that are context candidates based on the POI information about images. The unit calculates a POI score obtained by dividing the number of POI check-ins by the distance from the POI to the position where the image is recorded for each of a plurality of POI information, and the POI information having the highest POI score is calculated as a user context. It may be estimated as a POI candidate for deriving, and the most POI candidates in the same image group may be estimated as POI candidates for deriving the group context of the image group.

  By considering the number of POI check-ins, it is possible to easily estimate POI information that is likely to be visited by a user as a POI candidate. Thereby, erroneous estimation can be suppressed. Further, in the estimation in units of image groups, the POI for deriving the group context from the POI information that is likely to be visited by the user because the most POI candidates are the POI candidates for deriving the group context. Can be a candidate.

  Further, the plurality of image management information includes event information specified by information indicating a position and information indicating a date and time, and the candidate estimation unit estimates an event candidate that is a context candidate based on the event information. An event estimation unit, which includes event information whose location information matches the location information of the point where the image is recorded, and whose date and time information matches the date information where the image is recorded. As the event candidate for deriving the image, the position information that matches the position information of the point where the image included in the image group is recorded, and the date and time information is the oldest among the images included in the image group. The event information included between the date and time information when the image is recorded and the date and time information when the image with the newest date and time information is recorded It may be estimated as an event candidate for deriving the loop context. Thereby, the event candidate considered that the user was visiting can be appropriately estimated in consideration of the position and the date and time. That is, the accuracy of image search can be further improved.

  The plurality of image management information includes a character recognition result regarding the image, and the candidate estimation unit includes a character recognition estimation unit that estimates a character candidate that is a context candidate based on the character recognition result regarding the image, The character recognition estimation unit estimates a predetermined character among characters in the character recognition result as a character candidate for deriving a user context, and determines the most character candidate in the same image group as a group of the image group. You may estimate as a character candidate for deriving a context.

  By using only predetermined characters as character candidates, it is possible to set only appropriate characters as keywords indicating user behavior as character candidates. In addition, by setting the largest number of character candidates in the same image group as character candidates for deriving a group context, a group context can be derived using representative character candidates in an image group. The accuracy of the image search can be further improved.

  The combination unit may associate a tag corresponding to the user context with the image, and associate a tag according to the group context with the image group. Thereby, compared with the case where only one recognition result is conventionally used as a recognition result associated with an image, tagging can be performed with reduced ambiguity and reduced abstraction, and the accuracy of image search is improved. Can be improved.

  The information processing apparatus further includes a category assigning unit that associates one or more user contexts with a category defined by a higher concept with an image, and associates one or more group contexts with a higher concept with an image group. You may prepare. By associating a category with an image and an image group, it becomes possible to search for an image with a higher concept than a user context and a group context.

  Further, the group creation unit sets a plurality of image groups in which the date and time information of the included images is within a predetermined range among the created image groups as a common new image group, and the combination unit sets the new image group as a new image group. Of the plurality of included image groups, the group context of the image group having the largest number of images may be set as the group context of the new image group.

  Thereby, image groups with similar date and time information can be further collected, and similar image search results can be confirmed more easily. In addition, since the group context of the image group having the largest number of images is set as the group context of the new image group, it is possible to improve the accuracy of the image search for the new image group.

  ADVANTAGE OF THE INVENTION According to this invention, the information processing apparatus which improves the precision of an image search can be provided.

It is a figure which shows the function structure of the tagging system which concerns on embodiment of this invention. It is a figure which shows the hardware constitutions of the tagging apparatus contained in the tagging system shown in FIG. It is explanatory drawing of the grouping process by a group preparation part. It is a table | surface which shows an estimation result table. It is a table | surface which shows a tagging result table. It is explanatory drawing of an example which tags with a single image. It is explanatory drawing of an example which performs tagging per image group. It is explanatory drawing of an example which does not tag with a single image. It is explanatory drawing of an example which does not tag with a single image. It is explanatory drawing of an example which does not tag with a single image. It is explanatory drawing of an example which does not tag by an image group unit. It is explanatory drawing of an example of category provision. It is a flowchart which shows the process of the tagging apparatus which concerns on embodiment of this invention. It is a flowchart which shows the detail of a process of a character recognition estimation part. It is a flowchart which shows the detail of a process of an image recognition estimation part. It is a flowchart which shows the detail of a process of a POI estimation part. It is a flowchart which shows the detail of a process of an event estimation part. It is explanatory drawing of album creation of the tagging apparatus which concerns on a modification.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same reference numerals are used for the same or equivalent elements, and redundant descriptions are omitted.

  FIG. 1 is a diagram showing a functional configuration of a tagging system according to the present embodiment. As shown in FIG. 1, the tagging system 1 is a system for managing images, and includes a tagging device 10 (information processing device), a data display terminal 30, a POI conversion device 40, and a character recognition device. 50, an image recognition device 60, and an event estimation device 70.

  The tagging device 10 is a server related to image management, and is configured to be able to communicate with the data display terminal 30, the POI conversion device 40, the character recognition device 50, the image recognition device 60, and the event estimation device 70. More specifically, the tagging device 10 realizes facilitation of image search by attaching a search tag to an image managed by the data display terminal 30. Details of the tagging device 10 will be described later.

  The data display terminal 30 is a terminal that stores an image and a tagging result of the image and displays the image. An image is a visually recognizable figure or photograph. In the following description, it is assumed that the image is photograph data captured by the data display terminal 30. The data display terminal 30 transmits the captured image and the position information of the point where the image is captured to the tagging device 10. The data display terminal 30 receives the tagging result regarding the image derived by the tagging device 10 from the tagging device 10 and stores it. Further, the data display terminal 30 searches for and displays an image desired by the user in response to a search operation by the user of the data display terminal 30. The tagging result described above is used for the search.

  The data display terminal 30 is, for example, a mobile phone such as a smartphone, a tablet terminal, or a laptop personal computer that is used (carried) by a user. Alternatively, the data display terminal 30 is a desktop personal computer or the like installed in the user's home or workplace.

  The POI converter 40 is a server that outputs position information indicated by latitude and longitude and outputs POI information. The POI information includes a POI (point of interest) around the position indicated by the input position information, a POI category, and the number of check-ins to each POI. The POI is information indicating a specific place in which a certain user is interested, for example, information including latitude and longitude and a name (place name). The number of check-ins is the number of users who have visited the POI. The POI conversion device 40 updates the POI information to be output by adding and editing the POI, for example, by the user. The POI conversion device 40 outputs POI information to the tagging device 10 in response to a POI conversion request from the tagging device 10. The POI conversion device 40 may be a server managed by a telecommunications carrier that manages the tagging device 10, or may be a server related to POI conversion that has been widely publicized and used.

  The character recognition device 50 is a server that discriminates characters included in an image. The character recognition device 50 stores, for example, a character template, and determines a character included in the image by determining a match with the character of the template. The character recognition device 50 receives a character recognition request including an image to be recognized from the tagging device 10, discriminates characters included in the image, and uses the discrimination result (character recognition result) as the tagging device 10. Output to. The character recognition device 50 derives a category associated with the keyword based on the keyword that is the determined character. That is, the character recognition result includes a keyword indicating the determined character and a category associated with the keyword. The character recognition device 50 may be a server managed by a telecommunications carrier that manages the tagging device 10 or may be a server related to character recognition that has been widely disclosed and used.

  The image recognition device 60 is a server that determines an object or a scene included in an image. For example, the image recognition device 60 stores a template of an object, and determines an object included in the image by determining a match with the object of the template. The image recognition device 60 receives an image recognition request including an image to be recognized from the tagging device 10, discriminates an object included in the image, and uses the discrimination result (image recognition result) as the tagging device 10. Output to. The image recognition result includes a label indicating the determined object and a score indicating the degree of matching (similarity) with the template. The image recognition device 60 may be a server managed by a telecommunications carrier that manages the tagging device 10, or may be a server related to image recognition that is widely publicized and used.

  The event estimation device 70 is a server that outputs the event information with the POI of the point where the image is recorded (imaged) and the date / time information when the image is recorded as inputs. The event information includes an event name and an event category. The event estimation device 70 stores the event information, the POI at which the event was held, and the date and time in association with each other. Then, the event estimation device 70 receives the event estimation request including the POI of the point where the image is recorded (captured) and the date / time information when the image is recorded from the tagging device 10, specifies event information related to the image, and specifies The event information is output to the tagging device 10. The event estimation device 70 may be a server managed by a telecommunications carrier that manages the tagging device 10, or may be a server related to event estimation that is widely disclosed and used in general.

  Next, details of the tagging device 10 will be described. FIG. 2 is a diagram showing a hardware configuration of the tagging apparatus 10 included in the tagging system 1 shown in FIG. As shown in FIG. 2, the tagging apparatus 10 physically includes one or a plurality of CPUs 101, a RAM 102 and a ROM 103 which are main storage devices, an input device 104 such as a keyboard and a mouse which are input devices, and an output such as a display. The computer 105 includes a device 105, a communication module 106 that is a data transmission / reception device such as a network card, an auxiliary storage device 107 such as a semiconductor memory, and the like.

  Each function of the tagging device 10 is loaded with one or more predetermined computer software on the hardware such as the CPU 101 and the RAM 102 shown in FIG. 2, so that the input device 104 and the output device 105 are controlled under the control of the CPU 101. This is realized by operating the communication module 106 and reading and writing data in the RAM 102 and the auxiliary storage device 107.

  Returning to FIG. 1, the tagging apparatus 10 includes a communication unit 11 (acquisition unit), a group creation unit 12, a candidate estimation unit 13, a combination unit 14, a category assignment unit 15, and image metadata as functional configurations. A storage unit 16, a position information storage unit 17, and an estimation information storage unit 18 are provided.

  The communication unit 11 has a function of communicating with the data display terminal 30, the POI conversion device 40, the character recognition device 50, the image recognition device 60, and the event estimation device 70. The communication unit 11 receives the captured image and the position information of the spot where the image is captured from the data display terminal 30. The communication unit 11 stores the metadata of the received image in the image metadata storage unit 16 and stores the received position information in the position information storage unit 17. The image metadata includes date and time information (that is, image creation date and time) when the image is recorded, an image file format, and the like. In the position information storage unit 17, the position information and the image are stored in association with each other. The position information may indicate latitude and longitude, or may indicate POI received from the POI conversion device 40.

  The communication unit 11 transmits a POI conversion request including the position information of the point where the image is captured to the POI conversion device 40, and acquires POI information (image management information) from the POI conversion device 40 according to the request. To do. The communication unit 11 transmits a character recognition request including an image to the character recognition device 50, and acquires a character recognition result (image management information) from the character recognition device 50 in response to the request. The communication unit 11 transmits an image recognition request including an image to the image recognition device 60, and acquires an image recognition result (image management information) from the image recognition device 60 in response to the request. The communication unit 11 transmits an event estimation request including the POI of the point where the image is recorded (captured) and the date / time information when the image is recorded to the event estimation device 70, and the event estimation device 70 responding to the request transmits the event estimation request. Event information (image management information) is acquired. In this way, the communication unit 11 acquires a plurality of pieces of image management information indicating the time, place, or event related to the image. The communication unit 11 stores each piece of image management information in the estimation information storage unit 18 as estimation information used for context estimation, and outputs an estimation request to the candidate estimation unit 13.

  The group creating unit 12 has a function of creating an image group in which a plurality of images in which the position information of the point where the image is recorded and the date information when the image is recorded are within a predetermined range are grouped. The group creating unit 12 first determines an image (reference image) to be a grouping reference. The group creating unit 12 refers to the position information storage unit 17 to extract an image in which the difference in position information from the reference image is within a predetermined range. Further, the group creation unit 12 refers to the image metadata storage unit 16 to extract an image whose date / time information difference from the reference image is within a predetermined range. Then, the group creating unit 12 creates an image group by grouping a plurality of images in which the difference in position information and date / time information from the reference image is within a predetermined range. The group creation unit 12 outputs an estimation request including the grouping result to the candidate estimation unit 13. Note that the group creation unit 12 may exclude the image from the grouping target when there is no position information associated with the image. Moreover, the group creation part 12 may group the image which concerns on the same schedule, when the information to the effect that the schedule containing the date when the image was imaged exists is acquired.

FIG. 3 is an explanatory diagram of the grouping process performed by the group creation unit 12. In the example of FIG. 3, images b1, s1, s2, and b4, which are photographs of A amusement park, and images b2, s3, which are photographs of Park B, are set as grouping target images. As shown by the time axis t in FIG. 3, the captured timing is in the order of images b1, s1, s2, b2, s3, and b4, and all images have a difference in date and time information within a predetermined range. Suppose that In this case, for example, if the difference in position information from the reference image is within 500 m and the difference in position information is within a predetermined range, the image b1 that is the reference image and a location that is 400 m away from the image b1 The images s1 and s2 recorded in step 1 are set to the same image group (group 1). On the other hand, it is determined that the image b2 which is a photograph of the park B 1000 m away from the image b1 is not the group 1. Then, the image b2 is set as a new reference image, and the image b2 and the image s3 recorded at a location 200 m away from the image b2 are the same image group (group 2).
It is said. The image b4 recorded at a location 1000 m away from the image b2 is determined not to be the group 2 and is set as a new group 3 reference image.

  The candidate estimation unit 13 is an image that satisfies a predetermined condition regarding the accuracy of the image management information among a plurality of pieces of image management information (estimation information) acquired by the communication unit 11 and stored in the estimation information storage unit 18. This function estimates management information as a context candidate. Image management information that satisfies a predetermined condition regarding accuracy refers to image management information that can appropriately indicate a user's behavior. The candidate estimation unit 13 estimates a context candidate from each piece of image management information that is estimation information stored in the estimation information storage unit 18 by the communication unit 11. The candidate estimation unit 13 outputs a context derivation request including an estimation result table (described later) associated with the estimated context candidate to the combination unit 14. The candidate estimation unit 13 includes a character recognition estimation unit 13a, an image recognition estimation unit 13b, a POI estimation unit 13c, and an event estimation unit 13d.

  The character recognition estimation unit 13a acquires a character recognition result stored in the estimation information storage unit 18, and estimates a character candidate that is a context candidate based on the character recognition result. Specifically, when the keyword included in the character recognition result is a predetermined keyword (target keyword), the character recognition estimation unit 13a sets the keyword as a character candidate for deriving a user context. To do. The target keyword is a keyword considered to be able to appropriately indicate the user's behavior. For example, a keyword such as “Graduation Ceremony” that is recognized in the image of FIG. 8A indicates an event in which the user participates and can appropriately indicate the user's behavior, and thus is a target keyword. . On the other hand, a keyword or the like indicating a station name such as “Satsukawashiri” recognized in the image of FIG. 8B is not a target keyword because it is difficult to appropriately indicate the user's action. Similarly, when the category included in the character recognition result is a predetermined category (target category), the character recognition estimation unit 13a sets the character recognition result category and keyword as character candidates.

  Moreover, the character recognition estimation part 13a estimates a character candidate from the character recognition result also in the image group unit created by the group creation part 12. Specifically, the character recognition estimating unit 13a sets the character having the largest number as a single character candidate included in the same image group as a character candidate for deriving a group context (described later) of the image group. . Alternatively, all the character candidates for a single image included in the same image group may be listed, and all may be used as character candidates for deriving a group context (described later) of the image group.

  The image recognition estimation unit 13b acquires an image recognition result stored in the estimation information storage unit 18, and estimates an image candidate that is a context candidate based on the image recognition result. Specifically, the image recognition estimation unit 13b acquires labels and scores of a plurality of objects recognized in the same image, rearranges the scores in ascending order, and determines the label with the highest score as the label of the image. In addition, the labels of the top five scores are used as image candidates for deriving the user context. Or the image recognition estimation part 13b is good also considering the label whose score is higher than a predetermined threshold as an image candidate. For example, when the score threshold is 0.9 and the score of an object (label: animal) recognized in the koala photograph in FIG. 9A is 0.92, The label is an image candidate. On the other hand, if the score of the object (label: animal) recognized in the photograph of the lion and crocodile figurine in FIG. 9B is 0.44, the label of the object is not an image candidate. An image for which no image candidate exists is determined to be the others image. As a result of the determination, there may be a case where one or a plurality of image candidates exist in a single image, or a case where no image candidate exists.

  In addition, the image recognition estimation unit 13b estimates context candidates from the image recognition result even in units of image groups created by the group creation unit 12. Specifically, the image recognition estimation unit 13b sets the most image candidates in the same image group as image candidates for deriving a group context (described later) of the image group. Alternatively, the image recognition estimation unit 13b sets the label having the highest score sum in the same image group as an image candidate for deriving a group context (described later) of the image group. Further, the image recognition estimating unit 13b sets the most image candidates in the same image group as image candidates for deriving a group context (described later) of the image group, and when there are the same number of image candidates. The image candidate having a higher total score may be used as the image candidate. The image recognition estimation unit 13b may determine that there is no image candidate for deriving a group context (described later) for an image group that includes the above-described others image for a threshold value or more. For example, if the threshold value of the score is 0.9 and there is an image group consisting of the four images of FIGS. 11A to 11D, the score of FIG. ) Exceeds the threshold, and the scores of the images in FIGS. 11B to 11D do not exceed the threshold. Then, if the threshold of the ratio including the others image is 0.6 (60%), the ratio including the others image is now 0.75 (75%). Is determined that there is no image candidate for deriving a group context (described later). As described above, there are cases where one or a plurality of image candidates exist in image group units, and there are cases where no image candidates exist.

  The POI estimation unit 13c acquires the POI information stored in the estimation information storage unit 18, and estimates a POI candidate that is a context candidate based on the POI information. Specifically, for each of a plurality of POI information for the same image, the POI estimation unit 13c calculates a POI score obtained by dividing the number of POI check-ins by the distance from the POI to the position where the image is recorded, POI information having the highest POI score is set as a POI candidate for deriving a user context. More specifically, the POI estimation unit 13c sets the POI name and POI category of the POI information with the highest POI score as the POI candidate. When there are a plurality of pieces of POI information having the same POI score, the POI estimation unit 13c may use the POI information with the short distance as a POI candidate for the image. Alternatively, the POI estimation unit 13c may use the POI category as a POI candidate when the POI category is a predetermined keyword (target POI keyword). For example, when the target POI keyword includes “baseball field” and does not include “restaurant”, the POI category “baseball field” in the image of FIG. 10A is set as a POI candidate. The POI category “Restaurant” in the image of (b) is not a POI candidate.

  The POI estimation unit 13c also estimates the context candidates from the POI information even in the image group unit created by the group creation unit 12. Specifically, the POI estimation unit 13c estimates the most POI candidates in the same image group as POI candidates for deriving a group context (described later) of the image group. If there are a plurality of POI candidates, the POI estimation unit 13c may estimate a POI candidate having a high total POI score as a POI candidate for deriving a group context (described later) of the image group. .

  The event estimation unit 13d acquires event information stored in the estimation information storage unit 18, and estimates event candidates that are context candidates based on the event information. Specifically, the event estimation unit 13d displays event information in which the information indicating the position matches the position information of the point where the image is recorded and the information indicating the date and time matches the date and time information where the image is recorded. As an event candidate for deriving. The event estimation unit 13d first acquires the POI of the point where the image was recorded from the POI information stored in the estimation information storage unit 18, and recorded the image stored in the image metadata storage unit 16. Get date and time information. Thereafter, the event estimation unit 13d searches the event information stored in the estimation information storage unit 18 for event information that matches the POI and date / time information. If there is, the event name and event category of the event information are found. Are event candidates for the image.

  Further, the event estimation unit 13d estimates context candidates from the event information even in units of image groups created by the group creation unit 12. Specifically, if there is event information whose POI and date / time information match, the event estimation unit 13d sets the event name and event category as event candidates for deriving a group context (described later) of the image group. . Here, when determining whether the date / time information matches on an image group basis, the date / time information is obtained from the date / time information in which the image with the oldest date / time information is recorded among the images included in the image group. It is determined whether or not there is event information that is information indicating the date and time between the recorded date and time information.

  As described above, an estimation result table in which character candidates, image candidates, POI candidates, and event candidates that are context candidates are associated is output from the candidate estimation unit 13 to the combination unit 14. FIG. 4 is a table showing the estimation result table 180. In the estimation result table 180, an image ID, an image candidate, a POI candidate, position information, date / time information, a character candidate, a municipality name, an event candidate, and a file format are associated with each other.

  The image ID is an identifier that uniquely identifies the image. The image candidate, the POI candidate, the character candidate, and the event candidate are context candidates estimated by the above-described image recognition estimation unit 13b, POI estimation unit 13c, character recognition estimation unit 13a, and event estimation unit 13d, respectively. For example, the image indicated by the image ID “P0001” is “B park (park)” for the POI candidate. The “B park” is a POI name, and “(park)” is a POI category. The position information is the same information as the position information of the image stored in the position information storage unit 17. The date / time information is the date / time information of the image stored in the image metadata storage unit 16. The municipality name is the name of the municipality derived based on the location information. The file format is a file format of the image stored in the image metadata storage unit 16. Note that the estimation result table 180 is an estimation result table related to derivation of a user context of a single image, and therefore does not include group-unit context candidates.

  The combination unit 14 is a function for deriving a user context indicating a user's action related to the image by combining a plurality of pieces of image management information acquired by the communication unit 11. More specifically, the combination unit 14 derives a user context by combining the image management information that has been made a context candidate by the candidate estimation unit 13 among the plurality of image management information. Then, the combination unit 14 outputs a category assignment request including a tagging result table (described later) in which a tag corresponding to the derived user context is associated with an image to the category assignment unit 15. A tag is a specific information element related to an image, and is information serving as a search key for the image. The derived user context and tag may be the same keyword, or may be different corresponding keywords.

  The combination unit 14 identifies a context candidate associated with the image by referring to an estimation result table (for example, see FIG. 4). For example, for the image with the image ID “P0001” in FIG. 4, “park” as the image candidate, “B park (park)” as the POI candidate, and “athletic meet” as the character candidate are specified as the context candidates. The Then, by combining these context candidates, a user context with “park” is derived. Such derivation of the user context combining the context candidates may be derived, for example, by using a table in which the combination of context candidates (a plurality of context candidates) is associated with the user context. By referring to such a table, a user context can be uniquely determined from a combination of context candidates. In addition, the tag “park” is given according to the derived user context “park”. A tag is given based on a table in which a user context and a tag are associated, for example.

  Further, the combination unit 14 derives a group context indicating the user's action for each image group grouped by the group creation unit 12 based on the image management information of the images included in the image group. More specifically, the combination unit 14 derives a group context by combining image management information that is a context candidate (character candidate, image candidate, POI candidate, and event candidate) in units of image groups. Then, the combination unit 14 outputs a category assignment request including a tagging management table in which a tag corresponding to the derived group context is associated with an image group to the category assignment unit 15.

  The category assigning unit 15 is a function for associating an image with a category in which one or a plurality of user contexts are defined by a higher concept. The category assigning unit 15 associates a category with an image in the tagging management table input from the combination unit 14. The category assigning unit 15 derives a category from the user context based on, for example, a table in which one category is associated with a plurality of user contexts. The category assigning unit 15 may derive a plurality of categories for an image associated with a plurality of user contexts.

  FIG. 12 is a diagram for explaining an example of category assignment. For example, all the images in FIGS. 12A to 12C are associated with user contexts “B park” and “park”. The category assigning unit 15 associates the “outing” category shown in FIG. 12D with these images. In addition, the image of FIG. 12B is associated with a user context “lunch”. The category assigning unit 15 associates the “meal” category shown in FIG.

  The category assigning unit 15 associates a category that defines one or more group contexts with a superordinate concept with an image group. The category assigning unit 15 associates the category with the image group in the tagging management table input from the combination unit 14.

  FIG. 5 is a table showing the tagging result table 190. As shown in FIG. 5, in the tagging result table 190, the image ID, the tagging result of the single image, the category of the single image, the image candidate, the POI candidate, the character candidate, the municipality name, the event Candidates are associated with file formats. In the tagging result table 190 shown in FIG. 5, only the context candidates (character candidates, image candidates, POI candidates, and event candidates) for a single image are stored, but tagging and category assignment are performed in units of image groups. In the tagging result table, each context candidate for each image group is also stored (described later).

  As described above, the tagging device 10 performs tagging on an image basis and tagging on an image group basis. Hereinafter, with reference to FIGS. 6 and 7, an example of tagging in units of images and tagging in units of image groups will be described. FIG. 6 is an explanatory diagram of an example of performing tagging with a single image. FIG. 7 is an explanatory diagram of an example of performing tagging in units of image groups.

  6A is a diagram illustrating an example of an image, and FIG. 6B is a table illustrating an estimation result table 181 related to the image (image ID: P0001) illustrated in FIG. 6A. ) Is a table showing a tagging result table 191 related to the image (image ID: P0001) shown in FIG.

  As shown in the estimation result table 181 in FIG. 6B, for the image with the image ID “P0001”, “park” is associated with the image candidate and “B park (park)” is associated with the POI candidate. Yes. Then, the tagging result table 191 in FIG. 6C is derived according to the estimation result table 181. As shown in the tagging result table 191 in FIG. 6C, for the image with the image ID “P0001”, “Park” is the tagging result of the single image, and “Outing” is the category of the single image, respectively. Associated. The tagging result “park” of the image alone corresponds to the user context “park” derived based on the above-described context candidates “park” and “B park (park)”. Further, “outing” which is a category of the image alone corresponds to the above-described user context “park”.

  FIGS. 7A to 7C are diagrams showing examples of images, and FIG. 7D relates to the images (image IDs: P0001 to P0003) shown in FIGS. 7A to 7C. 7E is a table showing an estimation result table 182, and FIG. 7E is a table showing a tagging result table 192 relating to images (image IDs: P0001 to P0003) shown in FIGS. 7A to 7C. The images with image IDs P0001 to P0003 are included in the same image group (group ID: G0001). 7 (d) and 7 (e), only the context candidates for the single image are described, but in reality, the context candidates for each image group are also stored in these tables. Yes.

  As shown in the estimation result table 182 in FIG. 7D, for the images with the image IDs “P0001” and “P0003”, “park” is the image candidate, and “B park (park)” is the POI candidate, Associated. For the image with the image ID “P0002”, “meal” is associated with the image candidate, and “B park (park)” is associated with the POI candidate.

  In the estimation result table 182, “park”, which is the most image candidate in the image group, is set as an image candidate for deriving the group context of the image group. In the estimation result table 182, “B park (park)” which is the most POI candidate in the image group is set as a POI candidate for deriving the group context of the image group. Image candidates and POI candidates for deriving the group context of the image group are not described in FIG. 7D, but are actually stored in the estimation result table 182.

  The tagging result table 192 shown in FIG. 7E is derived according to the estimation result table 182. As shown in the tagging result table 192 of FIG. 7E, for the images with the image IDs “P0001” and “P0003”, “park” as the single image tagging result and “outing” as the single image category. Are associated with each other. The tagging result “park” of the image alone corresponds to the user context “park” derived based on the above-described context candidates “park” and “B park (park)”. “Odekake” which is a category of the single image corresponds to the user context “park” described above. For the image with the image ID “P0002”, “lunch” is associated with the tagging result of the single image, and “meal” is associated with the category of the single image. The tagging result “lunch” of the image alone corresponds to the user context “park” derived based on the above-described context candidates “meal” and “B park (park)”. The category “meal” of the single image corresponds to the above-described user context “lunch”.

  Furthermore, as shown in the tagging result table 192 of FIG. 7 (e), tags and categories are also associated with each image group including each image. As a result of tagging the image group, “park” “Outing” is derived as the category of the image group. The tagging result “park” of the image group corresponds to the group context “park” derived based on the above-mentioned image group unit context candidates “park” and “B park (park)”. “Odekake” which is the category of the image group corresponds to the group context “park” described above.

  For the images with the image IDs “P0001” and “P0003”, the tagging result “park” for the single image and the tagging result “park” for each image group overlap, and the category of the single image “Outing” and the category “outing” for each image group overlap. In this way, when the estimation result of the single image and the estimation result of the image group unit overlap, as shown in FIG. 7A and FIG. Display is made (the broken line portion in FIGS. 7A and 7C is a non-display location).

  Next, tagging processing by the tagging device 10 will be described with reference to FIG. FIG. 13 is a flowchart showing the processing of the tagging device 10.

  In the tagging process of the tagging device 10, first, estimation information used for context estimation is acquired by the communication unit 11 (step S1). Specifically, the communication unit 11 receives the POI information (image management information) of the image from the POI conversion device 40, the character recognition result (image management information) from the character recognition device 50, and the image recognition result (image management information) from the image recognition device 60. Image management information) and event information (image management information) from the event estimation device 70 as information for estimation. The communication unit 11 stores the estimation information in the estimation information storage unit 18.

  Subsequently, POI estimation is performed by the POI estimation unit 13c (step S2), image recognition estimation is performed by the image recognition estimation unit 13b (step S3), and character recognition estimation is performed by the character recognition estimation unit 13a (step S4). ). Although not shown in FIG. 13, event estimation may be performed by the event estimation unit 13d. And the estimation result containing the context candidate of an image unit is output to the combination part 14 by the candidate estimation part 13 (step S5).

  Here, the detailed processing of the character recognition estimation unit 13a, the image recognition estimation unit 13b, the POI estimation unit 13c, and the event estimation unit 13d, which are the components of the candidate estimation unit 13 described above, will be described with reference to FIGS. I will explain. FIG. 14 is a flowchart showing details of processing of the character recognition estimating unit 13a. FIG. 15 is a flowchart showing details of processing of the image recognition estimation unit 13b. FIG. 16 is a flowchart showing details of the processing of the POI estimation unit 13c. FIG. 17 is a flowchart showing details of processing of the event estimation unit 13d.

  As shown in FIG. 14, the character recognition estimation unit 13a obtains the character recognition result stored in the estimation information storage unit 18 (step S70). Then, the character recognition estimating unit 13a searches for a predetermined category, and determines whether or not the category associated with the keyword determined in the character recognition result is the predetermined category (step S71). ). In the case of the predetermined category, the keyword and category determined in the character recognition result are set as character candidates. Then, the character recognition estimating unit 13a searches for a predetermined keyword, and determines whether or not the keyword determined in the character recognition result is a predetermined keyword (step S72). In the case of the predetermined keyword, the keyword determined in the character recognition result is acquired as a character candidate.

  As shown in FIG. 15, in the image recognition estimation unit 13b, the label and score of the image recognition result stored in the estimation information storage unit 18 are acquired (step S60). Then, the image recognition estimation unit 13b rearranges the labels of the image recognition results in ascending order of scores (step S61), and determines the label with the highest score as the image label (step S62). Finally, the top five image recognition results of the score are acquired as image candidates by the image recognition estimation unit 13b (step S63).

  As shown in FIG. 16, the POI estimation unit 13c acquires the POI information stored in the estimation information storage unit 18 (step S50). Then, the POI estimation unit 13c calculates a POI score obtained by dividing the number of POI check-ins by the distance to the POI (step S51). The POI estimation unit 13c estimates the POI with the highest score as a POI candidate for the image (step S52). More specifically, the POI estimation unit 13c acquires the POI name and POI category of the POI information with the highest POI score as a POI candidate (step S53).

  As shown in FIG. 17, the event estimation unit 13d acquires the POI information stored in the estimation information storage unit 18 (step S80). Then, the event estimation unit 13d acquires the date / time information when the image is recorded from the image metadata storage unit 16, and the event whose POI and date / time information match the event information stored in the estimation information storage unit 18 It is determined whether there is information (step S81). If there is no matching event information in S81, the process ends. On the other hand, if there is matching event information in S81, the event estimation unit 13d specifies event information associated with the image (step S82), and the event name and event category of the event information indicate the event information of the image. Obtained as a candidate (step S83). The above is the process of each component of the candidate estimation unit 13.

  Returning to FIG. 13, when the process of S5 is completed, the combination unit 14 combines the image management information that has been made the context candidate by the candidate estimation unit 13, and estimates (derived) the user context in units of images (step S6). . And the tagging according to a user context is performed by the combination part 14 (step S7). Specifically, the combination unit 14 outputs a category assignment request including a tagging result table in which a tag corresponding to the derived user context is associated with an image to the category assignment unit 15. And in the category provision part 15, the category which prescribed | regulated the one or some user context by the high-order concept is linked | related with an image. The processing so far is tagging processing in units of images. Hereinafter, tagging processing in units of image groups will be described.

  In the tagging process for each image group of the tagging apparatus 10, first, an image group is created by the group creation unit 12 (step S8). Specifically, the group creating unit 12 creates an image group in which a plurality of images in which the position information of the point where the image is recorded and the date and time information when the image is recorded are within a predetermined range are grouped.

  Subsequently, context candidates for each image group are estimated by each configuration of the candidate estimation unit 13. That is, the POI estimation unit 13c identifies the most POI candidates in the same image group as POI candidates for deriving the group context of the image group (step S9). Further, the event estimation unit 13d identifies event information having the same POI and date / time information as an event candidate for deriving the group context of the image group (step S10). Further, the image recognition estimation unit 13b identifies the label having the highest total score in the same image group as an image candidate for deriving the group context of the image group (step S11). Further, the character recognition estimating unit 13a lists all the character candidates included in the same image group and specifies them as character candidates for deriving the group context of the image group (step S12).

  Then, the combination management unit 14 combines the image management information set as context candidates for deriving the group context by the candidate estimation unit 13, and estimates (derives) the group context (step S13). Thereafter, the combination unit 14 performs tagging in accordance with the group context (step S14), and the category assigning unit 15 associates a category that defines one or a plurality of group contexts with a higher concept (step S15). ).

  Next, the effect of the tagging apparatus 10 according to the embodiment will be described.

  The tagging apparatus 10 according to the present embodiment combines a communication unit 11 that acquires a plurality of image management information indicating time, place, or event related to an image and a plurality of image management information acquired by the communication unit 11. And a combination unit 14 for deriving a user context indicating the user's behavior related to the image.

  In the tagging device 10, a plurality of pieces of image management information, that is, information indicating when, where, and what is being combined is combined to derive a user context indicating the user's action related to the image. By using such a user context for image search, for example, compared to the case where image search is performed by associating information such as date and place with an image alone, it is more consistent with the image. Image retrieval can be performed on the basis of information. As a result, the accuracy of the image search can be improved.

  The tagging apparatus 10 further includes a group creating unit 12 that creates an image group in which a plurality of images in which the position information of the point where the image is recorded and the date and time information when the image is recorded are within a predetermined range are combined. The unit 14 derives, for each image group, a group context indicating the user's behavior based on the image management information of the images included in the image group.

  By deriving a group context for each group in consideration of the position and date and time, images related to a common event (event) can be easily obtained as the same search result. In other words, for example, if you have lunch in the middle of an event, it is possible to suppress another search result (output separately) because the location is a little different even though it is a common event. can do. In addition, for example, even when there is a misrecognition or estimation error in the user context of a single image, some errors can be ignored by considering the user context of a plurality of images, so the accuracy of image search is improved. It can be improved further.

  The tagging apparatus 10 further includes a candidate estimation unit 13 that estimates, as a context candidate, image management information that satisfies a predetermined condition regarding the accuracy of the image management information among the plurality of image management information acquired by the communication unit 11. The combining unit 14 derives a user context and a group context by combining the image management information that is selected as a context candidate by the candidate estimation unit 13 among the plurality of image management information.

  Rather than randomly selecting image management information to be combined, the accuracy of image search can be further improved by using information with high accuracy as image management information.

  The plurality of image management information includes a plurality of image recognition results regarding the image, and the candidate estimation unit 13 includes an image recognition estimation unit 13b that estimates an image candidate that is a context candidate based on the image recognition result regarding the image. The image recognition estimation unit 13b estimates information indicating an object whose score indicating the degree of similarity is equal to or greater than a predetermined threshold in the image recognition result regarding the image as an image candidate for deriving the user context, and within the same image group. Are estimated as image candidates for deriving the group context of the image group.

  By using information having a high score indicating the degree of similarity as an image candidate, a user context can be derived by combining information having high accuracy as image management information, and the accuracy of image search can be further improved. That is, for example, it is possible to prevent a user context from being derived from an image (photograph) that has been blurred during imaging as an image candidate. Further, by setting the most image candidates in the same image group as image candidates for deriving the group context, the group context can be derived using the representative image candidates in the image group. The accuracy of the image search can be further improved. In addition, in the estimation for each image group, it is possible to consider an image having a low score for a single image by determining image candidates as described above, instead of simply judging by a score.

  The plurality of image management information includes a plurality of pieces of POI information related to images, and the candidate estimation unit 13 includes a POI estimation unit 13c that estimates POI candidates that are context candidates based on the POI information about images, and performs POI estimation. The unit 13c calculates, for each of the plurality of POI information, a POI score obtained by dividing the number of POI check-ins by the distance from the POI to the position where the image is recorded, and the POI information having the highest POI score is obtained as the user context Are estimated as POI candidates for deriving and the most POI candidates in the same image group are estimated as POI candidates for deriving the group context of the image group.

  By considering the number of POI check-ins, it is possible to easily estimate POI information that is likely to be visited by a user as a POI candidate. Thereby, erroneous estimation can be suppressed. Further, in the estimation in units of image groups, the POI for deriving the group context from the POI information that is likely to be visited by the user because the most POI candidates are the POI candidates for deriving the group context. Can be a candidate.

  The plurality of image management information includes event information specified by information indicating a position and information indicating a date and time, and the candidate estimation unit 13 estimates an event candidate that is a context candidate based on the event information. The event estimation unit 13d receives event information in which the information indicating the position matches the position information of the point where the image is recorded and the information indicating the date and time matches the date and time information where the image is recorded. Estimated as an event candidate for deriving the context, and the information indicating the date and time coincides with the position information of the point where the image included in the image group is recorded, and the date and time information is the most among the images included in the image group. Event information included between the date and time information when the old image is recorded and the date and time information when the image with the newest date and time information is recorded. It estimated as the event candidate for deriving a group context of the loop.

  Thereby, the event candidate considered that the user was visiting can be appropriately estimated in consideration of the position and the date and time. That is, the accuracy of image search can be further improved.

  The plurality of image management information includes a character recognition result regarding the image, and the candidate estimation unit 13 includes a character recognition estimation unit 13a that estimates a character candidate that is a context candidate based on the character recognition result regarding the image, The character recognition estimation unit 13a estimates a predetermined character among characters in the character recognition result as a character candidate for deriving a user context, and determines the most character candidates in the same image group as the image group. It is estimated as a character candidate for deriving a group context.

  By using only predetermined characters as character candidates, it is possible to set only appropriate characters as keywords indicating user behavior as character candidates. In addition, by setting the largest number of character candidates in the same image group as character candidates for deriving a group context, a group context can be derived using representative character candidates in an image group. The accuracy of the image search can be further improved.

  The combination unit associates a tag according to the user context with the image and associates a tag according to the group context with the image group. Thereby, compared with the case where only one recognition result is conventionally used as a recognition result associated with an image, tagging can be performed with reduced ambiguity and reduced abstraction, and the accuracy of image search is improved. Can be improved.

  The tagging apparatus 10 further includes a category assigning unit 15 that associates one or more user contexts with a category defined by a superordinate concept with an image, and associates one or more group contexts with a superordinate concept with an image group. . By associating a category with an image and an image group, it becomes possible to search for an image with a higher concept than a user context and a group context.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. For example, a plurality of image groups may constitute a new image group (album) that is further summarized. That is, the group creation unit 12 sets a plurality of image groups in which the date and time information of the included images is within a predetermined range among the created image groups as a common new image group, and the combination unit 14 creates a new image group. The group context of the image group having the largest number of images among the plurality of image groups included in the group may be set as the group context of the new image group.

  FIG. 18 is an explanatory diagram of album creation of the tagging device according to the modification. As shown in the table 200 in FIG. 18A, a plurality of image groups indicated by different group IDs “G0001” and “G0002” are grouped into a new common image group (album ID “A0001”). . Assuming that the group context of the image group with the group ID “G0001” is “B park” and the group context of the image group with the group ID “G0002” is “B aquarium”, FIG. Thus, the image group with the group ID “G0001” has one more image, so the album title (album group context) is B park.

  Thereby, image groups with similar date and time information can be further collected, and similar image search results can be confirmed more easily. In addition, since the group context of the image group having the largest number of images is set as the group context of the new image group, it is possible to improve the accuracy of the image search for the new image group.

  DESCRIPTION OF SYMBOLS 10 ... Tagging apparatus, 11 ... Communication part (acquisition part), 12 ... Group creation part, 13 ... Candidate estimation part, 13 ... Candidate estimation part, 13a ... Character recognition estimation part, 13b ... Image recognition estimation part, 13c ... POI Estimation part, 13d ... event estimation part, 14 ... combination part, 15 ... category assignment part.

Claims (10)

An acquisition unit that acquires a plurality of image management information indicating the time, place, or event relating to the image;
An information processing apparatus comprising: a combination unit that derives a user context indicating a user action related to the image by combining a plurality of pieces of image management information acquired by the acquisition unit. A group creating unit that creates an image group in which a plurality of images in which the position information of the point where the image is recorded and the date and time information where the image is recorded are within a predetermined range are grouped;
The information processing apparatus according to claim 1, wherein the combination unit derives, for each image group, a group context indicating a user action based on the image management information of images included in the image group. A candidate estimation unit that estimates, as a context candidate, image management information that satisfies a predetermined condition regarding accuracy of the image management information among the plurality of image management information acquired by the acquisition unit;
3. The information according to claim 2, wherein the combination unit derives the user context and the group context by combining the image management information that is selected as the context candidate by the candidate estimation unit among the plurality of image management information. Processing equipment. The plurality of image management information includes a plurality of image recognition results regarding the image,
The candidate estimation unit includes an image recognition estimation unit that estimates an image candidate that is the context candidate based on an image recognition result related to the image;
The image recognition estimation unit
In the image recognition result regarding the image, information indicating an object whose score indicating the degree of similarity is equal to or greater than a predetermined threshold is estimated as the image candidate for deriving the user context,
The information processing apparatus according to claim 3, wherein the most image candidates in the same image group are estimated as the image candidates for deriving the group context of the image group. The plurality of image management information includes a plurality of POI information related to images,
The candidate estimation unit includes a POI estimation unit that estimates a POI candidate that is the context candidate based on POI information regarding the image,
The POI estimation unit
For each of the plurality of POI information, a POI score obtained by dividing the number of POI check-ins by the distance from the POI to the position where the image was recorded is calculated. Estimate as said POI candidate for derivation,
The information processing apparatus according to claim 3 or 4, wherein the most POI candidates in the same image group are estimated as the POI candidates for deriving the group context of the image group. The plurality of image management information includes event information specified by information indicating a position and information indicating a date and time,
The candidate estimation unit includes an event estimation unit that estimates an event candidate that is the context candidate based on the event information,
The event estimation unit
For deriving the user context, the event information in which the information indicating the position matches the position information of the point where the image is recorded and the information indicating the date and time matches the date and time information where the image is recorded. Estimated as the event candidate,
The information that matches the position information of the point where the image included in the image group is recorded, and the information indicating the date is from the date information that recorded the image with the oldest date information among the images included in the image group. The event information included between the date and time information recording the image with the newest date and time information is estimated as the event candidate for deriving the group context of the image group. The information processing apparatus according to any one of the above. The plurality of image management information includes a character recognition result regarding the image,
The candidate estimation unit includes a character recognition estimation unit that estimates a character candidate that is the context candidate based on a character recognition result regarding the image,
The character recognition estimation unit
Estimating a predetermined character among the characters in the character recognition result as the character candidate for deriving the user context,
The information processing apparatus according to claim 3, wherein the largest number of character candidates in the same image group is estimated as the character candidates for deriving the group context of the image group. The combination part is
Associate a tag according to the user context with the image,
The information processing apparatus according to claim 2, wherein a tag corresponding to the group context is associated with the image group. Associating a category that defines one or more user contexts in a generic concept with the image;
The information processing apparatus according to claim 2, further comprising a category assigning unit that associates a category in which one or a plurality of the group contexts are defined by a superordinate concept with the image group. The group creation unit, among the created image groups, a plurality of the image groups in which the date and time information of included images is within a predetermined range as a common new image group,
The combination unit sets the group context of the image group having the largest number of images among the plurality of image groups included in the new image group as the group context of the new image group. The information processing apparatus according to any one of the above.

Images