JP5834950B2 - Photo series similarity calculation device, method, and program - Google Patents

Description

  The disclosed technology relates to a photographic sequence similarity calculation apparatus, method, and program.

  For example, in a photo sharing service that allows users to share photographic images uploaded from multiple users on the server and share the stored photographic images, etc. You may want to browse photos.

  In this case, if a tag such as an event name is attached to each photo image, it becomes easy to view photos taken by others who participated in the same event as the event in which they participated. Is complicated.

  Therefore, based on the probability that the same object as the object included in the photographic image held by the person will appear in the photographic image held by another person, the similarity score between the two is calculated, and the two are associated based on the calculated similarity score. There is technology.

Special table 2009-533725

David G. Lowe, "Distinctive image features from scale-invariant keypoints", International Journal of Computer Vision, 60, 2 (2004), pp. 91-110. H.Bay, T.Tuytelaars, and L.V.Gool, "SURF: Speed Up Robust Features", Proc. Of Int. Conf. Of ECCV, (2006)

  The above technique is a technique for recognizing an object (person, building, landscape, etc.) included in each photographic image and determining that the similarity between photographic images including the same object is high. In order to recognize an object, for example, a person (individual), a building (Tokyo Tower, Empire State Building, etc.), a landscape (Mt. Fuji, Niagara Falls, etc.), and other reference data for identifying other objects (Model data) is required. For this reason, in order to determine similarity with high accuracy, a huge amount of model data is required.

  In recent years, research institutes such as universities have been making efforts to enhance image data sets with the aim of applying them to object recognition. For example, the experimental data set SIMPLIcity collection of PENN STATE UNIVERSITY is composed of a total of 1000 images in 10 categories, but this amount of model data is not sufficient to accurately determine similarity. .

  For example, in the photo sharing service, when attempting to group and share for each group of images taken of the same wedding, individual model data of all the venues and chapels used by users of the photo sharing site is required. That is, when users of a photo sharing service exist on a nationwide scale, it is difficult to realize this because model data of all wedding venues in the country is required.

  In the above technique, the objects included in the photographic image are classified into categories (people, famous buildings, pets, landscapes, celebrities, vehicles, non-famous buildings, etc.), and the importance of contributing to the similarity of the photographic images is determined. Set for each category. For example, 0.9 for people, 0.1 for famous buildings, 0.3 for non-famous buildings, and so on.

  However, it is difficult to set an appropriate level of importance in this way, and how to determine the appropriate level for each category and how to set the appropriate level for each category is described above. It is not disclosed in the technology.

  In addition, in order to automatically detect a photographic image taken by another person who participated in the same event as the event that he / she participated in, it is necessary to detect a partial component that clearly represents the characteristics of the event from the photographic image obtained by capturing the event. But it varies from event to event. For example, if it is a wedding of a certain person, it is the face of the person, clothes at that time, unique decorations at the venue, etc., and in the case of an exhibition or the like, it is a product exhibited. Therefore, it is extremely difficult to manually determine which partial component should be emphasized, which is a very complicated operation.

  An object of the disclosed technique is to accurately calculate the similarity of a photographic sequence without requiring an operation for setting the importance of partial components included in a photographic image.

  In the disclosed technology, the photo sequence generation unit receives the photo image from a photo image storage unit in which a plurality of photo images associated with shooting time information indicating a shooting time and user information indicating a corresponding user are stored. Acquire and generate a plurality of photo sequences for each user. The section photograph image set generation unit divides a period in which the photograph image is taken into a plurality of sections at arbitrary time intervals. Further, the section photograph image set generation unit assigns each of the photograph images acquired by the photograph series generation unit to any one of the plurality of sections based on the shooting time information. A section photograph image set is generated. The partial component extraction unit extracts a partial component included in the photographic image acquired by the photographic sequence generation unit. The frequent occurrence importance degree calculation unit calculates the frequent occurrence importance degree of the partial constituent elements in the photo series based on the appearance frequency of the partial constituent elements included in the photographic image belonging to the photo series. It calculates for every and every said partial component. The simultaneity importance calculation unit is configured to determine the simultaneity of the partial constituent elements based on the appearance frequency of the partial constituent elements included in the photographic images belonging to the section photographic image set generated by the section photographic image set generation section. The importance is calculated for each partial component. The importance composition unit calculates the importance of the subcomponents for the photo sequence based on the frequentity importance calculated by the frequentity importance calculation unit and the simultaneity importance calculated by the simultaneity importance calculation unit. Calculation is performed for each of a plurality of photograph series and each partial component. The similarity calculation unit calculates the similarity between the plurality of photo series based on the importance for each of the plurality of photo series and the partial components calculated by the importance synthesis unit.

  The disclosed technique has an effect that the similarity of a photographic sequence can be calculated with high accuracy without requiring an operation of setting the importance of a partial component included in a photographic image.

It is a functional block diagram of the similarity calculation apparatus of a photograph series. It is a functional block diagram of a partial component extraction part. FIG. 3 is a schematic block diagram of a computer that functions as a photographic sequence similarity calculation device. It is a flowchart which shows an example of the similarity calculation process of a photograph series. It is a figure for demonstrating the photograph series of the photograph which each user image | photographed. It is a figure which shows an example of a photograph management table. It is an image figure at the time of dividing | segmenting a designated period by a 10-minute space | interval. It is an image figure which shows the kind of photograph which each user image | photographed. (A) is a diagram illustrating an original image, and (B) is a diagram illustrating an image after a feature point extraction process is performed on the original image. It is a figure for demonstrating multidimensional feature-value calculation. It is a figure for demonstrating a clustering process. It is a figure for demonstrating a partial component extraction process. It is a figure which shows an example of a photograph management table. It is a figure which shows an example of a frequentity importance table. It is a figure for demonstrating calculation of frequent appearance importance. It is a figure which shows an example of a simultaneity importance table. It is a figure for demonstrating calculation of simultaneity importance. It is a figure for demonstrating calculation of importance. It is a figure which shows an example of an importance table. It is a figure for demonstrating similarity calculation. It is a figure which shows an example of a similarity table. It is a figure for demonstrating the service to which a similarity calculation process is applied. It is a figure for demonstrating the modification of a feature point extraction process.

  Hereinafter, an example of an embodiment of the disclosed technology will be described in detail with reference to the drawings.

  FIG. 1 shows a photographic sequence similarity calculation apparatus 10 according to this embodiment. The photo sequence similarity calculation device 10 obtains photo images taken within a specified period from a photo image storage unit in which photo images of a plurality of users are stored, and generates a photo sequence for each user. It is an apparatus for calculating the similarity between series.

  As shown in FIG. 1, a photographic sequence similarity calculation device 10 includes a photographic sequence generation unit 12, a section photographic image set generation unit 14, a partial component extraction unit 16, a frequentity importance calculation unit 18, and a simultaneity importance level. A calculation unit 20, an importance synthesis unit 22, and a similarity calculation unit 24 are provided. A photographic image storage unit 30 is connected to the photographic sequence generation unit 12. In addition, the photo sequence generation unit 12, the section photo image set generation unit 14, the partial component extraction unit 16, the frequentity importance calculation unit 18, the simultaneity importance calculation unit 20, the importance synthesis unit 22, and the similarity calculation unit A table storage unit 32 is connected to 24. In the present embodiment, the photo sequence similarity calculation device 10 is provided in a server 40 that stores photo images uploaded from a plurality of users and provides various photo sharing services such as a service for searching for similar photo sequences. The case will be described. The server 40 includes a photographic sequence similarity calculation device 10, a photographic image storage unit 30, and a table storage unit 32.

  The photographic image storage unit 30 stores a plurality of photographic images associated with shooting time information indicating the shooting time and user information indicating the user.

  The photographic sequence generation unit 12 is connected to the photographic image storage unit 30, acquires photographic images taken within a specified period from the photographic image storage unit 30, and generates a plurality of photographic sequences for each user. For example, the designated period may be designated by the user, or a predetermined period may be designated automatically.

  The designated period is set to, for example, a time suitable for event identification (for example, about half a day or one day). In this embodiment, as an example, a photo sequence refers to a set of photo images taken by the same user within a specified period. In addition, the photographic image is digital image data, and in the present embodiment, for example, a case of a still image taken by a photographing device such as a digital camera will be described as an example. The image of Further, it may be a read image obtained by reading a photograph with an image reading device such as a scanner.

  The section photograph image set generation unit 14 divides the specified period into a plurality of sections at regular time intervals, and each of the photograph images acquired by the photograph series generation unit 12 is used as shooting time information associated with each photograph image. Based on one of a plurality of sections based on this. Thereby, a section photograph image set is generated for each of a plurality of sections.

  The partial component extraction unit 16 extracts partial components from all of the photographic images acquired by the photo sequence generation unit 12. The partial components are classified according to characteristics including, for example, at least one of color and shape. As shown in FIG. 2, the partial component extraction unit 16 includes a feature point extraction unit 16A, a multidimensional feature quantity calculation unit 16B, and a clustering unit 16C.

  The feature point extraction unit 16A extracts a plurality of feature points from the photographic image. The multidimensional feature amount calculation unit 16B calculates multidimensional feature amounts of a plurality of feature points extracted by the feature point extraction unit 16A. The clustering unit 16C clusters a plurality of feature points on the multidimensional space represented by the multidimensional feature amount calculated by the multidimensional feature amount calculation unit 16B, and extracts the clustered feature points as partial constituent elements.

  The frequent occurrence importance calculation unit 18 calculates the frequent occurrence importance of the partial constituent elements in the photo series based on the appearance frequency of the partial constituent elements included in a photographic image belonging to a certain photo series. This is performed for each series and each partial component. The frequentity importance calculating unit 18 increases the frequency of occurrence of the partial component in the specific photo sequence and decreases the frequency of the partial component in the photo sequence other than the specific photo sequence. Calculation is performed so that the frequency of importance of components increases. This calculation method is an example of a method for calculating the frequency of importance.

  The simultaneity importance calculation unit 20 calculates the simultaneity importance of the partial constituent elements based on the appearance frequency of the partial constituent elements included in the photographic images belonging to the section photograph image set generated by the section photograph image set generation section 14. Calculate for each partial component. The simultaneity importance calculating unit 20 is configured to increase the appearance frequency of the partial component in the specific section photo image set and to decrease the appearance frequency of the partial component in the section photo image set other than the specific section photo image set. Calculation is performed so that the simultaneity importance of the constituent elements increases. This calculation method is an example of a method for calculating the simultaneity importance.

  Based on the frequentity importance calculated by the frequentity importance calculation unit 18 and the simultaneity importance calculated by the simultaneity importance calculation unit 20, the importance synthesis unit 22 applies to the photo sequence of the partial constituent elements. The degree of importance is calculated for each of a plurality of photograph series and for each partial component. In addition, as an example, the importance degree synthesis unit 22 sets the frequentity importance degree to a value obtained by multiplying the value obtained by weighting the simultaneity importance degree by a constant that defines the degree of influence of the simultaneity importance degree on the importance degree and the frequentity importance degree. The added value is calculated as the importance.

  The similarity calculation unit 24 calculates the similarity between a plurality of photo series based on the importance for each of a plurality of photo series and each partial component calculated by the importance synthesis unit 22.

  The photographic sequence similarity calculation device 10 can be realized by a computer 70 shown in FIG. 3, for example. The computer 70 includes a CPU 72, a memory 74, and a nonvolatile storage unit 76, which are connected to each other via a bus 78.

  The storage unit 76 can be realized by a flash memory, an HDD (Hard Disk Drive), or the like. The storage unit 76 as a recording medium stores a photo sequence similarity calculation program 80 for causing the computer 70 to function as the photo sequence similarity calculation device 10.

  The CPU 72 reads out the photo sequence similarity calculation program 80 from the storage unit 76 and expands it in the memory 74, and sequentially executes the processes of the photo sequence similarity calculation program 80.

  The similarity calculation program 80 includes a photographic image acquisition process 82, a section photographic image set generation process 84, a partial component extraction process 86, a frequent importance calculation process 88, a simultaneity importance calculation process 90, an importance synthesis process 92, And a similarity calculation process 94.

  The CPU 72 operates as the photo sequence generation unit 12 illustrated in FIG. 1 by executing the photo image acquisition process 82. The CPU 72 operates as the section photograph image set generation unit 14 shown in FIG. 1 by executing the section photograph image set generation process 84. The CPU 72 operates as the partial component extraction unit 16 shown in FIG. 1 by executing the partial component extraction process 86. The CPU 72 operates as the frequent occurrence importance degree calculation unit 18 shown in FIG. 1 by executing the frequent occurrence importance degree calculation process 88. The CPU 72 operates as the simultaneity importance calculation unit 20 shown in FIG. 1 by executing the simultaneity importance calculation process 90. The CPU 72 operates as the importance synthesis unit 22 shown in FIG. 1 by executing the importance synthesis process 92. The CPU 72 operates as the similarity calculation unit 24 illustrated in FIG. 1 by executing the similarity calculation process 94.

  As a result, the computer 70 that has executed the photo sequence similarity calculation program 80 functions as the photo sequence similarity calculation device 10. The photo sequence similarity calculation program 80 is an example of a photo sequence similarity calculation program in the disclosed technology.

  Next, as a function of the present embodiment, a similarity calculation process for a photo sequence will be described.

  A user of the photo sharing service uploads, to the server 40, a photographic image of a photograph taken by himself / herself at an event such as a wedding or sightseeing. The uploaded photographic image is stored in the photographic image storage unit 30. The photo image storage unit 30 stores photos of a plurality of users who use the photo sharing service.

  The user uploads a photographic image to the server 40 via a network such as the Internet using e-mail, a web application, a dedicated application, or the like. Each photographic image is stored in the photographic image storage unit 30 in association with photographing time information representing photographing time and user information representing a user. As a storage method, for example, a storage folder may be provided for each user, or a method such as RDB (Relational Data Base) may be used. In addition, since the shooting time can be automatically written in Exif information, which is a standard method for adding various information related to shooting to a photographic image, this information may be held as it is.

  In the photograph series similarity calculation apparatus 10 according to the present embodiment, the similarity calculation processing shown in FIG. 4 is executed periodically (for example, once a week, once a day, etc.). In this case, the designated period is, for example, one day, and the similarity calculation process shown in FIG. 4 is executed for a predetermined period (for example, the past several months).

  First, in step 100, the photo sequence generation unit 12 acquires all photo images taken during a specified period specified by the user from the photo image storage unit 30, and generates a plurality of photo sequences for each user. . FIG. 5 shows an image diagram in the case where the designated period is one day (24 hours) and the photograph images taken by a plurality of users acquired from the photograph series generation unit 12 are arranged in the order of the photographing times. Also, the photo sequence generation unit 12 assigns a unique photo ID to each photo image taken by all users within the specified period, and collects the photo images taken by each user within the specified period as one group. Each is assigned a unique photo series ID as a photo series.

  In the present embodiment, a photo sequence is a set of photo images taken by the same user within a specified period. For example, as shown in FIG. 5, a unique photo sequence ID (for example, photo sequence ID = 1) is assigned to each of the photo images X1 to X5 taken by the user A within a specified period. Thereby, a photo sequence for each user is generated.

  More specifically, the photo sequence generation unit 12 generates a photo sequence for each user by generating a photo management table as shown in FIG. As shown in the figure, the photo management table is a table including a record column in which each photo image can be set with a photo ID, a photo series ID, a section photo image set ID, and partial component information. In the example of the figure, 1 to N (N is a natural number) is assigned as a photo ID to a photo image taken by all users within a specified period. Also, 1 to M (M is a natural number) is assigned as a photo sequence ID to each photo sequence that is a set of photo images taken by each user within a specified period. In this embodiment, since the photo sequence is generated for each user, M is the same as the number of users.

  In step 102, the section photograph image set generation unit 14 divides the specified period into a plurality of sections at regular time intervals, and each of the photograph images acquired by the photograph series generation unit 12 is associated with each photograph image. Based on the shooting time information, it is assigned to one of a plurality of sections. Thereby, a section photograph image set in which the photograph images taken in the same section are grouped is generated for each section. Specifically, as shown in FIG. 6, the section photograph image set generation unit 14 sets 1 to L as section photograph image set IDs assigned to the respective photograph images in the section photograph image set ID column of the photograph management table. One of (L is a natural number) is written. FIG. 7 shows an image diagram when the designated period is divided into a plurality of sections at 10-minute intervals as an example. In this case, 24 hours are divided into 144 at 10 minute intervals (L = 144). Also, FIG. 8 shows a photographic image when users A, C, and D participate in a wedding and take pictures, and when users B, E, and F participate in Tokyo sightseeing and take pictures. An example of the type was given.

  The length of one section is set to a time (for example, 5 minutes or 10 minutes) suitable for identifying the simultaneity of the shooting time at an event highlight (for example, a cake cut if the event is a wedding). Note that the length of all sections is not limited to a constant length. For example, if the number of photographic images belonging to a certain section is less than a threshold, the length of the section is increased. May be an indefinite length.

  In Step 104, the partial component extraction unit 16 extracts partial components from all of the photographic images acquired by the photo sequence generation unit 12, that is, from all the photo sequences (all interval photo image sets). Note that the partial constituent elements are classified by at least one feature of color and shape, for example. Thereby, each photographic image can be expressed in the form of what partial components are included.

  As a method for extracting the partial constituent elements, for example, Visual Words representing an image with a feature vector can be used.

  Specifically, first, the feature point extraction unit 16A extracts a plurality of feature points from all the photographic images. For example, a method of extracting the feature points by determining the local shape of the image, such as a part with a large contrast change in the image, or setting a plurality of points at equal intervals or randomly in the image, and using the points as feature points There is a method of extracting as. FIG. 9B shows an example of an image after the feature point extraction process is performed on the original image shown in FIG.

  Next, the multidimensional feature amount calculation unit 16B calculates multidimensional feature amounts of a plurality of feature points extracted by the feature point extraction unit 16A. For example, a multidimensional feature amount is calculated from a partial image (for example, a small image such as 4, 8, 12, 16 pixels) obtained by cutting out the periphery of each feature point with a certain size. As a result, all feature points extracted from the photograph are expressed as points on a multidimensional space. FIG. 10 shows an example of a plurality of feature points on the multidimensional space represented by the multidimensional feature amount calculated by the multidimensional feature amount calculation unit 16B.

  In these feature point extraction processing and multidimensional feature amount calculation processing, the so-called SIFT (Scale Invariant Feature Transform) and SURF (Speed Up Robust Features) (refer to Non-Patent Documents 1 and 2 below) that speeds up SIFT, etc. Methods can be used, but are not limited to these. In SIFT (or SURF), first, a smoothed image is generated using a plurality of Gaussian filters with different scales with respect to the original image, and DoG (Difference) is a difference image between smoothed images of adjacent scales. of Gaussian) image. Then, the position of the feature point is determined by obtaining the maximum value from the DoG image. Next, a certain range around the feature points is divided into 16 blocks (4 each in length and width), and the gradient direction of the pixels in each block is quantized in 8 directions to generate a histogram to obtain a 128-dimensional feature value. be able to.

  Then, the clustering unit 16C clusters a plurality of feature points on the multidimensional space represented by the multidimensional feature value calculated by the multidimensional feature value calculation unit 16B, and extracts the clustered feature points as partial components. To do. Specifically, a cluster is created by collecting feature points that are close to each other in a multidimensional space. As a clustering method, for example, a general clustering method such as a K-means method (k-means method) is used. In this case, as shown in FIG. 11, each cluster is set to Visual Word, and names such as word 1 and word 2 are given for convenience.

  The K-average method generally performs the following processing. First, define the number of data as n and the number of clusters as K. Each data is randomly classified into K clusters. Next, the center of each cluster (average value of allocated data) is calculated. Next, the distance between each data and the center of each cluster is obtained, and each data is reclassified into the nearest cluster. Next, the calculation and reclassification of the center are repeated, and the process ends when the classification no longer changes. Thereby, all the photographic images can be expressed by which Visual words (partial components of the photographic image) are included.

  For example, as shown in FIG. 12, a wedding entrance photo image A1 taken by user A includes a partial image Y1 of a bouquet attached to the groom's jacket as a partial component 1, Includes various subcomponents. Similarly, the photo A2 of the wedding cake cut taken by the user A includes a partial image Y1 of the bouquet as a partial component 1 as well as the partial image Y2 of the cake. It is included as a partial component 3. In addition, the photographic image A2 includes various other partial components. In addition, the photographic image B1 taken by the user B of Tokyo Tower includes the partial image Y3 of the Tokyo Tower observation deck as the partial component 2, and also includes various other partial components.

  In this way, after extracting the partial components from all the photographic images, the partial component extraction unit 16 calculates how many of the partial components 1 to P (P is a natural number) is included in each photographic image. Then, the data is written in the photo management table as shown in FIG. In the example of FIG. 13, the photographic image with the photo ID 1 includes 10 partial components 1, 5 partial components 2, 3 partial components 3, 22 partial components 4, and partial component P. Is included.

  In step 106, the frequent-occurrence importance calculation unit 18 sets the frequent-occurrence importance of the partial constituent elements in the photo series for each of the plurality of photo series based on the appearance frequency of the partial constituent elements included in the photographic images belonging to the photo series. And for each partial component. This is because the partial components included in many photo series (for example, in the case of weddings, the bride and groom's face and a part of clothes) well represent the characteristics of the photo series, and photos containing similar partial components It is based on the fact that there is a high possibility that the series are taken of the same event.

  As a method of calculating the frequency of importance, the presence / absence of a partial component in a photo sequence may be expressed as a binary value to indicate the frequency of importance (binary expression), or the appearance frequency of the partial component in a photo sequence may be normalized. It is good also as frequent appearance importance. That is, the value may be a value that is an increasing function of the appearance frequency of the partial constituent elements in the photograph series.

  In the present embodiment, as a specific example of the method for calculating the frequency of importance, the case of calculating the frequency of importance of the subcomponents of the photo sequence using the tf-idf method used in document classification and search will be described. . tf (term frequency) represents the frequency of a certain word contained in one document, and idf (inverse document frequency) represents the reciprocal of the ratio of the word contained in the entire document set. The tf-idf (= tf * idf) value obtained by multiplying tf and idf is used as a value representing the feature of the word. This tf-idf value is large when a document contains many words in the document and not in the whole document set, that is, when the words represent the characteristics of the document well. It becomes. There are various methods for calculating tf and idf, but the most basic formula is shown below.

  Here, ni, j represents the number of words i included in document j. k represents the number of types of words included in the document j. | D | represents the total number of documents, and | {d: d ∋ ti} | represents the number of documents including the word i.

  When the tf-idf method is applied to a photo sequence, words may be replaced with partial constituent elements (such as the above-mentioned visual words) and each document may be replaced with a photo sequence. In this case, tf is (number of target partial components in the photo sequence / number of all partial components in the photo sequence). Idf is log (the number of all photo sequences / the number of photo sequences including the target partial component).

  When the tf-idf method is applied to a photo sequence, the tf-idf value becomes the frequency of importance. In this case, the frequency of occurrence of the subcomponents in the specific photo sequence increases as the frequency of occurrence of the subcomponents in the specific photo sequence increases and the frequency of appearance of the subcomponents in the photo sequence other than the specific photo sequence decreases. Increases importance. That is, a partial component having a high frequency of importance in a specific photo sequence is a good representation of the characteristics of the specific photo sequence.

  The frequent occurrence importance calculation unit 18 writes the frequent occurrence importance calculated for each partial component for each photograph series in the frequent occurrence importance degree table as shown in FIG.

  FIG. 15 shows a calculation example when the frequency importance is calculated by the tf-idf method.

  In the example shown in the figure, the photograph series of photograph series ID1 photographed by the user A includes five photograph images P1 to P5, of which five are the partial component 1 and 1 are the partial component 3. Included. Further, the number of all the partial constituent elements included in the photo series is 100, and the number of all the photo series is 100. In addition, the number of photograph series including the partial constituent elements 1 and 3 is three. In this case, the frequentity importance Wu (element 1) of the partial component 1 and the frequentity importance Wu (element 3) of the partial component 3 in the photo sequence of the photo sequence ID1 are respectively calculated by the following equations.

Wu (element 1) = tf * idf = (5/100) x log (100/3) = 0.0076

Wu (element 3) = tf * idf = (5/100) x log (100/3) = 0.0015

  Thus, since the partial component 1 is included more in the photographic image of the photo sequence ID 1 than the partial component 3, the frequentity importance of the partial component 1 is important for the frequentness of the partial component 3. The value is greater than the degree.

  In step 108, the simultaneity importance calculation unit 20 determines the simultaneous occurrence of the partial components based on the appearance frequency of the partial components included in the photographic images belonging to the section photo image set generated by the section photo image set generation unit 14. The gender importance is calculated for each partial component.

  The frequentity importance calculation unit 18 calculates the frequentity importance based on the appearance frequency of each partial component for each photograph series. The simultaneity importance calculation unit 20 calculates the simultaneity importance based on the frequency at which each partial component appears in the same section in a plurality of photograph series. Of the sub-components included in the photo series, those taken by multiple photographers at the same time (for example, a part of a cake-cut cake for weddings) may not appear in the entire photo series, It clearly shows the characteristics of the photo series. Moreover, there is a high possibility that photograph series including similar partial components are taken of the same event. For this reason, the simultaneity importance is calculated as described above.

  The simultaneity importance is calculated as a common value for all photo sequences, not for each photo sequence, and is written in the simultaneity importance table as shown in FIG.

  The calculation method of the simultaneity importance is that the occurrence frequency of the partial components in a specific section photo image set increases and the appearance frequency of the partial components in all the section photo image sets decreases. It is preferable to use a calculation method that increases the importance of sex. That is, it is preferable that the function is an increase function of the appearance frequency of partial components in a specific section photograph image set, and the appearance frequency of the partial components in other section photograph image sets is a decrease function.

  As such a calculation method, for example, the above-described if-idf method can be used. In this case, first, each section photograph image set may be regarded as a document, and a partial component included in the section photograph image set may be regarded as a word. Next, the maximum value of tf-idf calculated for each section or the average value in all the sections is obtained, and this is set as the degree of simultaneity for all photograph series (all photograph images). Thereby, the simultaneity importance of the partial components appearing in the same section can be increased, and the simultaneity importance of the partial components appearing in general regardless of the time can be calculated small.

  FIG. 17 shows a calculation example when the simultaneity importance is calculated by the tf-idf method. In the example shown in the figure, the total number of sections is 100, and all of the partial photo elements 1 and 3 are included in all the photo images included in a certain section photo image set K1. Further, the partial component 1 is included in all 100 section photograph image sets, and the partial component 3 is included in only one section photograph image set K1. In this case, tf * idf (element 1) of the partial component 1 and tf * idf (element 3) of the partial component 3 in the section photograph image set K1 are calculated by the following equations, respectively.

tf * idf (element 1) = (4/100) x log (100/100) = 0

tf * idf (element 3) = (4/100) x log (100/1) = 0.008

  Thus, since the partial component 3 appears only in one section photograph image set K1 as compared with the partial component 1, tf * idf of the partial component 3 is tf * idf of the partial component 1. It becomes a larger value.

  In this way, tf * idf of each partial component is obtained for all the section photograph image sets, and the average value of these is set as the simultaneity importance. In the example of FIG. 17, since the partial component 1 is included in all section photograph image sets, tf * idf is 0 in all section photograph image sets. For this reason, as shown in FIG. 17, the simultaneity importance Wt (element 1) of the partial component 1 is zero. Further, since the partial component 3 is included only in the section photograph image set K1, tf * idf in other section photograph image sets is all zero. For this reason, as shown in FIG. 17, the simultaneity importance Wt (element 1) of the partial component 3 is 0.08.

  In step 110, the importance synthesis unit 22 determines the importance of the partial constituent photo sequence based on the frequentity importance calculated in step 106 and the simultaneity importance calculated by the simultaneity importance calculation unit 20. The degree is calculated for each of a plurality of photograph series and each partial component.

  As a method of calculating the importance, for example, when the importance is W, the frequentity importance is Wu, the simultaneity importance is Ws, and the constant representing the influence degree of the simultaneity importance Wt on the importance W is d. There is a method of calculating the importance W of all the partial constituent elements in all the photo series by the following equations.

W = Wu + d × Wt (1)

  In the above equation (1), the importance level is obtained by weighting the concurrency importance level Wt by a constant d and adding the frequentity importance level Wu.

  Here, d is a constant that determines the degree of influence of the simultaneity importance. For this reason, the larger d is, the greater the influence of what is taken at the same time at an event (for example, a cake-cut cake for a wedding) on the importance. In addition, the smaller d is, the greater the influence of what appears in common in the entire event (such as the bride and groom's face at the wedding) on the importance.

  However, since the simultaneity importance is a value common to all the photo series, if the simultaneity importance of a certain subcomponent is high, the subcomponent is also included in a photo sequence that does not include the subcomponent at all. The weight of becomes high. Therefore, as described below, it is preferable to use a value obtained by multiplying the constant d by the frequentity importance Wu as the weight of the simultaneity importance.

W = Wu + d * Wu * Wt = Wu (1 + d * Wt) (2)

  When the importance W is calculated by the above equation (2), even if the sub-component that does not appear at all or hardly appears in the photograph series has a high simultaneity importance, the importance in the photograph series is 0 or small. . For example, if a cake does not appear in a concert photo series, the importance of the cake needs to be 0, but if the importance is calculated by the above equation (2), the cake in the concert photo series that does not include a cake Can be prevented from becoming more important.

  In FIG. 18, for example, when d = 100, the importance W (element 1) of the partial component 1 and the importance W of the partial component 3 in the photo sequence of the photo sequence ID1 including the photo image taken by the user A (element 1). The case where the element 3) is calculated by the above equation (2) is shown. As shown in the figure, since the partial component 1 is uniformly included in all the photo series, the frequentity importance Wt is larger than the partial component 3, but the simultaneity importance Wt is 0. Therefore, the importance is smaller than that of the partial component 3. In this way, the importance level synthesis unit 22 calculates the importance level W for each photo series and each partial component, and the calculated importance level W is written in the importance level table as shown in FIG.

  In step 112, the similarity calculation unit 24 calculates the similarity between a plurality of photo series based on the importance for each photo series and each partial component calculated by the importance synthesis unit 22. Each photo sequence can be expressed by a vector having the importance W of all the sub-components included in the photo sequence as an element. That is, as shown in FIG. 20, each photograph series can be expressed as a point in a multidimensional space with the importance W of all the partial components as an axis. For this reason, the similarity calculation part 24 calculates | requires each Euclidean distance of all the points in multidimensional space, and calculates those reciprocals as each similarity. The calculated similarity between each photo series is written in a similarity table as shown in FIG. FIG. 21 shows, as an example, the result of calculating the similarity between photo sequences (users) when the number of users on August 1 is 1000 (1000 photo sequences).

  Based on this similarity table, it is possible to determine that photo sequences having a similarity equal to or higher than a predetermined threshold are photo sequences obtained by photographing the same event. The processing shown in FIG. 4 may be executed periodically, for example, once a week or once a day to calculate the similarity table.

  As described above, in this embodiment, based on the frequentity importance level and the concurrency importance level, the importance level of the subcomponents for the photo series is calculated for each of the plurality of photo series and the subcomponents, and the calculated importance level is calculated. Based on the degree, the similarity between a plurality of photograph series is calculated. For this reason, the similarity of a photo sequence can be calculated with high accuracy without requiring a complicated setting operation of the importance of partial components.

  In the present embodiment, the case where the processing shown in FIG. 4 is periodically executed has been described. However, since a photographic image taken in the past may be uploaded later, in that case, FIG. The similarity table may be updated by executing the similarity calculation process shown in FIG.

  Next, functions provided by the server 40 using the similarity table generated by the similarity calculation process shown in FIG. 4 will be described. FIG. 22 shows a conceptual diagram of the service provided by the user in the photo sharing service provided by the server 40.

  A user uploads a photographic image to the server 40 via a network such as the Internet, using a photo management program 50 installed in a mobile terminal such as a personal computer (PC) or a mobile phone. The photo management program 50 includes a photo upload module 50A and a recommendation message display module 50B. The photo management program 50 is provided by a browser or a dedicated application.

  The server 40 stores the photographic image uploaded from the user in the photographic DB (the photographic image storage unit 30). Then, a similarity table is calculated by the event search program 60 including the processing program of the similarity calculation process shown in FIG. 4, and the photo series in which the same event is photographed between the photo series whose similarity is equal to or greater than a predetermined threshold value. It is judged that it is. Then, a recommendation message is generated by the recommendation message generation program 62. The generated recommendation message is transmitted to a mobile terminal such as a PC or a mobile phone of each user who has taken a series of photographs taken of the same event. For example, if the similarity between the photo sequences taken by user A and user B is greater than or equal to the threshold value, user A will be told, “Mr. “Would you like to view photos published by B?” And send a recommendation message. As a result, the user A can easily browse photos of others who participated in the same event.

  Further, when the similarity table is generated every day as shown in FIG. 21, for example, when there are a plurality of days when the similarity between the photograph series of the user A and the user C is equal to or more than a threshold, The person A and the user C are likely to participate in the same event many times. In such a case, for example, to user A, a recommendation message such as “Mr. C seems to have participated in the same event you participated many times. Would you like to register as a friend?” Generate and send. As a result, the user A can easily know the existence of others participating in the same event.

  In addition to this, the present invention can also be applied to a search service in which the user requests the server 40 to search for other users who photographed the same event as the user. In this case, the user requests the search by transmitting the date when the event occurred to the server 40. The server 40 searches the similarity table for the specified date for a photo sequence (user) whose similarity is equal to or greater than a threshold value, and transmits the result to the user terminal device.

  It can also be applied to a service for sharing additional information (event name, event description, GPS information, etc.) between photograph image sets of different users who photographed the same event.

  In this embodiment, the case where the partial constituent elements are extracted using Visual Words has been described. However, if a sufficient number of object model data can be prepared, the partial constituent elements can be extracted using the model data. You may make it extract. In this case, as shown in FIG. 23, in the case of a wedding photographic image D1, model data such as a groom A, a bride B, a tree, and a cake are required. In the case of the photographic image D2 of Tokyo Tower, Tokyo Tower and sky model data are required. Then, it is only necessary to determine whether or not the partial constituent elements represented by these model data are included in the photographic image D1, and calculate the frequent appearance importance and the concurrency importance by the above-described method.

  In the above description, a mode has been described in which a photographic sequence similarity calculation program, which is an example of a photographic sequence similarity calculation program according to the disclosed technology, is stored (installed) in the storage unit 76 in advance. It is not something. The image processing program according to the disclosed technology can be provided in a form recorded on a recording medium such as a CD-ROM or a DVD-ROM.

  All documents, patent applications and technical standards mentioned in this specification are to the same extent as if each individual document, patent application and technical standard were specifically and individually stated to be incorporated by reference. Incorporated by reference in the book.

  Regarding the above embodiment, the following additional notes are disclosed.

(Appendix 1)
The partial component extraction step includes a feature point extraction step for extracting a plurality of feature points from the photographic image, and a multidimensional feature amount for calculating a multidimensional feature amount of the plurality of feature points extracted by the feature point extraction step. And clustering the plurality of feature points on the multidimensional space represented by the multidimensional feature amount calculated by the calculating step and the multidimensional feature amount calculating step, and extracting the clustered feature points as the partial constituent elements And a clustering step. The similarity calculation method for a photo sequence according to claim 6.

(Appendix 2)
The frequency of importance calculation step includes increasing the frequency of appearance of the partial component in the specific photo sequence and decreasing the frequency of occurrence of the partial component in all the photo sequences. The method of calculating similarity of a photo series according to claim 6 or 1, wherein the frequency of importance is calculated so that the frequency of importance of the partial components in the series is increased.

(Appendix 3)
In the simultaneity importance calculation step, as the appearance frequency of the partial component in the specific section photo image set increases and the appearance frequency of the partial component in all the section photo image sets decreases, The photographic sequence similarity calculation method according to claim 6, wherein the simultaneity importance is calculated so that the simultaneity importance of a partial component increases.

(Appendix 4)
In the importance synthesis step, the importance is W, the frequentity importance is Wu, the simultaneity importance is Wt, and a constant representing the influence of the simultaneity importance on the importance W is d. Further, the importance W of the partial component in the photo sequence is calculated by the following equation: W = Wu + d × Wu × Wt
The method for calculating the similarity of a photographic sequence according to any one of claims 6 and 1-3.

(Appendix 5)
The partial component extraction step includes a feature point extraction step for extracting a plurality of feature points from the photographic image, and a multidimensional feature amount for calculating a multidimensional feature amount of the plurality of feature points extracted by the feature point extraction step. And clustering the plurality of feature points on the multidimensional space represented by the multidimensional feature amount calculated by the calculating step and the multidimensional feature amount calculating step, and extracting the clustered feature points as the partial constituent elements A photographic sequence similarity calculation program according to claim 7, comprising a clustering step.

(Appendix 6)
The frequency of importance calculation step includes increasing the frequency of appearance of the partial component in the specific photo sequence and decreasing the frequency of occurrence of the partial component in all the photo sequences. 6. The photo series similarity calculation program according to claim 7 or claim 5, wherein the frequency importance is calculated so that the frequency importance of the partial components in the series is increased.

(Appendix 7)
In the simultaneity importance calculation step, as the appearance frequency of the partial component in the specific section photo image set increases and the appearance frequency of the partial component in all the section photo image sets decreases, The photographic sequence similarity calculation method according to claim 7, wherein the simultaneity importance is calculated so that the simultaneity importance of a partial component increases.

(Appendix 8)
In the importance synthesis step, the importance is W, the frequentity importance is Wu, the simultaneity importance is Wt, and a constant representing the influence of the simultaneity importance on the importance W is d. Further, the importance W of the partial component in the photo sequence is calculated by the following equation: W = Wu + d × Wu × Wt
The method for calculating similarity of a photographic sequence according to any one of claims 7 and appendices 5-7.

DESCRIPTION OF SYMBOLS 10 Similarity calculation apparatus 12 Photo series production | generation part 14 Section photograph image set production | generation part 16 Partial component extraction part 16A Feature point extraction part 16B Multidimensional feature-value calculation part 16C Clustering part 18 Frequentity importance calculation part 20 Simultaneous importance Calculation unit 22 Importance composition unit 24 Similarity calculation unit 30 Photo image storage unit 32 Table storage unit 40 Server

Claims (7)

From the photographic image storage unit in which a plurality of photographic images associated with photographic time information representing the photographic time and user information representing the corresponding user are stored, the photographic image is obtained and a photographic sequence for each user is obtained. A plurality of photo sequence generation units;
A period during which the photographic image is taken is divided into a plurality of sections at arbitrary time intervals, and each of the photographic images acquired by the photographic sequence generation unit is selected from any of the plurality of sections based on the shooting time information. By assigning to the section photo image set generation unit for generating a section photo image set for each of the plurality of sections,
A partial component extraction unit that extracts a partial component included in the photo image acquired by the photo series generation unit;
Based on the appearance frequency of the partial component included in the photographic image belonging to the photographic sequence, the frequency of importance of the partial component in the photographic sequence is determined for each of the plurality of photographic sequences and for each partial component. A frequentity importance calculator to calculate,
Based on the appearance frequency of the partial component included in the photographic image belonging to the section photo image set generated by the section photo image set generation unit, the simultaneity importance of the partial component is determined for each partial component. A simultaneity importance calculation unit for calculating
Based on the frequentity importance calculated by the frequentity importance calculation unit and the simultaneity importance calculated by the simultaneity importance calculation unit, the importance of the partial component with respect to the photo sequence is determined. Importance synthesis unit to calculate for each of the plurality of photo series and for each partial component,
A similarity calculation unit that calculates a similarity between the plurality of photo series based on the importance for each of the plurality of photo series and the partial components calculated by the importance synthesis unit;
Apparatus for calculating similarity of photo series including The partial component extraction unit includes a feature point extraction unit that extracts a plurality of feature points from the photographic image, and a multidimensional feature amount that calculates a multidimensional feature amount of the plurality of feature points extracted by the feature point extraction unit. The plurality of feature points are clustered on a multidimensional space represented by the multidimensional feature amount calculated by the calculation unit and the multidimensional feature amount calculation unit, and the clustered feature points are extracted as the partial constituent elements A clustering unit;
The photographic sequence similarity calculation apparatus according to claim 1, comprising: The frequentity importance calculating unit increases the appearance frequency of the partial component in the specific photo sequence and decreases the appearance frequency of the partial component in the photo sequence other than the specific photo sequence. The photograph sequence similarity calculation device according to claim 1, wherein the frequency importance is calculated so that the frequency importance of the partial component in the specific photo sequence is increased. The simultaneity importance calculation unit increases the appearance frequency of the partial component in the specific section photo image set and the appearance frequency of the partial component in the section photo image set other than the specific section photo image set. The photographic sequence similarity calculation device according to any one of claims 1 to 3, wherein the simultaneity importance is calculated so that the simultaneity importance of the partial constituent elements increases as the value decreases. When the importance level is W, the frequentity importance level is Wu, the simultaneity importance level is Wt, and a constant representing the influence level of the simultaneity importance level on the importance level W is d. Further, the importance W of the partial component in the photo sequence is calculated by the following equation: W = Wu + d × Wu × Wt
The similarity calculation apparatus of the photograph series of any one of Claims 1-4. Computer
The photographic image is acquired from a photographic image storage unit in which a plurality of photographic images associated with photographing time information representing photographing time and user information representing a corresponding user is stored, and the use is performed by arithmetic processing of the computer. A plurality of photo sequences for each person , and a photo sequence generation step for storing in a memory ;
A period during which the photographic image is taken is divided into a plurality of sections at arbitrary time intervals by an arithmetic process of the computer, and each of the photographic images obtained by the photographic sequence generation step is based on the photographing time information. by assigning to one of said plurality of segments by the processing of the feeder the computer to generate an interval photographic image set for each of the plurality of sections, and the section image set generation step for storing in said memory,
A partial component extraction step of extracting a partial component included in the photographic image acquired by the photographic sequence generation step by a calculation process of the computer ;
Based on the appearance frequency of the partial component included in the photographic image belonging to the photographic sequence, the frequency of importance of the partial component in the photographic sequence is determined for each of the plurality of photographic sequences and for each partial component. A frequentity importance calculating step of calculating and storing in the memory ;
Based on the appearance frequency of the partial component included in the photographic image belonging to the section photo image set generated by the section photo image set generation step, the simultaneity processing of the simultaneity of the partial components is performed by the computer said calculated for each subcomponent, simultaneity importance calculating step you stored in the memory by,
The frequentity importance calculated by the frequent importance calculation step and stored in the memory and the simultaneity importance calculated by the simultaneity importance calculation step and stored in the memory are read from the memory. Based on the read frequentity importance and the simultaneity importance read out, the importance of the partial component with respect to the photograph series is calculated for each of the plurality of photograph series and for each of the partial constituent elements by arithmetic processing of the computer. Calculating the importance and storing it in the memory ;
The importance calculated for each of the plurality of photograph series and each partial component calculated by the importance combining step and stored in the memory is read from the memory, and for each of the plurality of photograph series read and for each partial component A similarity calculation step of calculating a similarity between the plurality of photo series by an arithmetic processing of the computer based on the importance of
A method for calculating the similarity of a photo series. On the computer,
The photographic image is acquired from a photographic image storage unit in which a plurality of photographic images associated with photographing time information representing photographing time and user information representing a corresponding user is stored, and the use is performed by arithmetic processing of the computer. A plurality of photo sequences for each person , and a photo sequence generation step for storing in a memory ;
A period during which the photographic image is taken is divided into a plurality of sections at arbitrary time intervals by an arithmetic process of the computer, and each of the photographic images obtained by the photographic sequence generation step is based on the photographing time information. by assigning to one of said plurality of segments by the processing of the feeder the computer to generate an interval photographic image set for each of the plurality of sections, and the section image set generation step for storing in said memory,
A partial component extraction step of extracting a partial component included in the photographic image acquired by the photographic sequence generation step by a calculation process of the computer ;
Based on the appearance frequency of the partial component included in the photographic image belonging to the photographic sequence, the frequency of importance of the partial component in the photographic sequence is determined for each of the plurality of photographic sequences and for each partial component. A frequentity importance calculating step of calculating and storing in the memory ;
Based on the appearance frequency of the partial component included in the photographic image belonging to the section photo image set generated by the section photo image set generation step, the simultaneity processing of the simultaneity of the partial components is performed by the computer said calculated for each subcomponent, simultaneity importance calculating step you stored in the memory by,
The frequentity importance calculated by the frequent importance calculation step and stored in the memory and the simultaneity importance calculated by the simultaneity importance calculation step and stored in the memory are read from the memory. Based on the read frequentity importance and the simultaneity importance read out, the importance of the partial component with respect to the photograph series is calculated for each of the plurality of photograph series and for each of the partial constituent elements by arithmetic processing of the computer. Calculating the importance and storing it in the memory ;
The importance calculated for each of the plurality of photograph series and each partial component calculated by the importance combining step and stored in the memory is read from the memory, and for each of the plurality of photograph series read and for each partial component A similarity calculation step of calculating a similarity between the plurality of photo series by an arithmetic processing of the computer based on the importance of
A similarity calculation program for a photo sequence for executing a process including