JP2013015902A - Image retrieval device, image retrieval system, image retrieval method and image retrieval program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval device capable of easily displaying a target image required by a user as a search result, an image retrieval system, an image retrieval method and an image retrieval program.SOLUTION: An image retrieval device 1 comprises: image obtaining means 41 for sequentially obtaining photographic images; image retrieval means 42 for retrieving retrieval target images similar to the photographic image; a result storage part 32 for storing the retrieval result; and result display means 44 for ranking retrieval target images calculated based on the stored retrieval results in the descending order of certainty factor to display the retrieval target images. The result display means 44 updates the display ranking of the displayed retrieval target images corresponding to the presence/absence of user's selection operation from starting the display of the retrieval target images until a predetermined time passes.

Description

  The present invention relates to an image search device, an image search system, an image search method, and an image search program for searching for a search target image similar to the shot image from a shot image.

2. Description of the Related Art Conventionally, there has been known an image search apparatus that searches for a target object from a captured image obtained by capturing the target object as an image search target (see, for example, Patent Document 1).
In the image search device described in Patent Document 1, an image optimal for searching for an object among a plurality of captured images is acquired by, for example, a user's selection operation, and the object is searched using the acquired image. We are carrying out.

JP 2010-66959 A

  By the way, in the image search device described in Patent Document 1, an image search process is performed using a captured image obtained by shooting the target object to search for the target object. Here, in the image search process, generally, an image of an object having a high degree of similarity to the acquired image is searched from a database in which the image of the object is stored, and a predetermined number of images are displayed. However, the state of the photographed image obtained by photographing the object changes depending on the photographing environment such as illuminance and angle at the time of photographing. Therefore, as described above, even if an object having a high similarity to the captured image is acquired, the target object cannot always be searched.

  On the other hand, it is also conceivable to perform an image search for each of a plurality of continuously captured images and display the result of the image search for each of the plurality of captured images. In this case, if the search target image of the displayed search result does not include an image (target image) requested by the user, it is necessary to repeatedly acquire a captured image and perform image search until the target image is displayed. There is. However, even when the captured image is acquired and the image search is repeatedly performed, the target image may not be displayed as the search result due to the influence of the shooting environment at the time of shooting. In this case, there is still a problem that a target image desired by the user cannot be obtained or a long time is required until the target image is displayed.

  An object of the present invention is to provide an image search device, an image search system, an image search method, and an image search program in which a target image desired by a user is easily displayed as a search result.

The image search device of the present invention is an image search device for acquiring a captured image and searching for a search target image similar to the captured image, wherein the image acquisition means for sequentially acquiring the captured image and the captured image are acquired. Each time the search target image similar to the photographed image is searched, an image search means for acquiring a search result including the searched search target image and its similarity, and a result of storing the search result Based on the certainty factor of each search target image calculated using the search result accumulated in the accumulation unit and the result accumulation unit, a result display that ranks the certainty factor in descending order and displays the search target image And the result display means displays a table of the displayed search target images according to whether or not a user has performed a selection operation until a predetermined time elapses after the search target images are displayed. And wherein the updating the rankings.
Here, the certainty factor described in the present invention is calculated based on the similarity between the captured image and the search target image. The certainty calculated based on the similarity is obtained by obtaining a similarity ranking obtained by rearranging the search target images searched when the image search is performed in descending order of similarity, ) May be set, or the certainty factor may be directly calculated based on the similarity between the captured image and the search target image.

  In the present invention, the captured images are sequentially acquired by the image acquisition means, and the image search means searches for a search target image having a high similarity from the image storage unit for each of the acquired captured images. Here, the image search means, for example, compares a parameter such as a feature amount of the captured image with a parameter such as a feature amount of the search target image, and searches for a search target image similar to the captured image. The search result searched by the image search means includes a search target image similar to the photographed image and the similarity of the search target image to the photographed image. The search result is sequentially stored in the result storage unit. Accumulated. In the image search device of the present invention, search target images are displayed in descending order of certainty calculated based on the search results stored in the result storage unit. That is, the result display means displays the search target image based on the certainty factor every time the captured image is acquired, and the search result display is updated in real time.

In the image search device of the present invention, the result display means updates the display ranking of the displayed search target image according to the presence / absence of the user's selection operation on the displayed search target image, and performs a new search. Display the target image.
That is, even when a search target image having a high degree of similarity to the captured image is captured, the search target image may not be the image that the user wants. In such a case, only the search result based on the similarity cannot provide a search result that satisfies the user. On the other hand, in the present invention, the display ranking is updated in accordance with the presence or absence of an operation in which the user selects a target image. In this case, even if the target image is not displayed in the search result by only one image search, the possibility that the target image is displayed as the search result increases with time. That is, according to the present invention, it is possible to perform an image search in which a target image desired by a user is easily displayed as a search result.

  In the image search device of the present invention, the result display means may reduce the display ranking of the search target image when there is no user selection operation for the displayed search target image and a predetermined time has elapsed. preferable.

In the present invention, the display ranking of the displayed search target image is updated after a predetermined time has elapsed without any selection operation by the user. That is, the fact that the displayed search target image is not selected by the user indicates that the displayed search target image is not a search target image (hereinafter referred to as a target image) desired by the user. If such a search target image is displayed, it is difficult to display a target image set with a lower certainty factor than the displayed search target image. For example, when performing a process that increases the certainty factor for the search target image searched by the image search, it is necessary to repeat the image search until the certainty factor of the target image becomes larger than the certainty factor of the search target image. . In such a case, it may take a long time before the target image is displayed.
In contrast, in the present invention, as described above, the searched search target images are ranked and displayed in descending order of certainty, and the display ranking of the displayed search target images is changed after a predetermined time has elapsed. It is decreasing. As a result, the search target image that has not been displayed is newly displayed as a search target image having a higher certainty factor, so that the user can find the target image earlier.

On the other hand, if the displayed search target image is deleted before a certain period of time elapses, the target image can be selected when the target image exists in the displayed search target image. Disappear. On the other hand, in the present invention, the display deletion processing of the search target image as described above is performed after a predetermined time has elapsed after the search target image is displayed. Here, although the predetermined time will be described later, for example, the determination time set to such an extent that the user can confirm all the search target images displayed, or after the search processing by the image search device of the present invention is started. The ratio (display rate) of the display time of each search target image with respect to the total time can be mentioned.
In such a configuration in which the display ranking is updated after the predetermined time elapses, it is possible to avoid the disadvantage that the displayed inspection target image is deleted before the user confirms the inspection target image.

  In the image search device of the present invention, the image search apparatus includes a certainty factor updating unit that calculates a certainty factor of each search target image using the search result accumulated in the result accumulation unit, and the certainty factor calculating unit includes the displayed When there is no selection operation of the user for the search target image, the reliability of the search target image is decreased and updated, and the result display unit is configured to search the search target in descending order of reliability based on the updated certainty factor. It is preferable to display an image.

  In the present invention, the result display means reduces the display ranking by lowering the certainty factor of the search target image when there is no selection operation of the search target image by the user. In this case, each time the certainty factor is updated, the result display unit only needs to display the search target image based on the updated certainty factor. For example, the result display unit updates the display ranking based on other parameters. There is no need to consider it, and the processing of the result display means can be simplified.

In the image search device of the present invention, it is preferable that the result display means excludes the search target image from the display when there is no user selection operation on the displayed search target image.
Here, when the result display unit updates the display ranking based on the certainty factor updated by the certainty factor calculating unit, the certainty factor of the search target image to be excluded is set to, for example, “0” by the certainty factor calculating unit. The minimum value can be set. Further, when the result table means directly updates the display ranking regardless of the certainty factor, the search target image to be excluded may be deleted from the display, and the display ranking of the other search target images may be moved up and displayed. .
In the present invention, search target images that are not selected by the user are immediately excluded from the display. In this case, since the search target image to be displayed can be switched more quickly, the probability that the target image is displayed early can be increased.

In the image search device of the present invention, it is preferable that the result display means lowers the display ranking of the search target image in a stepwise manner when there is no user selection operation on the displayed search target image.
Here, when the result display unit updates the display ranking based on the certainty factor updated by the certainty factor calculating unit, the certainty factor of the search target image to be excluded is, for example, “0.5” by the certainty factor calculating unit. It is only necessary to decrease the value step by step and update it in advance. In addition, when the result table means directly updates the display ranking regardless of the certainty factor, the display ranking is lowered by replacing the search ranking image of the search target image to be excluded with the search ranking image of the lower ranking. Just do it.
In the present invention, even when there is no user selection operation on the displayed search target image, it is not immediately excluded from the display, and the ranking of the search target image on the display gradually decreases step by step. In such a case, for example, even when the user cannot confirm the search target image displayed by looking aside, the search target image is not immediately excluded, and the time for selecting the search target image by the user is reduced. It can be secured sufficiently.

  The image search device of the present invention comprises display rate calculation means for calculating a display rate that is a ratio of the time during which each search target image is displayed to the total time during which the search target image is displayed by the result display means, The result display means preferably reduces the display ranking of the search target image when there is no user selection operation on the displayed search target image and the display rate of the search target image exceeds a threshold value. .

In the present invention, the display ratio is calculated by the display ratio calculation means, and the display ranking is updated when the display ratio exceeds the threshold.
That is, as described above, a search target image with a high display rate indicates that the search target image has been displayed for a long time, and that the search target image has not been selected by a selection operation by the user. Indicates that the target image is not the target image desired by the user. By removing such an image from display by updating the display ranking and displaying a new search target image, the target image desired by the user can be displayed as a search result more quickly as described above.

  In the image search device of the present invention, the result display means may display the display ranking of the search target image when there is no user selection operation on the displayed search target image and a preset determination time has elapsed. May be reduced.

  In the present invention, the result display means lowers the display ranking of the search target image when there is no search target image selection operation by the user before the preset determination time elapses. Even in this case, similarly to the above-described invention, the search target image that is not selected by the user can be removed from the display by updating the display ranking, and the new search target image can be displayed. Can be displayed as a search result more quickly.

In the image search device of the present invention, the result display means may increase the display ranking of the search target image when there is no user selection operation for the displayed search target image and the time is within the predetermined time. preferable.
In the present invention, when there is no selection by the user and a predetermined time has not elapsed, the result display means increases the display ranking of the search target images searched by the image search means. In other words, the higher the number of searches, the more similar it is to the captured image, so by setting such a search target image at the top of the display ranking, the target image desired by the user is displayed. Can increase the probability.

  In the image search device of the present invention, the image search apparatus includes a certainty factor updating unit that calculates the certainty factor of each search target image using the search result accumulated in the result accumulation unit, and the certainty factor calculating unit includes the captured image and Based on the similarity of the search target image and the number of times the search processing is performed by the image search means, an average value of the similarity of each search target image is calculated and used as the certainty of the search target image. Is preferred.

In the present invention, a certainty factor is given to the search target image based on the similarity between the photographed image and the search target image, and the similarity average for each search target image is calculated based on the similarity and the number of times the search process is performed. The value is calculated as a certainty factor.
As described above, in the image search, the search target images are searched in the descending order of similarity between the captured image and the search target image. Even if the similarity ranking is high, the similarity between the captured image and the search target image is determined. The degree may be low. In such a case, if the certainty factor is calculated based on the similarity ranking, there is a possibility that a search target image having a similarity greatly different from the target image is displayed. On the other hand, in the present invention, since the certainty factor is calculated based on the similarity, a high certainty factor can be given to a search target image having a high similarity regardless of the similarity ranking. For this reason, an appropriate search target image similar to the target image can be displayed.

  In the image search device of the present invention, when the displayed search target image is updated, the result display means displays a display form of the search target image to be updated in a display form different from other search target images. It is preferable.

According to the present invention, when the result display means changes the order of the display positions of the displayed search target images or displays a new search target image, the display mode of the search target image is changed to another search mode. Display in a different display form from the target image. Here, as the “different display forms”, for example, the display size of the search target image to be displayed is changed or the display color of the search target image is changed depending on whether the certainty factor is increased or decreased. To do.
Thus, by changing the display form of the search target image, the user can easily confirm the search target image whose display position has changed. For example, when one search target image is deleted and another search target image is newly displayed in place of the deleted search target image, the result display means adds a new search target image to the already displayed search target image. The search target images displayed on the screen are displayed in different display forms. In this case, the user can easily confirm the newly displayed search target image, and can suitably assist the user in confirming the target image.

  In the image search device of the present invention, the image acquisition unit acquires a captured image in which an object is captured, and the image acquisition unit searches for the search target image having an image of an object similar to the object of the captured image. It is preferable to do.

  In the present invention, a search target image including an image of an object similar to the object is searched from a plurality of search target images using a captured image obtained by shooting the object. That is, in the present invention, what is called an object search (object recognition) in which what an object included in a captured image is examined by an image search can be performed. In the present invention, in such object recognition, it is possible to increase the possibility that an image including an object that the user wants to investigate is displayed.

1 is a block diagram showing a schematic configuration of an image search device according to a first embodiment. The figure which shows the example of a display in the display part of the image search device of 1st embodiment. The figure which shows the example of a display change of the display part of 1st embodiment. The flowchart which shows the image search method in 1st embodiment. The flowchart which shows the image search method in 2nd embodiment. The figure which shows the example of a display change of the display part of 3rd embodiment. The figure which shows the example of a display of the display part in 6th embodiment. The figure which shows schematic structure of the image search system of 7th embodiment. The flowchart which shows the image search method in the image search system of 7th embodiment. The figure which shows schematic structure of the image search system of 8th embodiment. The flowchart which shows the image search method in the image search system of 8th embodiment.

[First embodiment]
Hereinafter, an image search apparatus according to a first embodiment of the present invention will be described with reference to the drawings.
[Configuration of image search device]
FIG. 1 is a block diagram illustrating a schematic configuration of the image search apparatus according to the first embodiment.
In FIG. 1, an image search apparatus 1 includes a camera 2, a storage means 3 for storing various programs and various data, a control means 4 configured by a CPU (Central Processing Unit) for performing various calculations, a display, and the like. Display unit 5 and an operation unit 6 such as a keyboard and a touch panel. As this image search apparatus 1, what is called computers, such as a mobile telephone with a camera function, a smart phone, a portable information terminal, a personal computer, can be used, for example.

The image search apparatus 1 is an apparatus that captures an object with the camera 2 and uses the captured image to search for and display a search target image similar to the captured image.
More specifically, when the control unit 4 of the image search device 1 is operated by the user to perform an image search used for a captured image, the camera 2 is first activated. When the camera 2 is pointed at one object, the control unit 4 automatically obtains a photographed image obtained by photographing the object at a certain photographing cycle, and uses each photographed image to perform an image search (object Recognition). At this time, the control means 4 periodically acquires captured images while the camera is activated, and performs object recognition (image search) for each of the acquired captured images. Then, the control unit 4 calculates the certainty factor of the search target image based on the recognition result (search result) of the object recognition for each captured image, and displays the image recognition result based on the certainty factor on the display unit 5 in real time. Let And the control means 4 will stop a series of search processes, if a camera is stopped by a user's operation.
That is, the image search apparatus 1 of the present embodiment searches for an optimal object image for a captured image based on the recognition results of a plurality of object recognitions. In the following description, a series of such search processes are performed. Is referred to as optimal image search processing.
Hereinafter, a configuration of the image search apparatus 1 for performing the above-described optimal image search process will be described.

  The storage unit 3 of the image search apparatus 1 includes an image database 31 (image storage unit) as various types of information for performing the optimum image search process as described above. In addition, the storage unit 3 is provided with a result storage unit 32 that stores the result of object recognition.

The image database 31 has a table structure in which search target images and index data are associated and stored as one record.
Here, the search target image stored in the image database 31 is an image including a search target object. In the present embodiment, the image database 31 is searched for so-called object recognition, in which a search target image having a search target object similar to the object is searched for a captured image obtained by shooting the object. The search target image including the target object is recorded, but the present invention is not limited to this. For example, when a landscape, a picture, or the like is used as a photographed image subjected to image search, an image (landscape, a picture, etc.) that does not include a specific search target object may be stored as a search target image.
Further, the index data stored in the image database 31 records a feature amount characterizing the search target image, an index for searching the feature amount, an image ID for determining the search target image, and the like.

The result accumulation unit 32 is a temporary storage area for accumulating recognition results when object recognition is performed using a captured image.
Here, as the recognition result of the object recognition, the search target image searched by the image search means 42 described later and the similarity between the search target image and the photographed image are recorded. Further, instead of the search target image, an image ID for specifying the search target image may be recorded.
Here, the similarity between the search target image and the captured image is a value calculated by comparing the feature amount of the search target image with the feature amount of the captured image, and how similar the search target image is to the captured image. It is a value indicating whether or not The similarity is expressed as a value from 0 to 1, with 1 being the case where the search target image and the captured image completely match, for example.

  The image search apparatus 1 includes, as the functional configuration of the control unit 4, an image acquisition unit 41, an image search unit 42, a certainty factor calculation unit 43, a result display unit 44, an image selection unit 45, a display rate calculation unit 46, and a search Exclusion means 47 is provided.

When an object recognition processing program for a captured image is executed by a user operation, the image acquisition unit 41 activates the camera 2 and acquires a captured image captured by the camera 2. Here, the image acquisition unit 41 acquires a captured image at a predetermined fixed imaging cycle.
In the present embodiment, an example in which a captured image is automatically acquired by the image acquisition unit 41 is shown. However, for example, a configuration may be adopted in which a captured image is acquired by a user's shutter operation.

The image search means 42 uses the captured image acquired by the image acquisition means 41 to perform object recognition for acquiring a search target image similar to the captured image from the image database 31 of the storage means 3.
Specifically, the image search means 42 calculates the feature amount of the photographed image, and searches the search target image corresponding to the feature amount of the photographed image from the feature amount index stored in the index data of the result storage unit 32. Extract. The image search means 42 compares the feature amount of the captured image with the feature amount of the search target image stored in the index data, calculates the similarity of the search target image to the captured image, and the similarity is A search target image that is equal to or greater than a preset threshold is searched.
Then, the image search unit 42 stores the recognition result of the object recognition in the result storage unit 32 of the storage unit 3.

The certainty factor calculation unit 43 calculates the certainty factor for each search target image searched by the image search unit 42 based on the past search results recorded in the result storage unit 32. This certainty factor is calculated based on the similarity between the search target image and the photographed image, and the reliability indicating whether or not the search target image is reliable as the target image desired by the user. It becomes the parameter shown.
The certainty factor calculating unit 43 calculates the certainty factor based on the stored search result every time the object is recognized by the image search unit 42 and the search result is stored in the result storage unit 32.
As a method of calculating the certainty factor by the display rate calculating means 46, for example, an average of the past K times of similarity can be used for each search target image. At this time, if the search is not performed by object recognition, the similarity may be calculated as 0. Further, in the series of optimum image search processes, when the certainty factor is calculated based on the result of first performing object recognition, the certainty factor is calculated based on one search result. In this case, the certainty factor calculation unit 43 can set the similarity of the search target image to the captured image as the certainty factor of the search target image.

The result display unit 44 displays the captured image acquired by the camera 2 and the searched search target image on the display unit 5 in real time.
Here, FIG. 2 shows an example of one display screen displayed on the display unit 5 in the optimum image search process.
As shown in FIG. 2, the result display unit 44 displays the shooting target image 51 shot by the camera 2 on the entire display unit 5. In addition, the result display unit 44 acquires the preset number of displays (for example, four) in the descending order of the certainty among the searched search target images, and displays the certainty higher image in the lower area of the display unit 5. 52.
In the lower area of the display unit 5, the leftmost area is the highest display position 52A, and the rightmost area is the lowest display position 52B. The result display means 44 displays the certainty level upper image 52 in an order of certainty by arranging from the most significant display position 52A to the least significant display position 52B. Therefore, the highest certainty factor among the certainty factor upper images 52 is displayed at the highest display position 52A, and the lowest certainty factor is displayed at the lowest display position 52B. That is, the result display means 44 ranks the certainty level upper images 52 in descending order of certainty level, and displays the display ranking on the display unit 5.
Here, an example is shown in which the certainty factor upper image 52 is shown in the lower region of the display unit 5, but the present invention is not limited to this. For example, it may be displayed in the upper area of the display unit 5 or may be displayed in another table area. Moreover, although it arranges and displays in order with large reliability, for example, the number which shows the order of reliability is superimposed on a part of high reliability image 52 so that the order with large reliability may be understood, and the display part 5 of FIG. The certainty level upper image 52 may be displayed at the four corners of the control means.

  Here, each time the certainty factor of each search target image is calculated (updated) by the certainty factor calculating unit 43, the certainty factor upper image 52 displayed on the display unit 5 is updated. That is, when the certainty factor of each search target image is calculated by the certainty factor calculating unit 43, if the certainty factor rank changes, the certainty factor upper image 52 displayed on the display unit 5 based on the changed rank. Update. As a result, the certainty factor upper image 52 displayed on the display unit 5 is updated and displayed as time elapses.

For example, when the search target images A, B, C, and D are searched in descending order of the certainty factor, the result display unit 44 displays the search target images A and B on the display unit 5 as shown in FIG. B, C, and D are displayed in order. Thereafter, when the certainty factor ranking is changed to A, B, E, and F by the certainty factor calculation by the certainty factor calculating unit 43, the result display unit 44 performs the search as shown in FIG. The target images C and D are deleted from the display, and the search target images E and F are displayed on the display unit 5 as the certainty level upper image 52 instead.
At this time, each time the certainty factor is updated, the result display means 44 scrolls the already displayed certainty factor upper image 52 out of the frame and scrolls the newly displayed certainty factor upper image 52 to the frame. A display method such as being inserted can be used.

Further, in the optimum image search process of the present embodiment, when the result display means 44 displays the higher confidence level image 52 in the order of confidence level on the display unit 5, any one of the displayed higher confidence level images 52 is displayed. Can be selected by the user. Therefore, when the certainty factor upper image 52 (target image) including the same object as the object to be imaged is present in the plurality of displayed certainty factor upper images 52, the user selects the certainty factor upper image 52. Thus, information regarding the certainty factor upper image 52 can be acquired.
On the other hand, for example, when there is no target image in the displayed higher certainty factor image 52, the certainty factor of the target image desired by the user is equal to or higher than the certainty factor of the higher certainty factor image 52 already displayed on the display unit 5. Until then, it is not displayed on the display unit 5. In this case, if the certainty factor of the certainty factor upper image 52 already displayed on the display unit 5 remains high, the target image is not displayed on the display unit 5 or it takes a long time to be displayed on the display unit 5. It may take.

On the other hand, in the image search device 1 of the present embodiment, in order to make it easy to display the target image, the result is obtained when the certainty factor upper image 52 is not selected by the certainty factor calculation unit 43 for a certain period of time. The display unit 44 updates the display ranking by deleting the display of the certainty factor upper image 52 (search target image) displayed on the display unit 5 or lowering the display ranking.
In this process, the result display means 44 is implemented based on the display rate calculated by the display rate calculation means 46 described later. That is, the result display unit 44 measures the time from the start of the optimum image search process, and after the predetermined set time has elapsed, the display rate calculated by the display rate calculation unit 46 is a preset threshold value. The certainty level upper image 52 exceeding (for example, 0.8) is deleted from the display unit 5 and a new search target image is displayed on the display unit 5. That is, the result display unit 44 excludes the search target image whose display rate exceeds the threshold, and updates the certainty level upper image 52 to be displayed on the display unit 5 based on the certainty level.

For example, as shown in FIGS. 3A to 3C, in a series of optimum image search processes, D seconds have elapsed since the first certainty factor upper image 52 was displayed on the display unit 5, It is assumed that the search target images A and B displayed on the display unit 5 are displayed on the display unit 5 for d seconds. In this case, the display rates of the search target images A and B are calculated by d / D.
When the display rates of the search target images A and B are equal to or higher than a preset threshold value X, the result display unit 44 displays the search target image A displayed on the display unit 5 as shown in FIG. , B confidence level upper image 52 is erased. Then, the result display unit 44 displays the search target images E and F at the positions where the search target images A and B were displayed. Further, the search target images G and H having the next highest certainty after the search target image F are newly displayed at the positions where the search target images E and F were displayed.

The image selection unit 45 selects at least one of the certainty factor upper images 52 displayed on the display unit 5 based on the user's selection operation.
Here, when the certainty level upper image 52 is selected by the image selection unit 45, the control unit 4 uses the network or the like, for example, for detailed information on the selected certainty level upper image 52 (for example, product information about the object). ) And the acquired detailed information is displayed on the display unit 5.

  The display rate calculation means 46 calculates the display rate of each certainty factor upper image 52 displayed on the display unit 5. As described above, the display rate calculation means 46 calculates the elapsed time D from the time when the higher certainty factor image 52 is first displayed on the display unit 5 to the present time, and each higher certainty factor image 52 in the optimum image search process. The display time d displayed on the display unit 5 is acquired, and the display rate (d / D) is calculated.

Further, the search exclusion unit 47 excludes the search target image deleted from the display unit 5 from the search target in the series of optimum image search processing, assuming that the display rate exceeds the threshold value.
In other words, the search exclusion unit 47 excludes the certainty factor upper image 52 (search target image) deleted by the result display unit 44 from the recognition result stored in the result storage unit 32 because the display rate has exceeded the threshold value. . In addition, when the search target image deleted from the display unit 5 is searched by the result display unit 44 in the subsequent object recognition by the image search unit 42, the search exclusion unit 47 selects the search target image from the recognition result. exclude. As a result, the search target image is not recorded in the result storage unit 32 and the certainty factor for the search target image is not calculated by the certainty factor calculation unit 43, and thus displayed on the display unit 5 by the result display unit 44. There is nothing.

[Image Search Method by Image Search Device]
Next, the optimum image search process of the image search apparatus 1 as described above will be described based on the drawings.
FIG. 4 is a flowchart of the optimum image search process in the operation of the image search apparatus 1 of the present embodiment.
When the optimal image search process is performed using the image search apparatus 1, the user first activates a program that executes the optimal image search process. As a result, the camera 2 is activated and the image captured by the camera 2 is displayed on the display unit 5 in real time by the result display means 44. Further, the control means 4 initializes the result accumulating unit 32 that accumulates the recognition result of each object recognition in the optimal image search process, and deletes the recognition result used in the previous optimal image search process ( Step S1).

  Next, the image acquisition unit 41 acquires a captured image captured by the camera 2 (step S2: image acquisition step). Here, the image acquisition means 41 acquires a captured image at a predetermined fixed imaging cycle, and performs the following steps S3 to S5 each time each captured image is acquired. In addition, as an imaging | photography period, the period set suitably by the user may be used and the period set beforehand may be used. In the present embodiment, in order to update the display of the display unit 5 in real time, it is set to about 10 seconds, for example.

Step S3 is an image search step of the present invention, and object recognition is performed using the photographed image acquired in step S2. In step S3, as described above, the image search unit 42 calculates the feature amount of the captured image acquired by the image acquisition unit 41, and the search target stored in the image database 31 is the feature amount of the captured image. The similarity of each search target image is obtained from the feature amount of the image. Then, the recognition results (the search target image and the similarity of the search target image to the captured image) searched in step S3 are stored and stored in the result storage unit 32.
Here, in this step S3, the search exclusion means 47 determines whether or not there is a search target image whose display rate exceeds the threshold in the search target images searched by object recognition. Then, if there is an image whose display rate exceeds the threshold among the searched search target images, the search excluding unit 47 excludes the search target image from the recognition result.
Accordingly, as the number of searches in object recognition in step S3, it is necessary to search for at least N + α search target images to be displayed on the display unit 5 in consideration of the number of search target images to be excluded. That is, extra search target images are acquired so that the search target images to be displayed are not shorted by excluding the search target images.

In step S4, the certainty factor is calculated based on the recognition result accumulated in the result accumulation unit 32 after step S3.
In this step S4, the certainty factor calculation means 43 calculates the average value of the similarity in each search target image i as shown in the following formula (1) based on the plurality of recognition results accumulated in the result accumulation unit 32. Calculated as certainty factor X (i).

In said formula (1), each code | symbol has the following meaning.
I = {A, B, C...}: Set of search target images A (k, i): similarity of the search target image i to the captured image in the kth object recognition

  Step S5 is a result display step of the present invention. Based on the certainty factor calculated in step S4, search target images corresponding to the number of displays with high certainty factor (for example, four) are displayed as the certainty factor upper image 52. 5 is displayed. At this time, the result display means 44 displays the certainty level upper image 52 in the lower area of the display unit 5 in such a way that the certainty level decreases in the direction from the most significant display position 52A to the least significant display position 52B.

  Further, in this step S5, the display rate calculation means 46 calculates the total time when the search target image is displayed by the result display means 44 and the display time when each search target image is displayed as the certainty factor upper image 52. Based on this, the display rate of each search target image is calculated. In other words, the display rate calculation means 46 calculates a display rate indicating how much each of the higher certainty factor images 52 is displayed on the display unit 5 after the series of optimum image search processing has been performed.

  Thereafter, the control unit 4 determines whether or not the next captured image has been acquired by the image acquisition unit 41 (step S6). In step S6, when the next captured image is acquired by the image acquisition unit 41, the process returns to step S2. That is, object recognition using the captured images in steps S2 to S5 and calculation of the certainty factor are performed, and the certainty factor upper image 52 based on the updated certainty factor is displayed.

In step S6, when the next captured image is not acquired by the image acquisition unit 41, the control unit 4 determines whether or not the image selection unit 45 has selected the higher confidence level image 52 based on the user's operation ( Step S7).
When the certainty factor upper image 52 is selected by the image selection unit 45 in this step S7, the control unit 4 displays detailed information on the selected certainty factor upper image 52 on the display unit 5 and provides it. A series of optimum image search processing is terminated (step S8).

On the other hand, if it is determined in step S7 that there is no selection by the user, it is determined whether the display rate of each certainty factor upper image 52 displayed in step S5 is equal to or greater than a threshold value (step S9).
That is, in this embodiment, when the display rate of the certainty factor upper image 52 is equal to or greater than the threshold value, the result display unit 44 deletes the certainty factor upper image 52 from the display unit 5 and confirms a new search target image. The upper image 52 is displayed.
The process in step S9 is performed after a preset set time has elapsed since the optimum image search process was started. That is, in the series of optimum image search processing, the certainty factor upper image 52 displayed on the display unit 5 for the first time has a display rate of 1. If such a higher certainty factor image 52 is deleted from the display unit 5 based on the display rate and is also excluded from the recognition result, the user may select if the higher certainty factor image 52 is the target image. Can not. In addition, if the display disappears from the display unit 5 immediately after the certainty level upper image 52 is displayed on the display unit 5, there arises a problem that the user cannot sufficiently confirm the certainty level upper image 52. On the other hand, in the present embodiment, the certainty factor is high even when the display rate of the certainty factor upper image 52 is high until the set time elapses after the optimum image search process is started. The upper image 52 is displayed on the display unit 5. Thereby, the above problems can be avoided and sufficient confirmation time for the certainty factor upper image 52 can be given to the user.

In step S9, when the display rate of each certainty factor upper image 52 displayed on the display unit 5 is smaller than the threshold value, the result display unit 44 causes the display unit 5 to continue displaying these certainty factor upper images 52. Then, the process returns to step S6.
On the other hand, in step S <b> 9, when there is a certainty factor upper image 52 whose display rate is equal to or higher than the threshold value among the certainty factor upper images 52 displayed on the display unit 5, the result display unit 44 displays the certainty factor upper image 52. The display is erased, and the search target image having the next highest certainty factor after the certainty factor upper image 52 is moved to the display position of the erased certainty factor upper image 52 and displayed (step S10). That is, the result display unit 44 updates the display ranking by excluding the certainty level upper image 52 whose display rate exceeds the threshold value and sequentially displaying the search target images on the display unit 5 based on the certainty level.
In step S <b> 10, the search exclusion unit 47 excludes the deleted certainty factor upper image 52 (search target image) from the recognition result of the result storage unit 32.
Thereafter, the process returns to the determination process in step S6 again.

[Operational effects of the first embodiment]
The image search device 1 according to the first embodiment described above can achieve the following operational effects.
When the captured image is acquired by the image acquisition unit 41, the image search unit 1 uses the acquired captured image to search the image database 31 for a search target image similar to the captured image. Then, the result is accumulated in the result accumulation unit 32. In addition, the certainty factor calculation unit 43 calculates the certainty factor for each search target image based on the recognition result accumulated in the result accumulation unit 32. Then, the result display unit 44 causes the display unit 5 to display a search target image having a high certainty factor among the plurality of searched search target images as the high certainty factor image 52 until the predetermined determination time elapses. In addition, when the certainty factor upper image 52 is not selected by the image selection unit 45, the display ranking of the certainty factor upper image 52 displayed on the display unit 5 is updated.

In such a configuration, when the initial certainty factor is given to the retrieval target image searched by the image search unit 42 for the photographed image, and the certainty factor is updated for each photographing cycle, the result display unit 44. Updates the certainty level upper image 52 to be displayed on the display unit 5 based on the updated degree of update. For this reason, in the search processing based on the similarity by one object recognition, even the search target image that is not displayed on the display unit 5 is displayed based on the certainty calculated based on the past recognition result. Thus, the possibility of being displayed on the display unit 5 is increased. Further, since the display ranking of the certainty factor upper image 52 displayed on the display unit 5 is updated as time passes, the search target image that has not been displayed on the display unit 5 is displayed on the display unit 5 by the update. The possibility of being further increased.
As described above, the probability that the target image desired by the user is displayed on the display unit 5 as the certainty factor upper image 52 can be increased. That is, it is possible to perform an optimum image search process in which the target image desired by the user is easily displayed.

In addition to this, the result display unit 44 deletes the certainty factor upper image 52 in which the certainty factor upper image 52 is not selected by the image selecting unit 45 and the display rate exceeds a predetermined upper limit value from the display unit 5, Next, the search target image with the highest certainty factor is displayed on the display unit 5 as the certainty factor upper image 52.
In such a configuration, an image that is not selected by the user can be erased from the display unit 5 as not being a target image. Therefore, the target image requested by the user can be displayed on the display unit 5 more quickly.
In addition, it is preferable to delete the display of the certainty factor upper image 52 by the result display unit 44 after a predetermined set time has elapsed since the optimum image search process was started. That is, immediately after the start of the optimum image search process, the elapsed time is small, so that the display rate of each certainty factor upper image 52 is also increased. On the other hand, by providing the set time as described above, the user's operation time can be sufficiently secured.

In the optimum image search process, the image acquisition unit 41 acquires a captured image for each shooting period, and the image search unit 42 performs object recognition for each of these captured images. And the certainty factor calculation means 43 calculates the certainty factor with respect to each search object image based on the recognition result of each of these object recognition.
In such a configuration, object recognition is performed on each of a plurality of photographed images photographed with respect to the same object, and a certainty factor that is comprehensively considered is calculated using the recognition results of these object recognitions. By performing display based on such a certainty factor, it is possible to display a higher certainty factor image 52 that is more similar to the object desired by the user than the recognition result based on the similarity of one object recognition.

Here, the certainty factor calculating unit 43 calculates the certainty factor based on the similarity A (k, i) of the search target image with respect to the photographed image and the number n of object recognition.
In such a certainty factor calculation method, it is possible to increase the certainty factor of a search target image having a large number of searches, and it is highly probable that the object desired by the user is included based on the similarity degree based on a series of search results for the photographed image. The detection result can be displayed as the certainty factor upper image 52.
In addition, the certainty factor may be a value that is simply calculated based on, for example, a similarity ranking, but the similarity ranking is data in which search target images searched by object recognition are arranged in order of similarity. For this reason, for example, when the similarity of all the search target images searched is low, the similarity may be low even in the search target image having a higher similarity ranking. On the other hand, when the certainty factor is obtained based on the similarity, the above-described problem does not occur, and a more reliable certainty factor can be calculated.

In the present embodiment, the result display unit 44 deletes the certainty factor upper image 52 in which the display rate calculated by the display rate calculating unit 46 exceeds the threshold value, and the search exclusion unit 47 deletes the certainty factor high level erased. Although the image 52 is excluded from the result accumulation unit 32, the present invention is not limited to this.
For example, when there is a certainty factor upper image 52 whose display rate exceeds the threshold, first, the certainty factor upper image 52 (search target image) is excluded from the recognition results accumulated in the result accumulation unit 32 by the search exclusion unit 47. The certainty factor calculation means 43 recalculates the certainty factor based on the recognition result of the result accumulation unit 32, and then the result display unit 44 displays the certainty factor upper image 52 on the display unit 5 based on the calculated certainty factor. It is also possible to display and update the display. Even in this case, recognition result display similar to that in the first embodiment can be performed.

[Second Embodiment]
In the image search device 1 of the first embodiment, the result display unit 44 uses the display rate of the certainty factor upper image 52 displayed on the display unit 5 based on the display rate calculated by the display rate calculation unit 46 as a threshold value. Is exceeded, the higher certainty factor image 52 is deleted from the display unit 5, and a new search target image is moved up and displayed on the display unit 5. On the other hand, in the image search device of the second embodiment, the result display means is convinced by the image selection means after a certain determination time elapses after the high confidence level image is displayed on the display unit. The difference is that when the higher-order image is not selected, the display of the certainty-higher-order image is deleted and a new search target image is displayed.

Since the image search device of the second embodiment has substantially the same configuration as the image search device 1 of the first embodiment, the second embodiment will be described below with reference to FIG.
Note that in the image search device 1 of the present embodiment, as a functional configuration of the control unit 4, the display rate calculation unit 46 may not be provided.
In the present embodiment, the result display unit 44 displays the image to be captured 51 captured by the camera 2 on the entire display unit 5 as in the first embodiment, and the higher certainty factor image in descending order of the certainty factor. 52. Further, the result display unit 44 changes the order of the certainty factor upper images 52 displayed on the display unit 5 when the certainty factor is updated by the certainty factor calculating unit 43 when the certainty factor rank is changed.

Then, the result display means 44 is operated by the image selection means 45 between the timing when the certainty factor upper image 52 is displayed on the display unit 5 and the elapse of a predetermined determination time (for example, 30 seconds). When the certainty factor upper image 52 is not selected, the certainty factor upper image 52 is excluded and the certainty factor upper image 52 is updated.
That is, the result display unit 44 switches between display and non-display (erase) of the certainty factor upper image 52 based on the time from when the certainty factor upper image 52 is displayed on the display unit 5 instead of the display rate. For this reason, in this embodiment, it is not necessary to provide the display rate calculation means 46 as mentioned above.

The image search method of the image search apparatus 1 according to the second embodiment as described above will be described with reference to the drawings.
FIG. 5 is a flowchart showing an operation in the image search apparatus 1 of the second embodiment.
As shown in FIG. 5, in the image search device 1 of the second embodiment, the optimum image search process is performed by a process substantially similar to that of the first embodiment, and “No” is determined in step S6. In addition, step S11 is performed instead of step S7.

That is, in this step S11, the result table means 44 measures the time from when each higher confidence level image 52 is displayed when it is determined in step S6 that there is no captured image acquired by the image acquisition means 41. To do. Then, it is determined whether or not the certainty level upper image 52 has been selected by the user's selection operation until the determination time has elapsed.
In step S11, if the certainty factor upper image 52 is not selected before the determination time elapses, step S10 is performed, and the result display unit 44 excludes the certainty factor upper image 52. Then, the display ranking of the certainty factor upper image 52 displayed on the display unit 5 is updated. That is, the result display unit 44 deletes the certainty factor upper image 52 that has not been selected by the image selecting unit 45 from the display unit 5 until the determination time elapses, and replaces the deleted certainty factor upper image 52 with the erased result. In addition, the search target image having the next highest certainty factor is displayed after the certainty factor upper image 52, and thereafter, the display position is sequentially raised and displayed.
On the other hand, if there is a selection operation by the user in step S11 and the certainty factor upper image 52 is selected by the image selection means 45, the processing in step S8 is performed.

[Operational effects of the second embodiment]
In the image search apparatus 1 according to the second embodiment as described above, the result display unit 44 has no user operation until the determination time elapses, and the image selection unit 45 selects the higher confidence level image 52. If there is not, the display ranking of the certainty factor upper image 52 to be displayed on the display unit 5 is updated by excluding the certainty factor upper image 52.
For this reason, since the image which is not selected by the user is excluded and the certainty level upper image 52 is updated, the probability that the target image is displayed on the display unit 5 more quickly can be increased.

[Third embodiment]
Next, a third embodiment according to the present invention will be described based on the drawings.
In the image search device 1 according to the second embodiment, the certainty factor upper image 52 by the image selecting unit 45 is displayed before the determination time elapses after the result display unit 44 displays the certainty factor upper image 52 on the display unit 5. When the image 52 is not selected, the result display unit 44 deletes the unreliable higher-rank image 52 from the display unit 5 and displays a new search target image on the display unit 5. On the other hand, in the image search device according to the third embodiment, the result display unit lowers the display position of the uncertainty higher image that is not selected when the higher-certainty image is not selected by the image selecting unit. The second embodiment is different from the second embodiment in that it is displayed at the rank display position.

Since the image search apparatus of the third embodiment has substantially the same configuration as the image search apparatus 1 of the first embodiment, the third embodiment will be described below based on FIG.
In the present embodiment, the image search device 1 selects an image from the timing when the certainty factor upper image 52 is displayed on the display unit 5 until a predetermined determination time (for example, 30 seconds) elapses. When the certainty factor upper image 52 is not selected by the means 45, the display rank of the certainty factor upper image 52 is lowered stepwise and the display is updated.

For example, as shown in FIG. 6A, the search target images of A, B, C, and D are displayed as the higher certainty factor image 52 in descending order of the certainty factor, and the result of performing object recognition multiple times. Assume that the certainty ranking has changed to B, E, F, and C. In this case, as shown in FIG. 6B, the result display unit 44 sets the display order of the certainty factor upper image 52 from the highest display position 52A to the lowest display position 52B as B, E, F, and C. , Update the display.
After the search target images B and C are displayed on the display unit 5, when a preset determination time has elapsed, the result display unit 44 sets the display position of the search target image B by the lowest display position for one rank. Move to the 52B side. That is, the display position of the search target image B and the display position of the search target image E are switched and displayed. Accordingly, when the search target image B is not selected by the user and is not selected by the image selection means 45, the display ranking is gradually lowered with time.
On the other hand, since the display position of the search target image C displayed at the lowest display position 52B cannot be moved by one rank, the result display unit 44 deletes the search target image C and continues to the search target image C. A search target image G having a high certainty factor is newly displayed at the lowest display position 52 </ b> B of the display unit 5.

Then, the search exclusion unit 47 excludes the search target image C deleted from the display from the recognition result of the result storage unit 32. As a result, the search target image C is excluded from the search target in the series of optimum image search processes, and is stored in the result storage unit 32 even when the search target image C is searched in the subsequent object recognition by the image search means 42. Since the certainty factor is not calculated, it is not displayed on the display unit 5.
If the search target image B is not continuously selected by the image selection means 45, the display position gradually moves to the lowest display position 52B side with the passage of time, and is finally erased from the display unit 5, It will be excluded from the search target.

Note that the result display unit 44 may perform a process of not deleting the search target image displayed at the lowest display position 52B, for example, only when the first determination time has elapsed.
That is, among the certainty factor upper images 52 displayed on the display unit 5, the certainty factor upper images 52 displayed at positions other than the lowest display position 52B are excluded only by the display position moving even after the determination time has elapsed. It will never be done. As a result, even when the determination time has elapsed due to an oversight of the user or the like, these higher certainty factor images 52 are displayed on the display unit 5, and the user can check these higher certainty factor images 52. . Therefore, for example, when these certainty factor upper images 52 are target images, the user can select a target image.
On the other hand, the certainty factor upper image 52 displayed at the lowest display position 52B is immediately deleted after the first determination time has elapsed. In this case, when the determination time has elapsed due to an oversight of the user or the like, the certainty factor upper image 52 displayed at the lowest display position 52B is excluded from the search target, and thus is not displayed as a recognition result. . On the other hand, for example, when the first determination time has elapsed, by executing a process of continuously displaying the certainty factor upper image 52 displayed at the lowest display position 52B without deleting it, the certainty factor upper image is displayed to the user. 52 confirmation can be prompted.

[Operational effects of the third embodiment]
In addition to the effects of the second embodiment, the image search device 1 of the third embodiment as described above has the following effects.
When there is no user operation and the higher certainty factor image 52 is not selected by the image selecting unit 45, the result display unit 44, when the display position of the higher certainty factor image 52 is other than the lowest display position 52B, The display position is moved to the lowest display position 52B side, and when the display position of the higher certainty factor image 52 is the lowest display position 52B, the display position 5 is deleted. Accordingly, the certainty factor upper image 52 displayed on the display unit 5 is not immediately deleted after the determination time has elapsed, and the ranking on the display gradually decreases, and is finally deleted from the display unit 5. Become.
For this reason, for example, even when the user has not seen the display on the display unit 5 by looking aside, it is possible to avoid the inconvenience that the higher certainty factor image 52 is immediately deleted. Therefore, sufficient time for the user to select the certainty level upper image 52 can be given, and a comfortable operation can be supported.

[Fourth embodiment]
Next, the image search apparatus of 4th embodiment of this invention is demonstrated based on drawing.
In the first to third embodiments, the certainty factor upper image 52 is selected by the image selection means 45 until the predetermined determination time elapses after the certainty factor upper image 52 is displayed on the display unit 5. In the case where there is not, the example in which the certainty factor upper image 52 is excluded is shown. On the other hand, in the image search apparatus of the fourth embodiment, when the certainty factor upper image is not selected by the image selecting unit, the certainty factor calculating unit selects the certainty factor upper image (search target image) that has not been selected. ) Is different in that it reduces the certainty.

Since the image search device of the fourth embodiment has substantially the same configuration as the image search device 1 of the first embodiment, the fourth embodiment will be described below based on FIG.
In the image search device 1 of the present embodiment, the display unit calculating unit 46 and the search excluding unit 47 may not be provided as a functional configuration of the control unit 4.
As in the first to third embodiments, the result display unit 44 of the image search device 1 according to the present embodiment displays the ranking of the updated certainty factor every time the certainty factor is updated by the certainty factor calculating unit 43. Accordingly, the order of the certainty factor upper images 52 to be displayed on the display unit 5 is changed.
In addition, the result display unit 44 of the present embodiment does not change the display order of the certainty factor upper image 52 on the display unit 5 when there is no change in the certainty factor. Therefore, the certainty level upper image 52 is displayed by the image selection means 45 between the timing when the higher certainty factor image 52 is displayed on the display unit 5 and the elapse of a predetermined determination time (for example, 30 seconds). Even if it is not selected, if there is no change in the ranking of the certainty factor, the display order of the certainty factor upper image 52 is not changed, and the certainty factor upper image 52 is not excluded from the display unit 5.

On the other hand, the certainty factor calculation unit 43 of the present embodiment performs the following processing in addition to calculating the certainty factor of each search target image based on the recognition result accumulated in the result accumulation unit 32.
That is, the certainty factor calculating means 43 is the image selecting means 45 for a predetermined determination time (for example, 30 seconds) set in advance from the timing when the certainty degree upper image 52 is displayed on the display unit 5 by the result display means 44. When the certainty factor upper image 52 is not selected by the above, processing is performed to reduce the certainty factor of the certainty factor upper image 52 (search target image).
Here, in the present embodiment, the certainty factor calculation unit 43 fixes the certainty factor of the certainty factor upper image 52 (search target image) that has not been selected by the image selection unit 45 to “0”, which is the lowest value.
In addition, since the result display unit 44 updates the display of the display unit 5 at the timing when the certainty factor is updated, the certainty factor upper image 52 whose certainty factor is updated to the lowest value by the certainty factor calculating unit 43 is displayed. It will be deleted from the part 5 and the certainty level upper image 52 based on the new certainty level will be displayed. This makes it possible to display the image recognition result as shown in FIG.
Here, the certainty factor calculating means 43 has the lowest certainty factor of the certainty factor upper image 52 (search target image) that has not been selected by the image selecting means 45 until the determination time elapses. However, the minimum value may be fixed to a negative value, for example. As a result, the search target image that has not been selected by the user is reliably not displayed on the display unit 5.

[Operation of image search device]
Next, an image search method of the image search device 1 according to the fourth embodiment will be described.
In the image search apparatus 1 of the present embodiment, an optimal image search process is performed by a process substantially similar to that of the second embodiment, and an image search method will be described with reference to FIG.
In the second embodiment, in step S10 of FIG. 5, the result display unit 44 excludes from the display without changing the certainty factor of the certainty factor upper image 52 that has passed the determination time and has not been selected by the user. The search exclusion means 47 excludes the recognition result from the recognition result.
On the other hand, in step S10 of the present embodiment, the certainty factor calculation unit 43 sets the certainty factor of the certainty factor upper image 52 that has passed the determination time and has no selection operation by the user to the lowest value. Then, the result display unit 44 causes the display unit 5 to display the top four search target images ranked in descending order of the certainty factor based on the updated certainty factor as the certainty factor upper image 52.

[Effects of Fourth Embodiment]
In the image search device 1 according to the fourth embodiment as described above, the result display means 44 has no user operation until the determination time elapses, and the image selection means 45 selects the higher confidence level image 52. If not, the certainty factor calculating unit 43 fixes the certainty factor of the certainty factor upper image 52 to the lowest value, and the result display unit 44 updates the certainty factor upper image 52 based on the certainty factor after the certainty factor is updated. Is displayed on the display unit 5.
For this reason, as in the first to third embodiments, since the images that are not selected by the user are excluded and the certainty level upper image 52 is updated, the probability that the target image is displayed on the display unit 5 can be increased. .

[Fifth embodiment]
Next, a fifth embodiment according to the present invention will be described with reference to the drawings.
In the image search device 1 of the fourth embodiment, the certainty level upper image 52 is displayed on the display unit 5 by the result display unit 44, and the certainty by the image selection unit 45 before the predetermined determination time elapses. When there is no selection of the higher degree image 52, the certainty factor calculation means 43 sets the certainty factor of the higher certainty factor image 52 that has not been selected to the lowest value. On the other hand, in the image search device of the fifth embodiment, the certainty factor changing unit calculates the certainty factor of the certainty factor upper image not selected when the higher certainty factor image is not selected by the image selecting unit. It differs from the fourth embodiment in that it is lowered by a predetermined set value.

Since the image search device of the fifth embodiment has substantially the same configuration as the image search device 1 of the first embodiment, the fifth embodiment will be described below based on FIG.
The certainty factor calculating means 43 of the image search device 1 of the present embodiment starts from a timing when the certainty factor upper image 52 is displayed on the display unit 5 by the result display means 44, and a predetermined determination time (for example, 30 seconds) set in advance. If the higher certainty factor image 52 is not selected by the image selection means 45, the certainty factor of the higher certainty factor image 52 (search target image) is decreased by a predetermined set value (for example, 0.5). Fix it.
Here, as the set value, it may be set in advance and stored in the storage means. For example, a value that can be appropriately set and changed by the user may be used.
In the present embodiment, the certainty factor is reduced by the certainty value by the certainty factor calculating unit 43, so that the certainty factor of the search target image not selected by the image selecting unit 45 is gradually reduced. Therefore, when the display of the display unit 5 is updated by the result display unit 44 every time the certainty factor is updated, the display position of the certainty factor upper image 52 that is not selected by the image selecting unit 45 is gradually lowered to the lowest display position 52B. Move to the side. Finally, the certainty factor of the certainty factor upper image 52 is not displayed on the display unit 5 because the certainty factor of the other search target images not displayed on the display unit 5 is smaller. That is, substantially the same display as the display example shown in FIG. 6 is performed, and the display ranking of the certainty factor upper image 52 not selected by the user is lowered step by step.

[Effects of Fifth Embodiment]
In the image search device 1 of the fifth embodiment as described above, it is possible to achieve substantially the same operational effects as the third embodiment.
That is, the certainty factor calculation unit 43 of the image search apparatus 1 sets the certainty factor of the certainty factor upper image 52 to a predetermined value when there is no user operation and the certainty factor upper image 52 is not selected by the image selection unit 45. Fixed to a value reduced by the value. For this reason, as time elapses, the certainty factor of the certainty factor upper image 52 that has not been selected by the image selection unit 45 gradually decreases. Therefore, every time the certainty factor is updated, the display of the display unit 5 is updated by the result display unit 44, whereby the display position of the certainty factor upper image 52 gradually moves to the lowest display position 52B side. Eventually, it is not displayed on the display unit 5.
In such a configuration, since the certainty level upper image 52 displayed on the display unit 5 is not immediately deleted after the determination time has elapsed, for example, the user does not see the display on the display unit 5 by looking aside. Even in this case, it is possible to avoid the inconvenience that the higher certainty factor image 52 is immediately deleted. Therefore, sufficient time for the user to select the certainty level upper image 52 can be given, and a comfortable operation can be supported.

[Sixth embodiment]
Next, the image search apparatus of 6th embodiment of this invention is demonstrated based on drawing.
In the image search apparatuses 1 and 1 of the first to fifth embodiments described above, the result display means 44 is arranged in the lower area of the display unit 5 from the highest display position 52A to the lowest display position 52B as shown in FIG. The certainty level upper image 52 of the same size was displayed in descending order of the certainty level. On the other hand, the sixth embodiment is different in that the display form is changed when the order of the certainty-higher-order images displayed by the result display means is changed.

FIG. 7 is a diagram illustrating a display example of the display unit in the sixth embodiment.
Moreover, since the image search device of the sixth embodiment has substantially the same configuration as the image search device 1 of the first embodiment, the sixth embodiment will be described below based on FIG.
As in the first to fifth embodiments, the result display unit 44 of the image search device 1 of the present embodiment displays the shooting target image 51 and the certainty factor upper image 52.
Here, the result display unit 44 of the present embodiment deletes the certainty factor upper image 52 when the certainty factor is newly calculated by, for example, the certainty factor calculating unit 43, or deletes the certainty factor upper image 52 in which the display rate exceeds the threshold. In the case of displaying the target image, the display size of the certainty factor upper image 52 displayed on the display unit 5 is changed.

  For example, in step S5 in FIG. 4, when the certainty factor of each search target image is calculated by the certainty factor calculating unit 43, if the ranking of the certainty factor varies, the result display unit 44 determines that the rank has increased. The higher-order image 52 is enlarged and displayed as shown in FIG. On the other hand, the result display means 44 reduces and displays the certainty factor upper image 52 whose rank has been lowered.

  Further, in step S10 in FIG. 4, if the certainty factor upper image 52 is not selected by the image selecting unit 45, the result display unit 44 erases the certainty factor upper image 52, and removes the higher certainty factor image. The search ranking image of the search target image with a low certainty factor is displayed in a higher order sequentially. At this time, the higher certainty factor image 52 to be displayed in a higher order is displayed in an enlarged manner. Further, the result display means 44 may display the higher certainty factor image 52 to be newly displayed at the lowest display position 52B with a larger display size.

[Operational effects of the sixth embodiment]
The image display device according to the sixth embodiment has the following functions and effects in addition to the functions and effects of the above embodiments.
That is, the result display means 44 increases the display size for the higher confidence level image 52 whose rank has been increased and the display size for the higher confidence level image 52 whose rank has decreased when the ranking of the confidence level has changed. Is displayed smaller. For this reason, the user can confirm how the ranking of the certainty factor fluctuates by confirming the display size of each displayed certainty factor upper image 52.
Further, when there is a new certainty level upper image 52 displayed on the display unit 5, the new certainty level upper image 52 displayed can be highlighted. As a result, the user's eyes are directed to the newly displayed high confidence level image 52 having a large display size, and the user can satisfactorily support the confirmation work of the high confidence level image 52.

  In the present embodiment, the example in which the display mode is changed and displayed on the result display unit 44 of the first embodiment has been described. However, the result display unit 44 of the second to fifth embodiments similarly displays the same. The display form of the certainty factor upper image 52 may be changed.

[Seventh embodiment]
Next, an image search system according to a seventh embodiment of the present invention will be described with reference to the drawings.
In the first to sixth embodiments, the image database 31 is stored in the storage unit 3 of the image search device, and the device for searching the search target image corresponding to the photographed image from the image database 31 is exemplified. On the other hand, in the seventh embodiment, an image database is stored in a storage unit provided in the server device, and the server device performs object recognition using a captured image transmitted from the terminal device, and selects a search target. It is different from the first to sixth embodiments in that it is transmitted to the image terminal device again.

[Image search system configuration]
FIG. 8 is a diagram showing a schematic configuration of an image search system according to the seventh embodiment. In the description of the seventh embodiment, the same reference numerals are given to the configurations described in the first to sixth embodiments, and the description thereof is omitted or simplified.
As illustrated in FIG. 8, in the image search system 100 according to the present embodiment, a terminal device 200 and a server device 300 are connected to be communicable via the Internet 110.
Here, in the present embodiment, the Internet 110 based on a general-purpose protocol such as TCP / IP is exemplified as the network, but is not limited thereto. For example, any configuration that transmits and receives data can be used, such as a network such as a communication network in which a plurality of base stations that can transmit and receive information via a wireless medium constitute a network.

[Configuration of terminal device]
As the terminal device 200, for example, a general-purpose computer such as a personal computer can be used in addition to mobile devices such as a mobile phone, a smartphone, and a mobile computer. The terminal device 200 includes a camera 2, a display unit 5, an operation unit 6, a storage unit 210, and a control unit 220 as a hardware configuration.

  The storage unit 210 stores a RAM (Random Access Memory) having a temporary area where data can be temporarily stored, an OS (Operating System: basic software), programs for controlling various servers, various applications, various data, and the like. ROM (Read Only Memory) or the like. The storage unit 210 includes a result storage unit 32 that stores the recognition result of the object recognition transmitted from the server device 300.

The control means 220 includes an integrated circuit such as a CPU and a communication means for communicating with the server device 300 via the Internet 110. The control unit 220 reads a program stored in the storage unit 210 and performs various processes according to the program.
Then, the control unit 220 includes, as functional configurations, an image acquisition unit 41, a captured image transmission unit 221, a result reception unit 222, a certainty factor calculation unit 43, a result display unit 44, an image selection unit 45, a display. The rate calculating means 46 and the search exclusion means 47 are provided.

  The captured image transmission unit 221 transmits the acquired captured image to the server device 300 via the Internet 110 every time the captured image is acquired by the image acquisition unit 41. At this time, the captured image transmission unit 221 transmits the captured image together with the user ID for specifying the terminal device 200.

  The result receiving unit 222 receives the recognition result of the object recognition for the captured image transmitted from the server device 300. As in the first embodiment, the recognition result includes a search target image having a high degree of similarity to the captured image and the similarity of the search target image. The result receiving unit 222 stores the received recognition result of the object recognition in the result storage unit 32 of the storage unit 210.

[Configuration of server device]
As shown in FIG. 9, the server device 300 includes a storage unit 310 and a control unit 320 as a hardware configuration.
The storage means 310 includes a RAM having a temporary area where data and the like can be temporarily stored, a ROM that stores programs for controlling the OS and various servers, various applications, various data, and the like.
The storage unit 310 stores an image database 31 for performing object recognition on a captured image.

The control unit 320 includes an integrated circuit such as a CPU and a communication unit for communicating with the terminal device 200 and the like via the Internet 110. In addition, the control unit 320 reads a program stored in the storage unit 310 and performs various processes according to the program.
And the control means 320 is provided with the image receiving means 321, the image search means 42, and the search result transmission means 322 as a function structure.

The image receiving unit 321 receives a captured image transmitted from the terminal device 200 via the Internet 110.
The image search means 42 is the same as the image search means 42 in the first embodiment, and uses the captured image received by the image receiving means 321 to search from the image database 31 of the storage means 310 and similar to the captured image. Search for an image.
The search result transmission unit 322 transmits the search target image and the similarity as the recognition result of the object recognition by the image search unit 42 to the terminal device 200 via the Internet 110. At this time, the search result transmission unit 322 transmits the search result to the terminal device 200 that is the transmission source of the captured image based on the user ID received together with the captured image by the image reception unit 321.

[Image Search Method in Image Search System]
Next, an image search method (optimum image search process) in the image search system 100 as described above will be described with reference to the drawings.
FIG. 9 is a flowchart of optimal image search processing in the image search system.

  As shown in FIG. 9, the optimum image retrieval process by the image retrieval system 100 is performed by substantially the same method as the optimum image retrieval process by the image retrieval apparatus 1. Accordingly, the same steps as those in the optimum image search process of the image search apparatus 1 are denoted by the same reference numerals, and the description thereof is omitted or simplified.

When a user who operates the terminal device 200 starts a program for executing the optimum image search process, the camera 2 is started, and an image captured by the camera 2 is displayed on the terminal device 200 in real time by the result display unit 44. Displayed in part 5. And the control means 220 of the terminal device 200 implements step S1, and initializes the result storage part 32. FIG.
Next, step S <b> 2 is performed, and the image acquisition unit 41 acquires a captured image captured by the camera 2.

Thereafter, the captured image transmission unit 221 of the terminal device 200 transmits the captured image acquired in step S2 to the server device 300 via the Internet 110 together with the user ID (step S21).
When the server device 300 acquires the captured image transmitted from the terminal device 200 (step S22), the server device 300 performs the same processing as that of step S3 in FIG. 4, and retrieves a search target image similar to the captured image acquired in step S22. A search is made from the image database 31 of the storage means 310.
And the search result transmission means 322 of the server apparatus 300 transmits the recognition result of step S2 to the terminal device 200 via the internet 110 (step S23).

When the terminal device 200 receives the recognition result transmitted from the server device 300 by the result receiving unit 222 (step S24), the terminal device 200 stores the received recognition result in the result storage unit 32. Thereafter, the terminal device 200 performs each process of step S4 to step S10 in the image search device 1 of the first embodiment, and gives and updates the certainty factor, and displays the certainty factor upper image 52 based on the certainty factor. Do.
In the display of the higher certainty factor image 52, the result display means 44 displays the higher certainty factor image 52 whose confidence factor rank has changed as different from the other higher certainty factor images 52 as in the sixth embodiment. It may be displayed in a form.

[Operation and effect of the seventh embodiment]
In the seventh embodiment, the same effects as those of the first embodiment can be obtained.
That is, in the image search system 100 of the present embodiment, the terminal device 200 stores the recognition result of object recognition transmitted from the server device 300 in the result storage unit 32, and each search target is based on this stored recognition result. Calculate the certainty of the image. And the result display means 44 of a terminal device displays the certainty degree upper image 52 on the display part 5 in order with a large certainty degree. When the certainty level upper image 52 is not selected by the image selection unit 45 after the certainty level upper image 52 is displayed on the display unit 5 until the predetermined determination time elapses, the result display unit 44 The certainty factor upper image 52 is excluded, and the certainty factor upper image 52 is updated based on the certainty factor.
For this reason, it is possible to increase the probability that the target image desired by the user is displayed on the display unit 5 as the certainty factor upper image 52.

In the image search system 100 of the present embodiment, the captured image acquired by the terminal device 200 is transmitted to the server device 300, the object recognition is performed by the server device 300, and the result is transmitted to the terminal device 200. In such a configuration, for example, even when a mobile device such as a mobile phone or a smartphone having a limited storage capacity of the storage unit 210 is used as the terminal device 200, the large-capacity storage unit 310 provided in the server device 300 is used. A search target image can be searched from the image database 31. Therefore, the target image desired by the user can be searched from the image database 31 in which more search target images are stored, and the search accuracy can be further improved.
Further, the image search means 42 of the server device 300 only needs to search for a search target image based on the similarity by an algorithm similar to that of a conventional search engine. Therefore, a new algorithm is used for the optimum image search processing of the present invention. There is no need to reassemble the search engine, and the configuration can be simplified.

In the present embodiment, the result display unit 44 updates the display ranking of the certainty factor upper image 52 displayed on the display unit 5 based on the display rate. For example, as shown in the second to fifth embodiments. Alternatively, the display ranking may be updated based on the determination time.
In addition, the result display unit 44 is configured to immediately exclude the certainty level upper image 52 that is not selected by the user when the display ranking is lowered. For example, as in the third embodiment, the result display unit 44 , A process of gradually reducing the display ranking of the certainty level upper image, a process of setting the confidence level to the lowest value as in the fourth embodiment, and a level of confidence level as in the fifth embodiment Processing may be performed.

[Eighth embodiment]
Next, an eighth embodiment of the present invention will be described based on the drawings.
In the image search system 100 according to the seventh embodiment, after the object recognition using the captured image is performed in the server device 300, the object recognition result is transmitted to the terminal device 200, and the terminal device 200 calculates the certainty factor. went. On the other hand, in the image search system of the eighth embodiment, the server device 300 performs the object recognition using the photographed image and the calculation of the certainty factor using the recognition result of the object recognition. This is different from the image search system 100 of the seventh embodiment.
In the description of the eighth embodiment, the same reference numerals are given to the same configurations as those in the first to seventh embodiments, and the description thereof is omitted or simplified.

[Image search system configuration]
FIG. 10 is a diagram showing a schematic configuration of the image search system of the present embodiment.
As in the seventh embodiment, the image search system of the eighth embodiment includes a terminal device 200 and a server device 300, and these terminal device 200 and server device 300 can communicate via a network such as the Internet 110. It is connected to the.

[Configuration of terminal device]
The terminal device 200 of the present embodiment includes a camera 2, a display unit 5, an operation unit 6, a storage unit (not shown), and a control unit 220 as hardware configurations.

  The storage means includes a RAM having a temporary area where data and the like can be temporarily stored, a ROM for storing programs for controlling the OS and various servers, various applications, various data, and the like. In this embodiment, as shown in FIG. 10, the image database 31 and the result accumulation unit 32 are not provided in the storage unit.

  Then, as shown in FIG. 10, the control unit 220 in the terminal device 200 of the present embodiment includes, as a functional configuration, an image acquisition unit 41, a captured image transmission unit 221, a result reception unit 222, and a result display unit 44. And an image selecting means 45.

  The captured image transmission unit 221 transmits the acquired captured image to the server device 300 via the Internet 110 every time the captured image is acquired by the image acquisition unit 41. At this time, the captured image transmission unit 221 transmits the captured image together with the user ID for specifying the terminal device 200.

The result receiving unit 222 receives the recognition result of the optimum image search process for the captured image transmitted from the server device 300. Here, the recognition result is a search target image to which a certainty factor is given, and the result receiving unit 222 may acquire the search target images for the number of displays to be displayed on the display unit 5.
The result display unit 44 causes the display unit 5 to display the search target images acquired by the result reception unit 222 in the descending order of the certainty factor as the certainty factor upper image 52. Further, the result display unit 44 updates the certainty factor upper image 52 displayed on the display unit 5 only when the recognition result is received by the result receiving unit 222.

  The image selection unit 45 selects the certainty factor upper image 52 displayed on the display unit 5 based on the user's operation, and specifies the selected certainty factor upper image 52 or the selected certainty factor upper image 52. The image ID to be transmitted is transmitted to the server device as the selected image.

[Configuration of server device]
In the server apparatus 300 of the present embodiment, as shown in FIG. 10, the storage unit 310 includes a result accumulation unit 32 in addition to the image database 31.

  Further, as shown in FIG. 11, the control unit 320 of the server apparatus 300 of the present embodiment has an image receiving unit 321, an image search unit 42, a certainty factor calculation unit 43, and a search result transmission unit 322 as shown in FIG. 11. And a selection image receiving unit 323 and a search excluding unit 47.

  The search result transmitting unit 322 transmits the search target image whose certainty factor is calculated by the certainty factor calculating unit 43 and the certainty factor of the search target image to the terminal device 200 via the Internet 110 as a recognition result. At this time, the search result transmission unit 322 transmits search target images for the number of displays to be displayed on the display unit 5 of the terminal device 200 as the certainty factor upper image 52 in descending order of the certainty factor.

The selection image receiving unit 323 receives the selection image transmitted by the image selection unit 45 of the terminal device 200.
The search exclusion unit 47 measures the time after the recognition result is transmitted by the search result transmission unit 322. Then, the search exclusion means 47 is the search object that is the recognition result transmitted by the search result transmission means 322 when the selection image reception means 323 does not receive the selection image until the predetermined determination time elapses. The image is excluded from the result accumulation unit 32.

[Image Search Method in Image Search System]
Next, an image search method (optimum image search process) in the image search system as described above will be described with reference to the drawings.
FIG. 11 is a flowchart of optimal image search processing in the image search system.
In the description of the optimum image search process by the image search system of this embodiment, the steps described in the above embodiment are denoted by the same reference numerals, and the description thereof is omitted or simplified.

In the image search system of the present embodiment, when a user who operates the terminal device 200 starts a program for executing an optimal image search process, the camera 2 is started and an image captured by the camera 2 is displayed as a result display unit. 44 is displayed on the display unit 5 of the terminal device 200 in real time. And step S2 is implemented by the image acquisition means 41, and the picked-up image image | photographed with the camera 2 is acquired periodically. Further, every time step S <b> 2 is performed, step S <b> 21 is executed, and the captured image transmission unit 221 transmits the acquired captured image to the server apparatus 300 via the Internet 110.
At this time, the captured image transmission unit 221 transmits a user ID for determining the terminal device 200 to the server device 300 together with the captured image. Further, in each terminal device 200, when the number of displayed high confidence images 52 displayed on the display unit 5 is different, the captured image transmission unit 221 may transmit the display number of the certainty higher images 52 together with the captured images. Good.

On the other hand, when the optimum image processing is executed, the server device 300 first initializes the result storage unit 32 (step S31).
Then, the image receiving unit 321 of the server device 300 performs Step S22 and receives the captured image transmitted from the terminal device 200.
Thereafter, the image search means 42 performs the object recognition process (image search process) in step S3, and searches the image database 31 of the storage means 310 for a search target image similar to the photographed image. In addition, the image search unit 42 stores the recognition result recognized by the object recognition in the result storage unit 32.
Furthermore, the server apparatus 300 performs the process of step S4, and calculates the certainty factor for each search target image based on the past recognition results accumulated in the result accumulation unit 32.

Thereafter, the search result transmission unit 322 of the server device 300 performs Step S <b> 23 and transmits the recognition result to the terminal device 200. At this time, the search result transmission unit 322 selects, from the searched search target images, search target images corresponding to the display count of the higher certainty factor images 52 in descending order of the certainty factor, and selects the selected search target images. Send to.
Here, in this embodiment, in step S23, the certainty factor of the search target image is transmitted to the terminal device 200 together with the search target image.

When the terminal device 200 receives the search target image as the recognition result from the server device 300 in step S24, the terminal device 200 executes step S5, and the result display unit 44 uses the received search target image as the certainty level upper image 52. The display units 5 are displayed in descending order.
Thereafter, the control unit 220 of the terminal device 200 performs steps S6 to S7, whether or not the next captured image is acquired by the image acquisition unit 41, and whether or not the next captured image is acquired by the image selection unit 45. Determine whether or not.
In step S7, the terminal device 200 does not need to determine whether or not the selected image has been acquired before the predetermined determination time has elapsed. That is, when the certainty factor upper image 52 is not selected by the image selection unit 45, Steps S6 and S7 are repeated, and the determination of whether or not the captured image is acquired and whether or not the selected image is present is repeated.
In step S7, when the certainty factor upper image 52 is selected by the image selecting unit 45 based on the user's operation, the image selecting unit 45 stores the selected certainty factor upper image 52 (selected image) in the server. It transmits to the apparatus 300 (step S32).

  On the other hand, in step S23, the server apparatus 300 waits for reception of a selected image (standby state). When the selected image is transmitted in step S32, the server apparatus 300 receives the selected image (step S33). Further, the server device 300 determines whether or not the selected image has been received from the transmission timing of the recognition result in step S23 until the predetermined determination time has elapsed (step S34). In step S34, if the selection image received in step S33 is within the determination time, detailed information regarding the selection image is transmitted to the terminal device 200 (step S35). Thereby, the terminal device 200 performs the process of step S8 and provides the user with detailed information regarding the selected image.

In addition, in the determination process in step S34, the server apparatus 300 determines that the search exclusion unit 47 receives the result from the result accumulation unit 32 in step S23 when the selection image is not received in step S33 until the determination time elapses. The search target image of the recognition result transmitted in step S36 is excluded (step S36). Thereafter, the server device 300 returns to step S4, causes the certainty factor calculation means 43 to calculate the certainty factor again, and transmits the result to the terminal device 200 in step S23.
Thereby, the higher certainty factor image 52 that the user did not select during the predetermined determination time is not displayed on the display unit 5 of the terminal device 200, and the higher certainty factor image 52 based on the new search target image is displayed. Will be displayed. That is, it is possible to display the same display ranking as in the first, second, and fourth embodiments described above.
Moreover, in the process of step S36, the same display ranking display as the third and fifth embodiments can be performed by performing a process of gradually reducing the certainty factor of the search target image transmitted in step S23. it can.

[Effects of Eighth Embodiment]
In the image search system according to the present embodiment, the server device 300 calculates the certainty factor of each search target image based on the recognition result accumulated in the result accumulation unit 32, and the search targets for the number of displays in descending order of the certainty factor. The image is transmitted to the terminal device 200. Thereby, the display unit 5 of the terminal device 200 can display the high-accuracy certainty upper image 52 based on the past recognition result.
If the selected image is not transmitted from the terminal device 200 to the server device 300 after the recognition result is transmitted from the server device 300 to the terminal device 200 until the predetermined determination time elapses, the search exclusion unit 47 The search target image corresponding to the transmitted recognition result is excluded from the result accumulation unit 32. For this reason, it is possible to exclude the certainty level upper image that the user has not selected from the search target, and to provide the target image more quickly.

In the image search system of this embodiment, the same effect as that of the seventh embodiment can be obtained. For example, when the mobile device such as a mobile phone or a smartphone having a limited storage capacity is used as the terminal device 200. However, the search target image can be searched from the image database 31 of the large-capacity storage unit 310 provided in the server device 300. Therefore, the target image desired by the user can be searched from the image database 31 in which more search target images are stored, and the search accuracy can be further improved.
Furthermore, the terminal device 200 according to the present embodiment does not calculate the certainty factor, and displays the search target image to which the certainty factor that is the recognition result transmitted from the server device 300 is given as the certainty factor upper image 52 as the display unit 5. It is only displayed on the screen. For this reason, the functional configuration of the terminal device 200 can be simplified. For example, even in a mobile phone with low processing capability, the optimum image search with high detection accuracy and high probability of displaying a target image as described above. Processing can be performed.
[Modification]
In addition, this invention is not limited to embodiment mentioned above, In the range which can achieve the objective of this invention, the deformation | transformation shown below is also included.
For example, in the first to eighth embodiments described above, the certainty factor calculation means 43 is based on Equation (1) based on the similarity obtained by object recognition and the number of times the object recognition has been performed. However, the present invention is not limited to this.
For example, the certainty factor calculation means is based on the similarity ranking set based on the similarity of each search target image obtained by object recognition with the captured image and the number of times the search target image has been searched (Hitcount). The certainty factor may be calculated.
In this case, the certainty factor calculation means calculates the certainty factor based on, for example, the following equation (2).

[Equation 2]

In the above formula, each symbol has the following meaning.
I = {A, B, C ...} Set of search target images R = {R1, R2, R3 ..., Rn} Set of similar rankings (recognition results) RANK (r, i) Rank in search target image i Hitcount (i) Number of similar rankings including search target image i (number of times search target image i is searched (recognized))
In the certainty factor calculation method as described above, the certainty factor can be calculated more easily based on the similarity ranking, and the processing speed can be increased.

  Furthermore, the certainty factor may be calculated based on the following equation (3).

[Equation 3]

In the above formula, each symbol has the following meaning.
I = {A, B, C ...} Set of search target images F = {F1, F2, F3 ..., Fn} Set of object recognition results S (f, i) Search target in f-th search Similarity of image i to photographed image Hitcount (i) Number of times search target image i is searched Even when the certainty factor calculation method as described above is used, from the search result display based on the similarity by one search process. In addition, a highly accurate detection result can be displayed.

In addition, the certainty factor calculating unit 43 calculates the certainty factor based on the recognition results of the past object recognition accumulated in the result accumulating unit 32. In addition, the certainty factor addition value based on the recognition result may be added to the certainty factor assigned to the search target image.
In this case, the storage unit stores the certainty factor for each search target image, and the certainty factor changing unit updates the stored certainty factor each time the object recognition is performed. In addition, as the certainty factor addition value, for example, an addition value may be set for each rank of the similar ranking, and the addition value may be calculated based on the similarity.

  Furthermore, although the image acquisition means 41 showed the example which acquires a picked-up image automatically, it is good also as a structure which acquires a picked-up image based on a user's shutter operation. Moreover, it is good also as a structure which acquires a some frame image from the image | photographed moving image. Alternatively, a plurality of captured images may be acquired at one time, and object recognition may be performed on each of the captured images to calculate a certainty factor.

  In the above-described embodiment, the image search apparatus 1 and the image search system 100 that search for a search target image similar to the captured image from the search target image having the image of the search target object with respect to the captured image of the object are illustrated. However, the present invention is not limited to this. For example, an image such as a landscape that does not have an object to be captured may be acquired as a captured image.

  Moreover, in 1st embodiment, although the reliability high-order image 52 whose display rate exceeded the threshold value was excluded from the recognition result, the example which updates a display ranking was shown, but it is not limited to this. For example, the result display unit 44 may gradually reduce the display ranking of the certainty factor upper image 52 whose display rate exceeds the threshold value. In addition, the certainty factor calculating means 43 updates the certainty factor by setting the certainty factor of the certainty factor upper image 52 whose display rate exceeds the threshold value to the lowest value or gradually decreasing the predetermined setting value step by step, The result display unit 44 may update the display ranking of the certainty factor upper image 52 to be displayed on the display unit 5 based on the updated certainty factor.

  In 6th Embodiment, although the result display means changed and displayed the display size with respect to the certainty degree upper image 52 with which the ranking of the certainty degree fluctuated, it was not limited to this. That is, the result display means may use any display form as long as it can display the certainty level upper image 52 so that the fluctuation of the certainty level ranking can be clearly understood. For example, the result display means may change the display color, brightness, resolution, etc. of the certainty factor upper image 52 in which the certainty factor fluctuates, display the outer frame on the outer periphery of the certainty factor upper image 52, You may change the frame width dimension and color of the said outer frame according to the fluctuation | variation of the ranking of reliability, and the variation | change_quantity of reliability. In addition, the calculated certainty value may be superimposed on the higher certainty factor image 52 and displayed.

  Furthermore, the result display means changed the display form based on the ranking of the certainty factor and the amount of change in the certainty factor. For example, the display size of the certainty factor upper image 52 is proportional to the magnitude of the certainty factor. May be changed. Even in this case, it is possible to easily confirm the increase or decrease in the certainty factor.

  In addition, the specific structure and procedure for carrying out the present invention can be appropriately changed to other structures and the like within a range in which the object of the present invention can be achieved.

The present invention can be used as an image search device, an image search system, an image search method, and an image search program for searching for a search target image similar to the shot image from the shot image.

DESCRIPTION OF SYMBOLS 1 ... Image retrieval apparatus 5 ... Display part 31 ... Image database (image storage part)
32 ... Result accumulation unit 41 ... Image acquisition means 42 ... Image search means 43 ... Confidence level calculation means 44 ... Result display means 45 ... Image selection means 46 ... Display rate calculation means 100 ... Image search system 110 ... Internet (network)
200: Terminal device 221: Captured image transmission means 300 ... Server device 322: Search result transmission means

Claims (15)

An image search device that acquires a captured image and searches for a search target image similar to the captured image.
Image acquisition means for sequentially acquiring the captured images;
Image retrieval means for retrieving a search target image similar to the captured image each time the captured image is acquired, and acquiring a search result including the searched search target image and its similarity,
A result storage unit for storing the search results;
Based on the certainty factor of each search target image calculated using the search result accumulated in the result accumulation unit, a result display unit that ranks the certainty factor in descending order and displays the search target image;
Comprising
The result display means updates the display ranking of the displayed search target image according to the presence / absence of a user's selection operation until a predetermined time elapses after the search target image is displayed. An image search device. The image search device according to claim 1,
The result display means reduces the display ranking of the search target image when there is no user selection operation on the displayed search target image and a predetermined time has elapsed. The image search apparatus according to claim 2,
Comprising a certainty factor update means for calculating the certainty factor of each search target image using the search result accumulated in the result accumulation unit;
The certainty factor calculating means updates the certainty factor of the search target image by reducing the certainty factor when there is no user selection operation on the displayed search target image,
The result display means displays the search target images in descending order of certainty based on the updated certainty factor. In the image search device according to claim 2 or 3,
The result display means excludes the search target image from the display when there is no user selection operation on the displayed search target image. In the image search device according to claim 2 or 3,
The result display means reduces the display ranking of the search target image in a stepwise manner when there is no user selection operation on the displayed search target image. The image search device according to any one of claims 2 to 5,
Display rate calculation means for calculating a display rate that is a ratio of the time during which each search target image is displayed to the total time during which the search target image is displayed by the result display means;
The result display means reduces the display ranking of the search target image when there is no user selection operation on the displayed search target image and the display rate of the search target image exceeds a threshold value. An image search device. The image search device according to any one of claims 2 to 5,
The result display means reduces the display ranking of the search target image when there is no user selection operation on the displayed search target image and a preset determination time has elapsed. apparatus. The image search device according to any one of claims 1 to 7,
The result display means increases the display ranking of the search target image when there is no user selection operation for the displayed search target image and the time is within the predetermined time. The image search device according to any one of claims 1 to 8,
Comprising a certainty factor update means for calculating the certainty factor of each search target image using the search result accumulated in the result accumulation unit;
The certainty factor calculating means calculates an average value of the similarities of each search target image based on the similarity between the captured image and the search target image and the number of times the search process is performed by the image search means. An image search apparatus characterized by having a certainty factor of the search target image. The image search device according to any one of claims 1 to 9,
The result display means, when updating the displayed search target image, displays the display form of the search target image to be updated in a display form different from other search target images. . The image search device according to any one of claims 1 to 10,
The image acquisition means acquires a captured image in which an object is captured,
The image search device, wherein the image acquisition means searches for the search target image having an image of an object similar to the object of the captured image. An image search system in which a terminal device that acquires a captured image and a server device that searches for a search target image similar to the captured image are connected to each other via a network.
The server device
An image storage unit for storing the search target image;
Image search means for searching for the search target image similar to the photographed image transmitted from the terminal device, and acquiring a search result including the searched search target image and its similarity;
Search result transmission means for transmitting the search result to the terminal device,
The terminal device
Image acquisition means for sequentially acquiring the captured images;
Captured image transmission means for sequentially transmitting the acquired captured images to the server device;
A result storage unit that stores the search results transmitted from the server device;
Based on the certainty factor of each search target image calculated using the search result accumulated in the result accumulation unit, a result display unit that ranks the certainty factor in descending order and displays the search target image;
With
The result display means updates the display ranking of the displayed search target image according to the presence or absence of the user's selection operation until a predetermined time elapses after the search target image is displayed. Image search system. An image search system in which a terminal device that acquires a captured image and a server device that searches for a search target image similar to the captured image are connected to each other via a network.
The server device
An image storage unit for storing the search target image;
Image search means for searching for the search target image similar to the photographed image transmitted from the terminal device, and acquiring a search result including the searched search target image and its similarity;
A search result storage unit for storing the search results;
A certainty factor calculating unit that calculates the certainty factor of each search target image searched by the image searching unit based on the search result stored in the search result storage unit;
Search target image transmitting means for transmitting the search target image and its certainty factor to the terminal device,
The terminal device
Image acquisition means for sequentially acquiring the captured images;
Captured image transmission means for transmitting the acquired plurality of captured images to the server device;
A result display means for displaying the search target images transmitted from the server device in descending order of the certainty factor,
The result display means updates the display ranking of the displayed search target image according to the presence or absence of the user's selection operation until a predetermined time elapses after the search target image is displayed. Image search system. An image search method for searching a search target image similar to a photographed image by a computer,
The computer
An image acquisition step for sequentially acquiring the captured images;
Each time the captured image is acquired, a search target image similar to the captured image is searched, a search result including the searched search target image and its similarity is acquired, and stored in a result storage unit. An image search step;
Based on the certainty factor of each search target image calculated using the search result accumulated in the result accumulation unit, a result display step of displaying the search target image ranked in descending order of the certainty factor;
With
The result display step updates the display ranking of the displayed search target image according to the presence / absence of a user's selection operation until a predetermined time elapses after the search target image is displayed. Image search method. An image search program for searching for a search target image similar to a captured image,
An image acquisition step for sequentially acquiring the captured images;
Each time the captured image is acquired, a search target image similar to the captured image is searched, a search result including the searched search target image and its similarity is acquired, and stored in a result storage unit. An image search step;
Based on the certainty factor of each search target image calculated using the search result accumulated in the result accumulation unit, a result display step of displaying the search target image by ranking in descending order of the certainty factor. Let the computer run,
The result display step updates the display ranking of the displayed search target image according to the presence / absence of a user's selection operation until a predetermined time elapses after the search target image is displayed. An image search program.

Images