JP5525737B2 - Server system, terminal device, program, information storage medium, and image search method - Google Patents

Description

  The present invention relates to a server system, a terminal device, a program, an information storage medium, an image search method, and the like.

  2. Description of the Related Art Conventionally, there has been known an image search system that searches for products or the like taken by a camera using image recognition processing. For example, Patent Literature 1 discloses a system for searching for a store where a product can be purchased by photographing a product desired by a user using a mobile phone with a camera and transmitting the photographed image via the Internet. ing. In such an image search system, registered images as reference images for image search are registered in an image database, and the image search is executed by comparing these registered images with captured images.

  However, in the conventional image search systems, the registered image that is the reference image for image search is prepared by the system side, so that the number of registered images registered in the image database is limited. Therefore, when performing an image search of a captured image from the user, there are many cases where a registered image that is similar (matched) to the captured image cannot be extracted from the image database and the image cannot be recognized.

  Further, in the image retrieval system so far, the photographed image is only used for product retrieval and the like, and the photographed image subject to image retrieval is stored in the system side and re-presented to the user. Was not proposed.

JP 2003-122757 A

  According to some aspects of the present invention, it is possible to provide a server system, a terminal device, a program, an information storage medium, an image search method, and the like that can increase registered images that serve as reference images for image search.

  According to one aspect of the present invention, a reception processing unit that performs processing for receiving transmission data of a captured image of a user who is an object of image search via a network, and a plurality of registered images similar to the captured image are stored in an image database. An image search unit for searching from a tag processing unit for acquiring a plurality of candidate name tags from a plurality of name tags associated with the plurality of registered images searched, and a search result screen for image search of the captured image An image management unit for registering the selected candidate name tag in the image database in association with the photographed image as a confirmed name tag of the photographed image when the user selects any of the plurality of candidate name tags; Related to server systems including

  According to one aspect of the present invention, when transmission data about a user's captured image is received via a network, a registered image similar to the received captured image is searched, and a captured image candidate is searched from the registered image name tag. A name tag is acquired. When the user selects one of the candidate name tags on the search result screen of the image search, the selected candidate name tag is registered in the image database in association with the captured image as a confirmed name tag of the captured image. As a result, the user's captured image can be registered in the image database together with the confirmed name tag, so that it is possible to increase the number of registered images that serve as reference images for image search.

  In one aspect of the present invention, the image management unit extracts a registered photographed image that is the photographed image registered in the image database from the image database, and the extracted registered photographed image satisfies a predetermined condition. If not, a process of deleting from the image database may be performed.

  In this way, the registered photographed image that does not satisfy the predetermined condition is deleted from the image database, so that it is possible to save the storage capacity of the image database and speed up the image recognition process.

  In one aspect of the present invention, the image management unit checks image recognition number information, which is information about the number of times the registered photographed image is recognized as an image similar to the search target image, and the image of the registered photographed image. If it is determined that the number of times of recognition is less than or equal to a predetermined number, the number of recognitions may be deleted from the image database.

  In this way, registered photographed images for which the number of times of image recognition is determined to be less than or equal to a predetermined number are deleted from the image database, so that it is possible to save the storage capacity of the image database. Become.

  Also, in one aspect of the present invention, the image management unit checks the selected selection number information, which is information about the number of times the registered photographed image is selected by the user on the search result screen, and extracts the registered photographed image extracted. May be deleted from the image database.

  In this way, registered photographed images for which the number of final selections is determined to be less than or equal to a predetermined number are deleted from the image database, so that it is possible to save storage space used for the image database. Become.

  In the aspect of the invention, the tag processing unit may count the number of appearances of the plurality of name tags, and acquire the plurality of candidate name tags based on the counting result.

  In this way, candidate name tags can be selected and acquired according to the number of appearance of name tags.

  In one aspect of the present invention, a position information acquisition unit that acquires user position information is included, and the image search unit acquires the user's position information when image recognition of the captured image is not successful. An image search may be performed by instructing the position information acquisition unit and narrowing down a search target based on the acquired position information.

  In this way, even when the image recognition is not successful, the search target can be narrowed down by using the user position information, and the image search can be executed again.

  In one aspect of the present invention, an album registration unit that performs a process of registering the photographed image in an album database, and a data creation unit that creates search result data including the plurality of candidate name tags acquired, When the user selects any one of the plurality of candidate name tags on the search result screen displayed by the search result data, the album registration unit selects the selected candidate name tag as the confirmed name of the captured image. A tag may be registered in association with the photographed image in the album database.

  In this way, it becomes possible to view the photographed image as the target of the image search as an album, and an image search of an unprecedented type can be realized.

  In the aspect of the invention, the reception processing unit performs processing for receiving a mail from a user to which the transmission data regarding the captured image is attached, and the album registration unit is configured to receive a mail address of the received mail. The photographed image may be registered in the storage location of the album database specified by.

  In this way, simply by sending a mail with a captured image attached thereto, the captured image is registered in the storage location specified by the mail address, so the convenience of the user can be improved.

  Further, in one aspect of the present invention, the apparatus includes a position information acquisition unit that acquires position information of a shooting location of the captured image based on the fixed name tag, and the album registration unit includes the position of the acquired shooting location. Information may be registered in association with the photographed image in the album database.

  In this way, the position information of the shooting location of the shot image can be specified by the confirmed name tag selected by the user and registered in the album database.

  Moreover, in one aspect of the present invention, an album browsing unit that performs processing for the user to browse album information in the album database may be included.

  In this way, the user can browse and enjoy the album in which the photographed image subject to image search is automatically registered.

  In the aspect of the invention, the album browsing unit may perform a process for displaying on the map screen at least one of the shooting location of the shot image registered in the album database and the movement history of the shooting location. You may go.

  In this way, the shooting location of the shot image remains as a trajectory on the map screen, so that the user can enjoy the afterglow of the trip.

  In one aspect of the present invention, the information processing apparatus includes a position information acquisition unit that acquires position information of the shooting location of the captured image based on the confirmed name tag, and the album browsing unit is based on the acquired position information. Then, the display position of the shooting location on the map screen or the movement history may be specified.

  In this way, it is possible to identify the location information of the shooting location using the confirmed name tag and display the shooting location and its movement history on the map screen.

  In one aspect of the present invention, the album registration unit may register an uploaded image in the album database when uploading of an image is selected by the user when the user browses the album.

  In this way, the user can register not only the photographed image but also a desired upload image in the album database.

  In one aspect of the present invention, the image search unit searches the image database for a plurality of registered images similar to the upload image when uploading of images is selected by a user, and the tag processing unit The plurality of upload image candidate name tags are acquired from the plurality of name tags associated with the plurality of registered images that have been searched, and the image management unit displays the plurality of upload images on the upload image search result screen. When one of the image candidate name tags is selected by the user, the selected candidate name tag may be registered in the image database in association with the uploaded image as an uploaded image fixed name tag.

  In this way, when the user selects a confirmed name tag from among the candidate name tags acquired by the image search for the uploaded image, the confirmed name tag is registered in the image database in association with the uploaded image.

  According to another aspect of the present invention, there is provided an image acquisition unit that acquires a captured image of a user who is an object of image search, and a transmission process that performs processing of transmitting transmission data about the acquired captured image via a network. A reception processing unit that performs processing for receiving search result data of an image search for the captured image via a network, and a display control unit that performs display control of the display unit based on the received search result data. And the display control unit retrieves a plurality of registered images similar to the captured image from an image database, and obtains a plurality of candidate name tags from a plurality of name tags associated with the retrieved plurality of registered images. The search result screen on which the obtained candidate name tags are displayed is displayed on the display unit, and the transmission processing unit performs the search When the user selects one of the plurality of candidate name tags on the result screen, the selected candidate name tag is transmitted via the network as the confirmed name tag of the captured image registered in the image database. This relates to a terminal device that performs processing. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to another aspect of the present invention, when a captured image that is a target of an image search is acquired, the captured image is transmitted via a network. When the search result data of the image search for the photographed image is received, the candidate name tag acquired from the photographed image is displayed on the search result screen. When the user selects one of the candidate name tags on the search result screen, the selected candidate name tag is transmitted via the network as a confirmed name tag of the captured image registered in the image database. As a result, the user's captured image can be registered in the image database together with the confirmed name tag, so that it is possible to increase the number of registered images that serve as reference images for image search.

  In another aspect of the present invention, the transmission processing unit registers the selected candidate name tag in the album database when the user selects any of the plurality of candidate name tags on the search result screen. As the fixed name tag of the captured image to be processed, a process of transmitting via a network may be performed.

  In this way, the photographed image that is the object of the image search is registered in the album database, and an unprecedented type of image search can be realized.

  In another aspect of the present invention, the display control unit may perform control to display an album browsing screen based on album information in the album database on the display unit.

  In this way, the user can browse the captured image that is the target of the image search on the album browsing screen.

  In another aspect of the present invention, the display control unit displays a map screen on which at least one of a shooting location of the shot image registered in the album database and a movement history of the shooting location is displayed. You may control to display on.

  In this way, the shooting location of the shot image remains as a trajectory on the map screen, so that the user can enjoy the afterglow of the trip.

  In another aspect of the present invention, the display control unit uploads an image by a user, searches a plurality of registered images similar to the uploaded image, and searches the plurality of registered images. When a plurality of upload image candidate name tags are acquired from a plurality of name tags associated with the search result screen, the acquired upload image candidate name tags are displayed on the display unit. When the user selects any of a plurality of upload image candidate name tags on the search result screen, the transmission processing unit displays the selected candidate name tag as the upload image fixed name. You may perform the process which transmits via a network as a tag.

  If it does in this way, if a user chooses a definite name tag from candidate name tags acquired by image search of an upload image, the definite name tag will be transmitted via a network as a definite name tag for upload images. It becomes like this.

  In another aspect of the present invention, a plurality of registered images similar to a user's captured image that is an object of image search are searched from an image database, and a plurality of name tags associated with the searched plurality of registered images are used. , Obtaining a plurality of candidate name tags, displaying a search result screen on which the obtained plurality of candidate name tags are displayed, and selecting one of the plurality of candidate name tags on the search result screen In this case, the present invention relates to an image search method in which the selected candidate name tag is used as a confirmed name tag of the photographed image and is registered in the image database in association with the photographed image.

The system configuration example of this embodiment. Explanatory drawing of operation | movement of this embodiment. 3A and 3B are examples of a result notification mail and a search result screen. An example of a tourist information screen. 5A and 5B are examples of the album browsing screen. An example of an image details screen. Explanatory drawing of the deletion process of a registration image. Process flow of registered image deletion processing. 9A and 9B show examples of the data structure of the album database and the image database. 10A and 10B are explanatory diagrams of tag processing. Example of upload image search result screen. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. System Configuration FIG. 1 shows the system configuration of the image search system of this embodiment. This image search system is realized by a server system realized by the management server 10 or the image search server 60, a terminal device 110 (communication terminal, portable terminal device, electronic device), or the like. The management server 10, the image search server 60, the terminal device 110, and the like are communicatively connected via a network 100. Other external servers and terminal devices may be connected to the network 100 for communication.

  In this embodiment, a tourist spot, facility, cultural property, store, building, or the like that is a subject of an image transmitted from a user by e-mail or the like is identified by image recognition, and the detailed information and a region such as a subsequent sightseeing route Provide information. In addition, when the user selects a desired name tag from among the candidate name tags of the photographed image on the search result screen of the image search, the selected candidate name tag is registered in the image database as a correct name tag of the photographed image. In addition, the image attached to the mail is automatically edited for each user, position information is added to the image, and the album is provided as an album for specifying and displaying the shooting location on the map.

  Specifically, when the user captures an image with the terminal device 110 such as a mobile phone with a camera, the captured image is transmitted from the terminal device 110. Then, the image search server 60 refers to the image database 90 to search for a registered image that is similar (matches) to the user's captured image, and based on the name tag (tag in a broad sense) of the searched registered image, Get the candidate name tag for the image. For example, when the subject of the photographed image is a temple, temple name candidates that are similar (match) to the subject of the photographed image are acquired as candidate name tags.

  Then, search result data including the acquired candidate name tag is created, and a search result screen as shown in FIG. As a result, the created search result screen is displayed on the display unit 180 of the terminal device 110. For example, in the search result screen of FIG. 3B, names of “E1 temple”, “E2 shrine”, and “E3 temple” are displayed as candidate name tags.

  The user selects a name tag that he / she thinks is correct from among these candidate name tags displayed on the search result screen. For example, in B4 of FIG. 3B, the user has determined that the temple photographed by himself is “E1 temple”, and therefore selects the name tag “E1 temple”. Thus, the name tag selected by the user is determined to be a confirmed name tag (correct tag). The captured image and the confirmed name tag are associated with each other and registered in the image database 90. As a result, not only the registered image prepared by the system side but also the photographed image requested by the user for the image search is registered in the image database 90 as a registered image serving as a reference image for the image search.

  The management server 10 also registers the user's captured image in the album database 40. As a result, the user can view a series of captured images for which image search has been performed in the past as his / her own album, as shown in FIGS. Further, the confirmed name tag selected by the user is also registered in the album database 40 in association with the photographed image of the user. In this way, for example, when the subject of the photographed image is a sightseeing spot, the user can remember the sightseeing spot that has been traveled in the past by browsing the album.

  Hereinafter, the configuration of the present embodiment will be specifically described with reference to FIG. The management server 10, the image search server 60, the terminal device 110, and the like are not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of these components or adding other components. Is possible. The server system according to the present embodiment can be realized by one of the management server 10 and the image search server 60, or can be realized by both the management server 10 and the image search server 60, for example.

  The network 100 (communication network) can include a mobile communication network, a public network such as the Internet network, or a fixed telephone network. The mobile communication network for communication connection with the terminal device 110 is a mobile telephone network such as a CDMA (Code Division Multiple Access) system or a PDC (Personal Digital Cellular) system, or mobile data such as a PDC-P (PDC-Packet) system. A communication network can be included. The network 100 can be realized by a WAN (Wide Area Network), a LAN (Local Area Network), or the like, regardless of wired or wireless.

  The management server 10 is a server in charge of management processing or the like in the server system, and is, for example, a web server or a mail server. The image search server 60 is a server in charge of image search and tag processing. The functions of the management server 10 and the image search server 60 can be realized by hardware such as an information processing apparatus and a program such as a CGI program.

  The terminal device 110 can transfer images from, for example, a camera-equipped cellular phone, a digital camera (digital still camera, digital video camera), an information processing terminal having a camera function, or an information processing terminal having a camera function, for example. A device (PC) or the like, and has a function of acquiring a captured image and a function of communicating with the management server 10 and the image search server 60 via the network 100. The information processing apparatus 200 is a store or user PC (Personal Computer) or the like.

  The management server 10 includes a processing unit 20, an album database 40, a personal information database 42, a content database 50, and a communication unit 58. The processing unit 20 performs overall system management processing, database management processing, data creation processing, and the like, and functions thereof include hardware such as various processors (CPU, etc.), PHPPHP (Hypertext Preprocessor), Perl, and the like. This can be realized by a CGI program written in a language.

  The album database 40, the personal information database 42, and the content database 50 are relational databases realized by, for example, MySQL. The album database 40 stores user's captured images as album information. In association with the photographed image, the name tag (determined name tag), the position information of the photographing location (user), and the like are stored as album information. The personal information database 42 stores various types of user information. The content database 50 stores various content information, and may include a tourist information database 52, a recommendation information database 54, an advertisement information database 56, and the like. The communication unit 58 is for communicating information with the terminal device 110, the information processing apparatus 200, and other external servers via the network 100.

  The image search server 60 includes a processing unit 70, an image database 90, and a communication unit 98. The processing unit 70 performs image search processing, tag processing, and the like, and its functions can be realized by various processors and programs.

  The image database 90 is a relational database that stores registered images for image search. The communication unit 98 is for communicating information with the management server 10 or the like via the network 100.

  The terminal device 110 includes a processing unit 120, a storage unit 140, an information storage medium 150, a communication unit 158, an imaging unit 160, an operation unit 170, and a display unit 180.

  The processing unit 120 performs various processes based on operation information from the operation unit 170, information received by the communication unit 158, and the like. The function of the processing unit 120 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program stored in the information storage medium 150, and the like.

  The storage unit 140 serves as a work area for the processing unit 120, the communication unit 158, and the like, and its functions can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like.

  The information storage medium 150 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by an optical disk (CD, DVD), an IC card (memory card), or the like. The processing unit 120 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 150. That is, in the information storage medium 150, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The communication unit 158 is for communicating information with the management server 10, the image search server 60, and other terminal devices via the network 100. The imaging unit 160 is for imaging (photographing) a subject, and can be realized by, for example, a CCD or a CMOS sensor. The operation unit 170 is for the user to perform various operations on the terminal device 110, and can be realized by various buttons (shutter buttons), GUI (Graphical User Interface), and the like. The display unit 180 is for displaying various display screens, and can be realized by, for example, a liquid crystal display, an organic EL display, a CRT, or the like.

  The processing unit 20 of the management server 10 includes a reception processing unit 22, a transmission processing unit 24, an album registration unit 26, a data creation unit 28, a position information acquisition unit 30, an album browsing unit 32, and a content presentation unit 34. Various modifications such as omission of some of these components and addition of other components are possible.

  The reception processing unit 22 performs data reception processing, and the transmission processing unit 24 performs data transmission processing. For example, the reception processing unit 22 receives transmission data of a user's captured image (search target image or image in a broad sense) as an image search target via the network 100 from, for example, the terminal device 110. Here, the transmission data regarding the photographed image may be the photographed image data itself or the photographed image recognition data (image recognition data). As the recognition data, for example, feature amount data (feature data) of a captured image, data specifying a search target image, and the like can be assumed.

  The album registration unit 26 performs processing for registering album information (images, tags, positions, etc.) in the album database 40. Specifically, a process of registering (writing) a captured image (captured image data) received via the network 100 in the album database 40 is performed. For example, it is assumed that the reception processing unit 22 receives an e-mail from a user attached with transmission data regarding a captured image. In this case, the album registration unit 26 takes a photographed image or a fixed name tag (correct tag) in the storage location (registration location) of the album database 40 specified by the email address (my page ID, user ID) of the received email. Register.

  The data creation unit (search result screen creation unit) 28 creates various data such as search result data. For example, a plurality of registered images similar to the photographed image are searched from the image database 90, and a plurality of candidate name tags (candidate tags) are acquired from a plurality of name tags (tags) associated with the searched plurality of registered images. Suppose. In this case, the data creation unit 28 creates search result data including a plurality of acquired candidate name tags. That is, data for displaying the search result screen shown in FIG.

  When the user selects one of a plurality of candidate name tags on the search result screen of FIG. 3B, the album registration unit 26 selects the selected candidate name tag as a confirmed name tag (correct tag, correct tag, As a selection tag, it is registered in association with the photographed image (album registered image) in the album database 40. As a result, the user can browse the album in which the image taken by the user and the name tag selected by the user are displayed.

  In the search result screen shown in FIG. 3B, when the user selects text search without selecting the confirmed name tag, the data creation unit 28 searches the content database 50 for content information by text search. To do. Then, search result data including content information such as the searched tourism information is created.

  The position information acquisition unit 30 acquires position information such as the shooting location (user location) of the shot image. Specifically, for example, the position information of the shooting location is acquired based on the confirmed name tag. For example, when the confirmed name tag selected by the user is “E1 temple” in FIG. 3B, the position of “E1 temple” is set as the shooting location of the captured image based on the tourist information in the tourism information DB. Obtained as location information. The album registration unit 26 registers the acquired location information of the shooting location in association with the shot image in the album database 40. By doing so, the position information of the shooting location of each captured image can be acquired by a simple process without using, for example, GPS.

  The album browsing unit 32 performs processing for the user to browse album information such as a photographed image in the album database 40. Specifically, various album browsing screens as shown in FIGS. 5A to 6 are created and displayed.

  For example, in C3 of FIG. 5A, a map screen (map) relating to the shooting location is displayed. On this map screen, the display position of the shooting location of the shot image registered in the album and its movement history (movement locus) are displayed. For example, it is displayed that the user shot “F2 Shrine” after shooting “F1 Tower” in Osaka, and then shot “F3 Castle”. In this case, the album browsing unit 32 specifies the display position and movement history of the shooting location on the map screen of FIG. 5A based on the position information acquired by the position information acquisition unit 30, for example.

  Also, when the user selects to browse the album information, the album browsing unit 32 creates and displays a list screen of photographed images registered in the album database 40 as shown in FIG. Further, when the user inputs a comment on the captured image displayed on the list screen, the album browsing unit 32 registers the input comment in association with the captured image in the album database 40. For example, when the user selects a desired captured image on the list screen of FIG. 5B, detailed image information as shown in FIG. 6 is displayed. In the detailed image information, a comment input by the user with respect to the captured image is displayed as shown in C11. Note that the input comment may be displayed in association with each captured image on the list screen of FIG.

  Also, the album registration unit 26 performs a process of registering the uploaded image in the album database 40 when the user selects to upload an image when browsing the album in FIGS. 5A and 5B.

  In this embodiment, an example in which various screens such as a search result screen and an album browsing screen are created on the server side has been described. However, these various screens may be created by an application program on the terminal device side. Good. For example, when the data creation unit 28 on the server side creates and transmits data for displaying various screens such as the search result screen (search result data and the like), the application program on the terminal device side is based on the created data. Various screens may be created and displayed on the display unit 180. Further, the blocks as software may be integrated on hardware, and for example, image recognition processing may be executed as application processing on the terminal device side.

  The content presentation unit 34 performs various types of content information presentation processing (display processing). For example, the content database 50 is referred to based on the confirmed candidate name tag selected by the user, and the content information corresponding to the confirmed name tag is acquired. Specifically, tourist information (detailed information on the subject) corresponding to the confirmed name tag is acquired as content information, and tourist information (tourist spot information) as shown in FIG. 4 is presented to the user.

  The processing unit 70 of the image search server 60 includes a reception processing unit 72, a transmission processing unit 74, an image search unit 76, a tag processing unit 78, and an image management unit 80.

  The reception processing unit 72 performs processing for receiving, for example, captured image data from the management server 10. Note that captured image data may be received from the terminal device 110.

  The image search unit 76 performs image search processing. Specifically, a plurality of registered images (one or a plurality of registered images) that are similar (match) to the captured image are searched from the image database 90.

  Note that the image search unit 76 instructs the position information acquisition unit 30 of the management server 10 to acquire the position information of the user when the image recognition of the captured image is not successful. Then, the search target may be narrowed down based on the position information acquired by the position information acquisition unit 30 to perform image search. For example, the position information may be acquired using GPS, simple position information, or the like.

  The tag processing unit 78 acquires a plurality of candidate name tags (candidate tags) from a plurality of name tags (tags) associated with the searched plurality of registered images. Specifically, the number of appearances (appearance frequency) of a plurality of name tags is tabulated, and a plurality of name tag sorting processes are performed based on the tabulation results to obtain a plurality of candidate name tags. For example, a plurality of name tags are rearranged in the descending order of appearance, and a name identification process for collecting duplicate name tags into one name tag is performed to obtain candidate name tags. Then, the transmission processing unit 74 performs processing for transmitting the acquired candidate name tag (tag information) to the management server 10.

  Note that the name tag counting process is not limited to the process of simply counting the number of appearance of name tags. For example, various modifications such as obtaining a weighting coefficient based on the degree of similarity of a registered image extracted as an image similar to a photographed image and performing a tabulation process based on the weighting coefficient and the number of appearances are possible. is there. For example, even if the number of appearances is small, when the similarity of the registered image to the photographed image is high, the total value of the name tags of the registered image may be increased.

  The image management unit 80 performs management processing of the image database 90. For example, an image registration process in the image database 90 or a deletion process (cleaning process) of a registered image in the image database 90 is performed. For example, it is assumed that the user selects one of a plurality of candidate name tags on the search result screen of the image search shown in FIG. In this case, the image management unit 80 registers the selected candidate name tag in the image database 90 in association with the captured image as a confirmed name tag of the captured image. That is, the user's captured image and the confirmed name tag selected by the user are registered in the image database 90.

  Further, the image management unit 80 extracts (reads out) a registered photographed image that is a photographed image registered in the image database 90 from the image database 90. If the extracted registered photographed image does not satisfy the predetermined condition, a process of deleting the registered photographed image from the image database 90 is performed.

  For example, the image management unit 80 checks the image recognition frequency information (image recognition frequency itself or information that changes according to the image recognition frequency) of the registered photographed image. When it is determined that the image recognition frequency information does not satisfy the predetermined condition, for example, when it is determined that the image recognition frequency of the registered captured image is equal to or less than the predetermined number, the registered captured image is deleted from the image database 90. To do. Here, the image recognition frequency information is information representing the number (frequency) at which the registered captured image is recognized as an image similar (matched) to the search target image (captured image, uploaded image).

  In addition, the image management unit 80 checks the fixed selection count information (the fixed selection count itself or information that changes according to the fixed selection count) of the registered photographed image. Then, when it is determined that the fixed selection number information does not satisfy the predetermined condition, for example, when it is determined that the fixed selection number is equal to or less than the predetermined number, the registered photographed image is deleted from the image database 90. Here, the fixed selection number information is information indicating the number of times the registered photographed image is selected by the user on the search result screen. Note that when the number of times of image recognition is less than a predetermined number or the number of final selections is less than a predetermined number, the registered photographed image does not necessarily have to be deleted. In consideration of this, the registered photographed image deletion process may be executed.

  The processing unit 120 of the terminal device 110 includes an image acquisition unit 122, a transmission processing unit 124, a reception processing unit 126, and a display control unit 128. Various modifications may be made such as omitting some of these components or adding other components.

  The image acquisition unit 122 acquires a captured image to be searched. For example, a captured image (subject image) obtained by imaging the subject with the imaging unit 160 is captured. For example, a captured image stored in a memory card or the like may be read and acquired.

  The transmission processing unit 124 performs processing for transmitting transmission data regarding the acquired captured image to the management server 10 (server system in a broad sense) via the network 100. The reception processing unit 126 performs a process of receiving search result data of a similar image search for captured images via the network 100.

  The display control unit 128 performs display control of the display unit 180 based on the received search result data, and displays a search result screen as shown in FIG. That is, when a registered image similar to the photographed image is searched and a candidate name tag is acquired based on the search result, a search result screen on which the acquired candidate name tag is displayed is displayed on the display unit 180. Further, the display control unit 128 performs control to display an album browsing screen based on the album information in the album database 40 as shown in FIGS. Specifically, as shown in FIG. 5A, control is performed to display a map screen on which shooting locations and movement histories of shot images registered in the album database 40 are displayed.

  Assume that the user selects one of the candidate name tags on the search result screen. Then, the transmission processing unit 124 performs processing for transmitting the selected candidate name tag as a confirmed name tag of a captured image registered in the image database 90 or the album database 40 via the network 100.

2. Operation Next, the operation of this embodiment will be described with reference to the processing flow of FIG. Hereinafter, “database” is described as “DB” as appropriate.

  When the user captures a subject with the camera (imaging unit) of the terminal device 110, the user sends an email with the captured image attached thereto to a specific email address (step S301). When receiving the mail from the user, the management server 10 registers (accumulates) the photographed image attached to the mail in the album DB and transmits the photographed image to the image search server 60 (steps S302, S303, S304). .

  When receiving the captured image, the image search server 60 performs image recognition processing with reference to the image DB (steps S305 and S306). Specifically, the photographed image (search target image) and each registered image in the image DB are collated, and among the registered images that are similar to the photographed image (including cases where they match), a predetermined upper number of registered images are obtained. Extract. And a candidate name tag is acquired based on the result of an image recognition process, and the acquired candidate name tag is transmitted to the management server 10 (steps S307 and S308). Specifically, a name tag associated with a predetermined number of registered images similar to the photographed image is collected (name identification process), and a candidate name tag is acquired and transmitted to the management server 10.

  When receiving the candidate name tag, the management server 10 extracts the tourism information corresponding to the candidate name tag from the tourism information DB, and transmits a result notification mail to the user terminal device 110 (steps S309, S310, and S311).

  FIG. 3A shows an example of the result notification mail. As shown in B1 and B2, the result notification mail is a mail including a candidate name tag of a photographed image, a URL (Uniform Resource Locator) of a My Page for displaying a search result screen, and the like. Then, when the user clicks on the My Page URL of B2, a search result screen as shown in FIG. 3B is displayed (Steps S313 and S314).

  In the search result screen of FIG. 3B, as shown in B3, a candidate name tag of the photographed image and an icon image (registered image similar to the photographed image) corresponding to each candidate name tag are displayed. Specifically, “E1 temple”, “E2 shrine”, “E3 temple” candidate name tags and icon images representing “E1 temple”, “E2 shrine”, “E3 temple” are displayed. The user refers to these candidate name tags and icon images, and selects a candidate name tag (icon image) that he / she considered to be correct (step S315). For example, in FIG. 3B, the candidate name tag “E1 temple” shown in B4 is selected.

  Then, the management server 10 refers to the content DB such as the tourist information DB, extracts various content information, and displays a content information screen (tourist information screen) as shown in FIG. 4 (steps S316 and S317). . As a result, the user can browse various pieces of content information such as sightseeing spot information, recommendation information, and advertisement information as shown in FIG. 4 (step S318).

  In this embodiment, when the user selects a desired name tag from the candidate name tags indicated by B3 in FIG. 3B, the selected name tag (E1 temple) is an image together with the photographed image as a confirmed name tag. Registered in the DB (step S319). That is, the user's captured image is registered as a registered captured image in the image DB, and the confirmed name tag selected by the user is registered as a correct tag of the registered captured image.

  Note that the confirmed name tag selected by the user may be registered in the album DB (step S320). That is, a fixed name tag of “E1 temple” is registered in association with the user's photographed image. As a result, the photographed image for which the user has performed an image search and the confirmed name tag selected by the user are sequentially registered in the album DB.

  Also, the user can execute the image search of the captured image again by clicking B7 in FIG. In addition, when the search result desired by the user cannot be obtained, the user can select the text search shown in B8 and perform a text search for information on the captured image. As a result, it becomes possible to search for sightseeing spot information and the like by referring to the sightseeing information DB and the like.

  As shown at B5 in FIG. 3B, the user is notified that the sent photographed image is registered in the user's album. When the user is interested in what album has been created and clicks on B6 album browsing, an album browsing screen as shown in FIG. 5A is displayed (steps S321 and S322).

  In FIG. 5A, as shown in C1, it is possible to browse albums of places where the user has traveled in the past, such as “Osaka album” and “Shiga album”. “Osaka Album” is displayed. Note that albums may be grouped by date (eg, shooting date).

  As shown at C2 in FIG. 5A, the user takes pictures of “F1 tower”, “F2 shrine”, and “F3 castle” on a trip in Osaka, and searches for images of the captured images. Then, the user selects the fixed name tags of these sightseeing spots on the search result screen of FIG. 3B, and these photographed images and fixed name tags are registered in the album DB. Then, the management server 10 identifies the location of the sightseeing spot from the fixed name tags of these “F1 tower”, “F2 shrine”, and “F3 castle” and displays them on the map screen shown in C3. For example, the star marks C4, C5, and C6 indicate the positions of “F1 tower”, “F2 shrine”, and “F3 castle”, respectively, and the arrow marks C7 and C8 indicate the movement history (movement) of the user (shooting place). Route). By viewing these marks, the user can remember what route he / she went sightseeing in Osaka.

  When the user clicks “view album” shown in C9, a list screen of photographed images in the Osaka album is displayed as shown in FIG. 5B. That is, a list screen of photographed images is created and displayed based on the album information in the album DB. By viewing such a list screen, the user can browse images taken in Osaka and enjoy travel memories.

  When the user clicks “view details” shown in C10 on the list screen of FIG. 5B, an image details screen as shown in FIG. 6 is displayed. On this image detail screen, as shown at C11, the user can input a comment for the photograph. As a result, the user can write down the situation / background etc. of the photograph taken in the comments, and can look back on the user's trip.

  In addition, when the user selects a candidate name tag in step S315 in FIG. 2, position information such as the sightseeing spot is acquired from the tourist information DB or the like based on the confirmed name tag (step S330). As a result, the location of the user is specified, and a notification e-mail notifying that the user is located (approaching) is transmitted to the information processing device (PC) 200 for the surrounding stores of the location (step S331). .

  When the information processing apparatus 200 receives the notification mail, the information processing apparatus 200 extracts a target user of the advertisement mail distribution, and selects a user who distributes the notification mail from the extracted target users (steps S332, S333, and S334). . For example, based on information from the user's personal information DB, the target users of advertisement mail distribution are narrowed down and selected according to the age group, sex, and the like. Then, the management server 10 is requested to distribute the advertisement mail corresponding to the registered advertisement content to the selected target user (step S335). It should be noted that registration of the advertisement content is preferably performed in advance before receiving the notification mail from the management server 10.

  Upon receiving the advertisement mail distribution request, the management server 10 reads advertisement information from the advertisement information DB and creates an advertisement mail (step S336). In the advertisement information DB, advertisement information having contents registered in advance by the information processing apparatus 200 is stored. Then, when the management server 10 transmits the advertisement mail created and the terminal device 110 receives the advertisement mail, the advertisement mail of the neighboring stores comes to be displayed on the display unit 180 of the terminal device 110 (steps S337 and S338). .

  For example, it is assumed that the user takes a photograph at the tourist destination “E1 temple” and selects “E1 temple” as the confirmed name tag in the search result screen of FIG. Then, the location information of “E1 temple” is extracted from the tourism information DB, the location of the user is specified based on this location information, and a notification mail is transmitted to the surrounding stores of that location. When there is an advertisement mail distribution request from the information processing apparatus 200 of the peripheral store, for example, as shown in B9 of FIG. 4, the advertisement information of the peripheral store is displayed. In this way, the confirmed name tag obtained by the image search can be effectively used to identify the location of the user and to present the advertisement information of the surrounding stores at the location to the user in real time.

  According to the method of the present embodiment described above, when a user takes a picture with a camera-equipped mobile phone or digital camera and requests the server to search for an image of the taken image, as shown in FIG. The search result screen is displayed. When the user selects a correct name tag from the candidate name tags on this search result screen, the selected name tag is registered in the image DB together with the captured image as a confirmed name tag. The registered captured image is used as a reference image for collation processing in image search. Therefore, various users use the image search system to transmit images taken by themselves to the server side and select a fixed name tag of the photographed images so that the number of registered images registered in the image DB increases. become. That is, as the number of users who use the image search system and the number of uses of each user increase, the number of registered images in the image DB also increases, and the image DB grows with the use of the user.

  Therefore, even when the number of registered images prepared by the system is small, the number of registrations naturally increases due to the use of the user, so that the burden of management and operation of the image search system can be reduced. In addition, the success rate of recognition in image recognition increases as the number of registered images increases. Therefore, it is possible to reduce the occurrence of a situation in which a registered image similar to a photographed image cannot be extracted from the image DB during image search and image recognition fails.

  In the present embodiment, during travel or sightseeing (on-travel), when the user requests the server to search the captured image, a search result screen as shown in FIG. 3B is displayed. Guide information such as sightseeing information as shown in FIG. Therefore, the guide information as shown in FIG. 4 is displayed just by taking a picture with a mobile phone or the like and sending the picture to a specific mail address, so that the user does not have to carry a guidebook while traveling. It will be over. This makes it possible to provide guide information that is more portable and convenient than a guide book.

  In addition, in the guide information of FIG. 4, information about neighboring stores as shown in B9 and recommendation information (recommended information) of other users as shown in B10 are displayed in real time. Information can be provided to the user.

  The photographed image requested by the user for image retrieval is automatically registered and stored in the album DB without the user being aware of it. When the user makes a selection to confirm the name tag of the photographed image on the search result screen of FIG. 3B, the confirmed name tag is registered in association with the photographed image, and the name of the photographed image is automatically assigned. Will be granted. Therefore, since the images taken by the user can be stored in individual units and provided to the user as a web album, the user can take a picture and save it on the server without worrying about the storage capacity. become.

  Also, when the user takes a picture for the purpose of image search, the user is informed that the photographed image is registered in the album, as indicated by B6 in FIG. 3B. ~ The photographed image can be viewed on the album browsing screen of FIG. Therefore, it is possible to provide not only guide information for on-travel but also information for looking back on memories of travel for after-travel. Further, as shown at C3 in FIG. 5A, by utilizing the location information of the shooting location of the shot image and leaving the travel history during the trip as a trajectory on the map screen, the user visually recognizes the afterglow of the trip. To be able to enjoy. Therefore, it is possible to prevent a situation in which many photographed images are stored without being browsed, and it is possible to effectively utilize the photographed images through browsing the album.

3. Image management of registered images Next, an image management method for registered images of this embodiment will be described. In the present embodiment, the user's captured image is registered as a reference image of the image DB, thereby increasing the registration number of the image DB and increasing the success rate of image recognition. However, if the number of registrations increases too much, there is a possibility that problems such as the use storage capacity of the image DB being compressed and the image search processing being delayed will occur.

  Therefore, in the present embodiment, a process (cleaning process) for deleting a registered photographed image that is a photographed image registered in the image DB is performed so that the registration number of the image DB does not increase too much. Specifically, a registered photographed image is extracted from the image DB, and when the registered photographed image does not satisfy a predetermined condition (a condition not to be deleted), the registered photographed image is deleted from the image DB.

  For example, in FIG. 7, it is assumed that registered images M1 to M6 whose image IDs are ID1 to ID6 are images registered by captured images of “H1 temple”. In the registered image M3, when the confirmed name tag is selected, the user has intentionally or mistakenly selected “H2 temple”. ”Name tag. The registered images M5 and M6 are unclear pictures taken in a dark situation, for example.

  In such a situation, when an image search is performed using the photographed image of “H1 temple” as a search target image, the registered images M5 and M6 are similar to the search target image because the registered images M5 and M6 are unclear pictures. The number of times of image recognition (image recognition frequency, image recognition rate) recognized as a (matching) image is small. That is, the number of times (frequency) that the registered images M5 and M6 are listed as higher-order registered images similar to the search target image is small. Therefore, if such registered images M5 and M6 are registered in the image DB, the storage capacity of the image DB is wasted.

  Therefore, for example, the image management unit 80 extracts a registered image (registered photographed image) from the image DB every predetermined period, and deletes the registered images M5 and M6 having a small number of image recognition times (number of upper candidate appearances) from the image DB.

  The registered image M3 is actually an image of “H1 temple”, but the name tag of “H2 temple” has been given. Therefore, even if such a registered image M3 is registered in the image DB, the possibility that the name tag of the registered image M3 is selected as the confirmed name tag in the search result screen of FIG. 3B is extremely low. For this reason, the storage capacity of the image DB is wasted.

  Therefore, the image management unit 80 extracts a registered image from the image DB, for example, every predetermined period, and deletes the registered image M3 with a small number of final selections on the search result screen in FIG.

  In this way, the registered images M5 and M6 that are less likely to be image recognition targets and the registered image M3 that is less likely to be selected as a confirmed name tag are deleted from the image DB. Only the storage capacity of the image DB can be saved. Further, since the registration number of the image DB is prevented from being increased unnecessarily, it is possible to prevent a situation where the image search process is delayed.

  FIG. 8 shows a process flow of the registered image deletion process (cleaning process) described above. First, a registered image is extracted from the image DB and image information is checked (steps S201 and S202). Next, the number of times of image recognition that is the number of hits of image recognition is checked (step S203). If the number of times of image recognition is equal to or less than a predetermined number (specified value), the registered image (registered photographed image) is deleted (steps S204 and S205).

  On the other hand, if the number of times of image recognition is greater than the predetermined number, the number of user's final selections (correct answer selections) on the search result screen is checked (step S206). If the number of final selections is less than or equal to the predetermined number, the registered image is deleted (steps S207 and S208). By performing the processing as described above, it is possible to delete the registered images M3, M5, and M6 in FIG. 7 and save the storage capacity of the image DB.

  In step S202, whether the image type is a master image (registered image or the like prepared in advance on the system side) or a user registered image, registration information (subject) is user registration information or official information (registered photographed image). Determine if there is. Then, according to the conditions set according to the combination of the image type and the registered information type, it is desirable to delete the image when the image recognition frequency information or the like meets the deletion condition.

  The ratio of the number of times of image recognition and the number of selected selections may be incorporated into the deletion condition, and the registered image (registered photographed image) may be deleted. For example, if the deletion condition is only the condition that the number of final selections is small, a registered image obtained by photographing “H1 temple” from an uncommon angle may also be deleted, which may reduce the image recognition rate. . For example, the registered image M3 in FIG. 7 has a large number of image recognitions but a small number of final selections. If such a registered image M3 is always deleted, the image recognition rate may be lowered. Accordingly, in this case, the ratio between the number of times of image recognition and the number of final selections is also included in the deletion condition. For example, registered images having a large ratio of the number of image recognition times to the number of final selections are not deleted even if the number of final selections is small. In other words, a registered image in which the number of confirmed selections is equal to or less than a predetermined number and the ratio of the number of image recognition times to the number of confirmed selections is equal to or less than a predetermined ratio is deleted.

4). Database Structure Next, the data structure of the album DB and the image DB used in this embodiment will be described. FIG. 9A shows an example of the data structure of the album DB. In FIG. 9A, my page ID corresponds to a user ID. For example, MPIDA is user A's my page ID, and MPIDB is user B's my page ID. For example, when a new user sends a mail with a photographed image attached to a specific mail address and performs user registration, for example, a My Page ID is assigned to the mail address of the registered user in a one-to-one correspondence The Then, the captured image attached to the user's mail is written in the storage location of the album DB corresponding to the My Page ID. For example, when user A performs user registration and my page ID = MPIDA is assigned, the captured image attached to the mail of user A is registered in the storage location of my page ID = MPIDA in FIG. Is done. When the user selects a fixed name tag from candidate name tags as shown at B4 in FIG. 3B, the fixed name tag is registered in association with the captured image as shown in FIG. 9A. As described above, in the present embodiment, the captured image and the fixed name tag are registered in the storage location of the album DB specified by the mail address of the received user's mail.

  For example, when position information is acquired by a fixed name tag, it is desirable to register the position information in association with a captured image as shown in FIG. In this way, as shown in C4 to C8 in FIG. 5A, a process of specifying the shooting location and movement history of the shot image using the registered position information and displaying it on the map screen is performed. It can be simplified. In this case, for example, the user may be able to edit the position information registered in the album DB. In this way, the user can finely adjust the position of the shooting location on the map screen of C3 in FIG. 5A, and the convenience for the user can be improved. The position information may not be stored in the album DB. For example, the location information may be acquired with reference to a tourist information DB or the like as necessary.

  FIG. 9B shows an example of the data structure of the image DB. In this image DB, each name tag is registered in association with each registered image data. A plurality of name tags may be associated with one registered image data.

  In this embodiment, for example, when an image search of the user's captured image is performed and the user selects a confirmed name tag as indicated by B4 in FIG. 3B, the captured image and the confirmed name for which the image search has been performed. Tags are also registered in the image DB. That is, the user's photographed image and its fixed name tag are registered in the image DB as a registered image for image search and its name tag. In this way, when various users search for images, the captured images are sequentially added as registered images to the image DB. That is, the number of registered images that serve as reference images for the image DB increases due to use by the user, and the image DB grows. Therefore, the image DB can cope with various photographed images, and user convenience and the like can be improved.

5. Acquisition of Candidate Name Tags Next, an example of a candidate name tag acquisition method will be described. For example, in FIG. 10A, IMT1, IMT2, IMT3, IMT4, IMT5, IMT6... Are extracted from the image DB as registered images similar to the user's captured image. In FIG. 10A, the registered images are listed in descending order of similarity, the registered image IMT1 has the highest similarity, and the registered image IMT2 has the next highest similarity.

  For registered images IMT1, IMT2, IMT3, IMT4, IMT5, and IMT6, name tags “E1 temple”, “E1 temple”, “E2 shrine”, “E1 temple”, “E2 shrine”, "E3 temple" is related. In this case, in the present embodiment, the number of appearances (appearance frequency) of these name tags is totaled, and candidate name tags are acquired based on the totalization results. For example, in FIG. 10A, the numbers of appearances of “E1 temple”, “E2 shrine”, and “E3 temple” are 3, 2, and 1, respectively. For this reason, by rearranging the name tags in FIG. 10A in descending order of appearance numbers and leaving only a predetermined number of name tags with higher appearance numbers, candidate name tags as shown in FIG. To be acquired. In this case, a name identification process is also performed in which duplicate name tags are deleted to be one.

  When counting the number of appearances in the counting process, a weighting count corresponding to the similarity of the image may be multiplied. Moreover, without counting the number of appearances, name identification processing may be performed on the name tags arranged in descending order of similarity, and a predetermined number of higher-level name tags after the name identification processing may be acquired as candidate name tags. .

6). Image Upload In this embodiment, not only photographed images but also images uploaded by a user using an information processing apparatus such as a PC can be registered in the album DB. Specifically, for example, on the album browsing screen shown in FIGS. 5A and 5B, the user can click an image shown in C12 or C13, and the user can select an image stored in the hard disk of the information processing apparatus. Upload and register uploaded images in the album DB.

  For example, when the name tag of the uploaded image is acquired by the image search of the uploaded image using the image DB, the album registration unit 26 in FIG. 1 registers the acquired name tag in the album DB in association with the uploaded image.

  When a plurality of registered images similar to the uploaded image are searched from the image DB and a plurality of uploaded image candidate name tags are acquired from a plurality of name tags associated with the searched plurality of registered images, data creation is performed. The unit 28 creates search result data including the plurality of uploaded image candidate name tags. Then, when the user selects any of the plurality of upload image candidate name tags on the search result screen displayed by the search result data, the album registration unit 26 selects the selected candidate name tag as the upload image fixed name. A tag is registered in association with the uploaded image in the album DB.

  FIG. 11 shows an example of an upload image search result screen (image candidate name selection screen). For example, if the name tag of the subject of the uploaded image can be uniquely determined by image search (match search), the name tag is given to the uploaded image as shown in D1 of FIG. Automatically saved in the corresponding folder. That is, when a registered image that matches (similar) the upload image can be searched, the name tag of the registered image is added to the upload image and registered in the album DB.

  On the other hand, as shown in D2 of FIG. 11, when the registered image matching the upload image cannot be searched, the upload is performed from the name tags of a plurality of registered images similar to the uploaded image, as in the case of the captured image. A candidate name tag for the image is acquired. Specifically, scoring is less than or equal to the specified value in the match search, but for images that have been searched in the similar search, name identification processing is performed on the name tag that is the search result, A name tag is acquired as a candidate name tag.

  Then, as shown by D3 in FIG. 11, these candidate name tags are displayed on the upload image search result screen. When the user selects one of these candidate name tags and clicks “determine name” as shown in D4, the selected name tag is associated with the uploaded image as a confirmed name tag for the uploaded image. To be registered in the album DB. If no registered image similar to the uploaded image can be retrieved, the uploaded image is stored in a folder of other images as shown in D5.

  As described above, if the uploaded image can be registered in the album DB, for example, an image that has not been searched after the user has gone home after traveling, or an image related to the tourist destination of the travel, etc. Can be uploaded using a PC or the like and registered in the album DB together with the photographed image during the trip. At this time, as shown in D1 and D2 of FIG. 11, the image search of the uploaded image is automatically performed, so that a name tag is automatically given to the uploaded image, or the user is selected from the candidate name tags. The name tag selected by is added to the uploaded image. Therefore, the user can organize desired upload images and register them in the album DB with little effort, and the convenience of the user can be improved.

  In this embodiment, such an uploaded image is also registered in the image DB. For example, when image upload is selected by the user, the image search unit 76 in FIG. 1 searches the image DB for a plurality of registered images similar to the upload image. The tag processing unit 78 acquires a plurality of upload image candidate name tags from a plurality of name tags associated with the searched plurality of registered images.

  Then, it is assumed that the user selects one of a plurality of upload image candidate name tags on the upload image search result screen. In this case, the image management unit 80 registers the selected candidate name tag as an upload image fixed name tag in association with the upload image in the image DB. In this way, not only photographed images but also uploaded images can be registered in the image DB as registered images that serve as reference images for image search, and the number of registered image DBs can be increased.

7). Image Recognition Next, a detailed example of image recognition processing will be described. Image recognition processing for image search includes, for example, image recognition using feature points and image recognition using features of the entire image. Image recognition using feature points is a match search used in D1 of FIG. 11 described above, and image recognition using features of the entire image is a similar search used in D2 of FIG.

7.1 Image recognition using feature points (match search)
In image recognition using feature points, first, a blur process is performed on an original image step by step to create a blur image. Then, the difference between the blur images is taken to create a difference image, and the pixel having the extreme value in the difference image (the coordinate whose value is larger or smaller than the surroundings) is used as the feature point. Usually, a plurality of feature points are extracted from one image.

  The feature quantity of each feature point is expressed by a 128-dimensional vector, which is called a feature vector. This feature vector is obtained by adding the differential value of the blur image in a certain range (hereinafter referred to as a pickup range) centered on the feature point obtained as described above to the corresponding dimension.

  The differential value to be added is weighted so that the value becomes larger as it is closer to the feature point. The index of the dimension to be added is obtained from the coordinates obtained by rounding the coordinates of each pixel in the pickup range (hereinafter referred to as pickup coordinates) to 4 × 4 and the orientation obtained by rounding the differential orientation of the pickup coordinates in eight directions. And this feature-value is calculated | required about all the some feature points.

  Next, the above processing is performed on the registered image and the question image (search target image) to extract feature points and feature amounts. Then, the feature amount closest to the feature amount of the feature point extracted from the question image is searched from the registered image. Here, the closeness of the feature amount is expressed by the distance of the feature vector.

  In this way, the feature points of the registered image corresponding to the feature points extracted from the search target image are obtained, and the registered image having the most corresponding feature points is returned as a correct image (similar image). When a plurality of images are returned as correct images, the top J images (J is an integer of 2 or more) are returned as correct images (similar images) in order from the image having many corresponding feature points.

7.2 Image recognition using whole image features (similarity search)
In image recognition using the characteristics of the entire image, the entire image is first converted to a gray scale and converted to a luminance image. The luminance image is reduced to 16 × 16 pixels, and for example, a 256-dimensional vector is used as a composition feature.

  The hue of the entire pixel is divided into 10 sections, for example, and the histogram is used as a color feature of the entire image and expressed as a 10-dimensional vector. Further, the entire image is divided into 512 colors, and the top 30 colors of the occupied area are used as representative color features.

  The above-described processing is performed on the registered image and the question image, and the feature amount is calculated. Then, the registered image having the feature closest to the question image is returned as a correct image (similar image). When a plurality of images are returned as correct images, the top J images are returned in order from the image closest to the question image. The proximity of the feature is expressed by the distance of each feature vector. Also, the weighting is changed depending on whether the composition feature or the color feature is important.

8). Detailed Processing Next, a detailed processing example of the present embodiment will be described with reference to the processing flows of FIGS.

  FIG. 12 and FIG. 13 are flow charts of processing for acquiring candidate name tags by image search of photographed images and registering the confirmed candidate name tags obtained by user selection in the album DB or image DB in association with the photographed images. .

  When the user photographs a subject with the terminal device 110 and sends an email with the photographed image attached thereto, the management server 10 receives the email and checks the email address (steps S1 to S4). If the user specified by the e-mail address is a new user, a My Page ID serving as a user ID is issued and the user information is registered in the personal information DB (steps S5 and S6). On the other hand, if the user is already using the service, the My Page ID is assigned to the photographed image, and the photographed image is transmitted to the image search server 60 and registered in the album DB (steps S7 to S7). S9).

  When receiving the captured image, the image search server 60 extracts feature points of the captured image and performs a matching process with the image DB (steps S10 to S12). That is, image recognition (match search) using the above-described feature points is executed. Then, scoring is calculated to determine whether or not the image recognition is successful (whether or not a recognition result is obtained) (steps S13 and S14).

  If the image recognition is not successful, the composition of the image is extracted and collation processing with the image DB is performed (steps S15 and S16). In other words, image recognition (similarity search) using the above-described overall image characteristics is executed. Then, scoring is calculated to determine whether the image recognition is successful (steps S17 and S18).

  If it is determined in step S14 or S18 that the image recognition has succeeded, the name tags of the registered images that have been searched are listed in order of similarity, and the name tag counting process (name identification process) is performed. Is transmitted to the management server 10 (steps S19 to S22). On the other hand, if it is determined in step S18 that the image recognition is not successful, the image search server 60 transmits an error result (step S51). When receiving the error result, the management server 10 creates an error mail and transmits it to the terminal device 110 (steps S52 to S54). When the user of the terminal device 110 confirms receipt of the error mail and clicks on the URL in the mail, a top screen is displayed (steps S55 to S57). When the user selects image recognition search on the top screen, the mailer is activated and the process returns to step S1, and when text search is selected, text search is performed (steps S58 to S61).

  When receiving the candidate name tag from the image search server 60, the management server 10 searches the tourism information DB based on the candidate name tag and creates a search result screen as shown in FIG. 3B (steps S23 to S25). ). Then, a result notification mail in FIG. 3A that informs the URL of the created search result screen is transmitted to the terminal device 110 (step S26).

  When the user of the terminal device 110 confirms receipt of the result notification mail and selects to browse the search result screen by URL, the search result screen of FIG. 3B is displayed (steps S27 to S29). When the user selects a candidate name tag that is considered to be correct as shown at B4 in FIG. 3B, a tourist information screen (guide information screen) as shown in FIG. 4 is displayed. Browse (steps S30 to S33). Then, the management server 10 determines that the selected candidate name tag is a confirmed name tag, and transmits the captured image and the confirmed name tag to the image search server 60 (steps S34 and S35). Further, by referring to the tourism information DB based on the confirmed name tag, the position information of the shooting location of the shot image is acquired (step S36). Taking B4 in FIG. 3B as an example, the location information of “E1 temple” is acquired from the tourism information DB. Then, the confirmed name tag and position information are registered in the album DB in association with the photographed image (step S37).

  On the other hand, the image search server 60 that has received the photographed image and the confirmed name tag registers the received photographed image and the confirmed name tag in the image DB (steps S38 and S39). As a result, the captured image is registered in the image DB as a registered image that serves as a reference for image search.

  As shown in B7 of FIG. 3B, when the user selects an image search on the search result screen, the mailer is activated and returns to step S1 of FIG. 12 (steps S40 and S41). When the text search is selected as shown in B8, a text search referring to the tourism information DB is executed, and when desired information is searched from the tourism information DB, a text search search result screen is displayed ( Steps S61 to S63, S29). On the other hand, if the information cannot be searched, the search by the external site is executed, the search result from the external site is received, and the search result screen is displayed (steps S64 to S66, S29).

  FIG. 14 and FIG. 15 are flowcharts of processing for acquiring image position information of a user when the image recognition of the photographed image is not successful, narrowing search targets based on the acquired position information, and performing image recognition. is there.

  Steps S71 to S83 in FIG. 14 are the same as steps S1 to S13 in FIG. If the image search server 60 determines in step S84 that the image recognition has not been successful, the image search server 60 instructs (requests) the management server 10 to acquire position information (step S91).

  Then, the management server 10 creates a position information acquisition screen and transmits an email informing the URL of the screen to the terminal device 110 (steps S92 and S93). When the user of the terminal device 110 confirms the reception of the mail and clicks on the URL in the mail, a position information acquisition screen is displayed (steps S94 to S96). When the user selects a position information acquisition command on this position information acquisition screen and permits acquisition of position information, the management server 10 performs a simple position information acquisition process (steps S97 to S99). For example, user location information is acquired based on base station information of the terminal device 110 such as a mobile phone. Or you may acquire position information using GPS which the terminal device 110 has. Then, the acquired position information is transmitted to the image search server 60 (step S100).

  When receiving the position information from the management server 10, the image search server 60 narrows down target images serving as a reference for the image search based on the received user position information (steps S 101 and S 102). Then, the composition of the image is extracted and collation processing with the image DB is performed (steps S103 and S104). That is, image recognition (similarity search) using the characteristics of the entire image is executed. Then, scoring is calculated and it is determined whether or not the image recognition is successful (steps S105 and S106).

  If it is determined that the image recognition has been successful, name tag counting processing or the like is performed, and the acquired candidate name tag is transmitted to the management server 10 (steps S107 to S110). Then, the management server 10 searches the tourism information DB based on the received candidate name tag, creates a search result screen, transmits an email notifying the URL of the screen, and the terminal device 110 receives the email (step) S111 to S115). Thereafter, the same processing as the normal flow described in FIGS. 12 and 13 is performed.

  If it is determined in step S106 that the image recognition has not been successful, the image search server 60 transmits an error result and the management server 10 receives it (steps S121 and S122). And the management server 10 produces and transmits an error mail, and the terminal device 110 receives (steps S123 to S125).

  According to the processes in FIGS. 14 and 15 described above, even when the first image recognition of the photographed image is not successful, the image search is performed by narrowing down the target image based on the user position information. The candidate name tag of the photographed image can be acquired and presented to the user.

  16, 17, and 18 are flowcharts of the album screen browsing process and the upload image registration process in the album DB.

  When the user of the terminal device 110 selects browsing of the album screen on the My Page top screen, the management server 10 extracts the target album from the album DB and displays the album selection screen (steps S131 to S134). When the user selects a desired album, a map screen for that album is displayed (steps S135 and S136). For example, in FIG. 5A, since the Osaka album is selected, the Osaka map screen is displayed. When the user clicks C9 in FIG. 5A and selects to browse the album, an album image list screen is displayed as shown in FIG. 5B (steps S137 and S138). When the user selects an image on this list screen, an image detail screen is displayed as shown in FIG. 6, and the user can input a comment as shown at C11 (steps S139 and S140). When the user inputs a comment, the input comment is associated with the photographed image and registered in the album DB (step S141).

  On the other hand, when browsing the album in FIG. 5A, when the user selects upload of an image as shown in C12, a mailer is activated in the case of a mobile such as a mobile phone, and the upload image is transmitted to the management server 10. In the case of a PC, the upload image attached by referring to the file is selected, and the upload image is transmitted (steps S151 to S153). Then, the management server 10 transmits the received upload image to the image search server 60 (step S154).

  When receiving the uploaded image, the image search server 60 extracts the feature points of the image, performs a matching process with the image DB, and calculates scoring (steps S155 to S158). If image recognition is successful through this match search, the tag of the registered image is acquired as a name tag, transmitted to the management server 10, and the uploaded image is registered in the image DB as a correct image (steps S159 to S159). S162). On the other hand, when receiving the name tag, the management server 10 acquires position information from the tourism information DB based on the name tag, and registers the uploaded image, name tag, and position information in the album DB (steps S163 to S165). Thus, the uploaded image can be uniquely recognized by the match search, and when the name tag is acquired, the uploaded image and the name tag are registered in the album DB. Then, in the upload image search result screen of FIG. 11, the user is informed as indicated by D1.

  If the image recognition is not successful in step S159, the composition of the image is extracted, collation processing with the image DB is performed, and scoring is calculated (steps S171 to S173). If the image recognition is not successful, the error result is transmitted to the management server 10, and the upload image is registered as another image as described in D5 of FIG. 11 (steps S174 to S177). On the other hand, when the image recognition is successful, a totaling process (name identification process) of the upper-listed name tags is performed, and candidate name tags are acquired (steps S178 to S180).

  When the acquired candidate name tag is transmitted from the image search server 60 to the management server 10, an unconfirmed folder is registered, and an upload image search result screen is created and displayed (steps S181 to S181). S185). When the user confirms the search result screen and selects the upload image candidate name tag as the confirmed name tag as shown in D3 and D4 of FIG. 11, the management server 10 extracts the target image from the album DB. Then, the location information of the tourist facility corresponding to the confirmed name tag is acquired from the tourist information DB (steps S186 to S189). Then, the upload image, the confirmed name tag, and the position information are registered in the album DB and transmitted to the image search server 60 (steps S190 and S191). Then, the image search server 60 registers the received upload image and confirmed name tag in the image DB (steps S192 and S193).

  By doing as described above, the user can register not only images actually captured by the user with the terminal device 110 but also desired upload images in the album DB and the image DB. In addition, image recognition of the uploaded image is performed, and the name tag is automatically acquired, or the confirmed name tag is determined by the user selecting from the candidate name tags, so the convenience of the user can be improved. .

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. The configurations and operations of the server system and the terminal device are not limited to those described in the present embodiment, and various modifications can be made.

10 management server, 20 processing unit, 22 reception processing unit, 24 transmission processing unit,
26 album registration unit, 28 data creation unit, 30 location information acquisition unit,
32 album browsing section, 34 content presentation section, 40 album database,
42 personal information database, 50 content database,
52 tourism information database, 54 recommendation information database,
56 advertising information database, 58 communication department,
60 image search server, 70 processing unit, 72 reception processing unit, 74 transmission processing unit,
76 image search unit, 78 tag processing unit, 80 image management unit,
90 image database, 98 communication department,
100 network, 110 terminal device, 120 processing unit, 122 image acquisition unit,
124 transmission processing unit, 126 reception processing unit, 128 display control unit,
140 storage unit, 150 information storage medium, 160 imaging unit, 170 operation unit,
180 display units, 200 information processing apparatus

Claims (19)

A reception processing unit that performs processing for receiving transmission data about a captured image of a user who is a target of image search via a network;
An image search unit for searching a plurality of registered images similar to the photographed image from an image database;
A tag processing unit for obtaining a plurality of candidate name tags from a plurality of name tags associated with the plurality of registered images searched;
When the user selects any of the plurality of candidate name tags on the search result screen of the image search of the photographed image, the selected candidate name tag is associated with the photographed image as a confirmed name tag of the photographed image. An image management unit registered in the image database;
Including
The image management unit
Extracting extracted registered photographed images from the plurality of registered images,
Check the image recognition number information, which is information about the number of times that the extracted registered photographed image is recognized as an image similar to the search target image,
Check the selected selection number information, which is information about the number of times that the extracted registered photographed image is selected by the user on the search result screen,
Check the ratio information, which is information about the ratio between the number of image recognition times of the extracted registered photographed image and the number of confirmed selections,
When it is determined that the ratio is equal to or less than a predetermined value, the extracted registered photographed image is deleted from the image database. In claim 1,
The tag processing unit
A server system, wherein the number of appearances of the plurality of name tags is totaled, and the plurality of candidate name tags are acquired based on a totaling result. In claim 1 or 2,
Including a position information acquisition unit for acquiring user position information;
The image search unit
When the image recognition of the captured image is not successful, the user is instructed to acquire the position information by the position information acquisition unit, and the search target is narrowed down based on the acquired position information, and the image search is performed. A server system characterized by that. In any one of Claims 1 thru | or 3,
An album registration unit for performing processing for registering the photographed image in the album database;
A data creation unit that creates search result data including the plurality of obtained candidate name tags,
The album registration unit
In the search result screen displayed by the search result data, when the user selects any of the plurality of candidate name tags, the selected candidate name tag is used as the confirmed name tag of the photographed image, and the album A server system for registering in association with the photographed image in a database. In claim 4,
The reception processing unit
A process of receiving an email from a user attached with the transmission data about the photographed image,
The album registration unit
A server system, wherein the photographed image is registered in a storage location of the album database specified by a mail address of the received mail. In claim 4 or 5,
Based on the fixed name tag, including a position information acquisition unit that acquires position information of the shooting location of the captured image,
The album registration unit
The server system, wherein the acquired location information of the shooting location is registered in association with the shot image of the album database. In any one of Claims 4 thru | or 6.
A server system comprising: an album browsing unit that performs processing for a user to browse album information in the album database. In claim 7,
The album browsing section
A server system for performing processing for displaying on a map screen at least one of a shooting location of the shot image registered in the album database and a movement history of the shooting location. In claim 8,
Based on the fixed name tag, including a position information acquisition unit that acquires position information of the shooting location of the captured image,
The album browsing section
A server system characterized by specifying a display position of the shooting location on the map screen or the movement history based on the acquired position information. In any one of Claims 4 thru | or 9.
The album registration unit
A server system that registers an uploaded image in the album database when uploading of an image is selected by the user when the user browses the album. In any one of Claims 1 thru | or 10.
The image search unit
When uploading an image is selected by the user, a plurality of registered images similar to the uploaded image are searched from the image database,
The tag processing unit
From a plurality of name tags associated with the searched plurality of registered images, obtain a plurality of upload image candidate name tags,
The image management unit
In the upload image search result screen, when the user selects any one of the plurality of upload image candidate name tags, the selected candidate name tag is associated with the upload image as the upload image fixed name tag. A server system registered in the image database. An image acquisition unit that acquires a captured image of a user who is a target of image search;
A transmission processing unit for performing processing for transmitting transmission data about the acquired captured image via a network;
A reception processing unit that performs processing for receiving search result data of an image search for the captured image via a network;
A display control unit that performs display control of the display unit based on the received search result data,
The display control unit
Acquired when a plurality of registered images similar to the photographed image are searched from the image database, and a plurality of candidate name tags are acquired from a plurality of name tags associated with the searched plurality of registered images. A search result screen on which the plurality of candidate name tags are displayed is controlled to be displayed on the display unit,
The transmission processing unit
In the search result screen, when the user selects any of the plurality of candidate name tags, the selected candidate name tag is used as a confirmed name tag of the captured image registered in the image database via the network. Process to send,
By a server system that is communicatively connected to a terminal device via a network,
Extracted registered photographed images are extracted from the plurality of registered images,
Image recognition number information, which is information about the number of times the extracted registered photographed image is recognized as an image similar to the search target image, is checked,
Confirmed selection number information that is information about the number of times that the extracted registered photographed image is selected by the user on the search result screen is checked,
Ratio information that is information about the ratio between the number of times of image recognition and the number of selected selections of the extracted registered photographed image is checked,
When it is determined that the ratio is equal to or less than a predetermined value, the extracted registered captured image is deleted from the plurality of registered images. In claim 12,
The transmission processing unit
In the search result screen, when the user selects any of the plurality of candidate name tags, the selected candidate name tag is used as the confirmed name tag of the captured image registered in the album database via the network. The terminal device characterized by performing the process of transmitting. In claim 13,
The display control unit
A terminal device that performs control to display an album browsing screen based on album information in the album database on the display unit. In claim 14,
The display control unit
A terminal device that controls to display on the display unit a map screen on which at least one of a shooting location of the shot image registered in the album database and a movement history of the shooting location are displayed. In any of claims 12 to 15,
The display control unit
An image is uploaded by the user, a plurality of registered images similar to the uploaded image are searched from the image database, and a plurality of uploaded images are retrieved from a plurality of name tags associated with the searched plurality of registered images. When a candidate name tag is acquired, control is performed to display a search result screen on which the acquired plurality of uploaded image candidate name tags are displayed on the display unit,
The transmission processing unit
In the search result screen, when the user selects any of the plurality of upload image candidate name tags, a process of transmitting the selected candidate name tag as the upload image fixed name tag via the network is performed. A terminal device characterized by that. An image acquisition unit that acquires a captured image of a user who is a target of image search;
A transmission processing unit for performing processing for transmitting transmission data about the acquired captured image via a network;
A reception processing unit that performs processing for receiving search result data of an image search for the captured image via a network;
As a display control unit that performs display control of the display unit based on the received search result data,
Make the computer work,
The display control unit
Acquired when a plurality of registered images similar to the photographed image are searched from the image database, and a plurality of candidate name tags are acquired from a plurality of name tags associated with the searched plurality of registered images. A search result screen on which the plurality of candidate name tags are displayed is controlled to be displayed on the display unit,
The transmission processing unit
In the search result screen, when the user selects any of the plurality of candidate name tags, the selected candidate name tag is used as a confirmed name tag of the captured image registered in the image database via the network. Process to send,
Extracted registered photographed images are extracted from the plurality of registered images,
Image recognition number information, which is information about the number of times the extracted registered photographed image is recognized as an image similar to the search target image, is checked,
Confirmed selection number information that is information about the number of times that the extracted registered photographed image is selected by the user on the search result screen is checked,
Ratio information that is information about the ratio between the number of times of image recognition and the number of selected selections of the extracted registered photographed image is checked,
When it is determined that the ratio is equal to or less than a predetermined value, the extracted registered photographed image is deleted from the plurality of registered images.   A computer-readable information storage medium, wherein the program according to claim 17 is stored. An image search step in which the image search unit searches the image database for a plurality of registered images similar to the captured images of the user who is the target of image search;
A tag processing step in which a tag processing unit obtains a plurality of candidate name tags from a plurality of name tags associated with the plurality of registered images searched;
A display step in which a display unit displays a search result screen on which the obtained plurality of candidate name tags are displayed;
In the search result screen, when the user selects any of the plurality of candidate name tags, the selected candidate name tag is used as a confirmed name tag of the photographed image and is associated with the photographed image in the image database. An image management step registered by the management unit;
Including
The image management step includes
Extracting extracted registered photographed images from the plurality of registered images,
Check the image recognition number information, which is information about the number of times that the extracted registered photographed image is recognized as an image similar to the search target image,
Check the selected selection number information, which is information about the number of times that the extracted registered photographed image is selected by the user on the search result screen,
Check the ratio information, which is information about the ratio between the number of image recognition times of the extracted registered photographed image and the number of confirmed selections,
If it is determined that the ratio is equal to or less than a predetermined value, the extracted registered captured image is deleted from the image database;
An image retrieval method using computer software, comprising steps.

Images