JP2010218373A - Server system, terminal apparatus, program, information storage medium, and image retrieving method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a server system etc. for acquiring positional information by effectively utilizing an image recognition result. <P>SOLUTION: The server system includes: a reception processing part for receiving transmission data on a user's photographic image for image retrieval through a network; a positional information acquisition part for acquiring the positional information on the place where the image is captured based on a result of image-recognizing the photographic image; and a content presentation part for extracting content information corresponding to the acquired positional information from a content database and presenting the content information to the user. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a server system, a terminal device, a program, an information storage medium, an image search method, and the like.

  2. Description of the Related Art Conventionally, there has been known an image search system that searches for products or the like taken by a camera using image recognition processing. For example, in Patent Document 1, when a user captures a desired product and transmits a captured image to a server via a network, the server searches for product information based on the captured image, and uses the retrieved product information as a user. A system for replying to the terminal device is disclosed.

  However, in the conventional image search system, the image recognition result is used only for image search of products and the like, and no system has been proposed for specifying position information using the image recognition result.

  Further, in the image retrieval system so far, the photographed image is only used for product retrieval and the like, and the photographed image subject to image retrieval is stored in the system side and re-presented to the user. Neither was proposed.

JP 2006-285654 A

  According to some aspects of the present invention, it is possible to provide a server system, a terminal device, a program, an information storage medium, an image search method, and the like that can acquire position information by effectively using an image recognition result.

  One embodiment of the present invention is based on a reception processing unit that performs processing for receiving transmission data regarding a captured image of a user who is an object of image search via a network, and based on an image recognition result of the captured image. A location information acquisition unit that acquires location information of a shooting location of an image, and a content presentation unit that extracts content information corresponding to the acquired location information from a content database and performs processing for presenting the content information to a user Related to the server system.

  In one aspect of the present invention, when transmission data about a user's captured image is received via a network, image recognition of the received captured image is performed, and position information of the shooting location is acquired based on the image recognition result. Is done. Then, content information corresponding to the acquired position information is extracted from the content database and presented to the user. Accordingly, while performing image recognition of a photographed image, content information can be provided by acquiring position information of a photographing location by effectively utilizing the result of the image recognition. Also, by acquiring position information from the image recognition result, it is possible to acquire meaningful position information (for example, what the user is looking at and what is interested in) that cannot be acquired from simple latitude / longitude information. It becomes possible.

  In one aspect of the present invention, the content presentation unit selects recommendation information at the shooting location from the content database based on the acquired position information, and presents the selected recommendation information to the user. You may perform the process for.

  In this way, recommendation information at the shooting location specified by the image recognition result of the shot image can be presented to the user who shot the image.

  Moreover, in one aspect of the present invention, the content presentation unit presents a recommended route to the user, and when the user selects the recommended route, the content presenting unit is for presenting the recommended information in the recommended route to the user. Processing may be performed.

  In this way, it is possible to present a recommended route at the photographing location specified by the image recognition result of the photographed image to the user who photographed the image.

  In the aspect of the invention, the content presentation unit may create the recommended route based on the position information acquired for another user.

  If it does in this way, based on the positional information acquired from the picked-up image of various users, a recommendation route | root can be created and it can show now to each user.

  Moreover, in one aspect of the present invention, a transmission process for performing a process of transmitting notification information notifying that the user is located to an information processing apparatus for a nearby store at the location of the user specified based on the position information. Part may be included.

  In this way, the location of the user is specified by the location information of the shooting location acquired from the image recognition result, and the user is located nearby with respect to the information processing device for stores around the location. It becomes possible to inform.

  Moreover, in one aspect of the present invention, the transmission processing unit may perform a process of transmitting the notification information to an information processing device for stores around the user's movement route predicted based on the position information. .

  In this way, it is possible to inform the store information processing apparatus around the predicted user movement path that the user is nearby.

  In the aspect of the invention, the content presentation unit may perform processing for presenting the advertisement information to a user when the information processing apparatus that has received the notification information requests distribution of the advertisement information. Also good.

  If it does in this way, the advertising information from the information processing apparatus for peripheral shops informed that the user is located nearby can be presented to the user.

  Moreover, in one aspect of the present invention, the aggregation processing unit that performs the aggregation processing of the position information acquired for a plurality of users and the information processing device for a store near the shooting location are obtained by the aggregation processing. You may include the transmission process part which performs the process which transmits total information.

  In this way, the total information of the position information acquired by the image recognition can be notified to the information processing apparatus for the stores near the shooting location.

  In the aspect of the invention, the transmission processing unit may perform a process of transmitting aggregate information of the number of users at the shooting location to the information processing apparatus.

  In this way, the information processing apparatus for neighboring stores can know the total information of the number of users located at the shooting location.

  In another aspect of the present invention, a plurality of registered images similar to the captured image are searched from an image database, and a plurality of candidate name tags are extracted from a plurality of name tags associated with the searched plurality of registered images. A data creation unit that creates search result data including the plurality of candidate name tags that have been acquired when acquired, wherein the position information acquisition unit is a search result screen displayed by the search result data, When one of a plurality of candidate name tags is selected by the user, the selected candidate name tag is determined to be a confirmed name tag of the captured image, and based on the confirmed name tag, the shooting location tag It relates to a server system that acquires position information.

  In this way, when the user selects any of the plurality of candidate name tags on the search result screen, the selected fixed name tag is determined to indicate the name of the subject of the captured image, and the shooting location is determined. The position information of can be acquired.

  In another aspect of the present invention, an image search unit that searches the image database for the plurality of registered images similar to the photographed image, and the plurality of name tags associated with the plurality of registered images that are searched. A tag processing unit that acquires the plurality of candidate name tags may be included.

If it does in this way, a candidate name tag will be acquirable by the image search of a picked-up image. Moreover, in the other aspect of this invention, the said tag process part will totalize the appearance count of these several name tags, and a total result The plurality of candidate name tags may be acquired based on the above.

  In this way, candidate name tags can be selected and acquired according to the number of appearance of name tags.

  In another aspect of the present invention, an album registration unit that performs processing for registering the photographed image in an album database is included, and the album registration unit selects any one of the plurality of candidate name tags on the search result screen. When the is selected, the fixed name tag of the photographed image may be registered in association with the photographed image of the album database.

  In this way, it becomes possible to view the photographed image as the target of the image search as an album, and an image search of an unprecedented type can be realized.

  In another aspect of the invention, the reception processing unit performs a process of receiving a mail from a user attached with the transmission data regarding the captured image, and the album registration unit is configured to receive the mail of the received mail. The captured image may be registered in a storage location of the album database specified by an address.

  In this way, simply by sending a mail with a captured image attached thereto, the captured image is registered in the storage location specified by the mail address, so the convenience of the user can be improved.

  In another aspect of the present invention, the album registration unit may register the acquired location information of the shooting location in association with the shot image in the album database.

  In this way, the position information of the shooting location of the shot image can be specified by the confirmed name tag selected by the user and registered in the album database.

  In another aspect of the present invention, an album browsing unit that performs processing for the user to browse album information in the album database may be included.

  In this way, the user can browse and enjoy the album in which the photographed image subject to image search is automatically registered.

  According to another aspect of the present invention, the album browsing unit displays at least one of the shooting location of the captured image registered in the album database and the movement history of the shooting location on a map screen. May be performed.

  In this way, the shooting location of the shot image remains as a trajectory on the map screen, so that the user can enjoy the afterglow of the trip.

  In another aspect of the invention, the album browsing unit may identify the display position of the shooting location on the map screen or the movement history based on the acquired location information of the shooting location. Good.

  In this way, it is possible to identify the location information of the shooting location using the confirmed name tag and display the shooting location and its movement history on the map screen.

  According to another aspect of the present invention, a reception processing unit that performs processing for receiving transmission data of a user's captured image that is an object of image search via a network, and a plurality of registered images that are similar to the captured image. An image search unit that searches from an image database, a tag processing unit that acquires a plurality of candidate name tags from a plurality of name tags associated with the searched plurality of registered images, and a search result of the image search of the captured image In the screen, when the user selects any one of the plurality of candidate name tags, the selected candidate name tag is determined as the confirmed name tag of the captured image, and based on the confirmed name tag, the captured image The present invention relates to a server system including a position information acquisition unit that acquires position information of a shooting location.

  According to another aspect of the present invention, when transmission data about a user's captured image is received via a network, a registered image similar to the received captured image is searched, and the captured image is searched from the registered image name tag. Candidate name tags are acquired. When the user selects one of the candidate name tags on the search result screen of the image search, the selected candidate name tag is determined as a confirmed name tag of the photographed image, and the location information of the photographing location is determined based on the confirmed name tag. Can be obtained. Therefore, meaningful position information that cannot be acquired from simple latitude / longitude information can be acquired.

  According to another aspect of the present invention, an image acquisition unit that acquires a captured image of a user who is a target of image search, and a transmission processing unit that performs processing of transmitting transmission data regarding the acquired captured image via a network. A reception processing unit that performs processing for receiving search result data of an image search for the captured image via a network, and a display control unit that performs display control of a display unit, wherein the display control unit includes the captured image When the position information of the shooting location of the captured image is acquired based on the image recognition result, and the content information corresponding to the acquired position information is extracted from the content database, the content information is displayed on the display unit. The present invention relates to a terminal device that performs display control. An aspect of the present invention relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to another aspect of the present invention, when a captured image that is a target of an image search is acquired, the captured image is transmitted via a network. Then, when image recognition of the captured image is performed, position information on the shooting location of the captured image is acquired based on the image recognition result, and content information corresponding to the acquired position information is displayed on the display unit. . Therefore, while performing image recognition of a captured image, it is possible to provide content information by acquiring position information by effectively using the result of the image recognition.

  In another aspect of the present invention, the display control unit retrieves a plurality of registered images similar to the captured image from an image database, and from a plurality of name tags associated with the retrieved registered images. When a plurality of candidate name tags are acquired, control is performed to display a search result screen on which the acquired plurality of candidate name tags are displayed on the display unit, and the transmission processing unit is configured to display the search result screen. When the user selects any of the plurality of candidate name tags, a process of transmitting the selected candidate name tag as a confirmed name tag for acquiring the position information is performed. Also good.

  In this way, when the user selects one of the plurality of candidate name tags on the search result screen, the selected candidate name tag is transmitted as a confirmed name tag for acquiring position information.

  In another aspect of the present invention, a plurality of registered images similar to a user's captured image that is an object of image search are searched from an image database, and a plurality of name tags associated with the searched plurality of registered images are used. , Obtaining a plurality of candidate name tags, displaying a search result screen on which the obtained plurality of candidate name tags are displayed, and selecting one of the plurality of candidate name tags on the search result screen In this case, the selected candidate name tag is determined as a confirmed name tag of the photographed image, and the present invention relates to an image search method for acquiring a photographing location of the photographed image based on the confirmed name tag.

The system configuration example of this embodiment. The 1st processing flow for demonstrating the basic operation | movement of this embodiment. The 2nd processing flow for demonstrating the basic operation | movement of this embodiment. The 3rd processing flow for demonstrating the basic operation | movement of this embodiment. The processing flow for demonstrating the detailed operation | movement of this embodiment. 6A and 6B show examples of a result notification mail and a search result screen. FIGS. 7A and 7B are examples of a sightseeing information screen and a peripheral store information screen. An example of a store information screen. 9A and 9B show examples of the album browsing screen. An example of an image details screen. 11A and 11B show examples of the data structure of the album database and the image database. 12A and 12B are explanatory diagrams of tag processing. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment. The detailed processing flow of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. System Configuration FIG. 1 shows the system configuration of the image search system of this embodiment. This image search system is realized by a server system realized by the management server 10 or the image search server 60, a terminal device 110 (communication terminal, portable terminal device, electronic device), an information processing device 200, and the like. The management server 10, the image search server 60, the terminal device 110, and the information processing device 200 are communicatively connected via a network 100. Other external servers and terminal devices may be connected to the network 100 for communication.

  FIG. 2 shows a first processing flow for explaining the basic operation of the present embodiment. In this embodiment, when the management server 10 receives transmission data about a captured image, the management server 10 acquires positional information on the shooting location of the captured image based on the image recognition result (candidate name tag or the like) of the captured image (Step S401, S402). Then, content information such as recommendation information and advertisement information corresponding to the acquired position information is extracted from the content database 50 and displayed on the display unit 180 of the terminal device 110 (steps S403 and S404).

  Specifically, when the user captures an image with the terminal device 110 such as a camera-equipped mobile phone, the image search server 60 refers to the image database 90 to search for a registered image that is similar (matches) to the user's captured image. To do. And based on the name tag (tag in a broad sense) of the searched registered image, the candidate name tag of the photographed image is acquired. For example, when the subject of the photographed image is a temple, temple name candidates that are similar (match) to the subject of the photographed image are acquired as candidate name tags.

  Then, search result data including the acquired candidate name tag is created, and a search result screen as shown in FIG. As a result, the created search result screen is displayed on the display unit 180 of the terminal device 110. For example, in the search result screen of FIG. 6B, the names “E1 temple”, “E2 shrine”, and “E3 temple” are displayed as candidate name tags that are image recognition results.

  The user selects a name tag that he / she thinks is correct from among these candidate name tags displayed on the search result screen. For example, in B4 of FIG. 6B, the user determines that the temple photographed by himself is “E1 temple”, and therefore selects the name tag “E1 temple”. Thus, the name tag selected by the user is determined to be a confirmed name tag (correct tag). Then, the position information of the shooting location of the shot image is acquired based on the fixed name tag, and the content information corresponding to the acquired position information is presented to the user. For example, when “E1 temple” is selected as the confirmed name tag as shown in B4 of FIG. 6B, the position information of “E1 temple” is acquired. Then, as indicated by D1 in FIG. 7A, based on the position information of “E1 temple”, recommendation information such as recommended routes around “E1 temple” is displayed to the user. In addition, as shown in D2, advertisement information of neighboring stores is also displayed. Further, for example, when the user clicks “search for nearby stores” of D3, peripheral store information as shown in FIG. 7B is displayed, and when D4 is clicked, detailed store information as shown in FIG. 8 is displayed. The

  FIG. 3 is a second processing flow for explaining the basic operation of the present embodiment. In FIG. 3, when the location information of the shooting location is acquired based on the image recognition result of the captured image, the management server 10 specifies the location of the user or predicts the movement route based on the acquired location information. (Steps S411, S412, and S413) And the notification information which notifies that a user is located is transmitted with respect to the information processing apparatus 200 for the shop of the periphery of a user's location or movement route (step S414). When a distribution request for advertisement information is received from the information processing apparatus 200 for the neighboring stores, the advertisement information is displayed on the display unit 180 of the terminal apparatus 110 as shown at D2 in FIG. 7A (step S415). , S416). In this way, the location of the user is identified from the image recognition result of the captured image, the travel route is predicted, and the advertisement information from the information processing device of the store in the vicinity of the user's location or travel route is displayed to the user. Can be presented.

  FIG. 4 is a third processing flow for explaining the basic operation of the present embodiment. In FIG. 4, when the position information of the shooting location is acquired based on the image recognition result of the captured image, the management server 10 performs a counting process such as the number of users based on the acquired position information (step S421, S422, S423). Then, the total information such as the number of users is transmitted to the information processing apparatus 200 for the nearby stores at the shooting location (step S424). When a distribution request for advertisement information is received from the information processing apparatus 200 for nearby stores, the advertisement information is displayed on the display unit 180 of the terminal device 110 (steps S425 and S426). In this way, the position information obtained from the image recognition result can be aggregated, and the aggregate information can be notified to the information processing apparatus in the store around the shooting location.

  Hereinafter, the configuration of the present embodiment will be specifically described with reference to FIG. The management server 10, the image search server 60, the terminal device 110, and the like are not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of these components or adding other components. Is possible. The server system according to the present embodiment can be realized by one of the management server 10 and the image search server 60, or can be realized by both the management server 10 and the image search server 60, for example.

  The network 100 (communication network) can include a mobile communication network, a public network such as the Internet network, or a fixed telephone network. The mobile communication network for communication connection with the terminal device 110 is a mobile telephone network such as a CDMA (Code Division Multiple Access) system or a PDC (Personal Digital Cellular) system, or mobile data such as a PDC-P (PDC-Packet) system. A communication network can be included. The network 100 can be realized by a WAN (Wide Area Network), a LAN (Local Area Network), or the like, regardless of wired or wireless.

  The management server 10 is a server in charge of management processing or the like in the server system, and is, for example, a web server or a mail server. The image search server 60 is a server in charge of image search and tag processing. The functions of the management server 10 and the image search server 60 can be realized by hardware such as an information processing apparatus and a program such as a CGI program.

  The terminal device 110 can transfer images from, for example, a camera-equipped cellular phone, a digital camera (digital still camera, digital video camera), an information processing terminal having a camera function, or an information processing terminal having a camera function, for example. A device (PC) or the like, and has a function of acquiring a captured image and a function of communicating with the management server 10 and the image search server 60 via the network 100. The information processing apparatus 200 is a store or user PC (Personal Computer) or the like.

  The management server 10 includes a processing unit 20, an album database 40, a personal information database 42, a content database 50, and a communication unit 58. The processing unit 20 performs overall system management processing, database management processing, data creation processing, and the like, and functions thereof include hardware such as various processors (CPU, etc.), PHPPHP (Hypertext Preprocessor), Perl, and the like. This can be realized by a CGI program written in a language.

  The album database 40, the personal information database 42, and the content database 50 are relational databases realized by, for example, MySQL. The album database 40 stores user's captured images as album information. In association with the photographed image, the name tag (determined name tag), the position information of the photographing location (user), and the like are stored as album information. The personal information database 42 stores various types of user information. The content database 50 stores various content information, and may include a tourist information database 52, a recommendation information database 54, an advertisement information database 56, and the like. The communication unit 58 is for communicating information with the terminal device 110, the information processing apparatus 200, and other external servers via the network 100.

  The image search server 60 includes a processing unit 70, an image database 90, and a communication unit 98. The processing unit 70 performs image search processing, tag processing, and the like, and its functions can be realized by various processors and programs.

  The image database 90 is a relational database that stores registered images for image search. The communication unit 98 is for communicating information with the management server 10 or the like via the network 100.

  The terminal device 110 includes a processing unit 120, a storage unit 140, an information storage medium 150, a communication unit 158, an imaging unit 160, an operation unit 170, and a display unit 180.

  The processing unit 120 performs various processes based on operation information from the operation unit 170, information received by the communication unit 158, and the like. The function of the processing unit 120 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program stored in the information storage medium 150, and the like.

  The storage unit 140 serves as a work area for the processing unit 120, the communication unit 158, and the like, and its functions can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like.

  The information storage medium 150 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by an optical disk (CD, DVD), an IC card (memory card), or the like. The processing unit 120 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 150. That is, in the information storage medium 150, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The communication unit 158 is for communicating information with the management server 10, the image search server 60, and other terminal devices via the network 100. The imaging unit 160 is for imaging (photographing) a subject, and can be realized by, for example, a CCD or a CMOS sensor. The operation unit 170 is for the user to perform various operations on the terminal device 110, and can be realized by various buttons (shutter buttons), GUI (Graphical User Interface), and the like. The display unit 180 is for displaying various display screens, and can be realized by, for example, a liquid crystal display, an organic EL display, a CRT, or the like.

  The processing unit 20 of the management server 10 includes a reception processing unit 22, a transmission processing unit 24, an album registration unit 26, a data creation unit 28, a position information acquisition unit 30, an album browsing unit 32, a content presentation unit 34, and a totalization processing unit 36. Including. Various modifications such as omission of some of these components and addition of other components are possible.

  The reception processing unit 22 performs data reception processing, and the transmission processing unit 24 performs data transmission processing. For example, the reception processing unit 22 receives transmission data of a user's captured image (search target image or image in a broad sense) as an image search target via the network 100 from, for example, the terminal device 110. Here, the transmission data regarding the photographed image may be the photographed image data itself or the photographed image recognition data (image recognition data). As the recognition data, for example, feature amount data (feature data) of a captured image, data specifying a search target image, and the like can be assumed.

  The album registration unit 26 performs processing for registering album information (images, tags, positions, etc.) in the album database 40. Specifically, a process of registering (writing) a captured image (captured image data) received via the network 100 in the album database 40 is performed. For example, it is assumed that the reception processing unit 22 receives an e-mail from a user attached with transmission data regarding a captured image. In this case, the album registration unit 26 takes a photographed image or a fixed name tag (correct tag) in the storage location (registration location) of the album database 40 specified by the email address (my page ID, user ID) of the received email. Register.

  The data creation unit (search result screen creation unit) 28 creates various data such as search result data. For example, a plurality of registered images similar to the photographed image are searched from the image database 90, and a plurality of candidate name tags (candidate tags) are acquired from a plurality of name tags (tags) associated with the searched plurality of registered images. Suppose. In this case, the data creation unit 28 creates search result data including a plurality of acquired candidate name tags. That is, data for displaying the search result screen shown in FIG. 6B is created.

  When the user selects one of a plurality of candidate name tags on the search result screen of FIG. 6B, the album registration unit 26 selects the selected candidate name tag as a confirmed name tag (correct tag, correct tag, As a selection tag, it is registered in association with the photographed image (album registered image) in the album database 40. As a result, the user can browse the album in which the image taken by the user and the name tag selected by the user are displayed.

  The position information acquisition unit 30 acquires position information such as the shooting location (user location) of the shot image based on the image recognition result of the shot image. Specifically, for example, the position information of the shooting location is acquired based on the confirmed name tag. For example, when the confirmed name tag selected by the user is “E1 temple” in FIG. 6B, the position of “E1 temple” is set as the shooting location of the shot image based on the sightseeing information in the tourism information DB. Obtained as location information. The album registration unit 26 registers the acquired location information of the shooting location in association with the shot image in the album database 40. By doing so, the position information of the shooting location of each captured image can be acquired by a simple process without using, for example, GPS. The position information may be acquired using GPS, simple position information, or the like.

  The album browsing unit 32 performs processing for the user to browse album information such as a photographed image in the album database 40. Specifically, processing for creating and displaying various album browsing screens as shown in FIGS.

  For example, in C3 of FIG. 9A, a map screen (map) relating to the shooting location is displayed. On this map screen, the display position of the shooting location of the shot image registered in the album and its movement history (movement locus) are displayed. For example, it is displayed that the user shot “F2 Shrine” after shooting “F1 Tower” in Osaka, and then shot “F3 Castle”. In this case, the album browsing unit 32 specifies the display position and movement history of the shooting location on the map screen in FIG. 9A based on the position information acquired by the position information acquisition unit 30, for example.

  When the user selects to browse the album information, the album browsing unit 32 creates and displays a list screen of photographed images registered in the album database 40 as shown in FIG. 9B. Further, when the user inputs a comment on the captured image displayed on the list screen, the album browsing unit 32 registers the input comment in association with the captured image in the album database 40. For example, when the user selects a desired captured image on the list screen of FIG. 9B, detailed image information as shown in FIG. 10 is displayed. In the detailed image information, a comment input by the user with respect to the captured image is displayed as shown in C11. Note that the input comment may be displayed in association with each captured image on the list screen of FIG.

  Also, the album registration unit 26 performs a process of registering the uploaded image in the album database 40 when the user selects to upload an image when viewing the album in FIGS. 9A and 9B.

  In this embodiment, an example in which various screens such as a search result screen and an album browsing screen are created on the server side has been described. However, these various screens may be created by an application program on the terminal device side. Good. For example, when the data creation unit 28 on the server side creates and transmits data for displaying various screens such as the search result screen (search result data and the like), the application program on the terminal device side is based on the created data. Various screens may be created and displayed on the display unit 180. Further, the blocks as software may be integrated on hardware, and for example, image recognition processing may be executed as application processing on the terminal device side.

  The content presentation unit 34 performs various types of content information presentation processing (display processing). For example, content information (recommendation information, advertisement information, etc. at the shooting location) corresponding to the position information acquired by the position information acquisition unit 30 is read from the content database 50. Then, a process for presenting the read content information to the user is performed. Specifically, processing for displaying content information as shown in FIGS. 7A to 8 on the display unit 180 of the terminal device 110 is performed.

  For example, the content presentation unit 34 presents a recommended route (recommended route) to the user, and when the user selects the recommended route, the content presentation unit 34 presents the recommended information (peripheral store and peripheral spot information) on the recommended route to the user. Perform the process. For example, when the user selects “E1 temple” as a confirmed name tag as shown in B4 of FIG. 6B, the recommended route around “E1 temple” is presented to the user as shown in D1 of FIG. 7A. To do. In this case, it is desirable to create a recommended route based on position information acquired for other users (including movement history information obtained by arranging position information on a map in time series). For example, in D1 of FIG. 7A, another user recommendation route or the like is created based on the position information (log of position information, movement information, movement history) acquired for other users.

  Further, based on the position information acquired by the position information acquisition unit 30, it is assumed that the user's location (stay place) is specified or the user's travel route is predicted. In this case, the transmission processing unit 24 notifies the information processing apparatus 200 prepared for a store around the user's location or a store around the predicted travel route that the user is located. A process of transmitting (notification information notifying the user of approach) is performed. Then, when the information processing apparatus 200 that has received the notification information requests distribution of advertisement information, the content presentation unit 34 performs processing for presenting advertisement information to the user. For example, the advertisement information registered in advance in the advertisement information database 56 is displayed on the display unit 180 of the terminal device 110 or presented to the user using e-mail or the like.

  The aggregation processing unit 36 performs aggregation processing on the acquired position information. For example, a totaling process of position information acquired for a plurality of users is performed. And the transmission process part 24 transmits the total information obtained by the total process with respect to the information processing apparatus 200 for surrounding stores of an imaging | photography place. Specifically, the total information of the number of users at the shooting location is transmitted to the information processing apparatus 200. For example, as shown in B4 of FIG. 6B, the number of users who have selected “E1 temple” is totaled, and the total information of the number of persons is transmitted to the information processing apparatus 200 of the peripheral store of “E1 temple”. As a result, the neighboring stores can grasp the number of users located around the “E1 temple”, and therefore can determine whether or not to transmit advertisement information to these users. For example, when the number of users is greater than a predetermined number, it is possible to make a determination such as delivering an advertisement information mail.

  The processing unit 70 of the image search server 60 includes a reception processing unit 72, a transmission processing unit 74, an image search unit 76, a tag processing unit 78, and an image management unit 80.

  The reception processing unit 72 performs processing for receiving, for example, captured image data from the management server 10. Note that captured image data may be received from the terminal device 110.

  The image search unit 76 performs image search processing. Specifically, a plurality of registered images (one or a plurality of registered images) that are similar (match) to the captured image are searched from the image database 90.

  The tag processing unit 78 acquires a plurality of candidate name tags (candidate tags) from a plurality of name tags (tags) associated with the searched plurality of registered images. Specifically, the number of appearances (appearance frequency) of a plurality of name tags is tabulated, and a plurality of name tag sorting processes are performed based on the tabulation results to obtain a plurality of candidate name tags. For example, a plurality of name tags are rearranged in the descending order of appearance, and a name identification process for collecting duplicate name tags into one name tag is performed to obtain candidate name tags. Then, the transmission processing unit 74 performs processing for transmitting the acquired candidate name tag (tag information) to the management server 10.

  Note that the name tag counting process is not limited to the process of simply counting the number of appearance of name tags. For example, various modifications such as obtaining a weighting coefficient based on the degree of similarity of a registered image extracted as an image similar to a photographed image and performing a tabulation process based on the weighting coefficient and the number of appearances are possible. is there. For example, even if the number of appearances is small, when the similarity of the registered image to the photographed image is high, the total value of the name tags of the registered image may be increased.

  The image management unit 80 performs management processing of the image database 90. For example, an image registration process in the image database 90 or a deletion process (cleaning process) of a registered image in the image database 90 is performed. For example, assume that the user selects one of a plurality of candidate name tags on the search result screen of the image search shown in FIG. In this case, the image management unit 80 registers the selected candidate name tag in the image database 90 in association with the captured image as a confirmed name tag of the captured image. That is, the user's captured image and the confirmed name tag selected by the user are registered in the image database 90.

  Further, the image management unit 80 extracts (reads out) a registered photographed image that is a photographed image registered in the image database 90 from the image database 90. If the extracted registered photographed image does not satisfy the predetermined condition, a process of deleting the registered photographed image from the image database 90 is performed.

  The processing unit 120 of the terminal device 110 includes an image acquisition unit 122, a transmission processing unit 124, a reception processing unit 126, and a display control unit 128. Various modifications may be made such as omitting some of these components or adding other components.

  The image acquisition unit 122 acquires a captured image to be searched. For example, a captured image (subject image) obtained by imaging the subject with the imaging unit 160 is captured. For example, a captured image stored in a memory card or the like may be read and acquired.

  The transmission processing unit 124 performs processing for transmitting transmission data regarding the acquired captured image to the management server 10 (server system in a broad sense) via the network 100. The reception processing unit 126 performs a process of receiving search result data of a similar image search for captured images via the network 100.

  The display control unit 128 acquires position information of the shooting location of the shot image based on the image recognition result (candidate name tag or the like) of the shot image, and extracts content information corresponding to the acquired position information from the content database 50. In such a case, control is performed to display content information such as recommendation information and advertisement information on the display unit 180.

  Specifically, the display control unit 128 displays a search result screen that displays the acquired candidate name tag when the registered image similar to the captured image is searched and the candidate name tag is acquired based on the search result. Control to display in 180 is performed. Then, when the user selects any of a plurality of candidate name tags on this search result screen, the transmission processing unit 124 uses the selected candidate name tag as a confirmed name tag for acquiring position information. 100 to transmit. Then, the position information is acquired based on the fixed name tag, and when the content information is read from the content database 50 based on the acquired position information, the display control unit 128 controls the display unit 180 to display the content information. I do. Further, the display control unit 128 performs control to display an album browsing screen based on the album information in the album database 40 as shown in FIGS. Specifically, as shown in FIG. 9A, control is performed to display a map screen on which shooting locations and movement histories of shot images registered in the album database 40 are displayed.

2. Operation Next, the detailed operation of this embodiment will be described with reference to the processing flow of FIG. Hereinafter, “database” is described as “DB” as appropriate.

  When the user captures a subject with the camera (imaging unit) of the terminal device 110, the user sends an email with the captured image attached thereto to a specific email address (step S301). When receiving the mail from the user, the management server 10 registers (accumulates) the photographed image attached to the mail in the album DB and transmits the photographed image to the image search server 60 (steps S302, S303, S304). .

  When receiving the captured image, the image search server 60 performs image recognition processing with reference to the image DB (steps S305 and S306). Specifically, the photographed image (search target image) and each registered image in the image DB are collated, and among the registered images that are similar to the photographed image (including cases where they match), a predetermined upper number of registered images are obtained. Extract. And a candidate name tag is acquired based on the result of an image recognition process, and the acquired candidate name tag is transmitted to the management server 10 (steps S307 and S308). Specifically, a name tag associated with a predetermined number of registered images similar to the photographed image is collected (name identification process), and a candidate name tag is acquired and transmitted to the management server 10.

  When receiving the candidate name tag, the management server 10 extracts the tourism information corresponding to the candidate name tag from the tourism information DB, and transmits a result notification mail to the user terminal device 110 (steps S309, S310, and S311).

  FIG. 6A shows an example of the result notification mail. As shown in B1 and B2, the result notification mail is a mail including a candidate name tag of a photographed image, a URL (Uniform Resource Locator) of a My Page for displaying a search result screen, and the like. When the user clicks on the My Page URL of B2, a search result screen as shown in FIG. 6B is displayed (Steps S313 and S314).

  On the search result screen of FIG. 6B, as shown in B3, a candidate name tag of the photographed image and an icon image (registered image similar to the photographed image) corresponding to each candidate name tag are displayed. Specifically, “E1 temple”, “E2 shrine”, “E3 temple” candidate name tags and icon images representing “E1 temple”, “E2 shrine”, “E3 temple” are displayed. The user refers to these candidate name tags and icon images, and selects a candidate name tag (icon image) that he / she considered to be correct (step S315). For example, in FIG. 6B, the candidate name tag “E1 temple” shown in B4 is selected.

  Then, the management server 10 refers to the content DB such as the tourist information DB, extracts various content information, and displays a content information screen (tourist information screen) as shown in FIG. 7A (step S316). , S317). As a result, the user can browse various pieces of content information such as sightseeing spot information, recommendation information, and advertisement information as shown in FIG. 7A (step S318).

  Specifically, when the user selects the name tag of “E1 temple” shown in B4 from the candidate name tags shown in B3 of FIG. 6B, the name tag of the selected “E1 temple” is the fixed name. Determined to be a tag. Then, the position information of the fixed name tag “E1 temple” is acquired from the tourism information DB, and recommended information such as a recommended route around “E1 temple” is displayed as indicated by D1 in FIG. .

  When the user clicks “Search for nearby stores” in D3 of FIG. 7A, a nearby store information screen as shown in FIG. 7B is displayed. Further, by clicking on a specific peripheral store as shown at D4 in FIG. 7B, detailed information on the peripheral store is displayed as shown in FIG.

  Note that the confirmed name tag selected by the user is registered in the image DB together with the captured image (step S319). That is, the user's captured image is registered as a registered captured image in the image DB, and the confirmed name tag selected by the user is registered as a correct tag of the registered captured image. Further, the confirmed name tag selected by the user is also registered in the album DB (step S320). That is, a fixed name tag of “E1 temple” is registered in association with the photographed image of the user. As a result, the photographed image for which the user has performed an image search and the confirmed name tag selected by the user are sequentially registered in the album DB.

  As shown at B5 in FIG. 6B, the user is notified that the sent photographed image is registered in the user's album. If the user is interested in what album has been created and clicks on B6 album browsing, an album browsing screen as shown in FIG. 9A is displayed (steps S321 and S322).

  In FIG. 9A, as shown in C1, albums of places where the user has traveled in the past such as “Osaka album” and “Shiga album” can be browsed. “Osaka Album” is displayed. Note that albums may be grouped by date (eg, shooting date).

  As shown at C2 in FIG. 9A, the user takes pictures of “F1 tower”, “F2 shrine”, and “F3 castle” on a trip in Osaka, and searches for images of the captured images. Then, the user selects the fixed name tags of these sightseeing spots on the search result screen of FIG. 6B, and these photographed images and fixed name tags are registered in the album DB. Then, the management server 10 identifies the location of the sightseeing spot from the fixed name tags of these “F1 tower”, “F2 shrine”, and “F3 castle” and displays them on the map screen shown in C3. For example, the star marks C4, C5, and C6 indicate the positions of “F1 tower”, “F2 shrine”, and “F3 castle”, respectively, and the arrow marks C7 and C8 indicate the movement history (movement) of the user (shooting place). Route). By viewing these marks, the user can remember what route he / she went sightseeing in Osaka.

  When the user clicks “view album” shown in C9, a list screen of photographed images in the Osaka album is displayed as shown in FIG. 9B. That is, a list screen of photographed images is created and displayed based on the album information in the album DB. By viewing such a list screen, the user can browse images taken in Osaka and enjoy travel memories.

  When the user clicks “view details” shown in C10 on the list screen of FIG. 9B, an image details screen as shown in FIG. 10 is displayed. On this image detail screen, as shown at C11, the user can input a comment for the photograph. As a result, the user can write down the situation / background etc. of the photograph taken in the comments, and can look back on the user's trip.

  When the user selects a candidate name tag in step S315 in FIG. 5, the location information such as the sightseeing spot is acquired from the tourist information DB or the like based on the confirmed name tag (step S330). As a result, the location of the user is specified, and a notification e-mail notifying that the user is located (approaching) is transmitted to the information processing device (PC) 200 for the surrounding stores of the location (step S331). .

  When the information processing apparatus 200 receives the notification mail, the information processing apparatus 200 extracts a target user of the advertisement mail distribution, and selects a user who distributes the notification mail from the extracted target users (steps S332, S333, and S334). . For example, based on information from the user's personal information DB, the target users of advertisement mail distribution are narrowed down and selected according to the age group, sex, and the like. Then, the management server 10 is requested to distribute the advertisement mail corresponding to the registered advertisement content to the selected target user (step S335). It should be noted that registration of the advertisement content is preferably performed in advance before receiving the notification mail from the management server 10.

  Upon receiving the advertisement mail distribution request, the management server 10 reads advertisement information from the advertisement information DB and creates an advertisement mail (step S336). In the advertisement information DB, advertisement information having contents registered in advance by the information processing apparatus 200 is stored. Then, when the management server 10 transmits the advertisement mail created and the terminal device 110 receives the advertisement mail, the advertisement mail of the neighboring stores comes to be displayed on the display unit 180 of the terminal device 110 (steps S337 and S338). .

  For example, it is assumed that the user takes a photo at “E1 temple” which is a tourist spot, and selects “E1 temple” as a confirmed name tag on the search result screen of FIG. Then, the location information of “E1 temple” is extracted from the tourism information DB, the location of the user is specified based on this location information, and a notification mail is transmitted to the surrounding stores of that location. Then, when an advertisement mail distribution request is received from the information processing device 200 of the peripheral store, for example, as shown in D2 of FIG. 7A, the advertisement information of the peripheral store is displayed. In this way, the confirmed name tag obtained by the image search can be effectively used to identify the location of the user and to present the advertisement information of the surrounding stores at the location to the user in real time.

  According to the method of the present embodiment described above, when a user takes a picture with a camera-equipped mobile phone or a digital camera and requests the server side to recognize the captured image, the image recognition result is shown in FIG. A search result screen (image recognition result screen) as shown in FIG. In the search result screen, when the user selects a name tag that is a correct answer from the candidate name tags, the selected name tag is determined to be a confirmed name tag. The location information of the shooting location is specified. Then, content information such as recommendation information and advertisement information corresponding to the specified position information is presented to the user. Therefore, since the location information of the shooting location is specified by the image recognition result of the shot image and the content information corresponding to the specified location information is extracted without using GPS, base station information, etc., the shooting location Appropriate content information corresponding to can be presented to the user.

  That is, the position information obtained by GPS or the like is latitude / longitude itself, and only such latitude / longitude uniquely identifies what object the user is interested in taking a picture at the shooting location. It is difficult to specify.

  On the other hand, as shown in B4 of FIG. 6B, when the user selects “E1 temple” as the fixed name tag of the photographed image, the user's photographing target is “E1 temple”, and the user It can be uniquely identified that the shooting location of “E1 temple” is. Therefore, if content information such as recommendation information and advertisement information corresponding to the position information of “E1 temple” is presented, appropriate content information corresponding to the shooting location of “E1 temple” can be presented to the user. In addition, it is also possible to use the positional information acquired by GPS or base station information together for specifying the positional information of the user.

  Moreover, in this embodiment, when a user's location is specified based on the positional information acquired by the image recognition result, the user is located with respect to the information processing apparatus of the surrounding store of the specified location. Notification information to notify is notified. When there is a request for distribution of advertisement information from this information processing apparatus, as shown at D2 in FIG. 7A, advertisement information of the surrounding stores is displayed on the terminal device 110 of the user. Accordingly, when the user takes a picture while moving and requests the server to search for an image of the taken image, the advertisement information displayed in D2 of FIG. 7A also changes in real time according to the user's shooting location. It becomes like this. Accordingly, it is possible to display appropriate advertisement information according to the user's shooting location on the user's terminal device, thereby realizing an unprecedented advertisement distribution system.

  Moreover, if the acquired positional information and its total information are transmitted to the information processing apparatus of a neighboring store, the neighboring store can be informed of appropriate judgment materials as to whether or not the advertisement information is distributed. For example, the information processing apparatus in a nearby store distributes advertisement information when it is determined that the number of users located at the location is greater than or equal to a predetermined number based on the total information of the number of users from the server system. Is possible. Further, if user information such as age group and gender is transmitted to the information processing apparatus in addition to the position information and the total information, more appropriate judgment materials regarding the distribution of the advertisement information can be provided to the neighboring stores.

  In the present embodiment, during travel or sightseeing (on-travel), the user takes a picture with a mobile phone with a camera or a digital camera, and requests the server to search for an image of the taken image. A search result screen as shown in FIG. 7B is displayed, and guide information such as sightseeing information as shown in FIGS. Therefore, by taking a picture with a mobile phone or the like and sending the picture to a specific email address, such guide information is displayed, so that the user does not have to carry a guidebook while traveling. become. This makes it possible to provide guide information that is more portable and convenient than a guide book.

  The photographed image requested by the user for image retrieval is automatically registered and stored in the album DB without the user being aware of it. When the user makes a selection to confirm the name tag of the photographed image on the search result screen of FIG. 6B, the confirmed name tag is registered in association with the photographed image, and the name of the photographed image is automatically set. Will be granted. Therefore, since the images taken by the user can be stored in individual units and provided to the user as a web album, the user can take a picture and save it on the server without worrying about the storage capacity. become.

  Also, when the user takes a picture for the purpose of image search, the user is informed that the photographed image is registered in the album, as indicated by B6 in FIG. 6B. ~ The photographed image can be viewed on the album browsing screen of FIG. Therefore, it is possible to provide not only guide information for on-travel but also information for looking back on memories of travel for after-travel. Further, as shown at C3 in FIG. 9A, by utilizing the location information of the shooting location of the shot image and leaving the travel history during the trip as a trajectory on the map screen, the user visually recognizes the afterglow of the trip. To be able to enjoy. Therefore, it is possible to prevent a situation in which many photographed images are stored without being browsed, and it is possible to effectively utilize the photographed images through browsing the album.

  As shown in C11 of FIG. 10, the user can input comments and evaluations for the album information of the web album. Accordingly, it is also possible to provide a web album map screen, a photographed image, a blog linked with comments, an SNS (Social Networking Service), album release, and album photo slideshow.

3. Database Structure Next, the data structure of the album DB and the image DB used in this embodiment will be described. FIG. 11A shows an example of the data structure of the album DB. In FIG. 11A, my page ID corresponds to a user ID. For example, MPIDA is user A's my page ID, and MPIDB is user B's my page ID. For example, when a new user sends a mail with a photographed image attached to a specific mail address and performs user registration, for example, a My Page ID is assigned to the registered user's mail address in a one-to-one correspondence. The Then, the captured image attached to the user's mail is written in the storage location of the album DB corresponding to the My Page ID. For example, when user A performs user registration and my page ID = MPIDA is assigned, the captured image attached to user A's mail is registered in the storage location of my page ID = MPIDA in FIG. Is done. When the user selects a fixed name tag from candidate name tags as indicated by B4 in FIG. 6B, the fixed name tag is registered in association with the photographed image as shown in FIG. 11A. As described above, in the present embodiment, the captured image and the fixed name tag are registered in the storage location of the album DB specified by the mail address of the received user's mail.

  For example, when position information is acquired by a fixed name tag, it is desirable to register the position information in association with a captured image as shown in FIG. In this way, as shown in C4 to C8 in FIG. 9A, a process of specifying the shooting location and movement history of the shot image using the registered position information and displaying it on the map screen is performed. It can be simplified. In this case, for example, the user may be able to edit the position information registered in the album DB. In this way, the user can finely adjust the position of the shooting location on the map screen of C3 in FIG. 9A, and the convenience for the user can be improved. The position information may not be stored in the album DB. For example, the location information may be acquired with reference to a tourist information DB or the like as necessary.

  FIG. 11B shows an example of the data structure of the image DB. In this image DB, each name tag is registered in association with each registered image data. A plurality of name tags may be associated with one registered image data.

  In the present embodiment, for example, when an image search of the user's captured image is performed and the user selects a fixed name tag as indicated by B4 in FIG. 6B, the captured image and the fixed name for which the image search has been performed. Tags are also registered in the image DB. That is, the user's photographed image and its fixed name tag are registered in the image DB as a registered image for image search and its name tag. In this way, when various users search for images, the captured images are sequentially added as registered images to the image DB. That is, the number of registered images that serve as reference images for the image DB increases due to use by the user, and the image DB grows. Therefore, the image DB can cope with various photographed images, and user convenience and the like can be improved.

4). Acquisition of Candidate Name Tags Next, an example of a candidate name tag acquisition method will be described. For example, in FIG. 12A, IMT1, IMT2, IMT3, IMT4, IMT5, IMT6... Are extracted from the image DB as registered images similar to the user's captured image. In FIG. 12A, the registered images are listed in descending order of similarity, the registered image IMT1 has the highest similarity, and the registered image IMT2 has the next highest similarity.

  For registered images IMT1, IMT2, IMT3, IMT4, IMT5, and IMT6, name tags “E1 temple”, “E1 temple”, “E2 shrine”, “E1 temple”, “E2 shrine”, "E3 temple" is related. In this case, in the present embodiment, the number of appearances (appearance frequency) of these name tags is totaled, and candidate name tags are acquired based on the totalization results. For example, in FIG. 12A, the numbers of appearances of “E1 temple”, “E2 shrine”, and “E3 temple” are 3, 2, and 1, respectively. For this reason, by rearranging the name tags in FIG. 12A in descending order of appearance numbers and leaving only the predetermined number of name tags having the highest number of appearances, candidate name tags as shown in FIG. To be acquired. In this case, a name identification process is also performed in which duplicate name tags are deleted to be one.

  When counting the number of appearances in the counting process, a weighting count corresponding to the similarity of the image may be multiplied. Moreover, without counting the number of appearances, name identification processing may be performed on the name tags arranged in descending order of similarity, and a predetermined number of higher-level name tags after the name identification processing may be acquired as candidate name tags. .

5). Image Recognition Next, a detailed example of image recognition processing will be described. Image recognition processing for image search includes, for example, image recognition using feature points and image recognition using features of the entire image.

5.1 Image recognition using feature points (match search)
In image recognition using feature points, first, a blur process is performed on an original image step by step to create a blur image. Then, the difference between the blur images is taken to create a difference image, and the pixel having the extreme value in the difference image (the coordinate whose value is larger or smaller than the surroundings) is used as the feature point. Usually, a plurality of feature points are extracted from one image.

  The feature quantity of each feature point is expressed by a 128-dimensional vector, which is called a feature vector. This feature vector is obtained by adding the differential value of the blur image in a certain range (hereinafter referred to as a pickup range) centered on the feature point obtained as described above to the corresponding dimension.

  The differential value to be added is weighted so that the value becomes larger as it is closer to the feature point. The index of the dimension to be added is obtained from the coordinates obtained by rounding the coordinates of each pixel in the pickup range (hereinafter referred to as pickup coordinates) to 4 × 4 and the orientation obtained by rounding the differential orientation of the pickup coordinates in eight directions. And this feature-value is calculated | required about all the some feature points.

  Next, the above processing is performed on the registered image and the question image (search target image) to extract feature points and feature amounts. Then, the feature amount closest to the feature amount of the feature point extracted from the question image is searched from the registered image. Here, the closeness of the feature amount is expressed by the distance of the feature vector.

  In this way, the feature points of the registered image corresponding to the feature points extracted from the search target image are obtained, and the registered image having the most corresponding feature points is returned as a correct image (similar image). When a plurality of images are returned as correct images, the top J images (J is an integer of 2 or more) are returned as correct images (similar images) in order from the image having many corresponding feature points.

5.2 Image recognition using whole image features (similarity search)
In image recognition using the characteristics of the entire image, the entire image is first converted to a gray scale and converted to a luminance image. The luminance image is reduced to 16 × 16 pixels, and for example, a 256-dimensional vector is used as a composition feature.

  The hue of the entire pixel is divided into 10 sections, for example, and the histogram is used as a color feature of the entire image and expressed as a 10-dimensional vector. Further, the entire image is divided into 512 colors, and the top 30 colors of the occupied area are used as representative color features.

  The above-described processing is performed on the registered image and the question image, and the feature amount is calculated. Then, the registered image having the feature closest to the question image is returned as a correct image (similar image). When a plurality of images are returned as correct images, the top J images are returned in order from the image closest to the question image. The proximity of the feature is expressed by the distance of each feature vector. Also, the weighting is changed depending on whether the composition feature or the color feature is important.

6). Detailed Processing Next, a detailed processing example of the present embodiment will be described with reference to the processing flows of FIGS.

  FIG. 13 and FIG. 14 are flowcharts of processing for acquiring candidate name tags by image retrieval of photographed images and registering the confirmed candidate name tags obtained by user selection in the album DB or image DB in association with the photographed images. .

  When the user photographs a subject with the terminal device 110 and sends an email with the photographed image attached thereto, the management server 10 receives the email and checks the email address (steps S1 to S4). If the user specified by the e-mail address is a new user, a My Page ID serving as a user ID is issued and the user information is registered in the personal information DB (steps S5 and S6). On the other hand, if the user is an existing user who already uses the service, the My Page ID is assigned to the captured image, and the captured image is transmitted to the image search server 60 and registered in the album DB (steps S7 to S7). S9).

  When receiving the captured image, the image search server 60 extracts feature points of the captured image and performs a matching process with the image DB (steps S10 to S12). That is, image recognition (match search) using the above-described feature points is executed. Then, scoring is calculated to determine whether or not the image recognition is successful (whether or not a recognition result is obtained) (steps S13 and S14).

  If the image recognition is not successful, the composition of the image is extracted and collation processing with the image DB is performed (steps S15 and S16). In other words, image recognition (similarity search) using the above-described overall image characteristics is executed. Then, scoring is calculated to determine whether the image recognition is successful (steps S17 and S18).

  If it is determined in step S14 or S18 that the image recognition has succeeded, the name tags of the registered images that have been searched are listed in order of similarity, and the name tag counting process (name identification process) is performed. Is transmitted to the management server 10 (steps S19 to S22). On the other hand, if it is determined in step S18 that the image recognition is not successful, the image search server 60 transmits an error result (step S51). When receiving the error result, the management server 10 creates an error mail and transmits it to the terminal device 110 (steps S52 to S54). When the user of the terminal device 110 confirms receipt of the error mail and clicks on the URL in the mail, a top screen is displayed (steps S55 to S57). When the user selects image recognition search on the top screen, the mailer is activated and the process returns to step S1, and when text search is selected, text search is performed (steps S58 to S61).

  Upon receiving the candidate name tag from the image search server 60, the management server 10 searches the tourism information DB based on the candidate name tag, and creates a search result screen as shown in FIG. 6B (steps S23 to S25). ). Then, a result notification mail in FIG. 6A that informs the URL of the created search result screen is transmitted to the terminal device 110 (step S26).

  When the user of the terminal device 110 confirms receipt of the result notification mail and selects to browse the search result screen by URL, the search result screen of FIG. 6B is displayed (steps S27 to S29). When the user selects a candidate name tag considered to be correct as shown in B4 of FIG. 6B, a tourist information screen (guide information screen) as shown in FIG. 7A is displayed. The information screen is browsed (steps S30 to S33). Then, the management server 10 determines that the selected candidate name tag is a confirmed name tag, and transmits the captured image and the confirmed name tag to the image search server 60 (steps S34 and S35). Further, by referring to the tourism information DB based on the confirmed name tag, the position information of the shooting location of the shot image is acquired (step S36). Taking B4 in FIG. 6B as an example, the location information of “E1 temple” is acquired from the tourism information DB. Then, the confirmed name tag and position information are registered in the album DB in association with the photographed image (step S37).

  On the other hand, the image search server 60 that has received the photographed image and the confirmed name tag registers the received photographed image and the confirmed name tag in the image DB (steps S38 and S39). As a result, the captured image is registered in the image DB as a registered image that serves as a reference for image search.

  As shown in B7 of FIG. 6B, when the user selects image search on the search result screen, the mailer is activated and returns to step S1 of FIG. 13 (steps S40 and S41). When the text search is selected as shown in B8, a text search referring to the tourism information DB is executed, and when desired information is searched from the tourism information DB, a text search search result screen is displayed ( Steps S61 to S63, S29). On the other hand, if the information cannot be searched, the search by the external site is executed, the search result from the external site is received, and the search result screen is displayed (steps S64 to S66, S29).

  FIG. 15 is a flow of processing for presenting content information such as recommendation information and advertisement information to the user based on the acquired position information.

  As the tourist information is selected by the user, a tourist information screen as shown in FIG. 7A is displayed, and the user browses the sightseeing spot information (steps S221, S222, and S223). Then, when the user clicks D1 in FIG. 7A and selects a recommended route, the management server 10 refers to the tourist information DB and acquires the position information of the shooting location (user location) (step S224, S225). Then, referring to the recommendation information DB, a peripheral facility search by information (route) is performed, and the peripheral facility information is extracted from the tourism information DB (steps S226 and S227). Further, referring to the log information DB, a log of the number of browsing times of other users is acquired, and a recommendation route screen is created (steps S228 and S229). That is, a recommended route is created based on position information acquired for other users (including movement history information obtained by arranging position information on a map in time series). As a result, the user can browse the recommended routes of other users (step S230).

  On the other hand, when the user clicks D3 in FIG. 7A and selects a nearby store search, the management server 10 refers to the tourist information DB and acquires position information of the shooting location (user location) ( Steps S231 and S232). Then, with reference to the store information DB, search / extraction of neighboring stores is executed (step S233). Further, the advertisement information store is searched for / extracted with reference to the advertisement information DB (step S234). Then, a peripheral store information screen as shown in FIGS. 7B and 8 is created (step S235). As a result, the user can browse the peripheral store information (step S236).

  FIG. 16 is a flow of processing for identifying the location of the user based on the acquired location information, transmitting notification information to the information processing apparatus of the nearby stores of the location, and presenting the advertisement information of the nearby stores to the user. It is.

  First, referring to the log information, the log information relating to the user position information and the like is analyzed to identify the current user location (steps S241 and S242). Further, based on the user log information, the user's travel route is predicted (step S243). For example, when the user moves from the place P1 to P2 and moves from the place P2 to P3, the next moving place P4 is predicted from the position information of the places P1, P2, and P3 and the movement locus thereof. And the notification mail which notifies the location and movement route of these users is transmitted to surrounding store PC (information processing apparatus) (step S244).

  When the neighboring store PC that has logged in to the management screen receives the notification mail, it extracts the person to whom the advertisement mail is to be distributed and determines the number of advertisement mail distribution (steps S245 to S248). For example, when user information such as the user's age group and gender is known from the management server 10 or when the total information of the number of users is known, based on these user information and total information, Extract recipients of advertising emails and determine the number of advertising emails to be delivered.

  Next, a template is extracted from the advertisement information DB, and if there is a template registered in the past, a desired template is selected from those templates (steps S249 to S251). On the other hand, when there is no template, the advertising store side newly inputs an advertisement content and registers it as a template in the advertisement information DB (steps S252 and S253). Then, the advertisement mail distribution request is registered in the advertisement information DB (step S254).

  The management server 10 sets the acquired location information, refers to the advertisement information DB, and confirms whether there is an advertisement placed by a nearby store (steps S261 and S262). If there is an advertisement, the store advertisement information registered by the neighboring stores is extracted from the advertisement information DB (steps S263 and S264). Then, a mail is created and distributed to the user terminal device 110 (step S265). When the user who receives the mail accesses the store information, the management server 10 creates and displays a store information screen (steps S266 to S268). As a result, the user can browse the neighboring store information screen and see advertisement information and the like of the neighboring stores (step S269). In addition, the mail itself delivered to the user in step S265 may be advertisement information, and the delivery of the advertisement information may be completed when the user views the mail. In addition, by identifying that the viewer is the recipient of the advertisement mail on the peripheral store information screen, and creating an advertisement information screen according to the user, the user has different preferential treatment (discount, service addition, etc.) for each user at the store It is also possible to receive.

  By doing as described above, the location of the user is specified by the acquired position information, and the advertisement information from the surrounding stores can be distributed in real time according to the location of the user.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings. The configurations and operations of the server system and the terminal device are not limited to those described in the present embodiment, and various modifications can be made.

10 management server, 20 processing unit, 22 reception processing unit, 24 transmission processing unit,
26 album registration unit, 28 data creation unit, 30 location information acquisition unit,
32 album browsing section, 34 content presentation section, 36 tabulation processing section,
40 album database, 42 personal information database,
50 content database, 52 tourism information database,
54 recommendation information database, 56 advertisement information database, 58 communication section,
60 image search server, 70 processing unit, 72 reception processing unit, 74 transmission processing unit,
76 image search unit, 78 tag processing unit, 80 image management unit,
90 image database, 98 communication unit, 100 network, 110 terminal device,
120 processing units, 122 image acquisition units, 124 transmission processing units, 126 reception processing units,
128 display control unit, 140 storage unit, 150 information storage medium, 160 imaging unit,
170 operation unit, 180 display unit, 200 information processing apparatus

Claims (24)

A reception processing unit that performs processing for receiving transmission data about a captured image of a user who is a target of image search via a network;
A position information acquisition unit that acquires position information of a shooting location of the shot image based on an image recognition result of the shot image;
A server system comprising: a content presentation unit that extracts content information corresponding to the acquired position information from a content database and performs a process for presenting the content information to a user. In claim 1,
The content presentation unit
A server system, wherein, based on the acquired position information, recommendation information at the shooting location is selected from the content database, and processing for presenting the selected recommendation information to a user is performed. In claim 2,
The content presentation unit
A server system, wherein a recommendation route is presented to a user, and when the user selects the recommendation route, a process for presenting recommendation information in the recommendation route to the user is performed. In claim 3,
The content presentation unit
A server system that creates the recommended route based on the location information acquired for another user. In any of claims 1 to 4,
It includes a transmission processing unit that performs a process of transmitting notification information notifying that the user is present to an information processing device for a nearby store at the location of the user specified based on the position information. Server system. In claim 5,
The transmission processing unit
The server system characterized by performing the process which transmits the said notification information with respect to the information processing apparatus for stores around the user's movement path | route estimated based on the said positional information. In claim 5 or 6,
The content presentation unit
When the information processing apparatus that has received the notification information requests distribution of advertisement information, the server system performs processing for presenting the advertisement information to a user. In any of claims 1 to 4,
A tally processing unit for tallying the position information acquired for a plurality of users;
The server system characterized by including the transmission process part which performs the process which transmits the total information obtained by the said total process with respect to the information processing apparatus for the surrounding stores of the said imaging | photography place. In claim 8,
The transmission processing unit
The server system characterized by performing the process which transmits the total information of the number of users in the said imaging | photography place with respect to the said information processing apparatus. In any one of Claims 1 thru | or 9,
Acquired when a plurality of registered images similar to the photographed image are searched from an image database, and a plurality of candidate name tags are acquired from a plurality of name tags associated with the searched plurality of registered images. Including a data creation unit for creating search result data including the plurality of candidate name tags,
The position information acquisition unit
When the user selects any of the plurality of candidate name tags on the search result screen displayed by the search result data, the selected candidate name tag is determined to be a confirmed name tag of the captured image. The server system, wherein the position information of the shooting location is acquired based on the confirmed name tag. In claim 10,
An image search unit that searches the image database for the plurality of registered images similar to the captured image;
A server system comprising: a tag processing unit that acquires the plurality of candidate name tags from the plurality of name tags associated with the searched plurality of registered images. In claim 11,
The tag processing unit
A server system, wherein the number of appearances of the plurality of name tags is totaled, and the plurality of candidate name tags are acquired based on a totaling result. In any of claims 10 to 12,
Including an album registration unit for performing processing for registering the photographed image in the album database;
The album registration unit
When the user selects any of the plurality of candidate name tags on the search result screen, the confirmed name tag of the photographed image is registered in association with the photographed image of the album database. Server system. In claim 13,
The reception processing unit
A process of receiving an email from a user attached with the transmission data about the photographed image,
The album registration unit
A server system, wherein the photographed image is registered in a storage location of the album database specified by a mail address of the received mail. In claim 13 or 14,
The album registration unit
The server system, wherein the acquired location information of the shooting location is registered in association with the shot image of the album database. In any of claims 13 to 15,
A server system comprising: an album browsing unit that performs processing for a user to browse album information in the album database. In claim 16,
The album browsing section
A server system for performing processing for displaying on a map screen at least one of a shooting location of the shot image registered in the album database and a movement history of the shooting location. In claim 17,
The album browsing section
A server system, wherein the display position of the shooting location on the map screen or the movement history is specified based on the acquired position information of the shooting location. A reception processing unit that performs processing for receiving transmission data about a captured image of a user who is a target of image search via a network;
An image search unit for searching a plurality of registered images similar to the photographed image from an image database;
A tag processing unit for obtaining a plurality of candidate name tags from a plurality of name tags associated with the plurality of registered images searched;
When the user selects any of the plurality of candidate name tags on the search result screen of the image search of the photographed image, the selected candidate name tag is determined as the confirmed name tag of the photographed image, and the confirmed name A position information acquisition unit that acquires position information of a shooting location of the shot image based on the tag;
A server system comprising: An image acquisition unit that acquires a captured image of a user who is a target of image search;
A transmission processing unit that performs a process of transmitting transmission data about the acquired captured image via a network;
A reception processing unit that performs processing for receiving search result data of an image search for the captured image via a network;
A display control unit that performs display control of the display unit,
The display control unit
Based on the image recognition result of the captured image, position information of the shooting location of the captured image is acquired, and when content information corresponding to the acquired position information is extracted from a content database, the content information is A terminal device that performs display control on a display unit. In claim 20,
The display control unit
Acquired when a plurality of registered images similar to the photographed image are searched from an image database, and a plurality of candidate name tags are acquired from a plurality of name tags associated with the searched plurality of registered images. A search result screen on which the plurality of candidate name tags are displayed is controlled to be displayed on the display unit,
The transmission processing unit
When the user selects any of the plurality of candidate name tags on the search result screen, the selected candidate name tag is transmitted as a confirmed name tag for acquiring the position information via the network. A terminal device that performs processing. An image acquisition unit that acquires a captured image of a user who is a target of image search;
A transmission processing unit that performs a process of transmitting transmission data about the acquired captured image via a network;
A reception processing unit that performs processing for receiving search result data of an image search for the captured image via a network;
As a display control unit that performs display control of the display unit,
Make the computer work,
The display control unit
Based on the image recognition result of the captured image, position information of the shooting location of the captured image is acquired, and when content information corresponding to the acquired position information is extracted from a content database, the content information is A program for performing control to be displayed on a display unit.   A computer-readable information storage medium, wherein the program according to claim 22 is stored. A plurality of registered images similar to the photographed image of the user who is the target of the image search are searched from the image database,
Obtaining a plurality of candidate name tags from a plurality of name tags associated with the searched plurality of registered images;
Display a search result screen on which the obtained plurality of candidate name tags are displayed,
In the search result screen, when the user selects any of the plurality of candidate name tags, the selected candidate name tag is determined as a confirmed name tag of the captured image, and based on the confirmed name tag, the An image search method characterized by acquiring a shooting location of a shot image.

Images