JP5687995B2 - Subject search device, subject search method, subject search information providing program, and subject search program - Google Patents

Description

  The present invention belongs to the field of information search technology, and relates to an apparatus, a method, and a program that can reliably search for accurate information about a photographed subject.

  Many people have a hobby of enjoying nature, such as walking in the mountains, strolling in satoyama, or strolling along the seaside or around the lake. Along with this, an increasing number of people are interested in collecting edible native plants (wildflowers, wild plants, persimmons, fruits, nuts, etc.) and edible seafood. However, some native plants and seafood are toxic and cannot be eaten without knowledge and experience.

  For this reason, in the case of users who have no knowledge or experience about collection targets such as native plants and seafood, many of them use pictorial books with photos to determine whether the collection of native plants and seafood is non-toxic and can be eaten. The user himself / herself determines whether or not. In relation to this, Patent Document 1 described later captures images of animals and plants, and searches a database based on image data obtained by taking the images and shooting conditions (shooting time and shooting location) at the time of shooting. An invention for obtaining an explanatory text relating to a subject is disclosed.

Japanese Patent Laid-Open No. 10-091634

  However, in the case of the above-described method using a pictorial book, it is necessary for the user to turn the page to check whether the contents described in the pictorial book are the same as the native plant or seafood to be collected. It is troublesome. In addition, since unique parts differ among native plants, seafood, and the like, there are many cases where accurate information about a subject cannot be searched by using only one piece of image data or shooting conditions. For this reason, even if it uses the invention of patent document 1 mentioned above, it is thought that appropriate information about collection objects, such as a native plant and seafood, may not be provided reliably.

  In view of the above, an object of the present invention is to ensure that accurate information regarding a photographed subject can be reliably searched without bothering the user.

In order to solve the above-described problem, a subject search device according to claim 1 is provided.
Photographing means for photographing an image of a subject to be searched and obtaining image data of the subject;
Acquisition means for acquiring related information about the image data including at least a shooting position and a shooting date and time for the image data acquired through the shooting means ;
Storage means for storing, for each of a plurality of objects that may be subjects, one or more image data forming one or more images of the object , and storing information relating to the object and map information ;
Search means for searching for information on the subject to be searched from stored information in the storage means based on the acquired image data and the related information;
First information requesting means for requesting first additional information when information on the subject to be searched cannot be searched;
First control means for controlling to acquire the first additional information through one or both of the photographing means and the acquiring means in response to a request for the first additional information;
First re-search means for re-searching information on the subject to be searched from the stored information in the storage means, taking into account the first additional information acquired in accordance with the control of the first control means. When,
Second information requesting means for requesting information on a specific part of the subject as second additional information when information on the subject that is a candidate for search can be retrieved;
Second control means for controlling to acquire the second additional information through one or both of the photographing means and the acquiring means in response to a request for the second additional information;
Second re-search means for re-searching information relating to the subject to be searched from the stored information in the storage means, taking into account the second additional information acquired in accordance with the control of the second control means. It is characterized by including these.

According to the subject search device of the first aspect of the present invention, the related information including the image data of the subject to be searched acquired by the shooting unit, and the shooting position and shooting date and time of the image data acquired by the acquiring unit. Based on the above, the search means searches for information on the subject to be searched from the stored information in the storage means. If the search means cannot find information related to the subject to be searched, the first addition request means requests the first additional information, and in response to this, the first control means causes the photographing means or The acquisition unit is controlled to acquire the first additional information. Then, the first re-search means re-searches information related to the subject to be searched from the stored information in the storage means, taking into account the newly acquired first additional information. In addition, when the information related to the subject that is a candidate for the search can be searched, the second addition request unit requests the second additional information, and in response to this, the second control unit performs the photographing unit or The acquisition unit is controlled to acquire the second additional information. Then, the second re-searching means re-searches information on the subject to be searched from the stored information in the storage means, taking into account the newly acquired second additional information.

  As described above, even if information about the subject to be searched cannot be searched by the first search, additional information (image data and related information) about the subject necessary for searching for information about the subject is obtained. The re-search is performed in consideration of this. As a result, it is possible to reliably retrieve accurate information regarding the subject to be retrieved. Moreover, it does not bother the user.

  According to the present invention, it is possible to reliably retrieve accurate information about a photographed subject without bothering the user.

It is a figure for demonstrating schematic structure of the subject search system which implement | achieves one Embodiment of the subject search apparatus of this invention. It is a block diagram for demonstrating the structural example of the mobile telephone terminal 4 of embodiment. It is a figure for demonstrating the outline | summary of an example of the format of an image data file. It is a block diagram for demonstrating the structural example of the server apparatus 1 of embodiment. It is a figure for demonstrating the example of the registration data of subject DB105. It is a flowchart for demonstrating the process in the mobile telephone terminal 4 when performing a to-be-photographed object search. It is a flowchart following FIG. It is a flowchart following FIG. 7 is a diagram for explaining an example of a screen displayed on the display screen of the mobile phone terminal 4 when performing a subject search. FIG. It is a flowchart for demonstrating the process in the server apparatus 1 in the case of performing a subject search. It is a flowchart following FIG.

  Hereinafter, embodiments of the apparatus, method, and program of the present invention will be described with reference to the drawings. The present invention can be realized by a single information processing apparatus having a camera function, or can be realized by combining a terminal apparatus having a camera function and an information processing apparatus. The present invention can also be realized as a system formed by connecting a terminal device having a camera function and a server device through a network.

  In the following, taking into account use during outdoor activities, a case where the present invention is realized as a system formed by connecting a terminal device having a camera function and a server device through a network will be described as an example. . There are various types of terminal devices having a camera function, such as information terminals called mobile phone terminals, tablet terminals, and pad terminals. However, in the embodiments described below, a terminal device having a camera function is assumed to use a mobile phone terminal with a camera that is currently widely used.

[System overall configuration]
FIG. 1 is a diagram for explaining a schematic configuration of a subject search system that realizes an embodiment of a subject search device of the present invention. That is, in this embodiment, the server device 1 and the mobile phone terminals 4 (1), 4 (2),... Is realized.

  As will be described in detail later, the server device 1 includes, for each of various plants and various seafood that may be subjects, one or more image data forming one or more images thereof, A subject database formed by accumulating detailed information is provided. The server device 1 can provide a subject search service using the subject database. Here, “something that can be a subject” means what is to be searched.

  In FIG. 1, the network 2 mainly means the Internet. The Internet is a network constructed by interconnecting networks based on the Internet protocol, which is constructed worldwide. The network 2 includes a public switched telephone network (PSTN) that provides a connection path to the Internet, a digital line network (ISDN (Integrated Services Digital Network)), an optical communication network, and the like. The network 2 also includes a mobile phone network, a wireless LAN (Local Area Network) called “wifi”, a wireless communication network conforming to a high-speed wireless communication standard called “wimax”, and the like. Accordingly, as shown in FIG. 1, each of the mobile phone terminals 4 (1), 4 (2),... Is connected to base stations 3 (1), 3 (2), 3 (3), It is connected to the network 2 through.

  Each of the mobile phone terminals 4 (1), 4 (2),... Has a camera function and a GPS (Global Positioning System) function in addition to a communication function for realizing a call function through a mobile phone network and a data communication function through the Internet. , A clock circuit, a direction sensor (geomagnetic sensor) and the like. Then, each of the mobile phone terminals 4 (1), 4 (2),... Can acquire image data that forms an image of the subject by performing shooting using the camera function.

  In addition, each of the mobile phone terminals 4 (1), 4 (2),..., Acquires a shooting position using a GPS function, acquires a shooting date and time using a clock circuit, and sets an orientation sensor. Various kinds of related information (metadata) relating to image data, such as using to acquire a shooting direction, is acquired. That is, in the mobile phone terminals 4 (1), 4 (2),..., Not only the subject image data is acquired by shooting, but also various related information (metadata) about the image data is acquired. It can be acquired and associated and stored.

  The GPS function of the mobile phone terminals 4 (1), 4 (2),... Determines the current position of the own device by receiving signals from at least three artificial satellites and performing arithmetic processing. Is something that can be done. Each of the GPS functions of the mobile phone terminals 4 (1), 4 (2),... Can measure the current position of the own device with high accuracy using a technique called so-called differential GPS. In FIG. 1, GPS satellites 5 (1), 5 (2),... Are arranged on the earth orbit to realize a GPS function on the ground, and transmit about 30 so-called GPS signals for current position measurement. A part of the base satellite is shown.

  Then, for example, it is assumed that the user of the mobile phone terminal 4 (1) goes out to pick a potato and finds the potato, but the potato has never been seen and cannot be eaten. In this case, the user accesses the server device 1 using the cellular phone terminal 4 (1), receives a subject search application or the like, and prepares an environment in which the subject can be searched with the server device 1. And according to a user's instruction | indication, the mobile telephone terminal 4 (1) image | photographs the image of the eyelid which the user found using a camera function, and acquires image data. At the same time, the cellular phone terminal 4 (1) acquires related information such as the shooting location, the shooting date and time, the shooting direction, etc., using the function of its own device. Then, the mobile phone terminal 4 (1) forms a search request including the image data and related information and transmits it to the server device 1.

  As described above, the server device 1 includes the subject database. When the server device 1 receives the search request from the mobile phone terminal 4 (1), the server device 1 uses the image data and related information included in the search request to obtain information on the subject of the image formed by the image data from the subject database. Search for. Then, the server device 1 returns the search result to the mobile phone terminal 4 (1) as the search request source. However, if only the image data and related information included in the search request, the corresponding data may not be found or the corresponding data may not be narrowed down.

  In view of this, the server device 1 adds information to the mobile phone terminal 4 (1) as the search request source when the information about the subject to be searched cannot be searched (when it cannot be specified) in the initial search processing. Request information. Additional information in this case requires photos (image data) of the top of the cocoon umbrella and the back of the cocoon umbrella, the diameter of the cocoon umbrella, the length of the stalk, the presence or absence of the viscosity of the stalk, and the smell of the cocoon Specific information such as requesting information on the kind of trees existing around the fence. That is, the server device 1 can narrow down the accumulated data that may correspond to the subject by the first search process. Therefore, the server device 1 specifies additional information necessary for the discrimination processing based on the information obtained by the initial search processing, and provides the additional information to the mobile phone terminal 4 (1) as the inquiry request source. Demand.

  In the mobile phone terminal 4 (1) for which provision of additional information is requested, additional information corresponding to the request from the server device 1 is acquired according to a user instruction, and is transmitted to the server device. In the server apparatus 1, the additional information from the mobile phone terminal 4 (1) is also taken into consideration, and the subject database is referred to and information on the subject to be searched is re-searched. As a result, it is possible to more reliably search for information related to the subject of the image formed by the image data (in this example, 茸) based on the image data and related information from the mobile phone terminal 4 (1). . Specifically, it is possible to know the name (name) of the candy, the presence or absence of toxicity, whether or not it is suitable for edible use, and what kind of cooking method is available for edible use. Here, the case of using the mobile phone terminal 4 (1) has been described as an example, but the same applies to the case of using another mobile phone terminal such as the mobile phone terminal 4 (2) having a camera function. In addition, the subject can be searched.

  As described above, in the first search process, the server device 1 according to the present embodiment provides the additional information necessary for the search when the information regarding the subject to be searched cannot be searched. Requests are made to the mobile phone terminals 4 (1), 4 (2),. And the server apparatus 1 can search the exact information regarding the subject of search object reliably by performing a re-search process using the additional information from the search request source.

  In this embodiment, each of the mobile communication terminals 4 (1), 4 (2),... Has the same basic configuration. Therefore, unless otherwise indicated below, the portable communication terminals 4 (1), 4 (2),... Are collectively referred to as the portable communication terminal 4. In the following, first, configuration examples of the mobile phone terminal 4 and the server device 1 constituting the subject search device (subject search system) of this embodiment will be described. Thereafter, the processing performed by the mobile phone terminal 4 and the server device 1 will be specifically described.

[Configuration example of mobile communication terminal 4]
FIG. 2 is a block diagram for explaining a configuration example of the mobile phone terminal 4 of this embodiment. As shown in FIG. 2, the cellular phone terminal 4 of this embodiment includes a transmission / reception antenna 401, a transmission / reception processing unit 402, a receiver (speaker) 403, and a transmitter (microphone) 404 as parts that perform communication through the network 2. Prepare. In addition, as a part that provides a user interface, a key operation unit 411, a display control unit 412, a display unit 413, an audio processing unit 414, and a speaker 415 are provided. In addition, the mobile phone terminal 4 includes a camera unit 416 that realizes a camera function.

  The key operation unit 411 is provided with numeric keys, various function keys, and the like, and receives an instruction input (operation input) from the user. The key operation unit 411 is also provided with a shutter key for using a camera function. The display control unit 412 and the display unit 413 are for displaying display information such as various images and display messages. As the display unit 413, for example, a so-called thin display element such as an organic electroluminescence display (LCD) or a liquid crystal display (LCD) is used. The voice processing unit 414 and the speaker 415 output ringtones, warning sounds, and other voice messages. The camera unit 416 includes an imaging lens, an imaging element such as a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor, a camera signal processing circuit, and the like.

  The mobile communication terminal 4 includes a control unit 420 that controls each unit. The control unit 420 is formed by connecting a CPU (Central Processing Unit) 421, a ROM (Read Only Memory) 422, a RAM (Random Access Memory) 423, and an EEPROM (Electrically Erasable and Programmable ROM) 424 through a CPU bus 425. A computer device. The CPU 421 reads out and executes a program stored in the ROM 422 or the EEPROM 424, forms a control signal to be supplied to each part, supplies this to each part, accepts data transmitted from each part, and receives it. Depending on the process. The ROM 422 is a program in which various data necessary for programs executed in the CPU 421 and processing are recorded in advance.

  In addition, the RAM 423 is mainly used as a work area, such as temporarily storing intermediate results of processing. The EEPROM 424 is a so-called nonvolatile memory, and stores and holds data and the like that should be retained even when the power of the mobile communication terminal 4 is turned off. The EEPROM 424 stores, for example, a new program for adding functions, various setting parameters, telephone directory data, application programs, additional programs for improving functions, and the like.

  Furthermore, the mobile communication terminal 4 includes a clock circuit 431 that provides the current date, current day, and current time, as well as a memory slot 432, an external input / output terminal 433, an external interface (hereinafter abbreviated as an external I / F). 434). The memory slot unit 432 includes a so-called memory card mounting unit and a data writing / reading unit for the memory card mounted on the memory card. The memory slot portion 432 can support a plurality of types of memory cards with different standards. In this embodiment, the external input / output terminal 433 and the external I / F 434 are digital interfaces compliant with, for example, the USB (Universal Serial Bus) standard.

  The mobile communication terminal 4 includes a GPS unit 435 and a GPS antenna 436. As described above, the GPS unit 435 and the GPS antenna 436 receive GPS signals (positioning information) from a plurality of GPS satellites and perform arithmetic processing, thereby accurately detecting the current position of the own device (positioning )can do. Although not shown in FIG. 2, the mobile phone terminal 4 includes various sensors, circuits, and the like for acquiring information included in related information (metadata) about image data such as an orientation sensor.

  The mobile communication terminal 4 having such a configuration transmits information to the network 2 through the transmission / reception processing unit 402 and the transmission / reception antenna 401 under the control of the control unit 420, or information addressed to the own device transmitted through the network 2. Or receive. Thereby, the mobile phone terminal 4 can perform various communication processes.

  The mobile communication terminal 4 also includes a menu for registering various executable application programs. The menu is displayed on the display screen of the display unit 413 through the display control unit 412 by performing a predetermined operation such as pressing a menu key provided in the key operation unit 411, for example. Can do. Then, the control unit 420 can receive a selection input of an item included in the menu from the user through the key operation unit 411 and execute a program corresponding to the item selected by the user.

  Then, the cellular phone terminal 4 of this embodiment can store the image data of the subject obtained by photographing through the camera unit 416 in the memory card 450 mounted in the memory slot unit 423. In this case, related information (metadata) about image data acquired at the time of shooting is also stored together with the image data. When using the subject search service provided by the server device 1 through the mobile phone terminal 4, a search request including image data obtained by photographing the subject to be searched is formed, and this is sent to the server device 1. Send. In this case, the search request includes related information (metadata) about the image data acquired at the time of shooting, and text information about the subject and the shooting location received through the key operation unit 411. That is, the text information input through the key operation unit 411 at the time of subject search is also related information (metadata) about the image data of the subject to be searched.

  FIG. 3 is a diagram for explaining a format example of a search request formed in the mobile phone terminal 4 of this embodiment. As shown in FIG. 3, the search request includes a terminal ID, header information, related information (metadata), one or more image data, and input information (text information). The terminal ID is information for identifying the mobile phone terminal 4 that is the source of the search request (search request source). The header information includes information indicating that the information is a search request, information indicating that the information is a re-search request described in detail later, information indicating what information is included in the search request, and the like. This information is necessary when processing according to the above is performed.

  The related information (metadata) is various information related to the image data stored in the search request. The related information related to the image data used in this embodiment includes shooting date / time, shooting direction, focal length, and GPS information. These pieces of information are information that can be acquired by the functions of the mobile phone terminal 4. In addition to this, information relating to a photographing device such as a photographing device manufacturer name and various photographing parameters such as resolution and shutter speed are also included in the search request as metadata.

  Further, as shown in FIG. 3, the search request can include a plurality of image data of the subject to be searched. Although this will be described in detail later, it is possible to cope with a case where image data of a plurality of different parts of the subject is requested as additional information from the server device 1 when the subject search service is used. Because. Of course, a plurality of image data is not necessarily required. In the case of the first search request, only one image data of the subject to be searched is included in the search request. Even if additional information is requested, if additional image data is not requested, the image data may not be included in the search request. In FIG. 3, the input information is text information regarding the subject, the shooting location, and the like input through the key operation unit 411 as described above.

  As described above, the cellular phone terminal 4 according to this embodiment captures a subject through the camera unit 416, forms a search request for the content described with reference to FIG. 3, and transmits it to the server device 1. It is possible to search for information on the subject to be searched. That is, the mobile phone terminal 4 realizes an access function to the server device 1, and by using the mobile phone terminal 4, a subject search service provided by the server device 1 can be used.

[Configuration Example of Server Device 1]
FIG. 4 is a block diagram for explaining a configuration example of the server apparatus 1 according to this embodiment. As illustrated in FIG. 4, the server apparatus 1 includes a connection terminal 101 for the network 2 and a communication I / F 102. In addition, a search request storage file 103 for storing a search request from the mobile phone terminal 4, a hard disk drive (hereinafter abbreviated as HDD) 104, a subject database (hereinafter abbreviated as subject DB) 105, a map database (hereinafter referred to as “DB”) (Abbreviated as map DB). In addition, a search processing unit 111, an answer information generation unit 112, and an additional information request unit 113 are provided as main parts for providing the subject search service. In addition, the server device 1 includes a control unit 120.

  The control unit 120 controls each unit of the server device 1, and is a computer device configured by connecting a CPU 121, a ROM 122, and a RAM 123 through a CPU bus 124 as shown in FIG. In the computer apparatus, the CPU 121 functions to read and execute so-called firmware such as BIOS stored in the ROM 122 and application software stored in the hard disk of the HDD 104 described later. Then, the CPU 121 generates a control signal to each unit, supplies it to each related unit, and receives data from each unit and processes it.

  The connection terminal 101 forms a connection end with the network 2. A communication line from the network 2 is connected to the connection terminal 101. The communication I / F 102 receives various information addressed to the own device transmitted through the network 2, converts this into information in a format that can be processed by the own device, and supplies the information to the control unit 120. Further, the communication I / F 102 converts various information for transmission supplied through the control unit 120 into a signal of a format to be transmitted to the network 2 and transmits the signal to the network 2.

  The search request storage file 103 is for temporarily storing a search request transmitted from the mobile phone terminal 4. In this search request storage file, a search request received through the connection terminal 101 and the communication I / F 102 is stored through the control unit 120. The HDD 104 includes a hard disk as a recording medium, and performs writing / reading on the hard disk in accordance with the control of the control unit 120. The hard disk of the HDD 104 stores, for example, various application programs and data obtained by various processes.

  As described above, the subject DB 105 includes one or more image data forming one or more images of each of various plants and various seafoods to be searched, and detailed information about them. Information is accumulated and formed. In this embodiment, plants are roughly classified into, for example, “fungi”, “grass / flowers”, “trees”, and “fruits”, and data is registered for each. In addition, seafood is classified into, for example, “fish”, “shellfish”, “seaweed”, and “other aquatic organisms”, and data is registered for each.

  FIG. 5 is a diagram for explaining an example of registration data in the subject DB 105, and shows an example of registration data of fungi. As shown in FIG. 5, the registration data of the subject DB 105 has an identification ID of the registration data and a subject type (subject classification) as header information. The registration data includes information about the subject, such as name, subject, scientific name, habitat, growing season, growing place, toxicity, etc., appearance characteristics, cooking information, and one or more image data. In this embodiment, each piece of information other than cooking information is a search target based on image data and related information (metadata) included in a search request from the mobile phone terminal 4. The cooking information includes information such as how to cook when there is no toxicity and it is suitable for food.

  As shown in FIG. 5, the image data of the subject included in the registration data includes image data for a plurality of images including the characteristic part of the subject. In the case of the example shown in FIG. 5, image data 1 (full shape 1), image data 2 (upper part of the umbrella), image data 3 (back part of the umbrella), image data 4 (stem part), image data 5 (cross section) ), Image data 6 (full form 2), image data 7 (full form 3), and image data 8 (full form 4). As described above, a plurality of pieces of image data are provided for one subject so that comparison from various angles is possible and accurate information regarding the subject to be searched can be reliably searched. It is to do. In addition, although FIG. 5 demonstrated the example of the registration data about fungi, the same registration data is formed also about another plant and seafood, and is accumulate | stored in object DB105.

  In this embodiment, the map DB 106 stores detailed map information. The map information includes the location of various landmarks such as mountains, rivers, towns, roads, railway lines, various stores, hospitals, clinics, schools, police stations, fire stations, shrines and temples, administrative boundaries, etc. The size can be accurately expressed. The map DB 106 stores map data (map drawing data) such as vector data and raster data for drawing a map in association with latitude / longitude information. The information in the map DB 106 can be used, for example, in determining what kind of place the shooting location is.

  In the server device 1, the search processing unit 111, the answer information generation unit 112, and the additional information request unit 113 are main units for providing a subject search service. Here, the case where the subject search for fungi is performed will be described as an example. The search processing unit 111 uses the image data and related information included in the search request stored in the search request storage file 103 to refer to the stored information in the subject DB 105 through the control unit 120 and search from the stored information in the subject DB 105. Search for information about the subject of interest.

  In this embodiment, in the registration data of the subject DB 105 described with reference to FIG. 5, it is assumed that the image data 1 (full form 1) forms a representative image of the subject. Therefore, the search processing unit 111 compares the image data included in the search request from the mobile phone terminal 4 with the entire image data 1 (image data 1) in the subject DB 105 to determine the similarity. At the same time, the search processing unit 111 compares the related information included in the search request from the mobile phone terminal 4 with the registration information in the subject DB 105.

  Here, the comparison between the related information included in the search request from the mobile phone terminal 4 and the registration information in the subject DB 105 can be performed with respect to the following points. That is, the shooting position included in the related information and the habitat area of the subject DB 105 can be compared, and the shooting date and time included in the related information and the growth time of the subject DB 105 can be compared. Further, when there is text information related to the appearance of the subject input by the user as the related information, the text information can be compared with the information on the appearance feature of the subject DB 105. Further, when there is text information related to the shooting location by user input as the related information, the text information can be compared with the information on the growth location of the subject DB 105.

  Then, the similarity between the image data included in the search request and the image data 1 of the entire form 1 (image data 1) in the subject DB 105 is higher than a predetermined threshold, and the related information included in the search request and the registration information in the subject DB 105 Suppose that they match. In this case, the search processing unit 111 can determine that the subject of the image formed by the image data included in the search request matches the subject indicated by the registration data in the subject DB 105. That is, information related to the subject to be searched can be searched from the accumulated data in the subject DB 105.

  In this case, the search processing unit 111 notifies the response information generation unit 112 of the identification ID of the registered data in the subject DB 105 that has been searched, and requests generation of response information. Based on the notified identification ID, the response information generation unit 112 reads the corresponding registration data from the subject DB 105 through the control unit 120, and based on this, responds to the search request source including name, toxicity, cooking information, and the like. Generate information. Then, the response information generation unit 112 sends the generated response information to the network 2 through the control unit 120, the communication I / F 102, and the connection terminal 101, and transmits it to the requesting mobile phone terminal 4.

  Further, it is assumed that the search processing unit 111 cannot search for information on the subject to be searched from the accumulated data in the subject DB 105. In this case, the search processing unit 111 identifies additional information that is requested to be provided to the search request source based on the current search result, and notifies this to the additional information request unit 113. In this case, when the corresponding registered data cannot be searched at all, the search processing unit 111 follows the predetermined condition to obtain one or more new image data, more detailed text information on the appearance of the subject, and more detailed shooting location. The text information related to In addition, the search processing unit 111 can search for registration data that is candidates to some extent, information that can be appropriately compared with them, for example, if there is a feature in a specific part of the subject, The image data of a part, text information about the specific part, etc. are used as additional information.

  Based on the notification from the search processing unit 111, the additional information request unit 113 forms a request for providing additional information to be transmitted to the search request source, and transmits the request to the network 2 through the control unit 120, the communication I / F 102, and the connection terminal 101. To the requesting mobile phone terminal 4. Thereby, the mobile phone terminal 4 as the search request source can grasp what additional information is requested from the server device 1. Then, the search request source mobile phone terminal 4 acquires additional information corresponding to the request from the server device 1, generates a re-search request including the additional information, and transmits it to the server device 1. The re-search request is received by the server device 1 through the connection terminal 101 and the communication I / F 102 and stored in the search request saving file 103.

  Then, using the image data and related information included in the re-search request, the search processing unit 111 performs a re-search process. As described above, when the search target is narrowed down to some extent by the initial search processing of the search processing unit 11, the information is managed by the search processing unit 111 and the control unit 120. Accordingly, the search processing unit 111 can perform the re-search process by narrowing the range of registration data in the subject DB 105 to be searched.

  As a result of the re-search process in the search processing unit 111, it is assumed that information related to the subject to be searched can be searched from the accumulated data in the subject DB 105. In this case, as described above, the search processing unit 111 notifies the response information generation unit 112 of the identification ID of the registered data in the subject DB 105 that has been searched, and requests generation of response information. Then, the response information generation unit 112 transmits the generated response information to the requesting mobile phone terminal 4 through the network 2 as described above.

  Further, it is assumed that the search processing unit 111 cannot search for information on the subject to be searched from the accumulated data in the subject DB 105. In this case, the search processing unit 111 requests the response information generation unit 112 to form response information (information indicating unsuccessful search) indicating that the search has failed. In this case, the answer information generation unit 112 forms a search unsuccessful message according to a predetermined template and transmits it to the search request source.

  The search failure message may be, for example, “No applicable data found. It may be toxic. It is not recommended to collect.”, “No applicable data found. Please check if there is any toxicity in the person who has, and make sure that it is not toxic before eating. " In addition to this, it is possible to introduce contact points that can inquire about the presence or absence of sputum toxicity, or to introduce search agencies that can test whether sputum is toxic. Also, if you collect and take it home, you can recommend searching again after washing it cleanly.

  As described above, the server device 1 according to this embodiment can provide the subject search service to the mobile phone terminal 4.

[Processing of Mobile Phone Terminal 4 and Processing of Server Device 1]
Next, processing in the case of subject search in the mobile phone terminal 4 having the configuration described with reference to FIGS. 2 and 3, and subject in the server apparatus 1 having the configuration described with reference to FIGS. The process for performing a search will be described with reference to a flowchart.

[Process when subject search is performed on the mobile phone terminal 4]
6 to 8 are flowcharts for explaining processing when subject search is performed on the mobile phone terminal 4. FIG. 9 shows a display screen 413G of the display unit 413 of the mobile phone terminal 4 in this processing. It is a figure for demonstrating the example of the screen displayed.

  When the control unit 420 of the mobile phone terminal 4 receives a predetermined operation input for using the subject search service provided by the server device 1 through the key operation unit 411, the control unit 420 executes the processes shown in FIGS. Here, the predetermined operation input is to select, for example, an icon or a menu item associated with the process. Then, the control unit 420 uses the URL (Uniform Resource Locator) of the site that provides the subject search service of the server device 1 to access the site, obtain the subject search application, and execute it (step S101). ).

  The subject search application and the like include necessary screen information and the like. Through the processing in step S101, preparation for searching for a subject is performed by transmitting and receiving predetermined information between the mobile phone terminal 4 and the server device 1, and the subsequent processing is performed. It is assumed that the URL of the site providing the subject search service of the server device 1 is registered in the EEPROM 424 in advance. Of course, it is also possible to search using the search function of the browser and access the site. The acquired subject search application or the like is stored in the EEPROM 424 and executed, for example.

  Then, the control unit 420 displays a subject type selection screen (FIG. 9A) on the display screen 413G of the display unit 413 through the display control unit 412 (step S102), and accepts an operation input from the user through the key operation unit 411. (Step S103). As shown in FIG. 9A, the subject type selection screen is used by the user of the mobile phone terminal 4 to select the type of subject to be searched. In step S103, the user can select a target subject type by using the up arrow key and the down arrow key of the key operation unit 411.

  Then, the control unit 420 determines whether or not the operation input received in step S103 is a selection operation of the “end” button (step S104). If the “end” button is not selected, the “OK” button is selected. It is determined whether or not the selection operation is performed (step S105). If it is determined in step S105 that the “OK” button is selected, the type of the subject on which the cursor CS is positioned on the subject type selection screen (FIG. 9A) is set as the type of the subject selected by the user. It is identified (step S106). The subject type specified here is included in the search request transmitted to the server device 1. Thereby, in the server apparatus 1, only registered data belonging to the specified subject type can be set as a search target.

  Next, the control unit 420 of the mobile phone terminal 4 displays the photographing guide screen (FIG. 9B) on the display screen 413G of the display unit 413 (Step S107), and accepts an operation input from the user through the key operation unit 411 (Step S107). S108). As shown in FIG. 9B, the shooting guide screen includes information for guiding how the user of the mobile phone terminal 4 should shoot a subject to be shot.

  Then, the control unit 420 determines whether or not the operation input received in step S108 is a selection operation of the “end” button (step S109). When the “end” button is not selected, the “OK” button is selected. It is discriminate | determined whether it is selection operation of (step S110). When it is determined in step S110 that the “OK” button is selected, the control unit 420 performs photographing of the subject and acquisition of related information (metadata) (step S111).

  Specifically, in step S111, the control unit 420 causes the camera unit 416 to function and shoots a subject according to a shutter operation from the user. In addition, the control unit 420 acquires related information from the GPS unit 435, the clock circuit 431, and the like when shooting is performed. Then, the control unit 420 temporarily stores, in the RAM 423, image data that forms an image of a subject obtained by shooting and related information about the image data. After the process of step S111, the control unit 420 of the mobile phone terminal 4 proceeds to the process of step S112 shown in FIG.

  Then, the control unit 420 displays the captured image display screen (FIG. 9C) based on the image data obtained by capturing in step S111 of FIG. 6 through the control unit 420 and the display control unit 412, and the display screen 413G of the display unit 413. (Step S112). As shown in FIG. 9C, the photographed image display screen displays an image based on the image data obtained by the photographing process in step S111. The captured image display screen is referred to in order to determine whether or not to re-capture the target object image by the user. Thereafter, the control unit 420 receives an operation input from the user through the key operation unit 411 (step S113).

  Then, the control unit 420 determines whether or not the operation input accepted in step S113 is a selection operation of the “end” button (step S114). It is determined whether or not the button is selected (step S115). If it is determined in the determination processing in step S115 that the “re-shoot” button is selected, the control unit 420 repeats the processing from step S111 in FIG. 6 to re-shoot the target subject and acquire related information. And do. That is, it is possible to re-shoot (re-shoot) an image of a target subject.

  If it is determined in step S115 that the “re-shoot” button is not selected, control unit 420 determines whether or not the accepted operation input is a “OK” button selection operation (step S116). . When determining in step S116 that the “OK” button is selected, the control unit 420 displays the shooting memo screen (FIG. 9D) on the display screen 413G of the display unit 413 (step S117).

  As shown in FIG. 9D, the shooting memo screen has an input field for the characteristics of the photographed subject and the characteristics of the shooting location. Text information can be input to the input field through the key operation unit 411. Note that the shooting memo screen is not limited to that shown in FIG. 9D. For example, if the subject follows the example of a cocoon, the characteristics of the photographed subject are divided into items such as “umbrella diameter”, “color of the top of the umbrella”, “color of the back of the umbrella”, etc. You can also ask for information. Further, as the characteristics of the shooting location, it is possible to divide items into details, such as “types of surrounding trees”, “types of surrounding flowers”, “degree of sunlight”, etc., and request information input.

  Then, the control unit 420 receives an operation input from the user through the key operation unit 411 (step S118), and determines whether or not the operation input received in step S118 is a selection operation of the “end” button (step S118). S119). If it is determined in step S119 that the “end” button is not selected, it is determined whether or not the “OK” button is selected (step S120). If it is determined in the determination process in step S120 that the “OK” button is selected, the control unit 420 acquires the image data and related information obtained by the process in step S111 and temporarily stored in the RAM 423, and step S118. The search request described with reference to FIG. 3 is generated from the input information (text data) for the shooting memo screen received in (step S121). Note that the header part of the search request includes the type of subject specified in step S106, information indicating that the search request is the first search request, and the like.

  Then, the control unit 420 transmits the search request generated in step S121 to the network 2 through the connection terminal 101 extending over the communication I / F 102, and transmits it to the site that provides the subject search service of the server device 1 (step S122). Thereafter, the process proceeds to step S123 shown in FIG. That is, the control unit 420 receives a reply from the site of the server device 1 in response to the search request transmitted from the own device (step S123), and waits for a reply from the site (step S124). When determining in step S124 that a reply from the site has been received, the control unit 420 determines whether the received reply requests provision of additional information (step S125).

  When it is determined in the determination process of step S125 that the provision of additional information is requested, the control unit 420 displays the additional information request screen (FIG. 9E) on the display screen 413G of the display unit 413 (step S126). ). As shown in FIG. 9E, the additional information request screen includes a display indicating that the search has been unsuccessful and information indicating what additional information is required. Thereafter, the control unit 420 receives an operation input from the user through the key operation unit 411 (step S127).

  Then, the control unit 420 determines whether or not the operation input received in step S127 is a selection operation of the “end” button (step S128). If the “end” button is not selected, the “OK” button is selected. It is determined whether or not the selection operation is performed (step S129). If it is determined in the determination process in step S129 that the “OK” button is selected, the control unit 420 re-photographs the subject image or inputs additional memo information in response to a request for providing additional information. (Step S130).

  In this step S130, in response to a request for providing additional information, a plurality of different portions of the subject are photographed, or text information relating to the requested subject and photographing location is input. Specifically, in the form shown in FIG. 9B, a shooting guide screen showing what kind of image of which part of the subject is necessary is displayed so that shooting can be performed according to this, and as shown in FIG. 9D. The shooting memo screen is displayed in such a manner that necessary text information can be input. Thereafter, the control unit 420 repeats the processing from step S121 shown in FIG. 7, forms a re-search request for the content described with reference to FIG. 3 (step S121), and transmits it to the site of the server device 1 (Step S122). In this case, the re-search request formed in step S121 includes information indicating that the request is a re-search request, information indicating what additional information is included, and the like.

  On the other hand, when it is determined that the provision of additional information is not requested in the determination process of step S125, the control unit 420 displays the search result screen (FIG. 9F) on the display screen 413G of the display unit 413 through the display control unit 412. (Step S131). As shown in FIG. 9F, the search result screen includes the name of the subject to be searched, the presence / absence of toxicity, whether or not it is suitable for food, and cooking information if it is suitable for food. It is. Although not shown, if the search is unsuccessful, as described above, the search is unsuccessful, and the subject may be toxic, so a message that recommends not to collect is included. It becomes.

  Thereafter, the control unit 420 receives an operation input from the user through the key operation unit 411 (step S132), and determines whether or not the operation input received in step S132 is a selection operation of the “return” button ( Step S133). When it is determined in the determination process in step S133 that the “return” button is selected, the control unit 420 notifies the site of the server device 1 that the use of the subject search service is continued (step S134). Thereafter, the control unit 420 repeats the processing from step S102 shown in FIG. 6 so that, for example, a search for a new object can be continuously performed.

  In the determination process of step S133, when it is determined that the selection of the “return” button has not been received, the control unit 420 determines whether or not the “end” button selection operation has been received (step S135). In the determination process of step S135, when it is determined that the selection of the “end” button has not been received, the control unit 420 repeats the process from step S132. If it is determined in step S135 that the selection of the “end” button has been received, the control unit 420 performs predetermined end processing such as returning the display screen to the state before using the subject search service ( Step S136). Then, the subject search process shown in FIGS. 6 to 8 ends.

  In step S104 in FIG. 6, step S109, step S114 in FIG. 7, step S119, and step S128 in FIG. 8, it is determined that the received operation input is a selection operation of the “end” button. In this case, the process proceeds to step S136 in FIG. 8, a predetermined end process is performed (step S136), and the processes shown in FIGS. 6 to 8 are ended. Also, assume that it is determined in step S105, step S110 in FIG. 6, step S116 in FIG. 7, step S120, and step S129 in FIG. 8 that the accepted operation input is not an “OK” button selection operation. In this case, the process returns to the previous operation input reception process and continues to receive operation input from the user.

  As described above, in the cellular phone terminal 4 of this embodiment, when the corresponding data cannot be retrieved by the retrieval process in response to the first retrieval request including the image data of the subject and the related information, the server device 1 You can receive a request for providing additional information. Then, in response to the additional information provision request, a re-search request including the additional information can be transmitted to the server device 1 to instruct to perform a re-search process using the additional information. As a result, it is possible to more reliably search for information regarding the subject to be searched.

[Process when subject search in server device 1]
FIG. 10 and FIG. 11 are flowcharts for explaining processing when the subject search is performed in the server device 1. The processes shown in FIGS. 10 and 11 are processes that are always executed in cooperation with the units related to the control unit 120 of the server device 1.

  The control unit 120 of the server device 1 accepts access from the mobile phone terminal 4 (step S201), and waits until accepting access from the mobile phone terminal 4 (step S202). When it is determined in step S202 that the access from the mobile phone terminal 4 has been accepted, the control unit 120 provides the access source mobile phone terminal 4 with an application or the like for performing subject search processing (step S203). ). As a result, an environment in which a subject search service can be provided to the access source mobile phone terminal 4 is prepared.

  Thereafter, the control unit 120 of the server device 1 accepts a request from the access source mobile phone terminal 4 (step S204), and determines whether or not the accepted request is an instruction to end the search process. (Step S205). In the determination process of step S205, it is determined that the received request or the like does not instruct the end of the search process. In this case, since the received request or the like is a search request, the control unit 120 stores the search request in the search request storage file 103 and instructs the search processing unit 111 to perform search processing ( Step S206).

  Then, under the control of the control unit 120, the search processing unit 111 reads the search request stored in the search request storage file 103, separates the image data and the related information, and makes each available (see FIG. Step S207). Then, the search processing unit 111 performs processing for searching for information related to the subject to be searched from registration data in the subject DB 105 based on the separated image data and related information (step S208).

  Thereafter, the search processing unit 111 determines whether or not the search process has been successful, that is, whether or not information related to the search target subject has been searched (step S209). If it is determined in step S209 that the search has not been successful, the search processing unit 111 controls the additional information request unit 113 to generate a request for providing additional information (step S210). In the determination process of step S209, when it is determined that the search is successful, the search processing unit 111 controls the answer information generation unit 112, and based on information about the search target subject obtained by the search process, etc. Response information is generated (step S211).

  Then, the additional information request unit 113 and the response information generation unit 112 return the additional information provision request generated in step S210 and the response information generated in step S211 to the requesting mobile phone terminal 4 (step S212). Thereafter, the control unit 120 of the server apparatus 1 proceeds to the process of step S213 in FIG.

  The control unit 120 receives an instruction or the like from the mobile phone terminal 4 of the access source (search request source) (step S213), and determines whether or not the received instruction is an instruction to continue the subject search process. (Step S214). In the determination process of step S214, when it is determined that the received instruction or the like is an instruction to continue the subject search process, the process from step S204 shown in FIG. Process).

  In the determination process of step S214, when it is determined that the received instruction or the like is an instruction to continue the subject search process, it is determined whether or not the received instruction is an instruction to end the subject search process ( Step S215). In the determination process of step S215, when it is determined that the received instruction or the like is an instruction to end the search process of the subject, a relationship in which the subject search service can be used with the mobile phone terminal 4 that is the search request source A predetermined termination process for terminating (environment) is executed (step S216). Thereafter, the processing from step S201 shown in FIG. 10 is repeated so that a subject search service can be provided to a new access source.

  When it is determined in step S205 in FIG. 10 that the accepted request is an end instruction, the process proceeds to step S216 in FIG. 11, and in step S215, it is determined that the accepted instruction is not an end instruction. If so, the processing from step S213 is repeated.

  As described above, the server device 1 provides a subject search service in response to a search request from the mobile phone terminal 4, and when information about the subject to be searched cannot be searched, the mobile phone terminal 4 as a search request source is provided. And request additional information. Then, in response to a request for provision of the additional information, when a re-search request including the additional information is provided from the search request source mobile phone terminal 4, a re-search process for the subject to be searched is performed accordingly. Can be executed.

[Effect of the embodiment]
Thus, in this embodiment, the subject search device is realized by the mobile phone terminal 4 and the server device 1. The server device 1 requests the mobile phone terminal 4 to provide additional information even if the target information cannot be searched in the search process corresponding to the first search request from the mobile phone terminal 4, and in response to this, The re-search process can be executed in consideration of the additional information provided. Thereby, it is possible to search and acquire accurate information related to the subject to be searched more reliably.

[Modifications, etc.]
In the above-described embodiment, the re-search is performed only once. However, if there is new additional information, the provision of new additional information is requested a plurality of times, and a plurality of times are requested accordingly. It is also possible to perform a re-search.

  Further, the map DB 106 is referred to based on the shooting position included in the related information of the image data, and it is determined whether the position of the mobile phone terminal 4 is a subject collection place or a home other than the collection place. The additional information requested can also be changed. For example, when the shooting location is a home other than the collection location, the subject can be easily separated, so image data of the cross section of the subject may be requested.

  Further, as described above, when requesting additional information, it is also possible to request that the entire subject to be searched or the size of a predetermined portion is measured and transmitted as additional information. Further, when requesting additional information, it is also possible to request detailed information of information related to the situation around the shooting location of the subject to be searched (for example, the base of Akamatsu, the tide pool of a rocky place, etc.) as additional information.

  As described with reference to FIG. 9, the subject DB 105 includes image data obtained by photographing the subject to be taken from various angles. Therefore, it is possible to request the provision of search target image data corresponding to the image data included in the accumulated data of the subject DB 105, and to perform recognition processing using a 3D image (three-dimensional stereoscopic image). That is, in order to perform 3D recognition, it is also possible to request that the subject to be searched be photographed from various directions and transmit the photographed image data.

  In the cellular phone terminal 4 according to the above-described embodiment, text information about the subject and text information about the shooting location are manually input through the key operation unit 411. However, the present invention is not limited to this. Information relating to the subject and information relating to the shooting location can be received by voice input, and the voice can be recognized and converted into text information, or can be held as voice data. Note that when the information about the subject and the information about the shooting location are held as voice data, the server apparatus 1 may recognize the voice and convert it into text information for use. Since the mobile phone terminal 4 includes the microphone 404, it can be dealt with by receiving input of audio data through the microphone 404.

  Further, in the above-described embodiment, the description has been made on the assumption that the subject is searched at the destination, but the present invention is not limited to this. For example, in the case of a cocoon or the like, image data suitable for a search may not be acquired even if the state grows and grows and the state is photographed as it is. For this reason, in the case of a bag or the like, it may be collected and taken home, and at home or the like, it may be divided into a single piece (one piece) and photographed for retrieval.

Further, as a result of performing the search processing on the subject to be searched, if similar registration data exists in the subject DB 105 and it is suspected that the subject is toxic, a check using a similar algorithm is performed. Thereby, it is possible to search for information on the subject more accurately.

  The cooking information about the subject to be searched is not only text data, but also shows a cooking example as a still image, or notifies the cooking method (cooking method) using one or both of video and audio. May be. In this case, the subject DB 105 may include not only text information but also necessary still image data, moving image data, and audio data in the registration data.

  In addition, by mounting the functions of the mobile phone terminal 4 and the server device 1 of the above-described embodiment on, for example, a tablet terminal or a laptop computer, a so-called stand-alone subject search device can be realized. .

  In addition, the functions of the server device 1 described above can be mounted on a personal computer installed at home or the like. In this case, the personal computer has a so-called local subject DB. Then, the mobile phone terminal 4 is connected to the personal computer through a wired or wireless digital interface, and predetermined information is exchanged between the mobile phone terminal 4 and the server device 1 according to the above-described embodiment. By doing so, the search process for the subject as the search target can be performed in substantially the same manner as in the above-described embodiment.

  In the embodiment described above, the subject to be searched can be applied to inanimate objects such as insects, animals, ores, as well as plants and seafood.

[Others]
In the server device 1 illustrated in FIG. 4, the functions of the search processing unit 111, the response information generation unit 112, and the additional information request unit 113 are realized as functions of the control unit 120 by a program executed in the control unit 120. You can also

Further, as can be seen from the description of the above-described embodiment, the camera unit 416 of the mobile phone terminal 4 realizes the function as the photographing unit. In addition, as an acquisition unit, various information acquisition functions such as a GPS unit 435, a clock circuit 431, a key operation unit 411, and an orientation sensor (not shown) are realized. Also, the storage means function is realized by the subject DB 105 of the server apparatus 1, and the search means and the first and second re-search means functions are realized by the search processing unit 111 of the server apparatus 1, and the first and second The additional information request unit 113 of the server device 1 realizes the function of the second addition request unit. The functions of the first and second control means are realized by the control unit 420 of the mobile phone terminal 4.

Also, the functions of the first and second answer information generating means are realized by the answer information generating unit 112 of the server device 1. Further, the first, second , and third forming means on the terminal device side are realized by the control unit 420 of the mobile phone terminal 4, and the first, second , and third transmitting means, first , The functions of the second receiving unit and the first and second answer information receiving units are realized mainly by the transmitting / receiving antenna 401 and the transmitting / receiving processing unit 402 of the mobile phone terminal 4. The functions of the first, second and third receiving means, first and second additional request transmitting means, and first and second answer information transmitting means on the information processing apparatus side are mainly the functions of the server device 1. The connection terminal 101 and the communication I / F 102 are realized.

  Further, the flowcharts of FIGS. 6 to 8 and the processes shown in FIGS. 10 and 11 correspond to the method of the present invention. A program for executing processing according to the flowcharts shown in FIGS. 6 to 8 corresponds to the subject search information providing program of the present invention. A program that executes processing according to the flowcharts shown in FIGS. 10 and 11 corresponds to the subject search program of the present invention. Therefore, a program for executing the processing shown in FIGS. 6 to 8 is formed, and is installed in a terminal device such as a mobile phone terminal so as to be executable in the device. Further, a program for executing the processing shown in FIGS. 10 and 11 is formed, and this is installed in an information processing apparatus such as a server apparatus or a personal computer, and is executable in the apparatus. Thereby, the subject search device of the present invention can be easily realized by the terminal device and the information processing device.

  DESCRIPTION OF SYMBOLS 1 ... Server apparatus, 101 ... Connection terminal, 102 ... Communication I / F, 103 ... Search request save file, 104 ... HDD, 105 ... Subject DB, 106 ... Map DB, 111 ... Search processing part, 112 ... Answer information generation part , 113: additional information request unit, 120: control unit, 2 ... network, 3 ... base station, 4 ... mobile phone terminal, 401 ... transmission / reception antenna, 402 ... transmission / reception processing unit, 403 ... receiver (speaker), 404 ... transmission 411: Key operation unit, 412 ... Display control unit, 413 ... Display unit, 414 ... Audio processing unit, 415 ... Speaker, 416 ... Camera unit, 420 ... Control unit, 431 ... Clock circuit, 432 ... Memory Slot part, 433 ... External input / output terminal, 434 ... External I / F, 435 ... GPS part, 436 ... GPS antenna, 5 ... GPS satellite

Claims (6)

Photographing means for photographing an image of a subject to be searched and obtaining image data of the subject;
Acquisition means for acquiring related information about the image data including at least a shooting position and a shooting date and time for the image data acquired through the shooting means ;
Storage means for storing, for each of a plurality of objects that may be subjects, one or more image data forming one or more images of the object , and storing information relating to the object and map information ;
Search means for searching for information on the subject to be searched from stored information in the storage means based on the acquired image data and the related information;
First information requesting means for requesting first additional information when information on the subject to be searched cannot be searched;
First control means for controlling to acquire the first additional information through one or both of the photographing means and the acquiring means in response to a request for the first additional information;
First re-search means for re-searching information on the subject to be searched from the stored information in the storage means, taking into account the first additional information acquired in accordance with the control of the first control means. When,
Second information requesting means for requesting information on a specific part of the subject as second additional information when information on the subject that is a candidate for search can be retrieved;
Second control means for controlling to acquire the second additional information through one or both of the photographing means and the acquiring means in response to a request for the second additional information;
Second re-search means for re-searching information relating to the subject to be searched from the stored information in the storage means, taking into account the second additional information acquired in accordance with the control of the second control means. And a subject search device. The subject search device according to claim 1,
First answer information generating means for generating first answer information based on the information obtained as a result of the search when the information on the subject to be searched can be searched by the search means;
Second answer information generating means for generating second answer information based on a search result in the first re-search means or the second re-search means;
A subject search device comprising: The subject search device according to claim 2,
The subject search device includes a terminal device and an information processing device,
The terminal device
The photographing means; the obtaining means;
A receiving means for receiving an instruction input from a user, and the first and second control means;
First forming means for forming a search request including the image data acquired by the photographing means and the related information acquired by the acquiring means;
First transmission means for transmitting the search request to the information processing apparatus;
First receiving means for receiving a request for the first additional information;
Second forming means for forming a first re-search request including information acquired by one or both of the photographing means and the acquiring means in response to control of the first control means;
Second transmission means for transmitting the first re-search request to the information processing apparatus;
Second receiving means for receiving the request for the second additional information;
Third forming means for forming a second re-search request including information acquired by one or both of the photographing means and the acquiring means in response to control of the second control means;
Third transmission means for transmitting the second re-search request to the information processing apparatus;
First answer information receiving means for receiving the first answer information;
And second answer information receiving means for receiving the second answer information,
The information processing apparatus includes:
The storage means, the search means, the first and second addition request means, the first and second re-search means, and the first and second answer information generation means,
First receiving means for receiving the search request from the terminal device;
First additional request transmitting means for transmitting a request for the first additional information to the terminal device;
Second receiving means for receiving the first re-search request from the terminal device;
Second addition request transmitting means for transmitting the request for the second additional information to the terminal device;
Third receiving means for receiving the second re-search request from the terminal device;
First answer information transmitting means for transmitting the first answer information to the terminal device;
Second answer information transmitting means for transmitting the second answer information to the terminal device;
A subject search device comprising: The subject search device according to claim 3,
The subject search device, wherein the terminal device and the information processing device are connected through a network. A photographing step of photographing an image of a subject to be searched through photographing means and obtaining image data;
An acquisition step of acquiring related information about the image data including at least a shooting position and a shooting date and time for the image data acquired through the imaging unit, through the acquisition unit ;
The search means includes one or more image data that forms one or more images of the appearance of the target for each of a plurality of targets that are subjects based on the acquired image data and the related information, and A search step for searching for information on a subject to be searched with reference to information in a storage means for storing information and map information ;
A first addition request step in which the first additional request forming means requests the first additional information when the information on the subject cannot be searched;
A first accepting step of accepting a first acquisition instruction of information in response to the request for the first additional information from a user through accepting means;
A first control step of controlling the first control means to acquire the first additional information through one or both of the photographing means and the acquisition means in response to the first acquisition instruction; ,
Considering also the first additional information newly acquired in accordance with the control of the first control means, the first re-search means refers to the information in the storage means, and the subject to be searched A first re-search process for re-searching information about
A second addition requesting step of requesting information on a specific part of the subject as second additional information when information on the subject that is a candidate for search can be retrieved;
A second accepting step of accepting a second acquisition instruction of information in response to the request for the second additional information from the user through accepting means;
A second control step of controlling the second control means to acquire the second additional information through one or both of the photographing means and the acquisition means in response to the second acquisition instruction; ,
In consideration of the second additional information newly acquired in accordance with the control of the second control means, the second re-search means refers to the information in the storage means, and the subject to be searched And a second re-search step for re-searching information regarding the subject search method. An information processing apparatus that includes one or more image data that forms one or more images of the appearance of a target for each of a plurality of targets that are subjects, and includes a storage unit that stores information about the target and map information A program executed by an installed computer,
When a search request including image data obtained by photographing a subject to be retrieved from a search request source and related information including at least a photographing position and a photographing date and time for the image data is received through a receiving unit. A search step of searching for information on the subject to be searched by referring to the information in the storage unit based on the image data and the related information included in the search request;
A first additional request forming step of forming a first additional request for requesting provision of first additional information when information on the subject cannot be retrieved;
A first transmission step of transmitting the first addition request to the search request source through transmission means;
When the first re-search request including the first additional information from the search request source is received through receiving means, the first additional information included in the first re-search request is also taken into consideration. A first re-searching step for re-searching information on the subject to be searched with reference to information in the storage means;
A second addition request forming step for forming a second addition request for requesting information on a specific part of the subject as second additional information when information on the subject as a candidate for search can be searched;
A second transmission step of transmitting the second addition request to the search request source through transmission means;
When the second re-search request including the second additional information from the search request source is received through receiving means, the second additional information included in the second re-search request is also taken into consideration. And a second re-search step for re-searching information relating to the subject to be searched with reference to the information in the storage means.

Images