JP2010205243A - Portable information retrieval apparatus for retrieving object from real world - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique which readily and surely retrieves a specified object from an image of circumference by means of image processing in portable information retrieval equipment. <P>SOLUTION: A retrieval result acquiring means 60 retrieves an object, on the basis of accepted retrieval conditions from the information for retrieval stored in an information storage means 30 for acquiring the information for collating the object. An image reception means 65 receives a subject image, from a terminal 1, in which a retrieval subject range is photographed. A matching means 70 detects a portion that matches the information for checking from the received subject information. The terminal 1, for example, displays only a book "AB, etc." matched with the retrieval conditions among several books photographed in the subject image displayed on a display unit D and displays its related information. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an improvement in search technology.

  In recent years, various information retrieval techniques using information processing have been devised. As an example, an image of a target object in the real world is transmitted to a server and matched with information stored in the server in advance. Thus, a technique for answering what a certain object is like a digital picture book has been proposed (see, for example, Patent Documents 1 and 2). In addition, as similar to them, the technique disclosed in Patent Document 3 reads information such as a barcode attached to an article in the real world and transmits it via a communication network, and the server searches for information such as a related URL. To answer.

JP 2003-44497 A Japanese Patent No. 4137756 Japanese Patent Laid-Open No. 11-143895

  However, the prior art as described above obtains relevant information from the object of interest that has already been found in the real world, and conversely, based on conditions such as name and type, There was no technology for searching for objects from the real world in the search range. For this reason, the task of searching for a target product and its candidates from a variety of display items has to be done manually, and there has been a potential demand for efficiency.

  The present invention solves the above-described problems, and an object of the present invention is to provide a technique for easily and reliably searching for a designated object from surrounding images by image processing in a portable information retrieval device. is there.

  Based on the above object, the present invention includes the following embodiments. First, the aspect (1) relates to a terminal side, and includes an input unit, a photographing unit, an information display unit, a communication unit that communicates with a server via a communication network, and a computer that performs control. A portable information retrieval device that continuously captures and displays images according to the movement of the photographing means, and receives a designation of an object from the input means and transmits it to the server as designation data representing the designation content Image acquisition means for acquiring a target image by imaging a search target range with the imaging means, and image transmission means for sequentially transmitting the acquired target images to the server. And, based on the transmitted target image, result receiving means for sequentially receiving, from the server, position information or a display image indicating a portion matching the target object The computer realizes display means for sequentially and continuously displaying the indication of the matched portion on the display unit using the position information or the display image sequentially received. To do.

  Aspect (11) captures aspect (1) from the viewpoint of a method, and controls an input means, a photographing means, an information display unit, and a communication means that communicates with a server via a communication network. An information processing method using a portable information retrieval device that continuously captures and displays images according to the movement of the photographing means, and accepts designation of an object from the input means and the designated content Target designation processing to be transmitted to the server as designation data representing the image, image acquisition processing for obtaining a target image by photographing the search target range sequentially by the photographing means, and each acquired target image Image transmission processing for sequentially transmitting to the server, and position information or a display image indicating a portion matching the object based on the transmitted target image. A result reception process for sequentially receiving from the bar, and a display for sequentially indicating the matched portion on the display unit using the position information or the display image sequentially received continuously. The processing is realized by a computer.

  Aspect (14) captures aspects (1) and (11) from the viewpoint of a computer program. The communication means communicates with the server via the input means, the photographing means, the information display unit, and the communication network. And a computer that performs control, and controls a portable information retrieval device that continuously captures and displays images according to the movement of the photographing means, and the program controls the computer Thus, the target specifying process for receiving the specification of the target object from the input means and transmitting it to the server as the specification data representing the specified contents, and successively capturing the search target range by the imaging means Image acquisition processing for acquiring images, image transmission processing for sequentially transmitting each acquired target image to the server, and transmission In addition, based on the target image, position information indicating a portion matching the target object or a result receiving process for sequentially receiving display images from the server, and the position information received sequentially. Alternatively, a display process of sequentially displaying on the display unit the indication of the matched portion using a display image is performed.

  Another aspect (4) relates to the server side, which is a server including communication means that communicates with a portable information search device via a communication network, and that is designated data representing the designated content of the object from the portable information search device. From the portable information retrieval device, the image receiving means for sequentially receiving the target images obtained by capturing the search target range from the portable information search device, and the specified data from each of the target images sequentially received. Matching means for sequentially detecting a portion that matches the object represented by, and a display image or a match subjected to graphic processing for pointing out the portion that matches the object based on each target image The computer realizes result transmission means for sequentially transmitting the position information indicating the position of the selected portion as the detection result to the portable information retrieval device successively And features.

  Aspect (12) captures aspect (4) from the viewpoint of a method, and is an information processing method using a server including a communication unit that communicates with a portable information search device via a communication network, wherein the portable information search Designation reception processing for receiving designation data representing the designation content of the target object from the device, image reception processing for sequentially receiving the target images obtained by photographing the search target range from the portable information search device, and sequential reception A matching process for successively and successively detecting a portion matching the object represented by the designated data from each of the target images, and pointing out a portion matching the object based on each target image A result transmission that sequentially transmits, as a detection result, position information indicating a position of a matching image or a display image subjected to graphic processing processing to the portable information retrieval device Wherein the process and, the computer is realized.

  Aspect (7) relates to a search system comprising the portable information search device according to any aspect of the present invention and the server according to any aspect of the present invention.

  As a basic function and effect of the present invention, an object such as a product image obtained in advance is detected by matching from a target image obtained by photographing a display shelf and the object is pointed out by graphic processing. Can be easily and reliably searched from the real world.

  In particular, aspects (1), (11), (14), (4), (12), and (7) related to the client-server system effectively support the limited storage and calculation capabilities of the portable information terminal with those of the server. In addition, the latest information can be used from a portable information search device as a client by updating information on the server.

  Further, in another aspect (2), in the portable information retrieval device relating to any aspect relating to the terminal side, the target designating means is accepted from the input means as search accepting means for accepting designation of search conditions. Request transmitting means for transmitting a search request accompanied with specification of the search condition to the server.

  Another aspect (5) corresponds to the portable information retrieval device according to aspect (2). In the server according to aspect (4), the designation receiving means includes information for collation for each object. From the information storage means storing information, the search request receiving means for receiving a search request with the specification of search conditions from the portable information search device, and the search information stored in the information storage means, And a search result acquisition means for searching for an object based on the accepted search condition and acquiring information for collation of the object and passing it to the matching means.

  In this way, by performing matching after searching the target object according to the search condition specified by the terminal and its matching information on the server, a series of processes from condition specification to matching can be performed quickly and easily. It becomes easy to find a desired product from a large number of candidates.

In addition, according to another aspect (3) of the present invention, in any aspect relating to the terminal side, the display means (display processing) relatively clearly displays the matched portion and relatively determines the remaining portion. It is characterized by unclear display.
In this way, a display that makes it easy to grasp the target object at a glance can be realized by mask processing that makes the matched target object part conspicuous and the other part is dull or covered with a dark color.

According to another aspect (6) of the present invention, in any aspect of the present invention relating to matching ((4), (5), etc.), the matching means (matching processing) uses a partial image matching algorithm. It is characterized by.
As described above, by applying the partial image matching algorithm that performs the matching calculation with the feature amount or the like being narrowed down, it is possible to search for a target object in a wide range and with high accuracy using a large-capacity target image such as a high resolution.

  Another aspect (8) of the present invention, which is completed on a terminal, is a portable information retrieval device comprising an input means, a photographing means, an information display section, and a computer for controlling. A designation receiving means for accepting designation of an object from the input means, an image obtaining means for obtaining a target image by photographing a search target range sequentially by the photographing means, and the acquired From each target image, matching means for sequentially detecting a portion that matches the target object sequentially, and sequentially processing each target image for graphic processing that points out the matched portion on the display unit The display means for displaying is realized by a computer.

  Aspect (13) captures aspect (8) from the viewpoint of a method, and uses a portable information retrieval device including an input means, a photographing means, an information display unit, and a computer that performs control. An information retrieval method, wherein a designation reception process for accepting designation of an object from the input unit, an image acquisition process for acquiring a target image by capturing a search target range with the imaging unit sequentially and sequentially, From each of the acquired target images, a matching process for sequentially detecting a matching portion with the target object, and a graphic processing process for sequentially pointing the target images to the matching portion are performed. The display processing displayed on the display unit is realized by a computer.

  In addition to the basic functions and effects of the present invention, the above-described aspects (8) and (13) of the portable information retrieval device alone eliminate the need for communication facilities and costs, simplify the configuration, and improve reliability.

  Another aspect (9) of the present invention relates to the terminal side, and is an input unit, a photographing unit, an information display unit, a communication unit that communicates with a server via a communication network, a computer that performs control, A portable information retrieval device that continuously captures and displays images according to the movement of the photographing means, wherein the search accepting means accepts designation of search conditions from the input means, and the accepted search conditions A request sending means for sending a search request with designation of the server to the server, an information receiving means for receiving the matching information corresponding to the sent search condition from the server, and successively, the subject to be searched by the photographing means Image acquisition means for acquiring a target image by photographing a range, and sequentially matching the received matching information from each of the target images acquired sequentially The computer realizes matching means for detecting the display and display means for displaying each target image on the display unit by performing graphic processing for pointing out the matched portion in succession. .

  Aspect (10) relates to the server side corresponding to aspect (9), and is a server provided with a communication means that communicates with a portable information retrieval device via a communication network, and includes a collation information for each object. From the information storage means storing information, the search request receiving means for receiving a search request with the specification of search conditions from the portable information search device, and the search information stored in the information storage means, Search means for searching for an object based on the received search condition and acquiring information for collation of the object; information transmitting means for transmitting the obtained information for collation to the portable information retrieval device; Is realized by a computer.

  In this way, the search is performed on the server side, but the image matching is performed on the terminal side, so that it is not necessary to send a relatively large-capacity image from the terminal to the server for matching. Packet charges can be saved.

  Further, the present invention is not limited to the above, and a method, a computer program (including a case where download is provided) corresponding to each device and system, and a storage medium storing such a computer program are also aspects of the present invention. Furthermore, the operations and effects of the present invention configured as described above, and the configurations, functions, and effects in other aspects of the present invention are as described later through the embodiments and the like.

  ADVANTAGE OF THE INVENTION According to this invention, in the portable information search device, the technique which searches the designated target object easily and reliably from the surrounding image by image processing can be provided.

The figure which shows the structure about 1st Embodiment of this invention (it is hereafter the same as FIG. 8). The flowchart which shows a process sequence. The figure which illustrates the information for search. The figure which illustrates the information for collation contained in the information for search. The figure which shows the state which has input the search keyword as a search condition in a terminal. The figure which shows the mode of the display shelf which displayed the book. The figure which shows the state which is displaying the target image image | photographed with the camera provided in the back surface of the terminal on the display part of the terminal surface. The figure which shows the example of a display which displays the relevant information of the matched book while masking except the matched part in the display part of a terminal. The figure which shows the state from which the range of a target image and the display content according to it change sequentially sequentially. The figure which shows the structure about 2nd Embodiment of this invention. The flowchart which shows a process sequence about 2nd Embodiment of this invention. The figure which shows the structure about 3rd Embodiment of this invention. The flowchart which shows a process sequence about 3rd Embodiment of this invention.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described with reference to the drawings. It should be noted that assumptions common to those already described in the background art and problems are omitted as appropriate.

[1. Constitution〕
In the first embodiment, as shown in FIG. 1 (configuration diagram), a portable information device (hereinafter also referred to as a “terminal”) 1 serving as a portable information search device such as a smartphone and a server 2 are connected to the Internet or a mobile phone. The present invention relates to a search system (hereinafter referred to as “the present system”) combined via a communication network N typified by a telephone network. The present invention relates to an information processing method and computer corresponding to the present system, the terminal 1 and the server 2. -It can be grasped as other categories of technology such as programs.

  As shown in FIG. 1, the terminal 1 includes an operation input unit T such as a touch panel, a photographing unit (hereinafter also referred to as “camera”) C such as a CCD camera, an information display unit D such as a liquid crystal display panel, A communication unit 17 (such as a client communication module such as a mobile phone or a wireless LAN) that communicates with the server 2 via the communication network N, and continuously captures and displays images according to the movement of the imaging unit. . In addition, the terminal 1 includes a calculation control unit 15 such as a CPU and a storage device 16 such as an external storage device (HDD, flash memory, etc.) and a main memory as a configuration of a computer that controls each unit.

  In the terminal 1, the computer realizes elements (35, 40, etc.) such as each means shown in FIG. More specifically, a predetermined computer program stored (installed) in advance in the storage device 16 controls the computer (particularly, the arithmetic control unit 15), whereby elements (35, 40). The computer program in this case is acquired and downloaded by downloading from a predetermined website via the communication network N, for example.

  The server 2 also communicates with the terminal 1 via a communication network N as a general server computer configuration, such as a calculation control unit 25 such as a CPU, a storage device 26 such as an external storage device (HDD or the like) or a main memory. Communication means 27. In the server 2, the server computer realizes elements (30, 55, etc.) such as the means shown in FIG. More specifically, a predetermined computer program stored (installed) in advance in the storage device 26 controls the server computer (especially the arithmetic control unit 25), whereby elements (30 such as the means shown in FIG. , 55, etc.).

  Among the above elements (35, 55, etc.) of the terminal 1 and the server 2, the means for storing information is a variable such as a file, various databases (also referred to as “DB”), an array, etc. in each storage device 16 or 26. It may be realized in an arbitrary format such as a system setting value, or instead, may be provided on a separate external server connected so as to constitute one element of the server 2. Of the above-described elements (35, 55, etc.) of the terminal 1 and the server 2, other elements that perform information processing are processing means that implement and execute the following functions and processes.

[2. Action)
A processing procedure in the system configured as described above will be described with reference to the flowchart of FIG. In the present application, step numbers and other symbols are for indicating parts in the figure and are not related to the order of processing.

[2-1. (Specify target and send target image)
First, in the terminal 1, the target designating means receives designation of the object from the input means T and transmits it to the server 2 as designation data representing the designated content (target designation processing). In this case, the target object is one or more tangible objects (characters, figures, symbols, shapes, patterns, colors, combinations thereof, etc.), and it does not matter whether it is a biological inanimate object, a real estate for real estate, or a paid item. In addition, the designation of the target object is specified by selecting or transmitting the target object image received by e-mail as a recommended product from an acquaintance, etc., or the target object image downloaded from the website. Alternatively, it may be specified as a search condition such as a character string or an attribute value.

  The example shown here is the case where the latter is specified as a search condition. The target specifying means is constituted by a search receiving means 35 and a request transmitting means 40, and the information storage means 30 of the server 2 is set for each object. The search information including the verification information is stored, and an example thereof is shown in FIG.

  In this example, for each book, as related information that also serves as an attribute for attribute search (attribute value), the title of the book (“title”), author name, year of publication, publisher name, and a given website from readers In addition to the evaluation score (“number of stars”) summarizing the evaluation and the explanatory text, the file name (for example, “0003.gif” etc.) of the information for collation for detecting the cover of the book from the image is included. . The verification information is stored in a predetermined directory in the server 2, and as illustrated in FIG. 4, for each of the books “AB”, “CD”, and “EF”, a characteristic appearance of the cover, an image, and other This is data expressed in the form features.

  When the search receiving unit 35 receives a search condition specification from the input unit T (step S35, search reception process), the request transmission unit 40 sends a search request with the received specification of the search condition to the server 2. Transmit (step S40).

  As the search condition, for example, an attribute value such as a character string or the number of stars included in the attributes such as the title and the author name illustrated in FIG. 3 is designated. Regarding the number of stars that are evaluation values, for example, a method of displaying a star having three or more stars that can take up to five values is conceivable. Moreover, FIG. 5 shows the surface provided with the display part D which also serves as a touch panel for the terminal 1 which is a smartphone. In this example, while confirming the input characters in the input field E displayed on the display part D, A state in which a search keyword is input with the finger group F using the displayed button group B is shown.

  Following the specification of such a search condition, the image acquisition means 45 acquires the target image by successively capturing the search target range with the camera C (step S45, image acquisition processing), and the image transmission means. 50 sequentially transmits the acquired target images sequentially to the server 2 (step S50, image transmission processing). Here, FIG. 6 is a diagram of a book display shelf R installed on the wall surface in front of the user having the terminal 1 as viewed from the front. FIG. 7 shows a state in which the display shelf R shown in FIG. 6 is photographed using a camera C (FIG. 1) provided on the back of the terminal 1 (not shown). By displaying the image on the display unit D, the appearance is such that the back display shelf can be seen through.

[2-2. Server processing)
In the server 2, the designation receiving means receives designation data representing the designated content of the target object from the terminal 1 (designation receiving process). At this time, regarding the reception of the designated content, the object image may be transferred from the terminal 1, or the data representing the search condition such as the character string or the attribute value is transferred, and the target is obtained by the search based on the transfer. You may specify. The example shown here is a case where designation is received as the latter search condition, and the designation receiving means is composed of an information storage means 30, a search request receiving means 55, and a search result acquiring means 60.

  In this case, in the server 2, when the search request receiving unit 55 receives a search request accompanied by specification of the search condition from the terminal 1 (step S 55. Search request receiving process), the search result acquiring unit 60 stores it in the information storage unit 30. The target information is searched based on the received search conditions from the search information that has been received, and information for matching the target object is acquired and passed to the matching means 70 (step S60, search result acquisition processing). In the server 2, the image receiving unit 65 sequentially receives the target images obtained by photographing the search target range from the terminal 1 (step S 65, image receiving process).

  Note that after receiving the search request, the search result may be transmitted once to the terminal 1 to allow the user to select or confirm, and the target image may be transmitted, or the search request and the target may be transmitted without such selection or confirmation. Images may be transmitted sequentially one after another.

  Then, the matching unit 70 sequentially and continuously detects a portion that matches the object represented by the designated data, that is, a portion that matches the matching information, from each of the target images that are successively received (step). S70: matching process). At the time of this detection, the type of matching technique used by the matching means 70 is arbitrary. For example, if a neural network technique is used, the matching information is not specified as a feature amount, and can be determined from an arbitrary position on the image. This pattern can also be detected.

  However, in the matching, it is particularly desirable that the matching means 70 uses a partial image matching algorithm (for example, Journal of Art and Science Vol. 6, No. 3, pp. 117-125). In this way, the processing load is reduced by applying the partial image matching algorithm that performs the matching operation by focusing on the feature amount, etc., so that the target can be searched in a wide range and with high accuracy even from high-capacity target images such as high resolution. However, the use of such a partial image matching algorithm is not essential and can be omitted.

  Then, the result transmission unit 75 detects, based on each of the target images, a display image that has been subjected to graphic processing processing that points out a portion that matches the object, or position information that points out the position of the matching portion. As a result, the data is sequentially and continuously transmitted to the terminal 1 (step S75, result transmission process).

[2-3. (Receiving and displaying the results)
In the terminal 1, the result receiving unit 80 sequentially receives, from the server 2, position information or display images that point out a portion that matches the target object that meets the search condition, based on the transmitted target image. Then (step S80, result reception processing), the display unit 85 sequentially displays the indication of the matched portion on the display unit D sequentially using the position information or the display image sequentially received. Perform (step S85, display processing).

  At the time of this display, the display means 85 is not essential, but it is desirable to display the matched portion relatively clearly with respect to its brightness, saturation, etc., and the remaining portion relatively unclearly. . FIG. 8 shows an example of such a display. Of the many books shown in the target image displayed on the display unit D in the example of FIG. 7, the book X “AB ...” that matches the search condition. Only is displayed brightly, and related information K such as the author name and publisher name illustrated in FIG. 3 is displayed.

  In this way, a display that makes it easy to grasp the target object at a glance can be realized by mask processing that makes the matched target object part conspicuous and the other part is dull or covered with a dark color. However, since such mask processing is not indispensable, for example, a graphic of another aspect such as surrounding the matched part with a line using position information indicating the matched part as the target object or blinking the part is used. You may perform the display which points out the matched part by highlighting by a display.

  Note that even if the position information (position coordinate range in the target image) of the matched part is sent from the server 2 to the terminal 1 as the detection result, the graphic display content indicating the part is created in the terminal 1 Alternatively, a display image that has been subjected to mask processing by graphic processing based on the target image may be transmitted and displayed on the terminal 1 as it is.

[2-4. (Repeated sequential processing)
Among the above (FIG. 2), processing from photographing to display of the peripheral image (steps S45 to S85 on the terminal 1 side) and processing from reception of the peripheral image to transmission of the detection result (step S65 on the server 2 side). To S75) are repeated at an appropriate time interval such as several frames / second to gradually change the position and orientation of the terminal 1 provided with the camera C, and the object from a display shelf or the like having a wider range than the image range of the camera. Can be matched and displayed.

  FIG. 9 shows an example. Books A, B, C, and D are arranged on a display shelf in the arrangement as shown in FIG. 9 (1), and only books A and D satisfy the search condition. It is assumed that the target object (shown by a solid line in FIG. 9A) and the other books B and C are non-objects (shown by a broken line in FIG. 9A). When the camera C of the terminal 1 is directed to the display shelf and the target image in the range shown on the left side of FIG. 9 (2) is sent to the server, only the book A, which is the object, is clearly displayed based on the matching result. The non-target book B is displayed unclearly (including non-display) by masking or the like (right side of FIG. 9 (2)).

  9 (3) shows that the camera has moved to the left by several frames (several frames) from this state. If the camera moves further to the left after this state, the object as shown on the left side of FIG. 9 (4). Based on the matching result, based on the matching result, as shown on the right side of FIG. 9 (4), the target books A and D are clearly displayed, and the non-target book C is unclearly displayed by mask processing or the like. It will be in the state.

[3. effect〕
As described above, in this embodiment, a target object such as a product image obtained in advance is detected by matching from a target image obtained by photographing a display shelf and the part is pointed out by graphic processing, so that the target object is displayed in the real world. It becomes possible to search easily and reliably. Further, in this embodiment, a series of processes from condition specification to matching can be quickly and easily performed by performing a search after searching for a target object corresponding to the search condition specified by the terminal and its matching information on the server. Thus, it becomes easy to find a desired product from a large number of unknown candidates.

  Further, in the present embodiment, the implementation mode of the client server system can effectively support the limited storage and calculation capabilities of the portable information terminal with those of the server, and the latest information is updated with the client by updating the information on the server. It can be used from a portable information device.

[4. Second Embodiment]
The above-described embodiment (referred to as “first embodiment”) has a configuration of a client server system. However, all the above processing is performed according to the storage and calculation capability of the portable information device. An implementation mode that is completed above is also conceivable (second embodiment). The configuration of the portable information device (terminal) 11 in the second embodiment is shown in FIG. That is, the terminal 11 is configured in accordance with the terminal 1 of the first embodiment, such as the input means T, the photographing means C, the information display section D, and the arithmetic control section 5 and the storage device 6 that are the configuration of a computer that performs control. However, the communication means 7 is not essential.

  In this terminal 11, as shown in the flowchart of FIG. 11, the designation receiving unit 37 receives the designation of the object from the input unit T (Step S <b> 37, designation receiving process). Further, when the image acquisition unit 47 acquires the target image by successively capturing the search target range with the camera C (step S47, image acquisition processing), the matching unit 72 acquires each of the acquired target images. From this, the portions that match the object are successively detected successively (step S72, matching processing). Then, the display means 87 displays each of the target images on the display unit D by performing graphic processing processing that points out the matched portion sequentially (step S87, display processing).

  As described above, in the second embodiment, which is an implementation mode of a stand-alone device, in addition to the operations and effects described for the first embodiment, communication facilities and costs are unnecessary, the configuration is simple, and reliability is improved. The In addition, the response time can be improved by reducing the communication time.

  In the second embodiment as described above, it is not indispensable to receive the search condition and search for and acquire the target object and its matching information from the search information. However, when performing such a search, Information storage means for preliminarily storing search information including collation information for each object is provided, the designation receiving means receives the designation of the target object from the input means, and the search result acquisition means is stored in the information storage means. The target information is searched from the received search information based on the received search condition, and the collation information of the target object is acquired. And a matching means detects the part which matches the information for collation from a target image.

[5. Third Embodiment]
In addition, when taking the configuration of the client server system as shown in the first embodiment, which function is arranged on which side can be flexibly changed. The image matching may be performed on the client (terminal) side (third embodiment). FIG. 12 shows the configuration of the third embodiment, and FIG. 13 shows the processing procedure.

  In the third embodiment, the hardware configuration of the operation control unit of the terminal 13 and the server 23, the reception of the search condition designation at the terminal, the transmission of the search request accompanied by it, the search request at the server The reception of information and the acquisition of verification information are in accordance with the first embodiment (FIG. 1). However, in the third embodiment, the information transmitting unit 76 transmits the acquired verification information to the terminal 13 (step S76, information return processing), and in the terminal 13, the information receiving unit 81 transmits in step S40. Information for collation corresponding to the search condition is received from the server 23 (step S81, information reception process).

  And the image acquisition means 45 acquires a target image by image | photographing a search object range sequentially with the camera C (step S45. Image acquisition process), and the matching means 71 was acquired sequentially sequentially. From each of the target images, a portion that matches the received verification information is successively detected (step S71, matching process), and the display unit 85 sequentially matches the target images with the match. A graphic processing process for pointing out the portion is performed and displayed on the display unit D (step S85, display process).

  In this way, the search is performed on the server side, but the image matching is performed on the terminal side, so that it is not necessary to send a relatively large-capacity image from the terminal to the server for matching. Packet charges can be saved.

[6. Other embodiments]
In addition, said each embodiment is only an illustration, and this invention includes what is illustrated below and other embodiment other than that. For example, in the first embodiment, an example including a search based on a designated search condition is shown. However, since the search is not essential, the client server system does not include a search as shown in the second embodiment. It is also possible to constitute the present invention.

  Moreover, although application to a book was illustrated in each said embodiment, this invention is applicable not only to a book but the display shelf of various articles, such as CD jacket and snacks. Also, each configuration diagram, data diagram, flowchart, conceptual diagram, screen display example diagram, and the like are merely examples, and the presence / absence of each element, its order, and specific contents can be changed as appropriate. In the present invention, elements such as each means may be realized by an electronic circuit based on a wired logic or the like without being limited to an arithmetic control unit of a computer.

N Communication network 1 Portable information device (terminal)
2 Servers 5, 15, 25 Arithmetic control units 6, 16, 26 Storage devices 7, 17, 27 Communication means 30 Information storage means 35 Search reception means 60 Search result acquisition means 70 Matching means 85 Display means

Claims (14)

An input unit, an imaging unit, an information display unit, a communication unit that communicates with a server via a communication network, and a computer that performs control, and continuously captures images according to the movement of the imaging unit; A portable information retrieval device for displaying,
Target designation means for accepting designation of an object from the input means and transmitting to the server as designation data representing the designation content;
Image acquisition means for acquiring a target image by capturing a search target range by the imaging means successively in succession;
Image transmitting means for sequentially and sequentially transmitting the acquired target images to the server;
Based on the transmitted target image, result receiving means for sequentially receiving, from the server, position information or a display image indicating a portion matching the object;
Using the position information or the display image sequentially received successively, a display means for sequentially displaying on the front display unit the indication of the matched portion;
A portable information retrieval device characterized by realizing a computer. The target specifying means includes
Search accepting means for accepting designation of search conditions from the input means;
Request transmitting means for transmitting a search request with specification of the accepted search condition to the server;
The portable information retrieval device according to claim 1, further comprising:   3. The portable information retrieval device according to claim 1, wherein the display unit displays the matched portion relatively clearly and the remaining portion relatively unclear. A server comprising a communication means for communicating with a portable information retrieval device via a communication network,
Designation receiving means for receiving designated data representing the designated content of the object from the portable information retrieval device;
Image receiving means for sequentially receiving a target image obtained by photographing a search target range from the portable information search device;
Matching means for successively and successively detecting a portion that matches the object represented by the designated data from each of the target images that are sequentially received.
Based on each of the target images, a display image that has been subjected to graphic processing for pointing out a portion that matches the object, or position information that points out the position of the matching portion is sequentially and continuously detected as a detection result. A result transmitting means for transmitting to the information retrieval device;
A server characterized in that a computer is realized. The designation receiving means is
Information storage means storing search information including collation information for each object;
Search request receiving means for receiving a search request accompanied by specification of search conditions from the portable information search device;
Search result acquisition means for searching for an object from the search information stored in the information storage means on the basis of the received search condition and acquiring information for collation of the object and passing it to the matching means When,
5. The server according to claim 4, further comprising:   The server according to claim 4 or 5, wherein the matching means uses a partial image matching algorithm. The portable information search device according to any one of claims 1 to 3,
A server according to any one of claims 4 to 6;
A search system comprising: A portable information retrieval device comprising an input means, a photographing means, an information display unit, and a computer for controlling,
Designation accepting means for accepting designation of an object from the input means;
Image acquisition means for acquiring a target image by capturing a search target range by the imaging means successively in succession;
Matching means for successively and successively detecting a portion matching the object from each acquired target image;
Display means for displaying each target image sequentially and continuously on the display unit by performing graphic processing for pointing out the matched portion;
A portable information retrieval device characterized by realizing a computer. An input unit, an imaging unit, an information display unit, a communication unit that communicates with a server via a communication network, and a computer that performs control, and continuously captures images according to the movement of the imaging unit; A portable information retrieval device for displaying,
Search accepting means for accepting designation of search conditions from the input means;
Request transmitting means for transmitting a search request with specification of the accepted search condition to the server;
Information receiving means for receiving information for verification corresponding to the transmitted search condition from the server;
Image acquisition means for acquiring a target image by capturing a search target range by the imaging means successively in succession;
From each of the target images acquired sequentially sequentially, matching means for sequentially detecting a portion that matches the received verification information;
Display means for displaying each target image sequentially and continuously on the display unit by performing graphic processing for pointing out the matched portion;
A portable information retrieval device characterized by realizing a computer. A server comprising a communication means for communicating with a portable information retrieval device via a communication network,
Information storage means storing search information including collation information for each object;
Search request receiving means for receiving a search request accompanied by specification of search conditions from the portable information search device;
Search means for searching for a target object based on the received search condition and acquiring collation information for the target object from the search information stored in the information storage means;
Information transmitting means for transmitting the acquired verification information to the portable information retrieval device;
A server characterized in that a computer is realized. An input unit, an imaging unit, an information display unit, a communication unit that communicates with a server via a communication network, and a computer that performs control, and continuously captures images according to the movement of the imaging unit; An information processing method using a portable information retrieval device to display,
A target designation process for receiving designation of the object from the input means and transmitting the designation data representing the designation content to the server;
Image acquisition processing for acquiring a target image by capturing a search target range with the imaging unit sequentially and continuously,
Image transmission processing for sequentially and successively transmitting the acquired target images to the server;
Based on the transmitted target image, a result reception process for sequentially receiving position information indicating a portion matching the object or a display image sequentially from the server;
Using the position information or display image sequentially received successively, a display process for sequentially displaying on the display unit the indication of the matched portion;
An information processing method characterized by realizing a computer. An information processing method using a server provided with a communication means for communicating with a portable information retrieval device via a communication network,
A designation receiving process for receiving designated data representing the designated content of the object from the portable information retrieval device;
Image receiving processing for sequentially receiving the target images obtained by capturing the search target range from the portable information search device;
From each of the target images received sequentially and sequentially, a matching process that sequentially and continuously detects a portion that matches the object represented by the designated data;
Based on each of the target images, a display image that has been subjected to graphic processing for pointing out a portion that matches the object, or position information that points out the position of the matching portion is sequentially and continuously detected as a detection result. A result transmission process to be transmitted to the information retrieval device;
An information processing method characterized by realizing a computer. An information search method using a portable information search device comprising an input means, an imaging means, an information display unit, and a computer for control,
A designation acceptance process for accepting designation of an object from the input means;
Image acquisition processing for acquiring a target image by capturing a search target range with the imaging unit sequentially and continuously,
From each acquired target image, a matching process that sequentially and successively detects a portion that matches the object;
Display processing for displaying each target image sequentially and continuously on the display unit by performing graphic processing processing for pointing out the matched portion;
An information retrieval method characterized in that a computer is realized. An input unit, an imaging unit, an information display unit, a communication unit that communicates with a server via a communication network, and a computer that performs control, and continuously captures images according to the movement of the imaging unit; A computer program for controlling a portable information retrieval device to be displayed,
The program controls the computer,
A target designation process for receiving designation of the object from the input means and transmitting the designation data representing the designation content to the server;
Image acquisition processing for acquiring a target image by capturing a search target range with the imaging unit sequentially and continuously,
Image transmission processing for sequentially and successively transmitting the acquired target images to the server;
Based on the transmitted target image, a result reception process for sequentially receiving position information indicating a portion matching the object or a display image sequentially from the server;
Using the position information or display image sequentially received successively, a display process for sequentially displaying on the display unit the indication of the matched portion;
A computer program for executing

Images