JP2015130631A - Information processing apparatus and electronic apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus and an electronic apparatus capable of providing a user with information on a search object.SOLUTION: The information processing apparatus comprises: an input unit (301) which receives an image from an electronic apparatus (10A) which can be mounted on a user; a detection unit (314) which detects whether the image includes the search object; and an informing unit (315) which gives information on the search object.

Description

  The present invention relates to an information processing apparatus and an electronic apparatus.

  Conventionally, a system that estimates and tracks the line of sight of a subject (user) wearing glasses capable of detecting the line of sight of a person is known (see, for example, Patent Document 1).

JP 2013-81762 A

  However, in the above-mentioned Patent Document 1, the user's attention to the product etc. is only evaluated from the user's line of sight, for example, information that is in the user's field of view but is not noticed or overlooked by the user There is no consideration for.

  The present invention has been made in view of the above problems, and an object thereof is to provide an information processing apparatus and an electronic apparatus that can notify a user of information related to a search target.

  An information processing apparatus (30) of the present invention includes an input unit (301) for inputting an image from an electronic device (10A) that can be worn by a user, and detection for detecting whether or not a search target exists in the image. A section (314) and a notification section (315) for performing notification regarding the search object.

  In this case, the input unit may input images from a plurality of electronic devices (10A, 10B), and the detection unit may detect the search object from the plurality of images input from the plurality of electronic devices. Good. Further, the information processing apparatus of the present invention may include a determination unit (313) that determines the search object.

  Moreover, in this invention, the said alerting | reporting part is good also as alerting | reporting the said search target object to the user different from the said user, good also as notifying the positional information regarding the said search target object, and the said search. Information on an article that is related to the object and is different from the search object may be notified.

  The electronic device (10C) of the present invention can be worn by a user, and includes an imaging unit (101) that captures an image, and a detection unit (114) that detects whether or not a search target exists in the image. And a notifying unit (115) for notifying the search target object according to the detection result.

  In this case, the notification unit may notify a user different from the user of the search target object. In the present invention, the electronic device may include a determination unit (113) that determines the search object.

  Moreover, the electronic device of the present invention may include a registration unit (112) for registering the search target, and the registration unit may register the search target based on information captured by the imaging unit. The registration unit may register an article obtained by the user as the search object.

  Further, in the present invention, the detection unit starts detecting whether or not the determined search object exists in the image after the determination unit determines the search object, The latest information among the information on the search object may be notified.

  In addition, the electronic apparatus of the present invention includes a display unit (102) that displays the image, and the notification unit is configured such that when the search target is present in an image that the imaging unit is currently capturing, Information indicating the presence of the search object may be displayed on the display unit. In the present invention, the notification unit may display the image when the search object is detected to be present in the image.

  In addition, in order to explain the present invention in an easy-to-understand manner, the above description has been made in association with the reference numerals of the drawings representing one embodiment. However, the present invention is not limited to this, and the configuration of an embodiment described later May be modified as appropriate, or at least a part thereof may be replaced with another component. Further, the configuration requirements that are not particularly limited with respect to the arrangement are not limited to the arrangement disclosed in the embodiment, and can be arranged at a position where the function can be achieved.

  The information processing apparatus and the electronic apparatus according to the present invention have an effect of being able to notify the user of information related to the search target.

It is a block diagram which shows the structure of the information processing system which concerns on 1st Embodiment. It is a perspective view which shows an example of a wearable apparatus. It is a figure which shows an example of a search target object information table. It is a flowchart which shows an example of the process which the control part of the information processing apparatus which concerns on 1st Embodiment performs. FIG. 5A to FIG. 5C are diagrams for explaining the processing of FIG. It is a flowchart which shows an example of the search target information storage process which the control part of the information processing apparatus which concerns on 1st Embodiment performs. FIG. 7A and FIG. 7B are diagrams for explaining the processing of FIG. It is a flowchart which shows an example of the search target object detection process which the control part of the information processing apparatus which concerns on 1st Embodiment performs. It is a figure for demonstrating the process of FIG. It is a block diagram which shows the structure of the information processing system which concerns on 2nd Embodiment. It is a flowchart which shows an example of the search object registration process which the control part of the wearable apparatus which concerns on 2nd Embodiment performs. It is a flowchart which shows an example of the search target object detection process which the control part of the wearable apparatus which concerns on 2nd Embodiment performs. It is a figure which shows the case where a search target object is a text.

<< First Embodiment >>
Hereinafter, the first embodiment will be described in detail with reference to FIGS. FIG. 1 is a block diagram showing the configuration of an information processing system 500A according to the first embodiment. The information processing system 500A is a system that supports a user to find a search object.

  As illustrated in FIG. 1, the information processing system 500A includes (uses) wearable devices 10A and 10B and the information processing apparatus 30. The wearable devices 10A and 10B and the information processing apparatus 30 are connected to the network 50. The network 50 includes, for example, a public wireless LAN (Local Area Network), the Internet, a mobile phone line network, and the like.

(Wearable device 10A, 10B)
The wearable devices 10A and 10B are eyeglass-type terminals worn by the user on the body (face). In the present embodiment, it is assumed that the father uses the wearable device 10A and the mother uses the wearable device 10B. In the present embodiment, since the wearable devices 10A and 10B have the same configuration, the wearable devices 10A and 10B will be described as the wearable device 10 and described below unless otherwise distinguished.

  As shown in FIG. 1, the wearable device 10 includes an imaging unit 101, a display unit 102, an operation unit 103, a microphone 104, a position detection unit 105, a speaker 106, a storage unit 107, and a communication unit 108. And a control unit 109. 2 shows the wearable device 10 in a perspective view. As shown in FIG. 2, the wearable device 10 includes a glasses-type frame 120. The configuration of the wearable device 10 illustrated in FIG. 1 and not illustrated in FIG. 2 is provided inside the frame 120 or a part of the frame 120.

  The imaging unit 101 includes a lens, an imaging element, an image processing unit, and the like, and captures still images and moving images. As shown in FIG. 2, the imaging unit 101 is provided near the end of the frame 120 (near the user's right eye). For this reason, in a state where the user wears the wearable device 10, an image in the direction in which the user is facing (looking at) can be taken.

  The display unit 102 includes a projector provided in or near the frame 120 and a prism for guiding a projection image from the projector to the eyes of the user. The display unit 102 displays various information under the control of the control unit 109.

  The operation unit 103 is a touch pad provided on the frame 120, detects the movement of the user's finger, receives an operation from the user, and transmits the received operation information to the control unit 109. Details of the imaging unit 101, the display unit 102, the operation unit 103, and the like are also disclosed in, for example, US Published Patent No. 2013/0044042.

  The microphone 104 is provided on the frame 120 and collects voices uttered by the user. The voice collected by the microphone 104 is voice-recognized by a voice recognition unit (not shown), and the voice recognition result is transmitted to the control unit 109. The control unit 109 executes processing based on the voice recognition result (for example, command execution processing). Note that the control unit 109 may perform voice recognition.

  The position detection unit 105 includes, for example, a GPS (Global Positioning System) sensor, detects the position of the wearable device 10, and outputs position information (latitude, longitude) of the wearable device 10 to the control unit 109. . Instead, the positional information (latitude and longitude) is associated with the map information and the location information and stored in the storage unit 107, and the control unit 109 is based on the positional information (latitude and longitude), for example, a company, home (living room) , Bedroom) may be provided.

  The speaker 106 is an audio output device that is provided in the frame 120 and outputs audio under the control of the control unit 109, for example. As the speaker 106, in addition to the earphone and the headphone, a speaker having directivity and capable of providing audio information mainly toward the ear of the user wearing the wearable device 10 can be employed. .

  The storage unit 107 is, for example, a nonvolatile semiconductor memory such as a flash memory, and stores image data captured by the imaging unit 101, map data, location data, display data displayed on the display unit 102, and the like. Remember.

  The communication unit 108 communicates with the information processing apparatus 30 via the network 50. In the first embodiment, the communication unit 108 transmits the image captured by the imaging unit 101 to the information processing apparatus 30 together with the position information detected by the position detection unit 105 under the control of the control unit 109, or the information An image including an article that the user is looking for (hereinafter referred to as a search object) is received from the processing device 30.

  The control unit 109 controls the wearable device 10 as a whole. The control unit 109 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The control unit 109 transmits or receives information necessary for the user to find the search target object to the information processing device 30 via the communication unit 108. For example, the control unit 109 transmits the image captured by the imaging unit 101 to the information processing apparatus 30 through the communication unit 108 together with the position information detected by the position detection unit 105. In addition, the control unit 109 transmits information regarding an instruction received from the user via the operation unit 103 or the microphone 104 to the information processing apparatus 30. In addition, the control unit 109 receives information on the search target from the information processing apparatus 30 and displays the information on the display unit 102 or outputs sound from the speaker 106 to notify the search target.

(Information processing apparatus 30)
The information processing apparatus 30 is, for example, a server, and includes a communication unit 301, a control unit 302, and a storage unit 303 as illustrated in FIG.

  The communication unit 301 performs wireless communication or wired communication with the wearable device 10 via the network 50. In the first embodiment, the communication unit 301 receives information necessary for the user to find a search target from the wearable device 10 or transmits the information to the wearable device 10. For example, the communication unit 301 receives an image captured by the imaging unit 101 of the wearable device 10 from the wearable device 10 and outputs the image to the control unit 302. In addition, the communication unit 301 transmits image data stored in a storage unit 303 described later to the wearable device 10 under the control of the control unit 302.

  The storage unit 303 is a hard disk drive, for example, and stores information related to the search target. In the first embodiment, the storage unit 303 stores a search object information table shown in FIG. The search object information table includes items of article name, registered user, reference image, and history image. In order to register data in the search object information table, for example, an article to be registered by the user is imaged by the imaging unit 101 of the wearable device 10, and the name of the imaged article is spoken. The microphone 104 collects the sound, and the control unit 109 only has to transmit the article name recognized by a sound recognition unit (not shown) and the image captured by the imaging unit 101 to the information processing apparatus 30. The information processing apparatus 30 registers the received article name in the article name item, and registers the image captured by the imaging unit 101 in the reference image item. Thereby, the user can perform registration operation hands-free. Instead of this, registration of data in the search object information table is performed using a user interface such as a multifunctional mobile device (smartphone), an imaging unit of a tablet terminal, a touch panel, or the like, and using the network 50. You may make it transmit this registration information to the processing apparatus 30. FIG.

  Note that when the image recognition unit is provided in the information processing apparatus 30, the input of the article name by the user can be omitted. In this case, an article category (eg, key) may be registered in the search object information table instead of the article name (eg, house key). In this case, the user may take an image of the article to be registered, touch the registration button, or utter the word registration. For the image recognition by the image recognition unit, a known feature amount detection algorithm such as SURF (Speeded Up Robust Features) or SIFT (Scale Invariant Feature Transform) may be used as the feature points of the image data.

  The user who registered the article name is registered in the item of registered user. In the item of the reference image, an image used when the detection unit 314 described later detects whether or not a search target is included in an image captured by the imaging unit 101 of the wearable device 10 is stored. . In the history image item, an image captured by the imaging unit 101 of the wearable device 10 is stored in time series together with information on the imaging date and time and the imaging position.

  The control unit 302 controls the information processing apparatus 30 as a whole. The control unit 302 includes a CPU, RAM, ROM, and the like. In the control unit 302, the CPU functions as the information storage unit 311, the registration unit 312, the determination unit 313, the detection unit 314, and the notification unit 315 illustrated in FIG.

  The information storage unit 311 inputs an image received from the wearable device 10 by the communication unit 301. The information storage unit 311 stores an image including an article registered in the search object information table among the input images in the search object information table (see FIG. 3) of the storage unit 303 as a history image of the article. To do. Details of processing executed by the information storage unit 311 will be described later.

  The registration unit 312 registers the article in the search target information table of the storage unit 303.

  The determination unit 313 determines an article to be searched from the articles registered in the storage unit 303, that is, a search target, and outputs the determined article to the detection unit 314.

  The detection unit 314 detects whether or not the search target determined by the determination unit 313 is present in the image captured by the imaging unit 101 of the wearable device 10, and outputs the detection result to the notification unit 315.

  The alerting | reporting part 315 alert | reports regarding a search target object based on a detection result. For example, the notification unit 315 transmits the search target object detection result to the wearable device 10 via the communication unit 301. The control unit 109 of the wearable device 10 that has received the detection result from the notification unit 315 notifies the user of information about the search target using the display unit 102, the speaker 106, and the like.

(Processing of information processing apparatus 30)
Next, processing executed by the control unit 302 of the information processing apparatus 30 will be described with reference to the flowchart of FIG. In the present embodiment, the process illustrated in FIG. 4 starts when the user turns on the wearable device 10 and communication is established between the wearable device 10 and the information processing apparatus 30.

  In the process of FIG. 4, first, the determination unit 313 determines whether or not a search request for a search target has been received from the wearable device 10 (step S <b> 10). For example, when the father pronounces “search object” toward the microphone 104 of the wearable device 10 </ b> A, a search request for the search object is transmitted from the wearable device 10 </ b> A to the information processing apparatus 30. When the determination unit 313 receives the search request from the wearable device 10, the determination unit 313 proceeds to step S40. The search request for the search object may be transmitted, for example, when a predetermined operation (for example, a tap) is performed on the operation unit 103 by the user.

  On the other hand, when the determining unit 313 has not received a search request for the search target from the wearable device 10, the registration unit 312 and the information storage unit 311 are connected to the imaging unit from the wearable device 10 via the communication unit 301. The image captured by 101 is input (step S11). At this time, the information storage unit 311 also receives the position information detected by the position detection unit 105 of the wearable device 10 from the wearable device 10. Note that the registration unit 312 and the information storage unit 311 receive images from the wearable devices 10A and 10B, respectively. The timing for inputting images from the wearable devices 10A and 10B may be the same or different.

  Next, the registration unit 312 analyzes the image input in step S11 using the above-described image recognition unit, and determines whether or not the user is holding an article in the image (step S13). If the determination here is negative, the information storage unit 311 executes the search target information storage process of step S20.

  On the other hand, as a result of analyzing the image as shown in FIG. 5A, the registration unit 312 determines that the user is holding an article (step S13 / YES). It is determined whether or not it is registered in the search object information table of the storage unit 303 (step S15). For example, when the image shown in FIG. 5A is input in step S11, the registration unit 312 extracts the key K10 image, the extracted key K10 image, and the search target information table (see FIG. 3). Pattern matching is performed with the reference image, and it is determined whether or not the key K10 held by the user is already registered in the search object information table.

  When the article that the user is holding is already registered in the search target information table (step S15 / YES), the information storage unit 311 executes the search target information storage process of step S20.

  On the other hand, when the article held by the user is not registered in the search target information table (step S15 / NO), the registration unit 312 registers the article held by the user in the search target information table ( Step S17). For example, as shown in FIG. 5A, it is assumed that the father holds the key K10 in the image input from the wearable device 10A used by the father. In this case, the registration unit 312 registers the image of the key K10 extracted in the process of step S13 as a reference image in the search object information table, and sets the father who is the user of the wearable device 10A as the registered user item. Register (see FIG. 5C). Also, the registration unit 312 performs matching between the template images of various articles stored in advance in the storage unit 303 and the reference image, identifies the article name of the extracted image, and registers it in the item name item ( In the example of FIG. 5C, “key”). It is assumed that the name of each article is associated with the template image of each article. At this time, as illustrated in FIG. 5B, the registration unit 312 may register an image obtained by interpolating an image of a portion hidden by the hand as a reference image using the above-described image recognition unit.

  After the process of step S17 is complete | finished, it returns to step S10. In the flowchart of FIG. 4, the search target is registered on the condition that the user is holding the article. However, the registration process may be executed when the user is watching the article. In this case, the imaging time by the imaging unit 101 lasts for 5 seconds or more, the article having the largest area captured by the imaging unit 101, or a predetermined size (an area of 1/5 or more of the imaging region) is present. You may register the article which was done as search object. Alternatively, the registration process may be executed by touching a registration button after an image of an article that the user wants to register is captured.

(Search object information accumulation process)
Next, details of the search target information accumulation process in step S20 of FIG. 4 will be described with reference to the flowchart of FIG. The article registered as the search target by the user tends to be forgotten by the user, so in this flowchart, the imaging history (imaging date, imaging position, imaging location, etc.) of the search target article is updated. Note that the information storage unit 311 may end this flowchart when it is recognized that there is no change in the imaging history.

  In the process of FIG. 6, the information storage unit 311 determines whether or not an article registered in the search target information table exists in the image input in step S11 of FIG. 4 (step S23). For example, the information storage unit 311 performs pattern matching between the reference image registered in the item of the reference image of the search target information table and the input image, and whether the search target article exists in the image. Judge whether or not. If the determination here is negative, the information storage unit 311 ends all the processes in FIG. 6 and returns to step S10 in FIG.

  On the other hand, when the search target article exists in the image (step S23 / YES), the information storage unit 311 displays the image input in step S11 of FIG. 4 together with the date / time information and the position information when the image is captured. For example, primary storage is performed in the RAM (step S25).

  Next, the information storage unit 311 again inputs an image from the wearable device 10 (step S27), and determines whether or not a search target article exists in the input image, similarly to step S23 (step S23). S29). If there is an article to be searched, the information storage unit 311 returns to step S25, and stores the information temporarily stored in the RAM, the image input in step S27, the date / time information and the position at which the image was captured. Overwrite with information.

  On the other hand, when the search target article does not exist in the input image (step S29 / NO), the information storage unit 311 registers the image temporarily stored in the RAM in the history image item of the search target information table. After (step S31), all the processes in FIG. 6 are terminated, and the process returns to step S10 in FIG. For example, as shown in FIG. 7A, it is assumed that a key K10, which is an article to be searched for by the father, exists in the image input from the wearable device 10B of the mother. In this case, the information storage unit 311 registers the image (FIG. 7A) input from the wearable device 10 </ b> B together with the date / time information and the position information of the captured image in the “key” history image item. As described above, in the history image item of the search target information table, in addition to the image captured by the imaging unit 101 of the wearable device 10 of the registered user (for example, father), a user (for example, mother) that is different from the registered user. When the imaging unit 101 of the wearable device 10 images a search target article, an image when the search target article is last imaged is registered as a history image.

  Of the images including the search target imaged by the imaging unit 101 of each user's wearable device 10 by the above-described processing, the image that is temporarily stored in the RAM or the like at the end of the processing of steps S25 to S29 being repeated. Is stored in the storage unit 303 as a history image.

(Search object detection process)
Next, the search target object detection process in step S40 of FIG. 4 will be described in detail with reference to the flowchart of FIG. In the following description, a case where the father searches for the key K10 will be described.

  In the process of FIG. 8, the determination unit 313 first determines a search object from the search object information table (step S41). For example, when receiving a search request for a search target from the wearable device 10A, the determination unit 313 is a user of the wearable device 10A in the registered user item in the search target information table stored in the storage unit 303. A list of article names in which “father” is registered or a list of reference images is transmitted to the wearable device 10A. If there is no search target article registered by the father, the determination unit 313 may refer to the search target article registered by the mother. In this way, articles to be searched that are registered with family members, friends, or specific groups may be shared. The control unit 109 of the wearable device 10 </ b> A displays the list of article names received from the information processing apparatus 30 on the display unit 102. For example, when the user selects a search target using the operation unit 103, the control unit 109 transmits information on the search target selected by the user to the information processing apparatus 30 via the communication unit 108. Note that the user may designate the search target by generating the name “key” of the search target article toward the microphone 104 of the wearable device 10. In this case, the determination unit 313 outputs key information stored in the search object information table such as a house key or a car key to the detection unit 314 as a search object.

  Next, the detection unit 314 inputs an image currently captured by the imaging unit 101 of the wearable device 10 (step S43). At this time, the detection unit 314 inputs images from the wearable devices 10A and 10B, respectively. The timing for inputting images from the wearable devices 10A and 10B may be the same or different.

  Subsequently, the detection unit 314 determines whether or not the search object input from the determination unit 313 exists in the image input in step S43 (step S45). For example, the detection unit 314 performs pattern matching between the image input in step S43 and the reference image of the key K10 (see FIG. 5B), and the key is included in the image captured by the imaging unit 101. It is determined whether or not K10 exists.

  When a search target exists in the image input in step S43 (step S45 / YES), the detection unit 314 is a wearable type of a client (here, a father) who requested the detection of the search target. It is determined whether or not the image is input from the device 10A (step S47).

  When the image in which the search target exists is an image input from the wearable device 10A of the client who requested the detection of the search target (here, father) (step S47 / YES), the notification unit 315 The wearable device 10A is notified of the position of the search target in the image (step S49), thereby notifying the search target. The control unit 109 of the wearable device 10A that has received the position of the search target object from the information processing apparatus 30 displays the image currently captured by the image capturing unit 101 on the display unit 102 and information indicating the search target object ( For example, an arrow, a marker, or the like (see the marker 330 in FIG. 9A) is displayed. Note that information (marker 330) indicating the search target may be displayed on the display unit 102 so as to be superimposed on the user's field of view without displaying the image currently captured by the imaging unit 101. Thereby, the user (father) of the wearable device 10 </ b> A can notice the search object that is in the field of view but has been overlooked.

  On the other hand, when the image in which the search target exists is, for example, an image input from the mother's wearable device 10B, the determination in step S47 is denied. In this case, the notification unit 315 notifies other users (for example, a mother using the wearable device 10B) of the wearable device 10 that has input an image in which the search object is present (notice about the search object) ( Step S51). For example, when the image (see FIG. 9B) in which the search target (key) exists is input from the wearable device 10B, the notification unit 315 searches for the search target (key). And the position of the search object in the image currently captured by the imaging unit 101 of the wearable device 10B is notified to the wearable device 10B. The control unit 109 of the wearable device 10B that has received the information displays information (for example, an arrow or a marker) indicating the search target on the display unit 102, and the father is searching for the search target. Is displayed on the display unit 102 or notified to the mother by voice using the speaker 106. As a result, the mother can know that the father is searching for the key K10, and can find a search object (key K10) in his field of view and take a response such as passing it to the father. it can.

  By the way, when the search object does not exist in the image input in step S43 (step S45 / NO), has the predetermined time (for example, 10 seconds) elapsed since the detection unit 314 started the process of step S43? It is determined whether or not (step S53).

  When the predetermined time has not elapsed, the detection unit 314 executes the processing from step S43 again. However, when the predetermined time has elapsed, the search target information table stored in the storage unit 303 searches the search target information table. It is determined whether or not a history image is registered for the object (step S55).

  When a history image is registered for the search target (step S55 / YES), the notification unit 315 transmits an image having the latest imaging date and time among the history images to the wearable device 10A (step S57). . The control unit 109 of the wearable device 10 </ b> A displays the image received from the information processing apparatus 30 on the display unit 102. Thereby, it is possible to notify the father of the place where the search object (key K10) lastly existed. The father can know where the wearable device 10 last imaged the search object (key K10), and can take measures such as changing the search place for the search object. At this time, the notification unit 315 may notify the position information (latitude, longitude) when the search object is imaged last. This makes it easier for the user to find the search object.

  On the other hand, when the history image is not registered for the search target (step S55 / NO), the notification unit 315 notifies the wearable device 10A that the search target cannot be detected (step S59).

  Subsequently, the determination unit 313 determines whether or not there is an instruction from the wearable device 10A to end detection of the search target (step S61). For example, when the father inputs an instruction to end detection of the search object using the operation unit 103 of the wearable apparatus 10A, the wearable apparatus 10A transmits an instruction to end detection of the search object to the information processing device 30. To do. When the determination unit 313 receives the instruction from the wearable device 10A, the determination unit 313 determines that an instruction to end detection of the search target has been received (step S61 / YES), ends all the processes in FIG. Return to step S10. On the other hand, when the instruction has not been received, the detection unit 314 executes the processing from step S43 again. Note that the processing in FIG. 4 is forcibly terminated when a predetermined command is input by voice or operation of the operation unit 103 or when the power of the wearable device 10 is turned off. In the flowchart of FIG. 8, the order of each step may be changed, for example, step S55 is performed prior to step S43.

  As described above in detail, according to the first embodiment, the information processing apparatus 30 inputs an image from the wearable device 10 that can be worn by the user, and whether a search target exists in the image. Since the detection unit 314 that detects whether or not and a notification unit 315 that performs notification regarding the search target are provided, even if the user overlooks the search target, it is input from the wearable device 10 without particular awareness. If there is a search object in the obtained image, the presence of the search object can be notified to the user.

  Further, according to the first embodiment, the detection unit 314 receives images from the plurality of wearable devices 10 and detects a search target from the plurality of images input from the plurality of wearable devices 10. Thereby, even when the search target is imaged by the wearable device 10 worn by a user different from the user who is searching for the search target, notification regarding the search target can be performed.

  Further, according to the first embodiment, the notification unit 315 includes the search target in the image currently captured by the wearable device 10 worn by a user different from the user who is searching for the search target. In doing so, the user is notified of the search object. Accordingly, it is possible to notify the user of the wearable device 10 that is capturing an image including the search target that the search target of another user exists in the image.

  In the first embodiment, the notification unit 315 performs notification related to the search target specified by the user. However, the notification unit 315 notifies information related to the search target that is different from the search target. Also good. For example, it is assumed that the search target specified by the user is a “key” and an image in which the key is hidden is captured. In this case, the determination unit 313 determines “鞄” instead of the “key” designated by the user as a search target based on the image, and the detection unit 314 has a wrinkle in the image captured by the wearable device 10. Detect if it exists. Based on the detection result of the detection unit 314, the notification unit 315 transmits information related to wrinkles (position information of wrinkles and images in which wrinkles are present) to the wearable device 10. The wearable device 10 displays information about the bag on the display unit 102 based on the information received from the information processing device 30. Thereby, even when the search object (key) is hidden or stored in another article (bag), information related to the search object can be provided to the user.

  In the first embodiment, the detection unit 314 inputs images from a plurality of wearable devices 10 and detects a search object from the plurality of images. However, the detection unit 314 inputs from one wearable device 10. The search object may be detected from the obtained image. In addition, the registration unit 312 has registered the search target article based on inputs from the plurality of wearable devices 10, but registers the search target article based on inputs from one wearable device 10. It may be. Further, the first embodiment may be configured by at least one wearable device 10 and the information processing apparatus 30.

  In the first embodiment, images are stored in the history image item of the search target information table (see FIG. 3) in time series. However, the latest image may always be stored. . In addition, for example, when a predetermined number (for example, 10) of history images are stored for each search target article, the oldest history image may be overwritten with the latest history image ( FIFO: First In, First Out). In the case of storing a plurality of history images, the notification unit 315 displays the history images on the display unit 102 of the wearable device 10 in time series, so that the search target is current in what manner. You may alert | report to a user whether it was moved to the position.

  In the first embodiment, any of the wearable devices 10A and 10B may have the function of the information processing apparatus 30 described above. In this case, for example, the wearable device 10A receives information to be registered in the search object information table from the wearable device 10B via the network 50, or transmits information related to the search object to the wearable device 10B. Good. Further, for example, when the wearable device 10A has the functions of the information processing apparatus 30 described above, a series of items for registering articles in the search target information table, accumulating search target information, and detecting search targets. The processing can be performed by the wearable device 10A alone.

<< Second Embodiment >>
Next, a second embodiment will be described. In the second embodiment, the wearable device has the function of the information processing apparatus 30 of the first embodiment.

  FIG. 10 is a block diagram illustrating the configuration of an information processing system 500B according to the second embodiment. As illustrated in FIG. 10, the information processing system 500 </ b> B includes a wearable device 10 </ b> C and an article-side device 40.

(Article side device 40)
The article-side device 40 is attached to an article that can be a search target, such as an article possessed by a user. The article-side device 40 includes a storage unit 401 and a communication unit 402.

  The storage unit 401 is, for example, a non-volatile semiconductor memory, and the article name of the article to which the article-side device 40 is attached and the article used when detecting whether or not the article exists in the image. The reference image is stored.

  The communication unit 402 communicates with other devices, and uses, for example, proximity communication such as Bluetooth (registered trademark), RFID (Radio Frequency Identification), TransferJet (registered trademark), or human body communication via a human body. be able to. In the present embodiment, the communication unit 402 includes an electrode unit 402B provided in a portion of the article that comes into contact with the user, and a human body communication unit 402A that performs human body communication using the electrode unit 402B. In the present embodiment, the human body communication unit 402A communicates with the human body communication unit 110A of the wearable device 10C in a state where the user wears the wearable device 10C and the user holds an article including the article-side device 40. The article name and the reference image used to detect the article stored in the storage unit 401 are transmitted to the wearable device 10C. Human body communication includes a current system in which a weak current is passed through the human body and the current is modulated to transmit information, and an electric field system in which information is transmitted by modulating an electric field induced on the surface of the human body. . In this embodiment, it is possible to use either the current method or the electric field method. Note that human communication is possible not only when the user is bare hands (that is, when the electrode is in contact with the hand) but also when wearing gloves (ie, when the electrode is facing the hand). It is.

(Wearable device 10C)
The wearable device 10C is different from the wearable devices 10A and 10B of the first embodiment in that the wearable device 10C includes the communication unit 110, information stored in the storage unit 107, and a function included in the control unit 109. Hereinafter, the difference between the wearable device 10C and the wearable devices 10A and 10B will be described in detail.

  The communication unit 110 communicates with other devices. For example, proximity communication such as Bluetooth (registered trademark), RFID, TransferJet (registered trademark), or human body communication via a human body can be used. In the present embodiment, for example, the communication unit 110 performs human body communication using the electrode unit 110B provided on a part of the frame 120 that contacts the user (for example, the user side of the frame 120) and the electrode unit 110B. A human body communication unit 110A. In the present embodiment, the human body communication unit 110A communicates with the human body communication unit 402A of the article side device 40 in a state where the user wears the wearable device 10C and the user holds an article including the article side device 40. The article name and the reference image stored in the storage unit 401 of the article side device 40 are received from the article side device 40.

  The storage unit 107 stores the search object information table shown in FIG.

  The control unit 109 functions as the information storage unit 111, the registration unit 112, the determination unit 113, the detection unit 114, and the notification unit 115 illustrated in FIG. 10 when the CPU executes a program.

  The information storage unit 111 executes a search target information storage process, and registers an image including the search target article among the images captured by the imaging unit 101 as a history image in the search target information table. The search target information storage process executed by the information storage unit 111 is the same as the flowchart shown in FIG. 6 except that an image is input from the imaging unit 101, and thus detailed description thereof is omitted.

  The registration unit 112 executes a search target registration process described later, and registers necessary information in the storage unit 107.

  Of the articles registered in the determination unit 113 and the storage unit 107, an article designated by the user using the operation unit 103 or the microphone 104 is determined as a search target and output to the detection unit 114.

  The detection unit 114 detects whether or not the search target determined by the determination unit 113 exists in the image captured by the imaging unit 101, and outputs the detection result to the notification unit 115.

  The notification unit 115 uses the display unit 102 or the speaker 106 to notify information related to the search target based on the detection result by the detection unit 114. For example, the notification unit 115 displays information indicating the presence of the search target on the display unit 102 together with the image (live view image) currently captured by the imaging unit 101.

(Search target registration process)
Next, the search target registration process executed by the registration unit 112 will be described with reference to the flowchart of FIG. In the present embodiment, the search target registration process is started when the user turns on the wearable device 10C. Instead, the wearable device 10C may be turned on when the human body communication or the proximity communication with the article-side device 40 is established, and the search target registration process may be started.

  In the process of FIG. 11, the registration unit 112 first determines whether or not human body communication with the article-side device 40 is established (step S101). In other words, the registration unit 112 determines whether the user has picked up an article provided with the article-side device 40. Thereby, even when the image of the user holding the article is not captured by the imaging unit 101, the article acquired by the user can be registered. When the human body communication with the article side device 40 is not established (step S101 / NO), the registration unit 112 repeats the determination at step S101 until the human body communication with the article side device 40 is established.

  On the other hand, when human body communication with the article-side device 40 is established (step S101 / YES), the registration unit 112 stores the article name and reference stored in the storage unit 401 of the article-side device 40 via the communication unit 110. The image is received from the article side device 40 (step S103).

  Subsequently, as in step S15 in FIG. 4, the registration unit 112 determines whether or not the article information acquired in step S103 has already been registered in the search target information table (see FIG. 3) (see FIG. 3). Step S105). If the determination here is affirmed, the registration unit 112 returns to step S101. If the determination is negative, the registration unit 112 uses the information on the article acquired in step S103 as the search target, as in step S17 of FIG. The information is registered in the information table (see FIG. 3) (step S107).

  Through the above processing, the article picked up by the user is automatically registered in the search object information table of the storage unit 107.

(Search object detection process)
Next, search object detection processing executed by the control unit 109 will be described with reference to the flowchart of FIG. In the present embodiment, the wearable device 10 </ b> C starts the search object detection process when receiving a search request for the search object from the user via the operation unit 103 or the microphone 104. For example, when a father who is a user of the wearable device 10C pronounces “search object” toward the microphone 104, the wearable device 10C determines that a search request for a search object has been received, and starts the process of FIG. .

  In the process of FIG. 12, first, the determination unit 113 included in the control unit 109 determines a search target (step S141). For example, when the determination unit 113 receives a search request for a search target from the user via the operation unit 103 or the speaker 106, a list of article names registered in the search target information table stored in the storage unit 107. Alternatively, a list of reference images is displayed on the display unit 102. The determination unit 113 determines, for example, an article selected by the user using the operation unit 103 from the list of the search target information table as a search target, and outputs it to the detection unit 114.

  Next, the detection unit 114 inputs an image currently captured by the imaging unit 101 (step S143).

  The detection unit 114 determines whether or not the search target input from the determination unit 113 is present in the image currently captured by the imaging unit 101 (step S145). When the determination here is affirmed, the notification unit 115 displays information indicating the search target in the image currently captured by the imaging unit 101 (for example, an arrow, a marker, etc .: the marker 330 in FIG. 9A). Reference) is added and displayed on the display unit 102 (step S147). As a result, the user of the wearable device 10 </ b> C can notice a search target that is in his or her field of view but is overlooked.

  On the other hand, when the search target does not exist in the image currently captured by the imaging unit 101 (step S145 / NO), the detection unit 114 starts a process of step S143 for a predetermined time (for example, 10 seconds). It is determined whether or not elapses (step S149).

  When the predetermined time has not elapsed, the detection unit 114 executes the processing from step S143 again. However, when the predetermined time has elapsed, the detection unit 114 performs steps S151 to S151 similarly to steps S55 to S61 in FIG. The process of step S157 is executed. And the determination part 113 will complete | finish all the processes of FIG. 12, if the instruction | indication which complete | finishes the detection of a search target object is received from a user (step S157 / YES).

  As described above in detail, according to the second embodiment, the wearable device 10C can be worn by the user, and the imaging unit 101 that captures an image and the image captured by the imaging unit 101 are Since the detection unit 114 that detects whether or not the search target exists and the notification unit 115 that performs notification regarding the search target according to the detection result are provided, even if the user overlooks the search target, If there is a search target in the image captured by the imaging unit 101, the presence of the search target can be notified to the user.

  Further, according to the second embodiment, the wearable device 10 </ b> C includes the registration unit 112 that registers an article that can be a search target, and the registration unit 112 is configured so that the user can perform a human body communication with the article-side device 40. Since the article obtained is registered, the article obtained automatically can be registered without forcing the user to perform a special operation. The registration method of the second embodiment may be applied to the first embodiment.

  Further, when there is no search target in the image currently captured by the imaging unit 101, the notification unit 115 displays the latest image from among the images in which the search target captured by the imaging unit 101 exists in the past. Since the image is displayed on the unit 102, the user can take an action such as changing the place where the search object is searched using the image in which the search object is finally captured.

  In addition, the notification unit 115 displays information indicating the presence of the search target on the display unit 102 when the search target exists in the image currently captured by the imaging unit 101. You can notice the presence of search objects that are in sight but have been overlooked.

  In the second embodiment, the determination unit 113 communicates with, for example, another wearable device 10 to receive information (reference image) of an article that another user is looking for, and the other user May be determined as an object to be searched. In this case, when the detection unit 114 detects a search target, the notification unit 315 captures an image of an article that another user is looking for with a user of the wearable device 10C that is different from the user who is searching for the search target. You may alert | report that the part 101 exists in the image imaged.

  Note that in the second embodiment, the registration unit 112 may perform registration based on the image captured by the imaging unit 101 as in the registration unit 312 of the first embodiment. That is, when the user is holding an article, the registration process may be performed by performing imaging by the imaging unit 101.

  In each of the above embodiments, the wearable devices 10A to 10C may not be glasses-type terminals. For example, it may be a wearable terminal that can be worn at any position on the body, and some of its functions may be separated. As an example, a wristwatch type terminal may have other functions as a contact lens type terminal provided with a display unit.

  Moreover, in each said embodiment, although the search object was registered automatically, a user may register manually. In this case, for example, an article that the user wants to register as a search target may be captured by the imaging unit 101 and registered in the search target information table by operating the operation unit 103. Further, an image of an article to be registered may be captured by a multi-function mobile phone (smart phone) or a tablet-type terminal, and the registration process may be performed.

  Further, in each of the above embodiments, an article (for example, key K10) is registered as a search target, but the search target is not limited to the article. For example, you may register the external appearance photograph of the store published on the homepage as a search object. In this case, the control unit 109 notifies the user when a store exists in the image captured by the imaging unit 101. Thereby, even if the said store is in the user's view, the user can notice the presence of the store when the user overlooks it.

  Further, for example, a keyword (text information) may be registered as a search target. In this case, the control unit 109 performs OCR processing on an image captured by the image capturing unit 101 on a paper medium that the user is viewing (reading), and a keyword (for example, the keyword “due date”) exists in the image. Determine whether or not. Then, when a keyword is present in the image, the control unit 109 notifies the user of the presence of the keyword by displaying a marker 330 or the like surrounding the keyword on the display unit 102 (see FIG. 13). Thereby, the user can notice the presence of the keyword when the keyword is overlooked even though the keyword is present in the sentence that the user is reading. The control unit 109 is not limited to detecting a keyword from a paper medium, but can detect a keyword in an image displayed on a display unit of a portable terminal or a landscape (for example, It is also possible to detect a keyword from a guidance display board or a signboard in the city.

  In each of the above embodiments, when detecting the search object, the detection units 314 and 114 may correct the reference image according to a time zone in which the search object is detected. For example, when a store exterior photo is registered as a reference image and the photo is taken during the day, and the time zone during which the store is detected is nighttime, the detection units 314 and 114 are set to brightness, etc. A store may be detected using a reference image in which is corrected.

  Further, in each of the above embodiments, the notification units 315 and 115 may perform notification regarding the search target at a timing at which the user can determine that the search target is overlooked. For example, when the imaging unit 101 of the wearable device 10 captures an image including a search target and then captures an image that does not include the search target, the notification units 315 and 115 perform notification. Good. Thereby, the troublesomeness of the user due to frequent notification can be reduced.

  In each of the above embodiments, the case where the control unit 109 displays an arrow, a marker, or the like as the search target object on the display unit 102 has been described. However, the present invention is not limited to this. For example, when the wearable device 10 includes a line-of-sight detection unit that can detect the direction of the user's line of sight, the control unit 109 detects the direction of the user's line of sight detected by the line-of-sight detection unit and the search target object. The user's line of sight may be guided based on the position. In this case, for example, the control unit 109 may display a mark in the direction of the user's line of sight using the display unit 102 and gradually move the mark in the direction of the search target. Further, the control unit 109 may output a sound such as “please look at the left side without moving the head” using the speaker 106 to guide the user's line of sight.

  The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the processing device (CPU) should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium (except for a carrier wave).

  When the program is distributed, for example, it is sold in the form of a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  The above-described embodiment is an example of a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.

10A, 10B, 10C Wearable device 30 Information processing apparatus 101 Imaging unit 102 Display unit 112 Registration unit 113 Determination unit 114 Detection unit 115 Notification unit 301 Communication unit 313 Determination unit 314 Detection unit 315 Notification unit

Claims (14)

An input unit for inputting an image from an electronic device attachable to the user;
A detection unit for detecting whether a search object exists in the image;
An information processing apparatus comprising: a notification unit that performs notification regarding the search object. The input unit inputs images from a plurality of electronic devices,
The information processing apparatus according to claim 1, wherein the detection unit detects the search object from a plurality of images input from the plurality of electronic devices.   The information processing apparatus according to claim 1, further comprising a determination unit that determines the search object.   The information processing apparatus according to any one of claims 1 to 3, wherein the notification unit notifies a user different from the user of the search target object.   The information processing apparatus according to any one of claims 1 to 4, wherein the notification unit notifies position information related to the search object.   The information processing apparatus according to any one of claims 1 to 5, wherein the notification unit is configured to notify information on an article that is related to the search target and is different from the search target. An imaging unit that is attachable to a user and that captures an image;
A detection unit for detecting whether or not a search object exists in the image;
An electronic device comprising: a notification unit that performs notification regarding the search object according to a result of the detection.   The electronic device according to claim 7, wherein the notification unit notifies a user different from the user about the search object.   The electronic device according to claim 7, further comprising a determining unit that determines the search object. A registration unit for registering the search object;
The electronic device according to claim 9, wherein the registration unit registers the search object based on information captured by the imaging unit. A registration unit for registering the search object;
The electronic device according to claim 9, wherein the registration unit registers an article obtained by the user as the search object. The detection unit starts detecting whether the determined search target exists in the image after the determination unit determines the search target,
The electronic device according to any one of claims 9 to 11, wherein the notification unit notifies the latest information of information related to the search target object. A display unit for displaying the image;
13. The notification unit displays information indicating the presence of the search target on the display unit when the search target exists in an image currently captured by the imaging unit. The electronic device as described in any one.   The electronic device according to any one of claims 7 to 12, wherein the notification unit displays the image when the search object is detected to be present in the image.

Images