JP2016192096A - Object recognition and selection device, object recognition and selection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to preferentially select an object that the user wants to select or control from a plurality of objects that are photographed and displayed according to object recognition.SOLUTION: There is provide an object recognition and selection device that includes an attachment part attached to the face or head of a user, acquires state information specifying the state of the user, photographs a plurality of objects and captures the images, and recognizes the plurality of objects from the captured images. The object recognition and selection device stores in advance association information in which the state information and the objects are associated with each other, and specifies candidates for an object selected or controlled by the user from the plurality of recognized objects on the basis of acquired direction information, acquired position information, and the stored association information. Also provided are an object recognition and selection method and a program.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an object recognition selection apparatus, an object recognition selection method, and a program for use in a wearable computer (terminal) of a type that is mounted on a head or face with a camera such as a smart glass (glasses type computer). It is.

In recent years, computers that can be worn and carried, so-called wearable computers, have begun to spread. Among these wearable computers, there are glasses-type computers called smart glasses. This smart glass is equipped with a camera and worn by the user as eyeglasses, so an object in front of the user is photographed with the camera and displayed on a display provided in the eyeglass lens part. Can be done. For this reason, in smart glasses, if the user can operate on an object displayed on a display close to the user's eyes, the user operates as if the user is actually looking at the object. Such a feeling will be obtained.
Japanese Patent Application Laid-Open No. 2011-209965 (Patent Document 1) discloses an information processing apparatus that is a portable computer including a head-mounted display (HMD) equipped with a camera and a display as a smart glass. Has been. In this information processing apparatus, when the user wears the HMD, an operation target device (hereinafter simply referred to as “device”) in front of the user is photographed by the camera, and the display in front of the user has a device on the display. In addition, an operation image such as a button is displayed, and the user can operate the device by virtually pressing the button displayed on the display.

Here, in the information processing apparatus, when there are a plurality of devices (for example, a stereo, a television, a fan, an air conditioner, and ceiling lighting) in front of the user (user), the information is extracted from the image captured by the camera by object recognition. There are a plurality of devices, and the plurality of extracted devices are displayed on the display together with operation images such as buttons and can be operated on each of them.
However, the user does not necessarily operate all of the plurality of devices displayed on the display, and often wants to operate only a specific device (for example, ceiling lighting).
In such a case, the user not only has the trouble of selecting and operating a specific device from a plurality of devices displayed on the display, but when a plurality of devices are close to each other, There is a problem that even if the buttons of each device are displayed, it is difficult to determine which button belongs to which device. In addition, instead of using buttons for each device, it is possible to create a list of multiple devices separately and display it on the display, and select the device you want to operate from this list. However, it is difficult to display a list of a plurality of devices.

By the way, when a user of a smart glass or HMD wants to select and control / operate one of a plurality of devices displayed on the display at a certain place (for example, in the user's home room). Usually, the face to be directed (control / operated) is directed in a certain direction (for example, when the ceiling lighting of the room is to be operated, the face is directed upward with the ceiling lighting). Also, even if there are multiple devices in the same direction, which device you want to select (control / operate) is roughly determined by the user's habits (for example, stereo, TV, If there is a fan, some users always operate the TV.)
Therefore, it can be said that the device that the user wants to select (control / operate) depends on the location of the user, the state of the user's face facing, and the like and the user's habits.

JP 2011-209965 A

The problem to be solved by the present invention is to enable a user to preferentially select an object that the user wants to select or control from a plurality of objects that have been photographed and recognized as objects.
The present invention is made by paying attention to the fact that the object that the user wants to select or control depends on the location of the user, the direction of the user's face, etc., and the user's habits. It is a thing.

  The present invention is an object recognition / selection device that includes a mounting unit that is mounted on a user's face or head, and that captures a plurality of objects and captures the images, and the object image capturing unit. Acquired by the object recognition means for recognizing the plurality of objects from the image captured by the capture means, the association information storage means for storing the association information in which the state information and the object are associated, and the state information acquisition means An object candidate that identifies a candidate for an object to be selected or controlled by the user from among a plurality of objects recognized by the object recognizing unit based on the state information that has been performed and the association information stored in the association information storage unit An object recognition / selection device including a specifying unit is provided to solve the above-described problem.

  The present invention provides an object recognition selection apparatus further comprising: means for displaying a plurality of objects recognized by the object recognition means; and object determination means for determining an object to be selected or controlled by the user. It solves the problem.

  According to the present invention, the association information is history information that associates the state information acquired by the state information acquisition unit with the object determined by the selected object determination unit when the object determination unit determines an object. An object recognition / selection device is provided to solve the above problem.

  The present invention further includes an object designating unit for designating the object, wherein the object determination unit selects the user candidate specified by the object candidate specifying unit or the object specified by the object specifying unit. Alternatively, an object recognition / selection device that determines the object to be controlled is provided to solve the above problem.

  The present invention provides an object recognition / selection device further comprising object control means for controlling the movement of the object determined by the object determination means, and solves the above-mentioned problems.

  The present invention provides an object recognition and selection device in which the state information is direction information that the user's face is facing and position information of the user, and solves the above problems.

  The present invention relates to an object recognition selection method by an object recognition selection device provided with a mounting unit to be worn on a user's face or head, which obtains state information for specifying the state of the user, and a plurality of objects And capturing the image, recognizing the plurality of objects from the captured image, storing association information that associates the state information with the object in advance, and acquiring the acquired state information and Provided is an object recognition selection method including a step of identifying a candidate for an object to be selected or controlled by the user from among the plurality of recognized objects based on the stored association information, Is a solution.

  The present invention solves the above problems by providing an object recognition selection method further comprising the step of displaying the plurality of recognized objects and determining an object to be selected or controlled by the user.

  The present invention provides an object recognition selection method in which the association information is history information in which the state information acquired when the user selects or controls an object to be determined is associated with the confirmed object. Thus, the above problems are solved.

  The present invention further comprises the step of specifying the object, and the step of determining the object to be selected or controlled by the user is to select or control the candidate of the specified object or the specified object by the user. An object recognition selection method, which is a step of determining as an object to be performed, is provided to solve the above problem.

  The present invention provides an object recognition and selection method further comprising the step of controlling the movement of the determined object, and solves the above-described problem.

  The present invention solves the above-mentioned problems by providing an object recognition selection method in which the state information is direction information in which the user's face is facing and position information of the user.

  The present invention provides a program for causing a computer to execute the object recognition selection method, thereby solving the above-described problems.

  The object recognition / selection device according to the present invention includes a mounting unit to be worn on the user's face or head, and an image obtained by photographing a plurality of objects is captured from the object image capturing unit. State information for identifying the state of the user when the object is recognized and the object image capturing unit captures the image is acquired from the state information acquiring unit, and the state information and the state are stored in the association information storage unit. Correlation information associated with an object is stored, and based on the state information acquired by the state information acquisition unit and the association information stored in the association information storage unit, the object candidate specifying unit is an object recognition unit. When the user can select or preferentially select an object that the user wants to control or control in order to identify a candidate for the object to be selected or controlled from the plurality of objects recognized by the user Achieve the cormorant effect.

  In the object recognition and selection apparatus of the present invention, the object display means further displays a plurality of objects recognized by the object recognition means, and the object determination means determines the object to be selected or controlled by the user. It is possible to preferentially select an object to be selected or controlled.

  In the object recognition / selection device of the present invention, the association information includes the state information acquired by the state information acquisition unit and the object determined by the selected object determination unit when the object determination unit determines an object. Since the history information is associated with each other, the object that the user wants to select or control can be preferentially selected based on the association information that is history information.

  In the object recognition / selection device of the present invention, the object designating unit further specifies the object, and the object determination unit selects the object candidate specified by the object candidate specifying unit or the object specified by the object specifying unit. Since the object to be selected or controlled by the user is determined, an object to be selected or controlled by the user can be preferentially selected, and the user can designate it.

  In the object recognition / selection device of the present invention, the object control means can preferentially select an object to be selected or controlled by the user because the object control means controls the operation of the object decided by the selected object decision means. The user can control it.

  In the object recognition / selection device of the present invention, the state information includes direction information that the user's face is facing and position information of the user. Therefore, the object that the user wants to select or control is given priority. There is an effect that it can be selected.

  An object recognition selection method according to the present invention is an object recognition selection method by an object recognition selection device having a mounting portion worn on a user's face or head, and acquires state information for specifying a user's state. , Capturing a plurality of objects, capturing the images, recognizing the object from the captured images, storing association information that associates the state information with the object in advance, and acquiring the acquired state Based on the information and the stored association information, the user selects or controls the object to be selected or controlled from among the plurality of recognized objects. There is an effect that can be selected.

  The object recognition selection method of the present invention displays the plurality of recognized objects, and determines the object to be selected or controlled by the user, so that the object to be selected or controlled by the user can be preferentially selected. Play.

  In the object recognition selection method of the present invention, the association information is history information in which the state information acquired when the user selects or controls an object to be determined is associated with the confirmed object. Based on the association information that is history information, there is an effect that an object that the user wants to select or control can be preferentially selected.

  The object recognition / selection method of the present invention further specifies an object, and determines or identifies the specified object candidate or the specified object as an object to be selected or controlled by the user. It is possible to preferentially select the desired object, and the user can designate it.

  The object recognition / selection method of the present invention further controls the movement of the determined object, so that the user can select or preferentially select the object to be controlled, and the user can control it. .

  In the object recognition selection method of the present invention, since the state information includes direction information of the user's face and position information of the user, the user selects or preferentially selects an object to be controlled. There is an effect that can be done.

It is an external view which shows the apparatus structure of one Embodiment at the time of utilizing the object recognition selection apparatus of this invention for a smart glass. It is a block diagram which shows schematic structure of the object recognition selection apparatus 1 shown in FIG. It is a block diagram which shows an example of the specific structure of the object recognition selection apparatus 1 shown in FIG. It is a figure for demonstrating the relationship between a user's attitude | position and the gravity which acts on the smart glass 1 (mounting part 2) with which the user was mounted | worn. It is a figure for demonstrating the relationship between the direction which the user's face is facing, and the direction of the magnetic field which acts on the smart glass 1 (mounting part 2) with which the user was mounted | worn. 6 is an explanatory diagram illustrating an example of an image of an object displayed on a display 50. FIG. It is explanatory drawing which showed the example of the log | history information table showing the correlation information which the correlation information storage part 60 memorize | stores. It is a figure explaining the example which displayed the candidate of the object which the object candidate specific | specification part 70 specified on the display 50 (screen 51). It is a figure for demonstrating the example which the object determination part 90 determines the object which a user selects. 3 is a flowchart showing an outline of the operation of the object recognition / selection device (smart glasses) 1. 4 is a flowchart illustrating details of an operation of acquiring direction information performed by the direction information acquisition unit 10. 5 is a flowchart showing details of an operation of acquiring position information performed by a position information acquisition unit 20. It is a flowchart which shows the detail of the operation | movement of the object image taking-in which the object image taking-in part 30 performs. It is a flowchart which shows the detail of the operation | movement of the object recognition which the object recognition part 40 performs. 5 is a flowchart showing details of an operation in which an image display unit (display) 50 displays an image of an object recognized by the object image capturing unit 40. It is a flowchart which shows the detail of the operation | movement of object candidate specification and selection object determination which the object candidate specific | specification part 70 and the selection object determination part 90 perform.

[Appearance configuration of object recognition selection device (smart glasses)]
An external configuration of the apparatus when the object recognition / selection apparatus of the present invention is used for smart glasses will be described.
FIG. 1 is an external view showing an external appearance of an apparatus according to an embodiment when the object recognition / selection apparatus of the present invention is used for a smart glass. In the figure, 1 is an object recognition / selection device (smart glass), 2 is a mounting part, 3 is a frame, and 4 is a lens part. The smart glass 1 includes a glasses-type mounting unit 2 and a controller (not shown) built therein. The controller is provided with a wireless unit and is connected to a server via the Internet. The mounting unit 2 includes a frame 3 and a pair of lens units 4. A camera is mounted on the frame, an acceleration sensor, a geomagnetic sensor, and the like, and a position detection sensor such as a GPS receiver is also incorporated. The wearing unit 2 is worn on the user's face in the same manner as glasses. It should be noted that the controller may not be built in the glasses-type mounting unit 2 but may be a separate one.

[Schematic Configuration of Object Recognition Selection Device (Smart Glass) 1]
FIG. 2 is a block diagram showing a schematic configuration of the object recognition / selection apparatus 1 shown in FIG. In the figure, 10 is a direction information acquisition unit, 20 is a position information acquisition unit, 30 is an object image capture unit, 40 is an object recognition unit, 50 is an object display unit, 60 is an association information storage unit, and 70 is an object candidate specification. , 80 is an object designation unit, 90 is an object determination unit, and 100 is an object control unit.
The direction information acquisition unit 10 determines in which direction the face of the user wearing the smart glass 1 is directed from the data measured by the acceleration sensor and the geomagnetic sensor built in the wearing unit 2 (frame 3) of the smart glass 1. Direction information that the user's face is facing is acquired.
The position information acquisition unit 20 acquires position information of a user wearing the smart glass 1 from data detected by a position detection sensor such as a GPS receiver built in the smart glass 1.
The object image capturing unit 30 is a camera mounted on the mounting unit 2 (frame 3) of the smart glass 1, and various devices (for example, a stereo, a television, a fan) existing in front of the user wearing the smart glass 1. A plurality of objects such as air conditioners and ceiling lights) and capture the images.
The object recognition unit 40 performs object recognition for recognizing a plurality of objects from an image including the plurality of objects captured by the object image capturing unit 30.
The object display unit 50 is attached to the lens unit 4 of the smart glass 1 and displays images of a plurality of objects recognized by the object recognition unit 40.
The association information storage unit 60 stores association information in which direction information that the user's face is facing and position information of the user are associated for each object such as various devices.
The object candidate specifying unit 70 is based on the user direction information acquired by the direction information acquisition unit 10, the user position information acquired by the position information acquisition unit 20, and the association information stored in the association information storage unit 60. Thus, a candidate for an object to be selected or controlled by the user is specified from among a plurality of objects displayed on the object display unit 50.
The object designating unit 80 designates an object to be selected or controlled by a user operation.
The selected object determination unit 90 selects an object candidate to be selected or controlled by the user specified by the object candidate specifying unit 70 or an object specified by the user by the object specifying unit 80 as an object to be selected or controlled by the user. Determine.
The object control unit 100 controls a communication unit (not shown), communicates with an object (such as various devices) determined by the selected object determination unit 90, and controls its operation (for example, ON / OFF of a switch). To do.

[Specific Configuration of Object Recognition Selection Device 1]
FIG. 3 is a block diagram showing an example of a specific configuration of the object recognition / selection apparatus 1 shown in FIG. 2, in which 5 is a control unit of the smart glass 1 and 6 is a server connected to the Internet. Hereinafter, a specific configuration of the object recognition / selection apparatus 1 will be described with reference to FIG.

[Direction information acquisition unit 10]
The direction information acquisition unit 10 shown in FIG. 2 includes an acceleration sensor 11, a data acquisition storage unit 12, a posture analysis specification unit 13, a geomagnetic sensor 14, a data acquisition unit 15, a direction calculation specification unit 16, a direction information generation unit 17, and the like. Is done.

  The acceleration sensor 11 is built in the vicinity of the lens unit 4 on the left side of the frame 3 and measures the gravitational acceleration acting on the smart glass 1 (mounting unit 2). The gravitational acceleration data measured by the acceleration sensor 11 is After being stored and accumulated in the data acquisition / storage unit 12, it is sent to the posture analysis specifying unit 13.

The posture analysis specifying unit 13 analyzes the gravitational acceleration data sent from the data acquisition and storage unit 12 and specifies the posture of the user.
FIG. 4 is a diagram for explaining the relationship between the posture of the user and the gravity acting on the smart glass 1 (mounting unit 2) attached to the user, where U is the user and UF is the use. A person's face, VD, is a range in a direction in which the face UF faces. The relationship between the user's posture and the gravity acting on the smart glass 1 will be described below with reference to the drawings. Here, the depth direction of the smart glass 1 (mounting part 2) is the X axis, and the left and right direction is the Z axis. The vertical direction is the Y axis.
As shown in FIG. 4A, when the user U is in an upright posture, the face UF faces in the horizontal direction, and the XZ axis direction of the smart glass 1 attached to the face UF is in the horizontal direction. Gravitational acceleration acts in the Y-axis direction.
On the other hand, when the user U is in a sleeping posture as shown in FIG. 4B, the face UF faces upward, and the YZ-axis direction of the smart glass 1 attached to the face UF becomes the horizontal direction. Therefore, the gravitational acceleration acts in the X-axis direction.
Thus, from the gravity acceleration data acting on the smart glass 1, it is possible to specify the posture of the user U, that is, the direction in which the face UF is directed in the vertical direction (vertical direction) VD. it can.
The posture of the user is standing up (face is facing horizontally), sleeping (face is facing directly upward), diagonally up (face is facing diagonally upward), diagonally downward (The face faces obliquely downward) may be provided, or may be set more finely. In addition, by analyzing the temporal change of gravity acceleration data measured continuously, the change in the vertical direction of the user's face is obtained, and the estimated posture (for example, sitting posture, prone position, etc.) May be set.
Information on the user's posture (which direction the face is facing in the vertical direction (vertical direction) VD) specified by the posture analysis specifying unit 13 in this way is sent to the direction information creating unit 17.

The geomagnetic sensor 14 is built in the vicinity of the lens unit 4 on the left side of the frame 3 and measures the direction of the magnetic field acting on the smart glass 1 (mounting unit 2). The data measured by the geomagnetic sensor 14 is After being once taken into the data acquisition unit 15, it is sent to the direction calculation specifying unit 16.
The azimuth calculation specifying unit 16 calculates the azimuth of north, northeast, east, etc. in units of 45 degrees from the data (magnetic field direction) sent from the data acquisition unit 15, and which direction the user's face is facing. To identify. The azimuth to be calculated may be a unit of 15 degrees including north, northeast, east, etc., and further north, northeast, etc., or may be further reduced.
FIG. 5 is a diagram for explaining the relationship between the orientation of the user's face and the direction of the magnetic field acting on the smart glass 1 (mounting unit 2) mounted on the user. In the figure, UD is the direction of the magnetic field acting on the smart glass 1, and HD is the range of the direction in which the face UF of the user U faces. As shown in the figure, the direction UD of the magnetic field measured by the geomagnetic sensor 14 is determined in which direction in the east, west, south, and north directions. Can ask.
Information on the user's azimuth (which direction the face faces in the horizontal direction HD) identified by the azimuth calculation / identification unit 16 in this way is sent to the direction information creation unit 17.
Since the smart glass 1 is often used while being tilted (the face of the user wearing the smart glass 1 is often tilted vertically), the geomagnetic sensor 14 is a three-axis type.

  The direction information creation unit 17 sends information about the user's posture (which direction the face is facing in the vertical direction (vertical direction) VD) sent from the posture analysis specifying unit 13 and the direction calculation specifying unit 16. Information on the orientation of the user (which direction the face is facing in the horizontal direction HD) is combined, and this is sent to the object candidate specifying unit 70 as direction information that the user's face is facing.

[Position information acquisition unit 20]
The position information acquisition unit 20 illustrated in FIG. 2 includes a position detection sensor 21, a data acquisition unit 22, a position information creation unit 23, and the like.

The position detection sensor 21 is a GPS receiver, WiFi, IMES, beacon or the like, and is built in the controller 6 of the smart glass 1 to detect and output the position of the user wearing the smart glass 1. The GPS receiver detects a position when the user is outdoors, receives a radio wave from a GPS satellite, and outputs it as position data. Wifi, IMES, beacons, and the like detect the position when the user is indoors. The position data output from the position detection sensor 21 is once taken into the data acquisition unit 22 and then sent to the position information creation unit 23.
The position information creation unit 23 creates user position information based on the position data sent from the data acquisition unit 22, and sends the created position information to the object candidate identification unit 70. The position information created by the position information creating unit 23 may be numerical data such as latitude and longitude output by the position detection sensor 21, or may be an address, facility / building name associated with the position data, It may be a room name specified by the user (living room, bedroom, etc.) or a part of the room (by the living window, living sofa) (in this case, the position data is associated with the room name or a part of the room in advance) .)

[Object Image Acquisition Unit 30]
The object image capturing unit 30 shown in FIG. 2 includes a camera 31, an image acquisition unit 32, a transmission unit 33, an Internet line 34, a reception unit 35, an image acquisition unit 36, and the like.
The camera 31 is attached in the vicinity of the lens unit 4 on the right side of the frame 3 of the smart glass 1 and photographs an object existing in front of the user wearing the smart glass 1 (mounting unit 2). The image acquisition unit 32 captures an image captured by the camera 31, and the transmission unit 33 transmits this image to the server 6 through the Internet line 34.
On the server 8 side, the receiving unit 35 receives an image sent from the Internet line 34, the image acquisition unit 36 takes in the received image, and sends it to the object recognition unit 40.

[Object recognition unit 40]
The object recognition unit 40 recognizes an object included in the image sent from the image acquisition unit 36 based on model data stored in a model data storage unit (not shown). In the model data storage unit, external data of an object such as various devices (for example, stereo, television, electric fan, air conditioner, ceiling lighting) is stored as model data with its name (identification symbol).
Therefore, the object recognizing unit 40 extracts the outer shape of the object included in the image sent from the image acquisition unit 36, and the extracted object outer shape and the model data (object outer shape data stored in the model data storage unit). ), The matched model data is recognized as an object included in the image, and the name (identification symbol) is associated.
Then, the object recognition unit 40 sends the recognized object image and the name (identification symbol) associated therewith to the transmission unit 41, and the transmission unit 41 transmits this to the smart glass 1 via the Internet line 42 (34).

On the smart glass 1 side, the receiving unit 43 receives the image of the object and its name (identification symbol) sent from the Internet line 42 (34), and the object image acquiring unit 44 acquires this, and the display 50 and the object The data is sent to the candidate specifying unit 70.
The object recognition 40 may be provided not on the server 6 side but on the smart glass 1 side. In this case, the transmission unit 33, the Internet line 34, the reception unit 35, the image acquisition unit 36, and the like are not necessary, and the object recognition 40 directly recognizes an image from the image acquisition unit 32.

[Image Display Unit (Display) 50]
A display 50, which is the image display unit 50 of FIG. 2, is provided on the upper part of the lens units 4 and 4 of the smart glass 1, and projects and displays an object image recognized by the object recognition unit 40 and acquired by the object image acquisition unit 44. .
FIG. 6 shows an example of an image of an object displayed on the display 50, and 51 is a screen. In this example, since the object recognition unit 40 has recognized stereo, television, electric fan, air conditioner, and ceiling lighting by object recognition, these images 200 are displayed on the display screen 51 in a state where they are actually arranged. . That is, a ceiling lighting image 201 and an air conditioner image 202 are displayed at the top of the screen 51, a television image 203 is displayed at the center, and a stereo image 204 and a fan image 205 are displayed at the bottom. Yes.
The user actually sees the stereo, television, fan, air conditioner, and ceiling lighting through the lens units 4 and 4 and sees these images 201 to 205 displayed on the screen 51.

[Association Information Storage Unit 60]
The association information storage unit 60 includes the direction information created by the direction information creation unit 17 and the position information of the user created by the camera 31 and the object recognition unit. The association information that associates the object recognized by 40 is stored.
FIG. 7 shows an example of a history information table representing the association information stored in the association information storage unit 60.
The history information table 61 in FIG. 7 records history information including items of “history ID”, “direction information (attitude and orientation)”, “position information”, “recognized object”, and “selected object”. It is.
“History ID” is a number or the like that distinguishes history information that the user has selected the object by wearing the smart glass 1 in the past, and “direction information” is the posture when the user selects an object ( Information on which direction the face is facing in the vertical direction (vertical direction)) and orientation (which direction the face is facing in the horizontal direction HD) is recorded. The user's position information is recorded, and the “recognized object” is a record of the object recognized by the object recognition unit 40 at that time and displayed on the display 50 (screen 51). The “selected object” is a record of the object selected by the user at that time.
For example, in the history ID “1”, when the user is in the “standing” posture, the face is facing “east”, and in the “living room”, the object “ceiling lighting, air conditioner, TV, stereo, electric fan” Is recorded, and history information that the object selected by the user is “ceiling lighting” is recorded.
The history information table 61 records all the history information when the user has selected an object in the past. In the past, “direction information (attitude, orientation)”, “position information”, “recognized object” Even if all the “selected objects” have the same history information, they are recorded as new history information with different history IDs. For example, in the history IDs “1” and “6”, all of “direction information (posture and orientation)”, “position information”, “recognized object”, and “selected object” are the same history information. .
In the history table 61, the postures of the direction information are “standing” and “sleeping”, but may be set more finely by adding “diagonally up”, “diagonally below”, etc. In addition to “north”, “east”, “south”, and “west”, “north-east”, “north-north-east”, and the like may be added for more detailed setting. In addition to the room such as “living room” and “bedroom”, the position information may be set finely to a part of the room such as “living window” and “living sofa”.

As described above, the user's direction information and position information are associated with the object based on the history information indicating which object is selected when the user is in which position and in which direction the face is facing. The reason is as follows.
What kind of object exists depends on the position of the user, and since the user generally turns his face toward the object he wants to select, the object selected by the user based on the user's direction information and position information is It can be roughly identified.
However, some users do not always turn their face toward the object they want to select. For example, when it is desired to select ceiling lighting, there are some users who do not turn their face toward the ceiling lighting and watch the ceiling lighting as an upper eye. Further, when there are a plurality of objects in the same direction as viewed from the user, the object to be selected cannot be specified only from the direction of the face.
In such a case, if the user's behavior is used, an object that the user wants to select can be selected. In other words, the direction in which the face is directed to the object to be selected often depends on the user's habits, and when there are multiple objects in the same direction as seen from the user, which object to select It often depends on the habits of the user (for example, when there is a stereo, a TV, or a fan in front of the user, a certain user always has the habit of selecting and operating the TV).
Therefore, based on the history information on which object is selected when the user is in which position and in which direction the face is facing, the user's habits and the like are reflected, and the object that the user wants to select is more accurate. Can be selected.

[Object candidate specifying unit 70]
The object candidate specifying unit 70 receives direction information (information on the user's posture and orientation) sent from the direction information creation unit 17 and information sent from the position information creation unit 23. Of the plurality of objects recognized by the object recognizing unit 40 and displayed on the display 50 based on the position information and the history information (association information) of the history information table 61 stored in the association information storage unit 60. The candidate of the object which a person selects is specified.
That is, the object candidate specifying unit 70 searches the history information table 61, and direction information (attitude and orientation) that matches the direction information (posture and orientation) from the direction information creation unit 17 and the position information from the position information creation unit 23. The history information of the history ID in which the (azimuth) and the position information are recorded is searched, and the object selected in the history information of the matching history ID is set as the object candidate.
For example, when the direction information (posture and direction) from the direction information creation unit 17 is “standing” and “east” and the position information from the position information creation unit 23 is “living”, the direction information in the history information table 61 The history IDs whose position information matches (posture and orientation) are “1” and “6”, and the “selected object” in the history information of the history IDs “1” and “6” is both “ceiling lighting”. Therefore, the object candidate specified by the object candidate specifying unit 70 is “ceiling lighting”.
In addition, the object candidate specifying unit 70 stores the direction information (posture and orientation) from the direction information creation unit 17 and the direction information (posture and orientation) that matches the position information from the position information creation unit 23 and the position information. If there are a plurality of history IDs and the selected objects of the history IDs are different, priorities are assigned to the object candidates, and objects with a higher number of matches are set as higher priority object candidates.
The object candidates specified by the object candidate specifying unit 70 in this way are displayed on the display 50 (screen 51), but when there are a plurality of object candidate candidates, they are displayed with priorities.

  FIG. 8 is a diagram for explaining an example in which object candidates specified by the object candidate specifying unit 70 are displayed on the display 50 (screen 51). In the figure, since the object candidate specified by the object candidate specifying unit 70 is “ceiling lighting”, an example is shown in which a selection frame 210 surrounding the image 201 representing ceiling lighting is displayed. The method of displaying the object candidate specified by the object candidate specifying unit 70 on the display 50 (screen 51) is not limited to displaying such a selection frame, and the specified object candidate may be displayed blinking. You may display the name of the candidate of the made object.

[Object designation unit 80, object determination unit 90]
When the object specifying unit 80 selects an object other than the object candidate specified by the object candidate specifying unit 70 as an object to be selected or controlled by the user, the object specifying unit 80 specifies the object to be selected by the user through the operation of the user.
The object determining unit 90 determines the object candidate specified by the object candidate specifying unit 70 or the object specified by the object specifying unit 80 as an object to be selected by the user.

FIG. 9 is a diagram for explaining an example in which the object determination unit 90 determines an object selected by the user. Hereinafter, a case where the object specifying unit 80 specifies another object “air conditioner” for the object candidate “ceiling lighting” specified by the object candidate specifying unit 70 will be described.
The object designating unit 80 includes a button 211 containing “other candidate selection” characters and a “confirm” character below a selection frame 210 surrounding the “ceiling lighting” image 210 displayed on the display 50 (screen 51). A button 212 containing is displayed (FIG. 9A). When the user presses the “determine” button 212, the object determination unit 90 determines the object candidate “ceiling illumination” specified by the object candidate specification unit 70 as the object to be selected by the user. On the other hand, when the user presses the “select other candidates” button 211, the selection frame 210 can be moved by the user's operation, and the user moves the selection frame 210 around the “air conditioner” image 202. Then, the object designating unit 80 displays a “select other candidate” button 211 and a “confirm” button 212 again below the selection frame 210. Here, when the user presses the “determine” button 212, the object determination unit 90 determines the object “air conditioner” designated by the object designation unit 80 as the object to be selected by the user.
In this way, the object determination unit 90 determines the object selected by the user, and associates the direction information (posture and orientation) and position information of the user at that time with the object determined by the object determination unit 90. The history information writing unit (not shown) additionally records the new history information in the history information table 61.
Then, the information on the object determined by the object determination unit 90 is sent to the object control unit 100.

[Object control unit 100]
The object control unit 100 controls a communication unit (not shown) that communicates with a server (not shown) that controls various operation devices, and the object (various devices) determined by the object determination unit 90 via this server. Communication) and control its operation (eg, switch ON / OFF).

[Outline of Operation of Object Recognition Selection Device (Smart Glass) 1]
FIG. 10 is a flowchart showing an outline of the operation of the object recognition / selection apparatus (smart glasses) 1. The outline of the operation of the object recognition / selection apparatus 1 will be described below with reference to FIG.
First, the direction information acquisition unit 10 analyzes which direction the face of the user wearing the smart glass 1 is facing from the data measured by the acceleration sensor 11 and the geomagnetic sensor 14, and the face of the user is facing. Direction information is acquired (S1).
Next, the position information acquisition unit 20 acquires the position information of the user wearing the smart glass 1 from the data detected by the position detection sensor 21 (S2).
Next, the object image capturing unit 30 captures and captures a plurality of objects existing in front of the user with the camera 31 mounted on the smart glass 1 (S3).
Next, the object recognition unit 40 performs object recognition for recognizing the plurality of objects from the image including the plurality of objects captured by the object image capturing unit 30 (S4), and displays the images of the recognized plurality of objects as images. It is displayed on the part 50 (S5).
Next, the object candidate identification unit 70 acquires the user direction information acquired by the direction information acquisition unit 10, the user position information acquired by the position information acquisition unit 20, and the association information stored in the association information storage unit 60. Based on the above, a candidate for an object to be selected or controlled by the user is specified from among a plurality of objects displayed on the object display unit 50 (S6).
Next, the object determination unit 90 selects the user candidate selected or controlled by the object candidate specifying unit 70 or the object specified by the user using the object specifying unit 80, or the object selected or controlled by the user. (S7).
Finally, the object control unit 100 communicates with an object (such as various devices) selected or controlled by the user determined by the object determination unit 90 and controls its operation (for example, switch ON / OFF) ( S8), the process ends.

[Operation of Direction Information Acquisition Unit 10]
FIG. 11 is a flowchart showing details of the direction information acquisition operation (S1 in FIG. 10) performed by the direction information acquisition unit 10, and the operation of the direction information acquisition unit 10 will be described below with reference to FIG.
First, the acceleration sensor 11 measures the gravitational acceleration acting on the smart glass 1 worn by the user (S10), and the data acquisition / storage unit 12 temporarily stores and stores the measured data (S11).
Next, the posture analysis specifying unit 13 analyzes the gravitational acceleration data sent from the data acquisition and storage unit 12 to determine which direction of the user's face is in the vertical direction (vertical direction), that is, the user Is identified (S12), and the identified posture information of the user is sent to the direction information creation unit 17 (S13).
User posture information includes standing (face is facing horizontally), sleeping (face is facing directly upward), diagonally upward (face is facing diagonally upward), diagonally downward ( The face is facing diagonally downward).
Next, the geomagnetic sensor 14 measures the direction of the magnetic field acting on the smart glass 1 (S14), and the direction calculation specifying unit 16 calculates the direction of north, northeast, east, etc. from the measurement data (direction of the magnetic field). Then, the orientation of the user's face is specified (S15), and the specified orientation information of the user is sent to the direction information creation unit 17 (S16).
Finally, the direction information creation unit 17 combines the user's posture information sent from the posture analysis specifying unit 13 and the user's heading information sent from the direction calculation specifying unit 16 to obtain the user's face. Is created (S17), and the direction information acquisition process is terminated.

[Operation of Position Information Acquisition Unit 20]
FIG. 12 is a flowchart showing details of the position information acquisition operation (S2 in FIG. 10) performed by the position information acquisition unit 20. The operation of the position information acquisition unit 20 will be described below with reference to FIG.
First, the position detection sensor 21 such as a GPS receiver detects the position of the user wearing the smart glass 1 and outputs position data (S20).
Next, the data acquisition unit 22 takes in the output position data (S21), and based on this position data, the position information creation unit 23 creates the user's position information (S22), and ends the position information acquisition process. .

[Operation of Object Image Capture Unit 30]
FIG. 13 is a flowchart showing details of the object image capturing operation (S3 in FIG. 10) performed by the object image capturing unit 30, and the operation of the object image capturing unit 30 will be described below with reference to FIG.
First, the camera 31 captures an object existing in front of the user wearing the smart glass 1 (S30), the image acquisition unit 32 captures the captured image (S31), and the transmission unit 33 captures the image on the Internet. The data is transmitted to the server 8 through the line 34 (S32).
Next, the receiving unit 35 on the server 6 side receives the image sent from the Internet line 34 (S33), captures the image received by the image acquisition unit 36 (S34), and ends the object image capturing process.

[Operation of Object Image Capture Unit 40]
FIG. 14 is a flowchart showing details of the object recognition operation (S4 in FIG. 10) performed by the object recognition unit 40. Hereinafter, the operation of the object recognition unit 40 will be described with reference to FIG.
First, the outer shape of the object included in the image sent from the image acquisition unit 36 is extracted (S40), and the extracted outer shape of the object, model data (object outer shape data) stored in the model data storage unit, and Are matched (S41).
Next, the matched model data is recognized as an object included in the image (S42), the name (identification symbol) is associated (S43), and the object recognition process is terminated.

[Operation of Image Display Unit (Display) 50]
FIG. 15 is a flowchart showing details of an operation (S5 in FIG. 10) in which the image display unit (display) 50 displays the image of the object recognized by the object image capturing unit 40. Hereinafter, the image is displayed based on FIG. The operation of the display unit (display) 50 will be described.
First, the transmission unit 41 on the server 6 side transmits the image of the object recognized by the object recognition unit 40 and the name (identification symbol) associated therewith via the Internet line 42 (34) (S50).
Next, the reception unit 43 on the smart glass 1 side receives the image of the object that has been sent (S51), the object image acquisition unit 44 acquires the image of this object (S52), and the image display unit (display) ) 50 displays the image of the object on the screen 51 (S53, see FIG. 6), and the process is terminated.

[Operations of Object Candidate Identification Unit 70 and Object Determination Unit 90]
FIG. 16 is a flowchart showing details of the object candidate specification and object determination operations (S6 and S7 in FIG. 10) performed by the object candidate specification unit 70 and the object determination unit 90. Hereinafter, the object candidate specification will be described based on FIG. Operations of the unit 70 and the object determination unit 90 will be described.
First, the object candidate specifying unit 70 searches the history information table 61, and direction information (attitude and orientation) that matches the direction information (posture and orientation) from the direction information creation unit 17 and the position information from the position information creation unit 23. It is determined whether there is a history ID in which the (azimuth) and position information is recorded (S60). If there is a matching history ID, the object selected in the history information of the matching history ID is specified as an object candidate (S61), and if there is no matching history ID, the process ends.
Next, the object candidate specifying unit 70 determines whether there are two or more (plural) object candidates (S62). If there are two or more (plural) object candidates, the object candidate specified on the display 50 (screen 51) is displayed. Displayed (S63, see FIG. 8).
Next, the object determination unit 90 determines whether an object other than the object candidate is specified from the object specifying unit 80 by the user's operation (S64). If there is no object specification, the displayed object candidate is displayed. Is determined as the object to be selected by the user (S65), and if the object is specified, the specified object is displayed as a candidate for the object (S63), and the process of S64 is performed (see FIG. 9).
Then, the direction information (attitude and orientation) and position information of the user at this time is associated with the object determined by the selected object determination unit 90 to obtain new history information, which is used as a history information writing unit (not shown). Are additionally recorded in the history information table 61 (S66).
If there are two or more (plural) object candidates in S62, the object candidate specifying unit 70 determines whether or not the object candidates are the same (S67). If they are not the same, the object candidates are different. In this case, priorities are assigned to the object candidates (S68), and the object candidates with priorities are displayed (S63). For example, an object having a higher number of times that the direction information and the position information coincide with each other has a higher priority. The selection frame 210 is displayed for an object having a high priority, and the processes after S64 are performed.

  An object recognition selection device, an object recognition selection method, and a program according to the present invention are provided from a wearable computer (terminal) of a type that is mounted on a head or a face by mounting a camera such as a smart glass (glasses type computer). An object (device) to be selected or controlled can be preferentially selected.

1 Object recognition selection device (smart glasses)
2 Mounting unit 3 Frame 4 Lens unit 5 Control unit 6 Server 10 Direction information acquisition unit 11 Acceleration sensor 12 Data acquisition storage unit 13 Attitude analysis specification unit 14 Geomagnetic sensor 15 Data acquisition unit 16 Direction calculation specification unit
17 direction information creation unit 20 position information acquisition unit 21 position detection sensor 22 data acquisition unit 23 position information creation unit 30 object image capture unit 31 camera 32 image acquisition unit 33 transmission unit 34, 42 internet line 35, 43 reception unit 36 image Acquisition unit 40 Object recognition unit 44 Object image acquisition unit 50 Object display unit (display)
51 Screen 60 Correlation Information Storage Unit 61 History Information Table 70 Object Candidate Identification Unit 80 Object Identification Unit 90 Selected Object Determination Unit 100 Object Control Units 200 to 205 Image 210 Selection Frames 211 and 212 Button U User UF User's Face VD Range UD in the direction facing the face UF Direction of the magnetic field acting on the smart glass 1 HD Range in the direction where the face UF of the user U faces

Claims (13)

An object recognition / selection device having a mounting portion to be mounted on a user's face or head,
Object image capturing means for capturing a plurality of objects and capturing the images;
Object recognition means for recognizing the plurality of objects from the image captured by the object image capture means;
State information acquisition means for acquiring state information for specifying the state of the user when the object image capturing means captures the image;
Association information storage means for storing association information in which the state information and the object are associated;
An object to be selected or controlled by the user from among a plurality of objects recognized by the object recognition unit based on the state information acquired by the state information acquisition unit and the association information stored in the association information storage unit An object candidate specifying means for specifying a candidate of
An object recognition / selection device comprising: Object display means for displaying a plurality of objects recognized by the object recognition means;
Object determining means for determining an object to be selected or controlled by the user;
The object recognition / selection apparatus according to claim 1, further comprising:   The association information is history information associating state information acquired by the state information acquisition unit with an object determined by the object determination unit when the object determination unit determines an object. The object recognition selection apparatus according to claim 2. An object designating unit for designating the object;
The object determination unit determines the object candidate specified by the object candidate specifying unit or the object specified by the object specifying unit as an object to be selected or controlled by the user. The object recognition selection apparatus according to claim 3.   5. The object recognition / selection apparatus according to claim 2, further comprising object control means for controlling an action of the object determined by the object determination means.   6. The object recognition / selection apparatus according to claim 1, wherein the state information includes direction information of the user's face and position information of the user. An object recognition selection method by an object recognition selection device provided with a mounting part to be worn on a user's face or head,
Obtaining state information identifying the state of the user;
Capture multiple objects and capture the images,
Recognize the plurality of objects from the captured image,
Preliminarily storing association information that associates the state information with the object,
Based on the acquired state information and the stored association information, a candidate for an object to be selected or controlled by the user from among the plurality of recognized objects is specified.
An object recognition selection method comprising steps.   8. The object recognition selection method according to claim 7, further comprising the step of displaying the plurality of recognized objects and determining an object to be selected or controlled by the user.   9. The object according to claim 8, wherein the association information is history information in which the state information acquired when the user selects or controls an object to be controlled is associated with the confirmed object. Recognition selection method.   The step of specifying the object is further provided, and the step of determining an object to be selected or controlled by the user is determined as the candidate for the specified object or the specified object as an object to be selected or controlled by the user. 10. The object recognition / selection method according to claim 8, wherein the object recognition / selection method is performed.   11. The object recognition / selection method according to claim 8, further comprising a step of controlling an operation of the determined object.   The object recognition selection method according to any one of claims 7 to 11, wherein the state information includes direction information of the user's face and position information of the user. A program that causes a computer to execute the object recognition selection method according to any one of claims 7 to 12.

Images