JP2012173772A - User interaction apparatus, user interaction method, user interaction program and integrated circuit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability of a user when the user performs selection of real objects targeted for operation.SOLUTION: A user operation monitoring part 102 identifies an operational classification corresponding to a current state of a user's hand based on a monitoring result of the state of the hand. A priority calculation part 103 calculates priority of each real object distinguished by a real object identification part 107. When the operational classification identified by the user operation monitoring part 102 indicates an operational classification of "in selection operation" before the user selects the real object targeted for operation, a display effect setting part 105 sets a degree of a display effect determined beforehand depending on priority to each real object distinguished by the real object identification part 107. A real object effect application part 104 applies a display effect corresponding to the degree of the display effect set by the display effect setting part 105 to each real object which is calculated the priority by the priority calculation part 103.

Description

  The present invention relates to a technique in which a user selects a real object to be operated without contact.

  In recent years, there has been provided an apparatus for realizing augmented reality (hereinafter referred to as “AR”) in which virtual electronic information is synthesized and displayed on an object of video data in real space and additional information about the object is presented. ing. For example, there is AR software such as Sekai Camera (registered trademark), and these software operate on a smartphone such as iPhone (registered trademark) or Android (registered trademark).

  In the future, the presentation of information using AR will not only present the information of an object in the real space (hereinafter referred to as a real object) to the user, but also an operation request for selecting a real object from the user as an operation target. And presenting information on the real object to the user. In this case, in AR, the user is allowed to select a real object without contact. Therefore, in AR, a real object that cannot be touched from the position of the user may be selected as an operation target.

  In Patent Document 1, a three-dimensional object corresponding to an operation device possessed by a user in the real space is displayed on the display screen, and the posture and position of the operation device in the real space are detected, and the detected posture and position are detected. A technique for changing the posture and position of a three-dimensional object based on the above is disclosed.

Japanese Patent No. 4085918

  However, in the conventional technology related to AR, when a user tries to select a certain real object as an operation target, the user cannot recognize whether or not the operation being performed is approaching the real object to be selected. There was a problem. Hereinafter, an example in which this problem occurs will be described with reference to FIG.

  FIG. 13A shows a state when the user starts a selection operation for selecting a certain real object as an operation target, and FIG. 13B shows a state when the selection operation is completed. In FIG. 13A, the user is trying to select a real object 304 using his / her hand 301. The real object 303 is the currently selected real object. Therefore, the contour of the real object 303 is emitted with high brightness. As the real object currently selected, for example, the real object closest to the user's hand 301 is selected.

  An arrow 302 represents a direction in which the user should move the hand 301 in order to select the real object 304. In the prior art, information that allows the user to confirm the user's operation as indicated by an arrow 302 has not been displayed on the screen. Therefore, when the user moves the hand 301, the user needs to recognize the real object whose outline is emitted, and to move the hand 301 by estimating the direction of the arrow 302 based on the recognition result.

  If the direction of the arrow 302 is well estimated and the hand 301 can be moved in that direction, the user can select the real object 304 as shown in FIG. In this case, the contour of the real object 304 is emitted with high brightness, and the user recognizes that the intended real object has been selected.

  As described above, the conventional technique has a problem that the user cannot recognize whether the direction of the hand 301 matches the arrow 302 during the selection operation, and the selection operation is difficult.

  In particular, when other real objects are concentrated in the real object to be selected, the selected real object is changed due to a slight difference in the direction of the hand 301, which makes the selection operation more difficult. On the other hand, when the arrow 302 is displayed on the screen and the user is notified of the moving direction of the hand 301, the display of the real object displayed on the screen is obstructed. Further, the arrow symbol is not suitable for the operation of selecting the real object by the action of putting the hand on the real object and grasping it in the field of view.

  In the method of Patent Document 1, no measures are taken to improve operability when performing an operation of selecting one real object from a plurality of real objects.

  An object of the present invention is to improve operability when a user performs an operation of selecting an actual object to be operated.

  (1) A user interaction device according to the present invention is a user interaction device in which a user selects a real object in a non-contact manner, a user view acquisition unit that acquires video data of the user's view, and a state of the user's hand And a user operation definition holding unit that holds user operation definition information that defines the operation type corresponding to the hand state, and the position coordinates of the hand in the real space of the user from the video data acquired by the user view acquisition unit The user's hand state, and a user action monitoring unit that identifies an operation type corresponding to the user's current hand state with reference to the user operation definition information; Included in the video data based on the real object information, a real object information holding unit that holds real object information indicating information about real objects in the real space in advance A real object identifying unit for identifying a real object that can be operated from among the real objects, a position coordinate of each real object identified by the real object identifying unit in the real space, and the user action monitoring unit. A priority calculation unit that calculates the priority of each real object identified by the real object identification unit based on the current hand position coordinates of the user, and the operation type specified by the user action monitoring unit Is a display effect setting unit that sets the degree of display effect that is predetermined according to the priority to each real object when the user indicates the type of operation being selected before the user selects a real object to be operated A real object effect applying unit that applies a display effect according to the degree of display effect set by the display effect setting unit to each real object, and a display effect that is applied to each real object by the real object effect applying unit. In the video data And a superimposed display output section for tatami displayed.

  A user interaction method according to the present invention is a user interaction method in which a user selects a real object in a non-contact manner, wherein a user interaction device acquires a user field of view image data of the user's field of view, and a user interaction device includes: , Monitoring the position coordinates of the user's hand in the real space and the state of the user's hand from the video data acquired by the user field of view acquisition step, and selecting an operation type corresponding to the current state of the user's hand A user operation monitoring step that identifies the user's hand state and an operation type corresponding to the hand state with reference to user operation definition information, and the user interaction device Included in the video data based on real object information indicating information about the real object. A real object identification step for identifying a real object that can be operated from among the real objects; and a user interaction device that detects the position coordinates of each real object identified in the real object identification step in the real space, and the user motion monitoring. A priority calculation step of calculating a priority of each real object identified in the real object identification step based on the position coordinates of the current hand of the user monitored in the step, and a user interaction device comprising: When the operation type specified by the action monitoring step indicates the operation type being selected before the user selects an actual object to be operated, the degree of display effect determined in advance according to the priority A display effect setting step for setting an actual object and a user interaction device that has the display effect set by the display effect setting step. A real object effect imparting step for imparting a display effect according to the degree to each real object, and a user interaction device for superimposing and displaying the display effect imparted to each real object by the real object effect imparting step on the video data A display output step.

  A user interaction program according to the present invention is a user interaction program that causes a computer to function as a user interaction device that allows a user to select a real object in a non-contact manner, and includes the user's hand state and an operation type corresponding to the hand state. A user operation definition holding unit that holds the defined user operation definition information, and from the video data indicating the user's field of view, the position coordinates of the user's hand in the real space and the state of the user's hand are monitored, and the user A user action monitoring unit that specifies an operation type corresponding to the current hand state with reference to the user operation definition information, and real object information indicating information about a real object in the real space that can be operated by the user in advance. A real object information holding unit for holding the real object included in the video data based on the real object information; A real object identifying unit for identifying a real object that can be operated from the inside, a position coordinate of each real object identified by the real object identifying unit in the real space, and the user monitored by the user motion monitoring unit Based on the current hand position coordinates, a priority calculating unit that calculates the priority of each real object identified by the real object identifying unit, and an operation type specified by the user action monitoring unit is the user Indicates an operation type being selected before selecting a real object to be operated, a display effect setting unit that sets a degree of display effect predetermined according to the priority to each real object, and the display A display effect according to the degree of display effect set by the effect setting unit is given to each real object, and the computer is caused to function as a display effect giving unit that superimposes the display effect given to each real object on the video data.

  An integrated circuit according to the present invention is an integrated circuit of a user interaction device that allows a user to select a real object in a non-contact manner, a user view acquisition unit that acquires image data of the user's view, and the state and hand of the user's hand. A user operation definition holding unit that holds user operation definition information that defines an operation type corresponding to the state of the user, and from the video data acquired by the user field of view acquisition unit, the position coordinates of the hand in the real space of the user and the A user motion monitoring unit that monitors a user's hand state and identifies an operation type corresponding to the user's current hand state with reference to the user operation definition information; and a real space operable by the user A real object information holding unit that holds real object information indicating information on the real object in advance, and from among the real objects included in the video data based on the real object information A real object identification unit for identifying a real object that can be created, a position coordinate of each real object identified by the real object identification unit in the real space, and a current state of the user monitored by the user action monitoring unit Based on the position coordinates of the hand, the priority calculating unit that calculates the priority of each real object identified by the real object identifying unit, and the operation type specified by the user action monitoring unit are operated by the user. In the case of indicating the type of operation being selected before selecting a target real object, a display effect setting unit that sets a degree of display effect predetermined according to the priority to each real object, and the display effect setting A real object effect applying unit that applies a display effect according to the degree of display effect set by the unit to each real object, and a display effect that is applied to each real object by the real object effect applying unit is superimposed on the video data. Overlay table And an output unit.

  According to these configurations, the priority is calculated for each real object based on the distance between the position coordinate of the user's hand and the position coordinate of the real object in the user's field of view. When the user performs an operation of selecting a certain real object as an operation target, the degree of display effect corresponding to the priority is set for each real object, and each real object is displayed.

  Therefore, the user can grasp the relationship between the position coordinates of his / her hand and the position coordinates of the real object from the degree of display effect. As a result, the user can perform an operation of selecting a target real object while correcting the current operation from the degree of display effect given to the real object to be selected, thereby improving operability. be able to.

  (2) It is preferable that the said user visual field acquisition part is a camera with which the head mounted display with which the said user wears is equipped.

  According to this configuration, video data of the user's field of view can be easily acquired.

  (3) The user view acquisition unit recognizes the user's line of sight from a plurality of fixed point cameras and video data captured by the fixed point camera, and acquires the video data of the user's view based on the recognized line of sight. It is preferable to include a video data acquisition unit.

  According to this configuration, it is possible to acquire video data of the user's field of view without attaching the camera to the user.

  (4) It is preferable that the user operation monitoring unit monitors a state of the user's hand from an image of the user's hand included in the video data acquired from the user view acquisition unit.

  According to this configuration, since the state is monitored from the hand image included in the video data acquired by the user view acquisition unit, there is no need to provide a sensor for monitoring the hand state, and the device configuration is simplified. Can be

  (5) It is preferable that the user operation monitoring unit monitors the state of the user's hand using information acquired from a sensor attached to the user's hand.

  According to this configuration, since the state of the user's hand is monitored using the sensor, the state of the user's hand can be accurately monitored.

  (6) The real object identifying unit further identifies the type of the real object that can be operated, and the display effect setting unit sets a display effect predetermined for each type of the real object that can be operated to each real object. It is preferable to do.

  According to this configuration, the user can recognize the type of the real object from the video data.

  (7) When the operation type specified by the user action monitoring unit indicates that the operation is being performed, the display effect setting unit extracts a predetermined number of real objects in descending order of priority, and extracts the extracted real objects. On the other hand, it is preferable to set the degree of display effect.

  According to this configuration, since the degree of display effect is set only for real objects with high priority, it is possible to notify the positional relationship with the user's hand only for real objects that are highly likely to be selected by the user. . As a result, only important information can be notified to the user, and unnecessary information can be eliminated to further improve operability.

  (8) When the operation type specified by the user action monitoring unit indicates that the operation is being performed, the display effect setting unit sets a degree of display effect that causes the brightness of the contour to be increased as the real object with higher priority is emitted. It is preferable to set.

  According to this configuration, the user can recognize that the current operation is correct when the contour of the real object desired to be selected emits light with high luminance. Further, when the contour of the real object desired to be selected emits light with low luminance, it can be recognized that the real object can be selected by moving the hand toward the real object. As a result, it is possible to intuitively notify the user of the positional relationship between the hand and the actual object desired to be selected.

  (9) When the operation type specified by the user action monitoring unit indicates the operation type in which the user has determined the real object to be operated, the display effect setting unit has a display effect level only for the real object. Is preferably set.

  According to this configuration, when the user determines a real object, the degree of display effect is set only for the real object, so the user can quickly recognize which real object the user interaction device has recognized as an operation target. can do.

  According to the present invention, the user can perform an operation of selecting a target real object while correcting the current operation from the degree of the display effect given to the real object, thereby improving operability. Can do.

It is a block diagram which shows the structure of the user interaction apparatus by embodiment of this invention. It is a flowchart which shows the process of the user interaction apparatus in embodiment of this invention when a user performs operation in selection. It is the figure which showed an example of the state and position coordinate of a user's hand in each immediately before and now. It is the figure which showed an example of user operation definition information. It is the schematic diagram which showed the calculation method of the priority in an XY coordinate system. It is the schematic diagram which showed the calculation method of the priority in Z coordinate system. It is the table | surface which put together the position coordinate of the real object shown in FIG. 5, FIG. It is the table | surface which put together the position coordinate of the hand shown to FIG. 5, FIG. (A) is a table summarizing the priorities of the XY coordinate system calculated for the real objects shown in FIGS. 5 and 6, and (b) is a table for the real objects shown in FIGS. 5 and 6. It is the table | surface which put together the calculated priority of the Z coordinate system. It is the table | surface which put together the comprehensive priority calculated with respect to the real object shown in FIG. 5, FIG. It is the figure which showed an example of display effect information. It is the table | surface which put together the degree of the display effect and display effect which the display effect setting part set with respect to the real object. (A) shows a state when the user starts a selection operation for selecting a certain real object as an operation target, and (b) shows a state when the selection operation is completed.

  Hereinafter, a user interaction device according to an embodiment of the present invention will be described with reference to the drawings. This user interaction device allows a user to select non-contact by performing an operation of grasping one real object from a plurality of real objects that are within the user's field of view and are not reachable by the user. Device.

  FIG. 1 is a block diagram showing a configuration of a user interaction device according to an embodiment of the present invention. The user interaction device shown in FIG. 1 includes a control unit 101 and an input / output unit 110. The control unit 101 includes a user action monitoring unit 102, a priority calculation unit 103, a real object effect providing unit 104, a display effect setting unit 105, a user operation definition holding unit 106, a real object identification unit 107, a real object information holding unit 108, And a display effect information holding unit 109. Here, the control unit 101 is configured by a microcomputer including, for example, a CPU, a ROM, a RAM, and the like, and the function of each block is realized by the CPU executing a user interaction program.

  The input / output unit 110 includes, for example, an HMD, and includes a user view acquisition unit 111 and a superimposed display output unit 112. Note that the input / output unit 110 and the control unit 101 may be configured as a single chip and configured as an integrated circuit, or the control unit 101 and the input / output unit 110 may be configured as a single chip and configured as individual integrated circuits.

  The HMD is equipped with a camera function and can acquire an image of a real environment reflected in the user's field of view. Here, as the HMD, a see-through HMD that superimposes and displays a virtual image on the image of the user's field of view in the real space is adopted.

  Moreover, you may implement | achieve the control part 101 as a computer connected to HMD via the network among the structures shown in FIG. 1, or the computer built in HMD. At this time, these functional blocks receive the video data acquired from the user view acquisition unit 111 as input, and output the superimposed image output unit 112 with virtual image data superimposed on the real space video data.

  The user view acquisition unit 111 includes, for example, an image sensor such as a depth sensor or a stereo camera attached to the HMD, and a processing circuit that processes the image data acquired by the image sensor, and stores the image data of the user's view in a predetermined manner. Get at a frame rate of. The user view acquisition unit 111 may be configured by a plurality of fixed point cameras instead of the image sensor attached to the HMD. Here, the fixed point camera is a camera fixed at a predetermined location in the real space.

  In this case, the user view acquisition unit 111 may recognize the user's line of sight from the video data acquired by the fixed point camera, and generate video data indicating the user's view from the recognized line of sight. Specifically, the user view acquisition unit 111 extracts a user's silhouette from each video data acquired by the fixed point camera. Then, the direction of the line of sight of the user in the real space is recognized from the extracted user silhouette.

  Then, the direction of the optical axis of each fixed-point camera when the recognized line-of-sight direction is used as a reference is specified. Then, a virtual screen having the same size as the user's field of view is set around the line of sight, and the video data acquired by each fixed-point camera is arranged on the virtual screen with reference to the direction of the optical axis with respect to the line of sight. Video data may be generated.

  The user motion monitoring unit 102 monitors the position coordinates of the user's hand in the real space from the video data acquired by the user view acquisition unit 111. Here, the user operation monitoring unit 102 is supplied with video data from the user view acquisition unit 111 at a predetermined frame rate, and performs a process of extracting feature points of a hand included in each video data.

  Specifically, first, the user action monitoring unit 102 performs a process of extracting feature points of real objects included in the video data. For example, the feature point is a point on the contour of the real object, which characterizes the shape of the real object such as a vertex or a point having a large curvature. Then, the user operation monitoring unit 102 clusters the feature points extracted from the video data, divides the feature points into a plurality of feature point groups, and extracts the feature points of the hand from the feature point arrangement pattern in each specific point group. In this case, the user action monitoring unit 102 holds in advance a reference arrangement pattern of the hand, which is an arrangement pattern of the feature points serving as a reference for the hand, and arrangement of the feature points extracted from the reference arrangement pattern of the hand and the video data. A feature point of the hand may be extracted by comparing the pattern.

  Then, the user action monitoring unit 102 specifies a representative point of the hand from the extracted feature points of the hand. Here, for example, the center of gravity of the hand can be employed as the representative point. Then, the user operation monitoring unit 102 obtains the position coordinates of the representative point of the hand in the real space as the position coordinates of the hand.

  Here, when a depth sensor is employed as the user view acquisition unit 111, the user motion monitoring unit 102 uses the horizontal and vertical coordinates of the representative point of the hand and the representative of the hand in the distance image data acquired by the depth sensor. What is necessary is just to obtain | require the three-dimensional position of a representative point from the depth coordinate of a point.

  When a stereo camera is employed as the user view acquisition unit 111, the user operation monitoring unit 102 extracts feature points of the hand from the video data acquired from one camera, and representative points are extracted from the extracted feature points. A corresponding point indicating the same location as the extracted representative point is extracted from the video data of the other camera. Then, the user action monitoring unit 102 obtains the depth coordinates of the representative point of the hand using the parallax between the representative point of the hand and the corresponding point. Then, the user motion monitoring unit 102 may calculate the position coordinates of the representative point in the real space from the obtained depth coordinates of the representative point and the vertical and horizontal coordinates of the representative point in the video data.

  In addition, the user action monitoring unit 102 monitors the state of the user's hand in the real space from the video data acquired by the user view acquisition unit 111.

  In the present embodiment, there are “release”, “grasping”, and “none” as the hand state. Therefore, the user action monitoring unit 102 monitors the hand state by determining which of the above three states the arrangement pattern of the feature points of the hand extracted from the video data indicates. Here, “release” indicates, for example, a state where the user's hand is naturally open. In this case, since the distance between the tip of the thumb of the user's hand and the tip of the index finger is a predetermined distance or more, the user operation monitoring unit 102 displays video data of a feature point indicating the tip of the index finger and a feature point indicating the tip of the thumb. The above distance or the distance in the real space is obtained, and when the tip of the index finger and the tip of the thumb are separated by a certain distance or more, it is determined that the hand state is “released”.

  The user operation monitoring unit 102 may detect the state of the user's hand by attaching a marker (an example of a sensor) to the finger of the user's hand and detecting the position of the marker. In this case, for example, markers are respectively attached to the user's thumb and index finger. Then, the user operation monitoring unit 102 may detect the position of the marker from the video data and detect the state of the user's hand from the distance of the marker in the video data. Here, as the marker, for example, a light emitter such as an LED, a white plate, or the like may be employed.

  Further, “grip” indicates a state where the user's hand has grasped something. In this case, since the tip of the thumb of the user's hand and the tip of the index finger come within a certain distance, the user operation monitoring unit 102 determines that the distance between the tip of the index finger and the tip of the thumb is within a certain distance. What is necessary is just to determine with a state being "holding."

  “None” indicates a state in which the state of the hand does not correspond to either “release” or “grip”. Specifically, this is a case where the user's hand monitoring point 102 is not included in the video data of the user's hand and the feature point of the hand cannot be extracted.

  Then, the user operation monitoring unit 102 refers to the user operation definition information held in the user operation definition holding unit 106 for the operation type corresponding to the current hand state of the user from the monitoring result of the hand state. Identify.

  FIG. 4 is a diagram illustrating an example of user operation definition information. As shown in FIG. 4, the user operation definition information includes columns of “operation type”, “hand state change”, and “absolute value of hand position coordinate difference”. The “operation type” column stores an operation type that can be identified by the user interaction apparatus. Here, the operation type is the type of operation that the user performs on the real object.

  In the example of FIG. 4, there are four operation types, “selected”, “decided”, “range designation”, and “cancel”. “Selecting” is an operation before the user selects one of the real objects existing in his / her field of view as an operation target, that is, an operation for selecting the real object to be operated. . “Determine” is an operation in which the user selects an actual object to be operated, that is, an operation in which the actual object to be operated is confirmed. “Range designation” is an operation indicating that the user starts the “selected” operation. Specifically, the “range designation” corresponds to an operation in which the user enters his / her hand from outside the angle of view of the user view acquisition unit 111 into the angle of view.

  “Release” is an operation for releasing a real object determined as an operation target by the user performing “decision” operation from the operation target.

  The “hand state change” column stores information indicating how the hand state has changed from immediately before to the present. Here, the user operation monitoring unit 102 determines the state of the user's hand at a certain period (for example, a frame period for acquiring video data). Therefore, “present” indicates the latest determination result of the hand state determined by the user action monitoring unit 102. Further, “immediately before” indicates a determination result when the user action monitoring unit 102 determines the hand state immediately before the “current” determination result. Therefore, “release → release” indicates that the previous hand state is release and the current hand state is release. “Release → Grip” indicates that the previous hand is released and the current hand is held. “None → release” indicates that the previous hand state is none and the current hand state is release. Further, “grasping → release” indicates that the state of the immediately preceding hand is grabbing and the current state of the hand is released.

  The “absolute value of hand position coordinate difference” column stores a condition regarding a difference in hand position coordinate from immediately before to the present. A condition “greater than 5 cm in any of the X, Y, and Z axis directions” is stored for the selected item. Here, the X, Y, and Z axes are coordinate axes for defining each position in the real space, and are orthogonal to each other. Specifically, the X axis is a coordinate axis parallel to the horizontal direction (lateral direction) of the rectangular video data. The Y axis is a coordinate axis parallel to the vertical direction (vertical direction) of the video data. The Z-axis is a coordinate axis that defines a direction orthogonal to the sensor surface of the image sensor included in the user view acquisition unit 111, that is, a depth coordinate of the user's view. Here, as the origin of the X, Y, and Z axes, for example, the center position of the video data is adopted.

  When the user performs the “selecting” operation, the user performs an operation of moving the hand to the target real object with the hand naturally opened. Therefore, the user motion monitoring unit 102 determines that the hand state change is “released → released”, and the position coordinates of at least one coordinate axis among the absolute values of the position coordinates of the X, Y, and Z axes of the hand position coordinates. If the absolute value of the difference is greater than 5 cm, it is determined that the operation type is being selected.

  When the user performs a “decision” operation, an operation is performed in which a hand is placed on the real object in the field of view and the finger is bent at that position to grasp the real object. Therefore, the user action monitoring unit 102 indicates that the hand state change is “release → grip”, and the absolute value of the position coordinate difference is the absolute value of the X, Y, Z axis position coordinate differences of the hand position coordinates. When all are 0 cm or more and 5 cm or less, it determines with an operation classification being determination.

  Here, the condition of 5 cm is assumed to be a shake of the user's hand. That is, the user wears this user interaction device and performs an operation of selecting a real object while walking in the city. In this case, even if the user stops the hand, the hand is shaken by about 5 cm due to the vibration of the walking motion. Therefore, in this embodiment, 5 cm is adopted as a condition. However, it is assumed that the hand shake when walking is different depending on the physical characteristics of the user. In addition, a user who stops walking when operating the user interaction apparatus and operates only in a stopped state is also assumed. A user who operates the user interaction apparatus only indoors is also assumed. Therefore, 5 cm is an example, and the user may be able to appropriately set a preferable value according to the user's physical characteristics and usage with respect to the user interaction device.

  When the user performs the “range specification” operation, the user performs an operation of intruding a hand from outside the angle of view of the user view acquisition unit 111 into the angle of view. Therefore, when the hand state change is “None → Release” and the absolute value of the difference between the position coordinates of the X, Y, and Z axes of the hand has not been calculated, Is determined to be a range specification. Here, in the case of “range specification”, “absolute value of hand position coordinate difference” is set to “none” because the position coordinate of the immediately preceding hand has not been calculated. This is because the value cannot be obtained.

  When the user performs a “release” operation, the user performs an operation of opening the finger from a state in which the finger is closed in order to grab a real object. Therefore, the user action monitoring unit 102 determines that the hand state change is “grasp → release” and the absolute values of the difference in the position coordinates of the X, Y, and Z axes of the hand position are all 0 cm or more and 5 cm or less. It is determined that the operation type is release.

  As described above, the user operation monitoring unit 102 determines the user operation type and updates the position coordinates of the user's hand while monitoring the state of the user's hand throughout.

  Returning to FIG. 1, the real object identification unit 107 is based on the real object information held in the real object information holding unit 108, and the real object in the real space that can be operated by the user from the real objects included in the video data. Identify

  Specifically, the real object identification unit 107 is supplied with video data from the user view acquisition unit 111 at a predetermined frame rate, extracts feature points of real objects included in each video data, and clusters the extracted feature points. Then, it is divided into a plurality of feature point groups, and an arrangement pattern of feature points in each feature point group is obtained. Then, the real object identification unit 107 reads a reference arrangement pattern, which is an arrangement pattern of feature points serving as a reference of an operable real object, from the real object information holding unit 108, and an arrangement pattern extracted from the reference arrangement pattern and video data Is compared to identify the real object that can be manipulated.

  The real object information holding unit 108 holds in advance real object information indicating information related to an operable real object. In the present embodiment, for example, the real object information includes a predetermined reference arrangement pattern and type of a real object that can be operated. As the type, for example, information indicating the type of a real object such as a building, a car, or a signboard is employed.

  The priority calculation unit 103 is based on the position coordinates of the real object identified by the real object identification unit 107 in the real space and the position coordinates of the current hand of the user monitored by the user action monitoring unit 102. The priority of the real object identified by the body identifying unit 107 is calculated.

  Here, the priority calculation unit 103 obtains the center of gravity of each real object from the arrangement pattern of the feature points of each real object extracted by the real object identification unit 107, and calculates the three-dimensional position coordinates of the center of gravity in the real space. Calculated as the position coordinates of each real object. In this case, the priority calculation unit 103 may obtain the position coordinates of the center of gravity of each real object using the same method as the method in which the user action monitoring unit 102 calculates the position coordinates of the representative point of the user's hand.

  And the priority calculation part 103 is the position coordinate of a user's hand in each of the XY coordinate system prescribed | regulated by the X-axis and a Y-axis, and the Z coordinate system prescribed | regulated by the Z-axis, and a real object The distance to each real object identified by the identification unit 107 is obtained, and the priority is calculated in order of increasing distance. And the priority calculation part 103 calculates | requires the total value of the priority of an XY coordinate system, and the priority of a Z coordinate system, and calculates | requires the total priority of each real object.

  Further, the priority calculation unit 103 may calculate the priority only for the real objects identified by the real object identification unit 107 within a predetermined distance from the position of the user's hand. In this way, the number of real objects for which priority is calculated is limited, and the processing speed can be increased.

  When the operation type specified by the user action monitoring unit 102 indicates the “selected” operation type, the display effect setting unit 105 displays the degree of display effect determined in advance according to the priority as the real object identification unit 107. Set for each real object identified by.

  Specifically, the display effect setting unit 105 specifies the display effect type corresponding to the operation type specified by the user action monitoring unit 102 from the display effect information held in the display effect information holding unit 109.

  FIG. 11 is a diagram illustrating an example of display effect information. The display effect information includes columns of “operation type”, “display effect type”, and “degree of display effect”. In the “operation type” column, four operation types specified by the user action monitoring unit 102 are stored. Since these four operation types have been described above, a description thereof will be omitted.

  In the display effect type column, the display effect type corresponding to each operation type is stored. In the example of FIG. 11, “light emission of the contour of the real object” is defined as the type of display effect for each of “selected”, “decision”, and “range designation”. Therefore, when an operation shown in these three operation types is performed by the user, the display effect setting unit 105 sets a display effect for emitting the outline of the corresponding real object.

  In addition, “swing a real object” with respect to “release” is defined as the type of display effect. Therefore, when the “release” operation is performed by the user, the display effect setting unit 105 sets a display effect that vibrates the corresponding real object for a certain period of time.

  In the example of FIG. 11, “light emission with strong, medium, and weak degrees” is stored as the “degree of display effect” corresponding to “selected” for the three real objects with the highest priority. Has been. Therefore, when the “selected” operation is performed, the display effect setting unit 105 extracts three real objects in descending order of priority calculated by the priority calculation unit 103, and the three real objects are extracted. Then, the degree of the display effect that causes the contour to emit light with strong, medium, or weak luminance is set.

  As a result, in the video displayed on the superimposed display output unit 112, the contours of the three real objects are emitted with strong, medium, and weak luminances in the order from the position coordinate of the user's hand, and the user The relationship with the position of the real object can be grasped from the video data. Therefore, when the contour of the real object to be selected emits light with strong luminance, the real object can be selected by performing an operation of holding and grasping the hand. Further, when the contour of the real object to be selected emits light with medium or weak luminance, the real object can be selected by moving the hand toward the real object.

  Further, when the contour of the real object to be selected is not illuminated, the user finds the real object closest to the real object to be selected among the real objects whose contours are illuminated, and toward the real object. By moving the hand, the hand can be brought close to the real object to be selected.

  As described above, by emitting the contour of the real object with the luminance according to the priority, the user can select the real object while confirming his / her operation, and the operability can be improved.

  Here, as a display mode for emitting the contour of the real object, for example, a display mode in which graphics obtained by tracing the contour of the real object with a predetermined color are superimposed on the real object may be employed. When light is emitted with high, medium, and weak luminance, the luminance of graphics that traces the contour may be emitted with predetermined luminance for each of strong, medium, and weak.

  In the example of FIG. 11, the contours of three real objects are displayed with strong, medium, and weak luminances. However, this is an example, and three real objects are displayed in different colors (for example, red, The relationship between the position of the hand and the position of the real object may be notified to the user. For example, when displaying a real object in red, image processing for highlighting the red component of each pixel may be performed on the image of the real object. Further, the red mark graphics may be superimposed and displayed with a predetermined transparency using α blending on the entire real object image.

  Moreover, you may employ | adopt the display effect which shakes three real objects. In this case, three real objects may be vibrated by setting at least one of the vibration frequency and amplitude to large, medium, and small in descending order of priority.

  Further, in the example of FIG. 11, when the operation type is “selected”, the contours of three real objects are emitted. However, the present invention is not limited to this, but other than three such as two, four, and five. The contours of a plurality of real objects may be emitted.

  In the example of FIG. 11, “light emission with a strong degree with respect to the determined real object” is stored as the “degree of display effect” corresponding to the operation type “decision”.

  Therefore, when the operation type specified by the user action monitoring unit 102 is “determine”, the display effect setting unit 105 displays the display effect of causing the outline to emit light with strong luminance only for the real object selected by the user. Set the degree. Here, the high luminance may be the same luminance as the high luminance when the operation type is being selected, or may be a luminance higher than that. In any case, when the operation type is “decide”, only the contour of one real object selected by the user is emitted, so that the user can recognize that the “decision” operation has been performed.

  In the example of FIG. 11, “light emission with a weak degree for all real objects” is stored as the “degree of display effect” corresponding to the operation type of “range specification”. Therefore, the display effect setting unit 105 sets the degree of display effect that causes the contour to emit light with low luminance for all the real objects identified by the real object identification unit 107. In this case, the user can recognize which real object among the real objects included in the video data is an operable real object. When the operation type is “range specification”, the display effect setting unit 105 weakens the contour of the real object located within a predetermined distance from the operable real object closest to the position of the user's hand. You may set the degree of the display effect made to light-emit by intensity | strength.

  In this way, when there are multiple real objects in the user's field of view, only the contours of the real objects that the user is likely to select are emitted with low brightness, improving operability. Can be made.

  Further, in the example of FIG. 11, “shake with a slight degree against the real object that has been deselected” is stored as the “degree of display effect” corresponding to the operation type “cancel”. Therefore, the display effect setting unit 105 sets the degree of the display effect that causes the real object whose selection has been canceled to emit light with low luminance and to be shaken. Here, “light emission with weak luminance” means that the luminance of the real object that has been deselected is reduced overall in the video data displayed by the superimposed display output unit 112, or the real object that has been deselected. This corresponds to displaying the graphics of the outline superimposed on the body with low luminance.

  Returning to FIG. 1, the display effect information holding unit 109 holds the display effect information shown in FIG. The real object effect imparting unit 104 imparts a display effect according to the degree of the display effect set by the display effect setting unit 105 to each real object whose priority is calculated by the priority calculating unit 103. Specifically, when emitting the contour of a certain real object, the real object effect imparting unit 104 generates graphics for causing the contour of the real object to emit light with a predetermined luminance, and the generated graphics is converted into video data. By outputting to the superimposed display output unit 112 a drawing command to be superimposed on the contour of the real object, a display effect is given to the real object. Further, when the real object effect imparting unit 104 shakes a certain real object, the image data of the real object is extracted from the video data, and a drawing command for shaking and displaying the extracted image data is displayed on the superimposed display output unit 112. By outputting, a display effect is given to the real object.

  The superimposed display output unit 112 includes, for example, a display panel, a drawing RAM, a processing circuit that writes video data to the drawing RAM and displays the image data on the display panel, and the display provided to each real object by the real object effect applying unit 104. The effect is superimposed on the video data. Specifically, the superimposed display output unit 112 writes the video data to the drawing RAM at a predetermined frame rate, and the graphic generated by the real object effect applying unit 104 according to the drawing command output from the real object effect applying unit 104. The graphics are superimposed on the video data by writing the graphics at the corresponding positions in the drawing RAM.

  FIG. 2 is a flowchart showing processing of the user interaction device in the embodiment of the present invention when the user performs an operation being selected.

  In FIG. 2, steps S (hereinafter abbreviated as “S”) 201 to S206 are processes performed by the user operation monitoring unit 102.

  First, the user motion monitoring unit 102 recognizes the user's hand from the video data of the user's field of view acquired by the user field of view acquiring unit 111, and acquires the position coordinates of the hand in the real space (S201).

  Next, the user operation monitoring unit 102 acquires the state of the user's hand from the hand feature point arrangement pattern extracted from the video data (S202). As described above, there are “release”, “grip”, and “none” as the hand state.

  Next, the user operation monitoring unit 102 specifies a change in the hand state (S203). Here, the user operation monitoring unit 102 identifies the hand state immediately before and the present as the hand state change. In this example, since the user is performing the “selecting” operation, “release → release” is specified as the hand state change. Next, the user operation monitoring unit 102 calculates the absolute value of the difference between the hand position coordinates immediately before and the current time (S204).

  Next, the user operation monitoring unit 102 refers to the user operation definition information shown in FIG. 4 from the change in the hand state specified in S203 and the absolute value of the difference in the hand position coordinates calculated in S204, and The current operation type is determined (S205).

  FIG. 3 is a diagram illustrating an example of a user's hand state and position coordinates immediately before and at present. In FIG. 3, a hand 401 indicates the user's hand immediately before, and a hand 403 indicates the current user's hand.

  Here, the position coordinate 402 of the hand 401 is (X, Y, Z) = (5 cm, 10 cm, 30 cm), and the position coordinate 404 of the hand 403 is (X, Y, Z) = (8 cm, 20 cm, 32 cm). Suppose that Further, it is assumed that the state of the hand 401 is “released” and the state of the hand 403 is “released”.

  In this case, the user action monitoring unit 102 indicates that the hand state change is “release → release”, and the absolute value (405) of the difference between the hand position coordinates 402 and 404 (= | position coordinates 402−position coordinates 404 |). However, (X, Y, Z) = (3 cm, 10 cm, 2 cm). Therefore, the user operation monitoring unit 102 refers to the user operation definition information illustrated in FIG. 4 and determines that the operation type of the current user is “selected”.

  Note that the user action monitoring unit 102 determines that if either one of the identified “hand state change” or the calculated “absolute value of hand position coordinate difference” corresponds to a pattern not defined in FIG. Is determined not to be operated, that is, no operation.

  In S206, the user operation monitoring unit 102 determines whether or not the user is performing an operation. Here, the user operation monitoring unit 102 determines that there is a user operation (YES in S206), and performs the process of S207, except in the case where it is determined in S205 that the user is not operating. On the other hand, if the user operation monitoring unit 102 determines in S205 that the user is not operating, it determines that there is no user operation (NO in S206), and returns the process to S201.

  S207 to S210 are processes performed by the real object identification unit 107. In step S <b> 207, the real object identification unit 107 determines whether or not the user's field of view has been changed due to the user's line of sight or body movement.

  This determination is performed by comparing the current video data acquired by the user view acquisition unit 111 with the previous video data. Specifically, the user view acquisition unit 111 obtains difference image data between the current image data and the immediately preceding image data, and the difference image data includes a predetermined number or more of pixels whose luminance is a certain value or more. In this case, it is assumed that the user's field of view has been changed (YES in S207), and the process of S208 is performed. On the other hand, when there is no change in the user's field of view (NO in S207), S208 and S209 are skipped, and the process of S210 is performed.

  In step S <b> 208, the real object identification unit 107 identifies a real object included in the video data acquired by the user view acquisition unit 111. Here, the real object identification unit 107 identifies a real object by extracting feature points from the video data and clustering the extracted feature points. Note that the real object identification unit 107 repeats the process of S208 as much as possible until, for example, the number of identified real objects reaches a predetermined number or until a predetermined timeout time elapses.

  In step S209, the real object identification unit 107 matches the reference arrangement pattern of each real object held in the real object information holding unit 108 with the arrangement pattern of the feature points extracted from the video data. Identify the body.

  S <b> 210 is processing performed by the priority calculation unit 103. In S210, the priority calculation unit 103 obtains the position coordinates of each real object identified in S209 in the real space, and uses each position of the real object and the position coordinates of the hand calculated in S201. The priority of is calculated.

  The processing performed in S210 will be described with reference to FIGS. FIG. 5 is a schematic diagram showing a priority calculation method in the XY coordinate system, and FIG. 6 is a schematic diagram showing a priority calculation method in the Z coordinate system. The priority calculation unit 103 calculates the priority of each real object by combining the determination methods in FIGS. 5 and 6.

  An area 501 illustrated in FIG. 5 indicates one piece of video data acquired by the user view acquisition unit 111. In the area 501, the horizontal coordinate axis is the X axis, the vertical coordinate axis is the Y axis, and the depth coordinate axis is the Z axis.

  Position coordinates P502 to P505 indicate the position coordinates of the real objects 502 to 505 identified by the real object identification unit 107. In the present embodiment, for example, the center of gravity of the real objects 502 to 505 is employed as the position coordinates P502 to P505. The position coordinate P506 indicates the position coordinate of the user's hand 506 acquired by the user action monitoring unit 102.

  The priority in the XY coordinate system is given in ascending order of the distance in the XY coordinate system between the position coordinate P506 of the hand 506 and the position coordinates P502 to P505 of the real objects 502 to 505.

  FIG. 7 is a table summarizing the position coordinates P502 to P505 of the real objects 502 to 505 shown in FIGS. FIG. 8 is a table summarizing the position coordinates P506 of the hand 506 shown in FIGS.

  7, ID 502 to ID 505 described in the real object column are identification information given to the real objects 502 to 505 shown in FIGS. In FIG. 8, ID 506 described in the column of the hand 506 is identification information given to the hand 506 shown in FIGS. 9, 10, and 12, ID 502 to ID 505 represent the identification information of the real objects 502 to 505 shown in FIGS. 5 and 6, and ID 506 represents the identification information of the hand 506 shown in FIGS. 5 and 6. To express.

  5 and 6, the position coordinates P502 to P505 of the real objects 502 to 505 have values of X, Y, and Z coordinates shown in FIG. 5 and 6, the position coordinate P506 of the hand 506 is assumed to have X, Y, and Z coordinate values shown in FIG.

  FIG. 9A shows the priority of the XY coordinate system calculated for the real objects 502 to 505 shown in FIGS. As illustrated in FIG. 9A, the real objects 502 to 505 are shorter in distance from the user's hand 506 in the order of the real objects 503, 505, 504, and 502.

  Therefore, the priority calculation unit 103 determines that the actual object is more likely to be selected as the operation target by the user as the distance is shorter, and sets a higher priority in order of increasing distance.

  In the example of FIG. 9A, since the distance between the real object 502 and the user's hand 506 is 18 cm, 4 which is the highest priority is set. Further, since the distance between the real object 503 and the user's hand 506 is 35 cm, 1 which is the lowest priority is set.

  In the example of FIG. 9A, since the number of manipulable real objects identified by the real object identifying unit 107 is four, priority is set for four real objects. The priority value varies depending on the number of operable real objects identified by the real object identification unit 107. For example, when the number of real objects that can be operated is n (n is a positive integer), priorities from n to 1 are set in order of increasing distance to each real object.

  Here, the priority calculation unit 103 may calculate the priority only for objects that are within a certain distance from the user among the operable real objects identified by the real object identification unit 107. Good. In addition, from the viewpoint of simplifying the priority calculation unit 103 and processing, for example, the priority is given only to the real object whose position coordinates in the Z coordinate system or the XY coordinate system exist within a certain range for the user. You may make it calculate.

  The priority calculation unit 103 calculates the priorities of the real objects 502 to 505 using the determination method shown in FIG. Specifically, the priority calculation unit 103 firstly selects the real object 503 farthest from the user among the real objects identified by the real object identification unit 107 and the real object 505 nearest to the user. The distance 601 in the Z coordinate is calculated. Next, the priority calculation unit 103 associates the distance 601 with the movable range 602 of the user's hand 506.

  Here, the movable range 602 employs a predetermined distance for each user, a position P701 that is a distance Z1 away from the position coordinate P506 of the hand 506 in the + Z-axis direction (depth side), and a −Z-axis direction (front side). ) Is determined in advance as a movable range 602 by a position P702 separated by a distance Z2. The distance Z1 is a value that is predetermined for each user based on the distance that the user is assumed to extend the hand 506 in the depth direction, and the distance Z2 is a distance that the user is assumed to extend the hand 506 in the forward direction. Based on the above, a value predetermined for each user is adopted.

  The priority calculation unit 103 sets a position 603 obtained by internally dividing the distance 601 by the ratio of the distance Z1 and the distance Z2 within the distance 601 as the position of the user's hand 506. Then, the priority calculation unit 103 obtains the distance on the Z axis from the position 603 for each of the real objects 502 to 505. And the priority calculation part 103 sets a high priority, so that the real object with which the calculated | required distance is short is short.

  In the present embodiment, the user can select a real object located outside the movable range of the hand 506 in a non-contact manner. In this case, if the priority of the Z coordinate system is calculated while completely ignoring the movable range 602 of the user's hand 506, the user feels uncomfortable with the operation. Therefore, in the present embodiment, by calculating the priority of the Z coordinate system in association with the distance 601 to the movable range 602, a sense of discomfort with respect to the user's operation is suppressed.

  FIG. 9B is a table summarizing the priorities in the Z coordinate system calculated for the real objects 502 to 505 shown in FIGS. 5 and 6. In FIG. 9B, the distance from the user's hand indicates the distance between the real objects 502 to 505 when the position 603 shown in FIG. 6 is used as a reference.

  In the example of FIG. 9B, the distance from the user's hand 506 is shorter in the order of the real objects 503, 505, 504, and 502. Therefore, priorities of 1, 2, 3, and 4 are set for the real objects 503, 505, 504, and 502.

  Next, the priority calculation unit 103 sums the priorities of the real objects 502 to 505 obtained in the XY coordinate system and the priorities of the real objects 502 to 505 obtained in the Z coordinate system. A total priority of 505 is calculated.

  FIG. 10 is a table summarizing the overall priorities calculated for the real objects 502 to 505 shown in FIGS. 5 and 6. As shown in FIG. 10, the real object 502 has a priority of 4 in the XY coordinate system and a priority of 4 in the Z coordinate system, and thus the total priority is calculated as 8. The priorities of other real objects 503 to 505 are calculated in the same manner, and the priorities are calculated as 2, 6, and 4, respectively. Therefore, the total priority of the real objects 502, 504, 503, and 505 is calculated as 4, 3, 2, and 1.

  Returning to FIG. 2, the processing from S211 to S213 will be described. S211 is processing performed by the display effect setting unit 105 and the display effect information holding unit 109.

  In S208, the display effect setting unit 105 refers to the display effect information shown in FIG. 11 for the display effect type and the display effect level corresponding to the current operation type of the user specified by the user action monitoring unit 102. And decide.

  Here, since the operation type is “selected”, the display effect setting unit 105 sets “light emission of the contour of the real object” as the display effect type, and the display effect degree is set in the order of “highest priority”. For a real object, set “strong, medium, weak”.

  FIG. 12 is a table that summarizes the display effects set by the display effect setting unit 105 for the real objects 502 to 505 and the degree of display effects. In this example, the priorities for the real objects 502 to 505 are 4, 2, 3, 1. Therefore, the real objects 502, 504, and 503 are set to “contour light emission / strong”, “contour light emission / medium”, and “contour light emission / weak”, respectively, as the display effect and display effect degree, and the real object 505 is displayed. None is set as the degree of the effect and the display effect.

  As described above, when the operation type is selected, three real objects 502, 504, and 503 are extracted from the real objects 502 to 505 in descending order of priority, and the real objects 502, 504, and 503 are extracted. Thus, the display effect of emitting light with strong, medium and weak outlines and the degree of display effect are set.

  Returning to FIG. 2, in S <b> 212, the real object effect imparting unit 104 imparts to the real objects 502, 504, and 503 the display effect and the degree of display effect set for the real objects 502, 504, and 503 in S <b> 210. . Here, a display effect and a degree of display effect are set for the real objects 502, 504, and 503 to emit light with strong, medium, and weak intensities, respectively. Therefore, the real object effect imparting unit 104 generates graphics indicating the contours of the real objects 502, 504, and 503. Then, the real object effect imparting unit 104 sets strong luminance for graphics indicating the contour of the real object 502, sets medium luminance for graphics indicating the contour of the real object 504, and sets the real object 503. Weak luminance is set for graphics showing outlines.

  Then, the real object effect imparting unit 104 outputs, to the superimposed display output unit 112, a drawing command for superimposing and displaying graphics indicating the contours of the real objects 502, 504, and 503 on the real objects 502, 504, and 503 in the video data. To do.

  In S213, the superimposed display output unit 112 displays the graphics corresponding to the contours of the real objects 502, 504, and 503 in the video data in a superimposed manner in accordance with the drawing command. Thus, the operation of the user interaction device is completed.

  In the above embodiment, the display effect and the degree of display effect are set for the real object. However, the present invention is not limited to the real object, and can be applied to a virtual object displayed on a display or the like. In this case, if the virtual object is handled as the real object and the above process is applied, the display effect and the degree of the display effect can be set for the virtual object.

  In the above-described embodiment, the user has described an operation of selecting one real object as an operation target. However, the present invention is not limited to this, and is applied to an operation of selecting a plurality of real objects. May be. Further, the present invention may be applied when notifying the user of one or more real objects that can be selected. In this case, the display effect and the degree of the display effect during the selection, that is, the display for emitting the outlines of a predetermined number of real objects with strong luminance in descending order of priority with respect to one or more real objects that can be selected. What is necessary is just to give the degree of an effect and a display effect.

  The present invention can also be applied to a case where a display effect and an effect degree of the display effect are set every time each operation is completed when one operation is completed by combining a plurality of operations.

  In the above-described embodiment, when the selected operation is performed by the user, the display mode in which the outline of the real object having a high priority is emitted is adopted. You may employ | adopt the display mode to alert | report. In this case, the real object identification unit 107 identifies the type of the identified real object from the type of the real object included in the real object information and notifies the display effect setting unit 105 when identifying the real object. Then, the display effect setting unit 105 may set a display effect for displaying the outline of the object in a predetermined color for each object type.

  Moreover, the said embodiment is an example and this invention is not limited to the said embodiment. Forms obtained by subjecting the above embodiments to modifications conceivable by those skilled in the art and other forms realized by arbitrarily combining the components in the embodiments are also included in the present invention.

  Note that each functional block of the user interaction device shown in FIG. 1 is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.

  The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. For example, you may implement | achieve each block shown in FIG. 1 using biotechnology. Of the blocks of the user interaction device, only the means for storing data may be configured separately instead of being integrated into one chip.

  In addition, the present invention can be realized not only as a user interaction device, but also as a method using a processing unit constituting the user interaction device as a step. The present invention can be realized as a program that causes a computer to execute the steps included in the method. Furthermore, the present invention can be realized as a computer-readable storage medium such as a CD-ROM storing the program.

  The user interaction device according to the present invention is useful for realizing a technique for selecting an actual object in a situation where the user cannot recognize a tactile sense, such as selecting an actual object without contact.

DESCRIPTION OF SYMBOLS 101 Control part 102 User action monitoring part 103 Priority calculation part 104 Real object effect provision part 105 Display effect setting part 106 User operation definition holding part 107 Real object identification part 108 Real object information holding part 109 Display effect information holding part 110 Input / output Unit 111 user view acquisition unit 112 superimposed display output unit

Claims (12)

A user interaction device in which a user selects a real object without contact,
A user view acquisition unit for acquiring video data of the user's view;
A user operation definition holding unit for holding user operation definition information defining the user's hand state and the operation type corresponding to the hand state;
From the video data acquired by the user view acquisition unit, the position coordinates of the user's hand in the real space and the state of the user's hand are monitored, and the operation type corresponding to the current hand state of the user is determined. A user operation monitoring unit that identifies the user operation definition information with reference to the user operation definition information;
A real object information holding unit that holds in advance real object information indicating information on real objects in the real space that can be operated by the user;
Based on the real object information, a real object identification unit for identifying a real object that can be operated from the real objects included in the video data;
Based on the position coordinates in the real space of each real object identified by the real object identification section and the current hand position coordinates of the user monitored by the user action monitoring section, the real object identification section A priority calculating unit that calculates the priority of each real object identified by
When the operation type specified by the user action monitoring unit indicates an operation type being selected before the user selects an actual object to be operated, the degree of display effect determined in advance according to the priority Display effect setting unit for setting each real object,
A real object effect applying unit that applies a display effect according to the degree of the display effect set by the display effect setting unit to each real object;
A user interaction device comprising: a superimposed display output unit configured to superimpose and display a display effect imparted to each real object by the real object effect imparting unit.   The user interaction device according to claim 1, wherein the user view acquisition unit is a camera provided in a head mounted display worn by the user. The user view acquisition unit
Multiple fixed-point cameras,
The user interaction device according to claim 1, further comprising: a video data acquisition unit that recognizes the user's line of sight from video data captured by the fixed point camera and acquires video data of the user's field of view based on the recognized line of sight. .   The user interaction according to claim 1, wherein the user action monitoring unit monitors a state of the user's hand from an image of the user's hand included in the video data acquired from the user view acquisition unit. apparatus.   The user interaction according to any one of claims 1 to 3, wherein the user action monitoring unit monitors a state of the user's hand using information acquired from a sensor attached to the user's hand. apparatus. The real object identification unit further identifies the type of real object that can be operated,
The user interaction device according to claim 1, wherein the display effect setting unit sets a display effect predetermined for each type of real object that can be operated on each real object.   The display effect setting unit extracts a predetermined number of real objects in descending order of priority when the operation type specified by the user action monitoring unit indicates the operation in progress, and displays the extracted real objects The user interaction device according to claim 1, wherein the degree of effect is set.   The display effect setting unit is configured to set a degree of a display effect that emits a higher brightness of an outline of a real object having a higher priority when the operation type specified by the user action monitoring unit indicates the operation in progress. Item 8. The user interaction device according to any one of Items 1 to 7.   The display effect setting unit sets the degree of display effect only for the actual object when the operation type specified by the user action monitoring unit indicates an operation type in which the user has determined an actual object to be operated. The user interaction device according to claim 1. A user interaction method in which a user selects a real object without contact,
A user interaction device, a user view acquisition step of acquiring video data of the user's view;
The user interaction device monitors the position coordinates of the user's hand in the real space and the state of the user's hand from the video data acquired by the user view acquisition step, and corresponds to the current hand state of the user A user operation monitoring step for specifying an operation type to be referred to by referring to user operation definition information that defines the user's hand state and the operation type corresponding to the hand state;
A real object identification step in which a user interaction device identifies a real object that can be operated from real objects included in the video data based on real object information indicating information on real objects in the real space that can be operated by the user. When,
The user interaction device is based on the position coordinates of each real object identified in the real object identification step in the real space and the position coordinates of the current hand of the user monitored in the user action monitoring step. A priority calculating step of calculating the priority of each real object identified by the real object identifying step;
When the user interaction device indicates an operation type being selected before the user selects an actual object to be operated, the operation type specified in the user action monitoring step is determined in advance according to the priority. Display effect setting step for setting the degree of display effect for each real object;
A real object effect applying step in which the user interaction device applies a display effect corresponding to the degree of the display effect set in the display effect setting step to each real object;
A user interaction method comprising: a superimposed display output step in which a user interaction device superimposes and displays the display effect imparted to each real object in the real object effect imparting step on the video data. A user interaction program that causes a computer to function as a user interaction device that allows a user to select a real object without contact,
A user operation definition holding unit for holding user operation definition information defining the user's hand state and the operation type corresponding to the hand state;
From the video data indicating the user's field of view, the position coordinates of the user's hand in the real space and the state of the user's hand are monitored, and the operation type corresponding to the current hand state of the user is determined by the user operation. A user action monitoring unit to identify and refer to the definition information;
A real object information holding unit that holds in advance real object information indicating information on real objects in the real space that can be operated by the user;
Based on the real object information, a real object identification unit for identifying a real object that can be operated from the real objects included in the video data;
Based on the position coordinates in the real space of each real object identified by the real object identification section and the current hand position coordinates of the user monitored by the user action monitoring section, the real object identification section A priority calculating unit that calculates the priority of each real object identified by
When the operation type specified by the user action monitoring unit indicates an operation type being selected before the user selects an actual object to be operated, the degree of display effect determined in advance according to the priority Display effect setting unit for setting each real object,
The computer is caused to function as a display effect applying unit that applies a display effect corresponding to the degree of the display effect set by the display effect setting unit to each real object and superimposes the display effect applied to each real object on the video data. User interaction program. An integrated circuit of a user interaction device in which a user selects a real object without contact,
A user view acquisition unit for acquiring video data of the user's view;
A user operation definition holding unit for holding user operation definition information defining the user's hand state and the operation type corresponding to the hand state;
From the video data acquired by the user view acquisition unit, the position coordinates of the user's hand in the real space and the state of the user's hand are monitored, and the operation type corresponding to the current hand state of the user is determined. A user operation monitoring unit that identifies the user operation definition information with reference to the user operation definition information;
A real object information holding unit that holds in advance real object information indicating information on real objects in the real space that can be operated by the user;
Based on the real object information, a real object identification unit for identifying a real object that can be operated from the real objects included in the video data;
Based on the position coordinates in the real space of each real object identified by the real object identification section and the current hand position coordinates of the user monitored by the user action monitoring section, the real object identification section A priority calculating unit that calculates the priority of each real object identified by
When the operation type specified by the user action monitoring unit indicates an operation type being selected before the user selects an actual object to be operated, the degree of display effect determined in advance according to the priority Display effect setting unit for setting each real object,
A real object effect applying unit that applies a display effect according to the degree of the display effect set by the display effect setting unit to each real object;
An integrated circuit comprising: a superimposed display output unit that superimposes and displays the display effect imparted to each real object by the real object effect imparting unit on the video data.

Images