JP2012181809A - Apparatus controller, apparatus control method, apparatus control program and integrated circuit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable gesture-based operation for an operation apparatus that does not have gesture registration, gesture recognition, or action execution functions.SOLUTION: When an apparatus selecting instruction is inputted by making a gesture of touching a certain operation apparatus with a finger, a function information acquiring section 506 acquires function information indicating a list of functions possessed by the operation apparatus from a function information storage section 507. A virtual UI generating section 508 generates a virtual UI based on the function information acquired by the function information acquiring section 506 and stores the generated virtual UI and the gesture made by the user at the time of inputting the apparatus selecting instruction in a virtual UI storage section 509 such that they are associated with each other. When the same gesture as the apparatus selecting instruction is made, a display superimposing section 511 superimposes the virtual UI corresponding to the gesture on image data for a real space so as to generate image data for an augmented reality space and causes a display 513 to display the generated image data.

Description

  The present invention relates to a technique for operating an operating device included in an augmented reality space by a user wearing a head mounted display.

  In recent years, with the development and spread of high-speed communication technology and the appearance of Wii (registered trademark) made by Nintendo and PLAYSTATION (registered trademark) MOVE made by Sony, new devices such as operating devices using gestures Games and applications that provide a user experience (hereinafter referred to as UX) are being actively developed.

  For example, in Nintendo's Wii, a function corresponding to a recognized gesture is recognized when an action corresponding to the user's gesture operation is registered in advance in the device so that the user actually recognizes the gesture. A user interface (hereinafter referred to as UI) is provided.

  FIG. 10 is a schematic diagram of the prior art. In FIG. 10, the operating device 100 is gripped when the user performs a gesture. First, in the gesture registration stage (stage 1), the operating device 100 recognizes a gesture made by the user and registers the gesture. After the gesture registration, when the gesture is performed by the user, the operating device 100 recognizes the gesture (stage 2: gesture recognition stage). When the operation device 100 recognizes a gesture made by the user, the operation device 100 instructs the device 101 to perform an action and display corresponding to the gesture (step 3: function execution step). The device 101 that has received the command performs an action in accordance with the command, and displays the result of the action on the display device 102 (step 4: display step). As described above, in the conventional technology, the actual operation device 100 is provided with the gesture registration function and the gesture recognition function, and the action execution function for executing the action and display corresponding to the gesture recognized by the operation device 100 is provided in the device 101 in advance. UX was realized by having it.

“What is NINTENDO Wii Motion Plus” [ONLINE], “Search December 15, 2010”, Internet <URL: http://www.nintendo.co.jp/wii/rztj/motionplus/index.html>

  However, in the above-described conventional technology, the operation device 100 and the device 101 must have a gesture recognition function, a gesture registration function, and an action execution function in advance. Therefore, there is a problem that a device that does not have these functions cannot realize a gesture registration function, a gesture recognition function, and an action execution function. For example, digital home appliances that are currently popular often do not have these functions. Therefore, there is a problem that the user cannot operate the digital home appliance using a gesture.

  On the other hand, when the above-described gesture registration function, gesture recognition function, and action execution function are installed in a digital home appliance, there is a problem that costs increase.

  The objective of this invention is providing the technique which enables operation using a gesture with respect to the operating device which does not have a gesture registration function, a gesture recognition function, and an action execution function.

  (1) A device control device according to the present invention is a device control device that operates an operation device included in an augmented reality space by a user wearing a head-mounted display, and recognizes a gesture performed by the user in the augmented reality space. A user input recognizing unit, a user input determining unit that determines an input instruction input by a user based on a gesture recognized by the user input recognizing unit, and the user input determining unit for selecting a certain operating device When it is determined that a device selection instruction is input as the input instruction, a function information acquisition unit that acquires a function information indicating a list of functions of the operation device, and the operation device selected by the user according to the device selection instruction Correspondence between function information and gesture made by user when inputting the device selection instruction A gesture information registration unit that generates input gesture registration information, a virtual user interface is generated based on the function information acquired by the function information acquisition unit, and the generated virtual user interface and the device selection instruction are input. A user interface registration information is generated by associating a gesture made by a user with the user, and the user input determination unit determines that the gesture registered in the user interface registration information is input A superimposing display unit that superimposes and displays a virtual user interface corresponding to the gesture on video data in real space, and a device operation instruction from the user using the virtual user interface is input as the input instruction by the user input determination unit. If it is determined that A function information search unit that searches the function information included in the input gesture registration information for a function corresponding to the device operation instruction, and the function device searched to execute the function searched by the function information search unit. A function execution requesting unit for executing instructions.

  The device control method, device control program, and integrated circuit according to the present invention have the same configuration as the above-described device control apparatus.

  According to this configuration, when the user performs a gesture to select a certain operation device and inputs a device selection instruction, the function information of the operation device selected by the user is acquired, and the acquired function information and gesture Are associated with each other to generate input gesture registration information. Also, a virtual user interface (virtual UI) is generated from the acquired function information, and user interface registration information is generated by associating the generated virtual UI with a gesture made by the user. As described above, the gesture performed by the user, the function information of the operation device, and the virtual UI of the operation device are associated with each other.

  When the user performs the same gesture as when the device selection instruction is input, a virtual UI associated with the gesture is displayed. When the user inputs an operation instruction to the displayed virtual UI, a function corresponding to the operation instruction is searched from the function information included in the input gesture registration information and executed.

  Therefore, even an operating device that does not have a gesture registration function, a gesture recognition function, and an action execution function can be operated using the gesture. Therefore, user convenience can be improved.

  (2) It is preferable that the user input recognition unit recognizes the gesture based on video data acquired by a camera mounted on the head mounted display or a camera installed in a real space.

  According to this configuration, since the gesture can be recognized by photographing the user with the camera, the gesture can be recognized with high accuracy.

  (3) It is preferable that the function information acquisition unit is provided in the head mounted display or server device.

  According to this configuration, the function information can be acquired by the head mounted display or the server device.

  (4) It is preferable that the virtual user interface is a virtual remote controller that simulates an actual remote controller of the operating device.

  According to this configuration, since the virtual UI has the same appearance as the actual remote controller of the operating device, the user can operate the virtual UI with the same feeling as an existing remote controller.

  (5) It is preferable that the said function information acquisition part acquires the said function information from the server apparatus which memorize | stores the said operation apparatus or the said function information previously.

  According to this configuration, when the operating device has a function of outputting the function information to the outside, it is possible to reliably obtain the same function information as the operating device by directly acquiring the function information from the operating device. When the function information of the operating device is managed by the server device, the latest function information of the operating device can be obtained by acquiring the function information from the server device.

  (6) When the gesture recognized by the user input recognition unit is a gesture that touches a certain operation device, the user input determination unit inputs the device selection instruction for selecting the operation device as the input instruction. It is preferable to determine that it has been done.

  According to this configuration, the user can input a device selection instruction by touching the operating device, and the user can input the device selection instruction using a gesture that is intuitively easy to understand.

  (7) When the user input determination unit determines that a display position specifying instruction is input by a user performing a gesture for specifying a position in the real space while the virtual user interface is displayed, the display A contact point recognizing unit that recognizes a position designated by a position designation instruction as a display position of the virtual user interface; and the superimposed display unit is displayed at a position recognized by the contact point recognizing unit. The virtual user interface is preferably superimposed on the video data in the real space.

  According to this configuration, the user can display the virtual UI at a desired position in the augmented reality space by inputting a display position designation instruction.

  (8) When it is determined by the user input determination unit that the display position designation instruction is input, the contact point recognition unit recognizes an object touched when the user inputs the display position designation instruction, It is preferable that the superimposing display unit superimposes the virtual user interface on video data in the real space so that the virtual user interface is pasted on the object recognized by the contact point recognition unit.

  According to this configuration, since the virtual UI is pasted and displayed on the object, when the user operates the virtual UI, the fingertip touches the object and can feel that the virtual UI has been operated.

  (9) It is preferable that the virtual UI generation unit customizes the layout of the virtual user interface for each user.

  According to this configuration, a user-friendly virtual UI can be provided.

  (10) Preferably, the virtual UI generation unit customizes the virtual user interface based on an input frequency of a device operation instruction corresponding to each function included in the function information.

  According to this configuration, buttons corresponding to functions that are rarely used by the user are omitted from the virtual UI, the amount of information on the virtual UI is reduced, and only buttons corresponding to functions that are truly required by the user are included in the virtual UI. Can be made.

  (11) It is preferable that the virtual UI generation unit customizes the virtual user interface based on a user's age or a selection instruction from the user.

  According to this configuration, an appropriate virtual user interface can be provided according to the user's age and the user's preference.

  ADVANTAGE OF THE INVENTION According to this invention, operation using a gesture is attained with respect to the operating device which does not have a gesture registration function, a gesture recognition function, and an action execution function. Therefore, user convenience can be improved.

It is the figure which showed the flow until a virtual UI is produced | generated after a user performs a gesture. FIG. 2 is a diagram illustrating a flow until a virtual UI generated and stored in FIG. 1 is displayed. FIG. 10 is a diagram illustrating a flow when a user designates a display position of a virtual UI in a state where a virtual UI is displayed. It is a functional block diagram of the apparatus control apparatus in embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus control apparatus by embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus control apparatus by embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus control apparatus by embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus control apparatus by embodiment of this invention. It is the figure which showed an example of the data structure of the user interface registration information of a certain user produced | generated by the virtual UI production | generation part. It is a schematic diagram of a prior art.

  1 to 3 are diagrams showing an outline of processing of the device control apparatus according to the embodiment of the present invention. 1 to 3, the same components are denoted by the same reference numerals, and description thereof is omitted. The device control apparatus according to the present embodiment includes a camera 201, a head mounted display (hereinafter referred to as “HMD”) 202, and a server device 203. The camera 201 acquires real space video data. The HMD 202 superimposes a virtual user interface (hereinafter referred to as “virtual UI”) or the like on the video data acquired by the camera 201 to generate and display video data in the augmented reality space. The server device 203 is connected to the HMD 202 via a predetermined communication line NT, and provides various services to the HMD 202. The server device 203 is connected to the operation device 204 through the communication line NT. Here, as the communication line NT, for example, the Internet, a mobile phone communication network, a dedicated line, or the like is adopted.

  FIG. 1 is a diagram illustrating a flow from when a user 200 performs a gesture until a virtual UI is generated. First, in step 1, the user 200 performs a gesture for touching the operation device 204 in order to select the operation device 204. In step 2, the camera 201 recognizes the gesture itself and recognizes the operation device 204 selected by the user.

  In step 3, the camera 201 notifies the HMD 202 of information indicating the recognized gesture and the operation device 204 selected by the gesture.

  In step 4, the HMD 202 transmits a request for generating a virtual UI to the server device 203. In step 5, the server device 203 acquires function information indicating a list of functions of the operation device 204 from the operation device 204. In step 6, the server device 203 generates a virtual UI based on the function information. In step 7, the server device 203 stores the gesture made by the user 200 in association with the generated virtual UI.

  FIG. 2 is a diagram illustrating a flow until the virtual UI generated and stored in FIG. 1 is displayed. In stage 1, the same gesture as when the user 200 selects the operating device 204 is performed. In step 2, the camera 201 recognizes the gesture. In step 3, the camera 201 notifies the HMD 202 of the recognized gesture. In step 4, the HMD 202 notifies the server device 203 which gesture has been performed. In step 5, the server device 203 determines whether a virtual UI corresponding to the gesture is stored. When the virtual UI corresponding to the gesture is stored, the server device 203 transmits the virtual UI to the HMD 202 in Step 6. In step 7, the HMD 202 displays the virtual UI transmitted from the server device 203.

  FIG. 3 is a diagram illustrating a flow when the user 200 designates the display position of the virtual UI in a state where the virtual UI is displayed. In stage 1, the user 200 makes a gesture. Here, it is assumed that the user 200 performs a gesture of touching a fingertip with an object in the real space, for example. In step 2, the camera 201 recognizes the position where the finger and the object are in contact. In step 3, the camera 201 notifies the HMD 202 of the recognized position. In step 4, the HMD 202 superimposes the virtual UI on the video data in the real space so that the virtual UI being displayed is pasted on the object. Then, the user operates the operation device 204 by operating the virtual UI. The above is the outline of the processing of the device control apparatus according to the present embodiment. Hereinafter, processing of the device control apparatus according to the present embodiment will be described in detail.

  FIG. 4 is a functional block diagram of the device control apparatus according to the embodiment of the present invention. As illustrated in FIG. 4, the device control apparatus includes a user input recognition unit 500, a user input determination unit 501, a gesture information registration unit 502, a gesture information storage unit 503, a function information search unit 504, a function execution request unit 505, and function information. An acquisition unit 506, a function information storage unit 507, a virtual UI generation unit 508, a virtual UI storage unit 509, a contact point recognition unit 510, a display superimposing unit 511, a video acquisition unit 512, and a display unit 513 are provided.

  In the following description, the video acquisition unit 512, the user input recognition unit 500, the user input determination unit 501, and the contact point recognition unit 510 are configured by the camera 201, and include a gesture information registration unit 502, a gesture information storage unit 503, and a function information search. The unit 504, the function execution request unit 505, the display superimposing unit 511, and the display unit 513 are configured by the HMD 202. The function information acquisition unit 506, the function information storage unit 507, the virtual UI generation unit 508, and the virtual UI storage unit 509 are servers. It is assumed that the device 203 is configured. However, this is merely an example, and each function other than the video acquisition unit 512 and the display unit 513 illustrated in FIG. 4 may be executed by any of the camera 201, the HMD 202, and the server device 203.

  The camera 201 may be provided in the HMD 202 or may be configured by a fixed point camera arranged at a position in the real space.

  The video acquisition unit 512 is constituted by, for example, a depth sensor, and acquires real space video data at a predetermined frame rate. That is, the video acquisition unit 512 acquires a moving image in the real space. Note that the video acquisition unit 512 is not limited to a depth sensor, and may be a stereo image sensor that can detect the position of depth coordinates by a stereo method.

  The user input recognition unit 500 recognizes a gesture performed by the user in the augmented reality space. Here, the user input recognizing unit 500 receives real-space video data sequentially from the video data acquired by the video acquisition unit 512 at a predetermined frame rate. Then, the user input recognition unit 500 detects the user's arm from the input video data. When the user input recognition unit 500 can detect the user's arm from the video data, the user input recognition unit 500 tracks the movement of the left and right arms of the user using the video data input thereafter. When the user input recognition unit 500 detects the movement of the left and right arms of the user, the user input recognition unit 500 periodically generates gesture recognition information including the movement direction and the movement amount of each part such as the elbow, hand, and each finger of the left and right arms of the user. To the user input determination unit 501.

  In the present embodiment, the video acquisition unit 512 is configured by a depth sensor, for example. Therefore, the movement direction and the movement amount included in the gesture recognition information are configured by three-dimensional data including a depth coordinate component in addition to a vertical coordinate component and a horizontal coordinate component. Here, the vertical coordinate component is a coordinate component in the vertical direction (vertical direction) of the video data of the depth sensor, the horizontal coordinate component is a coordinate component in the horizontal direction (lateral direction) of the depth sensor, and the depth coordinate component is the depth sensor. This is a coordinate component in the depth direction perpendicular to the sensor surface.

  The user input determination unit 501 determines an input instruction input by the user based on the gesture recognized by the user input recognition unit 500. In the present embodiment, a device selection instruction, a virtual UI display instruction, a device operation instruction, a display position designation instruction, and the like are included as input instructions. The device selection instruction is an input instruction for selecting a certain operation device by the user 200. For example, the user 200 inputs a device selection instruction by performing a gesture of touching a certain operation device 204 with a finger. Therefore, the user input determination unit 501 determines that a device selection instruction has been input by the user 200 when the gesture recognition information indicates a gesture that touches the operation device 204 with a finger of the user 200.

  Here, when the user input determination unit 501 is supplied with gesture recognition information indicating a gesture for pointing out a finger from the user input recognition unit 500, an object is positioned at the fingertip from the video data supplied from the video acquisition unit 512. It is determined whether or not. When the user input determination unit 501 determines that the object is located at the fingertip, the user input determination unit 501 determines whether the object is the predetermined operation device 204. When the user input determination unit 501 determines that the object is the predetermined operation device 204, the user 200 It may be determined that the device selection instruction is input.

  Specifically, when the gesture recognition information for pointing out a finger is supplied, the user input determination unit 501 extracts the object included in the video data by performing the process of detecting the feature point, and among the detected objects If there is an object having a portion whose coordinate data matches the fingertip, it may be determined that the fingertip of the user 200 touched the object. Then, if the arrangement pattern of the feature points of the object touched with the fingertip matches the arrangement pattern of the operation device 204 determined in advance, the user input determination unit 501 determines that the device selection instruction has been input by the user 200. Good.

  The virtual UI display instruction is an input instruction for displaying the virtual UI of the operation device 204 selected by the user 200 using the device selection instruction. The user 200 displays the virtual UI display by performing the same gesture as when the device selection instruction is input. For example, when the user 200 performs a gesture of touching the operation device 204 by pointing out the index finger and inputting a device selection instruction, when the gesture of pointing out the index finger (in this case, it is not necessary to touch the operation device 204), the virtual UI is displayed. Is displayed.

  The device operation instruction is an input instruction for the user 200 to operate the virtual UI displayed in the virtual reality space. The user 200 inputs a device operation instruction by performing, for example, a gesture touching an operation button included in the virtual UI. In this case, the user input determining unit 501 determines that the user 200 has input a device operation instruction when the coordinate data of the operation button of the virtual UI overlaps the coordinate data of the fingertip of the user 200 or is located within a certain range. What is necessary is just to judge.

  Here, the virtual UI is displayed in the augmented reality space. The augmented reality space is a three-dimensional space defined by the vertical coordinate component, the horizontal coordinate component, and the depth coordinate component, and the video data is generated by superimposing graphics on the video data in the real space.

  The display position designation instruction is an input instruction that the user 200 inputs to designate the display position of the virtual UI in the augmented reality space. In the present embodiment, the user 200 cannot operate the virtual UI only by displaying the virtual UI, but can specify the display position of the virtual UI to operate the virtual UI. Therefore, for example, after performing a gesture for displaying the virtual UI, the user 200 inputs a display position designation instruction by presenting a certain finger (for example, forefinger) and performing a gesture for stopping the finger presented at a desired position. To do. Therefore, when the virtual UI is displayed but the virtual UI is not operable, the user input determination unit 501 performs a display position designation instruction when the user 200 performs a gesture to stop by pointing out the index finger. Is determined to be input.

  In addition, the user input determination unit 501 indicates whether the virtual UI is being displayed, is not displayed, is being displayed but is operable, is being displayed, but is not operable, The current state of the virtual UI is maintained and the virtual UI is managed.

  When the user input determination unit 501 determines that a device selection instruction has been input, the function information acquisition unit 506 displays function information indicating a list of functions of the operation device 204 selected by the device selection instruction, as a function information storage unit 507. Get from. For example, when the operation device 204 is a DVD player, functions include “power on / off” for turning on / off the power, “play” for playing back video data, “recording” for starting recording, “Channel selection” to select a channel, “Stop” to stop playback and recording, “Fast forward” to advance a frame of video data to a desired scene, and “Rewind” to return a frame of video data to a desired scene "Return" etc. exist. Therefore, in this case, the function information is information indicating a list of functions such as power on / off, playback, recording, channel selection, stop, rewind, and fast forward.

  When the operation device 204 is a television, for example, there are functions such as “power on / off” for turning on / off the power, “channel selection” for selecting a channel, “volume” for adjusting the volume, and the like. To do. Therefore, in this case, the function information is information indicating a list of functions such as power on / off, channel selection, and volume.

  Note that these pieces of function information are merely examples, and other functions may be included. The operation device 204 is not limited to a DVD player and a television, and any device can be used as long as it can be operated by a remote controller, such as an audio, a mini component, an air conditioner, or an air purifier. Good.

  Here, when the operation device 204 has a device information notification function, the function information acquisition unit 506 directly acquires device information from the operation device 204 by transmitting a device information acquisition request to the operation device 204. Also good. In this case, the function information acquisition unit 506 communicates with the operation device 204 by using a communication protocol performed between the remote control and the device main body if the operation device 204 is a device operated by the remote controller, and the device information. Just get it. In this case, the function information storage unit 507 is provided in the operating device 204.

  When the function information acquisition unit 506 is provided in the HMD 202 and the server device 203 holds the function information in advance, the function information acquisition unit 506 sends the user input determination unit 501 to the operation device 204. The function information corresponding to the specified model number may be acquired from the device information storage unit 507. In this case, for example, the user input determination unit 501 may recognize the model number of the operation device 204 from the video data and notify the function information acquisition unit 506 of the model number. In this case, the function information storage unit 507 is provided in the server device 203.

  In addition, when the function information acquisition unit 506 is provided in the HMD 202, the function information acquisition unit 506 may constantly update the acquired function information to the latest function information. In this case, for example, every time the device selection apparatus is turned on, the function information acquisition unit 506 notifies the device information storage unit 507 of an update request for the acquired device information, and updates the device information to the latest device information. Update it.

  The function information storage unit 507 and the function information storage unit 507 may be provided on the memory of the HMD 202. In this case, the function information storage unit 507 may store the function information of each predetermined operation device 204 in advance. In this case, the user 200 may acquire the function information of the favorite operation device 204 from the server device 203 and store the function information in the function information storage unit 507.

  When it is determined by the user input determination unit 501 that a device selection instruction has been input, the gesture information registration unit 502 displays the function information of the operation device 204 and the gesture performed by the user 200 when inputting the device selection instruction. The input gesture registration information is generated in association with the gesture feature information shown, and the generated input gesture registration information is stored in the gesture information storage unit 503.

  Here, when the device control apparatus executes a program that prompts the user 200 to input a device selection instruction, the user 200 and the device control apparatus may interactively perform registration processing of input gesture registration information. For example, the display superimposing unit 511 may display a human body object that operates in response to the gesture of the user 200 and feed back the gesture performed by the user 200 to the user 200. Alternatively, the user 200 may be made to perform a gesture so as to imitate a gesture of a human body object and input a device selection instruction. For example, the device selection instruction may be registered interactively by causing the human body object to perform an operation of waving a hand and causing the user 200 to perform a gesture of waving accordingly.

  The input gesture registration information includes, for example, the gesture feature information that characterizes the gesture performed when inputting the device selection instruction, the function information of the operation device 204 selected by the device selection instruction, and the user 200 that has input the gesture. Information for associating the user ID shown is adopted. As the gesture characteristic information, gesture recognition information indicating a gesture performed by the user 200 when inputting a device selection instruction is employed.

  In the present embodiment, a device selection instruction is input by causing the user 200 to perform a gesture of touching the operation device 204 with a finger. Therefore, as the gesture feature information, for example, data indicating the type of finger, the moving direction of the fingertip, and the moving amount of the fingertip is employed. As the type of finger, information indicating any finger such as a thumb to a little finger is employed. As the movement direction of the fingertip and the movement amount of the fingertip, the movement direction of the corresponding finger included in the gesture recognition information supplied from the user input recognition unit 500 to the user input determination unit 501 when a device selection instruction is given, and What is necessary is just to employ | adopt movement amount.

  As the user ID, information for identifying the user 200 wearing the HMD 202 may be employed. For example, this user ID may be input to the user 200 when the HMD 202 is mounted. Specifically, a user ID is registered in advance in the HMD 202. When the power of the HMD 202 is turned on, a virtual UI for inputting a user ID is displayed on the display panel of the HMD 202, and the registered user ID may be input to the user 200 using the virtual UI.

  As the function information, the function information acquired by the function information acquisition unit 506 when a device selection instruction is given may be adopted. When the user input determination unit 501 determines that a device selection instruction has been input, the user input determination unit 501 supplies the function information acquired by the function information acquisition unit 506 to the gesture information registration unit 502. Thereby, the gesture information registration unit 502 can acquire function information.

  When the user input determination unit 501 determines that a device operation instruction from the user 200 using the virtual UI has been input as an input instruction, the function information search unit 504 inputs a function corresponding to the device operation instruction to the input gesture registration information. Search from the function information included in. For example, a case where a virtual UI of a DVD player is displayed will be described as an example. The virtual UI includes buttons such as power on / off, playback, recording, channel selection, stop, rewind, and fast forward, for example.

  Therefore, when the user input determination unit 501 determines that, for example, a power on / off button has been pressed, the function information search unit 504 searches the function information for a function corresponding to the power on / off. If the function corresponding to the power on / off is found, the function execution request unit 505 is notified to execute the power on / off function.

  The function execution request unit 505 instructs the operation device 204 to execute the function searched by the function information search unit 504. In this case, when the operation device 204 is a device that can be operated by a remote controller such as a DVD player, the function execution request unit 505 transmits a remote control signal that is transmitted to the DVD main body in order to cause the remote controller to execute a function in the DVD player. The same signal may be transmitted. For example, when the DVD player is turned on / off, the function execution request unit 505 transmits to the DVD player the same signal as the remote control signal that turns on / off the DVD player that the remote control of the DVD player transmits to the main body of the DVD player. Thus, the power is turned on when the DVD player is in the off state, and the power is turned off when the DVD player is in the on state.

  The virtual UI generation unit 508 generates a virtual UI based on the function information acquired by the function information acquisition unit 506, and displays the generated virtual UI and the gesture performed by the user 200 when inputting a device selection instruction. Corresponding user interface registration information is generated and stored in the virtual UI storage unit 509.

  FIG. 9 is a diagram illustrating an example of a data structure of user interface registration information 1000 of a certain user 200 generated by the virtual UI generation unit 508. As shown in FIG. 9, the user interface registration information 1000 includes columns for gesture feature information, a selected device, and a virtual UI. In the input gesture registration information column, gesture feature information indicating a gesture performed when the user 200 inputs a device selection instruction, such as an index finger or a middle finger, is stored.

  In the present embodiment, the user 200 inputs a device selection instruction by performing a gesture of touching a finger on the operation device 204. Therefore, in the example of FIG. 9, information indicating the finger of the user 200 touching the operation device 204 is employed as the gesture feature information.

  In the column of the selected device, identification information for specifying the operation device 204 selected by the user 200 inputting a device selection instruction is stored. The virtual UI column stores identification information for identifying a virtual UI for the operation device 204 selected by the user 200 inputting a device selection instruction.

  In the example of FIG. 9, the user 200 performs a gesture of touching the index finger on the DVD recorder to input a device selection instruction, and identification information “DVD_Rec_UI” is given to the virtual UI generated at that time. Therefore, the record in the row 1001 stores “index finger”, “DVD recorder”, and “DVD_Rec_UI” in the columns of the gesture feature information, the selected device, and the virtual UI, respectively.

  In the example of FIG. 9, the user 200 performs a gesture of touching the middle finger on the Blu-ray recorder to input a device selection instruction, and identification information of “BluRay_Rec_UI” is assigned to the virtual UI generated at that time. Therefore, the record in the row 1002 stores “middle finger”, “Blu-ray recorder”, and “BluRay_Rec_UI”.

  Similarly, since the user 200 inputs a device selection instruction by touching the thumb, pinky finger, and ring finger with respect to the television, component, or personal computer, the information shown in the columns of the records in the rows 1003 to 1005 includes Stored.

  Here, when the virtual UI is acquired together with the function information by the function information acquisition unit 506, the virtual UI generation unit 508 may generate the user interface registration information 1000 using the acquired virtual UI. Also, the virtual UI generation unit 508 tries to acquire a virtual UI from the server device 203 when the function information acquisition unit 506 does not acquire the virtual UI together with the function information. Then, when the virtual UI can be acquired, the virtual UI generation unit 508 may generate the user interface registration information 1000 using the virtual UI.

  In the present embodiment, a virtual remote controller having the same appearance as the remote controller of the operation device 204 is employed as the virtual UI. Therefore, when the function information acquisition unit 506 can acquire the virtual UI together with the device information, a virtual remote controller having the same appearance as the remote controller of the corresponding operation device 204 is employed as the virtual UI. Therefore, the same buttons as the buttons of the actual remote controller of the operation device 204 such as a “power on / off” button, a “play” button, and a “fast forward” button are displayed in the same layout on the virtual UI.

  In addition, when the virtual UI cannot be acquired, the virtual UI generation unit 508 may generate a virtual UI that lists each function included in the function information in a button format. In this case, the virtual UI generation unit 508 may generate a virtual UI that displays a list of operation buttons in which the name of each function is enclosed by a square.

  The virtual UI generation unit 508 may customize the virtual UI layout for each user 200. In this case, the virtual UI generation unit 508 may customize the virtual UI based on the input frequency of the device operation instruction corresponding to each function included in the function information. For example, it is assumed that the virtual remote controller has an A button for executing a function A and a B button for executing a function B.

  In this case, the virtual UI generation unit 508 counts the number of times the user 200 has touched the A button and the B button. Further, the virtual UI generation unit 508 also counts the number of times that the user 200 displays the virtual UI. Then, the virtual UI generation unit 508 divides the number of times the A button and the B button are touched by the display number when the user 200 displays the virtual UI exceeds a certain value, Find the input frequency. Then, when the input frequency of the A button or B button exceeds a predetermined value that is predetermined for each A button or B button, the virtual UI generation unit 508 leaves the A button or B button. On the other hand, when the input frequency of the A button or B button is equal to or less than a predetermined value predetermined for each A button or B button, the A button or B button may be omitted from the virtual UI. Thereafter, the virtual UI generation unit 508 may perform the above process every time the number of times the virtual UI is displayed exceeds a certain value, and determine whether to omit the A button or the B button.

  As a result, buttons corresponding to functions that are rarely used by the user 200 are omitted from the virtual UI, the amount of information on the virtual UI is reduced, and only buttons corresponding to functions that the user 200 really needs are included in the virtual UI. Can do.

  Note that the virtual UI generation unit 508 is supplied with information indicating which button of the virtual UI is touched by the user 200 from the user input determination unit 501, and calculates the input frequency for each function of the function information based on this information. That's fine.

  The virtual UI generation unit 508 may customize the virtual UI based on the age of the user 200 or a selection instruction from the user 200. For example, the elderly user 200 may use only a few (for example, 2 to 5) of the functions of the operation device 204. In the case of an elderly user 200, if a large number of buttons are arranged on the virtual UI, it may be difficult to visually recognize the buttons with presbyopia. Therefore, the virtual UI generation unit 508 may customize the virtual UI for the elderly user 200 when the age of the user 200 who uses the HMD 202 exceeds a certain age. In this case, the virtual UI generation unit 508 may adopt the virtual UI when the function information acquisition unit 506 acquires the virtual UI for the elderly person together with the function information. The virtual UI may be customized so that only the button is included. Note that the user 200 may input the age of the user 200 when the user 200 uses the HMD 202 for the first time.

  When the user input determination unit 501 determines that the user 200 has input a display position designation instruction, the contact point recognition unit 510 recognizes the position designated by the user 200 according to the display position designation instruction as the display position of the virtual UI. .

  Specifically, when it is determined that the display position designation instruction is input by the user input determination unit 501, the contact point recognition unit 510 receives coordinate data indicating the position of the tip of the finger of the user 200 from the user input determination unit 501. get. Then, the contact point recognition unit 510 outputs a command for superimposing and displaying the virtual UI on the position of the tip of the finger to the display superimposing unit 511.

  In the present embodiment, the user 200 inputs a display position designation instruction by performing a gesture of touching an object in the real space with a finger. Accordingly, when it is determined by the user input determination unit 501 that a display position designation instruction has been input, the contact point recognition unit 510 recognizes an object touched by the user 200 from the video data. In this case, the contact point recognition unit 510 extracts an object existing at the position of the fingertip of the user 200 from the video data in the augmented reality space, and displays information indicating the image area of the extracted object on the video data as a display superimposing unit. 511 may be notified.

  As a result, in the augmented reality space, the virtual UI is displayed by being pasted on the object touched when the user 200 performs a gesture for inputting a display position designation instruction. As described above, the virtual UI is composed of a plurality of buttons. When such a button is displayed so as to float in the augmented reality space, it is difficult for the user 200 to realize that the button is pressed. Therefore, in the present embodiment, the virtual UI is pasted and displayed on the object. Thereby, when the user 200 presses the button of the virtual UI, the fingertip touches the object, and can feel that the button has been pressed.

  Note that when a virtual UI that allows the user 200 to input a device operation instruction by making a gesture that causes the user 200 to grasp the object displayed in the augmented reality space is used, the virtual UI is not pasted on the object. What is necessary is just to display as if it floated in the real space.

  When it is determined by the user input determination unit 501 that a gesture registered in the user interface registration information is input and the user 200 has input a virtual UI display instruction, the display superimposing unit 511 displays a virtual UI corresponding to the gesture. The augmented reality space image data is generated by superimposing the image data on the actual space and displayed on the display unit 513. Specifically, when a command for displaying a virtual UI is supplied from the contact point recognition unit 510, the display superimposing unit 511 performs texture mapping or the like on the image area of the object on the video data defined by the command. A virtual UI is pasted using a technique.

  For example, the display superimposing unit 511 writes the video data acquired by the video acquiring unit 512 to a drawing buffer included in the display unit 513, and superimposes the virtual UI on the written video data to write the virtual UI. Just stick to the object. When the display position designation instruction is input and the object touched by the user 200 is recognized, the contact point recognition unit 510 subsequently tracks the object with respect to the video data by the video acquisition unit 512, and the video data is updated. Each time a command is displayed, a command for displaying a virtual UI indicating the image area of the object is supplied to the display superimposing unit 511.

  The display superimposing unit 511 performs texture mapping every time a command for displaying the virtual UI is supplied, and pastes the virtual UI on the image area of the object to be superimposed on the virtual UI. Thus, even if the orientation of the object to which the virtual UI is pasted is changed by the user 200 and the image area of the object changes, the virtual UI is changed so as to follow the change in the image area.

  Thereby, the virtual UI is displayed so as to be deformed along the surface shape of the object touched by the user 200, and the virtual UI can be pasted to the object without a sense of incongruity.

  The display unit 513 includes, for example, an HMD display panel, a drawing buffer in which video data in the augmented reality space displayed on the display panel is written, and supplies the video data written in the drawing buffer to the display panel. As a result, a moving image of the video data in the augmented reality space in which the virtual UI or the like is superimposed on the video data acquired by the video acquisition unit 512 is displayed on the display unit 513.

  Next, the operation of the device control apparatus in the embodiment of the present invention will be described. 5 to 8 are flowcharts showing the operation of the device control apparatus according to the embodiment of the present invention.

  First, when detecting that the user 200 has made a gesture (step S (hereinafter abbreviated as “S”) 600), the user input recognition unit 500 supplies the gesture recognition information to the user input determination unit 501.

  Next, the user input determination unit 501 determines whether or not the gesture recognition information supplied from the user input recognition unit 500 corresponds to a device selection instruction (S601). Next, when the user input determination unit 501 determines that a device selection instruction has been input by the user 200 (YES in S602), the user input determination unit 501 recognizes the operation device 204 touched by the fingertip of the user 200 (S603). On the other hand, if the user input determination unit 501 determines that the device selection instruction has not been input by the user 200 (NO in S602), the process proceeds to S604.

  Next, the function information acquisition unit 506 searches the function information storage unit 507 for the virtual UI of the operating device 204 recognized in S601 (S606).

  When the function information acquisition unit 506 determines that the corresponding virtual UI exists in the function information storage unit 507 (YES in S700), the function information acquisition unit 506 acquires the function information of the virtual UI and the corresponding operation device 204 from the function information storage unit 507. (S701).

  Next, the gesture information registration unit 502 inputs the function information acquired by the function information acquisition unit 506, the gesture feature information indicating the gesture recognized in S601, and the input associated with the user ID of the user 200 who uses the HMD 202. Gesture registration information is generated and stored in the gesture information storage unit 503 (S702).

  Next, the virtual UI generation unit 508 associates the received virtual UI with the gesture feature information indicating the gesture recognized in S601, generates user interface registration information, and stores the user interface registration information in the virtual UI storage unit 509 (S703).

  For example, when the user 200 touches the DVD recorder with the index finger, the gesture characteristic information indicating the gesture touched with the index finger, the virtual UI of the DVD recorder selected by the user 200 by the gesture, and the user ID are associated with each other. The user interface registration information is generated and stored in the generation unit 508 in the virtual UI.

  Accordingly, when the user 200 next performs a gesture of pointing out the index finger, the virtual UI of the DVD player is displayed on the display unit 513. Note that the virtual UI may have buttons and the like corresponding to all functions, or may be customized so that buttons that are rarely used by the user 200 are hidden.

  Next, the virtual UI generation unit 508 supplies the generated virtual UI to the display superimposing unit 511 and causes the display unit 513 to display the virtual UI (S704). In this case, the display superimposing unit 511 may display the virtual UI at a predetermined default position in the augmented reality space. In this case, the user input determination unit 501 may set the virtual UI state to “displaying” and “not operable”.

  On the other hand, when the function information acquisition unit 506 determines that the virtual UI of the operation device 204 does not exist in the function information storage unit 507 (NO in S700), the function information acquisition unit 506 acquires only the function information of the operation device 204 from the function information storage unit 507. (S705). Next, the gesture information registration unit 502 generates input gesture registration information by associating the function information acquired by the function information acquisition unit 506, the gesture feature information indicating the gesture recognized in S601, and the user ID. And stored in the gesture information storage unit 503 (S706).

  Next, the function information acquisition unit 506 notifies the virtual UI generation unit 508 of a virtual UI generation request (S707). Next, the virtual UI generation unit 508 generates a virtual UI from the device information acquired by the function information acquisition unit 506 (S708).

  Next, the virtual UI generation unit 508 generates user interface registration information by associating the generated virtual UI with the gesture feature information indicating the gesture recognized in step S601 and the user ID, and stores the user interface registration information in the virtual UI storage unit 509. (S709). In this case, for example, user interface registration information 1000 as shown in FIG. 9 is generated.

  Next, similarly to S709, the virtual UI generation unit 508 supplies the generated virtual UI to the display superimposing unit 511 and causes the display unit 513 to display the virtual UI (S710). In this case, the user input determination unit 501 may set the state of the virtual UI to “displaying” and “not operable”.

  On the other hand, in S604 of FIG. 5, the user input determination unit 501 determines that the gesture recognized in S601 is registered in the input gesture registration information stored in the gesture information storage unit 503 and is a virtual UI display instruction. If yes (YES in S604), the process proceeds to S605.

  Then, the virtual UI generation unit 508 reads the virtual UI corresponding to the gesture recognized in S601 from the virtual UI storage unit 509, supplies the virtual UI to the display superimposing unit 511, and causes the display unit 513 to display the virtual UI (S605).

  Next, when the user 200 performs a gesture of bringing a fingertip into contact with an object and the user input determination unit 501 determines that a display position designation instruction is input by the user 200 (YES in S800), the contact point recognition unit 510 The position where the fingertip contacts the object is recognized (S801).

  Next, the contact point recognition unit 510 recognizes the object by extracting the image area of the object touched by the user 200 with the fingertip from the video data (S802). Next, the display superimposing unit 511 pastes a virtual UI on the image area of the recognized object by texture mapping (S803). Next, the user input determination unit 501 sets the state of the virtual UI to “displaying” and “operable” (S804), and ends the process.

  On the other hand, if a display position designation instruction is not input in S800 (NO in S800), the process returns to S800, and the process waits until a display position designation instruction is input.

  In S604 illustrated in FIG. 5, when the user input determination unit 501 determines that the gesture recognized in S601 is not a virtual UI display instruction (NO in S604), the user input determination unit 501 determines whether the gesture is a device operation instruction. (S900).

  When the user input determination unit 501 determines that the gesture is a device operation instruction (YES in S900), the function information search unit 504 searches the gesture information storage unit 503 for a function corresponding to the device operation instruction. (S901).

  Next, the function execution request unit 505 outputs to the operating device 204 a command for causing the operating device 204 to execute the function searched in S901. Thereby, the function input by the user 200 according to the operation instruction is executed by the operation device 204.

  On the other hand, when the gesture recognized in S601 is not a device operation instruction (NO in S900), the user input determination unit 501 ends the process.

  In the above flowchart, the user 200 inputs the display position designation instruction by bringing the fingertip into contact with an object. However, the user 200 inputs the display position designation instruction by pointing the fingertip to an arbitrary position in the augmented reality space. Also good.

  In this case, the display superimposing unit 511 may expand the virtual UI in the vertical direction and the horizontal direction of the display unit 513 around the position of the fingertip pointed by the user 200. Further, when the size of the virtual UI displayed on the display unit 513 is small or large, the virtual UI may be enlarged or reduced to an appropriate size.

  In the above description, the contact point recognition unit 510 recognizes the position of the fingertip of the user 200 using the video data, but any technique can be used as long as it can recognize the position of the fingertip of the user 200. good. For example, a position sensor may be provided at the fingertip of the user 200 so that the fingertip of the user 200 may be recognized.

  As described above, according to the device control apparatus according to the present embodiment, when the user 200 performs a gesture of touching the operation device 204 that is a certain finger, the function information of the operation device 204 selected by the user 200 is acquired and acquired. The function information thus made and the gesture of the user 200 are stored in association with each other. In addition, a virtual UI is generated from the function information, and the generated virtual UI is stored in association with the gesture performed by the user 200. Thus, the gesture performed by the user 200, the function information of the operation device 204, and the virtual UI of the operation device 204 are associated with each other.

  When the user 200 performs a gesture for pointing out the fingertip in the same manner as when the operation device 204 is selected, a virtual UI associated with the gesture is displayed. Then, when the user 200 inputs an operation instruction to the displayed virtual UI, a function corresponding to the operation instruction is executed.

  Therefore, even a device that does not have a gesture registration function, a gesture recognition function, and an action execution function can provide a gesture registration function, a gesture recognition function, and an action execution function, which improves the convenience of the user 200. Can be improved.

  Moreover, in the said embodiment, the user 200 employ | adopted the aspect which displays a virtual UI, when performing the gesture which presents the fingertip of a certain finger | toe. However, in this case, when the user 200 does not want to display the virtual UI but performs the same gesture, the virtual UI may be displayed. Therefore, when the user 200 inputs a virtual UI display instruction, the operation mode of the device control apparatus may be set to the virtual UI display mode and the virtual UI display instruction may be input. Thereby, it is possible to prevent the virtual UI from being displayed when the user 200 does not want to display the virtual UI. In this case, a switch may be provided in the HMD 202 and the operation mode may be set by the user 200 by operating this switch. Further, the operation mode may be set by causing the user 200 to perform a gesture for setting the operation mode.

  The above problem also occurs in the device selection instruction. Therefore, when the user 200 inputs a device selection instruction, the operation mode of the device control apparatus may be set to the virtual UI registration mode.

  In the example of FIG. 9, a virtual UI of a different operation device 204 is associated with each finger of the user 200. Therefore, when the user 200 performs a gesture of spreading his / her hand, all the virtual UIs associated with each finger can be displayed. As a result, the user 200 can quickly confirm even if the user forgets which virtual UI of which operating device 204 is associated with which finger. Of course, after the confirmation, the user 200 may make a gesture to hide any virtual UI among all displayed virtual UIs.

  The embodiment is an example, and the present invention is not limited to the embodiment. Forms obtained by subjecting the embodiments to modifications conceived by those skilled in the art and other forms realized by arbitrarily combining the components in the embodiments are also included in the present invention.

  4 may be implemented as an LSI (Large Scale Integration) that is an integrated circuit. These components may be individually made into one chip, or may be made into one chip so as to include a part or all of them. Although referred to as LSI here, it may be called IC (Integrated Circuit), system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Furthermore, if integrated circuit technology that replaces LSI emerges as a result of advances in semiconductor technology or other derived technology, naturally, using that technology, the components included in the display area control device are integrated into an integrated circuit. Also good. Further, among the components of the device control apparatus, only the means for storing data may be configured separately instead of being integrated into one chip.

  In addition, the present invention can be realized not only as a device control apparatus, but also as a method using a processing unit constituting the device control apparatus as a step. The present invention can be realized as a program that causes a computer to execute the steps included in the method. Furthermore, the present invention can be realized as a computer-readable storage medium such as a CD-ROM storing the program.

  According to the device control apparatus according to the present invention, operation using gestures can be performed on operation devices that do not have a gesture registration function, a gesture recognition function, and an action execution function. This is considered to be particularly useful when used in an augmented reality space application equipped with a HMD.

200 User 201 Camera 202 HMD
203 Server device 204 Operation device 500 User input recognition unit 501 User input determination unit 502 Gesture information registration unit 503 Gesture information storage unit 504 Function information search unit 505 Function execution request unit 506 Function information acquisition unit 507 Function information storage unit 508 Virtual UI generation Unit 509 virtual UI storage unit 510 contact point recognition unit 511 display superimposition unit 512 video acquisition unit 513 display unit 1000 user interface registration information

Claims (14)

A device control device for operating an operation device included in an augmented reality space by a user wearing a head-mounted display,
A user input recognition unit for recognizing a gesture made by a user in the augmented reality space;
A user input determination unit that determines an input instruction input by the user based on the gesture recognized by the user input recognition unit;
When the user input determination unit determines that a device selection instruction for selecting a certain operation device is input as the input instruction, a function information acquisition unit that acquires function information indicating a list of functions of the operation device When,
A gesture information registration unit that generates input gesture registration information by associating the function information of the operation device selected by the user according to the device selection instruction with a gesture performed by the user when inputting the device selection instruction;
A virtual user interface is generated based on the function information acquired by the function information acquisition unit, and the generated virtual user interface is associated with a gesture made by the user when inputting the device selection instruction. A virtual UI generation unit for generating registration information;
When the user input determination unit determines that a gesture registered in the user interface registration information is input, a superimposed display unit that displays a virtual user interface corresponding to the gesture superimposed on video data in real space;
When the user input determination unit determines that a device operation instruction from the user using the virtual user interface is input as the input instruction, a function corresponding to the device operation instruction is included in the input gesture registration information. A function information search unit for searching from the function information;
A device control apparatus comprising: a function execution request unit that instructs the operation device so that the function searched by the function information search unit is executed.   The device control apparatus according to claim 1, wherein the user input recognition unit recognizes the gesture based on video data acquired by a camera mounted on the head mounted display or a camera installed in a real space.   The device control device according to claim 1, wherein the function information acquisition unit is provided in the head mounted display or server device.   The device control apparatus according to claim 1, wherein the virtual user interface is a virtual remote controller that simulates an actual remote controller of the operation device.   The device control apparatus according to claim 1, wherein the function information acquisition unit acquires the function information from the operation device or a server device that stores the function information in advance.   The user input determination unit determines that the device selection instruction for selecting the operation device is input as the input instruction when the gesture recognized by the user input recognition unit is a gesture that touches the operation device. The apparatus control apparatus in any one of Claims 1-5. When the user input determination unit determines that a display position designation instruction has been input by a user performing a gesture for designating a position in the real space while the virtual user interface is displayed, the display position designation instruction A contact point recognizing unit for recognizing the position specified by the display position of the virtual user interface,
The device control according to claim 1, wherein the superimposing display unit superimposes the virtual user interface on video data in the real space so that the virtual user interface is displayed at a position recognized by the contact point recognition unit. apparatus. The contact point recognition unit recognizes an object touched when a user inputs the display position designation instruction when the user input determination unit determines that the display position designation instruction is input;
The device control apparatus according to claim 7, wherein the superimposing display unit superimposes the virtual user interface on video data in the real space so that the virtual user interface is pasted on an object recognized by the contact point recognition unit.   The device control apparatus according to claim 1, wherein the virtual UI generation unit customizes a layout of the virtual user interface for each user.   The device control apparatus according to claim 9, wherein the virtual UI generation unit customizes the virtual user interface based on an input frequency of a device operation instruction corresponding to each function included in the function information.   The device control apparatus according to claim 9 or 10, wherein the virtual UI generation unit customizes the virtual user interface based on a user's age or a selection instruction from the user. A device control method for operating an operation device included in an augmented reality space by a user wearing a head mounted display,
A user input recognition step for recognizing a gesture made by a user in the augmented reality space;
A user input determination step of determining an input instruction input by the user based on the gesture recognized by the user input recognition step;
If the user input determination step determines that a device selection instruction for selecting a certain operating device has been input as the input instruction, a function information acquisition step for acquiring function information indicating a list of functions of the operating device When,
A gesture information registration step of generating input gesture registration information by associating the function information of the operating device selected by the user according to the device selection instruction with a gesture performed by the user when inputting the device selection instruction;
A virtual user interface is generated based on the function information acquired in the function information acquisition step, and the generated virtual user interface is associated with a gesture made by the user when inputting the device selection instruction. A virtual UI generation step of generating registration information;
When it is determined by the user input determination step that a gesture registered in the user interface registration information is input, a superimposed display step for displaying a virtual user interface corresponding to the gesture superimposed on video data in real space;
When it is determined in the user input determination step that a device operation instruction from the user using the virtual user interface is input as the input instruction, a function corresponding to the device operation instruction is included in the input gesture registration information. A function information search step for searching from the function information;
A device control method comprising: a function execution requesting step for instructing the operating device to execute the function searched in the function information searching step. A device control program that causes a computer to function as a device control device that allows a user wearing a head-mounted display to operate an operation device included in an augmented reality space,
A user input recognition unit for recognizing a gesture made by a user in the augmented reality space;
A user input determination unit that determines an input instruction input by the user based on the gesture recognized by the user input recognition unit;
When the user input determination unit determines that a device selection instruction for selecting a certain operation device is input as the input instruction, a function information acquisition unit that acquires function information indicating a list of functions of the operation device When,
A gesture information registration unit that generates input gesture registration information by associating the function information of the operation device selected by the user according to the device selection instruction with a gesture performed by the user when inputting the device selection instruction;
A virtual user interface is generated based on the function information acquired by the function information acquisition unit, and the generated virtual user interface is associated with a gesture made by the user when inputting the device selection instruction. A virtual UI generation unit for generating registration information;
When the user input determination unit determines that a gesture registered in the user interface registration information is input, a superimposed display unit that displays a virtual user interface corresponding to the gesture superimposed on video data in real space;
When the user input determination unit determines that a device operation instruction from the user using the virtual user interface is input as the input instruction, a function corresponding to the device operation instruction is included in the input gesture registration information. A function information search unit for searching from the function information;
A device control program that causes a computer to function as a function execution request unit that instructs the operating device to execute a function searched by the function information search unit. An integrated circuit of a device control device that causes a user wearing a head-mounted display to operate an operation device included in the augmented reality space,
A user input recognition unit for recognizing a gesture made by a user in the augmented reality space;
A user input determination unit that determines an input instruction input by the user based on the gesture recognized by the user input recognition unit;
When the user input determination unit determines that a device selection instruction for selecting a certain operation device is input as the input instruction, a function information acquisition unit that acquires function information indicating a list of functions of the operation device When,
A gesture information registration unit that generates input gesture registration information by associating the function information of the operation device selected by the user according to the device selection instruction with a gesture performed by the user when inputting the device selection instruction;
A virtual user interface is generated based on the function information acquired by the function information acquisition unit, and the generated virtual user interface is associated with a gesture made by the user when inputting the device selection instruction. A virtual UI generation unit for generating registration information;
When the user input determination unit determines that a gesture registered in the user interface registration information is input, a superimposed display unit that displays a virtual user interface corresponding to the gesture superimposed on video data in real space;
When the user input determination unit determines that a device operation instruction from the user using the virtual user interface is input as the input instruction, a function corresponding to the device operation instruction is included in the input gesture registration information. A function information search unit for searching from the function information;
An integrated circuit comprising: a function execution requesting unit that instructs the operating device so that the function searched by the function information searching unit is executed.

Images