JP6514416B2 - IMAGE DISPLAY DEVICE, IMAGE DISPLAY METHOD, AND IMAGE DISPLAY PROGRAM - Google Patents

Description

  The present invention relates to display technology of a user interface in a virtual space.

  BACKGROUND In recent years, techniques for causing users to experience virtual reality (hereinafter, also referred to as “VR”) are used in various fields such as games, entertainment, and vocational training. In the VR, usually, a glasses-type or goggle-type display device called a head mounted display (hereinafter also referred to as “HMD”) is used. By mounting the HMD on the head of the user and viewing the screen built in the HMD with both eyes, the user can view an image with a three-dimensional effect. The HMD incorporates a gyro sensor and an acceleration sensor, and the image displayed on the screen changes according to the movement of the head of the user detected by these sensors. As a result, the user can experience as if it were embedded in the displayed image.

  In the technical field of such VR, a user interface for performing operations by user's gesture has been studied. As an example, a technology that performs an operation according to the user's movement on the HMD by attaching a dedicated sensor to the user's body surface or arranging an external device for detecting the user's movement around the user It has been known.

  Further, as another example, a specific object for operation is disposed in a screen displayed on the HMD, and a change in the user's viewpoint with respect to this object is detected by a gyro sensor or an acceleration sensor built in the HMD. There is also known a technique for performing an operation according to a change in the user's viewpoint.

  Furthermore, as another example, in Patent Document 1, the user interface is disposed in a virtual space, and an operation unit used by the user to operate the user interface is in the field of view of the imaging unit, and the operation unit and the user An image processing apparatus is disclosed that determines that an operation has been performed on a user interface when the positional relationship with the interface is in a prescribed positional relationship.

JP, 2012-48656, A

  However, when a dedicated sensor or an external device is provided to detect a user's gesture, the entire system becomes large. In addition, since the amount of computation for processing signals detected by a dedicated sensor or an external device is also enormous, a high-spec computing device is required to perform real-time operation.

  In addition, when an object for operation is displayed on the screen of the HMD, this object always comes into the user's field of view, which may be disturbed for the user, and the immersive feeling peculiar to the VR experience may be reduced. There is. Moreover, the operation by the movement of the viewpoint is not intuitive for the user, and there is a possibility that an erroneous operation may occur.

  Furthermore, when a plurality of icons are displayed in the virtual space as a user interface, the positional relationship with each icon changes each time by moving the operation unit. Therefore, if the presence or absence of the operation is simply determined based on the positional relationship between the operation unit and each icon, it may be determined that the operation has been performed on the icon not intended by the user. With regard to this point, in Patent Document 1, when the selection instruction is input in the state where the positional relationship between the icon and the operation unit satisfies a predetermined condition, it is determined that the operation on the icon has been performed. . That is, two-step processing of icon selection and determination is performed. Therefore, it is difficult for the user to say intuitive operation.

  The present invention has been made in view of the above, and provides an image display device, an image display method, and an image display program that can realize an intuitive user interface with few erroneous operations with a simple configuration. Intended to be provided.

  In order to solve the above problems, an image display apparatus according to an aspect of the present invention is an image display apparatus capable of displaying a screen for causing a user to recognize a virtual space, which includes a plurality of selection targets and a plurality of the selection targets. A virtual space construction unit for arranging an image of a user interface including an operation object for selecting any of selection targets on a specific plane in the virtual space, and a real space in which the image display device is present An external information acquisition unit for acquiring information, an object recognition unit for recognizing a specific object existing in the real space based on the information, and an image of the specific object arranged on the plane in the virtual space A pseudo three-dimensionalization processing unit, a contact determination unit which determines whether or not an image of the specific object contacts the operation object in the plane, and an image of the specific object is the operation object An interaction operation unit that calculates an effect exerted on the operation object by the movement of the image of the specific object when it is determined that the plurality of objects have come into contact with the test object; And a selection determination unit that determines which of the selection targets has been selected.

  In the image display device, the interaction operation unit may calculate the action based on a velocity vector of the image of the specific object at a contact point between the image of the specific object and the operation object.

  In the image display device, the interaction operation unit may calculate the action by adding the velocity vector of the image of the specific object to a velocity vector of the operation object.

  In the above-mentioned image display device, the selection determination unit is configured to select the plurality of selections in the advancing direction of the operation object, the absolute value of the velocity vector of the operation object after the action being exerted is equal to or greater than a threshold. If any of the objects exists, it may be determined that the selection object existing in the traveling direction is selected.

  In the image display device described above, the selection determining unit is configured such that the operation object after the action is exerted is positioned in any of a plurality of determination areas respectively set for the plurality of selection targets. In this case, it may be determined that the selection target corresponding to the determination region located is selected.

  In the image display device, when the selection determining unit determines that none of the plurality of selection targets is selected, the virtual space configuration unit returns the operation object to a position before the action is exerted. Also good.

  The image display device further includes a motion detection unit that detects a motion of the image display device, and the virtual space configuration unit displays the image of the user interface when the motion of the image display device is within a predetermined range. It may be arranged in the virtual space.

  In the image display device, the virtual space configuration unit may delete the image of the user interface from the virtual space when the movement of the image display device exceeds the predetermined range.

  In the image display device, the external information acquisition unit may be a camera built in the image display device.

  In the image display device, each of the plurality of selection targets may be an item representing the content of an operation on the image display device.

  In the image display device, each of the plurality of selection targets may be a position specified in a two-dimensionally displayed drawing.

  An image processing method according to another aspect of the present invention is an image display method executed by an image display apparatus capable of displaying a screen for causing a user to recognize a virtual space, which includes a plurality of selection targets and a plurality of selection targets. Placing an image of a user interface including an operation object for selecting any of the objects on a specific plane in the virtual space; and (a) information on the real space in which the image display device is present Acquiring a specific object existing in the real space based on the information, and arranging an image of the specific object on the plane in the virtual space Step (d), Step (e) of determining whether the image of the specific object in the plane is in contact with the operation object, and the image of the specific object is the object of the operation Calculating the effect of the movement of the image of the specific object on the operation object when it is determined that the plurality of objects are touched by the operation object on which the action is exerted. And (g) determining which of the selection targets has been selected.

  An image processing program according to still another aspect of the present invention is an image display program for displaying on the image display device a screen for causing a user to recognize a virtual space, which includes a plurality of selection targets and the plurality of selections. Placing an image of a user interface including an operation object for selecting any of the objects on a specific plane in the virtual space; and (a) information on the real space in which the image display device is present Acquiring a specific object existing in the real space based on the information, and arranging an image of the specific object on the plane in the virtual space Step (d), Step (e) of determining whether the image of the specific object has come into contact with the manipulation object in the plane, and an image of the specific object Step (f) of calculating an effect exerted on the operation object by the movement of the image of the specific object when it is determined that the operation object is touched, and the operation object affected by the operation And (g) determining which one of the plurality of selection objects has been selected.

  According to the present invention, the image of the user interface is arranged on a specific plane in the virtual space, and the image of the specific object existing in the real space is arranged on the plane, and included in the user interface by the image of the object Since selection for a plurality of selection targets is performed via the operation object, it is possible to realize an intuitive and user interface with few erroneous operations by the user's gesture in the real space with a simple configuration.

FIG. 1 is a block diagram showing a schematic configuration of an image display apparatus according to a first embodiment of the present invention. FIG. 2 is a schematic view showing a state in which the image display apparatus is attached to the user. FIG. 3 is a schematic view showing an example of a screen displayed on the display unit shown in FIG. FIG. 4 is a schematic view showing an example of a virtual space corresponding to the screen shown in FIG. FIG. 5 is a schematic view showing an example of a screen on which an operation menu is displayed. FIG. 6 is a flow chart showing the operation of the image display device shown in FIG. FIG. 7 is a flowchart showing menu selection determination processing. FIG. 8 is a schematic diagram for explaining menu selection determination processing. FIG. 9 is a flowchart showing menu selection determination processing in the modification of the first embodiment of the present invention. FIG. 10 is a flowchart showing the operation of the image display device according to the second embodiment of the present invention. FIG. 11 is a schematic view illustrating the operation map. FIG. 12 is a flowchart showing determination processing of a movement destination on a map.

  Hereinafter, a display device according to an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited by these embodiments. Further, in the descriptions of the respective drawings, the same parts are denoted by the same reference numerals.

First Embodiment
FIG. 1 is a block diagram showing a schematic configuration of an image display apparatus according to a first embodiment of the present invention. The image display device 1 according to the present embodiment is a device that causes a user to view a screen with both eyes to recognize a three-dimensional virtual space. As shown in FIG. 1, the image display device 1 includes a display unit 11 on which a screen is displayed, a storage unit 12, an operation unit 13 that performs various arithmetic processing, and information on the outside of the image display device 1 (hereinafter referred to as It includes an external information acquisition unit 14 for acquiring external information) and a motion detection unit 15 for detecting the motion of the image display device 1.

  FIG. 2 is a schematic view showing a state in which the image display device 1 is attached to the user 2. For example, as shown in FIG. 2, the image display device 1 is configured by attaching a general-purpose display device 3 provided with a display such as a smartphone, a personal digital assistant (PDA), a portable game device and a camera to the holder 4. be able to. In this case, the display device 3 is mounted with the display provided on the front side facing the inside of the holder 4 and the camera 5 provided on the back side facing the outside of the holder 4. Inside the holder 4, lenses are respectively provided at positions corresponding to the left and right eyes of the user, and the user 2 views the display of the display device 3 through these lenses. In addition, the user 2 can see the screen displayed on the image display device 1 hands-free by wearing the holder 4 on the head.

  However, the appearances of the display device 3 and the holder 4 are not limited to those shown in FIG. For example, instead of the holder 4, a simple box-shaped holder with a built-in lens may be used. In addition, a dedicated image display device in which a display, an arithmetic device, and a holder are integrated may be used. Such a dedicated image display device is also referred to as a head mounted display.

  Referring back to FIG. 1, the display unit 11 is a display including a display panel and a drive unit formed of, for example, liquid crystal or organic EL (electroluminescence).

  The storage unit 12 is a computer-readable storage medium such as a semiconductor memory such as a ROM or a RAM. The storage unit 12 includes, in addition to the operating system program and the driver program, a program storage unit 121 that stores application programs for executing various functions, various parameters used during execution of these programs, and the like, and the display unit 11. An image data storage unit 122 for storing image data of content to be displayed (still image or moving image); and an object storage unit 123 for storing data of an image of a user interface used when performing an input operation during content display. Including. In addition to this, the storage unit 12 may store voice data of voices and sound effects output during execution of various applications.

  The arithmetic unit 13 is configured by using, for example, a central processing unit (CPU) or a graphics processing unit (GPU), and reads various programs stored in the program storage unit 121 to comprehensively control each unit of the image display device 1. While performing control, various arithmetic processing for displaying various images is executed. The detailed configuration of the calculation unit 13 will be described later.

  The external information acquisition unit 14 acquires information on a physical space in which the image display device 1 is present. The configuration of the external information acquisition unit 14 is not particularly limited as long as it can detect the position and the movement of an object existing in the real space, and for example, an optical camera, an infrared camera, an ultrasonic transmitter, a receiver, etc. It can be used as the acquisition unit 14. In the present embodiment, the camera 5 incorporated in the display device 3 is used as the external information acquisition unit 14.

  The motion detection unit 15 includes, for example, a gyro sensor and an acceleration sensor, and detects a motion of the image display device 1. Based on the detection result of the motion detection unit 15, the image display device 1 determines the state of the head of the user 2 (whether or not the user 2 is stationary), the line of sight direction of the user 2 (upper and lower), It is possible to detect a relative change or the like in the gaze direction.

  Next, the detailed configuration of the calculation unit 13 will be described. The operation unit 13 causes the display unit 11 to display a screen for causing the user 2 to recognize the three-dimensionally configured virtual space by reading the image display program stored in the program storage unit 121 and causes the gesture of the user 2 to be displayed. Execute the operation of accepting the input operation by.

  Here, FIG. 3 is a schematic view showing an example of a screen displayed on the display unit 11. FIG. 4 is a schematic view showing an example of a virtual space corresponding to the screen shown in FIG. When displaying the content of a still image or a moving image as a virtual space, as shown in FIG. 3, the display panel of the display unit 11 is divided into two areas, and two screens 11a and 11b provided with parallax each other are Display in the area of The user 2 can recognize a three-dimensional image (that is, a virtual space) as shown in FIG. 4 by viewing the screens 11a and 11b with the left and right eyes, respectively.

  As shown in FIG. 1, the operation unit 13 includes a motion determination unit 131, a virtual space configuration unit 132, an object recognition unit 133, a pseudo three-dimensionalization processing unit 134, a virtual space display control unit 135, and contact determination. And an interaction operation unit 137, a selection determination unit 138, and an operation execution unit 139.

  The movement determination unit 131 determines the movement of the head of the user 2 based on the detection signal output from the movement detection unit 15. Specifically, the movement determination unit 131 determines whether or not the head of the user is stationary, and in which direction the head is directed when the head is moving.

  The virtual space configuration unit 132 arranges objects in the virtual space to be recognized by the user. In detail, the virtual space configuration unit 132 reads the image data from the image data storage unit 122, and according to the state of the head of the user, a partial area (a field of view of the user Cut out an area in the range of (1) or change the sense of depth of an object in an image.

  Also, the virtual space configuration unit 132 reads from the object storage unit 123 the image data of the user interface used when the user performs the operation by the gesture, and arranges the image of this user interface on a specific plane in the virtual space .

  FIG. 5 is a schematic view showing an example of a screen on which a user interface is displayed. In FIG. 5, an operation menu 16 is displayed as a user interface. The operation menu 16 includes a plurality of menu items 16a to 16d to be selected, and an operation object 17 used when selecting one of the selection items. The menu items 16 a to 16 d represent the contents (menus A to D) of the operation on the image processing apparatus 1. Further, the operation object 17 is a tool for selecting a desired menu item by the user operating virtually. In the present embodiment, a circular icon is displayed as the operation object 17, but the shape or size of the operation object is not particularly limited. For example, an icon of a desired shape and size such as a bar or arrow shape may be used as the operation object.

  The object recognition unit 133 recognizes a specific object existing in the real space based on the external information acquired by the external information acquisition unit 14. As described above, when the camera 5 (see FIG. 2) is used as the external information acquisition unit 14, the object recognition unit 133 is preset by image processing on an image of the physical space acquired by the camera 5 taking an image. Recognize objects with features. The specific object to be recognized may be the hand or finger of the user 2 or an object such as a stylus pen or a stick.

  The features used when recognizing a specific object are preset according to the recognition target. For example, when recognizing the hand or finger of the user 2, the object recognition unit 133, for example, includes pixels having color feature amounts (pixel values of R, G, B, color ratio, color difference, etc.) included in the skin color range. An area where a predetermined number or more of these pixels are gathered is extracted as an area in which a finger or a hand appears. Alternatively, an area in which a finger or a hand appears may be extracted based on the area and the perimeter of the area in which the extracted pixels are collected.

  The pseudo three-dimensional processing unit 134 executes processing for arranging the image of the specific object recognized by the object recognition unit 133 on a specific plane in the virtual space. Specifically, the image of a particular object is placed in the same plane as the image of the user interface. Specifically, the pseudo three-dimensionalization processing unit 134 generates a two-dimensional image including an image of a specific object, and this two-dimensional image has the same sense of depth as the plane on which the image of the user interface is displayed in the virtual space By setting the parallax as described above, the user is made to recognize that the image of the object is present in a plane in a three-dimensional virtual space.

  The virtual space display control unit 135 combines the two-dimensional image generated by the pseudo three-dimensional processing unit 134 with the virtual space configured by the virtual space configuration unit 132 and causes the display unit 11 to display the combined image. Thus, for example, as shown in FIG. 5, an image (hereinafter referred to as a finger image) 18 of the user's finger is superimposed and displayed on the operation menu 16.

  The touch determination unit 136 determines whether or not the image for the specific object (for example, the finger image 18) recognized by the object recognition unit 133 is in contact with the operation object in the image for the user interface (for example, the operation menu 16). Do.

  When it is determined that the operation object and the image of the specific object are in contact with each other, the interaction calculation unit 137 calculates an effect exerted on the operation object by the movement of the image of the specific object.

  The selection determination unit 138 determines which of the plurality of selection targets is selected by the operation object, based on the action calculated by the interaction calculation unit 137. For example, by moving the finger image 18 in FIG. 5 to bring it into contact with the operation object 17, the operation object 17 is moved by the action of moving the finger image 18 in the upper left direction in the drawing. When the predetermined area in the quadrant of the item 16a is entered, it is determined that the menu item 16a (ie, "menu A") is selected.

  When it is determined that one of the plurality of selection targets is selected, the operation execution unit 139 executes an operation according to the selected selection target. The content of the operation is not particularly limited as long as the operation is executable in the image display device 1. Specific examples include an operation of switching on / off of image display, an operation of switching an image being displayed to another image, and the like.

  Next, the operation of the image display device 1 will be described. FIG. 6 is a flowchart showing the operation of the image display device 1, and shows an operation of accepting an input operation by a gesture of the user during execution of the image display program in the virtual space. In the following description, it is assumed that the operation menu 16 shown in FIG. 5 is displayed as an image of the user interface.

In step S101, the calculation unit 13 waits for the display of the operation menu 16.
In the subsequent step S102, the calculation unit 13 determines whether the head of the user is stationary. Here, that the head is at rest includes the case where the head is slightly moved as well as the state where the user's head is not completely moved. In detail, the movement determination unit 131 determines whether the acceleration and angular acceleration of the image display device 1 (that is, the head) are equal to or less than predetermined values based on the detection signal output from the movement detection unit 15 . When the acceleration and the angular acceleration exceed the predetermined values, the movement determination unit 131 determines that the head of the user is not stationary (No at Step S102). In this case, the operation of the calculation unit 13 returns to step S101 and continues to wait for the operation menu display.

  On the other hand, when the calculation unit 13 determines that the head of the user is stationary (step S102: Yes), subsequently, whether or not the user holds his hand in front of the camera 5 (see FIG. 2) It determines (step S103). Specifically, the object recognition unit 133 determines whether or not there is a region in which a predetermined number or more of pixels having the color feature amount of the hand (skin color) are gathered by image processing on the image acquired by the camera 5 . If there is no area where a predetermined number or more of pixels having the color feature value of a hand are gathered, the object recognition unit 133 determines that the user does not hold their hand (step S103: No). In this case, the operation of the calculation unit 13 returns to step S101.

  On the other hand, when it is determined that the user holds the hand over (step S103: Yes), the calculation unit 13 displays the operation menu 16 on the display unit 11 (step S104). If the user's head moves during this time, the operation unit 13 displays the operation menu 16 so as to follow the movement of the user's head (that is, the direction of the user's line of sight). If the operation menu 16 is fixed with respect to the background virtual space despite the change in the user's gaze direction, the operation menu 16 deviates from the user's field of view, and the user tries to operate from this For the screen to be unnatural.

  In steps S103 and S104, the user holding the hand over the camera 5 is used as a trigger to display the operation menu 16. However, in addition to the hand, for example, a stylus pen or a bar is set in advance. Holding an object in front of the camera 5 may be used as a trigger.

  In the subsequent step S105, the calculation unit 13 determines again whether the head of the user is stationary. When it is determined that the user's head is not stationary (step S105: No), the calculation unit 13 deletes the operation menu 16 (step S106). Thereafter, the operation of the calculation unit 13 returns to step S101.

  Here, in the steps S101 and S105, the fact that the head of the user is at rest is used as the condition for displaying the operation menu, in general, the user operates the image display apparatus 1 while moving the head widely. Because there is no need to Conversely, the fact that the user is moving the head greatly is considered that the user is immersed in the virtual space being watched, and displaying the operation menu 16 at such time This is because the user feels troublesome.

  When it is determined that the user's head is stationary (step S105: Yes), the calculation unit 13 executes recognition processing of a specific object in the real space (step S107). In the following, it is assumed that the user's finger is recognized as a specific object. In detail, the object recognition unit 133 performs image processing on the image in the physical space acquired by the external information acquisition unit 14 so that the region where a predetermined number or more of pixels having the skin color feature amount are collected is It is extracted as an area where the finger of. In addition, the pseudo three-dimensional processing unit 134 generates a two-dimensional image of the extracted area (that is, the finger image 18), and the virtual space display control unit 135 superimposes the two-dimensional image on the operation menu 16 It is displayed on the display unit 11.

  In the subsequent step S108, the computing unit 13 determines whether the image of the specific object is in contact with the operation object. Specifically, the touch determination unit 136 determines whether a part of the pixel area of the finger image 18 overlaps the pixel area of the operation object 17. If the two do not overlap, the touch determination unit 136 determines that the finger image 18 is not in contact with the operation object 17 (step S108: No). In this case, the operation of the calculation unit 13 returns to step S104.

  On the other hand, when it is determined that the image of the object is in contact with the operation object (Yes at step S108), the calculation unit 13 executes menu selection determination processing (step S109).

FIG. 7 is a flowchart showing menu selection determination processing. FIG. 8 is a schematic diagram for explaining the menu selection determination process.
First, in step S121, the interaction calculation unit 137 initializes the velocity vector of the operation object.

  In the following step S122, the interaction calculation unit 137 calculates the velocity vector of the object image from the portion of the image of the object in contact with the operation object. For example, in the case of FIG. 8, when the user moves his / her finger, the finger image 18 moves in the direction of the dashed arrow and contacts the operation object 17 at point P, the velocity vector of the finger image 18 at this point P v1 is calculated.

  In the following step S123, the interaction calculation unit 137 gives the velocity vector calculated in step S122 as the velocity vector of the operation object. For example, in the case of FIG. 8, a velocity vector v2 having the same direction and amount as the velocity vector v1 of the finger image 18 is given to the manipulation object 17.

  In the following step S124, the interaction calculation unit 137 updates the position of the manipulation object to which the velocity vector has been newly given. For example, in the case of FIG. 8, the operation object 17 moves to the position indicated by the broken line by the velocity vector v2 being given.

  In FIG. 8, since the circular operation object 17 is displayed, even if the velocity vector v2 is given, the direction of the operation object 17 does not change but only the position changes. However, when, for example, a bar-like object is used as the operation object, the direction may be changed by rotating the operation object according to the position at which the velocity vector is given. The point is that the position and orientation of the manipulation object when the velocity vector is given to the contact point with the finger image 18 may be calculated according to a general physical law. At this time, the position of the operation object 17 may be inertially changed. Thereby, the user can operate the operation object 17 in a more natural sense.

  In the subsequent step S125, the selection determining unit 138 determines whether a determination region of a specific menu item exists in the direction of the velocity vector of the operation object, and the absolute value of the velocity vector is equal to or greater than a threshold. Here, the determination area is an area in the image of the user interface, and is an area set in advance to determine whether a specific menu item is selected. As the determination area, it is possible to set a desired area such as an area where the menu item is displayed as text or an icon representing the menu item. In FIG. 8, determination areas 19 a to 19 d are respectively provided in the vicinity of the display areas of the menu items 16 a to 16 d.

  When the determination area of the specific menu item exists in the direction of the velocity vector of the operation object, and the absolute value of the velocity vector is equal to or more than the threshold (step S125: Yes), the selection determination unit 138 It is determined that the corresponding menu item has been selected (step S126). For example, in the case of FIG. 8, the determination area 19a exists on the extension of the velocity vector v2. In this case, it is determined that the menu item 16a corresponding to the determination area 19a is selected. Thereafter, the process returns to the main routine.

  On the other hand, if there is no determination region of a specific menu item in the direction of the velocity vector of the operation object, or if the absolute value of the velocity vector is less than the threshold (step S125: No), the selection determining unit 138 It is determined that the selection of the menu item has failed (step S127). In this case, the selection determination unit 138 further returns the operation object to the initial position (step S128). For example, in FIG. 8, when the operation object 17 pressed by the finger image 18 moves in the direction of the velocity vector v2 but reaches the determination area 19a, the operation object 17 returns to the original position and is displayed. Ru. Thereafter, the process returns to the main routine.

  In step S110 following step S109, the computing unit 13 determines whether the selection of the menu item has succeeded. When the selection of the menu item is successful (step S110: Yes), the acceptance of the input operation by the gesture of the user is ended. In this case, operation unit 13 executes an operation corresponding to the selected menu item. On the other hand, when the selection of the menu item fails (step S110: No), the operation of the calculation unit 13 returns to step S104.

  As described above, according to the first embodiment of the present invention, since the operation menu is displayed in the virtual space when the user causes the head to substantially rest, the intention of the user who intends to start the input operation is achieved. It can display along. That is, even if the user unintentionally places the hand over the camera 5 (see FIG. 2) of the image display device 1 or if an object similar to the hand is accidentally reflected on the camera 5, Since the operation menu is not displayed, the user can continue watching the virtual space without being interrupted by the operation menu.

  Further, according to the above-described embodiment, the selection target is selected through the operation object instead of directly selecting the selection target based on the image of the object existing in the real space, so that erroneous operations can be reduced. For example, in FIG. 8, assuming that the menu item of the judgment area which is in direct contact with the finger image 18 is selected, another judgment area 19d is touched when the finger image 18 is moved to touch the judgment area 19a. There is a risk of On the other hand, in the above embodiment, the selection target (determination area) is finally determined by the position and the motion of the operation object 17. Therefore, the user can easily perform an accurate operation without considering the positional relationship between the finger image 18 and the determination area to be selected.

  Further, according to the above-described embodiment, since the arithmetic processing for causing the movement of the finger image 18 to act on the operation object 17 is performed, the user obtains a realistic feeling as if operating the operation object 17 with his own finger. be able to. As a result, the user can perform intuitive operations and reduce unintended operations.

  Further, according to the above-described embodiment, the image of the user interface is arranged in a specific plane in the virtual space, and the image of a specific object (for example, a user's finger) for performing an operation by a gesture is arranged in the same plane. Thus, the user can perform an intuitive operation in the same manner as touching the operation object. For example, in the above-described embodiment, the user can operate the circular operation object 17 as if it were pushing or flipping as it was.

  Further, according to the above embodiment, since the image of the user interface and the image of the specific object are two-dimensionally displayed, the amount of operation required to display these and the operation required to determine which selection target is selected It is also possible to reduce the amount. Therefore, even when a general-purpose display device such as a smartphone is used, high-speed processing is possible, and the user can perform input operation in a sense of real time.

  Further, according to the above-described embodiment, the external information acquisition unit 14 provided in the image display device 1 acquires information related to the input operation by the user's gesture. That is, as in the prior art, it is not necessary to provide a dedicated sensor or an external device for detecting a user's gesture. As a result, the configuration of the entire system can be simplified, and the cost can be significantly reduced. In addition, calculation for analyzing information input from a dedicated sensor or the like is not necessary. Therefore, even when a small display device such as a smartphone is used, it is possible to realize an image display device capable of real-time input operation by a gesture.

  Further, according to the above-described embodiment, among the external information acquired by the external information acquisition unit 14, only the information related to the specific object set in advance is displayed in the virtual space. The desired operation can be performed without being confused by an object not involved in

  In the above embodiment, the user moving the head is a trigger for deleting the operation menu (see steps S105 and S106 in FIG. 6), but the trigger for deleting the operation menu is not limited to this. For example, a menu item "delete the operation menu" may be provided in the operation menu. Alternatively, when a specific operation is performed on the operation object 17, the operation menu may be deleted. As one example, an operation that repels strongly as the operation object 17 pops up from the operation menu 16 can be mentioned.

(Modification)
Next, a modification of the first embodiment of the present invention will be described. FIG. 9 is a flowchart showing menu selection determination processing in the present modification. The configuration and the operation of the image display device used in this modification are the same as those of the first embodiment (see FIGS. 1 and 6). Further, the processes in steps S121 to S124 in FIG. 9 are also the same as those in the first embodiment (see FIG. 7).

  In step S131 following step S124, the selection determination unit 138 determines whether the operation object is present in the determination area of the specific menu item. For example, as shown in FIG. 8, when the operation object 17 is displayed at the position indicated by the broken line, it is determined that the operation object 17 exists in the determination area 19a of the menu item 16a.

  If the operation object exists in the determination area of the specific menu item (step S131: Yes), the selection determination unit 138 determines that the menu item corresponding to the determination area in which the operation object exists is selected (step S132). ). Thereafter, the process returns to the main routine.

  On the other hand, when the operation object does not exist in any of the determination items of the menu items (step S131: No), the selection determination unit 138 further separates the image of the object (the finger image 18 in FIG. 8) from the operation object. It is determined whether or not it is (step S133).

  When the image of the object is not separated from the operation object (step S133: No), the process returns to step S122. In this case, the operation object is further moved according to the movement of the image of the object (see steps S122 to S124).

  On the other hand, when the image of the object is separated from the operation object (step S133: Yes), the selection determining unit 138 determines that selection of the menu item has failed (step S134). In this case, the selection determination unit 138 further returns the position of the operation object to the initial position (step S135). Thereafter, the process returns to the main routine.

  As described above, according to the present modification, since the operation object is carried by the image of the object until the operation object falls within any of the plurality of determination regions, the user can more reliably obtain the desired menu item. It is possible to select.

Second Embodiment
Next, a second embodiment of the present invention will be described. FIG. 10 is a flowchart showing the operation of the image display apparatus according to the second embodiment of the present invention, and shows an operation of receiving an input operation by a gesture of the user during execution of the image display program in the virtual space. FIG. 11 is a schematic view showing an example of an image of a user interface displayed in the present embodiment. The configuration of the image display apparatus according to the present embodiment is the same as that shown in FIG.

  As shown in FIG. 11, in the present embodiment, the operation map 20 is illustrated as a user interface. The operation map 20 includes a map 21 and an operation object 22. In the operation map 20, the real estate property is the selection target, and the area occupied by each property displayed on the map 21 is the determination area of the selection target. Also, as the operation object 22, a circular icon is shown as in the first embodiment, but of course, an icon having a desired shape and size such as a rod shape or an arrow shape is used as the operation object 22. You may use.

The operation of the image display apparatus according to the present embodiment will be described with reference to FIG.
First, in step S201, the calculation unit 13 waits for display of the operation map 20. The operations of the subsequent steps S202 and S203 are the same as the steps S102 and S103 shown in FIG.

  When it is determined that the user holds the hand in step S203 (step S203: Yes), the calculation unit 13 displays the operation map 20 illustrated in FIG. 11 on the display unit 11 (step S204).

  The operations of subsequent steps S205 to S208 are the same as steps S105 to S108 shown in FIG. In step S206, the operation map 20 displayed in step S204 is deleted.

  When it is determined that the image of the object (the finger image 18 in FIG. 11) is in contact with the operation object 22 in step S208 (step S208: Yes), the calculation unit 13 on the map 21 of the operation object 22 The movement destination determination processing is executed (step S209).

  FIG. 12 is a flowchart showing the determination processing of the moving destination on the map 21. The process of steps S221 to S224 is the same as that of steps S121 to S124 shown in FIG.

  In step S225 following step S224, the selection determination unit 138 determines whether the operation object 22 has been detached from the image of the object (finger image 18). When the image of the object is not separated from the operation object 22 (step S225: No), the process returns to step S222.

  On the other hand, when the image of the object is separated from the operation object 22 (step S225: Yes), the selection determination unit 138 further determines whether the operation object 22 exists on the map 21 (step S226).

  If the operation object 22 exists on the map 21 (step S226: Yes), the selection determination unit 138 determines that a specific point on the map 21 at which the operation object 22 is located is selected (step S227). ). Thereafter, the process returns to the main routine.

  On the other hand, when the operation object 22 does not exist on the map 21 (step S226: No), for example, when the operation object 22 is located in the margin area 23 around the map 21, the selection determination unit 138 It is determined that the selection of the movement destination has failed (step S228). Thereafter, the process returns to the main routine.

  In step S <b> 210 following step S <b> 209, the calculation unit 13 determines whether or not the selection of the movement destination on the map 21 has succeeded. When the selection of the movement destination is successful (step S210: Yes), the reception of the input operation by the gesture of the user is ended. In this case, the computing unit 13 executes a process of displaying the selected destination in the virtual space. On the other hand, when the selection of the movement destination fails (step S210: No), the operation of the calculation unit 13 returns to step S204.

  As described above, according to the second embodiment of the present invention, as in the case of a map, even when the determination areas are arranged at random or when the size of the determination areas is not constant, intuition Operations can be performed, and erroneous operations can be reduced.

  In the second embodiment, the case of selecting a property shown on the map has been described, but if it is a position (area or point) that can be identified on a two-dimensionally displayed drawing, this is the case. Embodiments can be applied. Specifically, the case of selecting a specific room displayed in the floor plan of a house or the case of selecting a specific floor in a longitudinal sectional view of a building may be mentioned.

  The present invention is not limited to the first and second embodiments and the modification, and by combining the plural components disclosed in the first and second embodiments and the modification as appropriate, Various inventions can be formed. For example, some of the components may be excluded from all the components shown in the first and second embodiments and the modifications, or they may be shown in the first and second embodiments and the modifications. You may combine and form a component suitably.

DESCRIPTION OF SYMBOLS 1 image display apparatus 2 user 3 display apparatus 4 holder 5 camera 11 display part 11a, 11b Screen 12 memory | storage part 13 calculating part 14 external information acquisition part 15 motion detection part 16 operation menu 16a-16d menu item 17, 22 object for operation 18 Finger images 19a to 19d Determination area 20 Operation map 21 Map 23 Margin area 121 Program storage section 122 Image data storage section 123 Object storage section 131 Motion determination section 132 Virtual space configuration section 133 Object recognition section 134 Pseudo three-dimensional processing section 135 Virtual space display control unit 136 Contact determination unit 137 Interaction operation unit 138 Selection determination unit 139 Operation execution unit

Claims (15)

An image display apparatus capable of displaying a screen for causing a user to recognize a three-dimensional virtual space,
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Virtual space components arranged two-dimensionally in a specific plane in a typical virtual space;
An external information acquisition unit that acquires information on a physical space in which the image display device is present;
An object recognition unit that recognizes a specific object existing in the real space based on the information;
A pseudo three-dimensional processing unit that arranges the image of the specific object two-dimensionally in the three-dimensional virtual space so as to have the same depth as the plane;
A touch determination unit that determines whether or not an image of the specific object contacts the operation object on the plane;
An interaction operation unit that calculates an action that the movement of the image of the specific object exerts on the operation object when it is determined that the image of the specific object contacts the operation object;
A selection determination unit that determines which one of the plurality of selection objects is selected by the operation object on which the action is exerted;
An image display apparatus comprising: An image display apparatus capable of displaying a screen for causing a user to recognize a three-dimensional virtual space,
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Virtual space components arranged in a specific plane in a virtual space,
An external information acquisition unit that acquires information on a physical space in which the image display device is present;
An object recognition unit that recognizes a specific object existing in the real space based on the information;
A pseudo three-dimensionalization processing unit that arranges an image of the specific object in the three-dimensional virtual space so as to have the same sense of depth as the plane;
A touch determination unit that determines whether or not an image of the specific object contacts the operation object on the plane;
An interaction operation unit that calculates an action that the movement of the image of the specific object exerts on the operation object when it is determined that the image of the specific object contacts the operation object;
A selection determination unit that determines which one of the plurality of selection objects is selected by the operation object on which the action is exerted;
Equipped with
Wherein each of the plurality of selection target is an item that represents the contents of operation on the image display device, images display. The image according to claim 1 or 2 , wherein the interaction calculating unit calculates the action based on a velocity vector of the image of the specific object at a contact point between the image of the specific object and the operation object. Display device. An image display apparatus capable of displaying a screen for causing a user to recognize a three-dimensional virtual space,
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Virtual space components arranged in a specific plane in a virtual space,
An external information acquisition unit that acquires information on a physical space in which the image display device is present;
An object recognition unit that recognizes a specific object existing in the real space based on the information;
A pseudo three-dimensionalization processing unit that arranges an image of the specific object in the three-dimensional virtual space so as to have the same sense of depth as the plane;
A touch determination unit that determines whether or not an image of the specific object contacts the operation object on the plane;
An interaction operation unit that calculates an action that the movement of the image of the specific object exerts on the operation object when it is determined that the image of the specific object contacts the operation object;
A selection determination unit that determines which one of the plurality of selection objects is selected by the operation object on which the action is exerted;
Equipped with
The interaction calculating unit calculates the action based on a velocity vector of an image of the specific object at a contact point between the image of the specific object and the operation object.
The selection determining unit is configured such that an absolute value of a velocity vector of the operation object after the action is exerted is equal to or greater than a threshold, and any one of the plurality of selection objects exists in the traveling direction of the operation object. If you determines that the selection target to be present in the traveling direction is selected, images display. When the operation object after the action is exerted is positioned in any of a plurality of determination areas respectively set for the plurality of selection targets, the selection determination unit The image display apparatus according to claim 3 , wherein it is determined that the selection target corresponding to the existing determination area is selected. The interaction calculation unit according to any one of claims 3 to 5 , wherein the action is calculated by adding the velocity vector of the image of the specific object to the velocity vector of the operation object. Image display device. The virtual space construction unit, when determining the selection determining section and none selected of the plurality of selection target, returning the operation object position before the action is exerted, of claim 3-6 The image display device according to any one of the items. A motion detection unit that detects a motion of the image display device;
A motion determination unit that determines whether or not the user's head is stationary based on the motion of the image display device detected by the motion detection unit;
And further
The virtual space construction unit, when the head of the user is determined to be stationary by the motion determination unit, placing images of the user interface in the virtual space, any one of the claims 1-7 The image display apparatus as described in a term. The virtual space configuration unit is configured to, when the motion determination unit determines that the head of the user is not stationary, when the image of the user interface is arranged in the virtual space, the image of the user interface The image display device according to claim 8 , wherein the image is erased from the virtual space. An image display method executed by an image display apparatus capable of displaying a screen for causing a user to recognize a three-dimensional virtual space, comprising:
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Arranging in a two-dimensional manner in a specific plane in a virtual space,
Acquiring information on a real space in which the image display device is present;
Recognizing a specific object existing in the real space based on the information;
Arranging the image of the specific object in a two-dimensional manner in the three-dimensional virtual space so as to have the same sense of depth as the plane;
(E) determining whether the image of the specific object has touched the manipulation object on the plane;
Calculating an action exerted on the operation object by the movement of the image of the specific object when it is determined that the image of the specific object contacts the operation object;
Determining which of the plurality of selection targets is selected by the operation object on which the action is exerted;
Image display method including. An image display method executed by an image display apparatus capable of displaying a screen for causing a user to recognize a three-dimensional virtual space, comprising:
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Placing in a specific plane in a virtual space (a),
Acquiring information on a real space in which the image display device is present;
Recognizing a specific object existing in the real space based on the information;
Arranging the image of the specific object in the three-dimensional virtual space to have the same sense of depth as the plane; (d)
(E) determining whether the image of the specific object has touched the manipulation object on the plane;
Calculating an action exerted on the operation object by the movement of the image of the specific object when it is determined that the image of the specific object contacts the operation object;
Determining which of the plurality of selection targets is selected by the operation object on which the action is exerted;
Only including,
An image display method , wherein each of the plurality of selection objects is an item representing the content of an operation on the image display device . An image display method executed by an image display apparatus capable of displaying a screen for causing a user to recognize a three-dimensional virtual space, comprising:
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Placing in a specific plane in a virtual space (a),
Acquiring information on a real space in which the image display device is present;
Recognizing a specific object existing in the real space based on the information;
Arranging the image of the specific object in the three-dimensional virtual space to have the same sense of depth as the plane; (d)
(E) determining whether the image of the specific object has touched the manipulation object on the plane;
Calculating an action exerted on the operation object by the movement of the image of the specific object when it is determined that the image of the specific object contacts the operation object;
Determining which of the plurality of selection targets is selected by the operation object on which the action is exerted;
Only including,
The step (f) calculates the action based on a velocity vector of the image of the specific object at the contact point between the image of the specific object and the operation object,
In the step (g), the absolute value of the velocity vector of the manipulation object after the action is exerted is equal to or greater than a threshold, and any one of the plurality of selection objects is selected in the traveling direction of the manipulation object. The image display method which determines that the selection object which exists in this advancing direction was selected, when it exists . An image display program for displaying a screen for causing a user to recognize a three-dimensional virtual space on an image display device,
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Arranging in a two-dimensional manner in a specific plane in a virtual space,
Acquiring information on a real space in which the image display device is present;
Recognizing a specific object existing in the real space based on the information;
Arranging the image of the specific object in a two-dimensional manner in the three-dimensional virtual space so as to have the same sense of depth as the plane;
(E) determining whether the image of the specific object has touched the manipulation object on the plane;
Calculating an action exerted on the operation object by the movement of the image of the specific object when it is determined that the image of the specific object contacts the operation object;
Determining which of the plurality of selection targets is selected by the operation object on which the action is exerted;
An image display program that causes a computer to execute. An image display program for displaying a screen for causing a user to recognize a three-dimensional virtual space on an image display device,
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Placing in a specific plane in a virtual space (a),
Acquiring information on a real space in which the image display device is present;
Recognizing a specific object existing in the real space based on the information;
Arranging the image of the specific object in the three-dimensional virtual space to have the same sense of depth as the plane; (d)
(E) determining whether the image of the specific object has touched the manipulation object on the plane;
Calculating an action exerted on the operation object by the movement of the image of the specific object when it is determined that the image of the specific object contacts the operation object;
Determining which of the plurality of selection targets is selected by the operation object on which the action is exerted;
On your computer ,
An image display program , wherein each of the plurality of selection targets is an item representing the content of an operation on the image display device . An image display program for displaying a screen for causing a user to recognize a three-dimensional virtual space on an image display device,
A three-dimensional image of a user interface which constitutes a three-dimensional virtual space consisting of still images or moving images, and which includes a plurality of selection targets and an operation object for selecting any of the plurality of selection targets Placing in a specific plane in a virtual space (a),
Acquiring information on a real space in which the image display device is present;
Recognizing a specific object existing in the real space based on the information;
Arranging the image of the specific object in the three-dimensional virtual space to have the same sense of depth as the plane; (d)
(E) determining whether the image of the specific object has touched the manipulation object on the plane;
Calculating an action exerted on the operation object by the movement of the image of the specific object when it is determined that the image of the specific object contacts the operation object;
Determining which of the plurality of selection targets is selected by the operation object on which the action is exerted;
On your computer ,
The step (f) calculates the action based on a velocity vector of the image of the specific object at the contact point between the image of the specific object and the operation object,
In the step (g), the absolute value of the velocity vector of the manipulation object after the action is exerted is equal to or greater than a threshold, and any one of the plurality of selection objects is selected in the traveling direction of the manipulation object. The image display program which determines that the selection object which exists in this advancing direction was selected, when it exists .