JP6159069B2 - Display device - Google Patents

Description

  The present invention relates to a display device, a control method, and a control program.

  Some display devices including a display unit can display an image or the like in a three-dimensional manner (for example, see Patent Document 1). Stereoscopic display is realized using parallax of both eyes.

JP 2011-95547 A

  Although the stereoscopic display is a display format familiar to the user, in the conventional display device, the stereoscopic display is used only for the purpose of viewing and has not been used for improving the convenience of operation. An object of the present invention is to provide a display device, a control method, and a control program capable of providing a user-friendly operation method.

The display device according to one aspect displays an image in which the overlap is adjusted so that the closer one of the real object and the predetermined object can be seen in front, corresponding to each of the user's eyes when worn. By doing so, a display unit that three-dimensionally displays the predetermined object, a detection unit that detects the first object and the second object in the real space, and the detected first object and second object A control unit configured to change the predetermined object according to the type associated with the predetermined object based on movement .

  The present invention has an effect that it is possible to provide a user with a highly convenient operation method.

FIG. 1 is a perspective view of a display device according to the first embodiment. FIG. 2 is a front view of the display device worn by the user. FIG. 3 is a diagram illustrating a modification of the display device. FIG. 4 is a diagram illustrating another modification of the display device. FIG. 5 is a diagram illustrating another modification of the display device. FIG. 6 is a block diagram of the display device according to the first embodiment. FIG. 7 is a diagram illustrating an example of control based on the function provided by the control program. FIG. 8 is a diagram illustrating an example of information stored in object data. FIG. 9 is a diagram illustrating an example of information stored in the action data. FIG. 10 is a diagram illustrating an example of information stored in the action data. FIG. 11 is a diagram illustrating an example of information stored in the action data. FIG. 12 is a diagram illustrating an example of information stored in the action data. FIG. 13 is a diagram illustrating an example of information stored in the action data. FIG. 14 is a diagram illustrating an example of information stored in the action data. FIG. 15 is a diagram for describing a first example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 16 is a diagram for describing a first example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 17 is a flowchart illustrating a processing procedure of contact detection processing in the first example. FIG. 18 is a flowchart showing the procedure of the operation detection process in the first example. FIG. 19 is a diagram for describing a second example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 20 is a flowchart showing the procedure of the operation detection process in the second example. FIG. 21 is a diagram for describing a third example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 22 is a diagram for describing a third example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 23 is a flowchart illustrating the procedure of the operation detection process in the third example. FIG. 24 is a diagram for describing a first example of detection of an operation performed by picking up a three-dimensional object. FIG. 25 is a flowchart illustrating a processing procedure of selection detection processing in the first example. FIG. 26 is a flowchart illustrating the procedure of the operation detection process in the first example. FIG. 27 is a diagram for describing a modified example of the first example of detecting an operation performed by picking up a three-dimensional object. FIG. 28 is a diagram for describing a second example of detecting an operation performed by picking up a three-dimensional object. FIG. 29 is a flowchart illustrating a processing procedure of selection detection processing in the second example. FIG. 30 is a diagram for explaining a modified example of the second example of the operation detection performed by picking up the three-dimensional object. FIG. 31 is a diagram for describing a third example of detection of an operation performed by picking up a three-dimensional object. FIG. 32 is a diagram for describing a third example of detection of an operation performed by picking up a three-dimensional object. FIG. 33 is a flowchart of the process procedure of the selection detection process in the third example. FIG. 34 is a flowchart illustrating the procedure of the operation detection process in the third example. FIG. 35 is a diagram for describing a modified example of the third example of detecting an operation performed by picking up a three-dimensional object. FIG. 36 is a perspective view of a display device according to the second embodiment. FIG. 37 is a block diagram of a display device according to the second embodiment. FIG. 38 is a diagram illustrating an example of display control in conjunction with a change in a three-dimensional object. FIG. 39 is a diagram illustrating an example of a trajectory of an operation of bringing a finger into contact with a three-dimensional object for a moment. FIG. 40 is a diagram illustrating an example of a trajectory of an operation for moving a finger along a three-dimensional object. FIG. 41 is a diagram illustrating an example of an operation trajectory for crushing a three-dimensional object with a finger. FIG. 42 is a flowchart illustrating a display control processing procedure executed in conjunction with a change in a three-dimensional object.

  Hereinafter, the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following description. In addition, constituent elements in the following description include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those in a so-called equivalent range.

  First, the overall configuration of the display device 1 according to the first embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a perspective view of the display device 1. FIG. 2 is a front view of the display device 1 attached by the user. As shown in FIG. 1 and FIG. 2, the display device 1 is a head mount type device that is worn on the user's head.

  The display device 1 includes a front surface portion 1a, a side surface portion 1b, and a side surface portion 1c. Front part 1a is arranged in front of the user so as to cover both eyes of the user when worn. The side surface portion 1b is connected to one end portion of the front surface portion 1a, and the side surface portion 1c is connected to the other end portion of the front surface portion 1a. The side surface portion 1b and the side surface portion 1c are supported by a user's ear like a vine of glasses when worn, and stabilize the display device 1. The side part 1b and the side part 1c may be configured to be connected to the back of the user's head when worn.

  The front part 1a is provided with a display part 32a and a display part 32b on the surface facing the user's eyes when worn. The display unit 32a is disposed at a position facing the user's right eye when worn, and the display unit 32b is disposed at a position facing the user's left eye when worn. The display unit 32a displays a right-eye image, and the display unit 32b displays a left-eye image. Thus, by providing the display unit 32a and the display unit 32b that display an image corresponding to each eye of the user when worn, the display device 1 realizes three-dimensional display using binocular parallax. Can do.

  The display unit 32a and the display unit 32b may be configured by a single display device as long as different images can be independently provided to the right and left eyes of the user. For example, one display device may be configured to independently provide different images for the right eye and the left eye by switching at high speed a shutter that blocks the displayed image so that only one eye can see it. . The front part 1a may be configured to cover the eyes of the user so that external light does not enter the eyes of the user when worn.

  The front surface portion 1a includes a photographing unit 40 and a photographing unit 42 on the surface opposite to the surface on which the display unit 32a and the display unit 32b are provided. The photographing unit 40 is disposed in the vicinity of one end portion (right eye side when worn) of the front surface portion 1a, and the photographing portion 42 is located in the vicinity of the other end portion (left eye side when worn) of the front surface portion 1a. Arranged. The imaging unit 40 acquires an image in a range corresponding to the field of view of the user's right eye. The imaging unit 42 acquires an image in a range corresponding to the field of view of the user's left eye. The field of view here is a field of view when the user is looking at the front, for example.

  The display device 1 displays an image photographed by the photographing unit 40 on the display unit 32a as an image for the right eye, and displays an image photographed by the photographing unit 42 on the display unit 32b as an image for the left eye. For this reason, the display device 1 can provide the user with the same scene as when the display device 1 is not worn, even if the field of view is blocked by the front surface portion 1a.

  In addition to the function of providing an actual scene to the user, the display device 1 displays virtual information three-dimensionally and allows the user to operate the virtual information. Have According to the display device 1, the virtual information is displayed so as to be superimposed on the actual scene as if it actually exists. Then, the user can operate the virtual information as if he / she actually touched the virtual information by using his / her hand, for example, to change the virtual information. As described above, the display device 1 provides an intuitive and highly convenient operation method for virtual information. In the following description, virtual information displayed three-dimensionally by the display device 1 may be referred to as a “three-dimensional object”.

  The display device 1 provides the user with the same wide field of view as when the display device 1 is not mounted. And the display apparatus 1 can arrange | position a three-dimensional object by arbitrary magnitude | sizes in the arbitrary positions in this wide visual field. As described above, the display device 1 can display three-dimensional objects having various sizes at various positions in a wide space without being restricted by the size of the display device.

  1 and 2 show an example in which the display device 1 has a shape like glasses (goggles), the shape of the display device 1 is not limited to this. For example, the display device 1 may have a helmet-type shape that covers substantially the upper half of the user's head, like the display device 2 shown in FIG. 3. Alternatively, the display device 1 may have a mask type shape that covers almost the entire face of the user, like the display device 3 shown in FIG. The display device 1 may be configured to be connected to an external device 4d such as an information processing device or a battery device by wire or wirelessly like the display device 4 shown in FIG.

  Next, a functional configuration of the display device 1 will be described with reference to FIG. FIG. 6 is a block diagram of the display device 1. As shown in FIG. 6, the display device 1 includes an operation unit 13, a control unit 22, a storage unit 24, display units 32a and 32b, photographing units 40 and 42, a detection unit 44, and a distance measurement unit 46. And have. The operation unit 13 receives basic operations such as starting and stopping the display device 1 and changing the operation mode.

  The display units 32 a and 32 b include display devices such as a liquid crystal display (LCD) and an organic EL (Organic Electro-Luminescence) panel, and display various types of information according to control signals input from the control unit 22. The display units 32a and 32b may be projection devices that project an image onto the user's retina using a light source such as a laser beam.

  The imaging units 40 and 42 electronically capture an image using an image sensor such as a charge coupled device image sensor (CCD) or a complementary metal oxide semiconductor (CMOS). The photographing units 40 and 42 convert the photographed images into signals and output the signals to the control unit 22.

  The detection unit 44 detects an actual object that exists in the imaging range of the imaging units 40 and 42. For example, the detection unit 44 detects an object that matches a pre-registered shape (for example, the shape of a human hand) among real objects existing in the imaging range. The detection unit 44 detects the range (shape and size) of the actual object in the image based on the brightness, saturation, hue edge, and the like of the object even if the shape is not registered in advance. It may be configured.

  The distance measuring unit 46 measures the distance to an actual object existing in the imaging range of the imaging units 40 and 42. The distance to the actual object is measured for each eye based on the position of each eye of the user wearing the display device 1. For this reason, when the reference position where the distance measuring unit 46 measures the distance is shifted from the position of each eye, the measured value of the distance measuring unit 46 indicates the distance to the eye position according to the difference. It is corrected to represent.

  In the present embodiment, the imaging units 40 and 42 also serve as the detection unit 44 and the distance measurement unit 46. That is, in this embodiment, an object within the shooting range is detected by analyzing the images shot by the shooting units 40 and 42. Furthermore, the distance from the object is measured (calculated) by comparing the object included in the image captured by the capturing unit 40 with the object included in the image captured by the capturing unit 42.

  The display device 1 may include a detection unit 44 separately from the imaging units 40 and 42. The detection unit 44 may be, for example, a sensor that detects an actual object existing in the imaging range using at least one of visible light, infrared light, ultraviolet light, radio waves, sound waves, magnetism, and capacitance. The display device 1 may include a distance measuring unit 46 separately from the imaging units 40 and 42. The distance measuring unit 46 is a sensor that detects the distance to an actual object existing in the imaging range using at least one of visible light, infrared light, ultraviolet light, radio waves, sound waves, magnetism, and capacitance, for example. Also good. The display device 1 may include a sensor that can also serve as the detection unit 44 and the distance measurement unit 46, such as a sensor using a TOF (Time-of-Flight) method.

  The control unit 22 includes a CPU (Central Processing Unit) that is a calculation unit and a memory that is a storage unit, and implements various functions by executing programs using these hardware resources. Specifically, the control unit 22 reads a program or data stored in the storage unit 24 and expands it in a memory, and causes the CPU to execute instructions included in the program expanded in the memory. The control unit 22 reads / writes data from / to the memory and the storage unit 24 and controls the operation of the display unit 32a and the like according to the execution result of the instruction by the CPU. When the CPU executes an instruction, the data developed in the memory and the operation detected via the detection unit 44 are used as a part of parameters and determination conditions.

  The storage unit 24 includes a nonvolatile storage device such as a flash memory, and stores various programs and data. The program stored in the storage unit 24 includes a control program 24a. The data stored in the storage unit 24 includes object data 24b, action data 24c, and virtual space data 24d. The storage unit 24 may be configured by a combination of a portable storage medium such as a memory card and a read / write device that reads from and writes to the storage medium. In this case, the control program 24a, object data 24b, action data 24c, and virtual space data 24d may be stored in a storage medium. Further, the control program 24a, the object data 24b, the action data 24c, and the virtual space data 24d may be acquired from other devices such as a server device by wireless communication or wired communication.

  The control program 24 a provides functions related to various controls for operating the display device 1. The functions provided by the control program 24a include a function for superimposing a three-dimensional object on the images acquired by the photographing units 40 and 42 and displaying the same on the display units 32a and 32b, a function for detecting an operation on the three-dimensional object, and a detected operation. A function for changing the three-dimensional object in response to this is included.

  The control program 24 a includes a detection processing unit 25, a display object control unit 26, and an image composition unit 27. The detection processing unit 25 provides a function for detecting an actual object existing in the imaging range of the imaging units 40 and 42. The function provided by the detection processing unit 25 includes a function of measuring the distance to each detected object.

  The display object control unit 26 provides a function for managing what three-dimensional objects are arranged in the virtual space and what state each three-dimensional object is in. The functions provided by the display object control unit 26 include detecting an operation on a three-dimensional object based on the movement of an actual object detected by the function of the detection processing unit 25, and changing the three-dimensional object based on the detected operation. The function to be included is included.

  The image composition unit 27 provides a function for generating an image to be displayed on the display unit 32a and an image to be displayed on the display unit 32b by synthesizing the image in the real space and the image in the virtual space. The function provided by the image compositing unit 27 includes the real object and 3 based on the distance to the real object measured by the function of the detection processing unit 25 and the distance from the viewpoint to the three-dimensional object in the virtual space. A function to determine the context of the dimension object and adjust the overlap is included.

  The object data 24b includes information regarding the shape and properties of the three-dimensional object. The object data 24b is used for displaying a three-dimensional object. The action data 24c includes information regarding how an operation on the displayed three-dimensional object acts on the three-dimensional object. The action data 24c is used to determine how to change the three-dimensional object when an operation on the displayed three-dimensional object is detected. The change here includes movement, rotation, deformation, disappearance, and the like. The virtual space data 24d holds information regarding the state of the three-dimensional object arranged in the virtual space. The state of the three-dimensional object includes, for example, a position, a posture, a deformation state, and the like.

  Next, an example of control based on the function provided by the control program 24a will be described with reference to FIG. The image P1a is an image obtained by the photographing unit 40, that is, an image corresponding to a sight of a real space viewed with the right eye. The table P1 and the user's hand H1 are shown in the image P1a. The display device 1 also acquires an image obtained by photographing the same scene by the photographing unit 42, that is, an image corresponding to a sight when the real space is viewed with the left eye.

  The image P2a is a right-eye image generated based on the virtual space data 24d and the object data 24b. In this example, the virtual space data 24d holds information related to the state of the block-shaped three-dimensional object BL1 existing in the virtual space, and the object data 24b holds information related to the shape and properties of the three-dimensional object BL1. The display device 1 reproduces the virtual space based on these pieces of information, and generates an image P2a obtained by viewing the reproduced virtual space from the viewpoint of the right eye. The position of the right eye (viewpoint) in the virtual space is determined based on a predetermined rule. Similarly, the display device 1 also generates an image obtained by viewing the reproduced virtual space from the viewpoint of the left eye. That is, the display device 1 also generates an image in which the three-dimensional object BL1 is displayed three-dimensionally by combining with the image P2a.

  In step S1 shown in FIG. 7, the display device 1 combines the image P1a and the image P2a to generate an image P3a. The image P3a is an image displayed on the display unit 32a as a right-eye image. At this time, the display device 1 determines the front-rear relationship between the real object in the shooting range of the shooting unit 40 and the three-dimensional object existing in the virtual space, with the position of the user's right eye as a reference. When the real object and the three-dimensional object overlap, the overlap is adjusted so that the object closer to the user's right eye can be seen in the front.

  Such overlapping adjustment is performed for each range of a predetermined size (for example, for each pixel) in the region on the image where the real object and the three-dimensional object overlap. For this reason, the distance from the viewpoint in the real space to the real object is measured for each range of a predetermined size on the image. Further, the distance from the viewpoint in the virtual space to the three-dimensional object is calculated for each range of a predetermined size on the image in consideration of the position, shape, posture, etc. of the three-dimensional object.

  In the scene of step S1 shown in FIG. 7, the three-dimensional object BL1 is arranged in a position corresponding to the position immediately above the position where the table T1 exists in the real space in the virtual space. Furthermore, in the scene of step S1 shown in FIG. 7, the user's hand H1 and the three-dimensional object BL1 exist in substantially the same direction and at substantially the same distance with respect to the position of the user's right eye. For this reason, by adjusting the overlap for each range of a predetermined size, in the combined image P3a, in the portion corresponding to the thumb of the hand H1 in the region where the hand H1 and the three-dimensional object BL1 overlap. The hand H1 appears on the front surface, and the three-dimensional object BL1 appears on the front surface in other portions. Furthermore, in the area where the table T1 and the three-dimensional object BL1 overlap, the three-dimensional object BL1 appears on the front surface.

  By adjusting the overlap, in step S1 shown in FIG. 7, an image P3a is obtained as if the three-dimensional object BL1 is placed on the table T1 and the user grasps the three-dimensional object BL1 with the hand H1. It has been. The display device 1 generates an image displayed on the display unit 32b as an image for the left eye by synthesizing the image captured by the imaging unit 42 and the image obtained by viewing the virtual space from the viewpoint of the left eye by the same processing. When generating an image for the left eye, the overlap between the real object and the three-dimensional object is adjusted based on the position of the user's left eye.

  The display device 1 displays the composite image thus generated on the display units 32a and 32b. As a result, the user can see a scene as if the three-dimensional object BL1 is placed on the table T1 and the user grasps the three-dimensional object BL1 with his / her hand H1.

  In the scene of step S2 shown in FIG. 7, the user moves the hand H1 in the direction of the arrow A1. In this case, the image obtained by the imaging unit 40 changes to an image P1b in which the position of the hand H1 has moved to the right. Further, the display device 1 determines that the movement of the hand H1 is an operation of moving the hand H1 to the right while holding the three-dimensional object, and moves the position of the three-dimensional object in the virtual space to the right according to the operation. The movement of the three-dimensional object in the virtual space is reflected in the virtual space data 24d. As a result, the image for the right eye generated based on the virtual space data 24d and the object data 24b changes to an image P2b in which the position of the three-dimensional object BL1 has moved to the right. Details of operation detection by the display device 1 will be described later.

  The display device 1 combines the image P1b and the image P2b to generate an image P3b for the right eye. The image P3b is an image as if the user is holding the three-dimensional object BL1 with the hand H1 at a position on the right side of the table T1 as compared with the image P3a. The display device 1 similarly generates a composite image for the left eye. Then, the display device 1 displays the composite image thus generated on the display units 32a and 32b. As a result, the user can see a scene as if he / she picked the three-dimensional object BL1 with his / her hand H1 and moved it to the right.

  Such a composite image for display is updated at a frequency (for example, 30 times per second) equal to the frame rate of a general moving image. As a result, the change of the three-dimensional object according to the user's operation is reflected in the image displayed on the display device 1 in almost real time, and the user can feel the three-dimensional without any sense of incongruity as if it actually exists. You can manipulate objects. Furthermore, in the configuration according to the present embodiment, the user's hand that operates the three-dimensional object is not positioned between the user's eyes and the display units 32a and 32b. The operation can be performed without worrying about the display of the object being interrupted.

  Next, the object data 24b and the action data 24c shown in FIG. 6 will be described in more detail with reference to FIGS. FIG. 8 is a diagram illustrating an example of information stored in the object data 24b. 9 to 14 are diagrams illustrating examples of information stored in the action data 24c.

  As shown in FIG. 8, the object data 24b stores information including type, shape information, color, transparency, etc. for each three-dimensional object. The type indicates the physical property of the three-dimensional object. The type takes values such as “rigid body” and “elastic body”, for example. The shape information is information indicating the shape of the three-dimensional object. The shape information is, for example, a set of vertex coordinates of the surfaces constituting the three-dimensional object. The color is the color of the surface of the three-dimensional object. Transparency is the degree to which a three-dimensional object transmits light. The object data 24b can hold information regarding a plurality of three-dimensional objects.

  In the example shown in FIGS. 9 to 14, information regarding a change when a pressing operation is detected is stored in the action data 24 c for each type of three-dimensional object. As shown in FIG. 9, when the type of the three-dimensional object is “rigid”, the change when the pressing operation is detected according to the presence / absence of the fulcrum, the presence / absence of the obstacle in the pressed direction, and the pressing speed changes. Different. The obstacle here may be another three-dimensional object or an actual object. Further, whether the pressing speed is fast or slow is determined based on a threshold value.

  If the three-dimensional object has no fulcrum and there is no obstacle in the pushed direction, the three-dimensional object is displayed so as to move according to the amount pushed in the pushed direction. The three-dimensional object displayed in this way is, for example, a building block, a pen, or a book. As to how to move, whether to slide or rotate may be determined based on the shape of the three-dimensional object. Further, whether to move with the pushed object or move away from the object so as to be repelled by the pushed object may be determined based on the pushing speed, and calculation of frictional resistance between the three-dimensional object and the bottom surface. You may determine based on a value or a setting value.

  If the 3D object has no fulcrum and there is an obstacle fixed in the pushed direction, the 3D object moves according to the amount pushed in the pushed direction and moves when it touches the obstacle. Appears to stop. The three-dimensional object displayed in this way is, for example, a building block, a pen, or a book. When the pushed speed is high, the three-dimensional object may destroy the obstacle and continue to move. Also, when the 3D object comes in contact with an obstacle while being moved away from the object so as to be repelled by the pressed object, the 3D object may be moved in the opposite direction so as to bounce off.

  If the 3D object has no fulcrum and there are other rigid bodies that are not fixed in the pushed direction, and the pushed speed is slow, the 3D object moves according to the amount pushed in the pushed direction, After contacting another rigid body, the other rigid bodies are displayed to move together. If the 3D object has no fulcrum and there is another rigid body that is not fixed in the pressed direction, and the pressed speed is fast, the 3D object moves according to the amount pressed in the pressed direction. Is displayed. After the three-dimensional object comes into contact with another rigid body, the other rigid body is displayed so as to be repelled and moved. After coming into contact with another rigid body, the three-dimensional object may stop on the spot or continue to move at a reduced speed. The combination of the three-dimensional object and the other rigid body displayed in this way is, for example, a combination of a bowling ball and a pin, or a combination of marbles.

  If there is another rigid body that does not have a fulcrum in the 3D object and is not fixed in the pressed direction, but the other rigid body can pass through, the 3D object will depend on the amount pressed in the pressed direction. After moving and contacting another rigid body, it is displayed to continue moving as it passes through the other rigid body. In reality, a rigid body does not slip through a rigid body, but by enabling such a slip-through, a novel experience can be provided to the user. The combination of the three-dimensional object and the other rigid body displayed in this way is, for example, a combination of a bowling ball and a pin, or a combination of marbles. A threshold may be provided for the pushed speed, and when the pushed speed is equal to or less than the threshold, the three-dimensional object may not pass through another rigid body.

  When the three-dimensional object has a fulcrum, the three-dimensional object is displayed so as to rotate around the fulcrum according to the pressed direction and amount. The rotation referred to here may be a rotation that rotates around 360 degrees, or a rotation that reciprocates within a predetermined rotation range. The three-dimensional object displayed in this way is, for example, a pendulum, a boxing sandback, or a windmill.

  Also, as shown in FIG. 10, when the type of the three-dimensional object is “elastic body”, the change when the pressing operation is detected differs depending on the material, the presence / absence of the change amount limitation, and the pressing speed. The material here is an assumed material of the three-dimensional object and is defined in the object data 24b.

  If the material of the 3D object is rubber, there is no limit on the amount of change, and the pressing speed is slow, the 3D object is deformed according to the amount pressed in the pressed direction and released from the pressed state. When it is done, it is displayed so as to return to the original shape. In addition, when the material of the three-dimensional object is rubber, there is no limit on the amount of change and the pressing speed is fast, the three-dimensional object is deformed according to the amount pressed in the pressing direction, and then the flip It is displayed to move in the pressed direction while returning to the original shape. The three-dimensional object displayed in this way is, for example, a rubber ball or an eraser.

  If the material of the 3D object is rubber, and the amount of change is limited, the 3D object will be deformed to the deformable range according to the amount pressed in the pressed direction, and the pressing operation will be detected thereafter. Then, it is displayed to move in the pushed direction while returning to the original shape. The three-dimensional object displayed in this way is, for example, a rubber ball or an eraser.

  If the material of the 3D object is metal, the 3D object will be deformed to the deformable range according to the amount pressed in the pressed direction, and will return to its original shape when released from the pressed state. To repeat (vibrate). When pressed in a direction other than the deformable direction, the three-dimensional object moves in the same manner as a rigid body. The three-dimensional object displayed in this way is, for example, a leaf spring or a string spring.

  As shown in FIG. 11, when the type of the three-dimensional object is “plastic”, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape changes. The three-dimensional object displayed in this way is, for example, clay.

  Also, as shown in FIG. 12, when the type of the three-dimensional object is “liquid”, the change when the pressing operation is detected differs depending on the pressing speed. When the pushing speed is slow, the object to be pushed is displayed so as to be in contact with the three-dimensional object, ie, the liquid. When the pressing speed is medium, the object to be pressed is displayed on the liquid so that the ripple spreads on the liquid. When the pushed speed is high, the object to be pushed is touched by the liquid, and it is displayed that the liquid splashes. The three-dimensional object displayed in this way is, for example, water in a cup.

  Also, as shown in FIG. 13, when the type of the three-dimensional object is “gas”, the change when the pressing operation is detected differs depending on the pressing speed. When the pushed speed is slow, a three-dimensional object, i.e., a gas is interrupted by the pushed object and is displayed to drift around it. When the pressing speed is medium, it is displayed that the gas is scattered by the pressing object. When the pushing speed is high, the vortex is generated in the gas behind the pushing object in the moving direction. The three-dimensional object displayed in this way is, for example, smoke.

  As shown in FIG. 14, when the type of the three-dimensional object is “aggregate”, the change when the pressing operation is detected differs depending on the coupling state of the elements of the aggregate. When there is no connection of the elements of the aggregate, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape of the aggregate changes. The three-dimensional object displayed in this way is, for example, sand or sugar.

  When there is a combination of the elements of the aggregate, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape of the aggregate changes. Further, an element other than the pressed part is displayed so as to be moved by being pulled by the element at the pressed part. The three-dimensional object displayed in this way is, for example, a chain.

  When there is no combination of the elements of the aggregate, but an attractive force or a repulsive force is applied to the pushing object, the three-dimensional object is displayed so as to move even if it does not contact the pushing object. When an attractive force acts between the pressed object and the pressed object, the three-dimensional object is attracted to the pressed object when it is within a predetermined distance from the pressed object without contacting the pressed object. In addition, when a repulsive force acts between the pressed object and the pressed object, the three-dimensional object moves away from the pressed object when it is within a predetermined distance from the pressed object without contacting the pressed object. The combination of the displayed three-dimensional object and the pressed object is, for example, a combination of iron powder and a magnet.

  As described above, by changing the three-dimensional object based on the information stored in the object data 24b and the information stored in the action data 24c, the three-dimensional object can be variously changed according to the pressing operation. it can. The information stored in the object data 24b and the action data 24c is not limited to the above example, and may be changed as appropriate according to the application. For example, the method of changing the three-dimensional object may be set according to the type and size of the pressed object and the size of the contact area between the pressed object and the three-dimensional object.

  Next, detection of an operation of pushing a three-dimensional object and change of the three-dimensional object according to the detected operation will be described with reference to FIGS. 15 and 16. In the following description, a space viewed by a user wearing the display device 1 may be referred to as a display space. The display device 1 can display an actual object and a three-dimensional object three-dimensionally (three-dimensionally) in the display space by providing images corresponding to the eyes of the user's right eye and left eye. . The display device 1 associates the virtual space reproduced based on the virtual space data 24d with the actual space photographed by the photographing units 40 and 42 based on a predetermined rule, and displays a space in which these spaces overlap. Display as space.

  15 and 16 are diagrams for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. In step S11 illustrated in FIG. 15, the display device 1 displays the three-dimensional object OB1 in a three-dimensional manner in the display space 50. The three-dimensional object OB1 is an object that imitates a ball, for example. In step S11, a bottom surface B1 that supports the three-dimensional object OB1 is displayed.

  In step S12, the user places the finger F1 in a position in contact with the three-dimensional object OB1 and keeps it still. The display device 1 determines that the three-dimensional object OB1 is selected as the operation target when a real object is detected in the display space and the state where the object is in contact with the three-dimensional object OB1 continues for a predetermined time or longer. . Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected as an operation target, for example, by changing the display mode of the three-dimensional object OB1.

  Whether the object is in contact with the three-dimensional object OB1 is determined based on the position of the object in the real space, the shape and orientation of the three-dimensional object OB1, the position in the virtual space, and the like. The comparison between the position in the real space and the position in the virtual space may be performed by converting the position in one space into the position in the other space based on the above-mentioned predetermined rule. You may perform by converting a position into the position of the space for a comparison. When a finger is detected as an actual object, the position of the finger toe may be processed as the position of the object. Since a person often uses a finger toe when operating something, by processing the position of the finger toe as the position of an object, a more natural operation feeling can be provided to the user.

  The notification of selection as the operation target may be, for example, changing the overall color of the three-dimensional object OB1, or changing the color near the position in contact with the object on the surface of the three-dimensional object OB1. Realized by changing. The display device 1 may perform notification by sound or vibration instead of such visual notification or in addition to such visual notification.

  As described above, the display device 1 determines that the three-dimensional object OB1 is selected as the operation target when a state where a real object such as a finger is in contact with the three-dimensional object OB1 is continuously detected for a predetermined time or longer. . By adding the condition that the contact state is continuously detected for a predetermined time or longer, for example, in the process of moving a finger to operate another three-dimensional object, an unintended three-dimensional object is selected as an operation target. The possibility of being made can be lowered.

  After the three-dimensional object OB1 is selected as the operation target, it is assumed that the user has intruded the finger F1 inside the three-dimensional object OB1 so as to press the three-dimensional object OB1, as shown in step S13. The display device 1 changes the three-dimensional object according to the operation when an operation for intruding the object into the three-dimensional object selected as the operation target is detected. How to change the three-dimensional object is determined by the type of the three-dimensional object defined in the object data 24b and the change rule defined in association with the type in the action data 24c.

  For example, in the object data 24b, the three-dimensional object OB1 is defined as an elastic body, and in the action data 24c, it is defined that the elastic body is deformed in the pressed direction according to the amount of pressing when pressed. Shall be. In this case, as shown in step S14, the display device 1 changes the three-dimensional object OB1 so that the portion into which the finger F1 has entered is pressed and recessed.

  Further, the object data 24b defines the three-dimensional object OB1 as a rigid body, and the action data 24c defines that the rigid body moves in the pressed direction according to the amount of pressing when pressed. And In this case, as shown in step S15 in FIG. 16, the display device 1 moves the three-dimensional object OB1 in the traveling direction of the finger F1 as if pressed by the finger F1. In step S15 of FIG. 16, since the three-dimensional object OB1 is supported by the bottom surface B1, it moves according to the horizontal component of the bottom surface B1 of the force applied by the rigid body.

  As described above, when an operation of pushing the three-dimensional object is detected, the three-dimensional object is changed variously according to the operation by changing the three-dimensional object OB1 based on the object data 24b and the action data 24c. Can do. The operation of pressing is an operation that is used in various scenes in the real world. By detecting the operation of pressing the three-dimensional object OB1 and executing the corresponding process, the operation is intuitive and highly convenient. Can be realized.

  An object used for operating a three-dimensional object is not limited to a finger, and may be a hand, a foot, a stick, an instrument, or the like. The mode of changing the three-dimensional object in accordance with the pressing operation may be in accordance with actual physical laws or may not be in reality.

  The display device 1 may limit a space for detecting an operation on the three-dimensional object to the operable range 51. The operable range 51 is, for example, a range that can be reached by a user wearing the display device 1. In this way, by limiting the space for detecting the operation on the three-dimensional object, it is possible to reduce the load of the arithmetic processing executed by the display device 1 to detect the operation.

  Next, a first example of a processing procedure executed by the display device 1 regarding an operation of pressing a three-dimensional object will be described with reference to FIGS. 17 and 18. FIG. 17 is a flowchart illustrating a processing procedure of contact detection processing of a three-dimensional object. The processing procedure illustrated in FIG. 17 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 17, first, the control unit 22 synthesizes and displays a virtual space image including a three-dimensional object and a real space image in step SA01.

  Subsequently, in step SA02, the control unit 22 determines whether a predetermined object is detected by the detection unit 44, that is, the imaging units 40 and 42. The predetermined object is, for example, a user's finger. When the predetermined object is not detected (step SA02, No), the control unit 22 determines whether the operation end is detected as step SA08.

  The operation end is detected, for example, when a predetermined operation is performed on the operation unit 13. When the operation end is detected (step SA08, Yes), the control unit 22 ends the contact detection process. When the operation end is not detected (No at Step SA08), the control unit 22 re-executes Step SA02 and the subsequent steps.

  When a predetermined object is detected (step SA02, Yes), the control unit 22 determines the type of the predetermined object as step SA03. The type of the predetermined object is determined based on, for example, the size, shape, color, and the like of the object in the images photographed by the photographing units 40 and 42. Subsequently, in step SA04, the control unit 22 searches for a three-dimensional object that is in contact with a predetermined object. If there is no three-dimensional object in contact with the predetermined object (step SA05, No), the control unit 22 proceeds to step SA08.

  When a three-dimensional object in contact with the predetermined object is found (step SA05, Yes), the control unit 22 determines, as step SA06, the three-dimensional object in contact with the predetermined object based on the object data 24b. Determine the type. And the control part 22 performs the operation detection process mentioned later as step SA07. Thereafter, the control unit 22 proceeds to Step SA08.

  FIG. 18 is a flowchart illustrating a procedure of the operation detection process. The processing procedure shown in FIG. 18 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 18, the control unit 22 first acquires the contact time between the predetermined object and the three-dimensional object as step SB01. Then, the control unit 22 determines whether the predetermined object has moved into the three-dimensional object as Step SB02. When the predetermined object has not moved inside the three-dimensional object (step SB02, No), the control unit 22 re-executes step SB01 and the subsequent steps.

  When the predetermined object has moved to the inside of the three-dimensional object (step SB02, Yes), the control unit 22 determines whether the contact time is a predetermined time or more as step SB03. When the contact time is shorter than the predetermined time (step SB03, No), it is determined that the three-dimensional object is not an operation target, and thus the control unit 22 ends the operation detection process.

  When the contact time is equal to or longer than the predetermined time (step SB03, Yes), the control unit 22 calculates the speed of the predetermined object as step SB04. In step SB05, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, in Step SB06, the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object. When the predetermined object has not moved to the outside of the three-dimensional object, that is, when the pressing operation is continued (No in Step SB06), the control unit 22 re-executes Step SB04 and the subsequent steps.

  When the predetermined object moves to the outside of the three-dimensional object, that is, when the three-dimensional object is released (step SB06, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step SB07. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step SB07, Yes), the control unit 22 changes the three-dimensional object as step SB08, and then re-executes step SB07 and the subsequent steps. When the change of the three-dimensional object is not continued (step SB07, No), the control unit 22 ends the operation detection process.

  As described above, in the first example, the three-dimensional object is variously changed according to the pressing operation, thereby providing a convenient operation method for the user.

  A second example of a processing procedure related to an operation of pressing a three-dimensional object will be described. The contact detection process in the second example is the same as the contact detection process in the first example. Therefore, in the second example, the description overlapping the first example is omitted, and the operation detection process will be mainly described.

  First, detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation will be described with reference to FIG. FIG. 19 is a diagram for describing detection of an operation of pressing a three-dimensional object and changes in the three-dimensional object according to the detected operation. In step S21 shown in FIG. 19, the user contacts the finger F1 with the three-dimensional object OB1, and in step S22, the user causes the finger F1 to enter the inside of the three-dimensional object OB1.

  When a real object is detected in the display space and the state in which the object moves to the inside of the three-dimensional object OB1 after the object contacts the three-dimensional object OB1 continues for a predetermined time or longer, the display device 1 It is determined that OB1 has been selected as the operation target. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected as an operation target, for example, by changing the display mode of the three-dimensional object OB1. Further, as shown in step S23, the display device 1 responds to the operation with the finger F1 after the contact detection as if the three-dimensional object OB1 was already selected as an operation target to be pressed in the step S21. Change.

  As described above, after detecting the contact between the object and the three-dimensional object, the user can quickly start the operation of pressing the three-dimensional object by making it possible to detect the pressing operation even if the object does not stay on the spot. can do. In addition, by adding the condition that the state in which the object moves to the inside of the three-dimensional object OB1 after contact continues for a predetermined time or longer, for example, in the process of moving a finger to operate another three-dimensional object, The possibility that an unintended three-dimensional object is selected as an operation target can be reduced.

  Next, the procedure of the operation detection process in the second example will be described with reference to FIG. FIG. 20 is a flowchart illustrating the processing procedure of the operation detection process. The processing procedure shown in FIG. 20 is realized by the control unit 22 executing the control program 24a. The processing procedure of the contact detection processing is the same as the procedure shown in FIG.

  As shown in FIG. 20, the control unit 22 first determines whether a predetermined object has moved inside the three-dimensional object as step SC01. If the predetermined object has not moved to the inside of the three-dimensional object (No at Step SC01), since the three-dimensional object is determined not to be an operation target, the control unit 22 ends the operation detection process.

  When the predetermined object has moved to the inside of the three-dimensional object (step SC01, Yes), the control unit 22 determines whether the elapsed time from the contact detection is a predetermined time or more as step SC02. When the elapsed time is shorter than the predetermined time (No at Step SC02), the control unit 22 re-executes Step SC01 and the subsequent steps.

  When the contact time is equal to or longer than the predetermined time (step SC02, Yes), the control unit 22 calculates the speed of the predetermined object as step SC03. In step SC04, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object and the type of the three-dimensional object. A specific method of change is determined according to the action data 24c.

  Subsequently, in step SC05, the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object. When the predetermined object has not moved outside the three-dimensional object, that is, when the pressing operation is continued (No at Step SC05), the control unit 22 re-executes Step SC03 and the subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (step SC05, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step SC06. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step SC06, Yes), the control unit 22 changes the three-dimensional object as step SC07, and then re-executes step SC06 and subsequent steps. When the change of the three-dimensional object does not continue (No at Step SC06), the control unit 22 ends the operation detection process.

  As described above, in the second example, since the pressing operation is recognized even when the object such as the finger is in contact with the three-dimensional object does not continue for a predetermined time or longer, the user can recognize the three-dimensional object. The operation of pushing the object can be started quickly.

  A third example of a processing procedure related to an operation of pressing a three-dimensional object will be described. The contact detection process in the third example is the same as the contact detection process in the first example. Therefore, in the third example, the description overlapping with the first example is omitted, and the operation detection process is mainly described.

  First, detection of an operation of pushing a three-dimensional object and change of the three-dimensional object according to the detected operation will be described with reference to FIGS. 21 and 22. 21 and 22 are diagrams for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. In step S31 shown in FIG. 21, the three-dimensional object OB1 is displayed three-dimensionally in the display space. In addition, the user is bringing the finger F1 into contact with the three-dimensional object OB1.

  Here, it is assumed that the user has caused the finger F1 to enter the inside of the three-dimensional object OB1. When the display device 1 detects that the object in contact with the three-dimensional object OB1 has moved to the inside of the three-dimensional object OB1, as shown in step S32, the display device 1 responds to the operation with the finger F1 from that point in time. Change. In the example shown in FIG. 21, in step S32, the three-dimensional object OB1 starts moving in accordance with the movement of the finger F1.

  Then, as shown in step S33, the display device 1 determines the three-dimensional object OB1 as an operation target when the finger F1 continues to move inward of the three-dimensional object OB1 for a predetermined time or more. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been determined as an operation target, for example, by changing the display mode of the three-dimensional object OB1. Thereafter, while the movement of the finger F1 to the inside of the three-dimensional object OB1 is detected, the display device 1 continues to change the three-dimensional object OB1.

  As shown in step S34 of FIG. 22, when the movement of the finger F1 to the inside of the three-dimensional object OB1 is not detected before the predetermined time has elapsed, the display device 1 is opposite to the change made so far. A change is applied to the three-dimensional object OB1. As a result, the three-dimensional object OB1 is displayed in the same state at the same position as in the step S31. The speed at which the reverse change is applied to the three-dimensional object OB1 may be faster than the speed at which the change has been applied to the three-dimensional object OB1 so far. That is, the three-dimensional object OB1 may be reversely changed as if it is reversely played back at high speed.

  In this way, by starting to apply changes to the 3D object from the stage where the intrusion of the object inside the 3D object is detected, the user can confirm that the 3D object is being selected before the selection is confirmed. Can be recognized from. As a result, the user can know at an early stage whether or not the intended three-dimensional object has been selected. When an unintended three-dimensional object is selected, the user can return the original three-dimensional object to an original state by canceling the operation before a predetermined time elapses.

  Until the movement of the finger F1 to the inside of the three-dimensional object OB1 continues for a predetermined time or longer, the mode is different from the state in which the selected three-dimensional object is changed as the operation target at both the normal time (for example, half-time). (Transparent) may be displayed. By changing the display mode in this way, the user can easily determine the state of the three-dimensional object.

  Next, the procedure of the operation detection process in the third example will be described with reference to FIG. FIG. 23 is a flowchart illustrating the processing procedure of the operation detection process. The processing procedure shown in FIG. 23 is realized by the control unit 22 executing the control program 24a. The processing procedure of the contact detection processing is the same as the procedure shown in FIG.

  As shown in FIG. 23, the control unit 22 first determines whether a predetermined object has moved inside the three-dimensional object as step SD01. If the predetermined object has not moved to the inside of the three-dimensional object (step SD01, No), since the three-dimensional object is determined not to be an operation target, the control unit 22 ends the operation detection process.

  When the predetermined object is moving inside the three-dimensional object (step SD01, Yes), the control unit 22 calculates the speed of the predetermined object as step SD02. In step SD03, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, as step SD04, the control unit 22 determines whether or not the elapsed time from the contact detection is a predetermined time or more. When the elapsed time is shorter than the predetermined time, that is, when the three-dimensional object is not fixed as the target of the pressing operation (No at Step SD04), the control unit 22 determines the three-dimensional object of the predetermined object as Step SD05. It is determined whether the movement in the internal direction continues.

  When the movement of the three-dimensional object in the internal direction continues (step SD05, Yes), the control unit 22 re-executes step SD02 and subsequent steps. When the movement of the three-dimensional object in the internal direction is not continued (No at Step SD05), the control unit 22 reversely changes the three-dimensional object and returns it to the original state as Step SD06. Then, the control unit 22 ends the operation detection process.

  When the elapsed time from the contact detection is equal to or longer than the predetermined time (step SD04, Yes), the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object as step SD07. When the predetermined object has not moved outside the three-dimensional object, that is, when the pressing operation is continued (No at Step SD07), the control unit 22 re-executes Step SD02 and the subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (step SD07, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step SD08. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step SD08, Yes), the control unit 22 changes the three-dimensional object as step SD09, and then re-executes step SD08 and subsequent steps. When the change of the three-dimensional object does not continue (No at step SD08), the control unit 22 ends the operation detection process.

  As described above, in the third example, since the three-dimensional object is changed in accordance with the operation from the time when the pressing operation is detected, the three-dimensional object that is the target of the pressing operation by the user. Easy to recognize objects.

  Although the operation of pushing the 3D object has been described as the operation related to the 3D object, the operation detected by the display device 1 concerning the 3D object is not limited to the pushing operation. The display device 1 can also detect an operation performed by a user picking a three-dimensional object. Hereinafter, an operation performed by picking a three-dimensional object will be described.

  With reference to FIG. 24, detection of an operation performed by picking a three-dimensional object will be described. FIG. 24 is a diagram for describing detection of an operation performed by picking up a three-dimensional object. In step S41 shown in FIG. 24, the three-dimensional object OB1 is displayed three-dimensionally in the display space 50.

  Here, it is assumed that the user wants to perform some operation by picking up the three-dimensional object OB1. In order to pick up the three-dimensional object OB1 and perform some operation, it is first necessary to select the three-dimensional object OB1 as an operation target. In order to select the three-dimensional object OB1, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2, as shown in step S42, and changes its state. Maintain for a predetermined time or longer.

  The display device 1 selects the three-dimensional object OB1 when two real objects are detected in the display space and the state where the three-dimensional object OB1 is located between the two objects continues for a predetermined time or longer. It is determined that the three-dimensional object OB1 is selected. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected, for example, by changing the display mode of the three-dimensional object OB1.

  Whether the three-dimensional object OB1 is located between the two objects is determined based on the positions of the two objects in the real space, the shape and orientation of the three-dimensional object OB1, the position in the virtual space, and the like. The comparison between the position in the real space and the position in the virtual space may be performed by converting the position in one space into the position in the other space based on the above-mentioned predetermined rule. You may perform by converting a position into the position of the space for a comparison. When a finger is detected as an actual object, the position of the finger toe may be processed as the position of the object.

  As described above, the display device 1 determines that the three-dimensional object OB1 is selected when the state in which the three-dimensional object OB1 is located between actual objects such as fingers is continuously detected for a predetermined time or more. The operation of placing a finger so that the three-dimensional object OB1 is sandwiched between the fingers is similar to the operation of a person picking something to select something in the real space. Therefore, such an operation is intuitive and easy to understand as an operation for selecting a three-dimensional object. Further, by adding a condition that the state is continuously detected for a predetermined time or longer, for example, an unintended three-dimensional object is selected in the process of moving a finger to select another three-dimensional object. The possibility can be reduced.

  After determining that the three-dimensional object OB1 is in the selected state, the display device 1 adds changes such as movement, deformation, and disappearance to the three-dimensional object OB1 according to the movement of the finger F1 and the finger F2.

  Next, a first example of a processing procedure executed by the display device 1 regarding an operation performed by picking a three-dimensional object will be described with reference to FIGS. FIG. 25 is a flowchart illustrating a processing procedure of a three-dimensional object selection detection process. The processing procedure shown in FIG. 25 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 25, the control unit 22 first synthesizes and displays a virtual space image including a three-dimensional object and a real space image in step SE01.

  Subsequently, in step SE02, the control unit 22 determines whether the detection unit 44, that is, the imaging units 40 and 42, has detected the first object and the second object. The first object and the second object are real objects, for example, user's fingers. When the first object and the second object are not detected (step SE02, No), the control unit 22 determines whether the operation end is detected as step SE10.

  The operation end is detected, for example, when a predetermined operation is performed on the operation unit 13. When the operation end is detected (step SE10, Yes), the control unit 22 ends the selection detection process. When the operation end is not detected (No at Step SE10), the control unit 22 re-executes Step SE02 and the subsequent steps.

  When the first object and the second object are detected (step SE02, Yes), the control unit 22 determines, as step SE03, the first object and the second object from the displayed three-dimensional objects. Search for the three-dimensional object displayed between. When there is no corresponding three-dimensional object (step SE04, No), the control unit 22 proceeds to step SE10.

  When a three-dimensional object displayed between the first object and the second object is found (step SE04, Yes), the control unit 22 determines that the first object, the second object, and the second object are step SE05. The time during which the three-dimensional object is located is acquired. When the acquired time is less than the predetermined time (step SE06, No), the control unit 22 proceeds to step SE10.

  When the acquired time is equal to or longer than the predetermined time (step SE06, Yes), the control unit 22 calculates the distance between the first object and the second object as step SE07. Moreover, the control part 22 makes the three-dimensional object currently displayed between the 1st object and the 2nd object a selection state as step SE08. And control part 22 performs operation detection processing mentioned below as Step SE09, and changes the three-dimensional object in a selection state according to detected operation in it. After the operation detection process ends, the control unit 22 proceeds to step SE10.

  FIG. 26 is a flowchart showing the procedure of the operation detection process. The processing procedure shown in FIG. 26 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 26, the control unit 22 first calculates the distance between the first object and the second object as step SF01. And control part 22 judges whether the distance of the 1st object after the start time of operation detection processing and the 2nd object is almost constant as Step SF02. The distance is substantially constant, for example, when the amount of change in the distance between the first object and the second object at the current time is compared with the distance at the start of the operation detection process in a predetermined range (first object and This means that the second object is within ± 10% of the maximum change in distance when moving at a normal speed. Also, when the distance between the first object and the second object continues to shrink after the start of the operation detection process (when the first object and the second object are moving in the direction of crushing the three-dimensional object) In addition, it may be determined that the distance is substantially constant. Further, when the distance between the two changes only within a range such as camera shake, the distance may be determined to be substantially constant.

  When the distance between the first object and the second object is substantially constant (step SF02, Yes), the control unit 22 calculates the moving speeds of the first object and the second object as step SF03. Subsequently, in step SF04, the control unit 22 determines whether the calculated moving speed is equal to or less than a threshold value. The threshold value used here is, for example, the moving speed of the fingertip when a person throws a thing. Further, the moving speed compared with the threshold value may be an average of the moving speed of the first object and the moving speed of the second object, whichever is faster, or whichever is slower It may be.

  When the moving speed is less than or equal to the threshold (step SF04, Yes), the control unit 22 changes the three-dimensional object according to the detected movements of the first object and the second object as step SF05. For example, when the rightward movement of the first object and the second object is detected, the control unit 22 moves the three-dimensional object to the right according to the movement of the first object and the second object. Let When the counterclockwise rotation of the first object and the second object is detected, the control unit 22 rotates the three-dimensional object counterclockwise in accordance with the rotation of the first object and the second object. When movement and rotation are detected simultaneously, movement and rotation are performed simultaneously. When there is an obstacle to the movement and rotation of the three-dimensional object, the movement and rotation of the three-dimensional object may be stopped when the three-dimensional object contacts the obstacle. The obstacle may be a real object or another three-dimensional object. Thereafter, the control unit 22 re-executes Step SF01 and subsequent steps.

  When the moving speed is faster than the threshold (No at Step SF04), the control unit 22 deletes the three-dimensional object as Step SF06. When deleting the three-dimensional object, an animation may be displayed so that the three-dimensional object flies in the moving direction of the first object and the second object. Then, the control unit 22 ends the operation detection process. As described above, when the first object and the second object move at high speed so as to throw the 3D object, the 3D object is erased by an intuitive operation by erasing the 3D object. be able to. Instead of the operation of moving the first object and the second object at a high speed, for example, erasure of the three-dimensional object may be assigned to an operation of crushing the three-dimensional object. Further, instead of deleting the three-dimensional object, the three-dimensional object may be returned to the original placement location. The display device 1 may not perform the processes of steps SF03, SF04, and SF06. That is, the display device 1 may execute step SF05 when the distance between the first object and the second object is determined to be substantially constant in step SF02, regardless of the moving speed of the two objects.

  When the distance between the first object and the second object is not substantially constant (step SF02, No), the control unit 22 determines whether the distance is a three-dimensional object, that is, the start point of the operation detection process, as step SF07. It is judged whether it is expanding. When the distance is increased (step SF07, Yes), the control unit 22 cancels the selected state of the three-dimensional object as step SF08. The operation of increasing the distance between the first object and the second object is similar to the operation of releasing the actual object being picked. Therefore, this operation is intuitive and easy to understand as an operation for canceling the selection of the three-dimensional object.

  Subsequently, in step SF09, the control unit 22 moves the selected three-dimensional object according to gravity or the like. Then, the control unit 22 ends the operation detection process. The movement here is displayed, for example, so that the three-dimensional object falls according to gravity and stops on the floor or table. Before stopping the movement of the three-dimensional object, the three-dimensional object may be bounced according to the elasticity of the three-dimensional object and the hardness of the floor or table. The magnitude of impact when the three-dimensional object collides with the floor or table may be calculated, and if the impact is greater than a predetermined value, the three-dimensional object may be displayed as damaged. Further, the three-dimensional object may be moved more slowly than when actual gravity works.

  When the distance between the first object and the second object is shorter than when the three-dimensional object is selected (step SF07, No), the control unit 22 deforms the three-dimensional object according to the distance as step SF10. . Then, the control unit 22 re-executes Step SF01 and subsequent steps. For example, the degree of deformation of the three-dimensional object may be changed according to the elasticity set as an attribute of the three-dimensional object. The control unit 22 increases the degree of deformation of an object having a low hardness as an attribute, such as a three-dimensional object simulating a rubber ball, as the distance between the first object and the second object decreases. It's okay. In addition, the control unit 22 reduces the degree of deformation of an object having a high hardness as an attribute, such as a three-dimensional object imitating a building block, even if the distance between the first object and the second object is reduced. You can keep it.

  When the distance between the first object and the second object is shorter than when the three-dimensional object is selected, the display device 1 may reduce the three-dimensional object instead of deforming it. When the distance between the first object and the second object becomes a predetermined value or less, the display device 1 may display the three-dimensional object as if it was damaged.

  As described above, in the first example, the three-dimensional object is selected when the state where the three-dimensional object is positioned between the objects such as the fingers continues for a predetermined time or longer. Selection can be realized by an intuitive and easy-to-understand operation.

  As illustrated in FIG. 27, the display device 1 selects a three-dimensional object that a state in which at least one of the first object and the second object is in contact with the three-dimensional object continues for a predetermined time or longer. It is good also as conditions. By making contact with a three-dimensional object as a selection condition, it becomes easier for a user to select a desired three-dimensional object when a plurality of three-dimensional objects are displayed close to each other.

  A second example of a processing procedure related to an operation performed by picking up a three-dimensional object will be described. The operation detection process in the second example is the same as the operation detection process in the first example. Therefore, in the second example, description overlapping with the first example is omitted, and mainly the selection detection process will be described.

  First, detection of an operation performed by picking a three-dimensional object will be described with reference to FIG. FIG. 28 is a diagram for explaining the detection of an operation performed by picking up a three-dimensional object. In step S51 shown in FIG. 28, the three-dimensional object OB1 is three-dimensionally displayed in the display space. In order to select the three-dimensional object OB1, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2.

  When two real objects are detected in the display space and the three-dimensional object OB1 is located between the two objects, the display device 1 monitors a change in the distance between the two objects. If the distance is substantially constant for a predetermined time or more, it is determined that the three-dimensional object OB1 has been selected, and the three-dimensional object OB1 is selected. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected, for example, by changing the display mode of the three-dimensional object OB1.

  While the display device 1 monitors a change in the distance between the two objects, the two objects do not need to stay at a position between which the three-dimensional object OB1 is sandwiched. That is, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2 as shown in step S51, and then maintains the state of the finger F1. The finger F2 may be moved to another position.

  Assume that the user has moved from the state of step S51 while keeping the distance D1 between the finger F1 and the finger F2 substantially constant as shown in step S52. In this case, as shown in step S53, the display device 1 puts the three-dimensional object OB1 into a selected state when the distance D1 between the finger F1 and the finger F2 is kept almost constant for a predetermined time or longer. Then, the display device 1 moves the three-dimensional object OB1 between the finger F1 and the finger F2 as if they were already selected in the step S51. The display device 1 may store the movements of the finger F1 and the finger F2 from step S51 to step S53, and rotate the three-dimensional object OB1 according to the stored movement. Thereafter, the display device 1 adds changes such as movement, deformation, and disappearance to the three-dimensional object OB1 in accordance with the movement of the finger F1 and the finger F2.

  In this way, once the two objects have moved to a position sandwiching the 3D object, the user can select the 3D object by making the 3D object selectable even if the object does not stay on the spot. After that, the operation can be started quickly.

  Next, the procedure of the selection detection process in the second example will be described with reference to FIG. FIG. 29 is a flowchart illustrating a processing procedure of a three-dimensional object selection detection process. The processing procedure shown in FIG. 29 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 29, first, the control unit 22 synthesizes and displays a virtual space image including a three-dimensional object and a real space image in step SG01. Subsequently, in step SG02, the control unit 22 determines whether the first object and the second object are detected by the detection unit 44, that is, the imaging units 40 and 42. When the first object and the second object are not detected (step SG02, No), the control unit 22 cancels the temporarily selected state of the three-dimensional object if there is a temporarily selected three-dimensional object as step SG14. To do. The provisional selection state is a state in which a state in which a three-dimensional object is displayed between two objects is detected and whether the distance between the two objects is maintained substantially constant is monitored.

  And control part 22 judges whether the end of operation was detected as Step SG15. When the operation end is detected (step SG15, Yes), the control unit 22 ends the selection detection process. When the operation end is not detected (No at Step SG15), the control unit 22 re-executes Step SG02 and the subsequent steps.

  When the first object and the second object are detected (step SG02, Yes), the control unit 22 determines whether there is a temporarily selected three-dimensional object as step SG03. When there is no temporarily selected three-dimensional object (step SG03, No), the control unit 22 displays the first three-dimensional object between the first and second objects as step SG04. Find the 3D object that is being used. When there is no corresponding three-dimensional object (step SG05, No), the control unit 22 proceeds to step SG15.

  When a three-dimensional object displayed between the first object and the second object is found (step SG05, Yes), the control unit 22 determines whether the first object, the second object, and the second object are step SG06. The three-dimensional object displayed during is temporarily set to the selected state. Moreover, the control part 22 calculates the distance of a 1st object and a 2nd object as step SG07. And the control part 22 progresses to step SG15.

  When the first object and the second object are detected and there is a provisionally selected three-dimensional object (step SG03, Yes), the control unit 22 sets the first object and the second object as step SG08. The distance is calculated. And the control part 22 determines whether distance is substantially constant as step SG09. When the distance is not substantially constant (No in Step SG09), the control unit 22 cancels the temporary selection state of the three-dimensional object in the temporary selection state as Step SG14. And the control part 22 progresses to step SG15.

  When the distance between the first object and the second object is substantially constant (step SG09, Yes), the control unit 22 determines whether the period during which the distance is kept substantially constant is a predetermined time or more as step SG10. judge. When the period during which the distance is kept substantially constant is less than the predetermined time (step SG10, No), the control unit 22 proceeds to step SG15.

  When the period during which the distance is maintained substantially constant is equal to or longer than the predetermined time (step SG10, Yes), the control unit 22 is displayed between the first object and the second object as step SG11. Put the dimension object in the selected state. Moreover, the control part 22 moves a three-dimensional object between a 1st object and a 2nd object as step SG12. And control part 22 performs operation detection processing shown in Drawing 26 as Step SG13, and changes the three-dimensional object in a selection state according to detected operation in it. After the operation detection process is completed, the control unit 22 proceeds to step SG15.

  As described above, in the second example, after a three-dimensional object is positioned between objects such as fingers, the three-dimensional object is selected when the distance of the object is kept substantially constant for a predetermined time or more. Therefore, the user can quickly start the operation after selecting the three-dimensional object.

  As in Step S61 to Step S63 illustrated in FIG. 30, the display device 1 determines the distance between the first object and the second object after at least one of the first object and the second object contacts the three-dimensional object. May be kept substantially constant for a predetermined time or more as a condition for selecting a three-dimensional object. By making contact with a three-dimensional object as a selection condition, it becomes easier for a user to select a desired three-dimensional object when a plurality of three-dimensional objects are displayed close to each other.

  A third example of a processing procedure related to an operation performed by picking up a three-dimensional object will be described. Regarding the third example, the description overlapping with the first example is omitted, and mainly the selection detection process and the operation detection process will be described.

  First, detection of an operation performed by picking up a three-dimensional object will be described with reference to FIGS. 31 and 32. In step S71 shown in FIG. 31, the three-dimensional object OB1 is displayed three-dimensionally in the display space. In order to select the three-dimensional object OB1, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2.

  When two real objects are detected in the display space and the three-dimensional object OB1 is located between the two objects, the display device 1 monitors a change in the distance between the two objects. If the distance is substantially constant for a predetermined time or more, it is determined that the three-dimensional object OB1 has been selected, and the three-dimensional object OB1 is selected. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected, for example, by changing the display mode of the three-dimensional object OB1.

  While the display device 1 monitors a change in the distance between the two objects, the two objects do not need to stay at a position between which the three-dimensional object OB1 is sandwiched. That is, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2 as shown in step S71, and then maintains the state of the finger F1. The finger F2 may be moved to another position.

  Assume that the user has moved from the state of step S71 while keeping the distance D1 between the finger F1 and the finger F2 substantially constant as shown in step S72. In this case, the display device 1 responds to the movement of the finger F1 and the finger F2 from the stage where it is detected that the three-dimensional object OB1 is displayed between the fingers F1 and F2, that is, from the stage of step S71. Changes such as movement, deformation, and disappearance are added to the three-dimensional object OB1. Then, as shown in step S73, the display device 1 places the three-dimensional object OB1 in a selected state when the distance D1 between the finger F1 and the finger F2 is kept substantially constant for a predetermined time or longer.

  As shown in step S74 of FIG. 32, when the distance D1 between the finger F1 and the finger F2 is expanded before the predetermined time has passed, that is, when the selection is not performed, the display device 1 has been added so far. A change opposite to the change is added to the three-dimensional object OB1. As a result, the three-dimensional object OB1 is displayed in the same state at the same position as in the step S71. The speed at which the reverse change is applied to the three-dimensional object OB1 may be faster than the speed at which the change has been applied to the three-dimensional object OB1 so far. That is, the three-dimensional object OB1 may be reversely changed as if it is reversely played back at high speed.

  Thus, by starting to apply a change to the 3D object from the stage where it is detected that the 3D object is displayed between the two objects, the user can confirm that the 3D object is being selected. It can be recognized before the selection is confirmed. As a result, the user can know at an early stage whether or not the intended three-dimensional object has been selected. The display device 1 has a mode in which the changed three-dimensional object is different from the normal state and the selected state (for example, translucent) until the distance between the two objects is kept substantially constant for a predetermined time or longer. ) May make it easier for the user to determine the state of the three-dimensional object.

  Next, a processing procedure executed by the display device 1 regarding an operation performed by picking up a three-dimensional object will be described with reference to FIGS. 33 and 34. FIG. 33 is a flowchart illustrating a processing procedure of a three-dimensional object selection detection process. The processing procedure shown in FIG. 33 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 33, the control unit 22 first combines and displays a virtual space image including a three-dimensional object and a real space image in step SH01. Subsequently, in step SH02, the control unit 22 determines whether the first object and the second object are detected by the detection unit 44, that is, the imaging units 40 and 42. When the first object and the second object are not detected (step SH02, No), the control unit 22 cancels the temporarily selected state of the three-dimensional object if there is a temporarily selected three-dimensional object as step SH10. To do.

  And the control part 22 determines whether operation completion was detected as step SH11. When the operation end is detected (step SH11, Yes), the control unit 22 ends the selection detection process. When the operation end is not detected (No at Step SH11), the control unit 22 re-executes Step SH02 and the subsequent steps.

  When the first object and the second object are detected (step SH02, Yes), the control unit 22 determines whether or not there is a temporarily selected three-dimensional object as step SH03. When there is no provisionally selected three-dimensional object (step SH03, No), the control unit 22 displays as a step SH04 between the first object and the second object among the displayed three-dimensional objects. Find the 3D object that is being used. When there is no corresponding three-dimensional object (step SH05, No), the control unit 22 proceeds to step SH11.

  When a three-dimensional object displayed between the first object and the second object is found (step SH05, Yes), the control unit 22 determines that the first object, the second object, and the second object are step SH06. The three-dimensional object displayed during is temporarily set to the selected state. Moreover, the control part 22 calculates the distance of a 1st object and a 2nd object as step SH07. Then, the control unit 22 proceeds to step SH11.

  When the first object and the second object are detected and there is a temporarily selected three-dimensional object (step SH03, Yes), the control unit 22 sets the first object and the second object as step SH08. It is determined whether at least one of them is moving. When neither the first object nor the second object has moved (step SH08, No), the control unit 22 proceeds to step SH11.

  When at least one of the first object and the second object is moving (step SH08, Yes), the control unit 22 executes the operation detection process shown in FIG. 34 as step SH09, and selects among them. The three-dimensional object in the state is changed according to the detected operation. After the operation detection process is completed, the control unit 22 proceeds to step SH11.

  FIG. 34 is a flowchart illustrating the procedure of the operation detection process. The processing procedure shown in FIG. 34 is realized by the control unit 22 executing the control program 24a. As shown in FIG. 34, the controller 22 first calculates the distance between the first object and the second object as step SI01. And control part 22 judges whether the distance of the 1st object after the start time of operation detection processing and the 2nd object is almost constant as Step SI02.

  When the distance between the first object and the second object is substantially constant (step SI02, Yes), the control unit 22 determines whether a predetermined time has elapsed since the operation detection process was started as step SI03. If the predetermined time has elapsed (step SI03, Yes), the control unit 22 sets the three-dimensional object in the selected state if there is a temporarily selected three-dimensional object as step SI04. If the predetermined time has not elapsed (step SI03, No), step SI04 is not executed.

  Subsequently, as step SI05, the control unit 22 calculates the moving speeds of the first object and the second object. And the control part 22 determines whether the calculated moving speed is below a threshold value as step SI06. When the moving speed is equal to or lower than the threshold (step SI06, Yes), the control unit 22 moves or rotates the three-dimensional object according to the detected movements of the first object and the second object as step SI07. To do. Then, the control unit 22 re-executes step SI01 and subsequent steps.

  When the moving speed is faster than the threshold (No in step SI706), the control unit 22 deletes the three-dimensional object as step SI08. When deleting the three-dimensional object, an animation may be displayed so that the three-dimensional object flies in the moving direction of the first object and the second object. Then, the control unit 22 ends the operation detection process. Instead of the operation of moving the first object and the second object at a high speed, for example, erasure of the three-dimensional object may be assigned to an operation of crushing the three-dimensional object. Instead of deleting the three-dimensional object, the three-dimensional object may be returned to the original placement location. The display device 1 does not have to perform the processes of steps SI05, SI06, and step SI08. That is, the display device 1 may execute step SI07 regardless of the moving speed of the two objects after determining NO in step SI03 or after executing step SI04.

  When the distance between the first object and the second object is not substantially constant (No in step SI02), the control unit 22 determines whether the distance is a three-dimensional object, that is, the start point of the operation detection process, as step SI09. It is judged whether it is expanding. When the distance is increased (step SI09, Yes), the control unit 22 determines whether the three-dimensional object displayed between the first object and the second object is in a temporarily selected state as step SI10. judge.

  When the three-dimensional object is in the temporary selection state (step SI10, Yes), the control unit 22 cancels the temporary selection state of the three-dimensional object as step SI11. Moreover, the control part 22 reversely changes a three-dimensional object and returns it to the original state as step SI12. Then, the control unit 22 ends the operation detection process.

  When the three-dimensional object is not in the temporary selection state, that is, in the selection state (step SI10, No), the control unit 22 cancels the selection state of the three-dimensional object as step SI13. Moreover, the control part 22 moves the three-dimensional object which canceled the selection state according to gravity etc. as step SI14. Then, the control unit 22 ends the operation detection process. The movement here is displayed, for example, so that the three-dimensional object falls according to gravity and stops on the floor or table. Before stopping the movement of the three-dimensional object, the three-dimensional object may be bounced according to the elasticity of the three-dimensional object and the hardness of the floor or table. The magnitude of impact when the three-dimensional object collides with the floor or table may be calculated, and if the impact is greater than a predetermined value, the three-dimensional object may be displayed as damaged. Further, the three-dimensional object may be moved more slowly than when actual gravity works.

  When the distance between the first object and the second object is shorter than when the three-dimensional object is selected (step SI09, No), the control unit 22 deforms the three-dimensional object according to the distance as step SI15. . Then, the control unit 22 re-executes step SI01 and subsequent steps. For example, the degree of deformation of the three-dimensional object may be changed according to the hardness set as an attribute of the three-dimensional object.

  As described above, in the third example, since it is detected that the three-dimensional object is positioned between objects such as fingers, the three-dimensional object is changed according to the operation. It is easy for the user to recognize the selection of the three-dimensional object.

  As in Step S81 to Step S83 shown in FIG. 35, after at least one of the first object and the second object comes into contact with the three-dimensional object, the distance between the first object and the second object is approximately equal to or longer than a predetermined time. It may be maintained as a condition for selecting a three-dimensional object. By making contact with a three-dimensional object as a selection condition, it becomes easier for a user to select a desired three-dimensional object when a plurality of three-dimensional objects are displayed close to each other.

  The display device 1 described in the above embodiment can be applied to various uses. The three-dimensional object (display object) to be operated may be an object that imitates an actual object such as a book, a building block, a spoon, a chopstick, a playing card, clay, or a musical instrument, or a virtual object. It may be an object that does not exist, such as an avatar, a game character, or a virtual reality AR tag. Further, the change applied to the three-dimensional object according to the detected operation is not limited to the above movement, deformation, disappearance, and the like. Further, the change applied to the three-dimensional object according to the pressing operation is not limited to the above-described embodiment, and may be changed according to the type of the three-dimensional object.

  For example, when a three-dimensional object imitating clay (hereinafter simply referred to as “clay”) is an operation target, the clay is deformed according to the pressing operation so that the user can shape the clay into an arbitrary shape. Also good. Moreover, the clay may be hardened as if the clay dries over time. Further, when an operation of pushing clay with a finger or hand immersed in a three-dimensional object of water is detected, the viscosity of the clay may be improved.

  In addition, when a three-dimensional object that imitates a record board (hereinafter simply referred to as “record board”) is an operation target, the record board rotates around a fulcrum and plays a sound according to the pressing operation. Also good. A technique such as scratching by a disc jockey may be virtually realized by linking rotation and sound reproduction.

  In the first embodiment, the example in which the display device detects an operation on the three-dimensional object and changes the three-dimensional object according to the detected operation has been described. However, the display device may perform an operation associated with the three-dimensional object according to the displacement of the predetermined object detected by the detection unit in response to an operation on the three-dimensional object. An example of a display device that performs an operation associated with a three-dimensional object according to the displacement of a predetermined object detected by the detection unit will be described.

  First, the configuration of the display device 5 according to the second embodiment will be described with reference to FIGS. 36 and 37. FIG. FIG. 36 is a perspective view of the display device 5. FIG. 37 is a block diagram of the display device 5. As shown in FIGS. 36 and 37, the display device 5 is the same as the display device 1 except that the display device 5 is further provided with a projector 34 and the storage unit 24 stores a control program 24e instead of the control program 24a. It has the composition of.

  The projector 34 projects an image from the projection unit 34 a according to the signal transmitted from the control unit 22. The projected image is displayed on a screen, a wall or the like, and can be viewed by a person other than the user wearing the display device 5. The method by which the projector 34 projects an image is not particularly limited. For example, the projector 34 may draw an image by reflecting laser light emitted from a light source with a MEMS (Micro Electro Mechanical System) mirror. The projector 34 may be configured by combining a light source such as a halogen light, LED, or LD and an optical system such as an LCD or DMD (Digital Micro-mirror Device). The display device 5 may include an external display instead of the projector 34.

  The control program 24e has the same configuration as that of the control program 24a except that it further includes an interlocking display control unit 28. The interlocking display control unit 28 provides a function of causing the projector 34 to project information related to information displayed on the display units 32a and 32b. The functions provided by the interlocking display control unit 28 include a function of changing information to be projected on the projector 34 in conjunction with a change of the three-dimensional object in accordance with an operation on the three-dimensional object in the display space.

  Next, an example of display control in conjunction with a change in a three-dimensional object will be described with reference to FIGS. FIG. 38 is a diagram illustrating an example of display control in conjunction with a change in a three-dimensional object. FIG. 39 is a diagram illustrating an example of a trajectory of an operation in which the finger F1 is momentarily brought into contact with a three-dimensional object. FIG. 40 is a diagram illustrating an example of a locus of an operation for moving the finger F1 along the three-dimensional object. FIG. 41 is a diagram illustrating an example of an operation trajectory for crushing a three-dimensional object with the finger F1.

  In step S91 shown in FIG. 38, the control unit 22 displays a spherical globe as a three-dimensional object OB2 in the display space. Further, the control unit 22 projects, from the projector 34, a map obtained by enlarging Japan located at the center of the three-dimensional object OB2 when viewed from the user wearing the display device 5 as a projection image P4. As described above, the control unit 22 causes the projector 34 to project an image related to the three-dimensional object displayed in the display space. When the projection image P4 is projected within the imaging range of the imaging units 40 and 42, the user can check the state of the projection image P4 based on the images captured by the imaging units 40 and 42.

  In this example, a map of Japan is projected as an image related to Japan located at the center of the three-dimensional object OB2, but instead of the map, other images related to Japan such as the Japanese flag and national flowers are projected. It may be projected. The image may be stored in advance in the storage unit 24, or may be acquired from another device by wired or wireless communication.

  In the state of step S91, it is assumed that an operation of moving the finger F1 obliquely downward as shown in FIG. 39 to make contact with the three-dimensional object OB2 for a moment and releasing it immediately is detected. This operation is similar to the tap operation in which the finger F1 is brought into contact with the touch panel for a moment and then released immediately, but the tap operation can select only a point on the plane, but can select an arbitrary place of the three-dimensional object OB2. Excellent in terms.

  When an operation as shown in FIG. 39 is detected, the control unit 22 determines that the position touched by the finger F1 has been selected, and executes processing corresponding to the selected position. The process corresponding to the selected position is, for example, a process of projecting detailed information corresponding to the selected position from the projector 34. In the present embodiment, as a process corresponding to the selected position, a process of rotating the three-dimensional object OB2 so that the selected position is the center is executed.

  In step S92 shown in FIG. 38, as a result of detecting the operation shown in FIG. 39, the control unit 22 rotates the three-dimensional object OB2 so that the vicinity of the touched Florida peninsula is the center. In conjunction with the rotation of the three-dimensional object OB2, the control unit 22 projects a map of the Florida peninsula moved to the center of the three-dimensional object OB2 from the projector 34 as a projection image P4. The direction in which the finger F1 is moved to make contact with the three-dimensional object OB2 for a moment is not limited to the diagonally downward direction, and may be any direction.

  In the state of step S91, it is assumed that an operation for moving the finger F1 as shown in FIG. 40 along the three-dimensional object OB2 is detected. When an operation as shown in FIG. 40 is detected, the control unit 22 rotates the three-dimensional object OB2 according to the moving direction and moving amount of the finger F1. As described above, it is intuitive and easy for the user to rotate the three-dimensional object OB2 as a process corresponding to the movement of the finger F1 along the three-dimensional object OB2.

  In step S93 shown in FIG. 38, as a result of detecting the operation shown in FIG. 40, the control unit 22 rotates the three-dimensional object OB2 leftward by an angle corresponding to the movement amount of the finger F1. In conjunction with the rotation of the three-dimensional object OB2, the control unit 22 projects a map of the Florida peninsula moved to the center of the three-dimensional object OB2 from the projector 34 as a projection image P4.

  In the state of step S91, it is assumed that an operation for crushing the three-dimensional object OB2 from above is detected using the finger F1 as shown in FIG. When an operation as shown in FIG. 41 is detected, the control unit 22 reduces the three-dimensional object OB2 according to the amount by which the user pushes the finger F1 into the three-dimensional object OB2. Thus, reducing the 3D object OB2 as a process corresponding to the operation of crushing the 3D object OB2 is intuitive and easy for the user to remember.

  In step S94 shown in FIG. 38, as a result of detecting the operation shown in FIG. 41, the control unit 22 reduces the three-dimensional object OB2. Further, in conjunction with the reduction of the three-dimensional object OB2, the control unit 22 reduces the map projected from the projector 34 as the projection image P4.

  Next, a display control processing procedure executed in conjunction with the change of the three-dimensional object will be described with reference to FIG. FIG. 42 is a flowchart illustrating a display control processing procedure executed in conjunction with a change in a three-dimensional object. The processing procedure shown in FIG. 42 is realized by the control unit 22 executing the control program 24e.

  As shown in FIG. 42, first, the control unit 22 synthesizes the image of the virtual space including the three-dimensional object and the image of the real space and displays them on the display unit 32a and the display unit 32b as step SJ01. Further, the control unit 22 causes the projector 34 to project an image corresponding to the three-dimensional object in step SJ02.

  Subsequently, the control unit 22 determines whether a finger is detected by the detection unit 44 as step SJ03. When the finger is not detected by the detection unit 44 (No at Step SJ03), the control unit 22 determines whether an end operation by the user is detected as Step SJ08. If the end operation has not been detected (No at Step SJ08), the control unit 22 re-executes Step SJ03 and the subsequent steps. On the other hand, when the end operation has been detected (step SJ08, Yes), the control unit 22 ends the processing procedure.

  When a finger is detected in step SJ03 (step SJ03, Yes), the control unit 22 determines the movement of the finger in the three-dimensional space based on the detection result of the detection unit 44 as step SJ04. Then, when an operation in which a finger touches the three-dimensional object for a moment is detected (step SJ05, Yes), the control unit 22 executes a process corresponding to the touched position as step SJ06. Then, the control unit 22 updates the image projected in association with the three-dimensional object as step SJ07, and proceeds to step SJ08.

  When a displacement corresponding to the movement of the finger along the three-dimensional object is detected (step SJ05, No, step SJ09, Yes), the control unit 22 responds to the movement direction and movement amount of the finger as step SJ10. To rotate the 3D object. Then, the control unit 22 updates the image projected in association with the three-dimensional object as step SJ07, and proceeds to step SJ08.

  When the displacement corresponding to the operation of crushing the three-dimensional object with the finger is detected (step SJ09, No, step SJ11, Yes), the control unit 22 responds to the amount of pushing the finger into the three-dimensional object as step SJ12. To reduce the 3D object. Then, the control unit 22 updates the image projected in association with the three-dimensional object as step SJ07, and proceeds to step SJ08.

  When none of the above operations is detected (No at Step SJ11), the control unit 22 maintains the projected image as it is and proceeds to Step SJ08.

  As described above, in the present embodiment, since the user's operation is detected based on the movement of the finger in the three-dimensional space, various operation methods can be provided to the user. Further, since the information to be displayed or projected to the outside is updated in conjunction with the change of the three-dimensional object according to the detected operation, the user can also perform the operation for updating the information provided to the third party. Various operation methods can be provided.

  In the above embodiment, three types of operations are detected based on finger movements in the three-dimensional space. However, operations detected based on finger and hand movements in the three-dimensional space include It is not limited. For example, in cooperation with the creation program, the user may be able to produce an image or a building by operating a block or viscosity displayed three-dimensionally with a finger. Further, in cooperation with the racing game program, the user may enjoy the race by operating the handle displayed three-dimensionally by hand.

  In addition, in cooperation with the musical instrument performance program, the user may enjoy the performance by operating the piano or keyboard displayed three-dimensionally with a finger. Further, in conjunction with the data display program, when an arbitrary portion of the three-dimensionally displayed shape is cut by hand, an image of the cut surface may be projected from the projector 34.

  In addition, the aspect of this invention shown by each said Example can be arbitrarily changed in the range which does not deviate from the summary of this invention. Moreover, you may combine each said Example suitably. For example, the control program shown in the above embodiment may be divided into a plurality of modules or may be integrated with other programs.

  In the above embodiment, the example in which the user himself / herself operates the three-dimensional object has been described. Also good. In addition, the display device may share the virtual space with other devices. That is, the display device may be configured such that a person other than the user of the display device can view and operate the three-dimensional object in the virtual space through another device.

  In the above embodiment, the display device alone detects an operation on the three-dimensional object. However, the display device may detect an operation on the three-dimensional object in cooperation with the server device. In this case, the display device sequentially transmits the image captured by the imaging unit or the information detected by the detection unit to the server device, and the server device detects the operation and notifies the display device of the detection result. With this configuration, the load on the display device can be reduced.

1 to 5 Display device 1a Front surface portion 1b Side surface portion 1c Side surface portion 4d External device 13 Operation unit 22 Control unit 24 Storage unit 24a, 24e Control program 24b Object data 24c Action data 24d Virtual space data 25 Detection processing unit 26 Display object control unit 27 Image composition unit 28 Interlocking display control unit 32a, 32b Display unit 34 Projector 34a Projection unit 40, 42 Imaging unit 44 Detection unit 46 Distance measurement unit

Claims (12)

By displaying the image with the overlap adjusted so that the closer of the real object and the predetermined object can be seen in the front, corresponding to each of the user's eyes, the predetermined object is displayed as 3 A display unit for displaying dimensions;
A detection unit for detecting a first object and a second object in real space;
A control unit configured to change the predetermined object according to the type associated with the predetermined object based on the detected movements of the first object and the second object. Display device. The detection unit detects a first state in which the predetermined object is located between the first object and the second object;
The display device according to claim 1, wherein the control unit changes the predetermined object when the first state is detected. The detection unit detects a second state in which the predetermined object is positioned between the first object and the second object for a predetermined time or more;
The display device according to claim 1, wherein the control unit changes the predetermined object when the second state is detected. In the detection unit, the predetermined object is positioned between the first object and the second object, and a distance between the first object and the second object is opened for a predetermined time or more. No third condition detected,
The display device according to claim 1, wherein the control unit changes the predetermined object when the third state is detected. The detecting unit moves at least one of the first object and the second object after the predetermined object is positioned between the first object and the second object, and Detecting a fourth state in which the distance between the first object and the second object does not open for more than time,
The display device according to claim 1, wherein the control unit starts changing the predetermined object when the movement is detected.   The controller is configured to move at least one of the first object and the second object after the predetermined object is positioned between the first object and the second object; and When it is detected that the distance between the first object and the second object is increased before a predetermined time elapses, a reverse change is displayed to reverse the change of the predetermined object. The display device according to claim 5. The display device according to claim 6, wherein the control unit displays the reverse change in a time shorter than a time required for the change. The control unit changes the predetermined object by moving, rotating, or combining the predetermined object with the movement of the first object or the second object. The display device according to any one of claims 1 to 7 . Wherein, with the movement of the first object or the second object, by deforming the predetermined object, of claims 1 to 7, characterized in that changing the predetermined object The display device according to any one of the above. Wherein, with the movement of the first object or the second object, by eliminating the said predetermined object, of claims 1 to 7, characterized in that changing the predetermined object The display device according to any one of the above. When the first object and the second object are in a position between which the predetermined object is sandwiched , the control unit determines the predetermined object according to the type associated with the predetermined object. The display device according to claim 1, further comprising: a control unit that changes the display unit. The display device according to claim 1, wherein action data is associated with each type associated with the predetermined object.

Images