JP6126195B2 - Display device - Google Patents

Description

  The present invention relates to a display device.

  Some display devices including a display unit such as a mobile phone terminal can display an image or the like in a three-dimensional manner (see, for example, Patent Document 1). Stereoscopic display is realized using parallax of both eyes.

JP 2011-95547 A

  Although the stereoscopic display is a display format that is familiar to the user, the stereoscopic display is used only for viewing purposes in conventional display devices, and has not been used to improve the convenience of operation. An object of this invention is to provide the display apparatus which can provide the operation method with high convenience with respect to a user.

  A display device according to the present invention includes a display unit that stereoscopically displays a display object, a detection unit that detects an object in a stereoscopic space where the display object is stereoscopically displayed, and a display unit that detects the object in the stereoscopic space. A control unit that changes the display object in the stereoscopic space according to the movement of the object when the movement is detected.

  In another aspect, the display device according to the present invention is a display unit that displays a display object in three dimensions, and when movement of an object is detected in a stereoscopic space in which the display object is displayed in three dimensions. And a control unit that changes the display object in the stereoscopic space in accordance with the movement of the object.

  The present invention has an effect that it is possible to provide a user with a highly convenient operation method.

FIG. 1 is a front view showing an appearance of a mobile phone terminal according to the first embodiment. FIG. 2 is a block diagram showing a functional configuration of the mobile phone terminal according to the first embodiment. FIG. 3 is a diagram for explaining detection of an operation of pushing a three-dimensional object in the first embodiment and changes in the three-dimensional object according to the detected operation. FIG. 4 is a diagram for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 5 is a diagram illustrating an example of information stored in object data. FIG. 6 is a diagram illustrating an example of information stored in the action data. FIG. 7 is a diagram illustrating an example of information stored in the action data. FIG. 8 is a diagram illustrating an example of information stored in the action data. FIG. 9 is a diagram illustrating an example of information stored in the action data. FIG. 10 is a diagram illustrating an example of information stored in the action data. FIG. 11 is a diagram illustrating an example of information stored in the action data. FIG. 12 is a flowchart illustrating the processing procedure of the contact detection process. FIG. 13 is a flowchart showing the procedure of the operation detection process. FIG. 14 is a diagram for explaining the detection of an operation of pushing a three-dimensional object in the second embodiment and the change of the three-dimensional object according to the detected operation. FIG. 15 is a flowchart illustrating the processing procedure of the operation detection process. FIG. 16 is a diagram for explaining the detection of an operation of pushing a three-dimensional object in the third embodiment and the change of the three-dimensional object according to the detected operation. FIG. 17 is a diagram for explaining the detection of an operation of pushing a three-dimensional object in the third embodiment and the change of the three-dimensional object according to the detected operation. FIG. 18 is a flowchart illustrating a procedure of the operation detection process. FIG. 19 is a front view showing an appearance of a mobile phone terminal according to the fourth embodiment. FIG. 20 is a block diagram showing a functional configuration of the mobile phone terminal according to the fourth embodiment. FIG. 21 is a diagram for explaining the detection of an operation on a three-dimensional object in the fourth embodiment. FIG. 22 is a diagram illustrating a modification of the mobile phone terminal according to the fourth embodiment.

  Hereinafter, the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following description. In addition, constituent elements in the following description include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those in a so-called equivalent range. In the following, a mobile phone terminal will be described as an example of a display device. However, the application target of the present invention is not limited to a mobile phone terminal. For example, a PHS (Personal Handyphone System), a PDA, a portable navigation device, a notebook The present invention can also be applied to personal computers, game machines, and the like.

  First, the configuration of the mobile phone terminal (display device) 1 according to the first embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a front view showing the appearance of the mobile phone terminal 1. FIG. 2 is a block diagram showing a functional configuration of the mobile phone terminal 1.

  As shown in FIGS. 1 and 2, the mobile phone terminal 1 includes an operation unit 13, a microphone 15, a receiver 16, a control unit 22, a storage unit 24, a communication unit 26, an audio processing unit 30, The touch panel 32, the photographing unit 40, and the photographing unit 42 are included. The operation unit 13, the microphone 15, the receiver 16, the touch panel 32, and the photographing unit 40 are partly exposed on the front surface of the mobile phone terminal 1.

  The operation unit 13 includes physical buttons and outputs a signal corresponding to the pressed button to the control unit 22. In the example shown in FIG. 1, the operation unit 13 has only one button, but the operation unit 13 may have a plurality of buttons.

  The microphone 15 acquires external sound. The receiver 16 outputs the other party's voice during a call. The sound processing unit 30 converts the sound input from the microphone 15 into a digital signal and outputs the digital signal to the control unit 22. In addition, the audio processing unit 30 decodes the digital signal input from the control unit 22 and outputs the decoded digital signal to the receiver 16.

  The communication unit 26 includes an antenna 26a, and establishes a radio signal line using a CDMA (Code Division Multiple Access) method or the like with a base station via a channel assigned by the base station. The communication unit 26 performs telephone communication and information communication with other devices through a radio signal line established with the base station.

  The touch panel 32 displays various types of information such as characters, graphics, and images, and detects an input operation on a predetermined area such as a displayed icon, button, or character input area. The touch panel 32 is configured by superimposing a display unit 32a and a touch sensor 32b.

  The display unit 32 a includes a display device such as a liquid crystal display (LCD) or an organic EL (Organic Electro-Luminescence) panel, and displays various information according to a control signal input from the control unit 22. The touch sensor 32 b detects an input operation performed on the surface of the touch panel 32 and outputs a signal corresponding to the detected input operation to the control unit 22. The touch sensor 32b may detect any operation using an arbitrary method such as a capacitance type, a resistance film type, or a pressure sensitive type.

  The touch panel 32 can display a three-dimensional object. The three-dimensional object is a display object such as an image or a shape created so as to be viewed stereoscopically using parallax. The method for displaying the three-dimensional object may be a method for realizing stereoscopic vision using an instrument such as glasses, or a method for realizing stereoscopic vision with the naked eye.

  The imaging units 40 and 42 electronically capture an image using an image sensor such as a charge coupled device image sensor (CCD) or a complementary metal oxide semiconductor (CMOS). The photographing units 40 and 42 convert the photographed images into signals and output the signals to the control unit 22. The imaging units 40 and 42 detect an object to be operated by selecting the three-dimensional object in a space in which the three-dimensional object is displayed in three dimensions (hereinafter, sometimes referred to as “stereoscopic space” or “viewing space”). It also functions as a detection unit.

  Note that it is preferable that the photographing units 40 and 42 have an angle of view and an arrangement so that an object such as a finger can be photographed wherever the object is located in the stereoscopic space. Further, the imaging units 40 and 42 may be a device that acquires an image of visible light, or may be a device that acquires an image of invisible light such as infrared rays.

  The control unit 22 includes a CPU (Central Processing Unit) that is a calculation unit and a memory that is a storage unit, and implements various functions by executing programs using these hardware resources. Specifically, the control unit 22 reads a program or data stored in the storage unit 24 and expands it in a memory, and causes the CPU to execute instructions included in the program expanded in the memory. The control unit 22 reads / writes data from / to the memory and the storage unit 24 and controls operations of the communication unit 26, the display unit 32a, and the like according to the execution result of the instruction by the CPU. When the CPU executes an instruction, data developed in the memory, a signal input from the touch sensor 32b, or the like is used as a part of parameters or determination conditions.

  The storage unit 24 includes a nonvolatile storage device such as a flash memory, and stores various programs and data. The program stored in the storage unit 24 includes a control program 24a. The data stored in the storage unit 24 includes object data 24b and action data 24c. In addition, the memory | storage part 24 may be comprised by the combination of portable storage media, such as a memory card, and the reading / writing apparatus which reads / writes with respect to a storage medium. In this case, the control program 24a, the object data 24b, and the action data 24c may be stored in a storage medium. Further, the control program 24a, the object data 24b, and the action data 24c may be acquired from other devices such as a server device by wireless communication by the communication unit 26.

  The control program 24 a provides functions related to various controls for operating the mobile phone terminal 1. The functions provided by the control program 24 a include a function for controlling the display of the three-dimensional object on the touch panel 32 and a function for detecting a user operation on the three-dimensional object displayed on the touch panel 32.

  The object data 24b includes information regarding the shape and properties of the three-dimensional object. The object data 24b is used for displaying a three-dimensional object. The action data 24c includes information regarding how an operation on the displayed three-dimensional object acts on the three-dimensional object. The action data 24c is used to change the three-dimensional object when an operation on the displayed three-dimensional object is detected. The change here includes movement, rotation, deformation, disappearance, and the like.

  Next, detection of an operation of pushing a three-dimensional object and change of the three-dimensional object according to the detected operation will be described with reference to FIGS. 3 and 4 are diagrams for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. In step S <b> 11 shown in FIG. 3, the three-dimensional object OB <b> 1 is stereoscopically displayed in the stereoscopic space 50 by the touch panel 32. The three-dimensional object OB1 is an object that imitates a ball, for example. In step S11, the touch panel 32 displays the bottom surface B1 that supports the three-dimensional object OB1.

  In step S12, the user places the finger F1 in a position in contact with the three-dimensional object OB1 and keeps it still. When the mobile phone terminal 1 detects a real object in the stereoscopic space and the state in which the object is in contact with the three-dimensional object OB1 continues for a predetermined time or more, the three-dimensional object OB1 is selected as the operation target. judge. Then, the cellular phone terminal 1 notifies the user that the three-dimensional object OB1 has been selected as an operation target, for example, by changing the display mode of the three-dimensional object OB1.

  Whether the object is in contact with the three-dimensional object OB1 is determined based on the actual position of the object in the stereoscopic space and the shape and the calculated position of the three-dimensional object OB1 in the stereoscopic space. The shape of the three-dimensional object OB1 is defined in the object data 24b.

  The position of the object is calculated based on images taken by the photographing units 40 and 42. The calculation of the position of the object may be performed based on the size of the object registered in advance and the size and position of the object in the image. The calculation of the position of the object may be performed by comparing the size and position of the object in the image captured by the imaging unit 40 with the size and position of the object in the image captured by the imaging unit 42. Note that detection of an object such as a finger may be realized using a known technique. When the finger is an object, it is preferable to process the position of the finger toe as the position of the object.

  The calculated position of the three-dimensional object OB1 in the stereoscopic space is calculated based on the position of the three-dimensional object OB1 on the display surface of the touch panel 32 and the lift amount of the three-dimensional object OB1 in the stereoscopic space. The The floating amount of the three-dimensional object OB1 in the stereoscopic space may be a value determined at the time of display, or a right-eye image and a left-eye image used for stereoscopically displaying the three-dimensional object OB1. It may be a value calculated from the difference in position of the three-dimensional object OB1 in the image.

  The notification of selection as the operation target may be, for example, changing the overall color of the three-dimensional object OB1, or changing the color near the position in contact with the object on the surface of the three-dimensional object OB1. Realized by changing. Instead of such visual notification, or in addition to such visual notification, notification by sound or vibration may be performed.

  In this way, the mobile phone terminal 1 determines that the three-dimensional object OB1 has been selected as the operation target when a state in which a finger or the like is in contact with the three-dimensional object OB1 is detected for a predetermined time or longer. By adding the condition that the contact state is continuously detected for a predetermined time or longer, an unintended three-dimensional object is selected as an operation target in the process of moving a finger to operate another three-dimensional object. Can be suppressed.

  After the three-dimensional object OB1 is selected as the operation target, it is assumed that the user has intruded the finger F1 inside the three-dimensional object OB1 so as to press the three-dimensional object OB1, as shown in step S13. The mobile phone terminal 1 changes the three-dimensional object according to the operation when the operation for intruding the object into the three-dimensional object selected as the operation target is detected. How to change the three-dimensional object is determined by the type of the three-dimensional object defined in the object data 24b and the change rule defined in association with the type in the action data 24c.

  For example, in the object data 24b, the three-dimensional object OB1 is defined as an elastic body, and in the action data 24c, it is defined that the elastic body is deformed in the pressed direction according to the amount of pressing when pressed. Shall be. In this case, as shown in step S14, the cellular phone terminal 1 changes the three-dimensional object OB1 so that the part into which the finger F1 has entered is depressed and depressed.

  Further, the object data 24b defines the three-dimensional object OB1 as a rigid body, and the action data 24c defines that the rigid body moves in the pressed direction according to the amount of pressing when pressed. And In this case, the cellular phone terminal 1 moves the three-dimensional object OB1 in the traveling direction of the finger F1 as if pressed by the finger F1, as shown in step S15 of FIG. In step S15 in FIG. 4, since the three-dimensional object OB1 is supported by the bottom surface B1, the three-dimensional object OB1 moves according to the horizontal component of the bottom surface B1 of the force applied by the rigid body.

  As described above, when an operation of pushing the three-dimensional object is detected, the three-dimensional object is changed variously according to the operation by changing the three-dimensional object OB1 based on the object data 24b and the action data 24c. Can do. The operation of pressing is an operation that is used in various scenes in the real world. By detecting the operation of pressing the three-dimensional object OB1 and executing the corresponding process, the operation is intuitive and highly convenient. Can be realized.

  Note that the object used for operating the three-dimensional object is not limited to the finger, and may be a hand, a foot, a stick, an instrument, or the like. In addition, the manner in which the three-dimensional object is changed according to the pressing operation may be in accordance with actual physical laws or may not be actual.

  Further, in the mobile phone terminal 1, when the direction in which the three-dimensional object is pressed is not parallel to the display surface of the touch panel 32, that is, the detected moving direction of the object intersects the display surface of the touch panel 32 or a horizontal plane with the display surface. Even in this case, the three-dimensional object is changed according to the operation. As described above, various operations can be performed on the three-dimensional object by three-dimensionally determining the operation of pushing the three-dimensional object. In order to three-dimensionally determine an operation of pressing a three-dimensional object, it is desirable to prepare a plurality of photographing units and photograph the finger F1 and the like from different directions so that the obstacle does not create a blind spot.

  Next, the object data 24b and the action data 24c shown in FIG. 2 will be described in more detail with reference to FIGS. FIG. 5 is a diagram illustrating an example of information stored in the object data 24b. 6 to 11 are diagrams illustrating examples of information stored in the action data 24c.

  As shown in FIG. 5, the object data 24b stores information including type, shape information, color, transparency, etc. for each three-dimensional object. The type indicates the physical property of the three-dimensional object. The type takes values such as “rigid body” and “elastic body”, for example. The shape information is information indicating the shape of the three-dimensional object. The shape information is, for example, a set of vertex coordinates of the surfaces constituting the three-dimensional object. The color is the color of the surface of the three-dimensional object. Transparency is the degree to which a three-dimensional object transmits light. The object data 24b can hold information regarding a plurality of three-dimensional objects.

  In the action data 24c, information regarding a change when a pressing operation is detected is stored for each type of three-dimensional object. As shown in FIG. 6, when the type of the three-dimensional object is “rigid”, the change when the pressing operation is detected according to the presence / absence of the fulcrum, the presence / absence of the obstacle in the pressed direction, and the pressing speed changes. Different. Here, the obstacle is another three-dimensional object. Further, whether the pressing speed is fast or slow is determined based on a threshold value.

  If the three-dimensional object has no fulcrum and there is no obstacle in the pushed direction, the three-dimensional object is displayed so as to move according to the amount pushed in the pushed direction. The three-dimensional object displayed in this way is, for example, a building block, a pen, or a book. As for the movement method, whether to slide or rotate may be determined based on the shape of the three-dimensional object. Further, whether to move with the pushed object or move away from the object so as to be repelled by the pushed object may be determined based on the pushing speed, and calculation of frictional resistance between the three-dimensional object and the bottom surface. You may determine based on a value or a setting value.

  If the 3D object has no fulcrum and there is an obstacle fixed in the pushed direction, the 3D object moves according to the amount pushed in the pushed direction and moves when it touches the obstacle. Appears to stop. The three-dimensional object displayed in this way is, for example, a building block, a pen, or a book. If the pushed speed is high, the three-dimensional object may destroy the obstacle and continue to move. Also, when the 3D object comes in contact with an obstacle while being moved away from the object so as to be repelled by the pressed object, the 3D object may be moved in the opposite direction so as to bounce off.

  If the 3D object has no fulcrum and there are other rigid bodies that are not fixed in the pushed direction, and the pushed speed is slow, the 3D object moves according to the amount pushed in the pushed direction, After contacting another rigid body, the other rigid bodies are displayed to move together. If the 3D object has no fulcrum and there is another rigid body that is not fixed in the pressed direction, and the pressed speed is fast, the 3D object moves according to the amount pressed in the pressed direction. Is displayed. After the three-dimensional object comes into contact with another rigid body, the other rigid body is displayed so as to be repelled and moved. After coming into contact with another rigid body, the three-dimensional object may stop on the spot or continue to move at a reduced speed. The combination of the three-dimensional object and the other rigid body displayed in this way is, for example, a combination of a bowling ball and a pin, or a combination of marbles.

  If there is another rigid body that does not have a fulcrum in the 3D object and is not fixed in the pressed direction, but the other rigid body can pass through, the 3D object will depend on the amount pressed in the pressed direction. After moving and contacting another rigid body, it is displayed to continue moving as it passes through the other rigid body. In reality, a rigid body does not slip through a rigid body, but by enabling such a slip-through, a novel experience can be provided to the user. The combination of the three-dimensional object and the other rigid body displayed in this way is, for example, a combination of a bowling ball and a pin, or a combination of marbles. It should be noted that a threshold value is provided for the pressing speed, and when the pressing speed is equal to or less than the threshold value, the three-dimensional object may not pass through another rigid body.

  When the three-dimensional object has a fulcrum, the three-dimensional object is displayed so as to rotate around the fulcrum according to the pressed direction and amount. The rotation referred to here may be a rotation that rotates around 360 degrees, or a rotation that reciprocates within a predetermined rotation range. The three-dimensional object displayed in this way is, for example, a pendulum, a boxing sandback, or a windmill.

  As shown in FIG. 7, when the type of the three-dimensional object is “elastic body”, the change when the pressing operation is detected differs depending on the material, whether or not the amount of change is limited, and the pressing speed. The material here is an assumed material of the three-dimensional object, and is defined in the object data 24b.

  If the material of the 3D object is rubber, there is no limit on the amount of change, and the pressing speed is slow, the 3D object is deformed according to the amount pressed in the pressed direction and released from the pressed state. When it is done, it is displayed so as to return to the original shape. In addition, when the material of the three-dimensional object is rubber, there is no limit on the amount of change and the pressing speed is fast, the three-dimensional object is deformed according to the amount pressed in the pressing direction, and then the flip It is displayed to move in the pressed direction while returning to the original shape. The three-dimensional object displayed in this way is, for example, a rubber ball or an eraser.

  If the material of the 3D object is rubber, and the amount of change is limited, the 3D object will be deformed to the deformable range according to the amount pressed in the pressed direction, and the pressing operation will be detected thereafter. Then, it is displayed to move in the pushed direction while returning to the original shape. The three-dimensional object displayed in this way is, for example, a rubber ball or an eraser.

  If the material of the 3D object is metal, the 3D object will be deformed to the deformable range according to the amount pressed in the pressed direction, and will return to its original shape when released from the pressed state. To repeat (vibrate). Note that when pressed in a direction other than the deformable direction, the three-dimensional object moves in the same manner as the rigid body. The three-dimensional object displayed in this way is, for example, a leaf spring or a string spring.

  As shown in FIG. 8, when the type of the three-dimensional object is “plastic”, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape changes. The three-dimensional object displayed in this way is, for example, clay.

  As shown in FIG. 9, when the type of the three-dimensional object is “liquid”, the change when the pressing operation is detected differs depending on the pressing speed. When the pushing speed is slow, the object to be pushed is displayed so as to be in contact with the three-dimensional object, ie, the liquid. When the pressing speed is medium, the object to be pressed is displayed on the liquid so that the ripple spreads on the liquid. When the pushed speed is high, the object to be pushed is touched by the liquid, and it is displayed that the liquid splashes. The three-dimensional object displayed in this way is, for example, water in a cup.

  As shown in FIG. 10, when the type of the three-dimensional object is “gas”, the change when the pressing operation is detected differs depending on the pressing speed. When the pushed speed is slow, a three-dimensional object, i.e., a gas is interrupted by the pushed object and is displayed to drift around it. When the pressing speed is medium, it is displayed that the gas is scattered by the pressing object. When the pushing speed is high, the vortex is generated in the gas behind the pushing object in the moving direction. The three-dimensional object displayed in this way is, for example, smoke.

  As shown in FIG. 11, when the type of the three-dimensional object is “aggregate”, the change when the pressing operation is detected differs depending on the coupling state of the elements of the aggregate. When there is no connection of the elements of the aggregate, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape of the aggregate changes. The three-dimensional object displayed in this way is, for example, sand or sugar.

  When there is a combination of the elements of the aggregate, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape of the aggregate changes. Further, an element other than the pressed part is displayed so as to be moved by being pulled by the element at the pressed part. The three-dimensional object displayed in this way is, for example, a chain.

  When there is no combination of the elements of the aggregate, but an attractive force or a repulsive force is applied to the pushing object, the three-dimensional object is displayed so as to move even if it does not contact the pushing object. When an attractive force acts between the pressed object and the pressed object, the three-dimensional object is attracted to the pressed object when it is within a predetermined distance from the pressed object without contacting the pressed object. In addition, when a repulsive force acts between the pressed object and the pressed object, the three-dimensional object moves away from the pressed object when it is within a predetermined distance from the pressed object without contacting the pressed object. The combination of the displayed three-dimensional object and the pressed object is, for example, a combination of iron powder and a magnet.

  As described above, by changing the three-dimensional object based on the information stored in the object data 24b and the information stored in the action data 24c, the three-dimensional object can be variously changed according to the pressing operation. it can. Note that the information stored in the object data 24b and the action data 24c is not limited to the above example, and may be changed as appropriate according to the application. For example, the method of changing the three-dimensional object may be set according to the type and size of the pressed object and the size of the contact area between the pressed object and the three-dimensional object.

  Next, a processing procedure executed by the mobile phone terminal 1 regarding an operation of pressing a three-dimensional object will be described with reference to FIGS. FIG. 12 is a flowchart illustrating a processing procedure of contact detection processing of a three-dimensional object. The processing procedure shown in FIG. 12 is realized by the control unit 22 executing the control program 24a when a predetermined operation is detected.

  As shown in FIG. 12, the control part 22 displays a three-dimensional object three-dimensionally based on the object data 24b as step S101 first. The object data 24b may be stored in the storage unit 24 in advance, or may be acquired from another device such as a server device by wireless communication by the communication unit 26.

  Subsequently, in step S102, the control unit 22 determines whether a predetermined object is detected by the detection unit, that is, the imaging units 40 and 42. The predetermined object is, for example, a user's finger. When the predetermined object is not detected (No at Step S102), the control unit 22 determines whether the operation end is detected as Step S108.

  The operation end may be detected, for example, when a predetermined operation is performed on the operation unit 13 or may be detected when a predetermined operation is performed on the touch panel 32. The end of the operation may be detected when a predetermined gesture by the user's hand is photographed by at least one of the photographing units 40 or 42. When the end of operation is detected (step S108, Yes), the control unit 22 ends the contact detection process. When the operation end is not detected (No at Step S108), the control unit 22 re-executes Step S102 and the subsequent steps.

  When a predetermined object is detected (step S102, Yes), the control unit 22 determines the type of the predetermined object as step S103. The type of the predetermined object is determined based on, for example, the size, shape, color, and the like of the object in the images photographed by the photographing units 40 and 42. Subsequently, in step S104, the control unit 22 searches for a three-dimensional object that is in contact with a predetermined object.

  When there is no three-dimensional object in contact with the predetermined object (No at Step S105), the control unit 22 determines whether the operation end is detected as Step S108. When the end of operation is detected (step S108, Yes), the control unit 22 ends the contact detection process. When the operation end is not detected (No at Step S108), the control unit 22 re-executes Step S102 and the subsequent steps.

  When a three-dimensional object in contact with the predetermined object is found (step S105, Yes), the control unit 22 determines in step S106 the three-dimensional object in contact with the predetermined object based on the object data 24b. Determine the type. And the control part 22 performs the operation detection process mentioned later as step S107. Thereafter, the control unit 22 determines whether the operation end is detected as step S108. When the end of operation is detected (step S108, Yes), the control unit 22 ends the contact detection process. When the operation end is not detected (No at Step S108), the control unit 22 re-executes Step S102 and the subsequent steps.

  FIG. 13 is a flowchart showing the procedure of the operation detection process. The processing procedure shown in FIG. 13 is realized by the control unit 22 executing the control program 24a.

  As illustrated in FIG. 13, the control unit 22 first acquires a contact time between a predetermined object and a three-dimensional object as step S201. Then, in step S202, the control unit 22 determines whether the predetermined object has moved into the three-dimensional object. When the predetermined object has not moved to the inside of the three-dimensional object (No at Step S202), the control unit 22 re-executes Step S201 and the subsequent steps.

  When the predetermined object is moving inside the three-dimensional object (step S202, Yes), the control unit 22 determines whether the contact time is equal to or longer than the predetermined time as step S203. When the contact time is shorter than the predetermined time (No at Step S203), the control unit 22 ends the operation detection process because it is determined that the three-dimensional object is not an operation target.

  When the contact time is equal to or longer than the predetermined time (step S203, Yes), the control unit 22 calculates the speed of the predetermined object as step S204. In step S205, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, in step S206, the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object. When the predetermined object has not moved outside the three-dimensional object, that is, when the pressing operation is continued (No at Step S206), the control unit 22 re-executes Step S204 and the subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (Yes at Step S206), the control unit 22 determines whether the change of the three-dimensional object continues as Step S207. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step S207, Yes), the control unit 22 changes the three-dimensional object as step S208, and then re-executes step S207 and subsequent steps. When the change of the three-dimensional object does not continue (No at Step S207), the control unit 22 ends the operation detection process.

  As described above, in the first embodiment, the three-dimensional object can be variously changed in accordance with the pressing operation, thereby providing a user-friendly operation method.

  The second embodiment will be described below. The cellular phone terminal 1 according to the second embodiment is different from the first embodiment in the processing procedure of the operation detection process executed based on the function provided by the control program 24a. It has the same configuration as the mobile phone terminal 1 according to the embodiment. Therefore, in the second embodiment, description overlapping with that in the first embodiment is omitted, and the operation detection process will be mainly described.

  First, detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation will be described with reference to FIG. FIG. 14 is a diagram for describing detection of an operation of pushing a three-dimensional object and a change in the three-dimensional object according to the detected operation. In step S21 shown in FIG. 14, the user contacts the finger F1 with the three-dimensional object OB1, and in step S22, the user causes the finger F1 to enter the inside of the three-dimensional object OB1.

  When a real object is detected in the stereoscopic space and the state in which the mobile phone terminal 1 moves to the inside of the three-dimensional object OB1 after contacting the three-dimensional object OB1 continues for a predetermined time or more, It is determined that the dimension object OB1 is selected as an operation target. Then, the cellular phone terminal 1 notifies the user that the three-dimensional object OB1 has been selected as an operation target, for example, by changing the display mode of the three-dimensional object OB1. Further, as shown in step S23, the cellular phone terminal 1 responds to the operation with the finger F1 after the contact detection as if the three-dimensional object OB1 was already selected as the target of the operation to be pressed in the step S21. Change.

  As described above, after detecting the contact between the object and the three-dimensional object, the user can quickly start the operation of pressing the three-dimensional object by making it possible to detect the pressing operation even if the object does not stay on the spot. can do. Further, by adding the condition that the state in which the object moves to the inside of the three-dimensional object OB1 after the contact continues for a predetermined time or longer is not intended in the process of moving the finger to operate another three-dimensional object. Selection of a three-dimensional object as an operation target can be suppressed.

  Next, the procedure of the operation detection process in the second embodiment will be described with reference to FIG. FIG. 15 is a flowchart illustrating the processing procedure of the operation detection process. The processing procedure shown in FIG. 15 is realized by the control unit 22 executing the control program 24a. Note that the processing procedure of the contact detection processing is the same as the procedure shown in FIG.

  As illustrated in FIG. 15, the control unit 22 first determines whether a predetermined object has moved inside the three-dimensional object in step S <b> 301. If the predetermined object has not moved to the inside of the three-dimensional object (No at Step S301), since the three-dimensional object is determined not to be an operation target, the control unit 22 ends the operation detection process.

  When the predetermined object is moving inside the three-dimensional object (step S301, Yes), the control unit 22 determines whether the elapsed time from the contact detection is equal to or longer than the predetermined time as step S302. When the elapsed time is shorter than the predetermined time (No at Step S302), the control unit 22 re-executes Step S301 and the subsequent steps.

  When the contact time is equal to or longer than the predetermined time (step S302, Yes), the control unit 22 calculates the speed of the predetermined object as step S303. In step S304, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, in step S305, the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object. When the predetermined object has not moved to the outside of the three-dimensional object, that is, when the pressing operation is continued (No in step S305), the control unit 22 re-executes step S303 and subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (Yes in step S305), the control unit 22 determines whether the change of the three-dimensional object continues as step S306. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step S306, Yes), the control unit 22 changes the three-dimensional object as step S307, and then re-executes step S306 and the subsequent steps. When the change of the three-dimensional object does not continue (No at Step S306), the control unit 22 ends the operation detection process.

  As described above, in the second embodiment, since the pressing operation is recognized even when the object such as the finger is in contact with the three-dimensional object does not continue for a predetermined time or longer, the user can The operation of pushing the dimension object can be started quickly.

  The third embodiment will be described below. The cellular phone terminal 1 according to the third embodiment is different from the first embodiment in the processing procedure of the operation detection process executed based on the function provided by the control program 24a. It has the same configuration as the mobile phone terminal 1 according to the embodiment. Therefore, in the third embodiment, descriptions overlapping with those in the first embodiment are omitted, and the operation detection process will be mainly described.

  First, detection of an operation of pushing a three-dimensional object and change of the three-dimensional object according to the detected operation will be described with reference to FIGS. 16 and 17. 16 and 17 are diagrams for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. In step S31 illustrated in FIG. 16, the three-dimensional object OB1 is stereoscopically displayed in the stereoscopic space by the touch panel 32. In addition, the user is bringing the finger F1 into contact with the three-dimensional object OB1.

  Here, it is assumed that the user has caused the finger F1 to enter the inside of the three-dimensional object OB1. When the cellular phone terminal 1 detects that the object that has touched the three-dimensional object OB1 has moved to the inside of the three-dimensional object OB1, as shown in step S32, the three-dimensional object OB1 is operated by the finger F1 from that point. Change accordingly. In the example shown in FIG. 16, in step S32, the three-dimensional object OB1 starts moving in accordance with the movement of the finger F1.

  Then, as shown in step S33, the mobile phone terminal 1 determines the three-dimensional object OB1 as an operation target when the finger F1 continues to move toward the inside of the three-dimensional object OB1 for a predetermined time or more. Then, the mobile phone terminal 1 notifies the user that the three-dimensional object OB1 has been determined as an operation target, for example, by changing the display mode of the three-dimensional object OB1. Thereafter, while the movement of the finger F1 to the inside of the three-dimensional object OB1 is detected, the mobile phone terminal 1 continues to change the three-dimensional object OB1.

  Note that, as shown in step S34 of FIG. 17, when the movement of the finger F1 to the inside of the three-dimensional object OB1 is not detected before the predetermined time has elapsed, the mobile phone terminal 1 changes so far. The opposite change is applied to the three-dimensional object OB1. As a result, the three-dimensional object OB1 is displayed in the same state at the same position as in the step S31. The speed at which the reverse change is applied to the three-dimensional object OB1 may be faster than the speed at which the change has been applied to the three-dimensional object OB1 so far. That is, the three-dimensional object OB1 may be reversely changed as if it is reversely played back at high speed.

  In this way, by starting to apply changes to the 3D object from the stage where the intrusion of the object inside the 3D object is detected, the user can confirm that the 3D object is being selected before the selection is confirmed. Can be recognized from. As a result, the user can know at an early stage whether or not the intended three-dimensional object has been selected. When an unintended three-dimensional object is selected, the user can return the original three-dimensional object to an original state by canceling the operation before a predetermined time elapses.

  It should be noted that until the movement of the finger F1 to the inside of the three-dimensional object OB1 continues for a predetermined time or more, the mode is different from the state in which the selected three-dimensional object is changed as the operation target at both the normal time (eg , Translucent). By changing the display mode in this way, the user can easily determine the state of the three-dimensional object.

  Next, the procedure of the operation detection process in the third embodiment will be described with reference to FIG. FIG. 18 is a flowchart illustrating a procedure of the operation detection process. The processing procedure shown in FIG. 18 is realized by the control unit 22 executing the control program 24a. Note that the processing procedure of the contact detection processing is the same as the procedure shown in FIG.

  As shown in FIG. 18, the control unit 22 first determines whether a predetermined object has moved inside the three-dimensional object in step S401. If the predetermined object has not moved to the inside of the three-dimensional object (No at Step S401), since the three-dimensional object is determined not to be an operation target, the control unit 22 ends the operation detection process.

  When the predetermined object is moving inside the three-dimensional object (step S401, Yes), the control unit 22 calculates the speed of the predetermined object as step S402. In step S403, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, in step S404, the control unit 22 determines whether the elapsed time from the contact detection is a predetermined time or more. When the elapsed time is shorter than the predetermined time, that is, when the three-dimensional object is not fixed as the target of the pressing operation (No at Step S404), the control unit 22 sets the three-dimensional object of the predetermined object as Step S405. It is determined whether the movement in the internal direction continues.

  When the movement of the three-dimensional object in the internal direction continues (step S405, Yes), the control unit 22 re-executes step S402 and subsequent steps. When the movement of the three-dimensional object in the internal direction is not continued (No in step S405), the control unit 22 reversely changes the three-dimensional object to return to the original state in step S406. Then, the control unit 22 ends the operation detection process.

  If the elapsed time from the contact detection is equal to or longer than the predetermined time (step S404, Yes), the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object as step S407. When the predetermined object has not moved to the outside of the three-dimensional object, that is, when the pressing operation continues (No at Step S407), the control unit 22 re-executes Step S402 and the subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (step S407, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step S408. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step S408, Yes), the control unit 22 changes the three-dimensional object as step S409, and then re-executes step S408 and subsequent steps. When the change of the three-dimensional object is not continued (step S408, No), the control unit 22 ends the operation detection process.

  As described above, in the third embodiment, since the three-dimensional object is changed according to the operation from the time when the pressing operation is detected, the user is the target of the pressing operation 3 Easy to recognize dimensional objects.

  The fourth embodiment will be described below. In each of the above-described embodiments, the object for manipulating the three-dimensional object is detected based on the image photographed by the photographing unit, but other detection methods may be used. For example, a capacitive touch sensor can detect the position of a finger not touching the touch sensor by increasing sensitivity. Therefore, the fourth embodiment shows an example in which a touch sensor is used as a detection unit that detects an object that operates a three-dimensional object. In the following description, the same parts as those already described are denoted by the same reference numerals as those already described. In addition, description may be omitted for overlapping descriptions.

  First, the configuration of a mobile phone terminal (display device) 2 according to a fourth embodiment will be described with reference to FIGS. 19 and 20. FIG. 19 is a front view showing the appearance of the mobile phone terminal 2. FIG. 20 is a block diagram showing a functional configuration of the mobile phone terminal 2.

  As shown in FIGS. 19 and 20, the mobile phone terminal 2 includes an operation unit 13, a microphone 15, a receiver 16, a control unit 22, a storage unit 24, a communication unit 26, a voice processing unit 30, A touch panel 32.

  The touch panel 32 displays various types of information such as characters, graphics, and images, and detects an input operation on a predetermined area such as a displayed icon, button, or character input area. The touch panel 32 is configured by superimposing a display unit 32a and a touch sensor 32b. In the present embodiment, the touch sensor 32b is a capacitive touch sensor. The touch sensor 32b also functions as a detection unit that detects a finger that operates a three-dimensional object.

  Next, detection of an operation on a three-dimensional object will be described with reference to FIG. FIG. 21 is a diagram for describing detection of an operation on a three-dimensional object. In FIG. 21, the three-dimensional object OB <b> 1 is stereoscopically displayed in the stereoscopic space by the touch panel 32. In FIG. 21, the user is bringing the finger F1 into contact with the three-dimensional object OB1.

  The mobile phone terminal 2 detects the position of the finger F1 using the touch sensor 32b. The touch sensor 32b detects the position of the finger F1 in the X-axis direction and the Y-axis direction by increasing sensitivity, for example, even when the distance in the Z-axis direction from the finger F1 to the surface of the touch panel 32 is about 10 cm. Can do. Further, the touch sensor 32b can detect the distance D2 in the Z-axis direction from the finger F1 to the surface of the touch panel 32 based on the capacitance.

  Based on the detected position of the finger F1 in the stereoscopic space, the mobile phone terminal 2 detects a contact between the finger F1 and the three-dimensional object OB1, or detects an operation in which the finger F1 presses the three-dimensional object OB1. Can be.

  As described above, in the fourth embodiment, the touch sensor is used as the detection unit. Therefore, even a display device that does not include a photographing unit can detect an operation on a three-dimensional object.

  Note that an imaging unit and a touch sensor may be used in combination to detect an operation on a three-dimensional object. When using an imaging | photography part and a touch sensor together, you may specify the position of the finger | toe F1 by averaging each detection result. In addition, in the area close to the touch panel 32, the imaging unit 40 hardly obtains the image of the finger F1, so the weight of the detection result of the touch sensor is increased, and in the area far from the touch panel 32, the detection accuracy of the touch sensor is low. You may use the weighted average which increased weighting of the detection result of the imaging | photography part 40. FIG.

  In addition, an operation on a three-dimensional object may be detected using a plurality of touch sensors so that it is difficult for another finger or the like to become an obstacle and the touch sensor cannot accurately detect the position of the finger. FIG. 22 is a diagram illustrating an example of a configuration of the mobile phone terminal 3 that detects an operation on a three-dimensional object using a plurality of touch sensors.

  The mobile phone terminal 3 includes a first housing 3a, a second housing 3b, and a hinge portion 3c. The hinge portion 3c connects the first housing 3a and the second housing 3b so that they can be opened and closed. The first casing 3a includes a touch panel 32 having a touch sensor 32b, and the second casing 3b includes a touch panel 34 having a touch sensor 34b. As shown in FIG. 22, the touch sensor 32b and the touch sensor 34b come into contact with the stereoscopic space at different angles when the first housing 3a and the second housing 3b are fixed at an angle of about 90 degrees. .

  The touch sensor 32b can detect the position of the finger F1 in the X-axis direction and the Y-axis direction. Further, the touch sensor 32b can detect the distance D2 in the Z-axis direction from the finger F1 to the surface of the touch panel 32 based on the capacitance. The touch sensor 34b can detect the position of the finger F1 in the X-axis direction and the Z-axis direction. The touch sensor 34b can detect the distance D4 in the Y-axis direction from the finger F1 to the surface of the touch panel 34 based on the capacitance.

  As described above, by detecting the finger F1 from different directions, the position of the finger F1 can be detected from any direction even when there is some obstacle. When the finger F1 is detected from a different direction, one touch panel may display a three-dimensional object and the other touch panel may not be displayed, or two-dimensional guidance may be displayed. The touch panel that does not display the three-dimensional object may be a simple touch sensor.

  Hereinafter, application examples and modifications of the display device described in each of the above embodiments will be described. The three-dimensional object (display object) to be operated may be an object that imitates an actual object such as a book, a building block, a spoon, a chopstick, a playing card, clay, or a musical instrument, or a virtual object. It may be an object that does not exist, such as an avatar, a game character, or a virtual reality AR tag. Further, the change applied to the three-dimensional object according to the detected operation is not limited to the above movement, deformation, disappearance, and the like. Further, the change applied to the three-dimensional object according to the pressing operation is not limited to the above-described embodiment, and may be changed according to the type of the three-dimensional object.

  For example, when a three-dimensional object imitating clay (hereinafter simply referred to as “clay”) is an operation target, the clay is deformed according to the pressing operation so that the user can shape the clay into an arbitrary shape. Also good. In addition, the viscosity of the clay may be lowered as if the clay dries over time. Further, when an operation of pushing clay with a finger or hand immersed in a three-dimensional object of water is detected, the viscosity of the clay may be improved.

  In addition, when a three-dimensional object that imitates a record board (hereinafter simply referred to as “record board”) is an operation target, the record board rotates around a fulcrum and plays a sound according to the pressing operation. Also good. A technique such as scratching by a disc jockey may be virtually realized by linking rotation and sound reproduction.

  In addition, the aspect of this invention shown by said Example can be arbitrarily changed in the range which does not deviate from the summary of this invention. Moreover, you may combine said Example suitably. For example, the control program 24a shown in the above embodiment may be divided into a plurality of modules or may be integrated with other programs. In the above embodiment, the three-dimensional object is operated with a finger. However, a rod-like object having a static charge at the tip may be used instead of the finger.

  Moreover, although the example which uses an imaging | photography part and a touch sensor as a detection part in order to detect a three-dimensional object was shown in the said Example, a detection part is not restricted to this. For example, a sensor using a TOF (Time-of-Flight) method may be used instead of the photographing unit. Further, if a proximity sensor or the like that can detect the movement in the plane direction in the stereoscopic space is arranged almost horizontally with the moving direction of the object, the displacement of the object can be detected even without contact. It is preferable that the displacement of the object can be detected without arranging a sensor or the like on the object. By not arranging a sensor or the like on the object, it is not necessary to attach an acceleration sensor to the finger or move the display device itself with acceleration, so that the cost can be reduced.

  In the above-described embodiment, the case where the three-dimensional object jumps out to the user side has been described. However, the present invention can also be applied to the case where the three-dimensional object is present behind the display unit. In this case, a sensor or a camera may be disposed on the back side of the display unit. When the display device is a mobile phone terminal, the display device often includes an in-camera for photographing the user himself and an out-camera for photographing a landscape or the like. Therefore, it may be possible to capture the displacement of the object on the back side by using this out camera.

  In the above embodiment, the display device alone detects an operation on the three-dimensional object. However, the display device may detect an operation on the three-dimensional object in cooperation with the server device. In this case, the display device sequentially transmits the information detected by the detection unit to the server device, and the server device detects the operation and notifies the display device of the detection result. By setting it as such a structure, the load of a display apparatus can be reduced.

1, 2, 3 Mobile phone terminal 22 Control unit 24 Storage unit 24a Control program 26 Communication unit 32, 34 Touch panel 32a Display unit 32b, 34b Touch sensor (detection unit)
40, 42 Imaging unit (detection unit)

Claims (6)

A display unit for displaying a display object in a viewing space;
A detection unit for detecting an object in the viewing space;
A control unit,
When the control unit detects that the object has moved in the internal direction of the display object after the object has contacted the display object, the control unit changes the display object, and a contact time from the contact is a predetermined time. Determine whether or not
If it determines with the said contact time being more than predetermined time, it will determine with the said display thing being an operation target. The display device according to claim 1, wherein the control unit determines that the display object is not an operation target when it determines that the contact time is shorter than the predetermined time. When the control unit determines that the contact time is shorter than the predetermined time, the control unit determines whether the object is moving in the external direction of the display object, and determines that the object is moving in the external direction, The display device according to claim 1, wherein a reverse change is displayed to restore the change of the display object. When the control unit determines that the contact time is shorter than the predetermined time, the control unit determines whether the object is moving in the external direction of the display object, and determines that the object is moving in the external direction, The display device according to claim 1, wherein the display object is vibrated. When the control unit detects that the object has moved in the internal direction of the display object after the object has contacted the display object, the control unit displaces the display object in accordance with the movement of the object, and the contact It is determined whether the contact time from is a predetermined time or more,
If it determines with the said contact time being more than predetermined time, it will determine with the said display thing being an operation target
The display device according to any one of claims 1 to 4. When the control unit determines that the display object is an operation target, the control unit changes the color of the display object or performs notification by sound or vibration.
The display device according to any one of claims 1 to 5.

Images