JP2021192178A - Device, control method, and program - Google Patents

Abstract

To implement various functions of a wearable device.SOLUTION: A device comprises a processing circuit and a first display. The processing circuit sets a second display that is virtual, and displays a part or all of video signals virtually projected on the second display via the first display based on the position and orientation of the first display with respect to the second display.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to devices, control methods and programs.

Wearable devices that are worn on the wrist have advantages over mobile devices such as smartphones in terms of portability, accessibility to functions when used, or when a notification is received. Today, wearable devices have not yet replaced smartphones.

One of the reasons for this is that the input / output interface that is optimal for the characteristics of the wearable device has not been realized. Currently, smartphone-type input / output devices are widely used as wearable devices. Mobile terminals such as smartphones output various graphical contents by taking advantage of the characteristics of a wide display, and realize an intuitive input / output interface by accepting direct gesture input to the contents. On the other hand, one of the characteristics of wearable devices is that their display area is narrow. Therefore, inconsistency occurs by applying the smartphone type input / output interface as it is.

This inconsistency has some problems in terms of both output and input. From the viewpoint of output, the content that can be displayed at one time is inevitably reduced due to the narrow area, and the content is small and dense in order to display the maximum amount of information in a narrow range, making it difficult to see. Be done. From the viewpoint of input, it is easy to make a mistake in finely controlling the position of touching a small and high-density content with a fingertip, and the operation becomes complicated. Furthermore, the fact that the content itself is hidden and invisible by the finger when inputting gestures, and that it is easy to make a mistake in the touching position by hiding is the same for smartphone-type terminals, but for wearable devices, the effect is that. Will also grow.

JP-A-2015-022589

Therefore, the present disclosure provides a wearable device having an input / output interface suitable for the characteristics of the device.

According to one embodiment, the device comprises a processing circuit and a first display. The processing circuit sets a virtual second display, and a part or all of the video signal virtually projected on the second display is first based on the position and orientation of the first display with respect to the second display. Show through the display. The processing circuit may be, for example, a processing circuit such as a CPU, a processing circuit such as an ASIC, or a circuit that performs processing with another digital or analog circuit.

The device may further include a sensor. The processing circuit may display a part or all of the video signal virtually projected on the second display on the first display based on the sensed information of this sensor.

The sensor may be an inertial sensor. The processing circuit may detect that the predetermined posture is reached by the inertial sensor and start displaying the first display. By providing the inertia sensor, the movement of the device can be acquired.

The processing circuit may detect that a predetermined gesture has been input from the inertial sensor and start displaying the first display.

The sensor may be equipped with a camera as one of them. By providing a camera, the surrounding environment can be acquired as video information.

The processing circuit may recognize the user's face or the user's line of sight based on the information acquired by the camera, and display the contents displayed on a part or all of the area of the second display on the first display. .. In this way, it is possible to detect a human face and execute the process, or to recognize the face of a specific person and execute the process.

The sensor may be a touch panel attached to the first display. A touch sensor may be provided in a place different from the display instead of the touch panel.

The processing circuit may execute a predetermined processing based on the gesture acquired by the sensor. By enabling various gesture inputs, for example, when a device is attached, a desired process can be executed by a simple movement.

The display of the first display may be started based on the gesture information sensed by the sensor. That is, it is also possible to display the screen from the standby state based on a predetermined gesture.

The area or posture of the second display displayed by the first display may be changed based on the gesture information sensed by the sensor. In this way, the area of the second display displayed on the first display can be changed by arbitrary gesture input.

The gesture sensed by the sensor may be displayed on the first display. By displaying the gesture, the user can determine whether or not the gesture performed by the user is surely input.

An avatar may be attached to the gesture detected by the sensor and displayed on the first display. For example, the avatar may be changed to an avatar such as a character that the user likes, or the avatar may be changed so that the image of the second display displayed on the first display is not disturbed.

The gesture may be a gesture for the first display. For example, the first display may be provided with a touch panel, and gesture input may be made to the first display.

The gesture may be a gesture for the second display. For example, the device may include sensors such as a camera and a distance measuring sensor so that a user can virtually contact an area of a virtual display and input a gesture in the virtual contact state. For example, a predetermined swipe operation on the second display may be recognized as a gesture input.

The processing circuit may change the position of the second display with respect to the first display based on the gesture. In this way, the position of the second display may be changed while the position of the first display is fixed.

A feedback processing unit, which executes feedback processing for gestures, may be further provided. As described above, when the gesture input is made to the second display, the user comes into contact with the virtual display in the space. In this case, the user cannot detect whether or not the user is in contact with the virtual display. Therefore, when the user is in contact with the virtual display, the feedback process may be performed on the user.

The feedback processing unit may include a vibrator. For example, a vibrator that vibrates by an actuator may be provided, and the user may be informed that the user is in contact with the virtual display by changing the vibration state according to the situation in which the user is in contact with the virtual display.

The feedback processing unit may include a speaker. A speaker may be provided in place of the above-mentioned vibrator or together with the vibrator to perform sound notification.

The sensor may be a microphone. The processing circuit may execute a predetermined process based on the voice information of the user sensed by the microphone. As an input device, a microphone may be provided and information of voice or environmental sound may be sensed.

The display of the first display may change based on the motion parallax due to the relative movement to the second display when the user or the first display moves. By changing the display area of the virtual display in the real display based on the motion parallax, usability can be improved and the sense of presence can be improved.

It may further include a communication unit that transmits / receives information to and from the outside. Then, the relative positional relationship with the external device may be received from the communication unit via the communication unit, and the display on the first display may be updated based on the positional relationship. For example, depending on the relative positional relationship with an external device, the virtual display may be used as a virtual space for arranging object information in AR space, and the real display may be superimposed on a specific object in this AR with the actual background. It can also function as a display to display.

The first display may be a two-dimensional display having a flat surface or a curved surface. As described above, the shape of the actual display is not limited to the plane.

The first display may be a virtual display having a shape of a part of the user's body that is virtually generated in the processing circuit. Then, a part of the user's body may be used as a display surface from the light emitting unit having a projector function. That is, the actual display may be a projection type display.

The first display may be a three-dimensional display. For example, it may be a display that projects an image or the like into a space such as a light field display.

The second display may be a three-dimensional display that projects a 3D object. The shape of the virtual display can be more flexible than the shape of the real display. For example, the virtual display may be defined as a virtual space (VR space, AR space, etc.) itself.

The second display may be defined between the user and the first display.

The second display may be defined farther than the first display from the user. As described above, the position of the first display with respect to the position of the second display, or the position of the second display with respect to the position of the first display can be arbitrarily set depending on the viewpoint.

The processing circuit may be switchable between a mode for displaying the content displayed on the second display via the first display and a mode for displaying the content directly on the first display. That is, the real display may be activated in principle as a display that projects a virtual display, or may be activated in principle as a display that is unique to the real display.

The device may be a wearable device worn on the user's arm. For example, it may be in the form of a watch type (smart watch or the like).

The device may be a glasses-type wearable device. In this case, the first display may be provided on the lens of a spectacle-type wearable device. By using the glasses type, it is possible to browse the actual display without being more conscious of it.

The first display may display the content displayed on the second display in an overlapping manner after displaying the content transparent to the viewpoint from the user. For example, in the case of a spectacle-type device, the content of the virtual display and the content actually viewed by the user can be superimposed and viewed.

The second display may be a display that projects a virtual reality in a virtual space that can be accessed by a plurality of users, and the device may allow a plurality of users to experience the same virtual reality. For example, when performing a remote conference system or the like, it is possible to view the contents shared by a plurality of users from different angles or the same angle.

The second display may be a display that projects the virtual reality in the virtual space, and the virtual reality may be reconstructed by the sensor sensing information of another device located in a remote place.

The processing circuit may reconstruct the content displayed on the first display based on the user's visual acuity.

The processing circuit may display an image having a converted dynamic range on the first display based on the environment of ambient brightness. In this way, the process for improving usability may be executed.

The processing circuit may display a part or all of the image of the second display via the telescope on the first display.

The processing circuit may display a part or all of the image of the second display through the microscope on the first display. In this way, you may use the camera to perform processing that allows you to observe the object of interest.

The device may be deformable. The device is not limited to one form, and may have two or more physically deformable forms, for example, a wristwatch type and a spectacle type.

The processing circuit is based on the first coordinate system based on a predetermined point of the user wearing the device, the second coordinate system based on the point on which the user is wearing the device, and the eyes of the user. The area and direction of the second display to be displayed on the first display may be set based on the third coordinate system and the fourth coordinate system based on a predetermined point of the second display.

The processing circuit may virtually define the position of the second display with respect to a predetermined point of the user.

The processing circuit converts the position of the first display in the second coordinate system into the first coordinate system, converts the position of the user's eye in the third coordinate system into the first coordinate system, and converts the position of the user's eye into the first coordinate system, and the first from the user's eye. The area to be displayed on the first display when the second display is viewed through the display may be calculated, and a part or all of the area of the second display may be displayed on the first display.

The processing circuit may perform coordinate transformation from the second coordinate system to the first coordinate system using Euler angles.

The processing circuit may perform a coordinate transformation from the second coordinate system to the first coordinate system using a quaternion.

The processing circuit may perform a coordinate transformation from the second coordinate system to the first coordinate system using a rotation matrix.

The processing circuit calculates the position and orientation of the first display relative to the second display, and based on the calculated position and orientation of the first display, the content displayed on a part or all of the second display is displayed. It may be projected and converted to the first display.

As the coordinate system, an orthogonal coordinate system, a cylindrical coordinate system, or a spherical coordinate system may be used. As described above, the processing circuit may realize a virtual display by various methods and means, and display the contents on the real display.

According to one embodiment, the control method is a method of causing the processing circuit to perform the process described in any of the above.

According to one embodiment, the program causes the processing circuit to perform the process described in any of the above. As described above, in the processing circuit, information processing by software may be specifically implemented by using hardware such as an actual display.

The figure which shows typically the use example of the wearable device which concerns on one Embodiment. The block diagram schematically showing an example of the wearable device which concerns on one Embodiment. The block diagram schematically showing an example of the wearable device which concerns on one Embodiment. The flowchart which shows an example of the processing of the wearable device which concerns on one Embodiment. The figure which shows typically the use example of the wearable device which concerns on one Embodiment. The figure which shows typically the use example of the wearable device which concerns on one Embodiment. The figure which shows an example of the area of the 2nd display output to the 1st display which concerns on one Embodiment. The figure which shows a part example of the image of the 2nd display which is output to the 1st display which concerns on one Embodiment. The figure which shows an example of the area of the 2nd display output to the 1st display which concerns on one Embodiment. The figure which shows a part example of the image of the 2nd display which is output to the 1st display which concerns on one Embodiment. The block diagram schematically showing an example of the wearable device which concerns on one Embodiment. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows an example of the gesture input which concerns on one Embodiment schematically. The figure which shows typically an example of the movement of the 2nd display by the gesture input which concerns on one Embodiment. The figure which shows typically an example of the movement of the 2nd display by the gesture input which concerns on one Embodiment.

Hereinafter, embodiments in the present disclosure will be described with reference to the drawings. The drawings are for illustration purposes only, and the shape, size, or size ratio of each part configuration to other configurations in an actual device need not be as shown in the figure. In addition, since the drawings are written in a simplified form, it is assumed that configurations necessary for mounting other than those shown in the drawings are appropriately prepared.

(First Embodiment)
FIG. 1 is a diagram schematically showing an example in which a user wears and uses a wearable device according to an embodiment. As shown in FIG. 1, as an example, the wearable device 1 is a wristwatch-type device worn by a user on the wrist. In the present embodiment, the wearable device 1 displays a display that is output to or is assumed to be output to the second display 2, which is a virtual display, via the first display, which is the display surface thereof. conduct. The second display 2 is, for example, a display virtually defined in the processing circuit in the wearable device 1, and the second display 2 does not have an actual physical display. For example, the second display 2 may be defined as a display having a wider display area than the first display provided in the wearable device 1 and displaying more information than the information output to the first display. The present invention is not limited to this, and the display area of the second display may have a display area equal to or narrower than the display area of the first display.

As a specific example, it is assumed that a wide area map is virtually displayed on the second display 2, and a user sees a part or all of the map through the window through the first display of the wearable device 1. Can be done.

In this disclosure, a wristwatch-type device will be mainly described as an unrestricted form, but this is given as an example. That is, the invention of the present disclosure is not limited to a wristwatch-type device. For example, it may be a wearable device such as eyeglasses, a device worn on an arm having a size such as a smartphone, a device that transforms from a wristwatch type to an eyeglass type, and the like. Not limited.

For example, even if it is not a wearable type, a device similar to a wearable type may be treated as a wearable device by using a band or the like. Furthermore, it is possible to realize the same processing for a device that is not a wearable type, for example, a device that is held in a hand for operation and display. That is, although it will be described as a wearable device, a device similar thereto is also included in the form of the present disclosure.

FIG. 2 is a block diagram schematically showing the wearable device 1 according to the embodiment. The wearable device 1 includes an input unit 10, a storage unit 12, a processing unit 14, and an output unit 16. This example shows a simple configuration, and may be appropriately provided with necessary parts.

The input unit 10 includes, for example, various sensors and acquires the state of the wearable device 1. Further, the input unit 10 may include a user interface for acquiring a request from the user.

The storage unit 12 is configured to include, for example, a non-temporary or temporary memory area. This implementation may be any implementation, and may include, for example, various ROMs (Read Only Memory) and RAMs (Random Access Memory). Further, a cache area may be provided. When the processing unit executes processing so that information processing by software is concretely realized by using hardware resources, a program or the like required for information processing of the software may be stored. The storage unit 12 may also store data input from the input unit 10 or store data required for the wearable device 1.

The processing unit 14 is configured to include, for example, a processing circuit. The processing circuit may be, for example, a CPU (Central Processing Unit) or the like. In addition to the CPU, the processing unit 14 may include various accelerators for appropriately executing processing, executing processing at high speed, or executing processing with high accuracy. For example, the above-mentioned second display 2 may be defined so that the processing unit 14 outputs a virtual image in the virtual space. In this case, information such as video may be stored in the storage unit 12. Then, the information output from the first display by the processing unit 14 may be cut out from the second display and output. As another example, even if the processing unit 14 executes an operation and outputs only the display of the second display (or the information including the information around it) in the area estimated to be displayed on the first display. good.

The output unit 16 includes, for example, a first display. This first display is a display provided in the wearable device 1. The processing unit 14 estimates a state in which the user looks into the information of the second display 2 through the first display, and performs arithmetic processing on a video or image (hereinafter referred to as video or the like) signal to be output to the first display. The acquired video, etc. is displayed on the first display.

The basic configuration of the wearable device 1 is as described above. Hereinafter, a more specific example will be described in detail.

FIG. 3 is a block diagram schematically showing an example of a wristwatch-type wearable device 1. The wearable device 1 includes an inertial sensor 100, a storage unit 12, a processing unit 14, and a first display 160.

The inertial sensor 100 is a sensor provided in the input unit 10 of the wearable device 1. The inertial sensor 100 is a sensor that acquires acceleration and the like about the movement of the wearable device 1, that is, how the wearable device 1 is moved when the user is stationary or inertial movement, and outputs the information. be. The inertial sensor 100 may be provided with, for example, an acceleration sensor, a gyro sensor, or the like, or may be provided with a plurality of these so as to capture movements with respect to a plurality of axes. In addition, the inertial sensor 100 may acquire an angular velocity, or may be a multi-axis sensor that captures acceleration in a plurality of directions.

The first display 160 is one of the output interfaces provided in the output unit 16 of the wearable device 1. The wearable device 1 outputs information to the user by displaying the information on the first display 160. The first display 160 acts as, for example, a window for peeping at the second display 2 from the user side. By moving the wearable device 1 to a position where the user wants to see the information or adjusting the posture of the wearable device 1 to an angle where the user wants to see the information, the user can virtually display the image on the second display 2. It is possible to see any part or all of the information from any posture. In this way, the wearable device 1 can see an image or the like displayed on the second display 2 existing in the VR (Virtual Reality) or AR (Augmented Reality) space.

For example, the user can input a request to the wearable device 1 by moving the arm on which the wearable device 1 is attached according to a predetermined gesture. This gesture is acquired by the inertial sensor 100 and processed by the processing unit 14. Further, based on the movement information of the wearable device 1 acquired by the inertial sensor 100, the processing unit 14 acquires the position and posture of the first display 160 with respect to the second display 2.

The processing unit 14 extracts the coordinates of the three points of the user's body, and displays a part or the whole of the second display 2 via the first display 160 from the relative position-posture relationship of the three points. Then, the user can view a part or all of the image or the like displayed on the second display 2 through the first display 160 of the wearable device 1.

The user can change the area that can be seen through the first display 160 in the image or the like displayed on the virtual second display 2. The user may implement the implementation in which the area of the second display 2 output to the first display 160 is changed by moving the wearable device 1, for example. The wearable device 1 senses that the relative position of the wearable device 1 itself has moved by the inertia sensor 100, and outputs this movement information to the processing unit 14. The processing unit 14 changes an area such as an image displayed on the first display 160 based on this movement information. Similar to the above, this process may be performed not only on the position of the wearable device 1 but also on the posture.

By changing the position and posture of the first display 160 in this way, the processing unit 14 appropriately updates the content displayed on the first display 160 and outputs the output. Therefore, the user can acquire information in different areas of the second display 2 by moving the wearable device 1, that is, his / her arm in the direction he / she wants to see.

FIG. 4 is a flowchart showing a processing flow of the wearable device 1 according to the present embodiment.

First, the inertial sensor 100 senses the state of the wearable device 1 (S100). As described above, state sensing is realized, for example, by acquiring a change in acceleration when the user is stationary or in inertial motion. For example, it is possible to acquire how the wearable device 1 has moved or how the posture has changed from the sensing information of the inertial sensor 100.

Next, the processing unit 14 acquires the relationship between the positions and postures of the predetermined three points of the user's body based on the information acquired by the inertial sensor 100 (S102). The predetermined three points of the user's body are, for example, the position of the user's chest, the position of the eyes, and the position where the wearable device 1 is worn.

The positions of these three points may be calculated from, for example, the mounting position of the wearable device 1 and the state of movement acquired by the inertial sensor 100. For example, assuming that the user wears the wearable device 1 on his left arm and the first display 160 faces the back of his hand, in this case, it can be calculated from the movement state of the wearable device 1 acquired by the inertial sensor 100. can.

More specifically, the wearable device 1 operates according to the constraint of the user's body. Based on this constraint condition, the processing unit 14 estimates and acquires the positions of the predetermined three points. As an example, assuming that the user moves the wearable device 1 from the arm down state to the front of the chest and makes it stand still, the chest is based on the position before the move and the resting position after the move. Estimate the position of. Such movements of the user are general movements. Further, as described below, the state in which the wearable device 1 is viewed is estimated by detecting the movement of the user's arm from an arbitrary state by the inertial sensor 100 without setting the reference state of the arm lowered state. It is also possible.

FIG. 5 is a diagram showing an example in which the user is looking at the wearable device 1. In this figure, three coordinate systems are shown, but they are not shown to identify the origin, but as axes that indicate the direction. The origin can be any point, but for example, the origin of the first coordinate system is the first predetermined point W, the origin of the second coordinate system is the second predetermined point R, and the origin of the third coordinate system is. , Third predetermined point E, etc. may be defined. The origin may also be defined so that the calculation can be performed faster or more accurately.

As shown in this figure, for example, a first coordinate system based on the position of the chest, which is the user's first predetermined point W, is defined. As an example, the axis is the z-axis in the height direction of the user, and the axis that intersects the z-axis at right angles, that is, the axis that is in the horizontal plane when the user stands upright, and the axis that connects the user's shoulders is x. The y-axis is the axis that intersects the axis, x-axis, and z-axis at right angles. The method of taking the axis is arbitrary as long as it is a coordinate system in which the three-dimensional coordinates are appropriately defined, and is not limited to the method of taking such a coordinate system.

Similarly, a second coordinate system with reference to a second predetermined point R, which is any one point of the wearable device 1, is defined. As an example, the axes are the x-axis and y-axis of the first display 160 and the z-axis in the depth direction that intersects these axes at right angles. Similarly, the method of taking the axis may be arbitrary as long as it is appropriate. The position of the second predetermined point R is set to be a point near the center of the first display 160 for easy understanding in the illustration. For example, a point indicating the upper left pixel of the first display 160, a center point, etc. , As long as the position in the first display 160 can be specified, the method of taking the second predetermined point R (reference point) may be arbitrary.

Further, a third coordinate system based on the third predetermined point E, which is the position of the user's eyes, is defined. As an example, the axis may be the axis connecting both eyes of the user as the x-axis, the direction of the user's head as the y-axis, and the axis perpendicular to these axes as the z-axis. Similar to the above, the method of taking the shaft may be arbitrary as long as it is appropriate. As another example, the direction from the third predetermined point E to the first display 160 (for example, the second predetermined point R) may be set as the z-axis, and the y-axis may be set as the direction perpendicular to these two axes. Alternatively, the third coordinate system may be a coordinate system having a different origin from the second coordinate system and having axes parallel to each axis, that is, a coordinate system obtained by translating the second coordinate system.

In FIG. 5, as an example, the third predetermined point E is the left eye of the user, but the present invention is not limited to this. For example, the third predetermined point may be the position of two points at the positions of both eyes, or may be the middle point between the eyebrows, that is, the positions of both eyes. For example, in the case of the positions of both eyes, the coordinate system may be formed by using the third predetermined points E and E'. In this way, by having two reference positions in the third coordinate system, for example, a stereo image may be supported.

By preparing the axes of the three series in this way, it is possible to define the axes centered on the positions of the three predetermined points of the user. That is, the posture of the periphery with respect to each predetermined point can be acquired from the direction of this axis. Specifically, the overall position of the user and the overall posture of the body can be acquired from the position of the first predetermined point W and the direction of the axis. From the position of the second predetermined point R and the direction of the axis, the position of the wearable device 1 and the posture with respect to the first coordinate system can be acquired. The position of the user's eyes and the direction of the line of sight can be obtained from the position of the third predetermined point E and the direction of the axis.

The above-mentioned predetermined points may be appropriately determined at the time of factory shipment of the device, for example. At the time of shipment from the factory, each position acquired by a plurality of users may be defined by statistical means as a predetermined point.

Further, at the initial booting of the device, the user may perform calibration by designating the position where the wearable device 1 is normally viewed, the position of the chest, and the position of the eyes. For example, the wearable device 1 is moved to the position of the arm, the position of the chest, and the position of the eyes when the user normally looks at the device, and a signal such as resting or tapping the screen or pressing a button is sent at this timing. Calibration may be performed by the processing unit 14 based on the sensing information of the inertial sensor 100 in the above. The parameters of this calibration may be stored in the storage unit 12. Further, the configuration may be such that the user can calibrate at an arbitrary timing as well as at the initial startup. If the calibration has not been executed for a predetermined period of time, the configuration may be such that recalibration is urged.

The processing unit 14 acquires the position and posture of a predetermined point of the user by using these three coordinate systems.

Returning to FIG. 4, the processing unit 14 then converts the content displayed on the second display 2 into the content displayed on the first display 160 (S104). This conversion is converted based on the position of each point acquired in S102 and the postures of the user and the device around each point. In this step, in addition to the above coordinate system, it is necessary to further associate the display area of the second display 2 with each coordinate system.

FIG. 6 is a diagram showing a user who uses the wearable device 1 and a coordinate system of the second display 2. The fourth coordinate system is a coordinate system defined with reference to a fourth predetermined point V, which is an arbitrary point on the second display 2. In the fourth coordinate system, for example, the x-axis and y-axis of the second display 2 may be the x-axis and the y-axis, respectively, and the depth direction when the user virtually sees the second display 2 may be the z-axis.

Since the fourth predetermined point V is a point on a virtual display, it can be set arbitrarily. The processing unit 14 may arbitrarily define the relationship between the first predetermined point W and the fourth predetermined point V. Conversely, the position of the second display 2 may be defined based on the first predetermined point W and the posture of the axis of the first coordinate system.

More specifically, when the first predetermined point W and the axis of the first coordinate system are set, the processing unit 14 forms the second display 2 such as the position, shape, size, posture, etc. of the second display 2. Set the necessary information to do so. This setting may be performed, for example, based on the data stored in advance in the storage unit 12, or may be performed based on the data that can be changed by the user. Then, a fourth coordinate system is set for the defined second display 2, and a video signal is virtually generated in this coordinate system. The order of formation of the second display 2 and the formation of the fourth coordinate system is not limited to this, and the fourth coordinate system may be defined first. Further, an area or the like for virtually displaying an image such as an area where the second display 2 exists may be defined in the first coordinate system without setting the fourth coordinate system.

By fixing the positional relationship between the first predetermined point W and the fourth predetermined point V, the wearable device 1 estimates the first predetermined point W, the second predetermined point R, and the third predetermined point E, and then the user When viewing the second display 2 from the eyes through the first display 160, it is possible to acquire and display an appropriate image or the like. That is, the relationship between the position and the axial direction of the first predetermined point W and the fourth predetermined point V, and the relationship between the estimated first predetermined point W, the second predetermined point R, and the third predetermined point E in the position and the axial direction. , The area and posture of the second display 2 when the first display 160 is viewed from the third predetermined point E can be acquired.

Therefore, the processing unit 14 calculates the area and the posture of the second display 2 to be displayed on the first display 160 based on the relationship between each predetermined point and the coordinate system. The processing unit 14 acquires, for example, an image output to the first display 160 by converting an image or the like displayed on the second display 2 by various projection conversion methods.

Here, an example of coordinate transformation will be specifically described. The conversion of this embodiment is not limited to the following examples.

First, the parameters of the positions and postures of the second predetermined point R and the third predetermined point E are acquired with reference to the first predetermined point W. Each of these parameters has 6 degrees of freedom: [3 degrees of freedom in position, 3 degrees of freedom in posture]. Since there are two points, it is an unknown parameter with 12 degrees of freedom.

As described above, as an example without limitation, the first predetermined point W indicates one point on the user's chest, the second predetermined point indicates any one point on the first display 160, and the third predetermined point E indicates the user's eyes. It is a point.

The parameter indicating the position can be acquired as the coordinates indicating each point.

The parameters indicating the posture are indicated by, for example, Euler angles. This Euler angle can be obtained as an angle in each coordinate system. For example, the position of the first predetermined point W is indicated by the coordinates of the chest in the first coordinate system, and the posture is indicated by the Euler angles indicating the direction of the chest in the first coordinate system. The position of the second predetermined point R is the coordinates in the second coordinate system, and the attitude is the direction of the first display 160 in the second coordinate system, for example, the normal vector (unit of) of the first display 160 at the second predetermined point R. It may be shown in Euler angle from the vector shown as (vector). The position of the third predetermined point E is indicated by the coordinates in the third coordinate system, and the posture is indicated by the Euler angles as the direction in which the user's eyes are facing in the third coordinate system.

The posture may be shown not only by Euler angles but also as a quaternion having 4 degrees of freedom or a rotation matrix. By indicating with parameters other than Euler angles, there is a possibility that it can be applied more flexibly without considering gimbal lock in the calculation between the axis. On the other hand, parameters can be reduced by using Euler angles. If the procedure for calculating from the information from the inertial sensor is fixed, it is possible to avoid this problem by setting the calculation formula in advance by a calculation method that avoids gimbal lock even for Euler angles. Further, based on the processing circuit provided in the processing unit 14, which of these methods is used as an index indicating the parameter may be appropriately selected.

The processing unit 14 acquires the position and orientation of the second predetermined point R from the information sensed by the inertial sensor 100. This acquisition may be realized, for example, by detecting and tracing the movement of the inertial sensor 100. For example, the user moves the wearable device 1 with his arm from the state where his hand is lowered to the position where the first display 160 can be seen. In the state of moving and resting in this way, the second coordinate system is set and the position from the first predetermined point W is estimated.

In addition, by acquiring the time-series data with an appropriate length, the user is trying to see the first display 160 of the wearable device 1 even if the user is not in the predetermined state of lowering his / her hand. It is also possible to estimate by a trained model. By associating the inertial sensor 100 with correct data such as motion capture in various movements, it is possible to detect whether the user has transitioned from an arbitrary state to a state in which the user is looking at the first display 160. can.

The estimation may be acquired as the output of a trained model based on the sensing information of the inertial sensor 100 when the arm is moved variously, for example, the time series data of acceleration and angular velocity. The trained model holds the time series of the sensing information of the inertial sensor 100 described above and the time series data of the correct position and posture acquired by another means such as motion capture as a database, and these data are used as teacher data. A trained model trained as may be used. Then, the trained model that outputs the correct position and posture of the wearable device 1 from the time series data of the inertial sensor 100 is optimized as an inference device. The processing unit 14 may estimate the position and orientation of the second predetermined point R using this inference device.

ここで、d(x, y)は、xとyの距離を求める関数であり、例えば、2次のノルムで表されるが、これに限られるものではなく、適切な関数であればよい。例えば、dは、勾配を取得できる関数であってもよい。 The training is realized as follows, as an example. Here, a is the sensing data of the inertial sensor 100, for example, 6-dimensional data composed of acceleration and angular velocity, and ^W p _R is 6-dimensional data indicating the correct position and orientation of the second predetermined point R in the first coordinate system. Let W be a weight matrix (6 × 6 dimensions), and optimize W by machine learning based on Eq. (1).

Here, d (x, y) is a function for finding the distance between x and y, and is represented by, for example, a quadratic norm, but the function is not limited to this and may be an appropriate function. For example, d may be a function that can get the gradient.

The weight parameter W is obtained by machine learning based on Eq. (1). Then, the processing unit 14 acquires the position and orientation of the second predetermined point R in the first coordinate system from the time series data sensed by the inertial sensor 100 by using the trained W parameter. That is, by calculating Wa for the actual data a sensed by the inertial sensor 100, the position and attitude of the second predetermined point R in the first coordinate system can be acquired. The machine learning method used for optimizing W may be any method.

Further, the calculation of the position of the second predetermined point R from the data sensed by the inertial sensor 100 is not limited to the model optimized by machine learning. It may be a linear model or a non-linear model that can appropriately acquire the position and orientation of the second predetermined point R in the first coordinate system from the sensing data of the inertial sensor 100.

Subsequently, the processing unit 14 acquires the position and attitude of the third predetermined point E from the information sensed by the inertial sensor 100. This acquisition may be realized, for example, by detecting the movement of the inertial sensor 100 and inputting it into the trained model in the same manner as described above.

ここで、bは、顔画像特徴のn次元データ、^Rp_Eは、第2座標系における第3所定点Eの正しい位置、姿勢を示す6次元データ、Vを重み行列（6 × n次元）とする。dは、上述と同様に、任意の適切な距離関数である。また、学習方法についても、上記と同様に、任意の手法としてもよい。 In addition to the above-mentioned database, the training of this trained model may be performed by holding the time-series data of the feature b of the facial image when the arm is moved in various ways as a database. Then, from the time-series data of the arm movement, a trained model for acquiring the position and posture of the second predetermined point E in the second coordinate system is acquired by optimization.

Here, b is the n-dimensional data of the facial image feature, ^R p _E is the 6-dimensional data indicating the correct position and posture of the third predetermined point E in the second coordinate system, and V is the weight matrix (6 × n-dimensional). And. d is any suitable distance function, as described above. Further, the learning method may be any method as described above.

As an example, b acquires the position of the eyes and the direction of the line of sight when the user moves his / her arm in various ways from the image as feature quantities. A model that estimates the position and posture of the third predetermined point E when the user moves the arm looking at the wearable device 1 by acquiring the movement of the arm, the position of the eyes, and the movement of the posture as time series data. Can be optimized.

Here, the representation of variables (scalars, vectors, matrices, etc.) used in the operations in the present disclosure will be described.

Equation (3) is expressed as a 4 × 1 homogeneous coordinate vector in the X coordinate system. Here, each coordinate system is represented by a code of a predetermined point as a reference. For example, the first coordinate system has the first predetermined point W, the second coordinate system has the second predetermined point R, the third coordinate system has the third predetermined point E, and the fourth coordinate system has the fourth predetermined point V. Expressed using. For example, when W is described in the equation, the point represents the first predetermined point W, and the coordinate system designation represents the first coordinate system.

For example, ^V x _V expresses the shape and display contents of the second display 2 in the fourth coordinate system. If the second display 2 is flat, it is _{fixed at z V} = 0. The second display 2 is not limited to this, and may have an arbitrary three-dimensional shape.

For example, ^R x _R expresses the shape and expression content of a real display in the second coordinate system. Again, if the first display 160 is flat, it is _{fixed at z R} = 0. The first display 160 may have any three-dimensional shape without being limited to this.

Equation (4) represents a 4 × 4 rigid transformation matrix from the X coordinate system to the Y coordinate system. Here, ^Y _X R indicates a 3 × 3 attitude matrix of the Y coordinate system with respect to the X coordinate system. ^Y _X t indicates the 3 × 1 relative position vector of the Y coordinate system with respect to the X coordinate system. The values of these ^Y _X R and ^Y _X t elements may be calculated, for example, from the above-mentioned position and posture parameters by a method generally well known in relation to computer vision.

^W _V M represents a first predetermined point W the second positional relationship between the display 2. This relationship may be preset, i.e. fixed.

ここで、h_xxは、変換行列の各成分を示す未知の要素である。空間的な整合性を確保するように投影するために、未知パラメータとしてこの変換行列の推定が必要となる。パラメータの推定を容易にするために、投影方法の制約を設定してもよい。投影方法の制約は、例えば、1点〜3点の透視投影、平行投影を行うことにより取得されてもよい。 Then, a matrix that transforms the display of the second display 2 into the display of the first display 160, that is, a 4 × 4 dimensional matrix that shows the projection transformation from the 4th coordinate system to the 2nd coordinate system is given in Eq. (5). It is defined as.

Here, h _xx is an unknown element indicating each component of the transformation matrix. In order to project so as to ensure spatial consistency, it is necessary to estimate this transformation matrix as an unknown parameter. Projection method constraints may be set to facilitate parameter estimation. The restrictions on the projection method may be acquired, for example, by performing perspective projection or parallel projection of 1 to 3 points.

These transformations have been shown as Cartesian coordinates (Cartesian), but are not limited to, and generate models in cylindrical (cylindrical), spherical, or other suitable coordinate systems. May be good.

Each parameter defined above is used to execute S104 in FIG. Specifically, by executing the coordinate transformation of the position and the posture as follows, a part or all of the image or the like displayed on the second display 2 is displayed on the first display 160.

In this step, first, the display of the second display 2 in the fourth coordinate system is converted to the third coordinate system via the first coordinate system, and the position of the point of the second display 2 as seen from the viewpoint is acquired. do. By using the above matrix, the coordinates of the 4th coordinate system are converted into the 1st coordinate system based on the equation (6).

ここで、Kは、第3座標系から視点への4 × 4次元の投影変換行列である。 On the other hand, the value of z of the fourth predetermined point V in the first coordinate system, that is, the distance from the first predetermined point W to the second display 2, and the fourth predetermined point V converted into the first coordinate system. The relationship is expressed by Eq. (7).

Here, K is a 4 × 4 dimensional projection transformation matrix from the third coordinate system to the viewpoint.

ここで、^W _Vm_zは、^W _VMの3行目の成分（z軸に対応する成分）を示す1 × 4次元のベクトルである。 Substituting Eq. (6) into Eq. (7) gives Eq. (8).

Here, ^W _V m _z is a 1 × 4 dimensional vector showing the component (component corresponding to the z axis) in the third row of ^W _{V M.}

ここで、^R _Vh_zは、^R _VHの3行目の成分（z軸に対応する成分）を示す1 × 4次元のベクトルである。 The first display 160 viewed from the viewpoint can be expressed as equation (11) from equations (9) and (10).

Here, ^R _V h _z is a 1 × 4 dimensional vector showing the component (component corresponding to the z axis) in the third row of ^R _{V H.}

この解を与える数式の1つとして、式(13)の関係式が導きだされる。

The projection transformation matrix ^R _V H is obtained from the condition that the points shown in Eq. (8) and the points shown in Eq. (11) are located, that is, ^E x _V = ^E x _R.

The relational expression of Eq. (13) is derived as one of the mathematical expressions that gives this solution.

The relationship of equation (13), as holds for any ^V x _V, to estimate the unknown parameters ^R _V H. This estimation method is not limited, and is performed by a general method. For example, it may be executed using an algebraic method (including a method by approximate calculation), a numerical analysis method by iterative calculation, a method by machine learning, or it may be executed based on various other optimization methods. May be good.

The matrix ^R _V H is factory, computed in advance a plurality of states based on the statistically acquired data may be stored in the storage unit 12, subjected to specific calibration to the user It may be calculated later and stored in the storage unit 12. Further, it may be calculated at an appropriate timing. For example, the processing unit 14 may update the parameter each time the inertia sensor 100 senses motion.

Processor 14, by the conversion of the second display content of the display 2 in the fourth coordinate system using the obtained matrix ^R _V H as described above formula (14), is displayed on the first display 160 Acquires information such as images on the second display 2.

Next, the processing unit 14 calculates the content to be displayed on the first display 160 based on the coordinate transformation of the equation (14), and displays the information (S106). By executing this process, a part or all of the image or the like displayed on the second display 2 updated in real time can be output to the first display 160.

For example, when the user moves the position of the arm, in S100, the inertial sensor 100 detects the movement of the arm, and the area such as the image of the second display 2 output to the first display 160 in accordance with the movement of the arm, Update and output at least one of the postures. For this output, the processing unit 14 may execute the parameter update operation in S104.

FIG. 7 is a diagram showing a case where the position of the wearable device 1 is moved in parallel with each display. From the state shown in the above figure, for example, it is assumed that the user moves the wearable device 1 to the right side. In such a case, as shown in the figure below, the processing unit 14 recalculates by updating the above parameters, and displays an image or the like about the area of the second display 2 different from the above figure on the first display 160. Let me.

In the above description, the stationary states are shown, but of course, the processing unit 14 updates the parameters in real time based on the movement of the inertial sensor 100 even while moving, and displays them on the first display 160. The content (area of the second display 2) may be changed in real time. It is possible to perform the same processing not only in the left-right direction but also in the up-down direction.

FIG. 8 is a diagram showing an example of a part of the area of the second display 2 output to the first display 160. For example, it is assumed that an image of a pet-type robot is displayed on the second display 2. In this case, when the first display 160 is positioned as shown in the upper figure of FIG. 7, for example, an image around the center of the second display 2 is output to the first display 160 as shown in the upper figure.

Then, when the user moves the first display 160 to the right side as shown in the lower figure of FIG. 7, the area of the second display 2 displayed on the first display 160 moves as shown in the lower figure of FIG. It changes with. Then, for example, an image on the right side of the center of the second display 2 in the vertical direction is output to the first display 160.

In this way, for example, the information of the virtual display having a wider area can be output by using the first display 160 having a narrower display area as compared with the second display 2. By moving the wearable device 1 (first display 160) in the direction in which the content to be viewed exists, the user can change the output area and view the video or the like corresponding to the moving direction.

Further, as an example, when the display area of the first display 160 is an area including the outside of the second display 2, padded display, for example, a blank display may be performed on the outer area. As another example, if the first display 160 moves from the edge of the second display 2 to an area that includes the outside, it will display on the first display 160 so that it stays in the area at the edge of the second display 2. You may.

FIG. 9 is a diagram showing a case where the position of the wearable device 1 is moved perpendicular to each display. From the state shown in the above figure, for example, it is assumed that the user moves the wearable device 1 in a direction closer to himself / herself. In such a case, as shown in the figure below, the processing unit 14 recalculates by updating the above parameters, and the first display displays an image or the like about the area of the second display 2 having a different angle of view from the above figure. Display at 160.

In the above description, the stationary states are shown, but similarly to the above, the processing unit 14 generates an image based on the movement of the inertial sensor 100 even while moving, and the first display 160 is displayed in real time. You may update the output. It is possible to perform the same processing not only in the approaching direction but also in the moving direction.

FIG. 10 is a diagram showing an example of a part of the area of the second display 2 output to the first display 160. When the first display 160 is positioned as shown in the upper figure of FIG. 9, for example, an image near the center of the second display 2 is output to the first display 160 as in the upper figure of FIG.

Then, when the user moves the second display 2 so as to approach itself as shown in the lower figure of FIG. 9, as shown in the lower figure of FIG. 10, the first display 160 is, for example, the second display 2. The position does not change, but the video etc. is output so as to output a wider range. However, in FIG. 10, the size of the first display 160 is drawn so as to change, but when the size of the first display 160 itself is fixed, the area is the same as the above figure. , A reduced and wider area is output.

Further, as an example, when the distance between the first display 160 and the user is closer than a predetermined distance, the first display 160 may display the entire area of the second display 2. In this case, the surroundings may be padded to display the second display 2 according to the reduction ratio, or the display may be fixed from the timing when the width or height of the second display 2 fits in the first display 160. ..

The above two patterns can be mixed. That is, the wearable device 1 may move with a component parallel to the display and a component perpendicular to the display. As described above, the processing unit 14 processes and draws a part or all of the area of the second display 2 on the first display 160, so that the second display is directed to the user (the first predetermined position W). It is possible to display so that the size of the display area of 2 does not change.

In the above example, the size of the second display 2 does not change, but it is not limited to this. For example, the amount of movement output to the first display 160 with respect to vertical, horizontal, and vertical movements based on the point at which the user first begins to see the display of the second display 2 through the first display 160. May be emphasized or suppressed. For example, the image may be moved vertically, horizontally, and vertically beyond the distance the user has moved the wearable device 1. On the contrary, the image may be moved vertically, horizontally, and vertically, less than the user moves the wearable device 1. It can also be mixed. For example, various mixing methods may be used, such as emphasizing in the left-right direction and suppressing in the vertical direction, or emphasizing in the parallel direction and suppressing in the vertical direction.

As described above, according to the present embodiment, the wearable device 1 can view a part or the whole area of the virtually defined display through the actual display. Furthermore, by freely moving the actual display, it is possible to change the area on which the virtual display is displayed. For example, a map based on GPS (Global Positioning System) is displayed on a virtual display, and a user can view an arbitrary position on this map on a real display, or enlarge or reduce this display. Can be seen.

As described above, from the user's point of view, it is possible to realize a device that changes the display content of the real display depending on the positional relationship between the real display and the virtual display of the wearable device. According to this wearable device, the virtual display is placed in a fixed position with respect to a user-dependent position, for example, the user's chest, so that the user always has a virtual display that is in front of him. It is possible to have an experience like looking through a window. According to the wearable device of the present embodiment, for example, it is possible to realize a display having a display area smaller than that of a smartphone but having a wide area that is substantially the same as that of a smartphone. That is, it has the same amount of information as the display of a smartphone, and the size of the content can be the same.

When the wearable device is a wristwatch type, it is possible to further set a constraint condition that the wearable device is attached to the arm and the position and posture are limited from the human skeleton. Based on this constraint condition, it is possible to limit the search space such as position and posture, simplify the calculation, and perform more interactive display.

Further, in the above examples of FIGS. 7 and 8, the display content is assumed to move in the same direction as the moving direction, but the display content may be changed so as to move in the opposite direction depending on the case. For example, the positions of the first display 160 and the second display 2 with respect to the first predetermined point W may be reversed. In this case, when the first display 160 moves to the right, the area on the left side of the second display 2 so that the user can perceive that the first display 160 is behind the second display 2. It may be displayed so as to move to. By outputting in this way, the user can more strongly perceive that the focus is on the second display 2. In this way, intuitive operation may be realized based on the motion parallax. Similarly, in the vertical direction, the moving directions of the first display 160 and the second display 2 may be reversed according to the intuitive operation of changing the pitch angle of the camera.

The image or the like initially displayed on the second display 2 is arbitrary. For example, a web browser may be displayed, a map may be displayed, or a specific application may be displayed. On the other hand, the user may execute an arbitrary application by inputting through the input interface. An example of the input interface is as described in the embodiments described below. As described above, since the second display 2 can have a display area equal to or larger than that of a smartphone or the like, the image or the like to be depicted on the virtual display is arbitrary, and the first display 160 can be appropriately displayed. 2 It should be noted that if a part or all of the area of the display 2 can be displayed, it is included in the technical contents of the present disclosure.

(Second embodiment)
In the above, it is assumed that the display on the first display 160 is started from an arbitrary state, but the display is not limited to this. For example, a predetermined gesture by the user may be sensed by the inertial sensor 100, and the display on each display may be started by this predetermined gesture.

For example, the user may start drawing by moving the wearable device 1 to a position where the wearable device 1 is moved to a position where the user raises his / her arm and then looks at the wristwatch. This predetermined gesture is determined based on the information sensed by the inertial sensor 100. For example, the inertial sensor 100 may be triggered and start processing by sensing a sudden change in the downward acceleration from the application of the sudden upward acceleration to the gravitational acceleration. Then, the second predetermined point R may be defined at a point stationary with respect to the gravitational acceleration, the first predetermined point W or the like may be set according to the above flow, and the subsequent processing may be executed.

In this way, a predetermined gesture may be used as a flag for starting output to the first display 160. In addition, the gesture may correspond not only to the start but also to the end of the application. Further, the processing unit 14 may execute an arbitrary application or any processing in the application by the gesture. As described above, the processing unit 14 may be configured to acquire gesture information based on the information sensed by the inertial sensor 100 and to execute arbitrary processing according to the gesture information.

The predetermined gesture is not limited to raising the arm. For example, the gesture may start with the user wearing the wearable device 1 and swinging the arm, and the gesture is not limited to this, and any gesture that can be naturally performed by a human can be used. .. Further, the user may arbitrarily set the gesture of each process including the start. Of course, in this case as well, a gyro sensor or the like may be used to detect that the wearable device 1 is stationary in a posture within a predetermined range and start the process. The display on the first display 160 is not special, and it can be executed as a kind of processing by sensing this gesture.

For example, as described in the above-described first embodiment, processing may be performed in the vertical and horizontal directions of the screen and in the depth direction, with the stationary state as the starting posture. This movement in the vertical and horizontal directions and the movement in the depth direction can also be regarded as a kind of gesture. That is, the processing unit 14 may execute an arbitrary process corresponding to the gesture by an arbitrary gesture.

(Third Embodiment)
In the first embodiment and the second embodiment described above, the wearable device is configured to include an inertial sensor as an input and a real display as an output, but the wearable device is not limited to these and has many more input / output interfaces. You may.

FIG. 11 is a block diagram schematically showing an example of the configuration of the wearable device 1 according to the third embodiment. The wearable device 1 further includes a communication unit 18 in addition to the configuration of the above-described embodiment.

In addition to the inertial sensor 100, the input unit 10 includes a camera 101, a distance measuring sensor 102, a touch sensor 103, a physical button 104, a microphone 105, a heartbeat sensor 106, a myoelectric sensor 107, a body temperature sensor 108, and a sweating sensor 109, as an example. , Brain wave sensor 110, position sensor 111, any combination is provided. As shown in the above-described embodiment, the input unit 10 can request the processing unit 14 to perform various processes. For example, the processing unit 14 may change the position and orientation of the display area of the second display 2 on the first display 160 based on the sensing information acquired from the input unit 10, and execute various other processes. Can be done.

As one of the processes, the processing unit 14 can automatically execute various processes based on the information from various sensors without the intention of the user. Further, as one of the processes, the processing unit 14 can execute various processes requested by the user based on the information input by the user's intention acquired from various sensors. That is, the processing unit 14 can execute an explicit command from the user or various processes in a non-explicit state based on the information acquired via the input unit 10.

In addition to the first display 160, the output unit 16 includes a speaker 161, an actuator 162, and a light emitting element 163 in any combination. Of course, the combination with each component of the input unit 10 is also arbitrary.

Further, the block diagram of FIG. 11 is shown as an example, and the internal components of the input unit 10 and the output unit 16 are not all indispensable and may be arbitrarily mounted.

The camera 101 is a sensor that acquires the surrounding environment as video information. A plurality of cameras 101 may be provided instead of a single camera 101. The camera 101 may be, for example, an in-camera provided on the user side (first display 160 side) of the wearable device 1, an out-camera provided on the opposite side, or both of them. You may be. Further, the present invention is not limited to this, and a plurality of cameras 101 may be provided for acquiring the environment of the wearable device 1 in any direction.

When there are a plurality of cameras 101, a pinhole camera having a fine hole structure or the like may be inconspicuous so as not to deteriorate the appearance from the user. When the in-camera is provided on the first display 160 side, it may be provided on the lower side of the display, that is, on a surface other than the user side of the display, on the lower side of the display light emitting element. In this case, the camera 101 may be configured as a back surface light receiving element or a front surface light receiving element.

Further, although the camera 101 is provided in the wearable device 1, it may be one that communicates with an external camera via the communication unit 18 and also uses the information of the camera.

For example, when the camera 101 is provided with an inner camera, in the above-mentioned first embodiment, the position and posture of the third predetermined point E determined by the information acquired from the inertial sensor 100 are acquired as image or video information. be able to. By using such a camera 101, it is possible to acquire more detailed data such as a user's viewpoint and a line-of-sight direction. For example, when obtaining the position and posture of the third predetermined point E using the evaluation function represented by the equation (2), the data of the feature points of the facial image can be used. Machine learning may be performed using the feature points of the image acquired by the camera 101 in various situations as training data to generate a trained model. Of course, it is also possible to more accurately acquire information on the position and posture of the first predetermined point W. The wearable device 1 may acquire this information not only by the inertial sensor 100 or the camera 101 but by using the data of both of them in a comprehensive manner.

Further, even for the movements shown in FIGS. 7 and 9, by acquiring the image information with the camera 101 and acquiring the user's eyes from the information of the feature points, etc., the first display 160 with higher accuracy can be obtained. It is possible to realize the conversion for the output.

Further, the gesture described in the second embodiment can also be acquired based on the information such as the image taken by the camera 101, not the sensing information of the inertial sensor 100. For example, various gestures can be photographed through the camera 101, and this can be used as training data to generate a trained model by machine learning. Then, the processing unit 14 can read out an arbitrary gesture by inputting information such as a video into the trained model. In this case, it is possible to input gestures in a state where the wearable device 1 itself can be acquired as an image or video such as the facial expression of the other arm or the face without moving the wearable device 1. For example, the area and posture of the second display 2 output to the first display 160 may be changed by changing the direction of the face or the direction of the line of sight.

As another example, the camera 101 is provided as an in-camera, and when the in-camera detects a human face, the output to the first display 160 may be started. Further, the face authentication may be executed based on the image acquired from the camera 101, and the output to the first display 160 may be performed based on the result of the face authentication. By providing the in-camera in this way, it is possible to improve security.

As described above, by providing the in-camera as the camera 101, it is possible to realize the processing of the first embodiment and the gesture processing of the second embodiment more clearly and accurately.

When the processing unit 14 recognizes the input of the gesture, the information of the gesture may be displayed on the first display 160. This display may be displayed superimposed on the display of the second display 2, or may be displayed in a predetermined window in the first display 160. Further, the user may be notified that the gesture input has been accepted by any of the output units described later. In this way, the user may be clearly informed that the gesture has been entered. As a result, the user can confirm that the gesture he / she has input is properly input to the wearable device 1.

The distance measuring sensor 102 is, for example, a sensor using sound waves, infrared rays, or the like, and is a sensor that measures the distance to a target object. The ranging sensor 102 may be provided, for example, in place of the above-mentioned in-camera. For example, when the positions of the first predetermined point W and the third predetermined point E are acquired by the inertia sensor 100 and the camera 101, the distance to the first predetermined point W or the third predetermined point E is determined by the distance measuring sensor 102. It may be acquired more accurately.

The touch sensor 103 is a sensor that reacts when touched by a human. For example, it may be configured as a touch panel having a touch sensor 103 above or below the display surface of the first display 160, or may be provided on the side surface of the device and used as a fingerprint sensor, a button, or the like. By using this touch sensor 103, the usability of the input interface can be improved.

In the above-described embodiment, each process is input by a gesture, but by further providing the touch sensor 103, it becomes possible to perform intuitive and more advanced input. Through the touch sensor 103, the user can request processing from the wearable device 1 by various methods such as tapping, double tapping, dragging, flicking / swiping, and pinch-in / pinch-out. Furthermore, gesture operations can be performed by dragging on the touch panel. For example, for the video displayed on the 2nd display 2, it is required to add a marker to the text, enlarge / reduce it, pin it to a map, and request playback / fast forward / rewind processing of the video. Is easily possible. These are shown as examples, and the scope of application is not limited to these.

The physical button 104 is an input interface that acts as a switch to physically turn on / off, trigger, or toggle. The physical button 104 may be provided, for example, as a power switch or the like. Further, the touch sensor 103 may be provided as an alternative to the physical button 104, or the touch sensor 103 may be further provided on the physical button 104. For example, the physical button 104 may be capable of requesting the processing unit 14 to perform a process capable of exerting the functions described in the above description of the touch sensor 103.

The microphone 105 is a sensor that acquires sound. For example, the microphone 105 senses a user's voice, surrounding environmental sounds, and the like. Instead of the above gesture or the like, voice input to the microphone 105 may be performed. When performing voice input, the processing unit 14 may appropriately process the information sensed by the microphone 105. A trained model trained by machine learning may be used for this process. Further, in addition to the voice input, the laughter of the user may be determined and the display of the first display 160 may be changed accordingly. For example, the user may be able to notify the processing unit 14 of the above-mentioned various processes by voice via the microphone 105.

The heart rate sensor 106 acquires the user's heart rate (or pulse). Based on the information sensed by the heart rate sensor 106, the processing unit 14 can execute processing based on the user's state. For example, if the user's heartbeat is higher than the predetermined heartbeat, it is possible to change the display output to the first display 160, assuming that the user is exercising or the like. For example, if the user's heartbeat is a heartbeat that is determined to be jogging, the processing unit 14 may determine that the first display 160 is not displayed. For example, if the user's heart rate is determined to be anaerobic exercise, a workout application is launched, the application is displayed on the second display 2, and the application is displayed via the first display 160. The display may be output so that the user can appropriately monitor his / her own condition.

The myoelectric sensor 107 acquires the current flowing through the user's muscles. For example, the myoelectric sensor 107 acquires a current flowing through the user's muscles from a weak current flowing through the user's skin. This myoelectric sensor 107 can also be used in the same manner as the heart rate sensor 106.

The body temperature sensor 108 acquires the user's body temperature. The body temperature sensor 108 can also be used in the same manner as the heart rate sensor 106. Further, by using the body temperature sensor 108, the user may acquire a sleepy state or the like. In such a case, the processing unit 14 lowers the display brightness of the second display 2 displayed via the first display 160, or changes the color to reduce the blue light (for example, changes the color temperature). May be good.

The sweating sensor 109 acquires the sweating state of the user. For example, the sweating sensor 109 acquires the sweating state of the user by sensing the wetness of the skin of the user. Similarly, the sweating sensor 109 may function as one sensor for knowing the exercise state of the user.

The brain wave sensor 110 acquires the state of the user's brain wave. For example, the electroencephalogram sensor 110, like the myoelectric sensor 107, estimates an electroencephalogram from a weak current flowing through the user's skin. This estimation may be realized by a trained model optimized by machine learning, for example, by acquiring data based on various light exercise states and using the acquired data as training data. For example, the electroencephalogram sensor 110 is mounted on a glasses-type wearable device 1. As an example, an electrode may be provided so that the temple of the spectacles of the wearable device 1 touches the temple, touches the back of the ear, or touches the nose in the pad of the spectacles.

If the heart rate sensor 106, the myoelectric sensor 107, the body temperature sensor 108, the sweating sensor 109, and the brain wave sensor 110 only acquire the exercise state of the user such as relaxing, walking, or running, for example. Instead, it may function as a sensor for acquiring the user's emotions, physical condition, muscle fatigue, and the like. The processing unit 14 may include a trained model capable of appropriately executing these processes, or may execute a determination based on a rule base based on statistically acquired data. Further, the processing unit 14 may acquire such a user's emotion, physical condition, etc. based on the user's voice sensed by the microphone 105.

Based on the acquired user's motor state, emotions, physical condition, etc., for example, the processing unit 14 appropriately switches the image to be output to the first display 160, or is appropriate from other components of the output unit 16. Information may be output.

The position sensor 111 includes, for example, GPS. The position sensor 111 senses and outputs the position of the wearable device 1. Based on the information acquired by the position sensor 111, the processing unit 14 may display a map on the second display 2, for example. The user can view this map via the first display 160. Further, the position sensor 111 is not limited to GPS, and may be a sensor that receives radio waves such as wi-fi (registered trademark) and a beacon and acquires a position from information contained in the radio waves. Further, the position sensor 111 may be a sensor that acquires a relative position. In this case, in combination with the inertial sensor 100, the processing unit performs processing corresponding to the motion parallax and arm movement of the above-described embodiment. It is also possible for 14 to run.

As described above, the output unit 16 can display a part or all of the area of the second display 2 via the first display 160. Then, for example, a feedback processing unit that outputs feedback such as user actions and gestures may be provided. A speaker 161, an actuator 162 (vibrator), and a light emitting element 163 may be provided as the feedback processing unit. Of course, these speakers 161 and the like may function not only as feedback but also as a general output unit.

The speaker 161 outputs sound information to the user.

The actuator 162 outputs to the user by vibrating the housing of the wearable device 1.

The light emitting element 163 is an LED (Light Emitting Diode) or the like provided separately from the light emitting element provided in the first display 160, and outputs light to the user by emitting light.

When functioning as a feedback processing unit, the components of these output units 16 perform feedback processing for input from the user. For example, the feedback processing unit may output different outputs depending on whether the input from the user is normally acquired or not. The feedback processing unit emits different sounds, outputs different vibrations, and emits light by different light emitting methods, for example, in the case of normal / abnormal. The present invention is not limited to this, and feedback output may be performed based on the gesture input by the user. For example, different feedback may be performed for the first gesture and for a second gesture that is different from the first gesture.

The wearable device 1 further includes a communication unit 18. The communication unit 18 is an interface that communicates with an external device, server, or the like by wired or wireless means. For example, the wearable device 1 is connected to the server via the communication unit 18.

The wearable device 1 can acquire various information from the server via the communication unit 18. For example, when the map information is virtually output to the second display 2, the map information may be acquired from an external server based on the position information acquired from the position sensor 111. However, the map information may be stored in the storage unit 12, or the map information acquired from the server via the communication unit 18 may be stored in the storage unit 12.

Further, the firmware may be updated and various data may be updated by the server via the communication unit 18.

Further, the calculation of the position, posture, etc. of each predetermined point in the above-described embodiment may be executed on the server side. That is, the processing unit 14 may transmit the sensing information from various sensors to the server via the communication unit 18 and receive the information processed by the server via the communication unit 18. Then, based on the position and posture information received from the server, the area of the second display 2 output to the first display 160, the posture, and the like may be calculated, and further, the data itself obtained by performing these calculations. May be obtained from the server. As described above, various operations may be appropriately executed by an external server instead of the processing unit 14.

For example, the wearable device 1 may be paired with a smartphone by a user via Bluetooth (registered trademark) or the like, and this smartphone can also be used as a server. In this way, the configuration can be configured in consideration of the privacy of the user.

The processing unit 14 may appropriately allocate the processing based on the communication status and the computing power of the computer, or the user may explicitly select which processing to perform. Further, the operation to be processed by the processing unit 14 and the operation to be processed by the server may be defined in advance, and the operation may be executed at an appropriate place based on this definition. That is, the analysis of various data and the processing of the data may be appropriately executed by either the processing unit 14 or the server.

The communication unit 18 is connected to, for example, a server, but the present invention is not limited to this. For example, it may be connected to a server and then connected to another user's device. Further, it may be connected to another device without going through a server. The wearable device 1 may be P2P connected to another device using the Internet, for example, via the communication unit 18, or may be P2P connected to another device by forming an ad hoc network.

As another example, a process of switching an image or the like to be displayed on the second display 2 or the first display 160 may be performed by detecting a position relative to another device via the communication unit 18. By performing such processing, in the wearable device 1, the other device can be virtually displayed on the second display 2, or a virtual object can be placed at the position of the other device. It is possible to expand the AR function according to the positional relationship of.

As described above, the wearable device 1 is not limited to the configurations of the first embodiment and the second embodiment described above, and may include various interfaces or components. By providing various sensors as the input unit 10, it is possible to process a wide range of display contents and display methods of the second display 2 displayed on the first display 160. By providing various modules as the output unit 16, it is possible to widely support the output method of the information acquired by the input unit 10. Further, by providing the communication unit 18, it is possible to execute communication and further improve usability.

(Fourth Embodiment)
In the above-described embodiment, various input / output interfaces have been described, but in the present embodiment, other embodiments of the first display 160 and the second display 2 will be described.

The first display 160 is said to be a flat display, but the present invention is not limited to this. For example, the first display 160 is not limited to being provided on the outside of the wristwatch-type wearable device 1, but may be provided on the inside, as a curved surface, or on the entire circumference. It may be something that can be done. Further, as another form, the first display 160 may be a foldable type, or may be provided with a light emitting unit that emits light to the outside and may project an image or the like on the back of the user's hand or the like.

Further, as another example, the first display 160 may be a 3D display. For example, a device having a fine hole array and projecting into space using light interference by outputting appropriate weak coherent light from each of the holes constituting the hole array (light field display). May be good. Further, the contents of the second display 2 may be output to the user by holography, a liquid convex lens, or a direct projection of the laser on the retina.

Since the shape of the second display 2 is virtually realized, it is possible to make the shape more flexible than that of the first display 160. For example, the second display 2 may be flat as described above. Further, the second display 2 may implement a 3D virtual reality space as a display in a broad sense. In this case, the video or the like may be an object, a background, or the like arranged in the space.

For example, the shape and size may be arbitrary. The second display 2 has an arbitrary shape such as a vertically long rectangle, a horizontally long rectangle, a circle, a polygon, and an annulus, and may have an arbitrary size such as a smartphone size and a newspaper size.

For example, it may have a three-dimensional shape instead of a two-dimensional shape. For example, it may be an L-shape, a curved surface, each face of a polyhedron, a sphere, a cylinder centered on the user, a cube, a dome shape, or the like.

For example, it may be a plurality of separated configurations instead of one. For example, windows or tabs may be arranged in the depth direction, or may be arbitrarily arranged around the user's body. For example, a remote controller for gesture input and a second display 2 indicating a keyboard may be near the user, and a screen-type second display 2 may be located further away from the user.

As mentioned above, it may be a 3D object rather than a display. The object may be any object, for example, an alarm clock, a television, a radio, a book, a game console, or the like.

Further, the shape and position of the second display 2 may be dynamically changed from the application or content side. For example, when watching a movie, it may be a 21: 9 display, it may be vertically long for browsing a web browser, and a bookmark area having another shape may be in front of the ring. For example, it may change to become darker or narrower depending on the state of the main character, the poisonous state, and the dark state in the game. For example, when calling the globe application, the second display 2 may be transformed into a spherical shape.

Further, the shape, position, etc. of the second display 2 may be dynamically changed depending on the state of the user. For example, when you are relaxed, you may be in front of you, and when you are sitting, you may look good. When walking, it may be a small display, or when running, it may be transformed into a minimum display suitable for a running form.

Any content may be displayed in the area outside the second display 2. For example, it may be a blue screen, an external image acquired by the outer camera 101 may be projected, or a predefined virtual space or theme may be displayed.

The processing unit 14 appropriately converts the video and the like on the second display 2 and outputs the video to the first display 160. Of course, as described above, the processing unit 14 appropriately calculates and outputs the area and posture of the second display 2 projected on the first display 160 according to the gesture or the instruction from the user.

The processing unit 14 appropriately executes conversion of an image or the like projected on the first display 160 by setting the origin and the axis of the third predetermined point E. In this case, the projection conversion method described above may be executed instead of an appropriate one. Further, the stabilization of the projection error may be corrected. For example, when an alias occurs in an image or the like to be displayed on the first display 160, the image or the like may be corrected by using various anti-aliasing methods (for example, smoothing). This is a process that can be applied also in each of the above-described embodiments.

As described above, the first display 160 and the second display 2 are not limited to flat displays, and may be displays having any shape and size. By appropriately performing conversion by the processing unit 14, for example, it becomes possible to appropriately display a part or all of the second display 2 on the first display 160 in response to the movement of the wearable device 1.

(Fifth Embodiment)
In this embodiment, an example of gesture input will be described. The gesture input may be a gesture for the first display 160 or a gesture for the second display 2. Further, as described above, the gesture may be a gesture caused by the user moving his / her own body, or may be a general gesture input to a touch panel or the like.

FIG. 12 is a diagram showing an example of gesture input to the first display 160. For example, when the first display 160 is a touch panel, it is possible to perform gesture input (including operations such as tapping) by touching the display surface of the first display 160. This is the same as a device such as a general smartphone.

FIG. 13 is a diagram showing an example of gesture input to the second display 2. For example, the wearable device 1 includes a camera 101 that acquires a state on the opposite side of the first display 160. When the user's finger is detected by the camera 101, the user's finger may be superimposed on the display of the second display 2 and output to the first display 160.

The processing unit 14 may enable gesture input when it is determined that the user's finger is at a distance of the second display 2. The distance of the second display 2 does not accurately indicate a plane or the like, and a distance within a certain range from a certain plane of the display may be an allowable range. In FIG. 13, the finger overlaps with the content, and the overlapped part is not visible, but this is not limited to this.

FIG. 14 is a diagram showing another example of gesture input to the second display 2. As shown in FIG. 14, the finger may be translucent and displayed on the first display 160 so as not to hide the information such as the image displayed on the second display 2. Further, instead of changing the transparency, the color of the user's finger may be changed and displayed on the first display 160.

FIG. 15 is a diagram showing another example of gesture input to the second display 2. As shown in FIG. 15, the finger of the user to which the avatar is applied may be displayed on the first display 160 by superimposing the image or the like displayed on the second display 2.

FIG. 16 is a diagram showing another example of gesture input to the second display 2. As shown in FIG. 16, an icon or a cursor pointing to the position of the user's finger may be displayed on the first display 160 without displaying the finger on the second display 2.

FIG. 17 is a diagram showing another example of gesture input to the second display 2. As shown in FIG. 17, an icon indicating the position of the user's finger may be displayed on the first display 160 without displaying the finger on the second display 2. Here, for example, in the state of FIG. 16, the icon may be changed when the user's face acquired by the in-camera has a predetermined facial expression such as a smile or a wink. For example, when it is determined that the user's face is smiling, the icon may be changed from the state shown in FIG. 16 to the state shown in FIG.

In these processes, for example, the processing unit 14 detects the user's finger based on the image acquired by the camera 101, and based on the detection result, processes the avatar or the like as described above at the position indicated by the fingertip. Run.

In addition, gestures are recognized as various gestures not only by the position of the finger, but also by, for example, pinching the air with the finger, wiping the hand, holding the hand, or performing the deco pin movement. It may be set to be done. The processing for each gesture may be stored in the storage unit 12 in advance, and appropriate processing may be performed by the processing unit 14.

When performing a gesture such as a touch operation on the second display 2, the user must position the virtual display, and it is difficult to recognize the position of the display. Therefore, each configuration of the feedback processing unit described in the above-described embodiment may notify the user that the user has touched the virtual display. For example, the user may be notified by making a sound or vibrating the wearable device 1.

FIG. 18 shows an example of a case where the position is specified in the line-of-sight direction with respect to the display. For example, the processing unit 14 estimates which position of the second display 2 the user is looking at based on the user's line-of-sight direction acquired by the in-camera. Based on this estimation result, the pointer may be displayed on the first display 160. As for the estimation of the line-of-sight direction, a trained model optimized by machine learning using a large number of training data may be used as in the various estimations described above, or a model based on a statistically obtained rule base. May be performed using.

FIG. 19 is a diagram showing another example of gesture input to the second display 2. As shown in FIG. 19, when the first display 160 is a projection to the back of the hand, the gesture to the back of the hand may be displayed on the first display 160 on the display of the second display 2. In this case, a light emitting unit that functions as the first display 160 is provided on the back side of the hand, and a camera 101 that acquires information on the direction of the back of the hand is provided. Then, the processing unit 14 detects the user's finger from the image acquired by the camera 101 and displays it on the first display 160 by superimposing it on the image of the second display 2.

Although the camera 101 is used for these gesture inputs, the distance measuring sensor 102 may also be used. By acquiring the information from the distance measuring sensor 102, for example, it becomes possible to more accurately acquire that the user's finger is at the position of the second display 2.

Also, gestures are not limited to those for displays. For example, the back of the hand, palm, fingers, or part of the face such as the user's chin, cheeks, ears, forehead, head, hair, etc., with or without the user's wearable device 1. , The gesture to other parts of the body such as the user's thighs and chest may be acquired. Other general gestures such as hand shape, fingertip trajectory, line of sight, blinking, facial expression, etc. may be used.

Of course, the input can also be performed by methods other than gestures. An input method recognized as a general user interface may be used in combination with a method other than this gesture. For example, various situations may be read by using the above-mentioned physical button 104, acquiring a biological signal by a microphone 105, or another sensor.

FIG. 20 is a diagram showing an example of processing by gesture. In the above-described embodiment, an example of changing the area of the second display 2 displayed on the first display 160 depending on the position of the wearable device 1 has been described. Not limited. For example, as shown in FIG. 20, it is also possible for the user to make a gesture of pinching the second display 2 with a finger to move the second display 2. When moved as shown in FIG. 20, the content displayed on the first display 160 is converted as shown in FIG. 8 by, for example, the processing unit 14.

For example, when the second display 2 is being picked up by hand, the feedback processing unit may continue to make a predetermined sound or vibrate, or the user may be notified by feedback that the second display 2 has transitioned to the pinched state. You may notify.

FIG. 21 is a diagram showing an example of processing by gesture. Of course, as shown in FIG. 21, it is also possible to perform a process of moving the second display 2 closer to or further away from the user with a gesture. In this case, the content displayed on the first display 160 by the processing unit 14 is converted as shown in FIG.

These transformations, for example, can be accommodated by changing the fixed value ^W _V M in the formula (13).

The user can end the movement of the second display 2 at any position. The ending gesture may be realized, for example, by transitioning from a pinched state to another state.

In this way, it is also possible to change the position of the second display 2 from a predetermined position. In this case, if the second display 2 is too close to the body, the content may be difficult to see. Therefore, the processing unit 14 may prevent the second display 2 from approaching within a predetermined distance. When the second display 2 reaches a predetermined distance, the feedback processing unit may give a predetermined feedback notification. Further, the distance of the second display 2 once set may be saved in the storage unit 12 so that the same setting is started at the next startup. In this case, the next startup, the processing unit 14 is stored in the storage unit 12, by reading the ^W _V M of the updated state, the first display 160 to properly second image of the display 2 such as It becomes possible to output.

The first display 160 and the second display 2 depend on, for example, the position of the wearable device 1, but the position of the wristwatch-type wearable device 1 differs depending on the individual. Therefore, right-handed and left-handed gesture authentication may be executed. Further, the orientation of the first display 160 may be such that it can be oriented outward or inward. For example, the first display 160 may be projected on the back of the hand when facing outward, and the first display 160 may be projected on the palm when facing inward.

The feedback when performing a gesture on the second display 2 may be various measures as well as the above.

For example, as tactile feedback, the actuator 162 may pulsately vibrate when in contact with the second display 2. Then, if the contact continues, it may be continuously vibrated. Further, the actuator 162 may notify by feedback a predetermined vibration indicating that the gesture has been accepted.

Of course, this feedback may be an auditory stimulus, that is, a sound output from the speaker 161 or a visual stimulus, that is, a light emission from the light emitting element 163. For example, feedback corresponding to various gestures can be executed by changing the tone color, frequency, sound length, and the emission color, intensity, interval, and the like.

As described above, various forms of the real display and the virtual display have been described in the present embodiment. As described in the present embodiment, the real display and the virtual display are not limited to the flat display, and can take various forms. Further, when the virtual display is touched, the contact state is fed back to the user, so that the operation of the display that does not actually exist can be realized more efficiently.

(Fifth Embodiment)
In each of the above embodiments, the first display 160 is intended to display a part or all of the area of the second display 2, but is not limited to this. For example, the first display 160 can display its own independent content directly on the first display 160 instead of browsing the virtual display like a normal portable device with a display. be.

The wearable device 1 may have a function of switching the display on the first display 160 between independent content, an image on the second display 2, and the like. For example, when exercising, the user may explicitly switch manually, or the processing unit 14 may automatically switch based on the sensing information from various sensors of the input unit 10. In addition, the wearable device 1 may display independent content when the user is in a situation where it is difficult to move or is tired. This setting can also be freely selected by the user.

The wearable device 1 may have a mode for displaying an image or the like of the second display 2 on the basis of displaying independent contents in the display of the first display 160. On the contrary, the wearable device 1 may have a mode of displaying independent contents on the first display 160 based on displaying the second display 2 in the display of the first display 160. That is, the mode that is the default as a function of the device may be the independent content display mode of the first display 160 or the content display mode of the second display 2.

Alternatively, these two modes may be completely independent of each other and may be related to being executed in parallel. In this case, the user may be able to select which mode is set as the default mode.

These display modes may be switched automatically based on the user's context as described above. This switching is performed, for example, when the processing unit 14 determines that the user is in a relaxed state at home based on the sensing information of the inertial sensor 100, the virtual display display mode, the user is on the train, or the like. If it is determined that the user is moving, the mode may be switched to the independent content display mode. In addition, when the processing unit 14 detects a context such as walking, running, muscle fatigue, or eye strain, based on the sensing information of the input unit 10. , Each may perform a process of switching to an appropriate display mode.

(Sixth Embodiment)
In each of the above-described embodiments, the wearable device 1 has been described as basically a wristwatch-type device, but the content of the present disclosure is not limited thereto. The wearable device 1 may be, for example, a smartphone, a tablet terminal, a ring terminal, a game machine, a digital camera, a glasses-type device, or the like.

In the wearable device 1 which is not a wristwatch type, the constraint condition used for processing the gesture of the processing unit 14 may not be used. Therefore, there may be a discrepancy between the image of the second display 2 displayed on the first display 160 and the position and posture of the first display 160. In order to avoid this, another constraint condition may be used to set the parameters of each mathematical expression. This may be preset as a parameter specific to the wearable device 1. Further, as another example, the parameters may be reset at the time of initial startup or by periodically performing calibration or the like. Further, as another example, the usage method such as how to hold the wearable device 1 may be restricted.

For example, in the case of a smartphone-type device, the way of holding the wearable device 1 may be restricted. Further, the individual user may perform the calibration by grasping or moving the device according to the instruction of the device at the time of initial startup, periodically or by the user's voluntary operation.

For example, in the case of a ring type or a spectacle type, it is possible to set many different restraint conditions from the case of wearing it on the arm. Based on these constraint conditions, the parameters in the above-mentioned mathematical formulas and the like may be set in advance. Of course, even in this case, the calibration may be performed periodically or voluntarily at the time of initial startup.

In the case of a ring-type device, the first display 160 may be provided in the wearable device 1 itself, or the first display 160 may be a projection-type display. The projection destination may be any place, for example, a desk, a wall, another flat surface, an unmounted back of the hand, or a palm. Of course, as described above, it may function as a light field display or a retinopathy-type display.

In the case of a spectacle-type device, the lens of the spectacles may be used as the first display 160. The lens used as the first display 160 may be one eye or both eyes. When both lenses are used as the first display 160, an image or the like that can be viewed stereoscopically may be output based on the parallax of both eyes. Further, as another example, different contents may be output.

When projecting the second display 2 onto the lens as the first display 160, it may be used completely as a display only by blocking what is visible in the background of the lens, or without blocking what is visible in the background of the lens. The image actually seen through the lens may be superimposed on the image of the second display 2.

In order to effectively utilize at least one of the movements of the muscles around the eyes such as the line of sight and blinking or the movement of the eyeball as a gesture, the frame part is equipped with one or more cameras 101 capable of acquiring the state of the eyes and the area around the eyes. You may. Further, various user states may be acquired by other sensors described in the above-described embodiment, such as the ranging sensor 102. For example, a sensor that can acquire brain waves because it is attached to a place close to the head and can detect the myoelectric state of facial muscles may be provided at an appropriate place, for example, a temple or a pad.

As another example, it may be configured as a first display 160 of a type that projects from a frame or the like to the back of the hand or the palm of the hand.

Gestures are not limited to eye movements, and may be night movements such as hands, as described above. In this case, a camera 101, a distance measuring sensor 102, or the like capable of acquiring the movement of a hand or the like may be appropriately provided in a place or the like where the outside is acquired from the frame.

Also, the wearable device 1 does not stop at having only one fixed shape. For example, the wearable device 1 may be a device capable of transforming a wristwatch type and a spectacle type. As another example, for example, it may function as a lens-type wearable device 1, and can be used as a wristwatch-type if it is attached to an arm band, or as a spectacle-type if it is attached to a spectacle-type wearer. .. Further, it may be a device that can be transformed into a wristwatch type, a spectacle type, or a plate type.

Further, as the first display 160, a display that can be folded or rolled may be used. In this way, by using the wearable device 1 whose shape can be physically deformed, it is possible to further improve usability, for example, the user can deform and use the wearable device 1 according to the situation.

(7th Embodiment)
In the above-described embodiment, the contents of the general processing have been described with specific examples, but in the present embodiment, more specific other use cases will be described.

Specific examples of gestures and the like are given below, but the contents of the present disclosure are not limited to these. For example, what gesture is used as long as the appropriate processing is executed for a predetermined gesture that can be executed by the user, can be appropriately sensed by the input unit 10, and can be appropriately detected by the processing unit 14. There may be.

In addition, each gesture or the like can be set or calibrated by the user. For example, in the case of a gesture in which blinking is continuous, the user may be able to set the blink interval, the number of blinks, and the like.

(1st example)
The wearable device 1 may execute an application optimized for SNS (Social Networking Service) by the processing unit 14. For example, when a notification arrives, the actuator 162 vibrates in a predetermined manner to notify the user that the notification has arrived. When the arm is raised, the content of the notification, for example, the content of the message is displayed on the second display 2 and can be viewed by the user through the first display 160.

Enter by gestures, such as moving your arms or moving your face. For example, "Like" may be sent to the message by inputting a gesture such as smiling or raising the thumb while the message is displayed.

You may prepare another gesture to reply to the message. For example, a keyboard mapped to the thumb of the hand wearing the wearable device 1 may be touched with the index finger to enable flick input.

In this case, the wearable device 1 may be provided with, for example, a camera 101 capable of photographing the thumb on the wearing side, or may be in a form in which minute finger movements can be acquired by the myoelectric sensor 107. Further, the keyboard mapped to the back of the hand on the wearing side may be in a form in which input can be performed by flicking with the finger of the other hand. Another predetermined gesture may be given to the transmission of the message after the character input is completed.

(2nd example)
The wearable device 1 may receive blinking as a gesture input. The second display 2 may be transformed into a cylindrical display centered on the user by blinking the user a predetermined number of times, for example, three times in a row. For example, this cylindrical display has applications lined up on its surface towards the user. The cylindrical display may rotate around the user by touching the surface of the cylinder and swiping, flicking, or the like.

The user starts or displays the application by inputting gestures such as blinking, arm movement, facial expression, etc. to the application to be started or displayed. For example, the application may be displayed by the user looking at the application and blinking twice. After displaying the application, the second display 2 may be a flat display.

(3rd example)
The wearable device 1 may be capable of launching a web browser optimized for gesture browsing. For example, when the web browser is placed, the second display 2 is transformed into a vertically long rectangle. The second display 2 may be movable by moving the arm and face, or inputting a gesture or the like to be picked by the hand of the person who does not wear the wearable device 1.

Further, the processing unit 14 may perform scroll display of the content by inputting a command to watch the scroll button. This scroll display may be implemented as a process of moving the area of the second display 2 displayed on the first display 160, or may be implemented as a process of scrolling the display of the second display 2. Further, the processing unit 14 may execute a process of increasing the character size or partially enlarging the character size when the gesture of squinting is detected via the input unit 10.

(4th example)
It may be an implementation that launches a music playback application by making a gesture that touches the ear with a hand that is not wearing the wearable device 1. In this case, the processing unit 14 may use the second display 2 as a ring-shaped display. The process of turning the annulus may be executed by making a gesture of turning the fingers of the hand on which the wearable device 1 is attached. By the process of turning this annulus, the song to be heard may be searched from the playlist, or it may be a volume operation. A gesture input for playback may be prepared separately.

Then, by making a gesture that pays for the second display 2, the actuator 162 may perform tactile feedback to minimize this application or go to background playback.

(5th example)
Here are some examples of applying the wearable device 1 as AR or VR.

The second display 2 is prepared as a display that surrounds the entire circumference of the user. Launch the map application and set your destination. When you start walking, the application may transform the second display 2 into the shape of an arrow-shaped 3D widget that points you in the direction you are walking to your destination.

In this case, not only when walking to the destination, but also when using transportation such as a train, the first display 160 is shown to show the way to the station instead of the way to the destination on foot. It may be displayed.

For example, the map may be superimposed on the arrow on the second display 2.

As another example, the second display 2 exists as a shape that transmits the shape of the arrow, and the processing unit 14 displays the actual rear view of the first display 160 and the deformed second display 2. It may be superimposed and displayed on the first display 160. In this case, the wearable device 1 has, for example, a camera 101 that acquires video information on the side opposite to the display surface of the first display 160. That is, the user can seamlessly display the view of the real world and the output of the first display 160 as if the first display 160 of the wearable device 1 had no wrist in the area. Therefore, the user can see an image showing the direction to go in the scenery, and can use a seamless navigation system.

Further, the time to the destination (including the waypoint) may be displayed on the background and the display of the arrow. As another example, after arriving at the station, the fare is displayed, the number of the platform on which the target train can be boarded is displayed, or the latest arrival, the time of the departing train, etc. are displayed. You may. The time display may be, for example, the remaining time until arrival and departure.

(6th example)
The wearable device 1 may operate as a translator. For example, when a user sees a foreign language that he / she does not understand, he / she can see the result of translating the foreign language into Japanese by pointing the wearable device 1 at that side. The processing unit 14 displays the translated result on the second display 2, and outputs the translation result to the user by displaying the display of the second display 2 superimposed on the background display on the first display 160. May be good.

As another example, instead of displaying the foreign language through see-through, the display of the native language may be displayed on the foreign language with the foreign language area filled in. By inputting a gesture such as a tap on the display in the native language, the display may be read aloud in the native language.

Conversely, translation into any language may be performed by holding the wearable device 1 over the mother tongue. In this case, if a gesture such as tapping is performed on the translated text, the translated text may be read aloud by voice.

(7th example)
Similar to the fifth example above, the wearable device 1 in which the second display 2 covers the entire circumference of the user is attached. By activating the remote camera application, for example, an image via a robot at home is displayed on the second display 2. The robot is equipped with, for example, a sensor capable of acquiring 360-degree video information, sound information, and the like.

By moving the arm or face, the area of the second display 2 displayed on the first display 160 may be moved in the direction corresponding to the movement of the arm or face. That is, by moving the arm or face, the state of the house displayed on the second display 2 via the first display 160 can be acquired as an image while maintaining physical and spatial consistency.

The robot may be able to move, for example, a flow line determined according to a user's instruction via the wearable device 1 or an arbitrary flow line. The robot can move between rooms, for example. The user's instruction may be analyzed, for example, a gesture such as an arm or a face, or a voice input, and the result may be transmitted from the wearable device 1 to the robot via the communication unit 18.

For example, it is possible to monitor the state of the home, the state of the pet, etc. while they are away.

Further, as another example, the remote-controlled robot may be provided as a monitoring device for remote care instead of at home. In this case, the robot moves in the house to be cared for, senses various information, and outputs it to the user via the virtual space projected by the second display 2. Of course, not only video but also sound information may be output. The user can check the status of the long-term care target. In this case, the target that is not preferable to be displayed for security reasons may be processed so as not to be displayed on the robot side, communicated, and displayed on the second display 2 of the wearable device 1.

The user may cause the robot to perform an action other than movement via the wearable device 1. For example, the voice of the user may be output from the robot. Conversely, the voice detected by the robot may be output from the speaker 161 of the wearable device 1. In this way, it may be possible to interact with another interactive user or a remote environment via a robot existing in a remote location.

In either case, the implementation may be such that the wearable device 1 can access other devices around the robot. For example, the screen of a PC, a smartphone, a tablet terminal, or the like registered in the robot in advance may be displayed on a part of the second display 2 of the wearable device 1. By viewing the second display 2 through the first display 160, the user can view the contents displayed on the device around the robot or operate the device.

(8th example)
Next, an example in which a plurality of devices operate in cooperation with each other will be described.

The remote desktop application may be configured so that the contents of the monitor of a remote PC can be acquired by using the wearable device 1. For example, a PC in a company is accessed by VPN (Virtual Private Network) or the like, and the monitor of the PC or the display area of the video signal output from the PC is matched to the shape and size of the display area of the second display 2. The shape and the like may be changed.

Then, the processing unit 14 processes the PC so that the user can operate the PC in a part or the whole area of the second display 2 displayed on the first display 160 via the remote desktop application. You may do it. For example, with such an implementation, it is possible to check documents and the like stored in the company's PC from the wearable device 1.

Subsequently, the shape of the second display 2 may be changed by disconnecting from the PC and accessing another device. For example, disconnect the wearable device 1 from the PC and establish communication to access the tablet terminal at home. The processing unit 14 changes the shape of the second display 2 according to the device to be accessed. For example, the shape of the second display 2, which was the shape of the display area of the monitor of the PC, changes to the shape of the display area of the tablet terminal. The user can execute the operation of the tablet terminal from the wearable device 1. Then, after returning to the home, it is possible to execute the continuation of the operation performed through the wearable device 1 by actually operating the tablet terminal.

(9th example)
A video or other signal may be notified to the wearable device 1 in synchronization with a TV or the like in a remote place. For example, when the home TV is turned on, the notification may be sent to the user via the wearable device 1. Then, the second display 2 may change the shape of the display area of the TV and display the contents displayed on the TV in real time. Images that conflict with each right under copyright law may not be transmitted to the wearable device 1. In this case, information about which channel or what kind of content is displayed may be displayed on the second display 2 side.

Similar to the above, it is also possible to operate the TV from the wearable device 1. For example, it is possible to change the channel, and when the game is running for a long time, a process of turning off the power of the TV may be executed.

(10th example)
Instead of using the plurality of devices by one user as described above, conversely, at least a predetermined area of one second display 2 may be shared by the plurality of users. In such a form, for example, a conference system using an object in a virtual space can be considered.

For example, a plurality of users may wear the wearable device 1 and display the shared virtual space on the second display 2. When one of them uses gesture input to pinch a 3D object in the virtual space and move it, the object in the second display 2 of another user who shares and browses the virtual space also Moving. Similarly, transforming an object also transforms the object in the second display 2 of another user. In this way, it is also possible to share 3D objects with other users over the network. Of course, the display of the second display 2 can be viewed by an individual user from a desirable position and posture via the first display 160.

Further, one user may move an object from another virtual space in his / her second display 2 to a shared virtual space. In this case, other users can share objects that did not exist in the virtual space that they have shared so far.

The above assumes, for example, a case where a plurality of people are pointing the wearable device 1 toward an empty space in the same conference room. The other user may be displayed in the second display 2 by grasping the relative position of the other user through the camera 101 provided on the opposite side to the user of the wearable device 1. Further, as in some of the above-described embodiments, it is also possible to perceive the VR space existing as the second display 2 as AR through the first display 160.

Furthermore, this conference may be held via a network. In this case, the position where each user is located may be virtually set based on the direction of the wearable device 1 and the like. Further, each user may be implemented so that each user virtually exists in the virtual space by designating a seat in the virtual space. In this case, the plurality of users do not have to exist at different positions, but may exist at the same position or at overlapping positions. In this case, when one user operates the object, the state of the object can be viewed from the second display 2 through the first display 160 from the same angle.

(Example 11)
Next, a use case for expanding visual ability using a virtual display will be described.

The wearable device 1 can display, for example, an application for reading a book on the second display 2. In such an application, the characters are enumerated, and depending on the eyesight, the user may not be able to read the characters accurately. For example, a myopia user can read nearby characters but has difficulty reading distant small characters, and a hyperopic user can read distant characters but can read nearby small characters. Is difficult.

Therefore, the second display 2 may display based on the visual acuity and output characters or the like to the first display 160 based on this display. For example, the wearable device 1 may include a light emitting element of a type that directly irradiates the retina with light as the first display 160. By providing such a first display 160 and a second display 2, characters and the like can be appropriately output to the user by appropriately outputting based on the user's eyesight and the distance between the eyes and the wearable device 1. Is possible. For example, the processing unit 14 sets an appropriate display based on the user's visual acuity to the second display 2, calculates the optimum light beam for the characteristics of the visual acuity based on the display of the second display 2, and displays the first display 160. It may emit light.

It can also be applied not only to just a reader application, but also to content such as a web browser that contains characters or contains detailed information.

(12th example)
The wearable device 1 can also operate as a night-vision camera. For example, an infrared camera may be provided as the input unit 10 in the darkness where the colors cannot be identified or the light is hardly perceived. The processing unit 14 may display the surrounding environment acquired by the infrared camera on the second display 2.

By holding the wearable device 1 in the desired direction in the dark, the user can display the dark-sighted image displayed on the second display 2 on the first display 160.

Also, instead of infrared light, the wearable device 1 may be equipped with a camera and an application that captures ultraviolet light. In this case, for example, an image considering the intensity of ultraviolet rays in the daytime is displayed on the second display 2, and the wearable device 1 is held over the range desired by the user to display the state of ultraviolet rays on the first display. It can be sensed through 160. In this case, it is also possible to superimpose the detection display of the ultraviolet rays displayed on the second display 2 and the image of shooting the same range and display them on the first display 160.

In either case, the sensor may include a light emitting element that not only senses light outside visible light but also irradiates light outside visible light. Then, the reflected light of the light emitted from these light emitting elements may be sensed by the sensor. However, as a thermograph that acquires infrared rays, such as a normal thermal sensor, it may be usable for situations other than night vision. When used as a thermograph, not only infrared rays but also ultraviolet rays may be acquired together.

In this way, the wearable device 1 can be used for various purposes by receiving light outside the visible light region invisible to humans and arbitrarily superimposing it on the actual background using the second display 2. It will be possible.

(13th example)
In the above, infrared rays and ultraviolet rays have been described, but the present invention is not limited to this. For example, depending on the light receiving element, there is also a sensor having a resolution of brightness finer than that of the human eye. By combining such a high-sensitivity sensor and a sensor having a lower sensitivity than a high-sensitivity sensor, the dynamic range may be corrected.

For example, the processing unit 14 may display an HDR (High Dynamic Range) image on the second display 2 based on the sensing information acquired from the camera 101. Then, the HDR image in the direction in which the user holds the wearable device 1 may be displayed on the first display 160. In this way, it is possible to easily convert only the range desired by the user into an HDR image by using the second display 2 and the first display 160.

(14th example)
The wearable device 1 may include a telescope application. For example, the camera 101 may acquire a telephoto image optically or digitally in the processing unit 14, and the processing unit 14 may display the telephoto image on the second display 2. The camera 101 is provided, for example, to photograph the side opposite to the display surface of the first display 160 of the wearable device 1. By holding the wearable device 1 in the direction in which the user wants to see far away, the first display 160 can display an image or the like acquired through a telescope.

Further, for example, when an image of the user's eyes is acquired by an in-camera and the user makes a gesture of squinting, the processing unit 14 makes the focal length farther or the angle of view more. It may be narrowed so that even a farther image can be accurately acquired. Then, by displaying on the second display 2, the user may acquire a farther image through the first display 160.

Further, as another example, a plurality of display surfaces according to the distance may be virtually set as the second display 2, and images and the like on each surface may be displayed. Upon detecting a gesture such as squinting the user, the processing unit 14 may switch the second display 2 projected on the first display 160 to output an image having a different focal length.

(15th example)
Similar to the telescope described above, the wearable device 1 may include a microscope application. The implementation is similar to the telescope application described above. In a microscope application, the user may squint to output a magnified image.

(16th example)
The wearable device 1 may be used for both a wristwatch type and eyewear. For example, when exercising, it may be used as a wristwatch-type wearable device 1, and when it does not move violently, it may be deformed and used as eyewear. For example, in a wristwatch-type wearable device 1, it may be deformed into eyewear that can be worn on the ear by removing it from the arm and bending the band.

By deforming in this way, it can be used as a wearable device 1 that requires mobility during sports, and can be used as eyewear when concentrating or when there is no vigorous movement. Of course, this wearable device 1 can also implement any of the above-mentioned applications, modules, and the like.

The user interface may be appropriately changed according to the mode of the form factor and the like.

(17th example)
Wearable device 1 may be used in a situation with many people. In such a case, the content displayed on the first display 160 of the wearable device 1 may be seen by other people. There is also the possibility that the wearable device 1 will be used by other people.

By providing an in-camera, it is possible to use the wearable device 1 more securely. For example, the user's face information, for example, the feature amount information is stored in advance in the storage unit 12 of the wearable device 1 so that face authentication can be performed. This certification may use a trained model trained by machine learning.

Then, when a plurality of humans are sensed by the in-camera, the processing unit 14 estimates the direction in which the user is from the plurality of humans based on the information acquired by the camera 101. This estimation may also use a trained model.

Then, the first display 160 may be made to emit light or the like so that the content displayed on the second display 2 is output to the user. The first display 160 may be, for example, a display capable of projecting with directivity. By using such a display, the user can view the content, but a person around him / her can view the content. From now on, it is possible to display the contents such as the video displayed on the second display 2 so that they cannot be viewed.

The above-described embodiment may have the following embodiments.

(1)
With the processing circuit
1st display and
Equipped with
The processing circuit is
Set up a virtual second display,
A part or all of the video signal virtually projected on the second display is displayed via the first display based on the position and orientation of the first display with respect to the second display.
device.

(2)
With more sensors,
The processing circuit is
Based on the sensing information of the sensor, the first display displays a part or all of the video signal virtually projected on the second display.
The device described in (1).

(3)
The sensor is an inertial sensor and
The processing circuit is
When the inertia sensor detects that the posture has been determined, the display of the first display is started.
The device described in (2).

(Four)
The processing circuit is
Detecting that a predetermined gesture has been input by the inertia sensor, the display of the first display is started.
The device described in (3).

(Five)
The sensor is a camera.
The device according to any one of (2) to (4).

(6)
The processing circuit is
Based on the information acquired by the camera, the user's face or the user's line of sight is recognized, and the content displayed in a part or all of the area of the second display is displayed on the first display.
The device described in any of (5).

(7)
The sensor is a touch panel attached to the first display.
The device according to any one of (2) to (6).

(8)
The processing circuit executes a predetermined processing based on the gesture acquired by the sensor.
The device according to any one of (2) to (7).

(9)
The processing circuit is
Based on the gesture information sensed by the sensor, the display of the first display is started.
The device described in (8).

(Ten)
The processing circuit is
The area or posture of the second display displayed by the first display is changed based on the gesture information sensed by the sensor.
The device according to (8) or (9).

(11)
The processing circuit is
The gesture sensed by the sensor is displayed on the first display.
The device according to any one of (8) to (10).

(12)
The processing circuit is
An avatar is attached to the gesture detected by the sensor and displayed on the first display.
The device described in (11).

(13)
The gesture is a gesture for the first display.
The device according to any one of (8) to (12).

(14)
The gesture is a gesture for the second display.
The device according to any one of (8) to (13).

(15)
The processing circuit is
The position of the second display with respect to the first display is changed based on the gesture.
The device according to any one of (8) to (14).

(16)
A feedback processing unit that executes feedback processing for the gesture.
The device according to any one of (8) to (15), further comprising.

(17)
The feedback processing unit includes a vibrator.
The device described in (16).

(18)
The feedback processing unit includes a speaker.
The device according to (16) or (17).

(19)
The sensor is a microphone
The processing circuit is
A predetermined process is executed based on the voice information of the user sensed by the microphone.
The device according to any one of (2) to (18).

(20)
The display of the first display changes based on the motion parallax due to the relative movement with respect to the second display when the user or the first display moves.
The device according to any one of (2) to (19).

(twenty one)
It also has a communication unit that sends and receives information to and from the outside.
The processing circuit is
A relative positional relationship with an external device is received from the communication unit, and the display on the first display is updated based on the positional relationship.
The device according to any one of (1) to (20).

(twenty two)
The first display is a two-dimensional display having a flat surface or a curved surface.
The device according to any one of (1) to (21).

(twenty three)
The first display is a virtual display having a shape of a part of the user's body that is virtually generated in the processing circuit, and uses a part of the user's body as a display surface from a light emitting unit having a projector function. ,
The device according to any one of (1) to (21).

(twenty four)
The first display is a three-dimensional display.
The device according to any one of (1) to (21).

(twenty five)
The second display is a three-dimensional display that projects a 3D object.
The device according to any one of (1) to (24).

(26)
The second display is defined between the user and the first display.
The device according to any one of (1) to (25).

(27)
The second display is defined farther from the user than the first display.
The device according to any one of (1) to (25).

(28)
The processing circuit is
A mode for displaying the content displayed on the second display via the first display, and
A mode in which the content is displayed directly on the first display, and
Can be switched,
The device according to any one of (1) to (27).

(29)
The device is a wearable device worn on the user's arm.
The device according to any one of (1) to (28).

(30)
The device is a spectacle-type wearable device.
The first display is provided in the lens of the eyeglass-type wearable device.
The device according to any one of (1) to (28).

(31)
The first display displays the content transparent to the viewpoint from the user, and then superimposes the content displayed on the second display.
The device according to any one of (1) to (30).

(31)
The second display is a display that projects virtual reality in a virtual space that can be accessed by a plurality of users.
With the device, multiple users experience the same virtual reality.
The device according to any one of (1) to (30).

(32)
The second display is a display that projects virtual reality in virtual space.
The virtual reality is reconstructed by the sensor detection information of another device located in a remote place.
The device according to any one of (1) to (30).

(33)
The processing circuit reconstructs the content displayed on the first display based on the user's visual acuity.
The device according to any one of (1) to (32).

(34)
The processing circuit displays an image having a converted dynamic range on the first display based on an environment of ambient brightness.
The device according to any one of (1) to (33).

(35)
The processing circuit displays a part or all of an image of the second display via a telescope on the first display.
The device according to any one of (1) to (34).

(36)
The processing circuit displays an image of a part or all of the second display via a microscope on the first display.
The device according to any one of (1) to (35).

(37)
The device is deformable,
The device according to any one of (1) to (36).

(38)
The processing circuit is
The first coordinate system relative to a predetermined point of the user wearing the device,
A second coordinate system based on the point at which the user wears the device,
A third coordinate system relative to the user's eyes,
A fourth coordinate system based on a predetermined point on the second display, and
The area and direction of the second display to be displayed on the first display are set based on the above.
The device according to any one of (1) to (37).

(39)
The processing circuit is
The position of the second display is virtually defined based on the predetermined point of the user.
The device described in (38).

(40)
The processing circuit is
The position of the first display in the second coordinate system is converted into the first coordinate system.
The position of the user's eyes in the third coordinate system is converted into the first coordinate system.
The area to be displayed on the first display when the second display is viewed from the user's eyes through the first display is calculated.
Displaying a part or all of the area of the second display on the first display,
The device according to (39).

(41)
The processing circuit performs coordinate transformation from the second coordinate system to the first coordinate system using Euler angles.
The device described in (40).

(42)
The processing circuit uses a quaternion to perform a coordinate transformation from the second coordinate system to the first coordinate system.
The device described in (40).

(43)
The processing circuit performs a coordinate transformation from the second coordinate system to the first coordinate system using a rotation matrix.
The device described in (40).

(44)
The processing circuit is
The position and orientation of the first display relative to the second display are calculated.
Based on the calculated position and orientation of the first display, the contents displayed on a part or all of the second display are projected and converted to the first display for display.
The device according to any one of (39) to (43).

(45)
A Cartesian coordinate system, a cylindrical coordinate system, or a spherical coordinate system is used as the coordinate system.
The device according to any one of (38) to (44).

(46)
Have the processing circuit perform the processing according to any one of (1) to (45).
Control method.

(47)
Cause the processing circuit to execute the process according to any one of (1) to (45).
program.

The embodiments of the present disclosure are not limited to the above-described embodiments, but include various possible modifications, and the effects of the present disclosure are not limited to the above-mentioned contents. The components in each embodiment may be applied in appropriate combinations. That is, various additions, changes and partial deletions are possible without departing from the conceptual idea and purpose of the present disclosure derived from the contents specified in the claims and their equivalents.

1: Wearable device,
10: Input section,
100: inertial sensor, 101: camera, 102: distance measurement sensor, 103: touch sensor, 104: physical button, 105: microphone, 106: heart rate sensor, 107: myoelectric sensor, 108: body temperature sensor, 109: sweating sensor, 110: Brain wave sensor, 111: Position sensor,
12: Memory,
14: Processing department,
16: Output section,
160: 1st display, 161: speaker, 162: actuator, 163: light emitting element,
18: Communication Department,
2: 2nd display,
W: 1st predetermined point,
R: 2nd predetermined point,
E: Third predetermined point,
V: 4th predetermined point

Claims (20)

With the processing circuit
1st display and
Equipped with
The processing circuit is
Set up a virtual second display,
A part or all of the video signal virtually projected on the second display is displayed via the first display based on the position and orientation of the first display with respect to the second display.
device. With more sensors,
The processing circuit is
Based on the sensing information of the sensor, the first display displays a part or all of the video signal virtually projected on the second display.
The device of claim 1. The sensor is an inertial sensor, a camera, a touch panel or a microphone.
The device according to claim 2. The processing circuit is
A predetermined process is executed based on the gesture acquired by the sensor.
The device according to claim 2. The processing circuit is
The area or posture of the second display displayed by the first display is changed based on the information of the gesture sensed by the sensor.
The device of claim 4. The processing circuit is
The position of the second display with respect to the first display is changed based on the gesture.
The device of claim 4. A feedback processing unit that executes feedback processing for the gesture.
4. The device of claim 4. The feedback processing unit includes a vibrator or a speaker.
The device of claim 7. The display of the first display changes based on the motion parallax due to the relative movement with respect to the second display when the user or the first display moves.
The device according to claim 2. The first display is a planar or curved two-dimensional display, a projector or a three-dimensional display that projects onto a part of the user's body.
The device of claim 1. The second display is a two-dimensional display having a planar or curved surface, or a three-dimensional display that projects a 3D object.
The device of claim 1. The processing circuit is
A mode for displaying the content displayed on the second display via the first display, and
A mode in which the content is displayed directly on the first display, and
Can be switched,
The device of claim 1. A wristwatch-type or eyeglass-type wearable device,
The device according to claim 1. The first display displays the content transparent to the viewpoint from the user, and then superimposes the content displayed on the second display.
The device of claim 1. The processing circuit is
The first coordinate system relative to a given point of the user wearing the device,
A second coordinate system based on the point at which the user wears the device,
A third coordinate system relative to the user's eyes,
A fourth coordinate system based on a predetermined point on the second display, and
The area and direction of the second display to be displayed on the first display are set based on the above.
The device of claim 1. The processing circuit is
The position of the second display is virtually defined based on the predetermined point of the user.
The device of claim 15. The processing circuit is
The position of the first display in the second coordinate system is converted into the first coordinate system.
The position of the user's eyes in the third coordinate system is converted into the first coordinate system.
The area to be displayed on the first display when the second display is viewed from the user's eyes through the first display is calculated.
Displaying a part or all of the area of the second display on the first display,
The device of claim 16. The processing circuit is
The position and orientation of the first display relative to the second display are calculated.
Based on the calculated position and orientation of the first display, the contents displayed on a part or all of the second display are projected and converted to the first display for display.
The device of claim 16. With the processing circuit
1st display and
It is a control method of a device equipped with
By the processing circuit
Set up a virtual second display,
A part or all of the video signal virtually projected on the second display is displayed via the first display based on the position and orientation of the first display with respect to the second display.
Control method. In the processing circuit
Set up a virtual second display,
A part or all of the video signal virtually projected on the second display is displayed through the first display based on the position and orientation of the first display with respect to the second display.
A program that runs the method.

Images