WO2019187167A1

WO2019187167A1 - Terminal display image display method

Info

Publication number: WO2019187167A1
Application number: PCT/JP2018/014029
Authority: WO
Inventors: 古賀信明
Original assignee: 株式会社スペシャルエフエックススタジオ
Priority date: 2018-03-31
Filing date: 2018-03-31
Publication date: 2019-10-03
Also published as: JP2019185076A

Abstract

[Problem] Whereas a display scrolling of a terminal such as a cellular phone has been carried out by detecting a movement with an acceleration sensor or the like, the application of such technology has not been possible in a moving vehicle or the like. [Solution] A three-dimensional measurement is carried out with a solid body in the vicinity of a terminal or a portion of a user's body as a reference solid body, and a position relation with regard to the terminal is continuously measured in realtime with the reference solid body treated as a "parent." With an arbitrary virtual reference plane determined by the user of the terminal treated as a "child," display data is virtually positioned in the virtual reference plane at an arbitrary position and size determined by the user, and the "child" is fixed to the "parent" in a three-dimensional position relation. When a display part of the terminal overlaps a space wherein the display data is virtually projected in parallel to the direction of a normal of the virtual reference plane, the display data which should be virtually projected in said display part at that position is displayed in realtime in the display part of the terminal, thereby enabling the user of the terminal to intuitively carry out a scrolling of an enlarged display screen.

Description

Display method of terminal display image

The present invention relates to terminal display control.

With the widespread use of mobile terminals, many people have come to use mobile terminals on a daily basis. However, the display density of display contents and display information has increased due to the higher functionality of mobile terminals.
Furthermore, users who feel that the display density is difficult to read are increasing due to the aging of users who read the display density.
The usual solution to this problem is simply to zoom in and scroll the mobile device screen, recognize the user's eyes with the camera, and enlarge the screen according to the distance between the user's viewpoint and the mobile device. Some applications changed rates.

JP2011-248811A JP-A-2005-251100 JP-A-10-254614

In order to make it easier to see the display of small characters on the display screen of a mobile terminal, the screen is most easily enlarged. However, it is a trade-off between downsizing and portability.
In addition, there is a method of enlarging and scrolling the screen of most mobile terminal display devices, but if you enlarge a small screen it will be easier to read, but only a part can be seen, so you can see the whole quickly. Will use scrolling by swiping, etc., so it may be necessary to scroll frequently depending on the enlargement ratio, and if you are unfamiliar with the operation, the intention of scrolling will perform other operations, etc. Intuitive and accurate operation will be difficult .

In addition, according to the methods described in References 1 to 3, the use of a mobile terminal while riding on a vehicle depends on whether the movement of the mobile terminal of the user or the movement of the vehicle is both Because it cannot be distinguished, it cannot be used in a moving vehicle.

Therefore, the terminal recognizes the relative positional relationship between the terminal and the user's periphery, fixes the virtual reference plane to an arbitrary position in the space around the user, and displays an arbitrary image at an arbitrary position and size of the virtual reference plane. Now we have devised a mechanism for virtual placement and viewing by holding the terminal in the virtual parallel projection display area of the image.

To solve the above-mentioned problem, it is not just the detection of the absolute movement amount, that is, the display control based on the movement amount detection amount of the mobile terminal by the acceleration sensor, etc., but it exists with the mobile terminal user space limited to the periphery of the mobile terminal As a solid parent, the relative positional relationship of the mobile terminal is detected, the virtual display area is set as “child” in the space, and “child” is fixed relative to “parent”. Provide a user-friendly display environment.

Specifically, it is obtained from a combination of one or more cameras built in the mobile terminal and sensors such as angular velocity, acceleration, infrared, ultrasonic, visible light, laser, radio wave, magnetism, GPS, or 3D depth sensor. Based on the information that is obtained, the 3D data in the space around the user with the feature points that do not deform with time, or the 3D data of the face or head or body part of the user using the mobile device, Remember.

The positional relationship between the three-dimensional data and the mobile terminal may be extracted in real time using a three-dimensional depth sensor from the beginning, but from the viewpoint of power saving and data saving, the three-dimensional sensor such as a three-dimensional depth sensor is used. First, the image feature point is extracted using the three-dimensional image data together, and the relative positional relationship between the portable terminal and the three-dimensional object is calculated in real time from the image sensor and the angular velocity sensor built in the portable terminal. Is good.
At this time, if the entire surrounding environment including the mobile terminal user does not move due to a vehicle or the like, an acceleration sensor may be added to the above.

After grasping three-dimensional three-dimensional information of a part of the mobile terminal user's own body or its surroundings and acquiring those images together, the relative positional relationship between the three-dimensional information and the mobile terminal is acquired in real time. Thus, the virtual reference plane can be fixed at an arbitrary location and size around the mobile terminal.

Virtual projection of data to be displayed such as a displayable image, video, text, etc. that the user wants to display on the mobile terminal in parallel with the normal direction of the virtual reference plane, and the virtual projection area and the display of the mobile terminal Part or all of the data to be virtually projected is displayed at the intersection of the surfaces.

In addition, the user said by this application refers to the user who uses a terminal or a portable terminal.
Since it is necessary to perform real-time processing, the minimum required data acquisition is desirable, but if the machine power is sufficient and the power consumption can be ignored, further information acquisition and analysis may be performed.

In addition, the real time referred to in the present application includes a delay (for example, about 10 msec) that does not interfere with practical use even if the reflection of the process result and the actual time are not strictly synchronized.

The text used in the drawings is cited from the author's own book "Nobuaki Koga, Analog Basic Course I (page 112)" of the present invention.

Although it seems that the portable terminal 001 of FIGS. 3, 4, and 5 does not have an image sensor or the like, it is omitted as an explanatory diagram of the display screen.

In the present specification, the description is mainly focused on the portable terminal, but the application of the present invention may be a terminal similar to a desktop personal computer or a notebook personal computer instead of the portable terminal.

Note that when acquiring three-dimensional data around the terminal in this specification or three-dimensional data of a part of the terminal user's body, it is not always necessary to move and measure the terminal.

According to the present invention, even if the user's space is a space that is moved by a vehicle or the like, after the user determines a virtual display area of an appropriate size, by holding the terminal in that area, When the portion corresponding to the virtual display content is displayed in the display (display) of the terminal and the terminal can be freely scrolled as a result of moving the terminal within the virtual display area. , The display is not scrolling, the fixed invisible display information is held over the display screen (frame) of the mobile terminal against a three-dimensional object around the mobile terminal user or a part of the user's own body, It feels like the information is displayed at the corresponding position, so it is very easy to see and there is little stress.

Also, by fixing the virtual reference plane to the mobile device user's head and face, move the terminal while gazing at the terminal without moving the head, or conversely, without moving the terminal, By changing the direction, you can scroll intuitively, even if you are a handicapped user.

It is explanatory drawing at the time of making a reference | standard solid ("parent") a portable terminal user's face. This is an example of the case where the mobile terminal is inclined and parallel to the virtual reference plane (“child”). It is an example of the display parallel-projected to a portable terminal with respect to a virtual reference plane (“child”). Similarly, it is an example of a display when the mobile terminal is tilted about the u axis. Similarly, it is an example of display when tilted in the u, v, and w axis directions. It is explanatory drawing of the imaging environment in which a three-dimensional analysis is possible when the angular velocity of an image sensor and the distance between feature points are known.

For example, information obtained by combining one or more built-in image sensors with sensors such as angular velocity, infrared, ultrasonic, visible light, laser, radio wave, magnetism, GPS, etc., or a three-dimensional depth sensor A program that allows a computer built in a mobile terminal to analyze 3D information around a mobile terminal or part of a mobile terminal user's body (3D modeling data generation) based on information obtained using a 3D sensor When,

Using a program that analyzes the positional relationship between the 3D modeling data generated using the information acquisition method and the terminal in real time, or using the 3D modeling data and an image sensor in combination, and using three or more feature points extracted, Using a combination of an image sensor and an angular velocity sensor, it has a program that analyzes the positional relationship with the 3D modeling data in real time,

On the virtual reference plane determined by the mobile terminal user based on the three-dimensional information obtained above, the mobile terminal user places an image, video, and text that can be displayed on the mobile terminal by the mobile terminal user at an arbitrary location. The data of the eyelid projected on the terminal display screen is displayed on the actual terminal display screen by intersecting the terminal display screen with the area where the data to be displayed is virtually projected in parallel with the normal direction of the virtual reference plane. Mobile terminal display device with a program.

The form described in this specification is one of the forms for carrying out the invention, and is not limited to this form as long as it matches the claims.

The application of the present invention is most likely to be a portable terminal, and since it can be scrolled by moving its face, it can also be used as a terminal operating means for people with reduced mobility. In this specification, an explanation will be given with an example of the portable terminal that most people will use.

First of all, the mobile terminal is one or more cameras built into the mobile terminal, or a position sensor that combines ultrasonic waves, lasers, radio waves, magnetism, infrared rays, visible light, acceleration, etc., or a three-dimensional depth sensor. The 3D spatial information of the mobile terminal and the user's periphery of the mobile terminal is acquired from the 3D sensor, and the 3D reference solid with the time axis obtained in this process is set as the “parent”.

At this time, 3D depth measurement using parallax if it is close to the mobile terminal, such as a part of the body of the mobile terminal user, time difference in which the light emitted from the mobile terminal is reflected and returned for things farther than the user, etc. You can use the three-dimensional depth measurement method based on the measurement method and the measurement method based on the measurement accuracy and power consumption.

In addition, the “parent” that is the reference in the three-dimensional space to be detected is determined to be a non-moving wall or window around the user or a part of the body such as the user's own face (head). This may be performed by the user himself / herself, or the computer inside the mobile terminal may automatically determine from the environmental conditions and learning results.

Next, the user determines the virtual reference plane by holding it over a position where the mobile terminal can be easily seen.
This virtual reference plane has a relationship of “child” fixed depending on the “parent”.

The user virtually arranges the display screen data at a desired position on the virtual reference plane, sets an easy-to-see size, and starts using it.
The 3D information of the “parent” and the location information of the mobile terminal are detected in real time, and unless the setting is changed, is the “child” always fixed to the “parent” based on the information? It moves dependently in real time like

The virtual display screen arranged by the user on the virtual reference plane is in a state of being virtually projected in parallel with the normal line of the virtual reference plane, and when the display screen of the mobile terminal intersects with the parallel projection area, The same data as the heel display screen data that is virtually projected onto the displayed part is displayed on the display screen of the terminal, and even if the mobile terminal is tilted with respect to the virtual reference plane, the virtual image projected in parallel on the display screen of the mobile terminal Since the display data is incident obliquely, the display screen looks distorted when viewed from the front with respect to the screen, but when viewed from the mobile terminal user, V20 in FIG. 2 and 4B and 5B in FIGS. Looks almost undistorted.

That is, when a white card with the same shape as the display screen of the terminal is inserted into the image projected in parallel by the projector, a part of the projected image is reflected on the card, and it is seen from the optical axis of the projector. Similar to the viewing state, as long as the observation is made from above the optical axis, even if the card is tilted with respect to the optical axis, the image shown on the card is deformed and cannot be seen.

When the face (head) of the user of the mobile terminal is “parent” as in 1A and 1B of FIG. 1, the virtual reference plane that is “child” in the virtual space is fixed to the face (head) of the user. It moves with the movement of the user's face (head) as if it were, so you can either fix the mobile device and change the orientation of the user's face (head), or move the mobile device if the user does not move The display contents are scrolled.

When the environment around the user is drastically changed, in an environment that hardly reflects light, or in a very dark environment, the light emitted from the display screen is used to illuminate the image sensor or use short-distance parallax. There is a big merit to make "parent" face for reasons such as improving accuracy.
In addition, when the wall or window around the mobile device user is set as the “parent”, the virtual display area is fixed in the space, so it is stable by moving the mobile device regardless of the posture and movement of the mobile device user. You can scroll the display content.

In a normal desktop or laptop computer instead of a portable terminal, a sensor equivalent to the portable terminal is built in, or an external three-dimensional sensor has a signal transmission means to the personal computer by wired or wireless connection, and those Analyzing the 3D information of the personal computer and its surroundings from the data and the 3D information of the PC user's body, for example, the 3D information of the face and head, and the positional relationship, based on the position of the personal computer user's face (head) The display location may be changed (scrolled).

Real-time detection of the positional relationship between the terminal and the terminal user requires that the power consumption of the sensor itself and the amount of data handled by it are greatly related to the battery drive time and the delay time due to image processing. is there.
In general, a three-dimensional sensor such as a three-dimensional depth sensor generally consumes more power than other sensors, and it is desirable to avoid using the device for a long time as much as possible, particularly in a portable terminal with limited power.

Therefore, we will explain the method of 3D analysis using a 2D image sensor and an angle sensor while minimizing the use of 3D sensors.
First, a three-dimensional solid that becomes a “parent” is acquired by using a three-dimensional depth sensor or the like together with the image data.
At this time, the part where the result is unstable during the three-dimensional measurement or the possibility of moving from the image data and the three-dimensional data determination is excluded.
Image data and 3D data judgment are collated with the library and past learning results, etc., and if it is recognized as wrinkled, uncollected hair, curtain-like, etc., there is a possibility of movement, Those that are recognized as flat surfaces such as walls, those that have a straight element, and those that have an angle close to a right angle may be determined not to move.

Subsequently, three or more feature points on the obtained “parent” image data are extracted as indicated by p1 to p3 in FIG. 6 and collated with the three-dimensional data matching the points.
At this time, the feature points that are easily discriminated by the image data and one of the three-dimensional data do not always match.
In that case, a plane is created using the three points of the three-dimensional data closest to the feature point on the image data that does not match the three-dimensional data, and the feature point of the non-matching image is at the center point on the plane. You may assume that there is.

In addition, for the same reference solid, an analysis using one image sensor, the three feature points, and the angular velocity may be performed at two or more locations and the average value may be used.
Further, with respect to the three feature points, disparity information obtained from each image sensor of two or more image sensors built in the terminal and having a fixed interval, and a triaxial angular velocity also fixed in the terminal. Three-dimensional analysis may be performed in the same manner as in the previous period using the angular velocity of each axis obtained from the sensor.
When the user determines the virtual reference plane and the display data to be arranged on the virtual reference plane, the measurement is switched from measurement using a three-dimensional sensor such as a three-dimensional depth sensor to analysis using an image sensor and an angular velocity sensor. At this time, the distance between the extracted feature points p1 to p3 and the image sensor is taken over, and is used as a starting point for starting the 3D analysis by 2D.

After that, as shown in 6B of FIG. 6, from the distance on the image formed by ip1 to ip3 on the image captured by the image sensor, the distance between the actual points, and the angular velocity of the uvw axis of the image sensor. The positional relationship between the “parent” and the terminal is calculated, and based on the result, the position in the space of the “child” that is the virtual reference plane 00 is fixed.

Hereinafter, a case where the user wants to read data in which a lot of text is written in an area larger than a portable screen such as a magazine will be described in more detail.
First, set the mobile device to the space measurement mode to obtain the 3D depth in the space where the mobile device is used, but before performing the 3D analysis, there is no possibility of movement in the interior or interior walls, windows, chairs, furniture, etc. The distinction between the interior and the human body or curtain that may move may be estimated from the actual image while referring to the learning results from the image, or set manually at the discretion of the user.
Next, three-dimensional measurement is performed in consideration of the time axis, it is determined whether it is actually moving, and the solid that is determined as “does not move” is set as the reference solid and is set as the “parent”.

You can continue to detect the positional relationship between the parent and the terminal using the 3D depth sensor, but feature point candidates are extracted from the image from the image sensor due to the power consumed by this process and the lightness of the data handled. On the other hand, it is practically desirable to estimate the position of the image sensor and the position of the feature point by corresponding the change in the three-axis angular velocity of the terminal incorporating the image sensor together with the time axis.
Even if it is determined that it does not move at the initial stage of measurement, a portion inconsistent with other data during use of position data is excluded as needed.
Therefore, it is desirable to list candidates that can be used as feature points on the reference solid as much as possible at the time of the first measurement.
The above measurement routine is repeated at any time and updated in real time to improve accuracy.

Needless to say here, images with lens distortion that causes contradiction in 3D analysis caused by the lens attached to the image sensor built in the mobile terminal are released by using image warping in the process before analysis. Perform three-dimensional analysis.

When the three-dimensional information that becomes the “parent” necessary for the virtual space display is obtained, the user sets the virtual reference plane 00 at an arbitrary place where it is easy to use.
Next, data such as text that the user wants to read is displayed at an arbitrary position and size on the display of the portable terminal.

When “parent” is set to a solid other than a part of the user's body, if you want to read the part that is off the display, you can read it by moving the mobile terminal in that direction, and you want to further enlarge or reduce the display It can be displayed by pinching out and pinching in the mobile terminal or by raising and lowering the mobile phone vertically with respect to the display surface.
In this case, it seems that it is easy to use a reduction when the terminal is vertically expanded with respect to the virtual reference plane 00 of the terminal, but conversely, depending on the user's intention, the virtual reference plane 00 with respect to the terminal. It may be set so that when it is vertically lowered, it is enlarged and when it is raised, it is reduced.

Instead of three-dimensional information around the terminal and the user, the positional information of feature points found on the user's eyes, eye corners, ears, glasses, or face, and the head, which are almost the same in the human body, is added. May be.
In particular, temporary changes in the placement due to changes in facial expressions, such as those seen on faces, often return to the original position after a certain amount of time. If there is no change in the positional relationship between the corners of the eyes and the eyes where there is no change in the relationship, even if the positional relationship between other feature points collapses, it can be judged as `` temporary disruption of the positional relationship due to facial expression changes '' good.
If a part of the user's body, for example, the head or face, is “parent” 000, the terminal display screen can be scrolled by moving the user's head, and conversely, the terminal can be scrolled by moving the terminal.

If the generation of the correct 3D space is unstable due to inadequate 3D data acquisition or a measurement environment that is not stable, smooth the path (trajectory) that moves the display so that the display on the mobile device does not become unstable. Also good.

The terminal computer may automatically determine what the positional relationship is not changed by the combined use of learned image recognition and stereoscopic recognition.
For example, “a solid with a shape of wrinkles” may move, “a flat surface without unevenness”, “an object with a 90 ° angle on acquired 3D data”, or an object with a straight outline May be judged as having a low possibility of movement.

The parts of the human body and the curtains that move by the wind, etc., in which the three-dimensional information in which the positional relationship between the point groups is fixed cannot be obtained are delicate, and their identification is very important for quick three-dimensional information acquisition.
In general, three-dimensional data with a smaller surface is more difficult to change with the passage of time than three-dimensional data with a large surface, for example, a human face is more likely to lose its fixed three-dimensional data due to changes in facial expressions. The shape is almost unchanged. In addition, it is preferable to reflect the fact that the three-dimensional shape of the knee, etc. when sitting down, even if it is a part of the same human body, hardly changes when acquiring the data as the “parent”.

When a user looks at a mobile device while walking, it is dangerous because it is judged as a “walking smartphone state” or the like because of the walking pattern obtained from an acceleration sensor, etc., or the small amount of information that is judged as a still life from one or more cameras. From the viewpoint of prevention, the display of the portable terminal may be interrupted when a message to that effect is displayed.

The most likely use of the present invention is in a portable terminal, in addition to the case where you want to see data larger than the display screen finely and quickly on a portable terminal,
Since it can be scrolled by moving its face, it can also be used as a terminal operation means for people who are handicapped.

000 Basic solid ("parent")
001 Mobile terminal 100 Virtual reference plane (“child”) and parallel projection area fixed relative to the reference solid (“parent”)
101 Portion where the display surface of the terminal intersects with the parallel projection area 00 The user's face or head (reference solid ("parent")) and the virtual reference plane ("child"")
10, 10 a, 10 b, 10 c, display example displayed on mobile terminal 11 state displayed on mobile terminal screen with high density 20 display example displayed on mobile terminal not parallel to virtual reference plane (“child”) V10 Display screens 10, 10 a, 10 b, 10 c viewed from the user V20 Display screens 20 viewed from the user V Line of sight 1 Line of sight that the user can move without moving the face 1A Virtual reference plane when the user looks at the front (“ Child ") and parallel projection area 1B Virtual reference plane (" child ") and parallel projection area when the user looks down 3A Perspective when projected from the virtual reference plane (" child ") on the display surface of the mobile terminal FIG. 3B is a view of a user's appearance when projected from a virtual reference plane (“child”) on the display surface of the 3B mobile terminal. 4A is a perspective view when projected from the virtual reference plane (“child”) on the display surface of the mobile terminal tilted 4A 4B from the virtual reference plane (“child”) on the display surface of the mobile terminal tilted 4B 5A is a perspective from the user when projected 5A is a perspective view when projected from a virtual reference plane ("child") on the display surface of the mobile terminal tilted in all three axes 5B A mobile terminal tilted in both the three axes 6A, which is the appearance from the user when projected from the virtual reference plane (“child”) on the display surface, and the point where the distance between the points that can be analyzed in three dimensions is known, and the angular velocity sensor and image sensor. 30 images (images) taken of points in 3D space with known distances p1, p2, p3 Points in 3D space with known distances between points ip1, ip2, ip3 Know the distance between the points Image data of points in the 3D space u, v, w Angle axis 30 Image sensor 301 Angle of view

Claims

In a terminal with a computer and a display device,
By a three-dimensional sensor built in the terminal or an external three-dimensional sensor connected to the terminal,
The terminal user or a solid around the terminal determined by the computer as a “parent” or a part of the terminal user's own body as a reference solid,
While performing 3D measurement in real time with the 3D sensor,
Using the three-dimensional measurement data, the “parent” corresponds to a virtual reference plane that is a “child” determined by the terminal user, and fixes the positional relationship between the “parent” and the “child”.
Virtually arrange the display image of the position and size determined by the terminal user on the virtual reference plane,
The display image is virtually projected in parallel with the normal direction of the virtual reference plane;
A terminal display method in which the display image of the eyelid virtually projected on a portion where the display screen of the terminal intersects is displayed on the display screen of the terminal.
In a terminal with a computer and a display device,
By a three-dimensional sensor built in the terminal or an external three-dimensional sensor connected to the terminal,
After performing a three-dimensional measurement of the solid around the terminal or a part of the terminal user's own body,
Obtained in real time from an image sensor built in the terminal based on information of a reference solid determined by the terminal user or the computer as a “parent” and a triaxial angular velocity sensor fixed to the image sensor Using 3D analysis
Using the three-dimensional analysis data, the “parent” corresponds to a virtual reference plane that becomes a “child” determined by the user, and fixes the positional relationship of the “child” to the “parent”.
Virtually arrange the display image of the position and size determined by the terminal user on the virtual reference plane,
The display image is virtually projected in parallel with the normal direction of the virtual reference plane;
A display method for a terminal, wherein the display image of the eyelid virtually projected onto a portion where the display screen of the terminal intersects is displayed on the display screen of the terminal.
A point in a three-dimensional space of three or more points that can be photographed, in which the distance between each point on the solid is known,
An image sensor that captures that point in real time,
A sensor that measures the three-axis angular velocity of the image sensor in real time,
Are synchronized on the time axis,
A method of detecting a positional relationship between the three or more points and the image sensor in a three-dimensional space from an image in which the three or more points are captured and an angular velocity of the image sensor that has captured the image.
Using two or more image sensors fixed in the terminal and three-axis angular velocity sensors fixed in the same terminal, the image data obtained by copying the three or more points on the reference solid by the two or more image sensors A method for acquiring in real time a positional relationship between the reference solid and the terminal in a three-dimensional space using the above parallax and angular velocity information obtained from the three-axis angular velocity sensor.