WO2018076609A1

WO2018076609A1 - Terminal and method for operating terminal

Info

Publication number: WO2018076609A1
Application number: PCT/CN2017/078581
Authority: WO
Inventors: 魏占婷
Original assignee: 中兴通讯股份有限公司
Priority date: 2016-10-27
Filing date: 2017-03-29
Publication date: 2018-05-03
Also published as: CN108008811A

Abstract

Provided in the embodiments of the present disclosure is a method for operating a terminal, the method comprising: collecting an image of a reference object in real time, the distance between the reference object and a terminal display screen exceeding a set value; analyzing the image of the reference object, to obtain, on the terminal display screen, a mapping point forming a preset mapping relationship with the reference object; generating a function instruction at the mapping point, and operating the terminal display screen on the basis of the function instruction. Further provided in the embodiments of the present disclosure is a terminal.

Description

Method and terminal for operating terminal

Technical field

The present disclosure relates to the field of human-computer interaction, and in particular, to a method and a terminal for operating a terminal.

Background technique

At present, the application range of a terminal with a touch screen becomes wider and wider; the touch screen on the terminal can be used as a device for realizing human-computer interaction. Illustratively, the resistive touch screen is actually a sensor, and its structure is basically It is a superimposed structure of film and glass. The adjacent side of the film and glass is coated with nano-indium tin oxide (ITO) coating. ITO has good conductivity and transparency. When a touch operation occurs, the ITO under the film contacts the ITO on the upper layer of the glass, and the corresponding electrical signal is transmitted through the inductor, sent to the processor through the conversion circuit, and converted into coordinate values on the screen (X, Y values). ), and complete the selected action and present it on the screen.

In the prior art, a touch response method, a device, and a wearable device of a wearable device are disclosed, so that the wearable device can feedback the touch operation effect to the user in real time, and improve the touch accuracy of the wearable device; The technical solution is: acquiring position information of the target fingertip collected by the binocular recognition device in the set touch action occurrence area; determining the mapping point of the target fingertip on the screen of the wearable device according to the position information of the target fingertip Location information; the cursor is displayed at the mapped point on the wearable device screen.

Summary of the invention

In order to solve the existing technical problems, embodiments of the present disclosure provide a method and a terminal for operating a terminal, which can implement an operation terminal without touching a screen with a finger.

In order to achieve the above object, the technical solution of the embodiment of the present disclosure is implemented as follows:

An embodiment of the present disclosure provides a method for operating a terminal, where the method includes:

Acquiring an image of the reference object in real time, the distance between the reference object and the display screen of the terminal exceeds a set value;

Performing an analysis on the image of the reference object to obtain a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object;

Generating a function instruction at the mapping point, and implementing an operation on a terminal display screen based on the function instruction.

Optionally, in the present disclosure, the reference object includes a pupil of a person;

Correspondingly, the image of the reference object is analyzed to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including:

Obtaining a spatial position of the fovea of the eye in real time, the pupil and the fovea of the eye being located in the same eye; and based on the image of the pupil, obtaining a spatial position of the central point of the pupil;

Based on the spatial position of the fovea of the eye and the center point of the pupil, the intersection of the line passing through the center point of the eye and the center point of the pupil is determined as a mapping relationship between the terminal display and the reference object. point.

Optionally, in the disclosure, the acquiring an image of the reference object in real time includes: separately acquiring the An image of the pupil;

The obtaining a spatial position of the pupil center point based on the image of the pupil includes: obtaining a spatial position of the pupil center point based on a spatial position of the two cameras and an image acquired by the two cameras.

Optionally, in the present disclosure, the acquiring an image of the reference object in real time includes: acquiring an image of the reference object in real time by using at least one camera;

Selecting at least one point in the image of the reference object as a reference point;

Determining the spatial position of each reference point based on the image of each reference point and the spatial position of each camera;

Determining a projection point of each reference point on the terminal display screen based on a spatial position of each reference point and a spatial position of the terminal display screen;

A mapping point that forms a preset mapping relationship with the reference object is determined based on the projection point.

Optionally, in the disclosure, the determining, according to the projection point, a mapping point that forms a preset mapping relationship with the reference object, includes: forming the projection point as a preset mapping with the reference object The mapping point of the relationship.

Optionally, in the present disclosure, the number of the reference points is two;

Determining, according to the projection point, a mapping point that forms a preset mapping relationship with the reference object, comprising: determining a projection point of the two reference points on the display screen of the terminal, and determining the connection of the two projection points The midpoint serves as a mapping point that forms a preset mapping relationship with the reference object.

Optionally, in the present disclosure, the reference object includes two eyes of a person;

Selecting at least one point in the image of the reference object as a reference point includes: using a pupil center point of the two eyes of the person as a reference point.

Optionally, in the disclosure, selecting at least one point in the image of the reference object as a reference point includes:

Determining a spatial location of the reference object based on an image of the reference object and a spatial location of each camera;

Based on the spatial position of the reference object, a point at which the vertical distance from the terminal display screen in the reference object is the smallest is used as a reference point.

Optionally, in the present disclosure, the acquiring an image of the reference object in real time includes: acquiring an image of the reference object in real time by using a camera;

Before analyzing the image of the reference object, the method further includes: acquiring a distance between the camera and the reference object in real time;

Correspondingly, determining the spatial location of the reference point based on the image of the reference point and the spatial position of each camera comprises:

A spatial position of the reference point is determined based on an image of the reference point, a spatial position of the camera, and a distance between the camera and the reference object.

Optionally, in the present disclosure, the reference object is an object having a minimum vertical distance from the display screen of the terminal, or the reference object is located in a human body.

Optionally, in the present disclosure, the reference object includes one eye of a person; the real-time image of the reference object is collected, Including: collecting images in the human eye in real time;

The analyzing the image of the reference object to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including: displaying the current display content of the terminal display screen and the collected human eyes The image matching area is determined as: a screen matching area; a point is selected in the screen matching area as a mapping point that forms a preset mapping relationship with the reference object.

Optionally, in the present disclosure, before analyzing the image of the reference object, the method further includes: determining a distance between the terminal display screen and the reference object;

The analyzing the image of the reference object to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including: when the determined distance is in the set interval, the reference object The image is analyzed to obtain a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.

Optionally, in the disclosure, the generating the function instruction at the mapping point comprises: determining that the mapping point is in a mapping point area on a terminal display screen; generating the mapping based on the determined size of time a function instruction at a point; the mapping point area includes an initial position of a mapping point on the display screen of the terminal.

Optionally, in the present disclosure, the generating, according to the determined size of the time, the function instruction at the mapping point, including:

When the determined time is in the first setting range, generating an instruction indicating that the current mapping point is clicked; when the determined time is in the second setting range, generating an instruction indicating that the current mapping point is long pressed; the determined time is at When the third setting range is set, an instruction indicating that the sliding operation is performed is generated;

No overlap is formed between the first setting range, the second setting range, and the third setting range.

Optionally, in the present disclosure, the first setting range is a range from a first time threshold to a second time threshold; the second setting range is greater than a second time threshold; the third setting The range is less than the first time threshold; the first time threshold is less than the second time threshold.

Optionally, in the disclosure, the generating an instruction to perform a sliding operation includes: acquiring a moving direction and a moving rate of the mapping point, and generating an indication to perform a sliding operation based on the moving direction and the moving speed of the mapping point. Instructions.

Optionally, in the disclosure, the generating, according to the moving direction and the moving rate of the mapping point, generating an instruction to perform a sliding screen operation includes:

The moving rate of the mapping point in the lateral direction of the display screen of the mobile terminal is used as the lateral moving rate of the mapping point, and the moving rate of the mapping point in the longitudinal direction of the display screen of the mobile terminal is taken as the longitudinal moving speed of the mapping point;

When the lateral movement rate of the mapping point is greater than the longitudinal movement rate of the mapping point, generating an instruction indicating that the horizontal sliding operation is performed; or, the lateral movement rate of the mapping point is greater than the longitudinal moving speed of the mapping point, and the horizontal direction of the mapping point When the moving rate satisfies the first setting condition, generating an instruction indicating that the horizontal sliding operation is performed;

When the longitudinal movement rate of the mapping point is greater than the lateral movement rate of the mapping point, generating an instruction indicating that the vertical sliding operation is performed; or, the longitudinal movement rate of the mapping point is greater than the lateral movement rate of the mapping point, and the vertical direction of the mapping point When the movement rate satisfies the second setting condition, an instruction is instructed to perform the vertical sliding operation.

Optionally, in the present disclosure, the first setting condition is: a lateral movement rate of the mapping point is within a fourth setting range; and the second setting condition is: a longitudinal movement rate of the mapping point is at a fifth Within the setting range.

Optionally, in the disclosure, the mapping point area includes an initial position of a mapping point on the display screen of the terminal; The area of the mapping point area is less than or equal to the set threshold.

Optionally, in the present disclosure, the mapping point area is a circular area centered on the initial position of the mapping point.

Optionally, in the disclosure, before the generating the function instruction at the mapping point, the method further includes: continuously performing image collection on the action of the user of the terminal, and obtaining an action image of the user; The motion image is image-recognized to obtain a recognition result;

Correspondingly, the generating the function instruction at the mapping point comprises: generating a function instruction at the mapping point based on the recognition result.

Optionally, in the present disclosure, the generating, according to the recognition result, the function instruction at the mapping point, when the recognition result is a blinking action, a mouth opening motion, or a mouth closing motion, generating an indication click mapping An instruction at a point; when the recognition result is a nodding action, generating an instruction to perform a downward screen operation; and when the recognition result is a head-up operation, generating an instruction to perform an upward sliding operation; the recognition result is shaking the left and right At the time of the action, an instruction is generated to instruct the lateral sliding operation.

Optionally, in the present disclosure, the function instruction at the mapping point is: an instruction to click a function instruction at a current mapping point, a function instruction to press and hold a current mapping point, or an instruction to perform a sliding operation.

An embodiment of the present disclosure further provides a terminal, including an image collection device and a processor;

The image capturing device is configured to collect an image of the reference object in real time, and the distance between the reference object and the display screen of the terminal exceeds a set value;

a processor configured to analyze an image of the reference object to obtain a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object; generate a function instruction at the mapping point, based on the function The instructions implement the operation of the terminal display.

The processor is further configured to acquire a spatial position of the fovea of the eye in real time; the pupil and the fovea of the eye are located in the same eye;

Correspondingly, the processor is arranged to obtain a spatial position of the pupil center point based on the image of the pupil; based on the spatial position of the eye fovea and the pupil center point, passing through the eye fovea and the pupil center point The intersection of the straight line and the terminal display screen is determined as a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.

Optionally, in the present disclosure, the image collection device includes at least one camera;

The processor is configured to select at least one point in the image of the reference object as a reference point; determine a spatial position of each reference point based on an image of each reference point and a spatial position of each camera; Determining a projection point of each reference point on the terminal display screen based on the spatial position of the point and a spatial position of the terminal display screen; and determining a mapping point that forms a preset mapping relationship with the reference object based on the projection point.

Optionally, in the present disclosure, the reference object includes one eye of a person;

The image capture device is configured to collect images in a human eye in real time;

The processor is configured to determine an area in the current display content of the terminal display screen that matches an image in the collected human eye as: a screen matching area; and select a point in the screen matching area as forming with the reference object The mapping point of the preset mapping relationship.

Optionally, in the present disclosure, the processor is further configured to determine a distance between the terminal display screen and the reference object before analyzing the image of the reference object;

The processor is configured to analyze an image of the reference object when the determined distance is in a set interval, and obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object.

Optionally, in the present disclosure, the processor is configured to determine a time when the mapping point is in a setting area; and generate a function instruction at the mapping point based on the determined size of the time.

Optionally, in the disclosure, the image collection device is further configured to continuously perform image collection on the action of the user of the terminal before generating the function instruction at the mapping point, to obtain an action image of the user;

The processor is further configured to perform image recognition on the motion image of the user to obtain a recognition result; and generate a function instruction at the mapping point based on the recognition result.

The method and the terminal for operating the terminal provided by the embodiment of the present disclosure collect an image of a reference object that is not in contact with the display screen of the terminal in real time; analyze the image of the reference object to obtain a reference image on the terminal display screen Forming a mapping point of the preset mapping relationship; generating a function instruction at the mapping point, and implementing an operation on the display screen of the terminal based on the function instruction; thus, analyzing the image of the reference object to obtain a display on the terminal The mapping point, and then based on the function instruction at the mapping point, completes the operation of clicking, long pressing, sliding screen, etc., realizes the non-touch screen operation terminal; can determine the mapping point of the fingertip, operate the mobile phone according to the fingertip trajectory, and can utilize The mapping point directly operates on the screen of the mobile phone; the user does not need to touch the screen with a finger, and the terminal can be operated only according to the image of the reference object; the present invention can solve the technical problem that the finger operating terminal caused by the size of the terminal display screen is large Effectively improve the efficiency of human-computer interaction, improve the ease of operation of the terminal, and enhance the user experience.

DRAWINGS

FIG. 1 is a flowchart of a method for operating a terminal according to an embodiment of the present disclosure;

2 is a first schematic diagram of a projection point of a reference point on a display screen of a terminal according to an embodiment of the present disclosure;

3 is a second schematic diagram of a projection point of a reference point on a display screen of a terminal according to an embodiment of the present disclosure;

4 is a flowchart of an implementation manner of a method for operating a terminal according to an embodiment of the present disclosure;

Figure 5 is a schematic view of the fovea and pupil of the eye in the embodiment of the present disclosure

FIG. 6 is a schematic diagram of a principle for determining a position of a space point by using a dual camera according to an embodiment of the present disclosure; FIG.

7 is a schematic diagram showing a positional relationship between a human eye line of sight and a terminal display screen according to an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of a terminal according to an embodiment of the present disclosure.

detailed description

The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the embodiments described herein are merely illustrative of the disclosure and are not intended to limit the disclosure.

The embodiment of the present disclosure describes a method for operating a terminal, which may be applied to a terminal, where the terminal may be a fixed terminal or a mobile terminal having a display screen; for example, the display screen may not have a touch response function, or may have a touch Responsive touch screen; mobile terminal can be a smartphone, tablet or wearable device (such as smart glasses, smart Watches, etc.) can also be smart cars, smart home appliances (such as smart refrigerators, smart batteries, set-top boxes, etc.); the operating system of smart phones can be Android operating system, IOS operating system or any other third-party developed to run on a microcomputer Operating system (including at least processor and memory) (such as mobile Linux, BlackBerry QNX, etc.).

The terminal described above includes an image capturing device for collecting an image of a reference object, where the reference object may be an object located at the terminal, for example, the reference object may be an object such as a human eye or a nose; and the image capturing device may include at least one camera.

The terminal described above further includes an image analyzing device for analyzing the image of the collected reference object. In actual implementation, the image analyzing device may be a processor on the terminal.

The following embodiments are proposed based on the display screen, the image pickup device, and the image analysis device provided on the terminal described above.

FIG. 1 is a flowchart of a method for operating a terminal according to an embodiment of the present disclosure. As shown in FIG. 1 , the process includes:

Step 101: Acquire an image of the reference object in real time, and the distance between the reference object and the display screen of the terminal exceeds a set value.

Here, the set value is greater than 0, and the set value can be set according to the actual application scenario; that is, the reference object does not form a contact relationship with the terminal display screen.

Here, the kind of the reference object is not limited. For example, the reference object may include: a person's eyes, a nose, or the reference object is an object having a minimum vertical distance from the display screen of the terminal, or the like.

Optionally, the image of the pupil may be separately collected by using at least one camera, for example, the number of cameras is 1 or 2; the camera may be disposed on the side of the terminal display, that is, the front camera is set on the terminal; the camera It can also be set on the back of the terminal, that is, the rear camera is set on the terminal.

Step 102: Perform an analysis on the image of the reference object to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object.

The following is divided into three cases to explain the implementation of this step.

First case

The reference includes a pupil of the person; the spatial position of the fovea of the eye is acquired in real time before the image of the reference is analyzed; the pupil and the fovea of the eye are located in the same eye; in step 101, two are utilized The camera separately captures an image of the pupil.

Correspondingly, the step includes: obtaining a spatial position of the pupil center point based on the image of the pupil; and a line and a terminal passing through the center of the eye and the center point of the pupil based on the spatial position of the central fovea and the pupil center point The intersection of the display screen is determined as a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.

In actual implementation, the above two cameras can be used to separately collect the image of the concave center of the eye, or the two cameras can be used to separately collect the image of the pupil center point; here, the image of the eye can be collected, and then the image recognition or image matching technology is utilized. An image of the pupil of the eye is obtained, and finally an image of the pupil center point is determined based on the image of the pupil of the eye.

Here, the spatial position of the pupil center point is obtained based on the image of the pupil, including: obtaining a space of a pupil center point based on a spatial position of the two cameras and an image acquired by the two cameras position.

It can be understood that the spatial position of each camera can be represented by three-dimensional space coordinates, in actual implementation, The coordinates of a point on the terminal can be set in advance, and then the three-dimensional space coordinates of each camera are determined according to the positional relationship between the point and each camera.

After obtaining the spatial position of the two cameras and the images acquired by the two cameras, the spatial position of the pupil center point can be determined based on binocular stereo vision technology.

It can be seen that the line passing through the center of the eye and the center point of the pupil actually represents the main line of sight of the human eye. Therefore, the determined mapping point is the intersection of a line of sight of the human eye and the display of the terminal.

Second case

In step 101, the image of the reference object is collected in real time by using at least one camera. Here, the type of the reference object is not limited. For example, the reference object is an object having the smallest vertical distance from the display screen of the terminal, or the reference The object is located in the human body.

Correspondingly, the step may include: selecting at least one point in the image of the reference object as a reference point; determining a spatial position of each reference point based on the image of each reference point and the spatial position of each camera; Determining a projection point of each reference point on the terminal display screen according to a spatial position of the reference point and a spatial position of the terminal display screen; and determining a mapping point that forms a preset mapping relationship with the reference object based on the projection point.

Illustratively, selecting a point in the image of the reference object as a reference point comprises: determining a spatial location of the reference object based on an image of the reference object and a spatial position of each camera; based on the reference The spatial position of the object is used as a reference point to minimize the vertical distance between the reference object and the terminal display screen. Optionally, in the present disclosure, the reference point is a point at which the vertical distance from the display screen of the terminal is the smallest, for example, when the reference object is a human nose, the reference point is a nose tip; When the terminal of the terminal display screen has the smallest vertical distance, if the finger of the person whose object has the smallest vertical distance from the display screen of the terminal, the reference point is the fingertip of the finger.

Here, when the number of cameras is 1, the distance between the camera and the reference object may be acquired in real time before analyzing the image of the reference object; based on the image of the reference point, the spatial position of the camera And determining a spatial position of the reference point by a distance between the camera and the reference object.

When the number of cameras is 2, a point may be selected as an reference point in the image of the reference object; based on the image of the reference point and the spatial position of the two cameras, the spatial position of the reference point is determined; Determining a projection point of the reference point on the display screen of the terminal by using a spatial position of the reference point and a spatial position of the display screen of the terminal; and forming the projection point as a preset on the terminal display screen and the reference object The mapping point of the mapping relationship.

In actual implementation, the distance sensing device or the distance detecting device may be disposed at a position on the terminal that does not exceed the first distance threshold from the camera, and the distance sensing device or the distance detecting device is configured to detect the distance between itself and the reference object, and the distance sensing device Or the distance detecting device may be a displacement sensor or a proximity sensor provided on the terminal. Here, the detected distance can be used as the distance between the camera and the reference object.

In actual implementation, the position of the terminal display screen may be determined according to the spatial position of each camera on the terminal and the relative positional relationship between each camera and the display screen of the terminal; each reference point is displayed on the terminal display screen with reference to the accompanying drawings. The projection point is visualized.

2 is a first schematic diagram of a projection point of a reference point on a display screen of a terminal according to an embodiment of the present disclosure. As shown in FIG. 2, the camera a and the camera b are two cameras disposed on the terminal, where the camera a and the camera b are located. Plane and terminal display The plane of the screen coincides. The coordinates of the reference point can be expressed as (X, Y, Z). The projection point of the reference point on the terminal display is: the intersection of the line perpendicular to the plane of the terminal display and the terminal display. .

3 is a second schematic diagram of a projection point of a reference point on a display screen of a terminal according to an embodiment of the present disclosure. As shown in FIG. 3, both the sensing point a and the sensing point b may be a camera disposed on the terminal, and the plane of the terminal display screen After the sensing point a and the sensing point b, the sensing layer represents the terminal display screen, and the coordinates of the reference point can be expressed as (X', Y', Z'), and the projection points of the reference point on the sensing layer are: over reference point and sensing The intersection of the line perpendicular to the plane and the sensing layer.

After determining the projection point, the projection point may be used as a mapping point for forming a preset mapping relationship with the reference object; or, when the reference point number is 2, determining two reference points on the terminal display screen The projection point is a mapping point of a line connecting the determined two projection points as a mapping point that forms a preset mapping relationship with the reference object.

In one implementation, the reference object includes two eyes of a person; the pupil center points of the two eyes of the person are respectively used as reference points; and then, the pupil center point of the two eyes of the person is determined at the terminal display screen The projection point on the upper point of the line connecting the determined two projection points is used as a mapping point for forming a preset mapping relationship with the reference object.

Third case

The reference object is one eye of a person; in step 101, an image in the human eye is collected in real time.

Correspondingly, the step may include: determining, in the current display content of the terminal display, an area matching the collected image in the human eye as: a screen matching area; selecting a point in the screen matching area as the reference object A mapping point that forms a preset mapping relationship.

Optionally, in the present disclosure, before analyzing the image of the reference object, the distance between the terminal display screen and the reference object may also be acquired; when the acquired distance is in the set interval, step 102 is performed; otherwise, If the acquired distance is not within the set interval, the process ends directly.

Here, the setting section is a section for indicating a distance, for example, the setting section is [0 cm, 5 cm], or [30 cm, 50 cm] or the like.

Optionally, in the present disclosure, after the mapping point is obtained, the cursor may also be displayed at the mapping point of the terminal display screen for the user to observe.

Step 103: Generate a function instruction at the mapping point, and implement an operation on the display screen of the terminal based on the function instruction.

Here, the function instruction at the mapping point generated by the step may be: a function instruction indicating that the current mapping point is clicked, a function instruction indicating that the current mapping point is long pressed, an instruction to perform a sliding operation, and the like. The instruction indicating the performing the sliding operation may be an instruction indicating a vertical sliding operation or an instruction indicating a horizontal sliding operation; the instruction indicating the vertical sliding operation may be an instruction or an instruction indicating that the upward sliding operation is performed to slide down The instruction of the screen operation, indicating that the instruction to perform the landscape sliding operation may be an instruction to perform a left-slide operation or an instruction to perform a right-slide operation.

In this step, the function instructions at the mapping point may be generated in the following two manners.

a first mode: determining a time when the mapping point is in a mapping point area on the terminal display screen; generating a function instruction at the mapping point based on the determined time size; the mapping point area includes The initial position of the mapped point.

In actual implementation, the time at which the mapping point is in the mapping point region may be determined by the processor of the terminal; afterwards, the processor of the terminal may generate a function instruction at the mapping point based on the determined size of the time.

Here, the mapping point area may be an area on the display screen of the terminal including the initial position of the mapping point; the area of the mapping point area is less than or equal to the set threshold. For example, if the initial position of the mapping point is point A, the mapping point area may be It is an area including the point A that is smaller than the set threshold, and the threshold is set to be 0.2 cm ² , 0.3 cm ^{2 ,} etc.; the shape of the boundary of the set area may be a circle, an ellipse, a polygon, or the like.

Preferably, the mapping point area may be a circular area centered on the initial position of the mapping point. For example, the mapping point area may be a circular area with the initial length of the mapping point as a center and a set length as a radius. .

Exemplarily, generating, according to the determined size of the time, the function instruction at the mapping point, comprising: when the determined time is in the first setting range, generating an instruction indicating that the current mapping point is clicked; the determined time When in the second setting range, generating an instruction to long press the current mapping point; when the determined time is in the third setting range, generating an instruction to perform a sliding operation; the first setting range, the second There is no overlap between the set range and the third set range, that is, there is no intersection between the first set range, the second set range, and the third set range.

For example, the first set range is represented as interval 1, the second set range is represented as interval 2, and the third set range is represented as interval 3, and interval 1, interval 2, and interval 3 are both used to represent the range of values of time. Each interval may be an open interval, a closed interval, or a half open half-closed interval, but there is no intersection between the interval 1, the interval 2, and the interval 3.

It can be seen that since there is no overlap between the first setting range, the second setting range and the third setting range, the time when the mapping point is in the setting area can only be in the above three setting ranges. Within a setting range, this ensures that at most one function instruction is generated.

In a preferred embodiment, the first setting range is [a1, a2], wherein a1 represents a first time threshold, a2 represents a second time threshold, a1 is less than a2, and the second setting range is (a2, ∞ ), the third setting range is (0, a1).

For example, the first time threshold a1 is 5 seconds, and the second time threshold a2 is 10 seconds; the mapping point area is a circular area centered on the initial position of the mapping point and having a radius of 0.3 cm; the mapping point area is a mapping The equivalent range of the initial position of the point; since the image of the reference object may change, the position of the mapping point also changes accordingly, so that the time of the mapping point in the mapping point area can be recorded by the processor of the terminal, at the mapping point When the duration of the mapping point area is greater than or equal to 5 seconds and less than 10 seconds, a function instruction indicating that the current mapping point is clicked is generated; when the mapping point has a duration of staying in the mapping point area of more than 10 seconds and less than 50 seconds, the generation is generated. Indicates that the function command at the current map point is long pressed.

Optionally, in the disclosure, if the duration of the mapping point in the mapping point area is in the third setting range, if the duration of the mapping point in the mapping point area is less than 5 seconds, it may be determined that the mapping point occurs. Large movement, in this case, an instruction to indicate a sliding operation is generated; in actual implementation, when the line of sight moves, the terminal continuously collects the coordinates of the mapped point; the time taken by the change and change of the position of the coordinate point , analyze the moving direction and moving speed of the mapping point.

In actual implementation, a two-dimensional Cartesian coordinate system can be established on the plane of the terminal display screen, and the two-dimensional Cartesian coordinate system is recorded as a screen coordinate system; the X-axis of the screen coordinate system is set to the lateral direction of the terminal display screen, The Y axis of the screen coordinate system is set to the longitudinal direction of the terminal display screen; in addition, the spatial position of the mapped point can also be represented by the coordinates of the screen coordinate system. The X-axis of the screen coordinate system is in the horizontal direction to the right, and the Y-axis in the screen coordinate system is in the vertical direction.

It can be understood that, according to the change of the X value of the coordinates of the mapping point in the screen coordinate system, the lateral moving distance of the mapping point is calculated, and then the time taken according to the change of the X value of the coordinate of the mapping point in the screen coordinate system is calculated. Horizontal shift of the mapped point Movement rate; here, the lateral movement direction is leftward or rightward; for example, the X-axis positive direction of the screen coordinate system is horizontal to the right direction; if the mapping point is in the screen coordinate system, the X value is within 1.5 seconds. Increase from 0 to 30, the lateral movement rate of the mapping point is 20 per second, and the lateral movement direction is to the right; in 1.5 seconds, if the X value of the coordinates of the mapping point in the screen coordinate system is reduced from 15 to 0 , the lateral movement rate of the mapping point is 10 seconds, and the lateral movement direction is leftward.

Similarly, the longitudinal movement distance of the mapping point can be calculated according to the change of the Y value of the coordinates of the mapping point in the screen coordinate system, and then calculated according to the time taken by the change of the Y value of the coordinate of the mapping point in the screen coordinate system. The longitudinal movement rate of the mapped point; here, the longitudinal movement direction is upward or downward; for example, the Y-axis of the screen coordinate system is in the vertical upward direction; if the mapping point is in the screen coordinate system within 1 second, The Y value increases from 0 to 18, the longitudinal movement rate of the mapping point is 18 per second, and the longitudinal movement direction is upward; in 1 second, if the Y value of the coordinate of the mapping point in the screen coordinate system is decreased by 0. To -10, the vertical movement rate of the mapped point is 10 per second, and the longitudinal movement direction is downward.

In an implementation manner, if the longitudinal movement rate of the mapping point is greater than the lateral movement rate of the mapping point, an instruction indicating that the vertical sliding operation is performed is generated, that is, an instruction indicating that the upper sliding screen operation is performed is generated; and the direction of the vertical sliding screen operation is Up or down; for example, the vertical movement rate of the mapping point is b1, the lateral movement rate of the mapping point is b2, and when b1 is greater than b2, an instruction is generated to indicate an upward sliding operation, or an indication is generated to perform a sliding screen operation. instruction.

If the lateral movement rate of the mapping point is greater than the longitudinal movement rate of the mapping point, generating an instruction indicating that the horizontal sliding operation is performed, that is, generating an instruction indicating that the left and right sliding operation is performed; and the direction of the horizontal sliding operation is leftward or rightward; For example, the lateral movement rate of the mapping point is b3, the longitudinal movement rate of the mapping point is b4, and when b3 is greater than b4, an instruction to perform a left-slide operation or an instruction to perform a right-slide operation is generated.

In particular, when the lateral movement rate of the mapping point is equal to the longitudinal movement rate of the mapping point, any function instruction may not be generated, and an instruction indicating a vertical sliding operation or an instruction indicating a horizontal sliding operation may be generated.

In actual implementation, the processor of the terminal may be used to record the starting point of the mapping point change and the coordinate of the ending point in the screen coordinate system; the starting point of the mapping point change to the longitudinal direction of the ending point may be set as a sliding operation. Orientation, the direction of the change from the endpoint of the mapping point change to the starting point is also set to the direction in which the sliding operation is performed; for example, the starting point of the mapping point change (the initial position of the mapping point) is in the screen coordinate system. The Y value of the coordinate is recorded as c1, and the Y value of the coordinate of the endpoint of the mapping point change in the screen coordinate system is recorded as c2. If c1 is smaller than c2, the longitudinal change direction of the starting point of the mapping point change to the end point is: In the upward direction, the direction in which the mapping point changes from the end point to the starting point is: the downward direction; at this time, the upward direction or the downward direction can be set to the direction in which the sliding operation is performed.

Similarly, the direction of the change from the starting point of the mapping point change to the end point of the mapping point may be set to the direction in which the sliding operation is performed, or the direction of the lateral change of the changing point of the mapping point to the starting point may be set as the sliding operation. For example, the X value of the coordinates of the starting point of the mapping point change in the screen coordinate system is recorded as d1, and the X value of the coordinate of the end point of the mapping point change in the screen coordinate system is recorded as d2, if d1 is smaller than D2, the direction of the horizontal change from the starting point of the mapping point change to the end point is: the direction to the right, the direction of the lateral change from the end point of the mapping point change to the starting point is: the direction to the left; at this time, the rightward direction can be The direction or the left direction is set to the direction in which the slide operation is performed.

In another implementation manner, when the lateral movement rate of the mapping point is greater than the longitudinal movement rate of the mapping point, the instruction indicating the horizontal sliding operation is not directly generated, but the determination of whether the lateral movement rate of the mapping point satisfies the first Setting a condition; if the lateral movement rate of the mapping point satisfies the first setting condition, generating an instruction indicating that the horizontal sliding screen is performed; If the lateral movement rate of the map point does not satisfy the first set condition, no instruction is generated.

Here, the first setting condition may be that the lateral movement rate of the mapping point is within the fourth setting range, and the fourth setting range may be greater than v1, or may be less than v2, or may be between v3 and v4. V3 is not equal to v4, and v1, v2, v3, and v4 can all be set by the user of the terminal, that is, v1, v2, v3, and v4 can all be set rate values.

Similarly, when the longitudinal movement rate of the mapping point is greater than the lateral movement rate of the mapping point, the instruction indicating the vertical sliding operation is not directly generated, but the determination of whether the longitudinal movement rate of the mapping point satisfies the second setting condition is continued. If the longitudinal movement rate of the map point satisfies the second set condition, an instruction is instructed to perform the vertical slide screen; if the longitudinal change rate of the map point does not satisfy the first set condition, no instruction is generated.

Here, the second setting condition may be that the longitudinal movement rate of the mapping point is within the fifth setting range, and the fifth setting range may be greater than v5, or may be less than v6, or may be between v7 and v8. V7 is not equal to v8, v5, v6, v7 and v8 can be set by the user of the terminal, that is, v5, v6, v7 and v8 can all be set rate values.

The second mode: before the function instruction at the mapping point is generated, continuously performing image acquisition on the action of the user of the terminal to obtain an action image of the user; performing image recognition on the action image of the user to obtain a recognition result Generating a function instruction at the mapping point based on the recognition result.

Here, the image capture device of the terminal may be used to collect the motion image of the user, and then the processor of the terminal is used to identify the motion image of the user, and the function instruction at the mapping point is generated based on the recognition result; for example, before using The camera captures the image changes of the user's head, and then collects the user's motion image.

Illustratively, when the recognition result is a blinking action, a mouth opening motion or a mouth closing motion, an instruction indicating that the click mapping point is generated is generated; and when the recognition result is a nodding motion, generating an instruction to perform the sliding screen operation And an instruction to generate an instruction to perform an upward sliding operation when the recognition result is a head-up operation; and an instruction to perform a left-slide operation or a right-slide operation when the recognition result is a left-right shaking action.

In actual implementation, the processor of the terminal may be used to generate a function instruction of the mapping point, and then the terminal automatically implements operation on the display screen of the terminal based on the function instruction, that is, the terminal may automatically implement the terminal based on the function instruction. The operation of the display does not require the user to touch the display.

Here, the operation of the terminal display screen corresponds to the function instruction; exemplarily, when the function instruction at the mapping point indicates that the instruction at the current mapping point is clicked, the click operation on the mapping point can be implemented based on the instruction; When the function instruction at the point is long pressing the instruction at the current mapping point, the long pressing operation on the mapping point can be implemented based on the instruction; when the function instruction at the mapping point is an instruction indicating the sliding operation, the instruction can be based on the instruction Achieve sliding operation.

After the operation of the display screen of the terminal is realized, the operation effect can be displayed on the display screen of the terminal. For example, when the click operation on the mapping point is implemented, the effect of opening the menu and exiting the menu can be achieved, and when the mapping point is realized, When the operation is pressed, the effect of long-pressing the menu can be realized; when the sliding operation is implemented, the effect of turning the page can be realized.

In the embodiment of the method for operating the terminal of the present disclosure, the mapping point on the display screen of the terminal can be obtained by analyzing the image of the reference object, and then clicking, long pressing, and sliding screen are completed based on the function instruction at the mapping point. Waiting for operation, realize non-touch screen operation terminal; do not need to touch the screen with a finger, and can operate the terminal only according to the reference object image; There is a technical problem that the finger operation terminal is inconvenient due to the size of the terminal display screen. The disclosure can effectively improve the efficiency of human-computer interaction, improve the operability of the terminal, and improve the user experience.

In order to further embodies the purpose of the present disclosure, further exemplifications are made on the basis of the above-described embodiments of the present disclosure.

FIG. 4 is a flowchart of an implementation manner of a method for operating a terminal according to an embodiment of the present disclosure. As shown in FIG. 4, the process includes:

Step 401: It is detected whether the positioning function is turned on by the terminal. If the positioning function is not enabled, the process is directly ended. At this time, the terminal does not respond and does not generate a function instruction; if the terminal opens the positioning function, then the process goes to step 402.

Step 402: Detect whether there is an object above the display screen of the terminal. If there is no object above the terminal, the process ends directly. At this time, the terminal does not respond and does not generate a function instruction; if there is an object above the display screen of the terminal, then the process goes to step 403.

Here, the distance detecting device and the distance sensing device may be used to determine whether there is an object above the display screen of the terminal within the sensing range of the distance detecting device and the sensing range of the distance sensing device.

Step 403: Detect whether a sensing space range is set on the terminal. If not, the terminal calculates a coordinate position of a mapping point of the reference object on the terminal screen, and the terminal according to the coordinate position of the mapping point, and the moving direction and movement of the mapping point. Rate, complete the command operation of clicking, long press, and sliding screen; if the sensing space range is set on the terminal, skip to step 404.

In this step, the sensing space range is the same as the setting interval of the display distance in the above embodiment.

Step 404: When the object is in the sensing space range, the terminal calculates the coordinate position of the reference point of the reference object on the terminal screen, and jumps to step 405.

It should be noted that when the object is not in the sensing space range, the terminal does not respond.

Step 405: The terminal completes the command operation of clicking, long pressing, and sliding screen according to the coordinate position of the mapping point and the moving direction and moving speed of the mapping point.

In the embodiment of the present disclosure, two cameras are disposed on the surface of the display screen of the terminal, and the two cameras are respectively labeled as the camera A3 and the camera B3, and the spatial positions of the camera A3 and the camera B3 can be represented by three-dimensional space coordinates, wherein The three-dimensional space coordinates of the camera A3 are (x _a3 , y _a3 , z _a3 ), and the three-dimensional space coordinates of the camera B3 are (x _b3 , y _b3 , z _b3 ); in actual implementation, the coordinates of a point on the terminal may be preset. Then, according to the positional relationship between the point and each camera, the three-dimensional space coordinates of each camera are determined.

Here, a reference object may be provided, and the reference object may be one eye of a person; FIG. 5 is a schematic view of a fovea and a pupil of the eye in the embodiment of the present disclosure, as shown in FIG. 5, the fovea of the eye is marked as E1, the center point of the pupil Marked as E2, the fovea of the eye is the clearest image of the human eye; the line passing through the fovea E1 of the eye and the center point E2 of the pupil may be a main line of sight of the human eye, and the main line of sight may be recorded as the line of sight L.

Here, the spatial position of the fovea of the eye and the center point of the pupil can be obtained, and the three-dimensional space coordinates of the spatial position of the acquired fovea E1 are (x1, y1, z1), and the spatial position of the acquired pupil center point E2 is three-dimensional. The space coordinates are (x2, y2, z2).

After the image acquisition is completed, based on the image of the fovea of the eye collected by the two cameras and the spatial position of the two cameras, the spatial position of the fovea of the eye is determined; and the center point of the pupil collected by the two cameras may also be The image and the spatial position of the two cameras above determine the spatial position of the pupil center point.

Here, in determining the spatial position of the fovea of the eye or the center point of the pupil, it can be realized by binocular stereoscopic vision; this will be described below with reference to FIG.

FIG. 6 is a schematic diagram of the principle of determining the position of a space point by using a dual camera according to an embodiment of the present disclosure. As shown in FIG. 6 , the two cameras are respectively recorded as O _l and O _r , and the focal lengths of the two cameras are f, and the two cameras are The distance between the two is T. Here, the XYZ three-dimensional Cartesian coordinate system is established with the position of a camera as the origin, wherein the X-axis direction is the continuous direction of the two cameras, the Y-axis is perpendicular to the X-axis, and the Z-axis direction is associated with each camera. The main optical axis (Principal Ray) direction is parallel; draw a left imaging plane perpendicular to the main optical axis of the camera O _l , the vertical distance from the camera optical center to the left imaging plane is the focal length f, and the left imaging plane is established in the left imaging plane two coordinate axes, the left imaging plane coordinate system are x _l y _l-axis and the axes, x _l axis parallel to X axis, y _l and Y axes parallel; Similarly, the camera main draw of O _r The right imaging plane perpendicular to the optical axis, the vertical distance from the optical center of the camera to the right imaging plane is the focal length f, and the right imaging plane coordinate system is established in the right imaging plane, and the two coordinate axes of the right imaging plane coordinate system are respectively x _l Axis and y _l axis, The x _l axis is parallel to the X axis, and the y _l axis is parallel to the Y axis.

In conjunction with FIG. 6, the intersection of the main optical axis camera O _l to the left imaging plane in the left imaging plane coordinate system is expressed as (c _x1, c _y1), the intersection of the main optical axis of the camera O _r of the right image plane of the right imaging plane The coordinate system is expressed as (c _x2 , c _y2 ); the intersection of the point P in the space with the optical center of the camera O _l and the left imaging plane is denoted as P _l , and the connection between the point P and the optical center of the camera O _r The intersection of the right imaging plane is denoted as P _r ; in actual implementation, according to the focal length f of the two cameras, the distance T between the two cameras, the point P _l in the coordinates of the left imaging plane coordinate system, the point P _{r is} on the right The coordinates of the plane coordinate system are imaged, and the coordinates of the point P in the above XYZ three-dimensional Cartesian coordinate system are obtained based on the binocular stereo vision principle.

After determining the spatial position of the fovea of the eye and the center point of the pupil, the intersection of the line passing through the center of the eye and the center point of the pupil can be determined based on the spatial position of the fovea of the eye and the center point of the pupil; An illustrative description.

7 is a schematic diagram showing the positional relationship between the human eye line of sight and the terminal display screen according to the embodiment of the present disclosure. Referring to FIG. 5 and FIG. 7, the eye center concave E1 and the pupil center point are determined based on the positions of the eye center concave E1 and the pupil center point E2. The line of sight L of E2; and based on the position of the known terminal display screen, determines the position of the intersection O of the line of sight L and the terminal display screen.

Here, the position of the terminal display screen can be determined according to the spatial position of each camera on the terminal and the relative positional relationship between each camera and the display screen of the terminal.

Here, the intersection point O of the line of sight L and the terminal display screen is a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object; optionally, in the present disclosure, the intersection of the line of sight L and the terminal display screen is determined. After the position of O, an indication point can also be displayed at the position of the intersection of the terminal display screen. Thus, it is convenient to visually display the mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.

After determining a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object, generating a function instruction at the mapping point based on the mapping point; the generated function instruction at the mapping point may be used to: indicate Click on the map point, indicate long press at the map point or indicate to perform a sliding screen operation; the slide screen operation can be up, down, left or right.

In an embodiment of the present disclosure, an implementation manner of generating a function instruction at the mapping point, and an implementation manner of implementing an operation on a terminal display screen based on the function instruction have been described in the foregoing embodiment of the present disclosure, and Let me repeat.

In an embodiment of the present disclosure, the reference object may be one eye of a person; a camera (front camera) is mounted on the side of the terminal display screen, and the image in the human eye is collected by using the front camera, that is, the object is collected in the human eye. Imaging; alternatively, in the present disclosure, the sharpest point of the image in the collected human eye can also be determined, where the center point of the image in the human eye can be determined as the sharpest point.

An area in the current display content of the terminal display that matches an image or a reference image in the collected human eye is recorded as: a screen matching area; the reference image is an area in the image of the collected human eye that includes the sharpest point The image, for example, the reference image may be an image of a region having a radius of 1 cm at the center of the sharpest point.

A point is determined in the screen matching area as a mapping point on the terminal display screen to form a preset mapping relationship with the reference object. For example, a point is determined as a point of the above mapping in a circular area centered on the center point of the screen matching area and having a radius of 0.5 cm.

Alternatively, in the present disclosure, the coordinates of the above mapping point in the screen coordinate system may also be determined.

In an embodiment of the present disclosure, an implementation manner of generating a function instruction at the mapping point, and an implementation manner of implementing an operation on a terminal display screen based on the function instruction have been described in the above embodiment of the present disclosure, where No longer.

In the embodiment of the present disclosure, the reference object may be two eyes of the person; two cameras are installed on the surface of the terminal display screen, and the two cameras are respectively recorded as the camera A5 and the camera B5, and the coordinates of the camera A5 in the three-dimensional coordinate system. For (x _a5 , y _a5 , z _a5 ), the coordinates of the camera B5 in the three-dimensional coordinate system are (x _b5 , y _b5 , z _b5 ); alternatively, in the present disclosure, the camera A5 and the camera B5 can be configured in the same In a plane, the plane formed by the two cameras is parallel or coincident with the terminal screen.

Each camera can separately capture images of two eyes, and then use image recognition or image matching technology to obtain an image of the pupil of each eye, and finally determine an image of the pupil center point of each eye according to the image of the pupil of each eye; Here, the pupil center point of each eye is the reference point.

After the camera A5 and the camera B5 both collect images of the pupil center points of the two eyes, the two images of the pupil center points of the two eyes collected by each camera and the spatial positions of the camera A5 and the camera B5 can be determined. The spatial position of the pupil's center point of the eye; here, the pupil center points of the two eyes can be recorded as C5 and D5, respectively; obviously, after determining the spatial position of the pupil center point of both eyes, each can be drawn The distance from the camera to point C5 and the distance from each camera to point D5.

Here, the principle of determining the spatial position of the pupil center point of both eyes has been explained in the above embodiment of the present disclosure, and details are not described herein again.

After determining the spatial position of the pupil center point of the two eyes, the processor of the terminal can determine the pupil center point of the two eyes on the terminal display according to the spatial position of the pupil center point of the two eyes and the position of the terminal display screen. The projection point on the top, where the projection point of the C5 point on the terminal display is recorded as E5 point, the projection point of the D5 point on the terminal display is recorded as F5 point, and the coordinate of the E5 point in the screen coordinate system is (X _A5 , Y _A5 ), the coordinates of the F5 point in the screen coordinate system are (X _B5 , Y _B5 ).

Here, the position of the terminal display screen can be determined according to the spatial position of the camera A5 and the camera B5 on the terminal, and the relative positional relationship between the camera A5 and the camera B5 and the terminal display screen.

After determining the projection point of the pupil center point of the two eyes on the terminal display screen, the midpoint of the line connecting the projection center points of the two eyes on the terminal display screen can be determined, and the determined connection is determined. The midpoint is a mapping point on the terminal display that forms a preset mapping relationship with the reference object.

In actual implementation, the midpoint O5 of the line connecting E5 and F5 can be determined according to the coordinates of the E5 point and the F5 point in the screen coordinate system. Here, the O5 point is a preset on the terminal display screen and the reference object. The mapping point of the mapping relationship, the sitting point of the O5 point in the screen coordinate system is (X _O5 , Y _O5 ).

Further exemplifications are made on the basis of the above-described embodiments of the present disclosure in order to further embody the purpose of the present disclosure.

In the embodiment of the present disclosure, a front camera and a rear camera are installed on the terminal, the front camera is used to collect the image of the reference object A in real time, and the rear camera is used to collect the image of the reference object B in real time;

Based on the above embodiment of the present disclosure, before analyzing the image of the reference object A, it is necessary to acquire the distance between the front camera and the reference object A in real time; before analyzing the image of the reference object B, it is necessary to acquire the rear camera and the reference object B in real time. the distance.

After the reference point is selected from the reference object A, the spatial position of the reference point may be determined based on the image of the reference point, the spatial position of the front camera, and the distance between the front camera and the reference object A; similarly, the reference object B is selected. After the reference point, the spatial position of the reference point can be determined based on the image of the reference point, the spatial position of the rear camera, and the distance between the rear camera and the reference object B.

For example, the reference point selected from the reference object A is represented as the point A6, and the reference point selected from the reference object B is represented as the point B6, and the coordinates of the spatial position of the point A6 can be recorded as (X _A6 , Y _A6 , Z _A6 ) The coordinates of the spatial position of the point B6 can be written as (X _B6 , Y _B6 , Z _B6 ).

After determining the reference point, determining an implementation manner of a projection point of each reference point on the terminal display screen, and determining, according to the projection point, a mapping point that forms a preset mapping relationship with the reference object, The above embodiments are disclosed in the above description, and are not described herein again.

It should be noted that the terminal can be provided with a front camera and a rear camera switch for selecting the front camera or the rear camera. Thus, the user can determine the relative position relationship between the reference object and the terminal according to the need to determine the camera before starting. , or after starting the camera.

Based on the above embodiments of the present disclosure, an embodiment of the present disclosure also proposes a terminal.

FIG. 8 is a schematic structural diagram of a terminal according to an embodiment of the present disclosure. As shown in FIG. 8, the terminal 800 includes: an image collection device 801 and a processor 802.

The image capturing device 801 is configured to collect an image of the reference object in real time, and the distance between the reference object and the display screen of the terminal exceeds a set value;

The processor 802 is configured to analyze an image of the reference object to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, and generate a function instruction at the mapping point, based on the The function instructions implement the operation of the terminal display.

The reference object includes a pupil of a person;

The processor 802 is further configured to acquire a spatial position of the fovea of the eye in real time; the pupil and the fovea of the eye are located in the same eye;

Correspondingly, the processor 802 is configured to obtain a spatial position of the pupil center point based on the image of the pupil; based on the spatial position of the eye fovea and the pupil center point, passing through the center fovea and the pupil center point The intersection of the straight line and the terminal display screen is determined as a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.

The image capture device 801 includes at least one camera;

The processor 802 is configured to select at least one point in the image of the reference object as a reference point; determine a spatial position of each reference point based on an image of each reference point and a spatial position of each camera; Determining a projection point of each reference point on the terminal display screen according to a spatial position of the reference point and a spatial position of the terminal display screen; and determining a mapping point that forms a preset mapping relationship with the reference object based on the projection point.

The reference object includes one eye of a person;

The image capturing device 801 is configured to collect images in a human eye in real time;

The processor 802 is configured to determine, in a current display content of the terminal display screen, an area that matches an image in the collected human eye as: a screen matching area; and select a point in the screen matching area as the reference object A mapping point that forms a preset mapping relationship.

Optionally, in the present disclosure, the processor 802 is further configured to determine a distance between the terminal display screen and the reference object before analyzing the image of the reference object;

The processor 802 is configured to analyze an image of the reference object when the determined distance is in a set interval, and obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object; When it is determined that the acquired distance is not within the set interval, the flow is directly ended.

The processor 802 is configured to determine a time when the mapping point is in a setting area; and generate a function instruction at the mapping point based on the determined size of the time.

Optionally, in the disclosure, the image collection device 801 is further configured to continuously perform image collection on the action of the user of the terminal before generating the function instruction at the mapping point, to obtain an action image of the user;

The processor 802 is further configured to perform image recognition on the motion image of the user to obtain a recognition result, and generate a function instruction at the mapping point based on the recognition result.

In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium comprising instructions, for example comprising The memory of the instructions, which may be executed by a processor of the terminal to perform the above method. For example, the non-transitory computer readable storage medium described above may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

Those skilled in the art will appreciate that embodiments of the present disclosure can be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or a combination of software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) including computer usable program code.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

The above description is only for the preferred embodiments of the present disclosure, and is not intended to limit the scope of the disclosure.

Industrial applicability

The present disclosure is applicable to the field of human-computer interaction, and can operate the terminal only according to the reference object image without touching the screen with a finger; it is inconvenient for the finger operation terminal that is currently caused by the size of the terminal display screen becoming large. The technical problem is that the invention can effectively improve the efficiency of human-computer interaction, improve the operability of the terminal, and improve the user experience.

Claims

A method of operating a terminal, comprising:

Acquiring an image of the reference object in real time, the distance between the reference object and the display screen of the terminal exceeds a set value;

Performing an analysis on the image of the reference object to obtain a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object;

Generating a function instruction at the mapping point, and implementing an operation on a terminal display screen based on the function instruction.
The method of claim 1 wherein said reference comprises a pupil of a person;

Correspondingly, the image of the reference object is analyzed to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including:

Obtaining a spatial position of the fovea of the eye in real time, the pupil and the fovea of the eye being located in the same eye; and based on the image of the pupil, obtaining a spatial position of the central point of the pupil;

Based on the spatial position of the fovea of the eye and the center point of the pupil, the intersection of the line passing through the center point of the eye and the center point of the pupil is determined as a mapping relationship between the terminal display and the reference object. point.
The method of claim 2, wherein the capturing an image of the reference object in real time comprises: separately acquiring an image of the pupil using a dual camera;

The obtaining a spatial position of the pupil center point based on the image of the pupil includes: obtaining a spatial position of the pupil center point based on a spatial position of the two cameras and an image acquired by the two cameras.
The method of claim 1, wherein the capturing an image of the reference object in real time comprises: acquiring an image of the reference object in real time using at least one camera;

Correspondingly, the image of the reference object is analyzed to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including:

Selecting at least one point in the image of the reference object as a reference point;

Determining the spatial position of each reference point based on the image of each reference point and the spatial position of each camera;

Determining a projection point of each reference point on the terminal display screen based on a spatial position of each reference point and a spatial position of the terminal display screen;

A mapping point that forms a preset mapping relationship with the reference object is determined based on the projection point.
The method according to claim 4, wherein the determining, based on the projection point, a mapping point that forms a preset mapping relationship with the reference object comprises: forming the projection point as a preset with the reference object The mapping point of the mapping relationship.
The method according to claim 4, wherein the number of the reference points is two;

Determining, according to the projection point, a mapping point that forms a preset mapping relationship with the reference object, comprising: determining a projection point of the two reference points on the display screen of the terminal, and determining the connection of the two projection points The midpoint serves as a mapping point that forms a preset mapping relationship with the reference object.
The method of claim 6 wherein said reference comprises two eyes of a person;

Selecting at least one point in the image of the reference object as a reference point includes: placing the two eyes of the person The pupil center points are used as reference points, respectively.
The method according to claim 4, wherein said selecting at least one point in the image of said reference object as a reference point comprises:

Determining a spatial location of the reference object based on an image of the reference object and a spatial location of each camera;

Based on the spatial position of the reference object, a point at which the vertical distance from the terminal display screen in the reference object is the smallest is used as a reference point.
The method of claim 4, wherein the capturing an image of the reference object in real time comprises: acquiring an image of the reference object in real time using a camera;

Before analyzing the image of the reference object, the method further includes: acquiring a distance between the camera and the reference object in real time;

Correspondingly, determining the spatial location of the reference point based on the image of the reference point and the spatial position of each camera comprises:

A spatial position of the reference point is determined based on an image of the reference point, a spatial position of the camera, and a distance between the camera and the reference object.
The method of claim 4, wherein the reference object is an object having a minimum vertical distance from the display screen of the terminal, or the reference object is located in a human body.
The method according to claim 1, wherein the reference object comprises one eye of a person; the real-time acquisition of an image of the reference object comprises: collecting an image in a human eye in real time;

The analyzing the image of the reference object to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including: displaying the current display content of the terminal display screen and the collected human eyes The image matching area is determined as: a screen matching area; a point is selected in the screen matching area as a mapping point that forms a preset mapping relationship with the reference object.
The method of claim 1, wherein prior to analyzing the image of the reference object, the method further comprises: determining a distance between the terminal display screen and the reference object;

The analyzing the image of the reference object to obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object, including: when the determined distance is in the set interval, the reference object The image is analyzed to obtain a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.
The method of claim 1, wherein the generating the function instruction at the mapping point comprises: determining a time at which the mapping point is in a mapping point area on a terminal display screen; generating a location based on the determined size of time a function instruction at a mapping point; the mapping point area includes an initial position of a mapping point on the display screen of the terminal.
The method of claim 13, wherein the generating the functional instructions at the mapping point based on the determined size of the time comprises:

When the determined time is in the first setting range, generating an instruction indicating that the current mapping point is clicked; when the determined time is in the second setting range, generating an instruction indicating that the current mapping point is long pressed; the determined time is at When the third setting range is set, an instruction indicating that the sliding operation is performed is generated;

No overlap is formed between the first setting range, the second setting range, and the third setting range.
The method of claim 14, wherein the first set range is a range from a first time threshold to a second time threshold; the second set range is greater than a second time threshold; the third The setting range is less than the first time threshold; the first time threshold is less than the second time threshold.
The method according to claim 14, wherein the generating an instruction to perform a sliding operation comprises: acquiring a moving direction and a moving rate of the mapping point, generating an indication to slide based on the moving direction and the moving speed of the mapping point Instructions for screen operations.
The method according to claim 16, wherein the generating an instruction to perform a sliding screen operation based on a moving direction and a moving rate of the mapping point comprises:

The moving rate of the mapping point in the lateral direction of the display screen of the mobile terminal is used as the lateral moving rate of the mapping point, and the moving rate of the mapping point in the longitudinal direction of the display screen of the mobile terminal is taken as the longitudinal moving speed of the mapping point;

When the lateral movement rate of the mapping point is greater than the longitudinal movement rate of the mapping point, generating an instruction indicating that the horizontal sliding operation is performed; or, the lateral movement rate of the mapping point is greater than the longitudinal moving speed of the mapping point, and the horizontal direction of the mapping point When the moving rate satisfies the first setting condition, generating an instruction indicating that the horizontal sliding operation is performed;

When the longitudinal movement rate of the mapping point is greater than the lateral movement rate of the mapping point, generating an instruction indicating that the vertical sliding operation is performed; or, the longitudinal movement rate of the mapping point is greater than the lateral movement rate of the mapping point, and the vertical direction of the mapping point When the movement rate satisfies the second setting condition, an instruction is instructed to perform the vertical sliding operation.
The method according to claim 17, wherein the first setting condition is that a lateral movement rate of the mapping point is within a fourth setting range; and the second setting condition is that a longitudinal movement speed of the mapping point is Within the fifth setting range.
The method according to claim 13, wherein the mapping point area comprises an initial position of a mapping point on the display screen of the terminal; and an area of the mapping point area is less than or equal to a set threshold.
The method of claim 19, wherein the map point area is a circular area centered on an initial position of the map point.
The method according to claim 1, wherein before the generating the function instruction at the mapping point, the method further comprises: continuously performing image acquisition on the action of the user of the terminal to obtain an action image of the user; Performing image recognition on the motion image of the user to obtain a recognition result;

Correspondingly, the generating the function instruction at the mapping point comprises: generating a function instruction at the mapping point based on the recognition result.
The method according to claim 21, wherein the generating the function instruction at the mapping point based on the recognition result comprises: generating an indication when the recognition result is a blink action, a mouth open action or a mouth closing action Clicking an instruction at the mapping point; when the recognition result is a nodding action, generating an instruction to perform a downward screen operation; and when the recognition result is a head-up operation, generating an instruction to perform an upward sliding operation; the recognition result is When the left and right shaking action is performed, an instruction to instruct a lateral sliding operation is generated.
The method of claim 1, wherein the function instruction at the mapping point is: an instruction to click a function instruction at a current mapping point, a function instruction to press and hold a current mapping point, or an instruction to perform a sliding operation.
A terminal comprising:

The image capturing device is configured to collect an image of the reference object in real time, and the distance between the reference object and the display screen of the terminal exceeds a set value;

a processor configured to analyze an image of the reference object to obtain a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object; generate a function instruction at the mapping point, based on the function The instructions implement the operation of the terminal display.
The terminal of claim 24, wherein the reference object comprises a pupil of a person;

The processor is further configured to acquire a spatial position of the fovea of the eye in real time; the pupil and the fovea of the eye are located in the same eye;

Correspondingly, the processor is arranged to derive a spatial position of the pupil center point based on the image of the pupil; a straight line passing through the center of the eye and the center of the pupil based on the spatial position of the central fovea and the pupil center point The intersection with the terminal display screen is determined as a mapping point on the terminal display screen that forms a preset mapping relationship with the reference object.
The terminal of claim 24, wherein said image acquisition device comprises at least one camera;

The processor is configured to select at least one point in the image of the reference object as a reference point; determine a spatial position of each reference point based on the image of each reference point and the spatial position of each camera; based on each reference point a spatial position and a spatial position of the terminal display screen, determining a projection point of each reference point on the terminal display screen; and determining a mapping point that forms a preset mapping relationship with the reference object based on the projection point.
The terminal of claim 24, wherein the reference object comprises one eye of a person;

The image capture device is configured to collect images in a human eye in real time;

The processor is configured to determine an area in the current display content of the terminal display screen that matches an image in the collected human eye as: a screen matching area; and select a point in the screen matching area as a pre-form with the reference object The mapping point of the mapping relationship.
The terminal of claim 24, wherein the processor is further configured to determine a distance between the terminal display screen and the reference object prior to analyzing the image of the reference object;

The processor is configured to analyze the image of the reference object when the determined distance is in the set interval, and obtain a mapping point on the display screen of the terminal that forms a preset mapping relationship with the reference object.
The terminal of claim 24, wherein the processor is configured to determine a time at which the mapping point is in a set region; and to generate a function instruction at the mapping point based on the determined size of the time.
The terminal according to claim 24, wherein the image collection device is further configured to continuously perform image acquisition on the action of the user of the terminal to generate an action image of the user before generating the function instruction at the mapping point;

The processor is further configured to perform image recognition on the motion image of the user to obtain a recognition result; and generate a function instruction at the mapping point based on the recognition result.