CN112819970A - Control method and device and electronic equipment - Google Patents
Control method and device and electronic equipment Download PDFInfo
- Publication number
- CN112819970A CN112819970A CN202110188853.2A CN202110188853A CN112819970A CN 112819970 A CN112819970 A CN 112819970A CN 202110188853 A CN202110188853 A CN 202110188853A CN 112819970 A CN112819970 A CN 112819970A
- Authority
- CN
- China
- Prior art keywords
- mode
- target
- state
- image
- feature points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000004364 calculation method Methods 0.000 claims description 16
- 230000002123 temporal effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 239000011521 glass Substances 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000033001 locomotion Effects 0.000 description 4
- 238000004873 anchoring Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/19—Sensors therefor
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Ophthalmology & Optometry (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- User Interface Of Digital Computer (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application discloses a control method, a control device and electronic equipment, wherein a target characteristic point is obtained by responding to the display of a virtual image of target equipment in a first mode, and the target characteristic point represents the characteristic point with consistent display coordinate positions in two collected images before and after the target characteristic point represents; determining a first number of target feature points in a first state; determining a second number of target characteristic points in a second state, wherein the image acquisition environments of the target equipment in the first state and the second state are different; whether to switch the first mode to the second mode is determined based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image. The switching of the display modes is realized based on the number of the target characteristic points in different image acquisition environments, so that the display modes of the target equipment are matched with the environments, and the actual application requirements are met.
Description
Technical Field
The present application relates to the field of information processing technologies, and in particular, to a control method, an apparatus, and an electronic device.
Background
Augmented Reality (AR) is a technology capable of fusing virtual information into a real environment, and implements analog simulation processing on entity information which is difficult to experience in a space range of the real world originally on the basis of scientific technologies such as computers, and virtual information content is effectively applied to the real world by superposition.
In the application of the AR equipment, the AR equipment needs to calculate the coordinate of the virtual image displayed in the space by identifying the environment, but not all environments are beneficial to the identification and calculation of the AR equipment, so that the actual requirements of the AR equipment cannot be met, and the user experience effect is influenced.
Disclosure of Invention
In view of this, the present application provides the following technical solutions:
a control method, comprising:
responding to the display of a virtual image by target equipment in a first mode, and acquiring target feature points, wherein the target feature points represent feature points with consistent display coordinate positions in two collected images in front and at back;
determining a first number of the target feature points in a first state;
determining a second number of the target characteristic points in a second state, wherein image acquisition environments of the target equipment in the first state and the second state are different;
determining whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image.
Optionally, the obtaining of the target feature point includes:
responding to the left-eye image and the right-eye image collected by the target equipment, and determining the characteristic points with consistent display coordinate positions in the left-eye image and the right-eye image as target characteristic points;
or,
responding to a left-eye two-dimensional image and a right-eye two-dimensional image acquired by the target equipment, and performing spatial projection on the left-eye two-dimensional image to obtain a first spatial point;
performing spatial projection on the right-eye two-dimensional image to obtain a second spatial point;
and determining the space point with consistent space coordinates in the first space point and the second space point as a target characteristic point.
Optionally, the obtaining of the target feature point includes:
acquiring a feature point at a first moment and a feature point at a second moment, wherein the first moment and the second moment have a time-sequence incidence relation;
and determining the feature point with the consistent display position in the feature point at the first moment and the feature point at the second moment as a target feature point.
Optionally, the image capturing environment comprises one of an image capturing temporal environment, an image capturing spatial environment, and an image capturing scene environment.
Optionally, the determining whether to switch the first mode to the second mode based on the ratio of the first number to the second number comprises:
and if the ratio of the first number to the second number is smaller than a target threshold, switching the first mode to a second mode to display the virtual image by the target device in the second mode, wherein the amounts of the pose data for generating the virtual image, which are respectively obtained in the first mode and the second mode, are different.
Optionally, the method further comprises:
determining a third number of the target feature points in a third state;
and if the ratio of the third quantity to the second quantity is not less than the target threshold, switching the second mode to the first mode.
Optionally, the method further comprises:
if the target equipment displays the virtual image in a second mode, acquiring first attitude data in a calculation mode corresponding to the second mode based on the current target characteristic point;
obtaining second posture data in a calculation mode corresponding to the first mode based on the current target feature point;
and if the difference value of the first position posture data and the second position posture data is larger than the target difference value, switching the second mode to a third mode, wherein the position posture data corresponding to the third mode is different from the first position posture data corresponding to the second mode.
A control device, comprising:
the acquisition unit is used for responding to the display of a virtual image of target equipment in a first mode and acquiring target characteristic points representing characteristic points with consistent display coordinate positions in the front and rear collected images;
a first determining unit, configured to determine a first number of the target feature points in a first state;
the second determining unit is used for determining a second number of the target characteristic points in a second state, and image acquisition environments of the target equipment in the first state and the second state are different;
a third determination unit configured to determine whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode differing in pose data utilized in generating the virtual image.
A storage medium storing computer executable instructions for performing a control method as claimed in any one of the preceding claims when executed by a processor.
An electronic device, comprising:
a memory for storing an application program and data generated by the application program running;
a processor for executing the application to implement:
responding to the display of a virtual image by target equipment in a first mode, and acquiring target feature points, wherein the target feature points represent feature points with consistent display coordinate positions in two collected images in front and at back;
determining a first number of the target feature points in a first state;
determining a second number of the target characteristic points in a second state, wherein image acquisition environments of the target equipment in the first state and the second state are different;
determining whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image.
According to the technical scheme, the control method, the control device and the electronic equipment disclosed by the application respond to the display of the virtual image of the target equipment in the first mode, and obtain the target characteristic points, wherein the target characteristic points represent characteristic points with consistent display coordinate positions in the two collected images in front and at the back; determining a first number of target feature points in a first state; determining a second number of target characteristic points in a second state, wherein the image acquisition environments of the target equipment in the first state and the second state are different; whether to switch the first mode to the second mode is determined based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image. The switching of the display modes is realized based on the number of the target characteristic points in different image acquisition environments, so that the display modes of the target equipment are matched with the environments, and the actual application requirements are met.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of a control method according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application;
fig. 3 is a schematic structural diagram of a control device according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application provides a control method, which is applied to AR (Augmented Reality) equipment, so that the AR equipment can switch corresponding display modes under different exchange conditions, the virtual image is guaranteed to be accurately displayed and anchored in a space, and the experience effect of a user is improved.
The AR (Augmented Reality) technology is a technology for fusing virtual information with a real world, and a user can view a virtual effect superimposed in a real scene through an AR device, that is, the user can view a virtual image, for example, a virtual article superimposed in a real indoor environment, thereby completing an experience of the virtual article.
Referring to fig. 1, a flow chart of a control method provided in an embodiment of the present application is shown, where the method may include the following steps:
s101, responding to the display of the virtual image of the target equipment in a first mode, and acquiring target characteristic points.
In this embodiment of the application, the target device is an AR device, such as AR smart glasses, and the virtual image displayed by the target device may be a final scene image presented to the user, that is, a virtual effect image obtained by superimposing virtual information on a real scene, or an image corresponding to the virtual information, such as a virtual effect image obtained by superimposing a virtual cup on a real indoor scene, or a virtual image corresponding to the virtual cup.
The first mode is one of target device display modes, where the display mode refers to a mode in which a virtual image is displayed and anchored, that is, a mode in which a virtual image is generated using what kind of pose data. The target feature points represent feature points with consistent display coordinate positions in two collected images before and after the target feature points represent feature points, wherein the feature points are feature points of point cloud data in a space obtained by target equipment (namely AR equipment) through a depth camera, namely the feature points can form feature point clouds, namely the point clouds are shortened. The feature points refer to visual difference feature points in a captured camera image, a certain number of target feature points are obtained by performing multi-layer repeated screening on the basis of the feature points, the target feature points can be used for calculating pose data, such as camera pose data of AR equipment, and the target feature points can also be changed into stable feature points in certain application scenes, namely the feature points with the same display coordinate position in the two frames of collected images.
S102, determining a first number of the target characteristic points in a first state.
S103, determining a second number of the target characteristic points in a second state.
And S104, determining whether to switch the first mode to the second mode based on the ratio of the first number to the second number.
It should be noted that the image capturing environments of the target device in the first state and the second state are different, and the difference in the image capturing environments may refer to one of a difference in image capturing time environments, a difference in image capturing space environments, or a difference in image capturing scene environments. The number of target feature points acquired by the target device in different states is also different. Since the target device is worn by the user, the state of the target device is related to the user corresponding to the target device, such as the environment in which the user wears the target device, the shift of the user's gaze point, or the movement of the user. The target device presents the virtual image in a real-time process, so that the target feature point is acquired in a real-time process, and in order to enable the display mode of the virtual image displayed by the virtual device to be in accordance with the environmental characteristics, the target feature points in different states need to be determined. For example, the first state may be a state of the target device when the user gazes at the first area, and the second state may be a state of the target device when the user gazes at the second area. The target feature points may be stable feature points, that is, in the process of continuously screening feature points, a display mode is determined according to the latest stable feature point number and the number of initially screened stable feature points, wherein the latest stable feature point number corresponds to the first number, and the number of initially screened determined feature points corresponds to the second number.
It is then determined whether it is necessary to switch the display mode of the target apparatus, i.e., whether to switch the first mode to the second mode, based on a ratio of the first number to the second number. The first mode and the second mode differ in pose data utilized in generating the virtual image. The pose refers to the position and direction of an object, wherein the position data can refer to latitude, longitude and altitude, and the direction is a direction angle, a pitch angle and a roll angle. The position of an object can be represented by (x, y, z). And the directions can be represented by (α, β, γ), which are angles representing rotations around three coordinate axes. Specifically, pose data, which is pose data of a camera of the target device and is also referred to as camera pose data, can be obtained by determining target feature points on the displayed virtual image, and then calculating based on coordinate parameters corresponding to the target feature points and motion data acquired by a sensor corresponding to the target device.
For example, the first mode may be a 6DoF (Degrees of freedom) mode, the second mode may be a 3DoF mode, and the 3DoF may refer to 3 Degrees of freedom of rotation, such as different directions of rotation of the head of the user wearing the target device, but it cannot detect spatial displacement of the head in front, back, left, and right. 6DoF can be on the basis of 3DoF, the change of up-down, front-back, left-right movement caused by the body movement of the brave wearing the target equipment is increased, and the tracking and positioning of the user can be better realized.
The embodiment of the application provides a control method, which comprises the steps of responding to the display of a virtual image by target equipment in a first mode, and obtaining target characteristic points, wherein the target characteristic points represent characteristic points with consistent display coordinate positions in two collected images before and after the target characteristic points represent; determining a first number of target feature points in a first state; determining a second number of target characteristic points in a second state, wherein the image acquisition environments of the target equipment in the first state and the second state are different; whether to switch the first mode to the second mode is determined based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image. The switching of the display modes is realized based on the number of the target characteristic points in different image acquisition environments, so that the display modes of the target equipment are matched with the environments, and the actual application requirements are met.
In the embodiment of the application, the state of the point cloud data can be detected in real time in a SLAM (Simultaneous Localization And Mapping) manner, And then the target feature point is determined according to the state of the feature point in the point cloud data. Specifically, the target feature point acquisition in the embodiment of the present application may be implemented in the following manner:
in one possible implementation manner, in response to the left-eye image and the right-eye image acquired by the target device, the feature point with the consistent display coordinate position in the left-eye image and the right-eye image is determined as the target feature point.
When the target device is an AR device, the AR device may generally include a left-eye display device and a right-eye display device, and when the left-eye display device and the right-eye display device display a left-eye image and a right-eye image corresponding to a virtual object, and when display coordinate positions of some feature points in the left-eye image and the right-eye image are consistent or have a small error, the feature points may be used as target feature points, that is, stable feature points. The feature points refer to visual difference feature points in a captured camera image, such as selected from points with large differences in light, shade, color and gray level in the image, specifically, edge points of a virtual object.
In another possible implementation, in response to the target device acquiring the left-eye two-dimensional image and the right-eye two-dimensional image, performing spatial projection on the left-eye two-dimensional image to obtain a first spatial point; performing spatial projection on the right-eye two-dimensional image to obtain a second spatial point; and determining the space point with consistent space coordinates in the first space point and the second space as the target characteristic point.
Specifically, a plurality of left-eye two-dimensional images may be acquired, and then a first spatial point corresponding to the three-dimensional spatial pose information may be obtained according to three-dimensional spatial position information of an object included in the plurality of left-eye two-dimensional images in the point cloud data. And determining the space point with consistent space coordinates in the first space point and the second space point as a stable space point, namely the target characteristic point.
In another possible implementation manner, the feature point at the first time and the feature point at the second time may be obtained, and the feature point with the same display position in the feature points at the first time and the feature points at the second time may be determined as the target feature point.
The first time and the second time have a time-sequential association relationship, for example, the first time is the current time, and the second time is the next second of the current time. That is, a plurality of feature points may be selected at a first time, then a corresponding number of feature points are selected at a second time, the feature points at the two times are matched, the matched feature points are determined as target feature points, and specifically, the feature points with the same display positions in the feature points may be determined as the target feature points. Correspondingly, a certain number of feature points can be randomly selected in batches for matching through an ICP (Iterative Closest Point algorithm). So that the point cloud data under different coordinates are merged into the same coordinate system, firstly, an available transformation is found, and the registration operation is to find a rigid transformation from the coordinate system 1 to the coordinate system 2. The ICP algorithm is essentially an optimal registration method based on the least squares method. The algorithm repeatedly selects the corresponding relation point pairs and calculates the optimal rigid body transformation until the convergence precision requirement of correct registration is met. The purpose of the ICP algorithm is to find a rotation parameter R and a translation parameter T between point cloud data to be registered and reference cloud data, so that the optimal matching between the two points of data meets a certain measurement criterion. Thereby obtaining stable feature points, i.e., target feature points.
It should be noted that, in the embodiment of the present application, when the number of target feature points is determined, the number is determined in different image acquisition environments respectively. Correspondingly, the number of the target feature points may be determined when the image acquisition time environments are different, although the process of screening the target feature points is performed in real time, in order to reduce the occupation of processing resources and better meet the requirements of practical application scenarios, the number may be determined after a fixed period and compared, for example, the number of the target feature points is obtained every 10 seconds, so as to determine whether to switch the display mode of the target device according to the number of the target feature points. In addition, the number of target feature points may also be obtained based on a change in an image acquisition space environment, for example, when an acquired image includes a certain reference object, the number of target feature points is obtained once, and then the number of target feature points is obtained again when a user gazes at a blank wall, so that whether to switch the display mode is determined according to the number. It is also possible to acquire the number of target feature points based on a difference in the image capturing scene environment, for example, determine the number of target feature points in a normal environment, i.e., an indoor environment, determine the number of target feature points once again when the wearing environment of the target device is in a traveling car, and then determine whether to switch the display mode of the target device.
In the embodiment of the application, the switching control of the display mode of the target device is a real-time judgment process, namely the display mode is switched in real time according to different image acquisition environments of the target device. If the third number of the target characteristic points in the third state is determined, whether the current second mode is switched to the first mode is determined according to the ratio of the third number to the second number. The third state is a state different from the second state.
Specifically, in the embodiment of the present application, determining whether to switch the first mode to the second mode based on the ratio of the first number to the second number includes:
and if the ratio of the first number to the second number is smaller than a target threshold, switching the first mode to a second mode to display the virtual image by the target device in the second mode, wherein the amounts of the pose data for generating the virtual image, which are respectively obtained in the first mode and the second mode, are different.
The target threshold may be a critical value for switching the display mode according to an actual application scenario. Taking the target device as an AR device for example, the AR device obtains spatial midpoint cloud data through a depth camera, repeatedly filters based on the feature points to obtain a first number of stable feature points, uses the stable feature points at this time as target feature points, and assumes the corresponding number as N, during this continuous feature point filtering process, determines whether switching of the display mode is required according to the ratio of the latest stable feature point number M to the initially filtered feature point N, and if the value of N/M is lower than the threshold value a, switches the first mode to the second mode, and when the value of N/M is higher than or equal to the threshold value a, switches to the first mode again. It should be noted that the number of pieces of pose data for generating a virtual image obtained in the first mode and the second mode, respectively, is different, that is, the degree of freedom of the rotation angle for generating the pose data in the first mode and the second mode.
In the embodiment of the present application, the mode in which the target device displays the virtual image may further include a third mode in addition to the first mode and the second mode, where pose data of the three modes are different, and the pose data includes the degrees of freedom of the rotation angle, and it may be that the first mode includes the most degrees of freedom, and the third mode includes the least degrees of freedom. For example, the first mode corresponds to a 6DoF mode, i.e., including 6 degrees of freedom in rotation angle, the second mode corresponds to a 3DoF mode, i.e., including 3 degrees of freedom in rotation angle, and the third mode corresponds to a Head loaded mode (Head aiming mode), i.e., a mode in which no anchor is made, and only a display mode in which an association relationship between the Head and the eyes is established is used.
Specifically, when determining whether to switch to the third mode, the following operations may be performed:
if the target equipment displays the virtual image in a second mode, acquiring first attitude data in a calculation mode corresponding to the second mode based on the current target characteristic point;
obtaining second posture data in a calculation mode corresponding to the first mode based on the current target feature point;
and if the difference value of the first position posture data and the second position posture data is larger than a target difference value, switching the second mode to a third mode.
And the pose data corresponding to the third mode is different from the first pose data corresponding to the second mode. It should be noted that, when the target device displays the virtual image in the second mode, whether to switch the second mode to the first mode or to switch the second mode to the third mode may be determined based on a change or calculation of the target feature point.
The calculation modes of the pose data corresponding to the display modes of different virtual images are different, and the data sources obtained in the calculation process are different, that is, the acquisition units for obtaining the data sources are different. For example, the first mode is a 6DoF mode, the second mode is a 3DoF mode, and the third mode is a Head loaded mode. In the 6DoF mode, the SLAM calculation main data source is two parts, namely a depth Camera (Camera) and an IMU (Inertial measurement unit), in the 3DoF display mode, the target device mainly calculates the pose data according to the IMU data, and if the pose data is calculated by adopting a calculation mode corresponding to the 6DoF mode, the obtained pose data has a larger difference with the coordinate value of the pose data obtained by only adopting the calculation mode corresponding to the IMU data due to the loss of the data acquired by the depth Camera. The reliability of the data calculated in the 6DoF mode is judged by these two values, and when the reliability is lower than standard B, the display mode of the target device is switched to the Head loaded mode. The IMU provides data including accelerations on three axes and gyro data of the three axes, namely angular velocity data, and can predict the position at the next moment through measurement of the accelerations and the angular velocity.
Therefore, when the display mode of the target equipment is a 6DoF mode, whether the display mode is switched to a 3DoF display mode is judged according to the numerical value proportion of the stable characteristic points, when the display mode is the 3DoF display mode, the credibility of the 6DoF data is judged according to the data obtained by calculating the 6DoF data and the data obtained by calculating only through the IMU according to the current depth camera and IMU fusion data, whether the display mode is switched to a Head loaded mode is judged according to the credibility, and the target equipment can be adapted to different use environments by switching the modes.
Referring to fig. 2, which shows a schematic diagram of an application scenario provided in an embodiment of the present application, in fig. 2, a user wears AR glasses, and recognizes an environment through the AR glasses to calculate coordinates of a virtual image displayed in a space, where the virtual image refers to an image of a tree in fig. 2, and a scene is a scene where a house is located, so that the user can see the virtual tree superimposed in the scene where the house is located through the AR glasses. However, the environment of the user wearing the AR glasses is not fixed, and some environments are not beneficial to the identification and calculation of the AR glasses, for example, a blank wall surface, an environment which continuously moves and changes in rotation, such as a driving automobile, an airplane and other places, may cause inaccurate anchoring of the virtual image in the space due to the fact that the characteristic points of the space environment are not done, or the continuous change causes inaccurate anchoring of the virtual image in the space, wherein the anchoring means a mode of recording the position of the virtual object in the image corresponding to the real scene. The switching to any display mode, such as the first mode, the second mode or the third mode, may be determined by the number of target points or by pose data calculated from the obtained target points. For a detailed description of how to switch, reference is made to the description of the above embodiments, which is not described in detail herein. It should be noted that the virtual image seen by the user in fig. 2 is an image corresponding to one of the display modes, and when the display mode is switched to a different display mode, the image seen by the user is matched with the corresponding display mode. Therefore, in the embodiment of the application, the target device can adapt to different use environments by switching the display mode of the target device.
There is also provided in an embodiment of the present application a control apparatus, see fig. 3, including:
the acquiring unit 10 is configured to respond to display of a virtual image by a target device in a first mode, and acquire a target feature point, where the target feature point represents a feature point with a consistent display coordinate position in two collected images in front and at back;
a first determining unit 20, configured to determine a first number of the target feature points in a first state;
a second determining unit 30, configured to determine a second number of the target feature points in a second state, where image capturing environments of the target device in the first state and the second state are different;
a third determining unit 40 configured to determine whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode being different in pose data utilized in generating the virtual image.
The embodiment of the application provides a control device, which responds to the display of a virtual image by target equipment in a first mode, and acquires target characteristic points representing characteristic points with consistent display coordinate positions in two collected images before and after the target characteristic points represent; determining a first number of target feature points in a first state; determining a second number of target characteristic points in a second state, wherein the image acquisition environments of the target equipment in the first state and the second state are different; whether to switch the first mode to the second mode is determined based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image. The switching of the display modes is realized based on the number of the target characteristic points in different image acquisition environments, so that the display modes of the target equipment are matched with the environments, and the actual application requirements are met.
Optionally, the obtaining unit 10 includes:
the first determining subunit is used for responding to the left-eye image and the right-eye image acquired by the target equipment, and determining the characteristic points with consistent display coordinate positions in the left-eye image and the right-eye image as target characteristic points;
or,
the second determining subunit is used for responding to the left-eye two-dimensional image and the right-eye two-dimensional image acquired by the target device, and performing spatial projection on the left-eye two-dimensional image to obtain a first spatial point; performing spatial projection on the right-eye two-dimensional image to obtain a second spatial point; and determining the space point with consistent space coordinates in the first space point and the second space point as a target characteristic point.
Optionally, the obtaining unit 10 further includes:
the third determining subunit is configured to acquire a feature point at a first time and a feature point at a second time, where the first time and the second time have a time-sequential association relationship; and determining the feature point with the consistent display position in the feature point at the first moment and the feature point at the second moment as a target feature point.
Optionally, the image capturing environment comprises one of an image capturing temporal environment, an image capturing spatial environment, and an image capturing scene environment.
Optionally, the third determining unit includes:
and a first switching subunit, configured to switch the first mode to a second mode if a ratio of the first number to the second number is smaller than a target threshold, so that the target device displays a virtual image in the second mode, where the numbers of pose data used for generating the virtual image, which are obtained in the first mode and the second mode respectively, are different.
Optionally, the apparatus further comprises:
the second switching subunit is used for determining a third number of the target characteristic points in a third state; and if the ratio of the third quantity to the second quantity is not less than the target threshold, switching the second mode to the first mode.
Optionally, the apparatus further comprises:
the third switching subunit is used for acquiring first pose data in a calculation mode corresponding to the second mode based on the current target feature point if the target equipment displays the virtual image in the second mode; obtaining second posture data in a calculation mode corresponding to the first mode based on the current target feature point; and if the difference value of the first position posture data and the second position posture data is larger than the target difference value, switching the second mode to a third mode, wherein the position posture data corresponding to the third mode is different from the first position posture data corresponding to the second mode.
It should be noted that, for the specific implementation of each unit in the present embodiment, reference may be made to the corresponding content in the foregoing, and details are not described here.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. The technical scheme in the embodiment is mainly used for matching the display mode of the target equipment with the environment, and meets the requirement of practical application.
Specifically, the electronic device in this embodiment may include the following structure:
a memory 401 for storing an application program and data generated by the application program;
a processor 402 for executing the application to implement:
responding to the display of a virtual image by target equipment in a first mode, and acquiring target feature points, wherein the target feature points represent feature points with consistent display coordinate positions in two collected images in front and at back;
determining a first number of the target feature points in a first state;
determining a second number of the target characteristic points in a second state, wherein image acquisition environments of the target equipment in the first state and the second state are different;
determining whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image.
According to the technical scheme, in the electronic equipment provided by the application, the target characteristic points are obtained in response to the fact that the target equipment displays the virtual image in the first mode, and the target characteristic points represent characteristic points with the same display coordinate position in the two collected images in the front frame and the back frame; determining a first number of target feature points in a first state; determining a second number of target characteristic points in a second state, wherein the image acquisition environments of the target equipment in the first state and the second state are different; whether to switch the first mode to the second mode is determined based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image. The switching of the display modes is realized based on the number of the target characteristic points in different image acquisition environments, so that the display modes of the target equipment are matched with the environments, and the actual application requirements are met.
It should be noted that, in the present embodiment, reference may be made to the corresponding contents in the foregoing, and details are not described here.
There is also provided in an embodiment of the present application a storage medium storing computer-executable instructions for performing the control method as described in any one of the above when executed by a processor.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A control method, comprising:
responding to the display of a virtual image by target equipment in a first mode, and acquiring target feature points, wherein the target feature points represent feature points with consistent display coordinate positions in two collected images in front and at back;
determining a first number of the target feature points in a first state;
determining a second number of the target characteristic points in a second state, wherein image acquisition environments of the target equipment in the first state and the second state are different;
determining whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image.
2. The method of claim 1, the obtaining target feature points, comprising:
responding to the left-eye image and the right-eye image collected by the target equipment, and determining the characteristic points with consistent display coordinate positions in the left-eye image and the right-eye image as target characteristic points;
or,
responding to a left-eye two-dimensional image and a right-eye two-dimensional image acquired by the target equipment, and performing spatial projection on the left-eye two-dimensional image to obtain a first spatial point;
performing spatial projection on the right-eye two-dimensional image to obtain a second spatial point;
and determining the space point with consistent space coordinates in the first space point and the second space point as a target characteristic point.
3. The method of claim 1, the obtaining target feature points, comprising:
acquiring a feature point at a first moment and a feature point at a second moment, wherein the first moment and the second moment have a time-sequence incidence relation;
and determining the feature point with the consistent display position in the feature point at the first moment and the feature point at the second moment as a target feature point.
4. The method of claim 1, the image acquisition environment comprising one of an image acquisition temporal environment, an image acquisition spatial environment, and an image acquisition scene environment.
5. The method of claim 1, the determining whether to switch the first mode to a second mode based on a ratio of the first number to the second number comprising:
and if the ratio of the first number to the second number is smaller than a target threshold, switching the first mode to a second mode to display the virtual image by the target device in the second mode, wherein the amounts of the pose data for generating the virtual image, which are respectively obtained in the first mode and the second mode, are different.
6. The method of claim 5, further comprising:
determining a third number of the target feature points in a third state;
and if the ratio of the third quantity to the second quantity is not less than the target threshold, switching the second mode to the first mode.
7. The method of claim 1, further comprising:
if the target equipment displays the virtual image in a second mode, acquiring first attitude data in a calculation mode corresponding to the second mode based on the current target characteristic point;
obtaining second posture data in a calculation mode corresponding to the first mode based on the current target feature point;
and if the difference value of the first position posture data and the second position posture data is larger than the target difference value, switching the second mode to a third mode, wherein the position posture data corresponding to the third mode is different from the first position posture data corresponding to the second mode.
8. A control device, comprising:
the acquisition unit is used for responding to the display of a virtual image of target equipment in a first mode and acquiring target characteristic points representing characteristic points with consistent display coordinate positions in the front and rear collected images;
a first determining unit, configured to determine a first number of the target feature points in a first state;
the second determining unit is used for determining a second number of the target characteristic points in a second state, and image acquisition environments of the target equipment in the first state and the second state are different;
a third determination unit configured to determine whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode differing in pose data utilized in generating the virtual image.
9. A storage medium storing computer-executable instructions for performing the control method of any one of claims 1 to 7 when executed by a processor.
10. An electronic device, comprising:
a memory for storing an application program and data generated by the application program running;
a processor for executing the application to implement:
responding to the display of a virtual image by target equipment in a first mode, and acquiring target feature points, wherein the target feature points represent feature points with consistent display coordinate positions in two collected images in front and at back;
determining a first number of the target feature points in a first state;
determining a second number of the target characteristic points in a second state, wherein image acquisition environments of the target equipment in the first state and the second state are different;
determining whether to switch the first mode to a second mode based on a ratio of the first number to the second number, the first mode and the second mode utilizing different pose data when generating the virtual image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110188853.2A CN112819970B (en) | 2021-02-19 | 2021-02-19 | Control method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110188853.2A CN112819970B (en) | 2021-02-19 | 2021-02-19 | Control method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112819970A true CN112819970A (en) | 2021-05-18 |
| CN112819970B CN112819970B (en) | 2023-12-26 |
Family
ID=75865488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110188853.2A Active CN112819970B (en) | 2021-02-19 | 2021-02-19 | Control method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112819970B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113932805A (en) * | 2021-10-12 | 2022-01-14 | 天翼数字生活科技有限公司 | Method for improving positioning accuracy and speed of AR virtual object |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150077434A1 (en) * | 2012-04-23 | 2015-03-19 | Sony Corporation | Image processing apparatus, image processing method, and program |
| US20170201709A1 (en) * | 2014-08-01 | 2017-07-13 | Sony Corporation | Information processing apparatus, information processing method, and program |
| CN107705333A (en) * | 2017-09-21 | 2018-02-16 | 歌尔股份有限公司 | Space-location method and device based on binocular camera |
| WO2018235923A1 (en) * | 2017-06-21 | 2018-12-27 | 国立大学法人 東京大学 | Position estimating device, position estimating method, and program |
| US20200410763A1 (en) * | 2019-06-28 | 2020-12-31 | Snap Inc. | 3d object camera customization system |
| CN112258658A (en) * | 2020-10-21 | 2021-01-22 | 河北工业大学 | Augmented reality visualization method based on depth camera and application |
-
2021
- 2021-02-19 CN CN202110188853.2A patent/CN112819970B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150077434A1 (en) * | 2012-04-23 | 2015-03-19 | Sony Corporation | Image processing apparatus, image processing method, and program |
| US20170201709A1 (en) * | 2014-08-01 | 2017-07-13 | Sony Corporation | Information processing apparatus, information processing method, and program |
| WO2018235923A1 (en) * | 2017-06-21 | 2018-12-27 | 国立大学法人 東京大学 | Position estimating device, position estimating method, and program |
| CN107705333A (en) * | 2017-09-21 | 2018-02-16 | 歌尔股份有限公司 | Space-location method and device based on binocular camera |
| US20200410763A1 (en) * | 2019-06-28 | 2020-12-31 | Snap Inc. | 3d object camera customization system |
| CN112258658A (en) * | 2020-10-21 | 2021-01-22 | 河北工业大学 | Augmented reality visualization method based on depth camera and application |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113932805A (en) * | 2021-10-12 | 2022-01-14 | 天翼数字生活科技有限公司 | Method for improving positioning accuracy and speed of AR virtual object |
| CN113932805B (en) * | 2021-10-12 | 2024-02-23 | 天翼数字生活科技有限公司 | Method for improving positioning accuracy and speed of AR virtual object |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112819970B (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11748907B2 (en) | Object pose estimation in visual data | |
| US10269177B2 (en) | Headset removal in virtual, augmented, and mixed reality using an eye gaze database | |
| JP6359644B2 (en) | Method for facilitating computer vision application initialization | |
| WO2019022803A1 (en) | Controller tracking for multiple degrees of freedom | |
| US11783443B2 (en) | Extraction of standardized images from a single view or multi-view capture | |
| TW202013149A (en) | Augmented reality image display method, device and equipment | |
| EP3128413A1 (en) | Sharing mediated reality content | |
| WO2004088348A1 (en) | Eye tracking system and method | |
| US10838515B1 (en) | Tracking using controller cameras | |
| US12026903B2 (en) | Processing of depth maps for images | |
| US11275434B2 (en) | Information processing apparatus, information processing method, and storage medium | |
| WO2020090316A1 (en) | Information processing device, information processing method, and program | |
| JP6723743B2 (en) | Information processing apparatus, information processing method, and program | |
| JP5086824B2 (en) | TRACKING DEVICE AND TRACKING METHOD | |
| CN112819970B (en) | Control method and device and electronic equipment | |
| CN117611664A (en) | Image information and inertial navigation data fusion method | |
| CN115223231A (en) | Sight direction detection method and device | |
| WO2024013137A1 (en) | Method and system for optimum positioning of cameras for accurate rendering of a virtual scene | |
| KR101473234B1 (en) | Method and system for displaying an image based on body tracking | |
| JP6487545B2 (en) | Recognition calculation device, recognition calculation method, and recognition calculation program | |
| US11972549B2 (en) | Frame selection for image matching in rapid target acquisition | |
| US20230122185A1 (en) | Determining relative position and orientation of cameras using hardware | |
| CN117710445A (en) | Target positioning method and device applied to AR equipment and electronic equipment | |
| Heidari et al. | A Solution for Cross-Calibration of Gaze Tracker System and Stereoscopic Scene System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |