CN113992885B - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN113992885B CN113992885B CN202111105917.4A CN202111105917A CN113992885B CN 113992885 B CN113992885 B CN 113992885B CN 202111105917 A CN202111105917 A CN 202111105917A CN 113992885 B CN113992885 B CN 113992885B
- Authority
- CN
- China
- Prior art keywords
- image
- moment
- gaze
- display output
- electronic device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
Abstract
The application discloses a data synchronization method and a device, wherein the method comprises the following steps: predicting a gaze position of a current user in a first display output region of a first electronic device at a first time, the first time being a time after the current time; obtaining a first image to be output to a first display output area at a first moment; sending the first image and the gaze position to the second electronic device, so that the second electronic device outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic device at the first moment; wherein the second image comprises at least part of the first image. Through implementing the application, the user of the second electronic equipment can be synchronously provided with the current user at the watching position of the first display output area of the first electronic equipment in real time, and the efficiency and the accuracy of the cooperative work are improved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data synchronization method and apparatus.
Background
At present, AR, VR, MR devices are widely used in systems such as remote expert consultation, remote expert assistance, and the like. However, AR, VR, and MR devices only transmit the image currently operated by the operator to the remote expert, and the remote expert cannot know the current gaze point of the operator on the image only through the image, so that the efficiency and accuracy of the cooperative work are greatly reduced.
Disclosure of Invention
In a first aspect, an embodiment of the present application provides a data synchronization method, including: predicting a gaze position of a current user at a first time in a first display output region of a first electronic device, the first time being a time after the current time; obtaining a first image to be output to a first display output area at a first moment; the first image and the gaze position are sent to second electronic equipment, so that the second electronic equipment outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic equipment at a first moment; wherein the second image comprises at least part of the first image.
Optionally, predicting the gaze location of the current user at the first time on the first display output region of the first electronic device comprises: the method comprises the steps of obtaining gaze position change information of a current user at a plurality of adjacent moments before a first moment, and predicting a gaze position of the current user at the first moment in a first display output area of first electronic equipment at least based on the gaze position change information; or obtaining gaze information of the current user at a plurality of adjacent moments before the first moment, determining a matched prediction model based on the gaze information, and inputting the gaze information into the prediction model to obtain a gaze position of the current user at the first display output area of the first electronic device at the first moment.
Optionally, predicting the gaze location of the current user at the first display output region of the first electronic device at the first time based on at least the gaze location change information includes: determining the maximum variation range of the gazing position of the current user in a first time period at a first moment, and predicting the gazing position at least based on the gazing position variation information and the maximum variation range; or, determining transmission delay information between the first electronic device and the second electronic device, and predicting the gaze position based on the transmission delay information and the gaze position change information;
alternatively, determining a matching predictive model based on the gaze information includes: the method comprises the steps of obtaining change characteristics of gazing information and display output data of a first display output area, determining a matched prediction model based on the change characteristics, and predicting a gazing position of a current user in the first display output area of first electronic equipment at a first moment by using the prediction model based on the gazing information and the display output data.
Optionally, obtaining a first image to be output to the first display output area at the first moment includes: acquiring a third image acquired by a second camera of the first electronic equipment at a second moment, and processing the third image at least based on the predicted gaze position to obtain a first image, wherein the second moment is earlier than the first moment, and the viewing range of the second camera is larger than that of the first camera; or obtaining a third image acquired by a second camera of the first electronic device at a second moment and a fourth image acquired by the first camera at the third moment, and processing the third image and the fourth image to obtain a first image, wherein the second moment and the third moment are both earlier than the first moment, the second moment is the same as or different from the third moment, and the viewing range of the second camera is larger than that of the first camera; or obtaining a fourth image acquired by a first camera of the first electronic device at a third moment, and processing the fourth image at least based on the predicted gaze position to obtain the first image, wherein the third moment is earlier than the first moment.
Optionally, processing the third image and the fourth image to obtain the first image includes: obtaining first gaze position change information of the predicted gaze position at the first time relative to the gaze position at the third time, and synthesizing and/or compensating the fourth image with the third image by referring to the first gaze position change information to obtain a first image; or acquiring a corresponding relation between the first display output area and the gaze position at the third moment and the predicted first moment, and synthesizing and/or compensating the fourth image by using the third image based on the change information of the corresponding relation to obtain a first image; or, processing the third image based on at least the predicted gaze location to obtain the first image, comprising: and cutting the third image based on the predicted gaze position to obtain the first image.
Optionally, the sending the first image and the gaze location to the second electronic device comprises: if the difference of the first image relative to at least one frame of image at the adjacent moment earlier than the first moment is in a first range, sending the first image and the gazing position to the second electronic equipment; or if the difference of the first image relative to at least one frame of image at the adjacent moment earlier than the first moment is in the second range, the determined corresponding content data and/or relationship data between the first image and the watching position are sent to the second electronic equipment.
Optionally, the step of sending the determined content data and/or relationship data between the first image and the gaze location to the second electronic device comprises: determining image content corresponding to the watching position in the first image, and sending the image content to the second electronic equipment; or determining the position relation between the gaze position and the second display output area based on the relation between the gaze position and the first image and the configuration relation between the first display output area and the second display output area, and sending the position relation to the second electronic device; or determining the image content corresponding to the gazing position in the first image and the relative position relationship between the image content and the gazing position, and sending the image content and the relative position relationship to the second electronic equipment.
Optionally, outputting the second image and the prompt information corresponding to the gazing position in the second display output area includes: and acquiring the configuration relation between the first display output area and the second display output area, and processing the first image and the gaze position at least based on the configuration relation so as to output the second image and the prompt information corresponding to the gaze position in the second display output area.
Optionally, processing the first image and the gaze location based on at least the configuration relationship to output the second image and the prompt information corresponding to the gaze location in the second display output area includes: configuring a difference processing first image based on display identification information between the first display output area and the second display output area so that a second image obtained by processing the first image matches the display identification information of the second display output area; and/or processing a first relative position relation between the first image and the gazing position based on the configuration relation, and configuring and outputting prompt information for highlighting the gazing position in a second image or a second display output area correspondingly by using a second relative position relation obtained by processing the first relative position relation.
In a second aspect, an embodiment of the present application provides a data synchronization apparatus, including: the prediction module is used for predicting the watching position of a current user in a first display output area of first electronic equipment at a first moment, wherein the first moment is a moment after the current moment; the device comprises an obtaining module, a display module and a display module, wherein the obtaining module is used for obtaining a first image to be output to a first display output area at a first moment; the sending module is used for sending the first image and the gazing position to the second electronic equipment so that the second electronic equipment can output a second image and prompt information corresponding to the gazing position in a second display output area of the second electronic equipment at a first moment; wherein the second image comprises at least part of the first image.
In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of data synchronization as in the first aspect or any of the embodiments of the first aspect.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to enable a computer to execute the data synchronization method in the first aspect or any implementation manner of the first aspect.
According to the data synchronization method and device, the gaze position of a current user in a first display output area of first electronic equipment at a first moment is predicted, and the first moment is a moment after the current moment; obtaining a first image to be output to a first display output area at a first moment; sending the first image and the gaze position to the second electronic device, so that the second electronic device outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic device at the first moment; wherein the second image comprises at least part of the first image; therefore, the images seen by the current user of the first electronic device and the user of the second electronic device at the first moment can be synchronized, and the gaze position of the current user of the first electronic device at the first moment is predicted at the current moment, so that when the prompt message corresponding to the gaze position is output on the second electronic device, the predicted gaze position can eliminate the delay of the prompt message of the output gaze position on the second electronic device caused by the network data transmission delay, and therefore, the actual gaze position of the current user of the first electronic device at the first moment is equivalent to the position synchronously given to the user of the second electronic device, so that the user of the second electronic device can know the position of the gaze image of the current user of the first electronic device in real time, and the efficiency and accuracy of cooperative work in systems such as remote expert consultation, remote expert assistance, remote teaching, remote game, remote diagnosis, remote design and manufacture and the like can be improved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
Fig. 1 is a schematic flowchart of a directory combining method based on a display interface in an embodiment of the present application;
FIG. 2 is a schematic diagram of a region of a gaze location formed by a plurality of gaze points in an embodiment of the present application;
FIG. 3 is a schematic view of a motion profile in an embodiment of the present application;
FIG. 4 is a diagram illustrating a fourth image being panned according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application;
fig. 6 is a schematic diagram of a hardware structure of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the present application provides a data synchronization method, which can be applied to AR, VR, and MR devices, and is used in cooperative work such as remote expert consultation, remote expert assistance, remote teaching, remote game, remote diagnosis, remote design and manufacture, and as shown in fig. 1, the data synchronization method includes:
s101, predicting a fixation position of a current user in a first display output area of first electronic equipment at a first moment, wherein the first moment is a moment after the current moment.
In this embodiment, the first electronic device may be any of AR, VR, and MR devices. The first display output area is a display screen of the first electronic device. The first display output area is for displaying an image. When the image is displayed in the first display output area, a gaze location is generated in the first display output area when the image is viewed by the current user's eyes. Therefore, in the embodiment, the eyeball tracking technology may be adopted to obtain the gaze position of the current user in the first display output area of the first electronic device in real time, and then predict the gaze position of the current user in the first display output area of the first electronic device at the first moment based on the gaze position obtained in real time.
Currently, most eye tracking systems use an infrared camera to capture an image of the eye, then process the captured image of the eye to determine the location of the pupil, and then estimate the direction and location of the eye gaze depending on the detected pupil location. The image processing process may sometimes be done on a PC, cell phone or other processor. Another method is a pupil-cornea reflex tracking method. The eye fixation position is determined by firstly shooting an eye image by using an eye camera, then obtaining the pupil center position through image processing, then taking a cornea reflection point (yellow spot) as a base point of the relative position of the eye camera and an eyeball, and obtaining the sight line vector coordinate according to the pupil center obtained through the image processing.
S102, a first image to be output to a first display output area at a first moment is obtained.
In this embodiment, at least a first camera is disposed on the first electronic device, and the first camera collects an image in real time, and then the first electronic device may process or not process the image and display the image in the first display output area. For situations that are not addressed, for example, the current user of the first electronic device is consulting a patient, the first camera may capture images of the patient in real-time. For the processing situation, the global image acquired by the first camera in real time can be processed into a local image, and then the local image is displayed in the first display output area; the image acquired before the first time may be processed using the predicted gaze position, the image that the first camera should acquire at the first time may be predicted, and the predicted image at the first time may be displayed in the first display output area at the first time.
S103, sending the first image and the gaze position to second electronic equipment so that the second electronic equipment outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic equipment at a first moment; wherein the second image comprises at least part of the first image.
In this embodiment, the second electronic device is disposed at a far end, and the second display output area may be a display screen of the second electronic device. In order for a remote user, such as a remote expert, to view the image viewed by the current user of the first electronic device, and to know that the current user of the first electronic device is currently gazing at a specific location of the first image, the first image and the gazing location need to be transmitted to the second electronic device. After receiving the first image and the gaze position, the second electronic device may form the first image according to the first image, output the second image in the second display output area, and form the prompt information at least according to a relationship between the gaze position and the first image.
The second image may be a global image, a local image, or a combination of global and local images of the first image, etc. The prompt message may be displayed by adding a virtual identifier, adjusting a display parameter (highlighted or highlighted), and the like. For example, the gaze location may be identified in the second image such that the image currently seen by the user of the first electronic device and the gaze point formed on the image may be displayed on the second display screen of the second electronic device in real time such that the images seen by the users of the first and second electronic devices and the gaze point on the images may be kept synchronized. The presentation mode of the prompt information and the second image can be a display layout, a display mode capable of enhancing the prompt effect, and the like.
According to the data synchronization method provided by the embodiment of the application, the gaze position of a current user in a first display output area of first electronic equipment at a first moment is predicted, and the first moment is a moment after the current moment; obtaining a first image to be output to a first display output area at a first moment; sending the first image and the gaze position to the second electronic device, so that the second electronic device outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic device at the first moment; wherein the second image comprises at least part of the first image; therefore, the images seen by the current user of the first electronic device and the user of the second electronic device at the first moment can be synchronized, and the gaze position of the current user of the first electronic device at the first moment is predicted at the current moment, so that when the prompt message corresponding to the gaze position is output on the second electronic device, the predicted gaze position can eliminate the delay of the prompt message of the output gaze position on the second electronic device caused by the network data transmission delay, and therefore, the actual gaze position of the current user of the first electronic device at the first moment is equivalent to the position synchronously given to the user of the second electronic device, so that the user of the second electronic device can know the position of the gaze image of the current user of the first electronic device in real time, and the efficiency and accuracy of cooperative work in systems such as remote expert consultation, remote expert assistance, remote teaching, remote game, remote diagnosis, remote design and manufacture and the like can be improved.
In an alternative embodiment, the step S101 of predicting the gaze location of the current user at the first time on the first display output region of the first electronic device includes: the method comprises the steps of obtaining gaze position change information of a current user at a plurality of adjacent moments before a first moment, and predicting a gaze position of the current user at the first moment in a first display output area of first electronic equipment at least based on the gaze position change information.
In the present embodiment, the gaze position change information includes the movement speed, the movement direction, the movement trajectory, and the like of the gaze position.
Since the average value of the pause time of human vision is 170ms, the region range of the gaze position every 170ms can be calculated by taking the interval of 170ms as one period. The first time is in a first time interval, a plurality of adjacent times before the first time are in a plurality of adjacent second time intervals before the first time, and the duration of the second time intervals is the same as that of the first time intervals. Since the current eye tracking device can acquire up to 2000 binocular images per second, and can calculate the coordinate position of the current user watching the screen within less than 3 milliseconds from the captured eye images, and feed back the coordinate position to the first electronic device, each 170ms period includes a plurality of gaze points, as shown in fig. 2, the plurality of gaze points form an area range of gaze positions. If the regions of the current actual gazing positions of the user are 540, 541, 542, 543, 545 and the predicted region of the gazing position of the eyeball is 592, the motion normal vectors from 540 to 592 can be obtained.
In this embodiment, the rate of transmitting the image from the first electronic device to the second electronic device is 34 ms/frame, and each 170ms includes 5 frames of images, and for the gaze position of each frame of image, the position average value of a plurality of gaze points generated when gazing at the frame of image may be taken to obtain the gaze position of each frame of image, and then the gaze position of 5 frames of images is averaged to obtain the gaze position of each 170ms formed gaze position region. As shown in fig. 3, the normal motion vectors (including the moving speed, moving direction, and moving trajectory of the gaze location) of the gaze locations of m 170ms consecutive gaze location regions are connected to form a curve, and the curve is fitted to the obtained motion curve according to the continuity of the human eye motion, thereby predicting the gaze location of the gaze location region formed by the next 170ms (for example, the x-th 170 ms). The gaze position of the first time period can be predicted based on the gaze position variation information (gaze position variation information of the actual gaze position region) of m second time periods before the first time period, since the first time is within the first time period, the gaze position of the first time is the gaze position of the first time period.
In this embodiment, since the eye movement has inertia, the gaze position of the current user in the first display and output area of the first electronic device at the first time can be predicted by the gaze position change information of the current user at a plurality of adjacent times before the first time, and the gaze position of the current user in the first display and output area of the first electronic device at the first time can be obtained quickly.
In an optional embodiment, predicting a gaze location of a current user at a first time on a first display output region of a first electronic device based at least on gaze location change information includes: determining the maximum variation range of the gazing position of the current user in the first time period at the first moment, and predicting the gazing position at least based on the gazing position variation information and the maximum variation range.
Specifically, since many gaze locations are actually generated during the first period, the maximum variation range of the gaze location of the current user during the first period at the first moment may be determined based on the actually generated gaze locations. And the gaze location for the first time period may be predicted based on the gaze location change information, so the gaze location for the first time may be determined based on the maximum range of change in the gaze location for the first time period and the predicted gaze location for the first time period. By predicting the gaze location based on at least the gaze location change information and the maximum change range, the predicted gaze location at the first time may be made more accurate.
In another optional embodiment, predicting the gaze location of the current user at the first time on the first display output region of the first electronic device based at least on the gaze location change information comprises: determining transmission delay information between the first electronic device and the second electronic device, and predicting a gaze position based on the transmission delay information and gaze position change information;
specifically, it may be considered to determine the latest time that can be accurately predicted according to the transmission delay information, such as considering to give the predicted gaze location at the first time to the second electronic device at a specific time before the first time according to the transmission delay information, so that the second electronic device can output the prompt information of the gaze location at the first time.
The transmission delay information between the first electronic device and the second electronic device can be determined through a plurality of tests, the gaze position of the first time period can be predicted based on the gaze position change information, and the gaze position of the first time can be determined according to the transmission delay information and the gaze position of the first time period. By predicting the gaze location based on the transmission delay information and the gaze location change information, the predicted gaze location at the first time may be made more accurate.
In an alternative embodiment, the step S101 of predicting the gaze location of the current user at the first time on the first display output region of the first electronic device includes: the method comprises the steps of obtaining gazing information of a current user at a plurality of adjacent moments before a first moment, determining a matched prediction model based on the gazing information, and inputting the gazing information into the prediction model to obtain a gazing position of the current user at the first display output area of first electronic equipment at the first moment.
In the present embodiment, the gaze information includes a gaze position, image data, and salient features in the image.
In an alternative embodiment, determining a matching predictive model based on the gaze information includes: the method comprises the steps of obtaining change characteristics of gazing information and display output data of a first display output area, determining a matched prediction model based on the change characteristics, and predicting a gazing position of a current user in the first display output area of the first electronic equipment at a first moment by using the prediction model based on the gazing information and the display output data.
Specifically, the display output data includes image data and salient features in the image. The changing characteristic of the gaze information may be a changing characteristic of the gaze location. The trained prediction models are different when the change characteristics of the gazing position are different, so that when the change characteristics are changed, the prediction models need to be adaptively changed, therefore, the change characteristics of the gazing information can be obtained based on the gazing information, and then the matched prediction models are determined based on the change characteristics.
The predictive model may be obtained by training a deep learning engine or neural network. The gaze point position of a user can be predicted by a deep learning engine or a neural network model based on eye movement speed, movement direction sequence, image data and significant features in image contents.
In an optional embodiment, if the current frame image acquired by the first camera is displayed in the first display output area in real time, step S102 is performed to obtain a first image to be output to the first display output area at the first moment. In this embodiment, the image collected by the first camera does not need to be processed, and then both the current users of the first electronic device and the second electronic device can see the original image shot by the first camera.
In an optional embodiment, the first electronic device may further include a second camera, wherein a viewing range of the second camera is larger than a viewing range of the first camera. For example, the first camera is a camera with a conventional video call function, the second camera is a wide-angle camera, the view range of the wide-angle camera includes the view range of the camera with the conventional video call function, and an image acquired by the wide-angle camera may include information in a wider range. When the images are transmitted through the network, a certain time delay exists, if the images displayed in the first display output area are completely synchronous with the images displayed in the second display output area, the third image acquired by the second camera at the second moment can be processed by combining the predicted gaze position, and the image which should be acquired by the first camera at the first moment is predicted, namely the first image to be displayed in the first display output area at the first moment. Step S102, obtaining a first image to be output to the first display output area at the first time may include: and obtaining a third image acquired by a second camera of the first electronic equipment at a second moment, and processing the third image at least based on the predicted gaze position to obtain the first image, wherein the second moment is earlier than the first moment.
In one embodiment, processing the third image based on at least the predicted gaze location to obtain the first image comprises: and cutting the third image based on the predicted gaze position to obtain the first image.
Specifically, since the predicted gaze position corresponds to the gaze image at the first time, the third image may be cropped according to the predicted gaze position, and the image at the predicted gaze position is retained, so as to obtain the first image, and since the gaze position is predicted, the first image is an image that the first camera should collect at the predicted first time.
In this embodiment, when the image is transmitted through the network, a certain delay exists, and the third image acquired by the second camera of the first electronic device at the second time is processed based on the predicted gaze position to obtain the predicted image that the first camera should acquire at the first time, so that when the first electronic device sends the first image to the second electronic device, the first image acquired by the second electronic device can cancel image frame lag caused by the network delay, and at the first time, the image displayed on the second electronic device and the image displayed on the first electronic device can be synchronized.
In an optional embodiment, because there is a certain delay when the images are transmitted over the network, if it is desired that the images displayed in the first display output area are synchronized with the images displayed in the second display output area, the images acquired by the first camera at the third time and the images acquired by the second camera at the second time may be synthesized, and the image that the first camera should acquire at the first time, that is, the first image to be output to the first display output area at the first time, is predicted. Step S102, obtaining a first image to be output to the first display output area at the first moment, including: and obtaining a third image acquired by a second camera of the first electronic equipment at a second moment and a fourth image acquired by the first camera at a third moment, and processing the third image and the fourth image to obtain a first image, wherein the second moment and the third moment are both earlier than the first moment, and the second moment is the same as or different from the third moment.
In an alternative embodiment, processing the third image and the fourth image to obtain the first image comprises: first gaze position change information of the predicted gaze position at the first time with respect to the gaze position at the third time is obtained, and the fourth image is synthesized and/or compensated using the third image with reference to the first gaze position change information, thereby obtaining a first image.
In particular, for head-mounted AR, VR, MR devices, the range of view of the acquired images may also change as the gaze location changes. Therefore, the first gaze position change information of the predicted gaze position at the first time relative to the gaze position at the third time may be obtained, and then the translation parameters of the image acquired by the first camera at the first time relative to the fourth image acquired at the third time, i.e. the distance and direction of the image movement, may be mapped based on the first gaze position change information. As shown in fig. 4, a new image can be obtained by translating the fourth image according to the translation parameters. And the new image has a part which does not comprise the fourth image, and the content of the part can be supplemented by the third image acquired by the second camera at the second moment. The location of the portion may be identified and then the contents of the location extracted from the third image filled into the new image, forming the first image.
In another alternative embodiment, processing the third image and the fourth image to obtain the first image includes: and acquiring a corresponding relation between the first display output area and the gaze position at the third moment and the predicted first moment, and synthesizing and/or compensating the fourth image by using the third image based on the change information of the corresponding relation to obtain the first image.
In this embodiment, the third image acquired by the second camera at the second time and the fourth image acquired by the first camera at the third time of the first electronic device are synthesized and/or compensated based on the first gaze position change information of the predicted gaze position at the first time relative to the gaze position at the third time, or based on the corresponding relationship between the third time and the predicted gaze position at the first time and the first display output area, so as to obtain the image that the first camera should acquire at the first time of the first electronic device, when the first electronic device transmits the first image to the second electronic device, the first image acquired by the second electronic device may counteract image picture lag caused by network delay, and at the first time, the second electronic device may synchronize with the image displayed on the first electronic device.
In an optional embodiment, since there is a certain delay in transmitting the images through the network, if the images displayed in the first display output area are intended to be synchronized with the images displayed in the second display output area, the fourth image acquired by the first camera at the third time may be processed in combination with the predicted gaze position, so as to predict the first image to be displayed in the first display output area at the first time. Step S102, obtaining a first image to be output to the first display output area at the first time, including: and acquiring a fourth image acquired by a first camera of the first electronic equipment at a third moment, and processing the fourth image at least based on the predicted gaze position to obtain the first image, wherein the third moment is earlier than the first moment.
Specifically, since the predicted gaze position corresponds to the gaze image at the first time, the fourth image acquired by the first camera at the third time may be cropped according to the predicted gaze position, and the image at the predicted gaze position is retained, so as to obtain the first image, and since the gaze position is predicted, the first image is the image that the first camera should acquire at the predicted first time.
In this embodiment, when the image is transmitted through the network, a certain delay exists, and the fourth image acquired by the first camera of the first electronic device at the third time is processed based on the predicted gaze position to obtain the predicted image that the first camera should acquire at the first time, so that when the first electronic device sends the first image to the second electronic device, the first image acquired by the second electronic device can cancel image frame lag caused by the network delay, and at the first time, the second electronic device and the image displayed on the first electronic device can be synchronized.
In an alternative embodiment, step S103, sending the first image and the gaze location to the second electronic device, includes: the first image and the gaze location are sent to the second electronic device if the first image differs from at least one frame of image at an adjacent time earlier than the first time by a first amount.
Specifically, if the difference of the first image with respect to at least one frame of image at the adjacent time earlier than the first time is within the first range, which indicates that the difference of the first image with respect to at least one frame of image at the adjacent time earlier than the first time is large, the first image and the gaze position need to be transmitted to the second electronic device, so that the second electronic device can display the image having the difference with the current image at the first time in time. The disparity may be determined based on changes in gaze parameters, including gaze movement speed, movement trajectory, and the like.
In an alternative embodiment, step S103, sending the first image and the gaze location to the second electronic device, includes: and if the difference of the first image relative to at least one frame of image at the adjacent moment earlier than the first moment is in a second range, sending the determined corresponding content data and/or relationship data between the first image and the gazing position to the second electronic equipment.
Specifically, if the difference of the first image with respect to at least one frame of image at the adjacent time earlier than the first time is in the second range, it indicates that the difference of the first image with respect to at least one frame of image at the adjacent time earlier than the first time is small, and at this time, it may be possible to transmit only the content data corresponding to the first image and the gaze position and/or determine the relationship data corresponding to the first image and the gaze position without transmitting the first image. The content data includes image content corresponding to the gaze position, and the image content may be an original-scale image corresponding to the gaze position or an enlarged image. The relationship data includes a relative positional relationship between the image content and the gaze position.
In an alternative embodiment, the step of sending the determined content data and/or relationship data between the first image and the gaze location to the second electronic device comprises: and determining the image content corresponding to the watching position in the first image, and sending the image content to the second electronic equipment.
Because the difference of the first image is smaller than that of at least one frame of image at the adjacent time which is earlier than the first time, the image content corresponding to the watching position in the first image is sent to the second electronic equipment, and then the second electronic equipment can directly form prompt information according to the image content to prompt the watching position of a user of the second electronic equipment.
In another alternative embodiment, the step of sending the determined content data and/or relationship data between the first image and the gaze location to the second electronic device comprises: and determining the position relation between the watching position and the second display output area based on the relation between the watching position and the first image and the configuration relation between the first display output area and the second display output area, and sending the position relation to the second electronic equipment.
Specifically, the configuration relationship between the first display output area and the second display output area may be a proportional relationship or a size relationship, or may be a difference in display configuration parameters, such as resolution, brightness, color temperature, and the like. The position relation between the watching position and the second display output area can be determined based on the relation between the watching position and the first image and the configuration relation between the first display output area and the second display output area, and after the position relation is sent to the second electronic device, the second electronic device can directly display the position relation without processing the position relation.
In another alternative embodiment, the step of sending the determined content data and/or relationship data between the first image and the gaze location to the second electronic device comprises: and determining the image content corresponding to the gazing position in the first image and the relative position relation between the image content and the gazing position, and sending the image content and the relative position relation to the second electronic equipment.
Specifically, the image content may be an enlarged image or an original-scale image corresponding to the gaze position, and the relative positional relationship between the image content and the gaze position may be determined based on the enlargement scale of the image content. And sending the image content and the relative position relation to the second electronic equipment, so that the second electronic equipment can form prompt information according to the image content and the relative position relation, and a user of the second electronic equipment is prompted to look at the position.
In an optional embodiment, outputting the second image and the prompt information corresponding to the gazing position in the second display output area includes: and acquiring the configuration relation between the first display output area and the second display output area, and processing the first image and the gaze position at least based on the configuration relation so as to output the second image and the prompt information corresponding to the gaze position in the second display output area.
In an optional embodiment, processing the first image and the gaze location based on at least the configuration relationship to output the second image and the prompt corresponding to the gaze location in the second display output area comprises: the first image is processed based on the display identification information configuration difference between the first display output region and the second display output region such that a second image resulting from processing the first image matches the display identification information of the second display output region.
Specifically, the Display Identification information includes Extended Display Identification Data (EDID). Since there may be a difference between the display identification information of the first display output area and the display identification information of the second display output area, the first image needs to be processed according to the difference so that the processed image (second image) can be matched with the display identification information of the second display output area.
In another alternative embodiment, processing the first image and the gaze location based on at least the configuration relationship to output the second image and the prompt corresponding to the gaze location in the second display output region includes: and processing a first relative position relation between the first image and the gazing position based on the configuration relation so as to configure and output prompt information for highlighting the gazing position in a second image or a second display output area correspondingly by using a second relative position relation obtained by processing the first relative position relation.
Specifically, since there may be a mismatch in the ratio or size between the first display output region and the second display output region, when the second image is displayed in the second display output region, in order to match the cue information that enables the gaze position with the second image, it is necessary to convert the first relative positional relationship between the first image and the gaze position according to the size ratio of the first display output region and the second display output region that are gazing between the first image and the gaze position at the first display output region to obtain the second relative positional relationship, so that the cue information of the gaze position that is configured according to the second relative positional relationship and displayed in the second display output region matches the first relative positional relationship displayed in the first display output region.
An embodiment of the present application further provides a data synchronization apparatus, as shown in fig. 5, including:
the prediction module 21 is configured to predict a gaze position of a current user in a first display output area of the first electronic device at a first time, where the first time is a time after the current time; the detailed description of the specific implementation manner is given in step S101 of the above method embodiment, and is not repeated herein.
An obtaining module 22, configured to obtain a first image to be output to a first display output area at a first time; the detailed description of the specific implementation manner is given in step S102 of the above method embodiment, and is not repeated herein.
The sending module 23 is configured to send the first image and the gaze position to the second electronic device, so that the second electronic device outputs the second image and the prompt information corresponding to the gaze position in the second display output area of the second electronic device at the first moment; wherein the second image comprises at least part of the first image. For details of the specific implementation, reference is made to the description of step S103 in the above method embodiment, and details are not described herein again.
The data synchronization device provided by the embodiment of the application predicts the gaze position of the current user in the first display output area of the first electronic device at the first moment, wherein the first moment is a moment after the current moment; obtaining a first image to be output to a first display output area at a first moment; sending the first image and the gaze position to the second electronic device, so that the second electronic device outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic device at the first moment; wherein the second image comprises at least part of the first image; therefore, the images seen by the current user of the first electronic device and the user of the second electronic device at the first moment can be synchronized, and the gaze position of the current user of the first electronic device at the first moment is predicted at the current moment, so that when the prompt message corresponding to the gaze position is output on the second electronic device, the predicted gaze position can eliminate the delay of the prompt message of the output gaze position on the second electronic device caused by the network data transmission delay, and therefore, the actual gaze position of the current user of the first electronic device at the first moment is equivalent to the position synchronously given to the user of the second electronic device, so that the user of the second electronic device can know the position of the gaze image of the current user of the first electronic device in real time, and the efficiency and accuracy of cooperative work in systems such as remote expert consultation, remote expert assistance, remote teaching, remote game, remote diagnosis, remote design and manufacture and the like can be improved.
Based on the same inventive concept as the data synchronization method in the foregoing embodiment, an embodiment of the present application further provides an electronic device, as shown in fig. 6, including: a processor 31 and a memory 32, wherein the processor 31 and the memory 32 may be connected by a bus or other means, and the connection by the bus is illustrated in fig. 6 as an example.
The processor 31 may be a Central Processing Unit (CPU). The Processor 31 may also be other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 32, which is a non-transitory computer readable storage medium, can be used for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the data synchronization method in the embodiments of the present application. The processor 31 executes various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in the memory 32, that is, implements the data synchronization method in the above method embodiments.
The memory 32 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 31, and the like. Further, the memory 32 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 32 may optionally include memory located remotely from the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more of the modules described above are stored in the memory 32 and, when executed by the processor 31, perform the data synchronization method as in the embodiment shown in fig. 1.
The details of the electronic device may be understood with reference to the corresponding related description and effects in the embodiment shown in fig. 1, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable information processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable information processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable information processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A method of data synchronization, comprising:
predicting a gaze position of a current user at a first time in a first display output region of a first electronic device, the first time being a time after the current time;
obtaining a first image to be output to the first display output area at a first moment;
sending the first image and the gaze position to a second electronic device, so that the second electronic device outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic device at the first moment;
wherein the second image comprises at least a portion of the first image.
2. The method of claim 1, wherein predicting a gaze location of a current user at a first time on a first display output region of a first electronic device comprises:
the method comprises the steps of obtaining gaze position change information of a current user at a plurality of adjacent moments before a first moment, and predicting a gaze position of the current user at the first moment in a first display output area of first electronic equipment at least based on the gaze position change information; or the like, or, alternatively,
the method comprises the steps of obtaining gaze information of a current user at a plurality of adjacent moments before a first moment, determining a matched prediction model based on the gaze information, and inputting the gaze information into the prediction model to obtain a gaze position of the current user at the first moment in a first display output area of first electronic equipment.
3. The method of claim 2, wherein,
predicting the gaze position of the current user at a first time in a first display output area of the first electronic device based at least on the gaze position change information, comprising:
determining the maximum variation range of the gaze position of the current user in the first time period at the first moment, and predicting the gaze position at least based on the gaze position variation information and the maximum variation range; or the like, or a combination thereof,
determining transmission delay information between the first electronic device and the second electronic device, and predicting the gaze position based on the transmission delay information and the gaze position change information;
alternatively, the first and second electrodes may be,
determining a matched predictive model based on the gaze information, comprising:
the method comprises the steps of obtaining change characteristics of the gazing information and display output data of a first display output area, and determining a matched prediction model based on the change characteristics so as to predict the gazing position of a current user in the first display output area of the first electronic equipment at a first moment by utilizing the prediction model based on the gazing information and the display output data.
4. The method of claim 1, the obtaining a first image to be output to the first display output region at a first time, comprising:
obtaining a third image acquired by a second camera of the first electronic device at a second moment, and processing the third image at least based on the predicted gaze position to obtain the first image, wherein the second moment is earlier than the first moment, and the viewing range of the second camera is larger than that of the first camera; or the like, or, alternatively,
the method comprises the steps of obtaining a third image collected by a second camera of first electronic equipment at a second moment and a fourth image collected by a first camera at the third moment, processing the third image and the fourth image to obtain a first image, wherein the second moment and the third moment are both earlier than the first moment, the second moment and the third moment are the same or different, and the framing range of the second camera is larger than that of the first camera; or the like, or, alternatively,
and obtaining a fourth image acquired by a first camera of the first electronic equipment at a third moment, and processing the fourth image at least based on the predicted gaze position to obtain the first image, wherein the third moment is earlier than the first moment.
5. The method of claim 4, wherein processing the third image and the fourth image to obtain the first image comprises:
obtaining first gaze position change information of the predicted gaze position at the first time relative to the gaze position at the third time, and synthesizing and/or compensating the fourth image with the third image with reference to the first gaze position change information to obtain the first image; or the like, or, alternatively,
acquiring a corresponding relation between a third moment and a predicted gazing position at the first moment and the first display output area, and synthesizing and/or compensating the fourth image by using the third image based on change information of the corresponding relation to obtain the first image;
or the like, or a combination thereof,
processing the third image based on at least the predicted gaze location to obtain the first image, comprising:
and cutting the third image based on the predicted gaze position to obtain the first image.
6. The method of claim 1, the sending the first image and the gaze location to a second electronic device, comprising:
if the difference of the first image relative to at least one frame of image at the adjacent moment earlier than the first moment is in a first range, the first image and the watching position are sent to the second electronic equipment; or the like, or, alternatively,
and if the difference of the first image relative to at least one frame of image at the adjacent moment earlier than the first moment is in a second range, sending the determined corresponding content data and/or relationship data between the first image and the gazing position to the second electronic equipment.
7. The method of claim 6, wherein transmitting the determined content data and/or relationship data between the first image and the gaze location to the second electronic device comprises:
determining image content corresponding to the watching position in the first image, and sending the image content to the second electronic equipment; or the like, or, alternatively,
determining the position relation between the watching position and the second display output area based on the relation between the watching position and the first image and the configuration relation between the first display output area and the second display output area, and sending the position relation to the second electronic equipment; or the like, or a combination thereof,
determining the image content corresponding to the gaze position in the first image and the relative position relationship between the image content and the gaze position, and sending the image content and the relative position relationship to the second electronic device.
8. The method of claim 1, wherein outputting a second image and a prompt corresponding to the gaze location at a second display output region comprises:
and acquiring the configuration relation between a first display output area and a second display output area, and processing the first image and the gaze position at least based on the configuration relation so as to output a second image and prompt information corresponding to the gaze position in the second display output area.
9. The method of claim 8, wherein processing the first image and the gaze location based at least on the configuration relationship to output a second image and a prompt corresponding to the gaze location at the second display output region comprises:
processing the first image based on a display identification information configuration difference between a first display output area and the second display output area so that a second image obtained by processing the first image matches the display identification information of the second display output area; and/or the presence of a gas in the gas,
and processing a first relative position relation between the first image and the gazing position based on the configuration relation, so that a second relative position relation obtained by processing the first relative position relation is configured in a second image or the second display output area to correspondingly output prompt information for highlighting the gazing position.
10. A data synchronization apparatus, comprising:
the prediction module is used for predicting the watching position of a current user in a first display output area of the first electronic equipment at a first moment, wherein the first moment is a moment after the current moment;
the acquisition module is used for acquiring a first image to be output to the first display output area at a first moment;
the sending module is used for sending the first image and the gaze position to second electronic equipment so that the second electronic equipment outputs a second image and prompt information corresponding to the gaze position in a second display output area of the second electronic equipment at the first moment;
wherein the second image comprises at least a portion of the first image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111105917.4A CN113992885B (en) | 2021-09-22 | 2021-09-22 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111105917.4A CN113992885B (en) | 2021-09-22 | 2021-09-22 | Data synchronization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113992885A CN113992885A (en) | 2022-01-28 |
| CN113992885B true CN113992885B (en) | 2023-03-21 |
Family
ID=79736169
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111105917.4A Active CN113992885B (en) | 2021-09-22 | 2021-09-22 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113992885B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107317987A (en) * | 2017-08-14 | 2017-11-03 | 歌尔股份有限公司 | The display data compression method and equipment of virtual reality, system |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100066802A1 (en) * | 2008-09-16 | 2010-03-18 | Brian Dross | Remote communications device and method for facilitating manual operations |
| US8964013B2 (en) * | 2009-12-31 | 2015-02-24 | Broadcom Corporation | Display with elastic light manipulator |
| US10359842B2 (en) * | 2014-09-16 | 2019-07-23 | Ricoh Company, Limited | Information processing system and information processing method |
| JP2016066351A (en) * | 2014-09-16 | 2016-04-28 | 株式会社リコー | Information processing system, information processing apparatus, information processing method, and program |
| WO2017025487A1 (en) * | 2015-08-07 | 2017-02-16 | SensoMotoric Instruments Gesellschaft für innovative Sensorik mbH | System and method for displaying a stream of images |
| CN105491414B (en) * | 2015-11-19 | 2017-05-17 | 深圳市鹰硕技术有限公司 | Synchronous display method and device of images |
| US10127680B2 (en) * | 2016-06-28 | 2018-11-13 | Google Llc | Eye gaze tracking using neural networks |
| CN108334191B (en) * | 2017-12-29 | 2021-03-23 | 北京七鑫易维信息技术有限公司 | Method and device for determining fixation point based on eye movement analysis equipment |
| CN110708384B (en) * | 2019-10-12 | 2020-12-15 | 西安维度视界科技有限公司 | Interaction method, system and storage medium of AR-based remote assistance system |
| JP7239511B2 (en) * | 2020-02-26 | 2023-03-14 | 株式会社日立製作所 | Image prediction system |
| EP4134867A4 (en) * | 2020-04-30 | 2023-05-31 | Huawei Technologies Co., Ltd. | Identification method and identification device for object of interest of user |
-
2021
- 2021-09-22 CN CN202111105917.4A patent/CN113992885B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107317987A (en) * | 2017-08-14 | 2017-11-03 | 歌尔股份有限公司 | The display data compression method and equipment of virtual reality, system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113992885A (en) | 2022-01-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7094266B2 (en) | Single-depth tracking-accommodation-binocular accommodation solution | |
| US8411171B2 (en) | Apparatus and method for generating image including multiple people | |
| CN108919958B (en) | Image transmission method and device, terminal equipment and storage medium | |
| US20170053447A1 (en) | Augmented Reality | |
| US20170053445A1 (en) | Augmented Reality | |
| CN109743626B (en) | Image display method, image processing method and related equipment | |
| CN111627116A (en) | Image rendering control method and device and server | |
| JP2000354257A (en) | Image processor, image processing method and program provision medium | |
| CN109901710A (en) | Treating method and apparatus, storage medium and the terminal of media file | |
| CN110780742B (en) | Eyeball tracking processing method and related device | |
| CN113362450B (en) | Three-dimensional reconstruction method, device and system | |
| CN111880654A (en) | Image display method and device, wearable device and storage medium | |
| CN109799899B (en) | Interaction control method and device, storage medium and computer equipment | |
| CN107065190B (en) | Method and device for displaying information on VR equipment and VR equipment | |
| CN114415826A (en) | Data processing method and equipment thereof | |
| US11842453B2 (en) | Information processing device, information processing method, and program | |
| CN113992885B (en) | Data synchronization method and device | |
| US10922825B2 (en) | Image data processing method and electronic device | |
| US11521297B2 (en) | Method and device for presenting AR information based on video communication technology | |
| CN113515193B (en) | Model data transmission method and device | |
| CN113448428B (en) | Sight focal point prediction method, device, equipment and computer storage medium | |
| WO2022004130A1 (en) | Information processing device, information processing method, and storage medium | |
| CN109756728B (en) | Image display method and apparatus, electronic device, computer-readable storage medium | |
| EP3429186A1 (en) | Image registration method and device for terminal, and terminal | |
| JPWO2021199184A5 (en) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |