CN117319574A

CN117319574A - Time axis adjusting method and electronic equipment

Info

Publication number: CN117319574A
Application number: CN202311121523.7A
Authority: CN
Inventors: 张维; 刘微; 孙萁浩; 田友强; 栾赫; 孟令邦
Original assignee: Hisense Group Holding Co Ltd
Current assignee: Hisense Group Holding Co Ltd
Priority date: 2023-08-31
Filing date: 2023-08-31
Publication date: 2023-12-29

Abstract

The embodiment of the application provides a time axis adjusting method and electronic equipment, which are used for solving the problem that time axes of a plurality of acquisition devices are not synchronous in the prior art. Because in this embodiment of the application, the electronic device adjusts the time axis of the acquisition device based on the first time stamp of the first RTP packet sent by the acquisition device, the second time stamp of the first RTP packet received, and the third time stamp of the acquisition device and the fourth time stamp of the electronic device when the acquisition device starts to acquire video, the time axis of the acquisition device can be adjusted to be synchronous with the electronic device, and then the time axis of each acquisition device can be adjusted to be synchronous with the electronic device, so that the time axes of a plurality of acquisition devices can be synchronous.

Description

Time axis adjusting method and electronic equipment

Technical Field

The present disclosure relates to the field of image generation or image synthesis, and in particular, to a time axis adjustment method and an electronic device.

Background

According to the actual application scene, a scheme for synchronizing multiple paths of video streams is formulated, and the method is an important means for realizing methods such as image splicing, image synthesis and the like.

In the related art, the electronic device is usually used for timing the acquisition device, synchronizing the video according to the time of the acquisition device, and transmitting the synchronized video or video frame to the multi-view video encoder, however, when the electronic device is used for timing the acquisition device, the problem that the time axis of each acquisition device is not synchronized may occur, for example, the time of the acquisition device 1 is 10 points at the same time, the time of the acquisition device 2 is 10 minutes at the 10 points, and further the problem that the video or video frame transmitted to the multi-view video encoder is not synchronized may occur.

Disclosure of Invention

The embodiment of the application provides a time axis adjusting method and electronic equipment, which are used for solving the problem that time axes of a plurality of acquisition devices are not synchronous in the prior art.

In a first aspect, an embodiment of the present application provides a method for adjusting a time axis, where the method includes:

acquiring a first RTP packet sent by acquisition equipment, a first timestamp of the first RTP packet and a second timestamp of the first RTP packet according to a packet sequence number of each received data transmission (Real-time Transport Protocol, RTP) packet sent by the acquisition equipment;

the third time stamp of the acquisition equipment when the acquisition equipment starts to acquire the video and the fourth time stamp of the instruction of the acquisition equipment to start acquiring the video, which are received by the electronic equipment, are acquired;

and determining a deviation of adjusting the time axis of the acquisition equipment to be consistent with the time axis of the electronic equipment according to the first time stamp, the second time stamp, the third time stamp and the fourth time stamp, and adjusting the time axis of the acquisition equipment according to the deviation.

In a second aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes at least a processor and a memory, where the processor is configured to execute the steps of any one of the above-mentioned time axis adjustment methods when executing a computer program stored in the memory.

In the embodiment of the application, the electronic device acquires a first RTP packet sent by the acquisition device, a first timestamp of the first RTP packet and a second timestamp of the received first RTP packet according to the received packet sequence number of each RTP packet sent by the acquisition device; acquiring a third timestamp of the acquisition equipment when the acquisition equipment starts to acquire the video and a fourth timestamp of an instruction received by the electronic equipment when the acquisition equipment starts to acquire the video; and determining a deviation for adjusting the time axis of the acquisition device to be consistent with the time axis of the electronic device according to the first time stamp, the second time stamp, the third time stamp and the fourth time stamp, and adjusting the time axis of the acquisition device according to the deviation. Because in this embodiment of the application, the electronic device adjusts the time axis of the acquisition device based on the first time stamp of the first RTP packet sent by the acquisition device, the second time stamp of the first RTP packet received, and the third time stamp of the acquisition device and the fourth time stamp of the electronic device when the acquisition device starts to acquire video, the time axis of the acquisition device can be adjusted to be synchronous with the electronic device, and then the time axis of each acquisition device can be adjusted to be synchronous with the electronic device, so that the time axes of a plurality of acquisition devices can be synchronous.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of a time axis adjustment process according to an embodiment of the present application;

fig. 2 is a schematic position diagram of an acquisition device deployment according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a process for determining deviation according to an embodiment of the present application;

fig. 4 is a schematic diagram of a process of acquiring a synchronization frame according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a time axis adjusting device according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The present application will be described in further detail below with reference to the attached drawings, wherein it is apparent that the described embodiments are only some, but not all embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

In order to synchronize time axes of a plurality of acquisition devices, a time axis adjustment method and an electronic device are provided.

The time axis adjustment method comprises the following steps: the electronic equipment acquires a first RTP packet sent by the acquisition equipment, a first timestamp of the first RTP packet and a second timestamp of the received first RTP packet according to the received packet sequence number of each RTP packet sent by the acquisition equipment; acquiring a third timestamp of the acquisition equipment when the acquisition equipment starts to acquire the video and a fourth timestamp of an instruction received by the electronic equipment when the acquisition equipment starts to acquire the video; and determining a deviation for adjusting the time axis of the acquisition equipment to be consistent with the time axis of the electronic equipment according to the first time stamp, the second time stamp, the third time stamp and the fourth time stamp, and adjusting the time axis of the acquisition equipment according to the deviation so as to synchronize the time axes of the acquisition equipment.

Fig. 1 is a schematic diagram of a time axis adjustment process according to an embodiment of the present application, where the process includes the following steps:

s101: and acquiring a first RTP packet sent by the acquisition equipment, a first time stamp of the first RTP packet and a second time stamp of the first RTP packet according to the received packet sequence number of each RTP packet sent by the acquisition equipment.

The time axis adjustment method provided by the embodiment of the application is applied to electronic equipment, and the electronic equipment can be intelligent equipment such as a PC or a server.

In order to realize synchronization of time axes of a plurality of acquisition devices, when the acquisition devices acquire videos, the acquisition devices send image frames of the acquired videos to the electronic devices through RTP packets, specifically, the acquisition devices also send packet serial numbers of each RTP packet to the electronic devices, and the electronic devices can acquire a first RTP packet sent by the acquisition devices, a first timestamp of the first RTP packet and a second timestamp of the first RTP packet received by the electronic devices according to the received packet serial numbers of each RTP packet sent by the acquisition devices.

The acquisition module of the acquisition device can send the video stream from the acquisition device to the video synchronous processing module of the electronic device through a network cable according to a streaming media protocol. Specifically, the collecting device and the electronic device establish a real-time streaming (Real Time Streaming Protocol, RTSP) protocol, and after completing the Description (DESCRIBE), SETUP (SETUP) and PLAY (PLAY) interaction instruction of the RTSP protocol, the collecting device may send the audio/video data to the server. In an actual scenario, the electronic device may obtain a packet sequence number of each RTP packet based on a PLAY instruction of the RTSP protocol, and further obtain a first RTP packet based on the packet sequence number of each RTP packet, that is, a packet sequence number corresponding to the first RTP packet is specified by a pay instruction of the RTSP protocol.

S102: and acquiring a third timestamp of the acquisition equipment when the acquisition equipment starts to acquire the video and a fourth timestamp of an instruction received by the electronic equipment, wherein the instruction starts to acquire the video by the acquisition equipment.

The electronic device further obtains a third timestamp of the acquisition device when the acquisition device starts to acquire the video, specifically, a business person can click a preset button of the acquisition device, and the acquisition device can start to acquire the video at the moment. After the acquisition device starts to acquire the video, the instruction of starting to acquire the video can be sent to the electronic device, and the electronic device can acquire a fourth timestamp of the instruction of starting to acquire the video, which is sent by the acquisition device, received. The third timestamp and the fourth timestamp may be considered as the time of the capture device and the time of the electronic device, respectively, when the capture device starts capturing video.

Wherein the third timestamp is an initial build time of the virtual timeline of the acquisition device. The time axis of the acquisition equipment is based on the system time of a server for deploying the acquisition equipment, and the server is used for carrying out time service on the acquisition equipment.

S103: and determining a deviation of adjusting the time axis of the acquisition equipment to be consistent with the time axis of the electronic equipment according to the first time stamp, the second time stamp, the third time stamp and the fourth time stamp, and adjusting the time axis of the acquisition equipment according to the deviation.

In order to accurately adjust the time axis of the collection device, the electronic device may determine, according to the first time stamp, the second time stamp, the third time stamp and the fourth time stamp, a deviation in which the time axis of the collection device is adjusted to be consistent with the time axis of the electronic device, specifically, the electronic device may determine a difference between the first time stamp and the second time stamp, for convenience of distinction, the difference may be referred to as a first difference, and determine a difference between the third time stamp and the fourth time stamp, for convenience of distinction, the difference may be referred to as a second difference, and after determining the first difference and the second difference, the electronic device may determine an average value of the first difference and the second difference, as a deviation in which the time axis of the collection device is adjusted to be consistent with the time axis of the electronic device. After determining a deviation that adjusts the time axis of the acquisition device to coincide with the time axis of the electronic device, the electronic device may adjust the time axis of the acquisition device according to the deviation.

It should be noted that, the electronic device may store an adjustable threshold range in advance for the acquisition device, if the deviation determined by a certain acquisition device corresponding to the deviation is within the threshold range, the time axis of the acquisition device may be adjusted based on the deviation, if the deviation determined by a certain acquisition device corresponding to the deviation is not within the threshold range, a first absolute value of a difference between the deviation and a maximum value of the threshold range may be determined, a second absolute value of a difference between the deviation and a minimum value of the threshold range may be determined, a larger value may be selected from the first absolute value and the second absolute value, if the first absolute value is larger, the time axis of the acquisition device may be adjusted based on the maximum value of the threshold range, and if the second absolute value is larger, the time axis of the acquisition device may be adjusted based on the minimum value of the threshold range.

Fig. 2 is a schematic position diagram of deployment of an acquisition device according to an embodiment of the present application.

As can be seen from fig. 2, the plurality of capturing devices are disposed at a certain angle, and the optical axes of the plurality of capturing devices all point to the same target to be photographed.

Because in this embodiment of the application, the electronic device adjusts the time axis of the acquisition device based on the first time stamp of the first RTP packet sent by the acquisition device, the second time stamp of the first RTP packet received, and the third time stamp of the acquisition device and the fourth time stamp of the electronic device when the acquisition device starts to acquire video, the time axis of the acquisition device can be adjusted to be synchronous with the electronic device, and then the time axis of each acquisition device can be adjusted to be synchronous with the electronic device, so that the time axes of a plurality of acquisition devices can be synchronous.

In order to accurately synchronize the time axes of the plurality of acquisition devices, based on the disclosure of the above embodiments, in an embodiment of the present application, before the determining adjusts the time axes of the acquisition devices to a deviation consistent with the time axes of the electronic devices, the method further includes:

sequencing each RTP packet according to the packet sequence number of each RTP packet sent by the received acquisition equipment, extracting data in the sequenced RTP packet, and decoding the data by a decoder to obtain each image frame; extracting a first Intra picture (I) frame in each image frame, obtaining a first display time stamp (Predetermined Time System, PTS) corresponding to the first I frame, and receiving, by an electronic device, a second PTS of an RTP packet corresponding to the first I frame and a fifth time stamp at which the I frame is decoded;

The determining, according to the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp, a deviation that adjusts the time axis of the acquisition device to coincide with the time axis of the electronic device includes:

and determining a deviation for adjusting the time axis of the acquisition device to be consistent with the time axis of the electronic device according to the first time stamp, the second time stamp, the third time stamp, the fourth time stamp, the first PTS, the second PTS and the fifth time stamp.

In order to accurately synchronize time axes of the plurality of acquisition devices, the electronic device may further sort each RTP packet according to a packet sequence number of each RTP packet sent by the received acquisition device, extract data in each RTP packet after sorting each RTP packet, decode the extracted data by a decoder, obtain each image frame, extract a first I frame in each image frame after obtaining each image frame, and obtain a first PTS corresponding to the first I frame, and the electronic device receives a second PTS corresponding to the first I frame and a fifth timestamp decoded by the I frame.

After obtaining the first PTS, the second PTS, and the fifth timestamp, the electronic device may determine, according to the first timestamp, the second timestamp, the third timestamp, the fourth timestamp, the first PTS, the second PTS, and the fifth timestamp, a deviation that adjusts the time axis of the acquisition device to be consistent with the time axis of the electronic device. Specifically, the electronic device may determine a first difference between the first timestamp and the second timestamp, determine a second difference between the third timestamp and the fourth timestamp, convert the first PTS and the second PTS into a sixth timestamp and a seventh timestamp, respectively, determine a third difference between the sixth timestamp and the fifth timestamp, and determine a fourth difference between the seventh timestamp and the fifth timestamp, and after determining the first difference, the second difference, the third difference, and the fourth difference, the electronic device may determine an average of the first difference, the second difference, the third difference, and the fourth difference as a deviation that adjusts a time axis of the acquisition device to be consistent with a time axis of the electronic device.

Fig. 3 is a schematic diagram of a process for determining deviation according to an embodiment of the present application, where the process includes the following steps:

s301: and acquiring the first RTP packet sent by the acquisition equipment, a first time stamp of the first RTP packet and a second time stamp of the received first RTP packet according to the received packet sequence number of each RTP packet sent by the acquisition equipment.

S302: and acquiring a third timestamp of the acquisition equipment when the acquisition equipment starts to acquire the video and a fourth timestamp of an instruction received by the electronic equipment when the acquisition equipment starts to acquire the video.

S303: and sequencing each RTP packet according to the packet sequence number of each RTP packet sent by the received acquisition equipment, extracting data in the sequenced RTP packet, and decoding the data by a decoder to obtain each image frame.

S304: and extracting the first I frame in each image frame, acquiring the PTS corresponding to the first I frame, and receiving the second PTS of the RTP packet corresponding to the first I frame and the fifth timestamp of the decoded I frame by the electronic equipment.

S305: and determining a deviation for adjusting the time axis of the acquisition device to be consistent with the time axis of the electronic device according to the first time stamp, the second time stamp, the third time stamp, the fourth time stamp, the first PTS, the second PTS and the fifth time stamp.

Because the electronic device determines the deviation of adjusting the time axis of the acquisition device to be consistent with the time axis of the electronic device, the deviation determination accuracy can be further improved, and the synchronization of the time axes of a plurality of acquisition devices can be further accurately realized according to the time stamp of the RTP packet corresponding to the first I frame, the time stamp of the RTP packet corresponding to the first I frame received by the electronic device and the time of decoding the first I frame.

The PTS time of the RTP packet corresponding to each I frame and the PTS time of the electronic device receiving the RTP packet are in a linear relationship, and if the PTS time changes, problems such as abnormal network fluctuation, network packet loss, and abnormal decoding may occur.

Wherein, the decoding is introduced by a certain group of pictures GOP sequence: assuming that the sequence of the image frames collected by the GOP group is ib B P B P in turn, the PTS of each image frame may be 1 2 3 4 5 67, the coding sequence may be 1 4 2 3 7 5 6, the pushing sequence is IP B P B for pushing according to the coding sequence, the video frame received by the electronic device may be IP B P B, and the decoding sequence is 1 2 3 4 5 6 7. Wherein, the PTS corresponding to each image frame obtained by decoding is not sequential, and in order to correctly display the video stream, the decoded image frames need to be adjusted according to the PTS. Specifically, how to decode and how to adjust the decoded image frames are all in the prior art, and are not described herein.

In order to accurately determine a deviation of adjusting the time axis of the acquisition device to be consistent with the time axis of the electronic device, in the embodiment of the present application, the determining, based on the first time stamp, the second time stamp, the third time stamp, the fourth time stamp, the first PTS, the second PTS, and the fifth time stamp, the deviation of adjusting the time axis of the acquisition device to be consistent with the time axis of the electronic device includes:

Respectively determining a first coordinate point with the first time stamp as an ordinate and the second time stamp as an abscissa, a second coordinate point with the third time stamp as an ordinate and the fourth time stamp as an abscissa, a third coordinate point with the first PTS as an abscissa and the second PTS as an ordinate, and a fifth time stamp as a fourth coordinate point with the abscissa and the ordinate, wherein the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point are sequentially connected with each other to generate a curve of the acquisition equipment;

if the number of the points where the curves of any two acquisition devices overlap is smaller than the preset number, performing approximation adjustment on the curves of the two acquisition devices until the number of the points where the adjusted curves overlap reaches the preset number; and determining the adjustment quantity of the ordinate when any curve is subjected to approximation adjustment to adjust the time axis of the acquisition equipment to be a deviation consistent with the time axis of the electronic equipment.

After the electronic device obtains the first timestamp, the second timestamp, the third timestamp, the fourth timestamp, the first PTS, the second PTS, and the fifth timestamp, the electronic device may establish a coordinate system with a time related to the collecting device as a vertical axis and a time related to the electronic device as a horizontal axis, determine a first coordinate point with the first timestamp as an ordinate and the second timestamp as an abscissa respectively, determine a second coordinate point with the third timestamp as an ordinate and the fourth timestamp as an abscissa respectively, and generate a curve of the collecting device after obtaining the first coordinate point, the second coordinate point, the third coordinate point, and the fourth coordinate point. In this way, the electronic device can generate a curve for each acquisition device.

After generating the curve of each acquisition device, the electronic device may determine whether the two acquisition devices are similar according to the number of points where the curves of any two acquisition devices overlap, if the number of points where the curves of the two acquisition devices overlap is not less than a preset value, then the two acquisition devices are described as similar, that is, the time axes of the two acquisition devices are synchronous, if the points where the curves of the two acquisition devices overlap are not less than the preset value, then the time axes of the two acquisition devices are described as not synchronous, in order to synchronize the time axes of the two acquisition devices, the electronic device may perform approximation processing on the curves of the two acquisition devices, specifically, when the electronic device performs approximation processing on the two curves, the ordinate of each point on one curve is increased by a preset value, the ordinate of each point on the other curve is decreased by the preset value, then the adjusted two curves are obtained, if the point where the two adjusted overlapping points are not less than the preset value, then the two adjusted points are not overlapped, then the adjusted to be the preset value is continued to be the adjusted to be the value, and if the point where the two adjusted points are not overlapped, then the ordinate of each point is increased by the preset value, and the value is continued to be increased by the value. At this time, the electronic device may determine the adjustment amount of the ordinate when any curve approaches adjustment, which is the deviation of synchronization of the time axes of the two collecting devices, that is, the deviation of adjusting the time axis of the collecting device to be consistent with the time axis of the electronic device.

Table 1 is a table containing the time associated with each acquisition device as determined in the embodiments of the present application.

Acquisition device 1	Acquisition device 2	Acquisition device 3	Acquisition device 4	Acquisition device 5
					a11	a12	a13	a14	a15
a21	a22	a23	a24	a25
					a31	a32	a33	a34	a35
a41	a42	a43	a44	a45

TABLE 1

Wherein, the second behavior in table 1 is the third timestamp of the capturing device when each capturing device starts capturing video, the third behavior in table 1 is the first timestamp of the first RTP packet sent by each capturing device, the fourth behavior in table 1 is the first PTS of the RTP packet corresponding to the first I frame of each capturing device, and the fifth behavior in table 1 is the fifth timestamp of the first I frame of each capturing device decoded.

In order to accurately adjust the time axis of the acquisition device, on the basis of the disclosure of the above embodiments, in the embodiment of the present application, the adjusting the time axis of the acquisition device according to the deviation includes:

sequencing each RTP packet according to the packet sequence number of each RTP packet sent by the received acquisition equipment, extracting data in the sequenced RTP packet, and decoding the data by a decoder to obtain each image frame;

for each image frame, determining the PTS of an RTP packet corresponding to the image frame as a third PTS of the image frame, determining the inverse of the difference between the image frame and the third PTS of the image frame last to the image frame as a change rate, determining the difference between the image frame and the third PTS of the image frame next to the image frame, determining the product of the difference and the change rate, determining the dynamic time of the image frame according to the product and the third PTS of the image frame, determining the time of the electronic device when the image frame is acquired according to the deviation of the dynamic time and the acquisition device, and adjusting the time axis of the acquisition device based on the time of the electronic device when the image frame is acquired.

When the electronic device adjusts the time axis of the acquisition device, each RTP packet can be sequenced according to the packet sequence number of each RTP packet sent by the acquisition device, based on the data in the sequenced RTP packet, the decoder decodes the data to obtain each image frame, after each image frame is obtained, the electronic device can determine the PRT value of the RTP packet corresponding to the image frame as the PTS of the image frame for each image frame, in order to facilitate discrimination, the PTS of the image frame can be referred to as the third PTS, the electronic device can determine the difference value of the image frame and the third PTS of the previous image frame of the image frame, and determine the inverse of the difference value as the change rate of the image frame, the electronic device can also determine the product of the difference value and the change rate of the image frame, determine the dynamic time of the image frame according to the third PTS of the product and the image frame, the electronic device can determine the dynamic time of each image frame by the way, and the electronic device can determine the dynamic time of each image frame by combining the dynamic time with the dynamic time of the image frame in the way, and the electronic device can determine the dynamic time can be referred to as the dynamic time of the electronic device when the electronic device is acquired according to the dynamic time of the image frame. After determining the time of the electronic device when the image frame was acquired, the electronic device may adjust the time axis of the acquisition device based on the time such that the time axis of the acquisition device is synchronized with the time axis of the electronic device.

Specifically, the electronic device may determine the dynamic time of a certain image frame using the following formula:

dTime+delta_f delta_t-1, where delta_t=PTS_n-PTS_ (n-1), delta_f=1/delta_t

Where dTime is the third PTS of the image frame, delta_f is the rate of change, delta_t-1 is the difference of the third PTS of the next image frame of the image frame, PTS_n is the third PTS of the last image frame of the image frame, and PTS_ (n-1) is the third PTS of the image frame.

By the method provided by the embodiment of the application, the dynamic time of the dynamic time axis can be calculated according to the change rate and the frame rate change of the PTS of the image frame, so that the time axis of the acquisition equipment is dynamically adjusted, wherein the difference value of the third PTS of the image frame and the next image frame can be the frame rate change. The specific adjustment method can adopt a GOP group mode, and the adjusted data is used as the offset basic offset of PTS of the subsequent frames in the GOP group. Then, the PTS of each image frame is used as one data in the time series, and the time axis of the acquisition device is adjusted. Finally, the method is equivalent to establishing the relation between the time axis of the electronic equipment and the time axis of the acquisition equipment so as to realize the effect of synchronizing the time axes of the multi-view acquisition equipment. It should be noted that this approach requires ensuring that the settings and environmental conditions of the individual acquisition devices are the same to ensure the accuracy of the synchronization effect. The dynamic time value of the dynamic time axis is calculated according to the change rate of PTS of the image frame and the change of the frame rate, so that the problems of partial network fluctuation and packet loss can be solved.

The electronic device calculates according to the time related to the first I frame when determining the deviation of the acquisition device, wherein the first I frame is the image frame in the first GOP group, which is equivalent to waiting for the arrival of the next I frame after the processing of the first GOP group is completed, and calculates the dynamic time based on the calculated deviation and the data of the next I frame and each subsequent frame.

The dynamic time axis described in the embodiments of the present application may be used to perform matching correction on time of an acquisition device, and provide a time correction scheme for a video synchronization frame for a technical scheme related to a free view angle. The dynamic time axis is a time axis which is obtained through calculation and is changed continuously, the time axis takes the time of the electronic equipment as a reference, and the time of the acquisition equipment, the time related to the first RTP packet and the like are used for correction, so that the dynamic time axis is determined.

And establishing a relation between the time axis of the acquisition equipment and a standard time axis according to the established dynamic time axis of the acquisition equipment, and establishing a relation between the PTS of the dynamic time axis of the acquisition equipment and the dynamic time axis on the standard time axis in a projection mode. Specific standard time axis value = acquisition device dynamic time axis value + acquisition device deviation.

The conventional multi-view video acquisition synchronization time axis method needs to be marked before shooting, and is manually synchronized in post-production or synchronized by using a time code. The dynamic time axis can be established in real time in the video acquisition process, and marking and post-production are not needed. The method can realize the establishment of a dynamic time axis by carrying out real-time processing and analysis on the acquired data of a plurality of acquisition devices. The method can improve the viewing experience of the audience, so that the audience can more conveniently switch pictures with different angles. The dynamic time axis can be adaptively adjusted according to actual conditions. During shooting, the acquired data of each camera may be affected by factors such as environmental conditions, and thus cause a deviation of the time axis. The dynamic time axis can be adjusted in a self-adaptive manner through real-time analysis and adjustment, so that the synchronization effect of the multi-view video is ensured.

In order to accurately determine the time of the electronic device when the image frame is acquired, based on the disclosure of the above embodiments, in the embodiments of the present application, the determining, according to the deviation between the dynamic time and the acquisition device, the time of the electronic device when the image frame is acquired includes:

Determining network delay time when the acquisition equipment interacts with the electronic equipment;

and determining the sum of the dynamic time, the deviation of the acquisition equipment and the network delay time as the time of the electronic equipment when the image frame is acquired.

In the actual scene, since the network delay exists between the acquisition device and the electronic device when the acquisition device interacts with the electronic device, in order to accurately determine the time of the electronic device when the image frame is acquired, the electronic device may determine the network delay time when the acquisition device interacts with the electronic device, specifically, the network delay time may be pre-saved, and when the electronic device determines the time of the electronic device when a certain image frame is acquired, the determined dynamic time of the image frame, the deviation of the acquisition device and the sum value of the network delay time may be determined as the time of the electronic device when the image frame is acquired.

The electronic device may calculate a network delay time for each GOP group, where the network delay time is the network delay time of each image frame in the GOP group, where the network delay time may be determined by a packet internet discovery (Packet Internet Groper, ping) command, and specifically, how to determine the network delay time by the ping command is the prior art and is not described herein. When determining the time of the electronic device when a certain image frame is acquired, the electronic device may acquire the network delay time of the GOP group to which the image frame belongs, and determine based on the network delay time.

In order to acquire the synchronization frame, based on the disclosure of the foregoing embodiments, in an embodiment of the present application, the method further includes:

if a synchronous frame acquisition instruction is received, acquiring standard time carried in the synchronous frame acquisition instruction, determining the system time of the acquisition equipment when the time of the electronic equipment is the standard time according to the deviation and the standard time, acquiring a synchronous frame acquired by the system time, and transmitting the synchronous frame.

When a user has a requirement of checking the synchronous frame, the user can select standard time from a preset page of the device which is used by the user, the standard time is the time of the image frame which the user wants to check, and click a preset button, at the moment, the electronic device can receive a synchronous frame acquisition instruction, the synchronous frame acquisition instruction carries the standard time, and the electronic device can determine that the time of the electronic device is the system time of the acquisition device according to the standard time and the deviation. After acquiring the system time of the acquisition device, the electronic device may acquire a synchronization frame acquired by the acquisition device at the system time, and send the synchronization frame.

Fig. 4 is a schematic diagram of a process of acquiring a synchronization frame according to an embodiment of the present application.

S401: and receiving a synchronous frame acquisition instruction.

S402: and acquiring standard time carried in the synchronous frame acquisition instruction.

S403: and determining the system time of the acquisition equipment when the time of the electronic equipment is the standard time according to the deviation and the standard time of the acquisition equipment.

S404: and acquiring a synchronous frame acquired by the acquisition equipment at the system time, and transmitting the synchronous frame.

By the method provided by the embodiment of the application, the electronic equipment can acquire the synchronous frames of the plurality of acquisition equipment in the standard time, and the acquisition of the multi-view synchronous frames can be realized. The embodiment of the application is equivalent to a mode of setting a dynamic time axis, acquiring video streams of a plurality of acquisition devices and extracting a synchronous frame.

In order to accurately acquire the synchronization frame, on the basis of the disclosure of the foregoing embodiments, in an embodiment of the present application, after the acquisition of the synchronization frame acquired by the system time and before the transmission of the synchronization frame, the method further includes:

acquiring a preset number of collected candidate synchronous frames adjacent to the synchronous frames;

the transmitting the synchronization frame includes:

transmitting the synchronous frame and each acquired candidate synchronous frame; so that the user can screen the standard synchronous frames from the synchronous frames and each candidate synchronous frame.

Because the obtained synchronization frame may not meet the requirement of the user, in order to accurately obtain the synchronization frame meeting the requirement of the user, the electronic device may further obtain a preset number of candidate synchronization frames adjacent to the synchronization frame, for example, may obtain a first preset number of candidate synchronization frames with acquisition time before the synchronization frame and obtain a second preset number of candidate synchronization frames with acquisition time after the synchronization frame, and send the obtained synchronization frame and each candidate synchronization frame, specifically, may send the synchronization frame and each candidate synchronization frame to a device sending a synchronization frame obtaining instruction, and the user may view the synchronization frame and each candidate synchronization frame on a page of the device and may screen the synchronization frame and each candidate synchronization frame for a standard synchronization frame meeting the requirement of the user.

The synchronization frame acquisition instruction may further carry error time, where the error time is a time that a user can accept by the synchronization frame within the error time, for example, may be 200ms, and the electronic device may acquire a corresponding time range according to standard time and error time carried in the synchronization frame acquisition instruction, and determine a system time range of the acquisition device when the time of the electronic device is the time range, acquire a synchronization frame acquired in the system time range, and send the synchronization frame, that is, acquire an optimal picture of the acquisition device through the standard time axis according to the set error time of the synchronization frame.

After determining the system time range of each acquisition device, the electronic device can send the synchronous frames acquired in the system time range to the device used by the user, the user judges whether the synchronous frames meet the requirements based on the device used by the user, if not, the electronic device can send an instruction for searching the synchronous frames again to the electronic device, the electronic device expands the system time range until the synchronous frames meeting the requirements of the user are found, and at the moment, the electronic device can record the expanded system time range as the subsequent searching. The electronic device may establish a matrix for marking the current standard timeline time, whether synchronization frame data is present or not. May be used for a user to extract sync frames while playing back video.

In order to improve accuracy and efficiency of synchronous frame acquisition, based on the disclosure of the above embodiments, in the embodiments of the present application, the method further includes:

and receiving a standard synchronous frame, acquiring a frame sequence number correspondingly stored for the standard synchronous frame, and storing the corresponding relation between the standard time and the frame sequence number.

In order to improve accuracy and efficiency of synchronization frame acquisition, when a user screens a standard synchronization frame meeting own requirements in the synchronization frame and each candidate synchronization frame, a user can select the standard synchronization frame from the self-used device and click a preset button, for example, a confirm button, so that the user can send the standard synchronization frame to the electronic device, and after decoding each image frame, the electronic device can determine the frame number of each image frame and correspondingly store the frame number of each image frame, and specifically how to determine that the frame number of each image frame is in the prior art is not described herein. After receiving the standard synchronization frame, the electronic device may acquire a frame number stored corresponding to the standard synchronization frame, and store a correspondence between the standard time and the frame number.

In order to improve accuracy and efficiency of the synchronization frame acquisition, in the embodiment of the present application, after the standard time carried in the synchronization frame acquisition instruction is acquired, before determining the system time of the acquisition device at the standard time according to the deviation and the standard time, the method further includes:

judging whether a frame sequence number is correspondingly stored for the standard time;

if yes, sending a synchronous frame with the frame number being the target frame number correspondingly stored in the standard time;

if not, executing the subsequent step of determining the system time of the acquisition equipment at the standard time according to the deviation and the standard time.

In order to improve accuracy and efficiency of synchronous frame acquisition, after acquiring a standard time carried in a synchronous frame acquisition instruction, the electronic device can determine whether a frame number is correspondingly stored for the standard time, if the frame number is correspondingly stored for the standard time, the electronic device can acquire a target frame number correspondingly stored for the standard time, acquire a synchronous frame with the frame number being the target frame number, send the acquired synchronous frame, and if the frame number is not correspondingly stored for the standard time, execute the following steps of acquiring the system time of the device according to deviation and standard time, determining the system time of the device when the standard time, and acquiring the synchronous frame based on the system time.

Fig. 5 is a schematic structural diagram of a time axis adjusting device according to an embodiment of the present application, where the device includes:

a receiving module 501, configured to obtain, according to a packet sequence number of each RTP packet sent by a received acquisition device, a first RTP packet sent by the acquisition device, a first timestamp of the first RTP packet, and a second timestamp of the received first RTP packet;

the acquiring module 502 is configured to acquire a third timestamp of the acquiring device when the acquiring device starts to acquire the video, and a fourth timestamp of an instruction received by the electronic device for the acquiring device to start acquiring the video;

and a processing module 503, configured to determine, according to the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp, a deviation of adjusting the time axis of the collecting device to be consistent with the time axis of the electronic device, and adjust the time axis of the collecting device according to the deviation.

Further, the processing module 503 is further configured to sort each RTP packet according to a packet sequence number of each RTP packet sent by the received acquisition device, extract data in the sorted RTP packets, and decode the data by a decoder to obtain each image frame; extracting a first internal picture I frame in each image frame, acquiring a PTS corresponding to the first I frame, and receiving a second PTS corresponding to the first I frame and a fifth timestamp of the decoded I frame by electronic equipment;

The processing module 503 is specifically configured to determine, according to the first timestamp, the second timestamp, the third timestamp, the fourth timestamp, the first PTS, the second PTS, and the fifth timestamp, a deviation that adjusts the time axis of the acquisition device to be consistent with the time axis of the electronic device.

Further, the processing module 503 is specifically configured to determine, with a time related to the collecting device as a vertical axis and a time related to the electronic device as a horizontal axis, a first coordinate point where the first timestamp is an ordinate and the second timestamp is an abscissa, a third timestamp is an ordinate and the fourth timestamp is an abscissa second coordinate point, the first PTS is an abscissa and the second PTS is an ordinate third coordinate point, and the fifth timestamp is an abscissa and an ordinate fourth coordinate point, and sequentially connect the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point to generate a curve of the collecting device; if the number of the points where the curves of any two acquisition devices overlap is smaller than the preset number, performing approximation adjustment on the curves of the two acquisition devices until the number of the points where the adjusted curves overlap reaches the preset number; and determining the adjustment quantity of the ordinate when any curve is subjected to approximation adjustment to adjust the time axis of the acquisition equipment to be a deviation consistent with the time axis of the electronic equipment.

Further, the processing module 503 is specifically configured to sort each RTP packet according to a packet sequence number of each RTP packet sent by the received acquisition device, extract data in the sorted RTP packet, and decode the data by a decoder to obtain each image frame; for each image frame, determining the PTS of an RTP packet corresponding to the image frame as a third PTS of the image frame, determining the inverse of the difference between the image frame and the third PTS of the image frame last to the image frame as a change rate, determining the difference between the image frame and the third PTS of the image frame next to the image frame, determining the product of the difference and the change rate, determining the dynamic time of the image frame according to the product and the third PTS of the image frame, determining the time of the electronic device when the image frame is acquired according to the deviation of the dynamic time and the acquisition device, and adjusting the time axis of the acquisition device based on the time of the electronic device when the image frame is acquired.

Further, the processing module 503 is specifically configured to determine a network delay time when the collecting device interacts with the electronic device; and determining the sum of the dynamic time, the deviation of the acquisition equipment and the network delay time as the time of the electronic equipment when the image frame is acquired.

Further, the processing module 503 is further configured to, if a synchronization frame acquisition instruction is received, acquire a standard time carried in the synchronization frame acquisition instruction, determine, according to the deviation and the standard time, a system time of the acquisition device when a time of the electronic device is the standard time, acquire a synchronization frame acquired by the system time, and send the synchronization frame.

Further, the processing module 503 is further configured to obtain a preset number of collected candidate synchronization frames adjacent to the synchronization frame;

the processing module 503 is specifically configured to send the synchronization frame and each obtained candidate synchronization frame; so that the user can screen the standard synchronous frames from the synchronous frames and each candidate synchronous frame.

Further, the processing module 503 is further configured to receive a standard synchronization frame, obtain a frame number corresponding to the standard synchronization frame, and store a correspondence between the standard time and the frame number.

Further, the processing module 503 is further configured to determine whether a frame number is stored for the standard time correspondence; if yes, sending a synchronous frame with the frame number being the target frame number correspondingly stored in the standard time; if not, executing the subsequent step of determining the system time of the acquisition equipment at the standard time according to the deviation and the standard time.

Fig. 6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application, and on the basis of the foregoing embodiments, the embodiment of the present application further provides an electronic device, as shown in fig. 6, including: processor 601, communication interface 602, memory 603 and communication bus 604, wherein processor 601, communication interface 602, memory 603 complete the communication each other through communication bus 604;

the memory 603 has stored therein a computer program which, when executed by the processor 601, causes the processor 601 to perform the steps of:

acquiring a first RTP packet sent by acquisition equipment, a first timestamp of the first RTP packet and a second timestamp of the first RTP packet according to a received packet sequence number of each RTP packet sent by the acquisition equipment;

Further, the processor 601 is further configured to sort each RTP packet according to a packet sequence number of each RTP packet sent by the received acquisition device, extract data in the sorted RTP packets, and decode the data by a decoder to obtain each image frame; extracting a first internal picture I frame in each image frame, acquiring a PTS corresponding to the first I frame, and receiving a second PTS corresponding to the first I frame and a fifth timestamp of the decoded I frame by electronic equipment;

the processor 601 is specifically configured to determine, according to the first timestamp, the second timestamp, the third timestamp, the fourth timestamp, the first PTS, the second PTS, and the fifth timestamp, a deviation that adjusts the time axis of the acquisition device to be consistent with the time axis of the electronic device.

Further, the processor 601 is specifically configured to determine, with a time related to the acquisition device being a vertical axis and a time related to the electronic device being a horizontal axis, a first coordinate point where the first timestamp is an ordinate and the second timestamp is an abscissa, a third timestamp is an ordinate and the fourth timestamp is an abscissa second coordinate point, the first PTS is an abscissa and the second PTS is an ordinate third coordinate point, and the fifth timestamp is an abscissa fourth coordinate point, and sequentially connect the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point to generate a curve of the acquisition device;

Further, the processor 601 is specifically configured to sort each RTP packet according to a packet sequence number of each RTP packet sent by the received acquisition device, extract data in the sorted RTP packets, and decode the data by a decoder to obtain each image frame;

for each image frame, determining a PTS corresponding to the image frame as a third PTS of the image frame, determining the inverse of the difference between the image frame and the third PTS of the previous image frame of the image frame as a change rate, determining the difference between the image frame and the third PTS of the next image frame of the image frame, determining the product of the difference and the change rate, determining the dynamic time of the image frame according to the product and the third PTS of the image frame, determining the time of the electronic device when the image frame is acquired according to the deviation of the dynamic time and the acquisition device, and adjusting the time axis of the acquisition device based on the time of the electronic device when the image frame is acquired.

Further, the processor 601 is specifically configured to determine a network delay time when the collecting device interacts with the electronic device;

Further, the processor 601 is further configured to, if a synchronization frame acquisition instruction is received, acquire a standard time carried in the synchronization frame acquisition instruction, determine, according to the deviation and the standard time, a system time of the acquisition device when a time of the electronic device is the standard time, acquire a synchronization frame acquired by the system time, and send the synchronization frame.

Further, the processor 601 is further configured to obtain a preset number of collected candidate synchronization frames adjacent to the synchronization frame;

the transmitting the synchronization frame includes:

Further, the processor 601 is further configured to receive a standard synchronization frame, obtain a frame number corresponding to the standard synchronization frame, and store a correspondence between the standard time and the frame number.

Further, the processor 601 is further configured to determine whether a frame number is stored for the standard time correspondence;

The communication bus mentioned by the server may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the electronic device and other devices.

The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; but also digital instruction processors (Digital Signal Processing, DSP), application specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

On the basis of the above embodiments, the embodiments of the present application further provide a computer readable storage medium, in which a computer program executable by an electronic device is stored, which when executed on the electronic device, causes the electronic device to perform the following steps:

the memory has stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of:

In one possible implementation, before the determining adjusts the time axis of the acquisition device to a deviation consistent with the time axis of the electronic device, the method further includes:

sequencing each RTP packet according to the packet sequence number of each RTP packet sent by the received acquisition equipment, extracting data in the sequenced RTP packet, and decoding the data by a decoder to obtain each image frame; extracting a first internal picture I frame in each image frame, acquiring a PTS corresponding to the first I frame, and receiving a second PTS corresponding to the first I frame and a fifth timestamp of the decoded I frame by electronic equipment;

In one possible implementation manner, the determining, according to the first timestamp, the second timestamp, the third timestamp, the fourth timestamp, the first PTS, the second PTS, and the fifth timestamp, the deviation that adjusts the time axis of the acquisition device to be consistent with the time axis of the electronic device includes:

In a possible embodiment, the adjusting the time axis of the acquisition device according to the deviation comprises:

In a possible implementation manner, the determining, according to the deviation of the dynamic time from the acquisition device, the time of the electronic device when the image frame is acquired includes:

In one possible embodiment, the method further comprises:

In a possible implementation manner, after the acquiring the synchronization frame acquired by the system time and before the transmitting the synchronization frame, the method further includes:

the transmitting the synchronization frame includes:

In one possible embodiment, the method further comprises:

In a possible implementation manner, after the acquiring the standard time carried in the synchronization frame acquiring instruction, before determining the system time of the acquisition device at the standard time according to the deviation and the standard time, the method further includes:

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.

Claims

1. A time axis adjustment method, applied to an electronic device, the method comprising:

acquiring a first RTP packet sent by acquisition equipment, a first timestamp of the first RTP packet and a second timestamp of the received first RTP packet according to a packet sequence number of each data transmission RTP packet sent by the acquisition equipment;

2. The method of claim 1, wherein the determining adjusts the time axis of the acquisition device to a deviation consistent with the time axis of the electronic device is preceded by the determining, the method further comprising:

sequencing each RTP packet according to the packet sequence number of each RTP packet sent by the received acquisition equipment, extracting data in the sequenced RTP packet, and decoding the data by a decoder to obtain each image frame; extracting a first internal picture I frame in each image frame, and acquiring a first display time stamp PTS of an RTP packet corresponding to the first I frame, a second PTS of the RTP packet corresponding to the first I frame received by electronic equipment, and a fifth time stamp of the decoded I frame;

3. The method of claim 2, wherein the determining, based on the first timestamp, the second timestamp, the third timestamp, the fourth timestamp, the first PTS, the second PTS, and the fifth timestamp, a deviation that adjusts the time axis of the acquisition device to coincide with the time axis of the electronic device comprises:

4. The method of claim 1, wherein said adjusting the time axis of the acquisition device according to the deviation comprises:

5. The method of claim 4, wherein determining the time of the electronic device when the image frame was acquired based on the deviation of the dynamic time from the acquisition device comprises:

6. The method according to claim 1, wherein the method further comprises:

7. The method of claim 6, wherein after the acquisition of the synchronization frame acquired by the system time and before the transmission of the synchronization frame, the method further comprises:

The transmitting the synchronization frame includes:

8. The method of claim 7, wherein the method further comprises:

9. The method of claim 8, wherein after the acquiring the standard time carried in the synchronization frame acquisition instruction, the determining, according to the deviation and the standard time, a system time of the acquisition device at the standard time is preceded by the method further comprising:

10. An electronic device comprising at least a processor and a memory, the processor being adapted to perform the steps of the time axis adjustment method of any of claims 1-9 when executing a computer program stored in the memory.