WO2024021999A1

WO2024021999A1 - Synchronization method, system, and electronic device

Info

Publication number: WO2024021999A1
Application number: PCT/CN2023/104098
Authority: WO
Inventors: 罗鹏; 林霖
Original assignee: 华为技术有限公司
Priority date: 2022-07-26
Filing date: 2023-06-29
Publication date: 2024-02-01
Also published as: CN117499712A

Abstract

Provided in the embodiments of the present application are a synchronization method, a system, and an electronic device. The method comprises: a server receives image frames sent by a plurality of IPCs and time information corresponding to the image frames, the image frames being obtained by encoding original images, and the plurality of IPCs comprising a first IPC and a second IPC; and when determining that the first IPC and the second IPC are asynchronous in frame time according to the time information corresponding to the image frames sent by the first IPC and the time information corresponding to the image frames sent by the second IPC, sending an encoding adjustment instruction to the first IPC, wherein the encoding adjustment instruction is used for triggering the first IPC to perform encoding adjustment, so as to adjust the time of the first IPC reading a next frame of an original image to be encoded. Thus, frame time synchronization of a plurality of IPCs is realized by data interaction between a server and the IPCs. Compared with the prior art, the present application does not need hardware, thus achieving good robustness, reducing the cost, and facilitating deployment of a plurality of IPCs.

Description

Synchronization method, system and electronic device

This application claims priority to the Chinese patent application submitted to the State Intellectual Property Office of China on July 26, 2022, with application number 202210883502.8 and the application name "Synchronization Method, System and Electronic Device", the entire content of which is incorporated herein by reference. Applying.

Technical field

Embodiments of the present application relate to the field of communications, and in particular, to a synchronization method, system and electronic device.

Background technique

IPC (IP Camera) is generally connected to servers (such as MP (Media Process, media processing terminal) equipment, MEP (Multi-access Edge Platform, multi-access edge platform) equipment, etc.) through the network. Usually, multiple IPCs collect original images and encode the original images into image frames and send them to the server; then the server selects one image frame from the image frames sent by each IPC for video splicing and FVV (Free-Viewpoint Video, Free-angle video) video production and other processing.

Since the startup time of different IPC encodings may be inconsistent, this will lead to differences in the frame outgoing times of different IPCs, which will cause the server to receive frames within the same time period (such as the time period of 8:10:22, 10 milliseconds to 20 milliseconds of the day). The pictures captured by multiple image frames sent by multiple IPCs will have time differences (i.e. frame time differences), that is, the frame times of multiple IPCs are not synchronized; thus, in the spliced image, the connection points of image frames from different IPCs There are problems such as frustration and unsmoothness in the screen.

In order to solve the problem of unsynchronized frame times of multiple IPCs, the existing technology uses data lines to connect two IPCs, and then the hardware control circuit controls the encoded video frame time of the IPCs so that all IPCs output video frame times are consistent. Frame time synchronization. The disadvantage of using hardware to achieve frame time synchronization of multiple IPCs is that it requires additional hardware and is costly; and the use of data lines to connect is not conducive to IPC deployment; in addition, if the hardware line contact is poor, it will cause the frame time to be out of synchronization and have poor robustness. etc.

Contents of the invention

In order to solve the above technical problems, this application provides a synchronization method, system and electronic device. In this method, frame time synchronization between multiple IPCs can be achieved without resorting to hardware through data interaction between the server and the IPC. It has good robustness and can reduce costs and facilitate the deployment of multiple IPCs.

In the first aspect, embodiments of the present application provide a synchronization method. The method includes: first, the server receives image frames sent by multiple network cameras IPC, and time information corresponding to the image frames, where the image frames are obtained by processing the original images. From the encoding, the multiple IPCs include the first IPC and the second IPC. The second IPC is a reference for whether the frame times of the first IPC and the second IPC are synchronized; then, when the server sends time information based on the image frame sent by the first IPC , and the time information corresponding to the image frame sent by the second IPC. When it is determined that the frame time of the first IPC and the second IPC is not synchronized, the server sends a coding adjustment instruction to the first IPC. The coding adjustment instruction is used to trigger the first IPC to perform encoding. Adjust so that the time for the first IPC to read the next frame of the original image to be encoded is adjusted. In this way, the server triggers the first IPC to perform one or more encoding adjustments by sending encoding adjustment instructions to the first IPC one or more times, so that the first IPC reads the time of the next frame of the original image to be encoded one or more times. The adjustment occurs to achieve synchronization of the frame time of the first IPC and the second IPC; thereby avoiding time differences in the pictures captured by multiple image frames received by the server in the same time period, and achieving frame time synchronization of multiple IPCs.

Compared with the existing technology that uses hardware to synchronize the frame time of multiple IPCs, this application can achieve frame time synchronization between multiple IPCs without using hardware through data interaction between the server and the IPC, and is robust. Good; and can reduce costs and facilitate the deployment of multiple IPCs.

Exemplarily, the plurality of IPCs include at least a first IPC and a second IPC.

For example, the original image may refer to an image collected by the photoreceptor of the IPC, that is, a RAW (unprocessed) image.

For example, the server may refer to a single server or a server cluster (such as a cloud server). This application does not make restrictions. For example, the server can be an MP device, a MEP device, etc.

For example, an image frame may also be called a code stream.

For example, the image frame may be a video frame.

According to the first aspect, the server sends a coding adjustment instruction to the first IPC, including: the server sends a coding control message to the first IPC; wherein the coding control message includes an offset time, and the offset time is an image frame sent by the server according to the first IPC The corresponding time information is determined by the time difference between the time information corresponding to the image frame sent by the second IPC; the encoding control message is used to trigger the first IPC to obtain the next frame of the original image to be encoded based on the offset time. Adjustment. In this way, when the first IPC receives the encoding control message, it can move the time to obtain the next frame of the original image to be encoded by the offset time in the encoding control message; and then the first IPC can achieve the same result as the first IPC through one encoding adjustment. Two IPC frame time synchronization.

For example, the offset time may be the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

For example, the offset time may be the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC, and the sum or difference between the preset period. . The preset period may refer to a preset encoding period such as 30 ms, which can be specifically set according to requirements, and this application does not limit this.

According to the first aspect, or any implementation of the above first aspect, the server sends an encoding adjustment instruction to the first IPC, including: the server sends an encoding restart message to the first IPC, and the encoding restart message is used to trigger the first IPC to perform encoding. Restart. In this way, when the first IPC receives the encoding restart message, the encoding can be restarted; after the encoding restart is successful, the next frame of the original image to be encoded can be read and encoded; in turn, the first IPC can read the next frame of the original image to be encoded. The timing of the original image is adjusted. In this case, the first IPC can achieve frame time synchronization with the second IPC through one or more encoding restarts.

According to the first aspect, or any implementation of the above first aspect, the server sends a coding adjustment instruction to the first IPC, including: the server sends a coding refresh message to the first IPC, and the coding refresh message is used to trigger the first IPC to frame refresh. In this way, when the first IPC receives the encoding refresh message, it can refresh the frame and start the next encoding. It will also read the original image to be encoded in the next frame, and encode the original image to be encoded in the next frame. This causes the time for the first IPC to read the next frame of the original image to be encoded to be adjusted. In this case, the first IPC can achieve frame time synchronization with the second IPC through one or more frame refreshes.

According to the first aspect, or any implementation of the above first aspect, determining that the frame time of the first IPC and the second IPC is not synchronized includes: the server determines synchronization time information corresponding to the image frame sent by the first IPC, and The time difference between the synchronization time information corresponding to the image frames sent by the two IPCs is greater than the threshold.

For example, when the time difference between the synchronization time information corresponding to the image frame sent by the first IPC and the synchronization time information corresponding to the image frame sent by the second IPC is less than or equal to the threshold, it can be determined that the first IPC and the second IPC Frame time synchronization.

According to the first aspect, or any implementation of the above first aspect, the time information corresponding to the image frame sent by each IPC in the plurality of IPCs is the time for each IPC to read the next frame of the original image to be encoded. In this way, the time information corresponding to each image frame sent by each IPC is closer to the actual time of the picture captured by each image frame, so that the time information corresponding to the image frame sent by the first IPC determined by the server corresponds to the image frame sent by the second IPC The time difference between the time information is more accurate, and the server sends fewer encoding adjustment instructions to the first IPC, which can achieve frame time synchronization between the first IPC and the second IPC and reduce the time required to synchronize the first IPC and the second IPC. The duration of the frame time.

In the second aspect, embodiments of the present application provide a synchronization method. The method includes: first, the first IPC reads the original image to be encoded; then, the first IPC encodes the original image into an image frame, and the first IPC determines the image Time information corresponding to the frame; subsequently, the first IPC sends the image frame and time information corresponding to the image frame. After that, the first IPC can receive the encoding adjustment instruction. The encoding adjustment instruction is based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC. It is generated and sent when the frame time of the first IPC and the second IPC is not synchronized. The second IPC is a reference for whether the frame time of the first IPC and the second IPC is synchronized; then, the first IPC performs coding adjustment according to the coding adjustment instruction. , so that the time for the first IPC to read the next frame of the original image to be encoded is adjusted. In this way, the first IPC implements one or more encoding adjustments according to one or more encoding adjustment instructions sent by the server, so that the time to read the next frame of the original image to be encoded is adjusted one or more times to achieve the same goal. The outgoing frame time of the second IPC is synchronized; thereby avoiding time differences in pictures captured by multiple image frames received by the server within the same time period, and achieving frame time synchronization of multiple IPCs.

For example, the second IPC can perform the following steps: first, the second IPC reads the original image to be encoded; then, the second IPC encodes the original image into an image frame, and the second IPC determines the time information corresponding to the image frame; Subsequently, the second IPC sends the image frame and time information corresponding to the image frame.

For example, an image frame may also be called a code stream.

For example, the image frame may be a video frame.

According to the second aspect, the first IPC determines the time information corresponding to the image frame, including: the first IPC determines the time information corresponding to the image frame based on the time when the original image to be encoded is read. In this way, the time information corresponding to each image frame sent by each IPC is closer to the time when the picture captured by each image frame actually occurred, so that the time information corresponding to the image frame sent by the first IPC determined by the server is consistent with the image frame sent by the second IPC. The time difference between the corresponding time information is more accurate, and the first IPC performs fewer encoding adjustments to achieve frame time synchronization with the second IPC, reducing the length of time to synchronize the frame time of the first IPC and the second IPC.

It should be noted that the time information can be determined based on any time before the image frame is sent. For example, the time information corresponding to the image frame can be determined based on the time when the original image is read. For another example, the time information corresponding to the image frame is determined based on the time at which the image frame is obtained by encoding. For example, the time information corresponding to the image frame is determined based on the time of various coding steps (such as coding prediction, quantization, entropy coding, etc.) performed during the coding process; etc., this application does not limit this.

According to the second aspect, or any implementation of the above second aspect, the first IPC sends the image frame and the time information corresponding to the image frame, including: the first IPC encapsulates the time information corresponding to the image frame into the extension field of the image frame , send the encapsulated image frame.

For example, the first IPC may first encapsulate the image frame according to the video encoding protocol, and when encapsulating the image frame according to the video encoding protocol, encapsulate the time information into the first extension field of the image frame (that is, the first extension field of the video encoding protocol). extension field). Then, the first IPC performs a second encapsulation on the first encapsulated image frame according to the network transmission protocol, and sends the second encapsulated image frame.

For example, the first IPC can perform the first encapsulation of the image frame according to the video encoding protocol; then, perform the second encapsulation of the first encapsulated image frame according to the network transmission protocol, and when encapsulating the image frame according to the network transmission protocol, The time information is encapsulated into the second extension field of the image frame (that is, the extension field of the network transmission protocol); the first IPC then sends the second encapsulated image frame.

According to the second aspect, or any implementation of the second aspect above, the encoding adjustment indication is a encoding control message, the encoding control message includes an offset time, and the offset time is time information corresponding to the image frame sent by the server based on the first IPC. The time difference between the time information corresponding to the image frame sent by the second IPC is determined; the first IPC performs encoding adjustment according to the encoding adjustment instruction, including: the first IPC reads the original image to be encoded in the next frame according to the offset time time to adjust. In this way, the first IPC can achieve frame time synchronization with the second IPC through one encoding adjustment.

According to the second aspect, or any implementation of the above second aspect, the encoding adjustment indication is a encoding restart message, and the first IPC performs encoding adjustment according to the encoding adjustment instruction, including: the first IPC performs encoding restart according to the encoding restart message. Since the first IPC will perform encoding immediately after restarting encoding, after the encoding restarts successfully, the next frame of the original image to be encoded can be read, and then the next frame of the image to be encoded can be encoded. In this way, the first IPC can also read the original image to be encoded in the next frame. The time is adjusted, that is, the time for the first IPC to read the next frame of the original image to be encoded can be adjusted to the encoding restart success time. In this case, the first IPC can achieve frame time synchronization with the second IPC through one or more encoding restarts.

According to the second aspect, or any implementation of the above second aspect, the coding adjustment indication is a coding refresh message, and the first IPC performs coding adjustment according to the coding adjustment instruction, including: the first IPC performs frame refresh according to the coding refresh message. The frame refresh starts the next encoding. At this time, the first IPC can read the original image to be encoded in the next frame, and then encode the original image to be encoded in the next frame. Furthermore, in this way, the time for the first IPC to read the next frame of the original image to be encoded can also be adjusted, that is, the time for the first IPC to read the next frame of the original image to be encoded is adjusted to the current time. In this case, the first IPC can achieve frame time synchronization with the second IPC through one or more frame refreshes.

The second aspect and any implementation manner of the second aspect respectively correspond to the first aspect and any implementation manner of the first aspect. The technical effects corresponding to the second aspect and any implementation manner of the second aspect may be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, which will not be described again here.

In a third aspect, embodiments of the present application provide a synchronization system. The system includes: multiple network camera IPCs and a server. The multiple IPCs include a first IPC and a second IPC. The second IPC is a link between the first IPC and the second IPC. Reference for frame time synchronization;

The first IPC is used to read the first original image to be encoded; encode the first original image into a first image frame, and determine the time information corresponding to the first image frame; send the first image frame and the first image to the server Time information corresponding to the frame; and receiving a coding adjustment instruction; performing coding adjustment according to the coding adjustment instruction, so that the time for reading the first original image to be encoded in the next frame is adjusted;

The second IPC is used to read the second original image to be encoded; encode the second original image into a second image frame, and determine the time information corresponding to the second image frame; send the second image frame and the second image to the server The time information corresponding to the frame;

The server is configured to receive the first image frame sent by the first IPC and the time information corresponding to the first image frame, and to receive the second image frame sent by the second IPC and the time information corresponding to the second image frame; when according to the first IPC When the time information corresponding to the sent first image frame and the time information corresponding to the second image frame sent by the second IPC are determined to be out of synchronization between the first IPC and the second IPC, a coding adjustment instruction is sent to the first IPC.

The third aspect and any one implementation manner of the server of the third aspect respectively correspond to the first aspect and any one implementation manner of the first aspect. The technical effects corresponding to the third aspect and any implementation of the server of the third aspect can be found in the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be described again here.

Any implementation of the third aspect and the first IPC of the third aspect corresponds to the second aspect and any implementation of the second aspect respectively. For the technical effects corresponding to any implementation of the third aspect and the first IPC, please refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the first IPC, and will not be described again here.

In the fourth aspect, embodiments of the present application provide a network camera IPC, which can be used to implement the second aspect and any implementation manner of the second aspect.

The technical effects corresponding to the fourth aspect and any one of the first IPC implementation methods of the fourth aspect may be referred to the technical effects corresponding to the above-mentioned second aspect and any one implementation method of the second aspect, and will not be described again here.

In a fifth aspect, embodiments of the present application provide an electronic device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the first aspect or A synchronized method in any possible implementation of the first aspect.

The fifth aspect and any implementation manner of the fifth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. The technical effects corresponding to the fifth aspect and any implementation manner of the fifth aspect may be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, and will not be described again here.

In a sixth aspect, embodiments of the present application provide an electronic device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the second aspect or The second aspect is optional synchronization method in the possible implementation.

The sixth aspect and any implementation manner of the sixth aspect respectively correspond to the second aspect and any implementation manner of the second aspect. For the technical effects corresponding to the sixth aspect and any implementation manner of the sixth aspect, please refer to the technical effects corresponding to the above-mentioned second aspect and any implementation manner of the second aspect, which will not be described again here.

In a seventh aspect, embodiments of the present application provide a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, and the signals include the memory computer instructions stored in; when the processor executes the computer instructions, the electronic device is caused to execute the synchronization method in the first aspect or any possible implementation of the first aspect.

The seventh aspect and any implementation manner of the seventh aspect respectively correspond to the first aspect and any implementation manner of the first aspect. The technical effects corresponding to the seventh aspect and any implementation manner of the seventh aspect may be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, and will not be described again here.

In an eighth aspect, embodiments of the present application provide a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, and the signals include the memory computer instructions stored in; when the processor executes the computer instructions, the electronic device is caused to execute the synchronization method in the second aspect or any possible implementation of the second aspect.

The eighth aspect and any implementation manner of the eighth aspect respectively correspond to the second aspect and any implementation manner of the second aspect. For the technical effects corresponding to the eighth aspect and any implementation manner of the eighth aspect, please refer to the technical effects corresponding to the above-mentioned second aspect and any implementation manner of the second aspect, which will not be described again here.

In a ninth aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is run on a computer or processor, it causes the computer or processor to execute the first aspect or the second aspect. A synchronized method in any possible implementation on the one hand.

The ninth aspect and any one implementation manner of the ninth aspect respectively correspond to the first aspect and any one implementation manner of the first aspect. The technical effects corresponding to the ninth aspect and any implementation manner of the ninth aspect may be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, and will not be described again here.

In a tenth aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is run on a computer or processor, it causes the computer or processor to execute the second aspect or the second aspect. Synchronization methods in any possible implementation of both aspects.

The tenth aspect and any one implementation manner of the tenth aspect respectively correspond to the second aspect and any one implementation manner of the second aspect. For the technical effects corresponding to the tenth aspect and any one of the implementation methods of the tenth aspect, please refer to the technical effects corresponding to the above-mentioned second aspect and any one implementation method of the second aspect, which will not be described again here.

In an eleventh aspect, embodiments of the present application provide a computer program product. The computer program product includes a software program. When the software program is executed by a computer or a processor, it causes the computer or processor to execute the first aspect or any possibility of the first aspect. The synchronization method in the implementation.

The eleventh aspect and any implementation manner of the eleventh aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the eleventh aspect and any implementation manner of the eleventh aspect, please refer to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, which will not be described again here.

In a twelfth aspect, embodiments of the present application provide a computer program product. The computer program product includes a software program. When the software program is executed by a computer or a processor, it causes the computer or processor to execute the second aspect or any possibility of the second aspect. The synchronization method in the implementation.

The twelfth aspect and any one implementation manner of the twelfth aspect respectively correspond to the second aspect and any one implementation manner of the second aspect. For the technical effects corresponding to the twelfth aspect and any one of the implementation methods of the twelfth aspect, please refer to the technical effects corresponding to the above-mentioned second aspect and any one implementation method of the second aspect, which will not be described again here.

Description of drawings

Figure 1a is a schematic diagram of an exemplary application scenario;

Figure 1b is a schematic diagram of an exemplary application scenario;

Figure 1c is a schematic diagram of an exemplary application scenario;

Figure 1d is a schematic diagram illustrating an exemplary IPC deployment manner;

Figure 1e is a schematic structural diagram of an exemplary IPC;

Figure 1f is a schematic structural diagram of an exemplary synchronization component;

Figure 2 is a schematic diagram of an exemplary synchronization process;

Figure 3a is a schematic diagram of an exemplary synchronization process;

Figure 3b is a schematic diagram of the frame structure of an encapsulated image frame;

Figure 3c is a schematic diagram of the frame structure of an encapsulated image frame;

Figure 3d is a schematic diagram of the frame structure of an encapsulated image frame;

Figure 3e is a schematic diagram of the message structure of an exemplary encoded control message;

Figure 4 is a schematic diagram of an exemplary synchronization process;

Figure 5 is a schematic diagram of an exemplary synchronization process;

Figure 6 is a schematic diagram of an exemplary synchronization system;

Figure 7 is a schematic structural diagram of an exemplary device.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

The term "and/or" in this article is just an association relationship that describes related objects, indicating that three relationships can exist. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and they exist alone. B these three situations.

The terms “first” and “second” in the description and claims of the embodiments of this application are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first target object, the second target object, etc. are used to distinguish different target objects, rather than to describe a specific order of the target objects.

In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "such as" in the embodiments of the present application is not to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary" or "such as" is intended to present the concept in a concrete manner.

In the description of the embodiments of this application, unless otherwise specified, the meaning of “plurality” refers to two or more. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

Figure 1a is a schematic diagram of an exemplary application scenario.

Referring to Figure 1a, in an exemplary embodiment, n (n is an integer greater than 1) IPCs are connected to the same MP device via a router. After n IPCs are started, each IPC can collect the original image of the target object (that is, RAW (unprocessed) image), then encode the original image into an image frame, and send the image frame to the router; the router converts the image frame Forwarded to MP device. After receiving n image frames sent by IPC, the MP device can process the n image frames sent by IPC, such as video splicing, FVV video production, etc. This application does not limit this. Subsequently, the MP device can send the processed image frames to terminal devices such as mobile phones, tablets, etc.

Figure 1b is a schematic diagram of an exemplary application scenario.

Referring to Figure 1b, in an exemplary embodiment, n IPCs are connected to the same MEP device via a base station and a core network. After n IPCs are started, each Each IPC can collect the original image of the target object, then encode the original image into an image frame, and send the image frame to the base station. After receiving n image frames sent by IPC, the base station can send the n image frames sent by IPC to the core network. The UPF (User Plane Function) network element of the core network will send the n image frames sent by IPC. Sent to MEP equipment. After receiving n image frames sent by IPC, the MEP device can process the n image frames sent by IPC, such as video splicing, FVV video production, etc. This application does not limit this. Subsequently, the MEP device can send the processed image frames to terminal devices such as mobile phones, tablets, etc.

Figure 1c is a schematic diagram of an exemplary application scenario.

Referring to Figure 1c, in an exemplary embodiment, n IPCs are connected to the same MP device through communication satellites. After n IPCs are started, each IPC can collect the original image of the target object, then encode the original image into an image frame, and send the image frame to the communication satellite. After the communication satellite receives n image frames sent by IPC, it can send n image frames sent by IPC to the MP device. After receiving n image frames sent by IPC, the MP device can process the n images sent by IPC, such as video splicing, FVV video production, etc. This application does not limit this. Subsequently, the MP device can send the processed image frames to terminal devices such as mobile phones, tablets, etc.

It should be noted that MP devices and MEP devices may be servers, and the server may refer to a single server or a server cluster (for example, a cloud server, etc.), which is not limited in this application.

It should be noted that the IPC can continuously send multiple image frames to the MP device/MEP device. In one possible way, multiple image frames sent continuously can form a video. In this case, the image frames can also be called video frames.

Figure 1d is a schematic diagram illustrating an exemplary IPC deployment manner. In Figure 1d, the deployment manner of multiple IPCs relative to the target object is shown.

Referring to Figure 1d(1), for example, n (n=6) IPCs (including IPC1, IPC2, IPC3, IPC4, IPC5 and IPC6) can be deployed around the target object, and each IPC is deployed at a different location relative to the target object. angle. It should be understood that Figure 1d(1) is only an example of multiple IPC deployment methods. The number of IPCs deployed around the target object in this application may be more or less than that shown in Figure 1d(1). This application is This is not a restriction.

Exemplarily, the application scenarios using the deployment method shown in Figure 1d(1) can include a variety of applications, such as artistic dance performances, sports competitions, film and television variety shows, Internet celebrity performances, scenic spots, live broadcasts, and exhibition traffic. Movement teaching, etc. In this way, after the MP device/MEP device performs FVV video production on n image frames sent by IPC, it can provide users with a variety of different viewing angles.

Referring to Figure 1d(2), for example, n (n=5) IPCs (including IPC1, IPC2, IPC3, IPC4, and IPC5) are deployed in a fan shape with the user position as the center and toward the target object. It should be understood that Figure 1d(2) is only an example of multiple IPC deployment methods. This application takes the user location as the center and the number of IPCs deployed in a fan shape toward the target object can be greater than that shown in Figure 1d(2). More or less, this application does not limit this.

Exemplarily, application scenarios using the deployment method of Figure 1d(2) may include a variety of scenarios, such as VR (Virtual Reality, virtual reality) scenarios, AR (Augmented Reality, augmented reality) scenarios, etc. In this way, after the MP device/MEP device performs video splicing on the image frames sent by two adjacent IPCs among the n IPCs, a panoramic image for VR scene and AR scene display can be formed.

Figure 1e is a schematic structural diagram of an exemplary IPC.

Referring to FIG. 1 e , by way of example, an IPC may include a photoreceptor, an image cache, an encoding component, and a communication component. It should be understood that Figure 1e is only an example of the present application, and the IPC may include more components than shown in Figure 1e, which is not limited by this application.

Referring to Figure 1e, an exemplary photoreceptor is a light sensor that receives an optical signal through external exposure and converts it into an electrical signal. The photoreceptor and the image buffer can be connected through a cable, which is photoelectrically converted into image data (hereinafter referred to as the original image) and output to the image buffer in real time. The encoding component can read the original image to be encoded from the image cache according to the preset cycle, encode the original image, and output the image frame; the encoded image frame is packaged in the communication component and sent to the MP device or MEP via the network equipment.

It should be noted that the interval at which the IPC continuously sends image frames to the MP device/MEP device may be equal to the preset period.

For example, before MP equipment/MEP equipment performs video splicing or FVV video production, it needs to synchronize the frame times of multiple image frames sent by multiple IPCs (that is, synchronize the frame times within the same time period (such as 8:10:22 of the day) 10 milliseconds to 20 milliseconds time period) to ensure that there is no time difference in the pictures captured by multiple image frames in the same group for video splicing or FVV video production.

Currently, the frame time of multiple IPCs is not synchronized (that is, the server receives multiple image frames sent by multiple IPCs within the same time period (such as the time period of 8:10:22, 10 milliseconds to 20 milliseconds of the day) One of the reasons why there is a time difference in the captured pictures is that different IPC encoding start times (that is, the time when the original image to be encoded is read from the image cache for the first time) are inconsistent. Based on this, this application proposes a synchronization method. This method sends the time information corresponding to the image frame while the IPC sends the image frame; then, the MP device/MEP device determines based on the time information corresponding to the image frame sent by each IPC. Whether the frame times of multiple IPCs are synchronized; when it is determined that the frame times of multiple IPCs are not synchronized, coding adjustment instructions can be sent to some IPCs to trigger some IPCs to adjust their coding, thereby controlling the frame outgoing times of some IPCs, so that multiple IPC outgoing frame time synchronization, thereby achieving frame time synchronization of multiple IPCs.

Figure 1f is a schematic structural diagram of an exemplary synchronization component. Figure 1f shows the synchronization component in the server (such as MP device, MEP device), which is used to ensure that the server side implements the synchronization method of the present application.

Referring to Figure 1f, in an exemplary embodiment, the synchronization component in the MP device may include a time comparison module, a decision-making module and a control module. Among them, the time comparison module is used to compare the time information corresponding to the two image frames sent by the two IPCs to determine the time difference; the decision-making module is used to decide whether to send a coding adjustment instruction based on the time difference; the control module is used to send a coding adjustment instruction.

It should be understood that Figure 1f is only an example of this application, and the synchronization component may include more or fewer modules than shown in Figure 1f, which is not limited by this application.

The synchronization method of this application is described below.

Figure 2 is a schematic diagram illustrating an exemplary synchronization process. In the embodiment of Figure 2, the number of IPCs is n, and the n IPCs are all connected to the same server. The server can refer to a single server or a server cluster; the server can be an MP device or a MEP device.

S201, IPC reads the original image to be encoded.

For example, n IPCs may include 1 second IPC and (n-1) first IPCs, where the second IPC is a reference for whether the frame times of the first IPC and the second IPC are synchronized, and the first IPC It is the IPC except the second IPC among the n IPCs.

In one possible way, n IPCs may negotiate with each other to determine the first IPC and the second IPC among the n IPCs.

In one possible way, the server can determine the first IPC and the second IPC among the n IPCs when starting to synchronize the frame times of multiple IPCs.

For example, when the server starts to synchronize the frame times of multiple IPCs, the server may determine the first IPC and the second IPC among the n IPCs according to preset rules. For example, the preset rules can be set according to requirements, such as random selection, selection according to a preset order, etc. This application does not limit this.

For example, after n IPCs are started, these n IPCs can perform system time synchronization. For example, after each IPC is started, it can interact with the time server and then synchronize with the system time of the time server; in this way, the system time of these n IPCs can be synchronized.

For example, S201 includes S201a and S201b:

S201a, the first IPC reads the original image to be encoded.

S201b, the second IPC reads the original image to be encoded.

For example, after the first IPC is started, the photoreceptor in the first IPC can receive the optical signal and convert the optical signal into an electrical signal; and then output the original image obtained through photoelectric conversion to the image cache in real time. In this way, the image cache can store multiple frames of the original image.

For example, after the first IPC is started, the encoding component is also started; after the encoding component is started, the original image to be encoded can be read from the image cache according to a preset cycle. In this way, each of the (n-1) first IPCs can read the original image to be encoded from the corresponding image cache. It should be noted that the encoding component can read a frame of the original image to be encoded from the image cache in a preset cycle.

For example, after the second IPC is started, the encoding component in the second IPC can also read the original image to be encoded from the image cache according to a preset cycle.

S202, IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

Illustratively, S202 includes S202a and S202b:

S202a: The first IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

S202b: The second IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

For example, after the encoding component of the first IPC reads the original image, on the one hand, the encoding component can encode the original image and output the image frame (that is, the code stream). For example, the encoding method of the original image by the first IPC can be determined according to the requirements. For example, the original image can be encoded by encoding protocols such as H.264, H.265, H.266, etc. This application does not limit this.

For example, after the encoding component of the first IPC reads the original image, on the other hand, the encoding component or the communication component can determine the time information corresponding to the image frame. Among them, the time information corresponding to the image frame can be used by the server to synchronize the frame times of multiple IPCs. The time information can be used to identify the time. For example, the time information can be a timestamp, which is not limited in this application.

In one possible way, the encoding component can determine the time information corresponding to the image frame based on the time when the encoding component reads the original image to be encoded from the image cache. For example, the time when the encoding component reads the original image to be encoded from the image cache can be used as the time information corresponding to the image frame; for another example, the time when the encoding component reads the original image to be encoded from the image cache can be used , the difference from the preset time (such as 0:00:00:10 milliseconds on January 1, 1970) is used as the time information corresponding to the image frame; this application does not limit this. For example, the unit of the time information corresponding to the image frame may be ms (millisecond).

In one possible way, the encoding component can determine the time information corresponding to the image frame based on the time when the image frame is obtained by encoding the encoding component. For example, the coding component can be coded to obtain the time of the image frame as the time information corresponding to the image frame; for another example, the coding component can be coded to obtain the difference between the time of the image frame and the preset time as the time information corresponding to the image frame; This application does not limit this.

In one possible way, the encoding component can determine the time information corresponding to the image frame based on the time of various encoding steps (such as encoding prediction, quantization, entropy encoding, etc.) performed by the encoding component during the encoding process. For example, the encoding component uses the time when the encoding component starts to perform coding prediction as the time information for determining the image frame correspondence; for another example, the encoding component uses the time when the encoding component starts to perform quantization as the time information for determining the image frame correspondence; and so on. There are no restrictions on applications.

In one possible way, the communication component can determine the time information corresponding to the image frame based on the time when the image frame is received. For example, the time when the communication component receives the image frame is used as the time information corresponding to the image frame; for another example, the difference between the time when the communication component receives the image frame and the preset time is used as the time information corresponding to the image frame.

It should be understood that any time before sending the image frame to the server can be used as the time information corresponding to the image frame, and this application does not limit this.

In this way, each of the (n-1) first IPCs can obtain the image frame and the time information corresponding to the image frame.

For example, after reading the original image, the encoding component of the second IPC can refer to the above method to obtain the image frame and the time information corresponding to the image frame.

S203. The IPC sends the image frame and the time information corresponding to the image frame to the server.

Illustratively, S203 includes S203a and S203b:

S203a: The first IPC sends the image frame and time information corresponding to the image frame to the server.

S203b: The second IPC sends the image frame and time information corresponding to the image frame to the server.

For example, each of the (n-1) first IPCs can send the obtained image frame and the time information corresponding to the image frame to the same server.

For example, the second IPC may also send the obtained image frame and the time information corresponding to the image frame to the server.

S204: Based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC, the server determines that the frame time of the first IPC and the second IPC are not synchronized, and sends the code to the first IPC. Adjustment instructions.

For example, the server can use the time information corresponding to the image frame sent by the second IPC as a reference, and determine whether the frame time of each first IPC and the second IPC is synchronized based on the time information corresponding to the image frame sent by each first IPC. .

For example, for a first IPC, the server may determine the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC. If the time difference is greater than the threshold, it may be determined that the frame times of the first IPC and the second IPC are not synchronized. If the time difference is less than or equal to the threshold, the frame time of the first IPC and the second IPC can be determined. Synchronize. The threshold can be set according to requirements, and this application does not limit this.

In this way, according to the above method, it can be judged whether the frame time of each first IPC among the (n-1) first IPCs is synchronized with the second IPC; that is, whether the frame time of n IPCs is synchronized is judged.

It should be understood that the server can serially determine whether the frame times of the (n-1) first IPC and the second IPC are synchronized, or it can also determine in parallel the frames of the (n-1) first IPC and the second IPC. Whether the time is synchronized; this application does not limit this.

For example, for a first IPC, if the server determines that the first IPC and the second IPC frame time are not synchronized, the first IPC needs to be synchronized. At this time, a coding adjustment instruction can be generated; and then the coding adjustment Instructions are sent to the first IPC. If the server determines that the first IPC and the second IPC frame time are synchronized, the synchronization adjustment of the first IPC can be ended, and there is no need to generate a coding adjustment instruction at this time.

S205: The first IPC performs encoding adjustment according to the encoding adjustment instruction, so that the time for the first IPC to read the original image of the next frame is adjusted.

For example, among the (n-1) first IPCs, after receiving the encoding adjustment instruction sent by the server, any first IPC can perform encoding adjustment according to the encoding adjustment instruction, so that the first IPC reads the next frame to be processed. The encoding of the original image is adjusted in time. In this way, when it is time to read the next frame of the original image to be encoded, you can return to execute the above-mentioned S201 to S205, and so on until the frame times of (n-1) first IPCs and second IPCs are all synchronized. .

It should be noted that in one possible way, among the (n-1) first IPCs, there may be only a part of the first IPCs and the second IPCs whose frame times are out of sync. The IPC sends a coding adjustment indication (not shown in Figure 2). In one possible way, the frame times of the (n-1) first IPCs and the second IPCs are not synchronized. At this time, the server can send coding adjustment instructions to the (n-1) first IPCs (as shown in Figure 2 shown).

It should be noted that for a first IPC, frame time synchronization with the second IPC may be achieved after receiving a coding adjustment instruction sent by the server once and performing coding adjustment according to the coding adjustment instruction; that is, the The first IPC can achieve frame time synchronization with the second IPC by performing one coding adjustment. It is also possible that frame time synchronization with the second IPC can be achieved only after receiving the encoding adjustment instructions sent multiple times by the server and performing encoding adjustments according to the encoding adjustment instructions sent multiple times; that is, the first IPC performs multiple encoding time adjustments. Only then can frame time synchronization with the second IPC be achieved.

In this way, the first IPC performs one or more encoding conditions according to the encoding adjustment instructions sent one or more times by the server to adjust the time of reading the next frame of the original image to be encoded one or more times. Realize frame time synchronization with the second IPC; thereby avoiding time differences in pictures captured by multiple image frames received by the server in the same time period, and realizing frame time synchronization of multiple IPCs.

Figure 3a is a schematic diagram illustrating an exemplary synchronization process. In the embodiment of Figure 3a, the coding adjustment indication is a coding control message.

S301, IPC reads the original image to be encoded.

For example, S301 includes S301a and S301b:

S301a, the first IPC reads the original image to be encoded.

S301b: The second IPC reads the original image to be encoded.

S302, IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

Exemplarily, S302 includes S302a and S302b:

S302a: The first IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

S302b: The second IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

For example, for S301 to S302, reference may be made to the description of the above S201 to S202, which will not be described again here.

S303. The IPC sends the image frame and the time information corresponding to the image frame to the server.

Exemplarily, S303 includes S303a and S303b:

S303a: The first IPC sends the image frame and time information corresponding to the image frame to the server.

S303b: The second IPC sends the image frame and time information corresponding to the image frame to the server.

For example, the first IPC may encapsulate the time information corresponding to the image frame and the image frame together, and then send the encapsulated image frame and the time information corresponding to the image frame to the server. The following takes a first IPC as an example to describe the process of the first IPC sending image frames and time information corresponding to the image frames.

Figure 3b is a schematic diagram of the frame structure of an encapsulated image frame.

For example, the time information corresponding to the image frame can be encapsulated into the extension field of the image frame; and then the encapsulated image frame is sent to the server.

For example, an information header (header) and an extension field can be encapsulated for the image frame; and the time information corresponding to the image frame can be encapsulated in the extension field; as shown in Figure 3b.

Figure 3c is a schematic diagram of the frame structure of an encapsulated image frame. In the embodiment of Figure 3c, this application encapsulates the time information corresponding to the image frame in the extension field of the video coding protocol.

Exemplarily, after the encoding component of the first IPC encodes the original image to obtain the image frame, the image frame can be first encapsulated according to a video encoding protocol (such as H.264, H.265, H.266 and other encoding protocols); While first encapsulating the image frame according to the video encoding protocol, time information corresponding to the image frame is encapsulated into the first extension field of the image frame. For example, in the process of first encapsulating the image frame according to the video encoding protocol, the image frame may be encapsulated with the information header (header) of the video encoding protocol (hereinafter referred to as information header 1 (header1)) and the first extension field. (That is, the extension field of the video coding protocol); and the time information corresponding to the image frame is encapsulated in the first extension field; as shown in Figure 3c (1).

In this case, the time information corresponding to the image frame can be determined based on any time before the image frame is obtained by encoding, such as the time when the encoding component reads the original image to be encoded from the image cache, or various various tasks performed by the encoding component during the encoding process. The time of the encoding step, such as the time of encoding the image frame, etc., is not limited in this application.

Then, the encoding component can send the first encapsulated image frame to the communication component, and the communication component performs a second encapsulation on the first encapsulated image frame according to the network transmission protocol to obtain a second encapsulated image frame. For example, during the second encapsulation process of the first encapsulated image frame according to the network transmission protocol, the first encapsulated image frame may be encapsulated with a header of the network transmission protocol (hereinafter referred to as the information header). 2 (header2)) and the extension field of the network transmission protocol (hereinafter referred to as the second extension field); as shown in Figure 3c(2). Subsequently, the second encapsulated image frame is sent to the server.

Figure 3d is a schematic diagram of the frame structure of an encapsulated image frame. In the embodiment of Figure 3d, this application encapsulates the time information corresponding to the image frame in the extension field of the network transmission protocol.

For example, after the encoding component of the first IPC encodes the original image to obtain the image frame, the image frame can be first encapsulated according to the video encoding protocol to obtain the first encapsulated image frame. For example, in the process of first encapsulating the image frame according to the video encoding protocol, the image frame may be encapsulated with the information header (header) of the video encoding protocol (hereinafter referred to as information header 1 (header1)) and the first extension field. (That is, the extension field of the video encoding protocol); as shown in Figure 3d(1).

Then, the encoding component can send the first encapsulated image frame to the communication component, and the communication component performs a second encapsulation on the first encapsulated image frame according to the network transmission protocol, and performs a third encapsulation on the first encapsulated image frame. While encapsulating, the time information corresponding to the image frame is encapsulated into the second extension field of the image frame. For example, during the second encapsulation process of the first encapsulated image frame according to the network transmission protocol, the image frame may be encapsulated with a header (header) of the network transmission protocol (hereinafter referred to as header 2 (header2)). and the second extension field (that is, the extension field of the network transmission protocol); and the time information corresponding to the image frame is encapsulated in the second extension field; as shown in Figure 3d(2). Subsequently, the second encapsulated image frame is sent to the server.

In this case, the time information corresponding to the image frame can be determined based on any time before the second encapsulation of the first encapsulated image frame, such as the time when the encoding component reads the original image from the image cache, or the encoding process of the encoding component The time of various coding steps performed in the process, such as the time of encoding to obtain the image frame, the time of first encapsulation of the image frame, etc., are not limited by this application.

For example, the second IPC may also send the image frame and the time information corresponding to the image frame to the server by encapsulating the time information corresponding to the image frame in the extension field of the image frame in the above manner, which will not be described again here.

S304: The server determines whether the first IPC and the second IPC are synchronized based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

For example, S304 may refer to the description of S204 above, which will not be described again here.

S305: When the server determines that the first IPC and the second IPC are out of synchronization, it sends an encoding control message to the first IPC.

For example, for a first IPC, when the server determines that the first IPC and the second IPC are not synchronized, the time information corresponding to the image frame sent by the first IPC can be compared with the time information corresponding to the image frame sent by the second IPC. The time difference between time information generates a coded control message (Control Message). The encoded control message may include a message header and a time offset field, as shown in Figure 3e. The time offset field is used to carry the offset time. The offset time can be determined based on the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

In one possible way, the offset time may be the time difference obtained by subtracting the time information corresponding to the image frame sent by the first IPC from the time information corresponding to the image frame sent by the first IPC. At this time, the offset time may be a positive number or a negative number.

In one possible way, if the time information corresponding to the image frame sent by the second IPC is less than the time information corresponding to the image frame sent by the second IPC, then the time information corresponding to the image frame sent by the second IPC is minus the first The time information corresponding to the image frame sent by IPC is a negative number. At this time, the sum of the time difference plus the preset period can be calculated, and the sum is used as the offset time. If the time information corresponding to the image frame sent by the second IPC is greater than the time information corresponding to the image frame sent by the second IPC, then the time information corresponding to the image frame sent by the second IPC is subtracted from the time information corresponding to the image frame sent by the first IPC. Time information, the resulting time difference is a positive number. At this time, the time difference can be used as the offset time. In other words, in this way, the offset times obtained are all positive numbers.

In one possible way, if the time information corresponding to the image frame sent by the second IPC is greater than the time information corresponding to the image frame sent by the second IPC, then the time information corresponding to the image frame sent by the second IPC is minus the first The time information corresponding to the image frame sent by IPC is a positive number. At this time, the time difference minus the preset period can be calculated, and the difference can be used as the offset time. If the time information corresponding to the image frame sent by the second IPC is less than the time information corresponding to the image frame sent by the second IPC, then the time information corresponding to the image frame sent by the second IPC is subtracted from the time information corresponding to the image frame sent by the first IPC. Time information, the resulting time difference is a negative number. At this time, the time difference can be used as the offset time. In other words, in this way, the offset times obtained are all negative numbers.

It should be understood that this application does not limit the method of determining the offset time.

S306: The first IPC adjusts the time for reading the original image to be encoded in the next frame according to the offset time.

For example, after receiving the encoding control message, the first IPC can parse the encoding control message and obtain the offset time carried in the encoding control message. Then, the first IPC determines the time to read the next frame of the original image to be encoded according to the preset period; then, the time to read the next frame of the original image to be encoded is moved by the offset time. In this way, there will be no time difference between the next image frame sent by the first IPC and the next image frame sent by the second IPC, thereby achieving frame time synchronization between the first IPC and the second IPC.

In this way, after the first IPC receives an encoding control message sent by the server once and adjusts the time to read the original image to be encoded in the next frame according to the offset time in the encoding control message, it can realize communication with the second IPC. Frame time synchronization. That is to say, in the embodiment of Figure 3a, for each first IPC, frame time synchronization with the second IPC can be achieved through one encoding adjustment.

In addition, if the time information of the image frame is determined based on the time when the original image to be encoded is read, the time information corresponding to each image frame sent by each IPC is closer to the time when the picture captured by each image frame actually occurred; in this way, The time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC determined by the server is more accurate, and then the server sends fewer encoding adjustment instructions to the first IPC, and the third By performing fewer encoding adjustments on one IPC, the frame time synchronization of the first IPC and the second IPC can be achieved, and the duration of synchronizing the frame time of the first IPC and the second IPC can be reduced.

It should be noted that for the embodiment of Figure 3a, for an IPC that does not have the function of processing encoding control messages, a service/application program for processing encoding control messages can be added to the IPC; in this way, the IPC can complete Figure 3a Medium S306. Thereby, it cooperates with the server according to the steps in the embodiment of Figure 3a to achieve frame time synchronization.

Figure 4 is a schematic diagram illustrating an exemplary synchronization process. In the embodiment of Figure 4, the coding adjustment indication is a coding restart message.

S401, IPC reads the original image to be encoded.

For example, S401 includes S401a and S401b:

S401a, the first IPC reads the original image to be encoded.

S401b: The second IPC reads the original image to be encoded.

S402, IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

Exemplarily, S402 includes S402a and S402b:

S402a: The first IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

S402b: The second IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

For example, for S401 to S402, reference may be made to the description of S201 to S202 above, which will not be described again here.

S403. The IPC sends the image frame and the time information corresponding to the image frame to the server.

Exemplarily, S403 includes S403a and S403b:

S403a: The first IPC sends the image frame and time information corresponding to the image frame to the server.

S403b: The second IPC sends the image frame and time information corresponding to the image frame to the server.

For example, S403 may refer to the description of S303 above, which will not be described again here.

S404: The server determines whether the first IPC and the second IPC are synchronized based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

For example, S404 may refer to the description of S204 above, which will not be described again here.

S405: When the server determines that the first IPC and the second IPC are out of synchronization, it sends an encoding restart message to the first IPC.

For example, the encoding restart message (Restart Message) is used to trigger the first IPC to restart encoding, that is, to restart the encoding component.

S406: The first IPC performs encoding restart according to the encoding restart message.

For example, after receiving the encoding restart message, the first IPC can restart the encoding component; after the encoding component restarts successfully, it can start to obtain the next frame of original image to be encoded from the image cache, and then execute the above S402-S403. In this way, the time for the first IPC to read the next frame of the original image to be encoded from the image cache is adjusted to the encoding restart success time.

It should be noted that one situation may be that after the first IPC restarts encoding, the time when it reads the next frame of the original image to be encoded from the image cache is the same as the time when the second IPC reads the next frame from the image cache. The time of the original image to be encoded is the same; in this way, after the first IPC completes S401a to S403a again, and the second IPC completes S401b to S403b again, the server can receive the next image frame and image frame sent by the first IPC The corresponding time information, as well as the next image frame sent by the second IPC and the time information corresponding to the image frame, and the frame time synchronization of the first IPC and the second IPC can be determined without generating a coding restart message. In other words, the first IPC can achieve frame time synchronization with the second IPC after a coding restart.

One situation may be that after the first IPC restarts encoding, the time when it reads the next frame of the original image to be encoded from the image cache is the same as the time when the second IPC reads the next frame of the original image to be encoded from the image cache. The times are still different; in this way, after the first IPC finishes executing S401a~S403a again, and the second IPC finishes executing S401b~S403b again, the server can receive the next image frame and the time corresponding to the image frame sent by the first IPC information, as well as the next image frame and the time information corresponding to the image frame sent by the second IPC; and it can be determined that the frame time of the first IPC and the second IPC is not synchronized. At this time, the server can send the encoding restart message to the first IPC again. , and then the first IPC performs coding restart according to the coding restart message again. This cycle repeats until the frame times of the first IPC and the second IPC are synchronized. In other words, the first IPC needs to undergo multiple encoding restarts before it can achieve frame time synchronization with the second IPC.

That is to say, in the embodiment of FIG. 4 , the first IPC needs to undergo one or more encoding restarts before it can achieve frame time synchronization with the second IPC.

Figure 5 is a schematic diagram illustrating an exemplary synchronization process. In the embodiment of Figure 5, the encoding adjustment indication is a encoding refresh message.

S501: At the beginning, the IPC reads the original image to be encoded.

For example, S501 includes S501a and S501b:

S501a, at the beginning, the first IPC reads the original image to be encoded.

S501b, at the beginning, the second IPC reads the original image to be encoded.

S502, IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

Exemplarily, S502 includes S502a and S502b:

S502a: The first IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

S502b: The second IPC encodes the original image into an image frame, and determines the time information corresponding to the image frame.

For example, for S501 to S502, reference may be made to the description of S201 to S202 above, which will not be described again here.

S503. The IPC sends the image frame and the time information corresponding to the image frame to the server.

Exemplarily, S503 includes S503a and S503b:

S503a: The first IPC sends the image frame and time information corresponding to the image frame to the server.

S503b: The second IPC sends the image frame and time information corresponding to the image frame to the server.

For example, S503 may refer to the description of S303 above, which will not be described again here.

S504: The server determines whether the first IPC and the second IPC are synchronized based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC.

For example, S504 may refer to the description of S204 above, which will not be described again here.

S505: When the server determines that the first IPC and the second IPC are out of synchronization, it sends a coding refresh message to the first IPC.

For example, the encoding refresh message (Refresh Messgae) is used to trigger the first IPC to perform frame refresh, and is also used to trigger the encoding component in the first IPC to perform frame refresh.

S506: The first IPC performs frame refresh according to the coding refresh message.

For example, after receiving the encoding refresh message, the first IPC can trigger the encoding component to perform frame refresh, that is, trigger the encoding component to immediately encode an image frame. At this time, the encoding component can immediately obtain the next frame to be encoded from the image cache. original image, and then execute the above S502~S503. In this way, the time when the first IPC reads the next frame of the original image to be encoded from the image buffer is adjusted to the current time.

It should be noted that one situation may be that after the first IPC performs this encoding refresh, the time when it reads the next frame of the original image to be encoded from the image cache is the same as the time when the second IPC reads the next frame from the image cache. The time of the original image to be encoded is the same; in this way, after the first IPC completes S501a ~ S503a again, and the second IPC completes S501b ~ S503b again, the server can receive the next image frame and image frame sent by the first IPC The corresponding time information, as well as the next image frame sent by the second IPC and the time information corresponding to the image frame can be determined, and the frame time synchronization of the first IPC and the second IPC can be determined without generating a coding refresh message. In other words, the first IPC can achieve frame time synchronization with the second IPC after one frame refresh.

One situation may be that after the first IPC performs this encoding refresh, the time when it reads the next frame of original image to be encoded from the image cache is the same as the time when the second IPC reads the next frame of original image to be encoded from the image cache. The times are still different; in this way, after the first IPC finishes executing S501a~S503a again, and the second IPC finishes executing S501b~S503b again, the server can receive the next image frame and the time corresponding to the image frame sent by the first IPC information, as well as the next image frame sent by the second IPC and the time information corresponding to the image frame; and it can be determined that the frame time of the first IPC and the second IPC is not synchronized. At this time, the server can send a coding refresh message to the first IPC again. , and then the first IPC performs encoding refresh according to the encoding refresh message again. This cycle repeats until the frame times of the first IPC and the second IPC are synchronized. In other words, the first IPC can achieve frame time synchronization with the second IPC after multiple frame refreshes.

That is to say, in the embodiment of FIG. 5 , the first IPC needs one or more frame refreshes to achieve frame time synchronization with the second IPC.

It should be noted that the existing IPC has a coding restart function and a coding refresh function, so S406 in the embodiment of FIG. 4 and S506 in the embodiment of FIG. 5 can be executed. In this way, multiple IPCs cooperate with the server according to the steps in the embodiment of Figure 4, or cooperate with the server according to the steps in the embodiment of Figure 5, so that frame time synchronization can be achieved.

It should be noted that the server can periodically synchronize the frame times of multiple IPCs according to a set time period. The set time period can be set according to requirements and can be greater than the preset period. This application does not limit this.

Figure 6 is a schematic diagram of an exemplary synchronization system. In the embodiment of Figure 6, the synchronization system in Figure 6 may include multiple IPCs and servers. The multiple IPCs may include at least a first IPC and a second IPC. The second IPC is a combination of the first IPC and the second IPC. Reference for frame time synchronization; where,

The first IPC is used to read the first original image to be encoded; encode the first original image into a first image frame, and determine the time information corresponding to the first image frame; send the first image frame and the first image to the server Time information corresponding to the frame; and receiving coding adjustment instructions; performing coding adjustments according to the coding adjustment instructions;

The server is configured to receive the first image frame sent by the first IPC and the time information corresponding to the first image frame, and to receive the second image frame sent by the second IPC and the time information corresponding to the second image frame; when according to the first IPC When the time information corresponding to the first image frame sent and the time information corresponding to the second image frame sent by the second IPC are determined to be out of sync between the frame times of the first IPC and the second IPC, a coding adjustment instruction is sent to the first IPC.

Exemplarily, the server is configured to send a coding control message to the first IPC; wherein the coding control message includes an offset time, and the offset time is based on the time information corresponding to the first image frame sent by the first IPC, and is different from the time information sent by the second IPC. The time difference between the time information corresponding to the second image frame is determined; the encoding control message is used to trigger the first IPC to adjust the time to obtain the first original image to be encoded in the next frame according to the offset time.

Exemplarily, the server is configured to send an encoding restart message to the first IPC, and the encoding restart message is used to trigger the first IPC to restart encoding.

Exemplarily, the server is configured to send a coding refresh message to the first IPC, and the coding refresh message is used to trigger the first IPC to refresh the frame.

For example, if the time difference between the synchronization time information corresponding to the first image frame sent by the first IPC and the synchronization time information corresponding to the second image frame sent by the second IPC is greater than the threshold, it is determined that the first IPC and the second The frame time of IPC is not synchronized.

Exemplarily, the first IPC is used to determine the time information corresponding to the first image frame based on the time when the first original image to be encoded is read.

Exemplarily, the first IPC is used to encapsulate the time information corresponding to the first image frame into an extension field of the first image frame, and send the encapsulated first image frame.

Exemplarily, the coding adjustment indication is a coding control message, and the coding control message includes an offset time. The offset time is based on the time information corresponding to the first image frame sent by the first IPC and the time corresponding to the second image frame sent by the second IPC. The time difference between the information is determined; the first IPC is used to adjust the time to read the first original image to be encoded in the next frame according to the offset time.

For example, the coding adjustment indication is a coding restart message; the first IPC is used to restart coding according to the coding restart message.

Exemplarily, the coding adjustment indication is a coding refresh message, and the first IPC is used to refresh the frame according to the coding refresh message.

Exemplarily, the second IPC is used to determine the time information corresponding to the second image frame based on the time when the second original image to be encoded is read.

Exemplarily, the second IPC is used to encapsulate the time information corresponding to the second image frame into an extension field of the second image frame, and send the encapsulated second image frame.

It should be understood that the technical effects of the steps performed by the server and the IPC can be referred to the technical effects described in any of the embodiments in Figure 2, Figure 3a, Figure 4 and Figure 5, and will not be described again here.

In one example, FIG. 7 shows a schematic block diagram of a device 700 according to an embodiment of the present application. The device 700 may include: a processor 701 and a transceiver/transceiver pin 702, and optionally, a memory 703.

The various components of device 700 are coupled together by bus 704, which includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are referred to as bus 704 in the figure.

Optionally, the memory 703 may be used to store instructions in the foregoing method embodiments. The processor 701 can be used to execute instructions in the memory 703, and control the receiving pin to receive signals, and control the transmitting pin to send signals.

The device 700 may be the electronic device or a chip of the electronic device in the above method embodiment.

All relevant content of each step involved in the above method embodiments can be quoted from the functional description of the corresponding functional module, and will not be described again here.

This embodiment also provides a computer-readable storage medium. Computer instructions are stored in the computer-readable storage medium. When the computer instructions are run on an electronic device, the electronic device causes the electronic device to execute the above-mentioned related method steps to implement the above-mentioned embodiments. sync method.

This embodiment also provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform the above related steps to implement the synchronization method in the above embodiment.

In addition, embodiments of the present application also provide a device. This device may be a chip, a component or a module. The device may include a connected processor and a memory. The memory is used to store computer execution instructions. When the device is running, The processor can execute computer execution instructions stored in the memory, so that the chip executes the synchronization method in each of the above method embodiments.

Among them, the electronic devices, computer-readable storage media, computer program products or chips provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the above provided The beneficial effects of the corresponding methods will not be described again here.

Through the description of the above embodiments, those skilled in the art can understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In practical applications, the above functions can be allocated to different modules according to needs. The functional module is completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another device, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separate. A component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

Any contents of various embodiments of this application, as well as any contents of the same embodiment, can be freely combined. Any combination of the above is within the scope of this application.

Integrated units may be stored in a readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or contribute to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium , including several instructions to cause a device (which can be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code.

The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Inspired by this application, many forms can be made without departing from the purpose of this application and the scope protected by the claims, all of which fall within the protection of this application.

The steps of the methods or algorithms described in connection with the disclosure of the embodiments of this application can be implemented in hardware or by a processor executing software instructions. Software instructions can be composed of corresponding software modules. Software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read only memory (Read Only Memory, ROM), erasable programmable read only memory ( Erasable Programmable ROM (EPROM), electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), register, hard disk, removable hard disk, compact disc (CD-ROM) or any other form of storage media well known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage media may be located in an ASIC.

Those skilled in the art should realize that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a general purpose or special purpose computer.

Claims

A synchronization method, characterized in that the method includes:

The server receives image frames sent by multiple network cameras IPC, and time information corresponding to the image frames, wherein the image frames are obtained by encoding the original images, and the multiple IPCs include a first IPC and a second IPC. , the second IPC is a reference for whether the frame time of the first IPC and the second IPC is synchronized;

When the server determines that the frame time of the first IPC and the second IPC are different based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC. During synchronization, the server sends a coding adjustment indication to the first IPC. The coding adjustment indication is used to trigger the first IPC to perform coding adjustment so that the first IPC reads the original to be coded in the next frame. The timing of the image is adjusted.
The method according to claim 1, characterized in that the server sends a coding adjustment instruction to the first IPC, including:

The server sends an encoding control message to the first IPC;

Wherein, the encoding control message includes an offset time, the offset time is the time information corresponding to the image frame sent by the server according to the first IPC, and the time information corresponding to the image frame sent by the second IPC The time difference between the two frames is determined; the encoding control message is used to trigger the first IPC to adjust the time to acquire the next frame of the original image to be encoded according to the offset time.
The method according to claim 1, characterized in that the server sends a coding adjustment instruction to the first IPC, including:

The server sends an encoding restart message to the first IPC, where the encoding restart message is used to trigger the first IPC to restart encoding.
The method according to claim 1, characterized in that the server sends a coding adjustment instruction to the first IPC, including:

The server sends a coding refresh message to the first IPC, where the coding refresh message is used to trigger the first IPC to refresh the frame.
The method according to any one of claims 1 to 4, wherein determining that the frame time of the first IPC and the second IPC is out of synchronization includes:

The server determines that the time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC is greater than a threshold.
The method according to any one of claims 1 to 5, characterized in that,

The time information corresponding to the image frame sent by each IPC among the plurality of IPCs is the time at which each IPC reads the original image to be encoded.
A synchronization method, characterized in that the method includes:

The first network camera IPC reads the original image to be encoded;

The first IPC encodes the original image into an image frame, and the first IPC determines the time information corresponding to the image frame;

The first IPC sends the image frame and time information corresponding to the image frame;

The first IPC receives a coding adjustment indication, and the coding adjustment indication is that the server determines the first IPC based on the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC. Generated and sent when the frame time of the second IPC is not synchronized, the second IPC is a reference for whether the frame time of the first IPC and the second IPC are synchronized;

The first IPC performs encoding adjustment according to the encoding adjustment instruction, so that the first IPC reads the original to be encoded in the next frame. The timing of the image is adjusted.
The method of claim 7, wherein the first IPC determines the time information corresponding to the image frame, including:

The first IPC determines the time information corresponding to the image frame based on the time when the original image to be encoded is read.
The method according to claim 7 or 8, characterized in that the first IPC sends the image frame and the time information corresponding to the image frame, including:

The first IPC encapsulates the time information corresponding to the image frame into an extension field of the image frame, and sends the encapsulated image frame.
The method according to any one of claims 7 to 9, characterized in that the encoding adjustment indication is an encoding control message, the encoding control message includes an offset time, and the offset time is the time specified by the server according to the The time difference between the time information corresponding to the image frame sent by the first IPC and the time information corresponding to the image frame sent by the second IPC is determined;

The first IPC performs encoding adjustment according to the encoding adjustment instruction, including:

The first IPC adjusts the time to read the original image to be encoded in the next frame according to the offset time.
The method according to any one of claims 7 to 9, characterized in that the coding adjustment indication is a coding restart message, and the first IPC performs coding adjustment according to the coding adjustment indication, including:

The first IPC performs encoding restart according to the encoding restart message.
The method according to any one of claims 7 to 9, characterized in that the coding adjustment indication is a coding refresh message, and the first IPC performs coding adjustment according to the coding adjustment indication, including:

The first IPC performs frame refresh according to the encoding refresh message.
A synchronization system, characterized in that the frame time synchronization system includes: multiple network camera IPCs and servers, the multiple IPCs include a first IPC and a second IPC, the second IPC is the first IPC A reference to whether the frame time of the second IPC is synchronized;

The first IPC is used to read the first original image to be encoded; encode the first original image into a first image frame, and determine the time information corresponding to the first image frame; send to the server The first image frame and the time information corresponding to the first image frame; and receiving an encoding adjustment instruction; performing encoding adjustment according to the encoding adjustment instruction so as to read the time of the first original image to be encoded in the next frame Adjustments occur.

The second IPC is used to read the second original image to be encoded; encode the second original image into a second image frame, and determine the time information corresponding to the second image frame; send to the server The second image frame and the time information corresponding to the second image frame;

The server is configured to receive the first image frame sent by the first IPC and the time information corresponding to the first image frame, and to receive the second image frame sent by the second IPC and the time information corresponding to the second image frame. ; When the frames of the first IPC and the second IPC are determined according to the time information corresponding to the first image frame sent by the first IPC and the time information corresponding to the second image frame sent by the second IPC. When the time is not synchronized, a coding adjustment instruction is sent to the first IPC.
A network camera IPC, characterized in that it is used to perform the synchronization method described in any one of the above claims 7 to 12.
An electronic device, characterized by including:

a memory and a processor, the memory coupled to the processor;

The memory stores program instructions that, when executed by the processor, cause the electronic device to execute the synchronization method executed by any server in claims 1 to 6.
An electronic device, characterized by including:

a memory and a processor, the memory coupled to the processor;

The memory stores program instructions that, when executed by the processor, cause the electronic device to execute the synchronization method executed by any one of the first IPCs in claims 7 to 12.
A chip, characterized in that it includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send the signals to the processor, and the The signal includes computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to perform the synchronization method performed by any server in claims 1 to 6.
A chip, characterized in that it includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send the signals to the processor, and the The signal includes computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to perform the synchronization method performed by any first IPC in claims 7 to 12.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program. When the computer program is run on a computer or a processor, the computer or the processor causes the computer or the processor to execute the instructions as claimed. The synchronization method according to any one of claims 1 to 12.
A computer program product, characterized in that the computer program product includes a software program, and when the software program is executed by a computer or a processor, the steps of the method described in any one of claims 1 to 12 are performed. implement.