WO2022021145A1

WO2022021145A1 - Media augmented reality labeling method, computer device, and storage medium

Info

Publication number: WO2022021145A1
Application number: PCT/CN2020/105530
Authority: WO
Inventors: 洪家明; 程之龙; 李鑫
Original assignee: 海能达通信股份有限公司
Priority date: 2020-07-29
Filing date: 2020-07-29
Publication date: 2022-02-03

Abstract

Disclosed in the present application are a media augmented reality labeling method, a computer device, and a storage medium. The method comprises: associating a label stream with at least one media stream by means of a uniform resource locator and a timestamp, the time when the label stream is displayed being determined by the timestamp, and the uniform resource locator being used for locating a source of the media stream; sending the label stream to a media server, such that the media server sends the media stream and the label stream to a display terminal, and the display terminal displays the media stream and the label stream, the media stream and the label stream sharing a transmission port; or sending the label stream to the display terminal, such that the display terminal displays the media stream and the label stream, transmission ports of the media stream and the label stream being different. By means of the approach, in the present application, no change of the media stream is involved, the label stream has good timeliness and strong independence, and the adaptability of the label stream is enhanced.

Description

Media augmented reality labeling method, computer device and storage medium

【Technical field】

The present application relates to the field of multimedia technology, and in particular, to a media augmented reality labeling method, a computer device, and a computer-readable storage medium.

【Background technique】

Real-time Transport Protocol (RTP for short) is a network transport protocol and a transport layer protocol for multimedia data streams on the Internet. The RTP protocol describes the standard packet format for delivering audio and video over the Internet. Real-time Transport Control Protocol (Real-time Transport Control Protocol or RTP Control Protocol, abbreviated RTCP) is a sister protocol of RTP. Usually RTP and RTP control protocol RTCP are used together. At present, RTP and RTCP are usually used in streaming media systems, and tags are added. In the video, the tags are time-sensitive and independent.

[Content of the invention]

The present application provides a media augmented reality labeling method, computer equipment and storage medium, which do not involve the modification of the media stream, the label stream has good timeliness and strong independence, and enhances the adaptability of the label stream.

In order to solve the above technical problem, a technical solution adopted in the present application is to provide a method for labeling a media augmented reality. The method includes: associating a tag stream with at least one media stream through a uniform resource locator and a timestamp, the time when the tag stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream; sending the tag stream to The media server, so that the media server sends the media stream and the label stream to the display terminal, the display terminal displays the media stream and the label stream, and the media stream and the label stream share the transmission port; or send the label stream to the display terminal, so that the display terminal can The media stream and the label stream are displayed, and the transmission ports of the media stream and the label stream are different.

In order to solve the above technical problem, another technical solution adopted in the present application is to provide a method for labeling a media augmented reality. The method includes: receiving a label stream from a label server and forwarded by the media server, and a media stream sent by the media server, wherein the label stream is associated with at least one media stream through a uniform resource locator and a timestamp, and the time when the label stream is displayed Determined by the timestamp, the Uniform Resource Locator is used to locate the source of the media stream, the tag stream and the media stream share the transmission port; or receive the tag stream from the tag server and the media stream sent by the media server, wherein the tag stream and at least one The media stream is associated with the uniform resource locator and the time stamp. The time when the label stream is displayed is determined by the time stamp. The uniform resource locator is used to locate the source of the media stream. The transmission ports of the label stream and the media stream are different; the display terminal displays the label stream. and media stream content.

In order to solve the above technical problem, another technical solution adopted in the present application is to provide a computer device. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned media augmented reality labeling method is implemented.

To solve the above technical problem, another technical solution adopted in the present application is to provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is executed by the processor to implement the above-mentioned media augmented reality labeling method.

The beneficial effects of the present application are: by associating the tag stream and at least one media stream through a uniform resource locator and a timestamp, the time when the tag stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream, and the The label stream is sent to the media server, so that the media server sends the media stream and the label stream to the display terminal, and the display terminal displays the media stream and the label stream, and the media stream and the label stream share the transmission port. Or send the label stream to the display terminal, so that the display terminal displays the media stream and the label stream, and the transmission ports of the media stream and the label stream are different. By associating the label stream and the media stream, this application does not involve the modification of the media stream, the label stream has good timeliness and strong independence, the label stream can be transmitted together with the media stream, and the transmission port is the same and will not conflict, and the label stream is enhanced. flow adaptability. The label server can also directly transmit the label stream to the display terminal, which increases the transmission efficiency. When the media server encounters problems such as receiving damage, the display terminal can also receive the label stream and display the label stream.

【Description of drawings】

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1A is a first schematic flowchart of a first embodiment of a media augmented reality labeling method provided by the present application;

1B is a first schematic flowchart of a second embodiment of a media augmented reality labeling method provided by the present application;

Fig. 2A is the transmission schematic diagram of media stream and label stream in Fig. 1A provided by this application;

Fig. 2B is the transmission schematic diagram of the media stream and the label stream in Fig. 1B provided by this application;

3A is a second schematic flowchart of the first embodiment of the media augmented reality labeling method provided by the present application;

3B is a second schematic flowchart of the second embodiment of the media augmented reality labeling method provided by the present application;

4A is a schematic diagram of a tag RTP packet message body in the media augmented reality tagging method provided by the present application;

4B is a schematic diagram of a tag RTP packet header structure in the media augmented reality tagging method provided by the present application;

5 is a schematic diagram of an RTCP packet message body in the media augmented reality labeling method provided by the present application;

6A is a first schematic flowchart of a third embodiment of a media augmented reality labeling method provided by the present application;

6B is a second schematic flowchart of the third embodiment of the media augmented reality labeling method provided by the present application;

7A is an example schematic diagram of a first example of inserting a tag RTP packet in the media augmented reality tagging method provided by the present application;

7B is an example schematic diagram of a second example of inserting a label RTP packet in the media augmented reality labeling method provided by the present application;

8A is an example schematic diagram of the first embodiment of inserting a label and controlling the transmission of a message in the media augmented reality labeling method provided by the present application;

8B is a schematic diagram of an example of a second embodiment of inserting a label and controlling message transmission in the media augmented reality labeling method provided by the present application;

9A and 9B are schematic flowcharts of updating labels in the media augmented reality labeling method provided by the present application;

10 is a schematic plan view of performing position calculation on a label in the media augmented reality labeling method provided by the present application;

11A is a first schematic flowchart of a first embodiment of a display terminal in a media augmented reality labeling method provided by the present application;

11B is a first schematic flowchart of a second embodiment of a display terminal in a media augmented reality labeling method provided by the present application;

11C is a second schematic flowchart of the display terminal in the first embodiment of the media augmented reality labeling method provided by the present application;

11D is a second schematic flowchart of a second embodiment of the display terminal in the media augmented reality labeling method provided by the present application;

12 is a schematic structural diagram of a computer device provided by the present application;

FIG. 13 is a schematic structural diagram of a computer-readable storage medium provided by the present application.

【detailed description】

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Labels can be added to the media, such as dynamic advertisement labels of advertising systems, scene-style advertisement labels embedded in live TV broadcasts, etc. After long-term research, the inventor of the present application found that in the prior art, the label is loaded in the SEI (Supplemental Enhancement Information) unit of the video, and the label needs to be decoded to the H264 layer to obtain the label. At the same time, the adaptability is not enough, and the Not all streams have SEI units, and not all streams are video streams. In addition, when the label is added to the video, the label has poor timeliness, poor independence, poor adaptability, difficult label insertion and editing, and poor multi-stream correlation. In order to solve the above technical problems, the present application provides the following embodiments.

This application provides a method for media augmented reality labeling, the application provides a first embodiment, please refer to FIG. 1A and FIG. 2A , the method may include the following steps:

S11: Associate the tag stream with at least one media stream through a uniform resource locator and a timestamp, the time when the tag stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream.

The tag server receives the media stream sent by the media server, associates the tag stream with at least one media stream through a timestamp, the time when the tag stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream. The timestamp can also be an absolute timestamp. Timestamps can determine the specific frame in the media stream where the tag stream should be displayed. The RTP (Real-time Transport Protocol) timestamps of different media streams may grow at different rates and may have independent random offsets. RTP timestamps can be used to synchronize different media streams. NTP (Network Time Protocol, Network Time Protocol) timestamps solve the problem of random offsets in RTP timestamps. Timestamp pairs include RTP timestamps and NTP timestamps. It is used to determine the corresponding relationship between the RTP timestamp and the NTP timestamp, so as to synchronize the media stream and the tag stream. Among them, the Uniform Resource Locator (URL for short) used to locate the source of the media stream is a concise representation of the location and access method of resources that can be obtained from the Internet, and can represent the addresses of standard resources on the Internet. . The URL in this application includes the source IP (Internet Protocol, Internet Protocol) address and port of the media stream, the destination IP address and port, the synchronization source identifier, and the type (payload), and one media stream associated with the tag stream can be determined through the URL. The tag stream is associated with at least one media stream through a uniform resource locator and a timestamp, and is synchronously associated in real time according to the timestamp of the media stream. It does not need to change any encoding and encapsulation methods of the existing media stream, and can be associated with the specific media stream. Frame data loosely coupled associations.

The tag stream defined in this application may be something related to the media stream, or may be unrelated to the media stream. Tags can be text, voice, picture, communication, video, etc. It can also be the structure of intelligent speech recognition, voiceprint recognition, vehicle recognition, face recognition, or video structured information. For example, fixed location equipment, buildings, etc. at a specific location, or moving targets around a fixed location at a specific time point, such as equipment, video, telephone, personnel, vehicles, etc., can also be data, information, intelligence, alarms, etc. The application can also add tags manually, such as circle selection behavior, curve or polygon, etc. The media stream in this application may be a video stream, an audio stream, or a data stream, or the like. The association of tag streams and media streams in this application can be performed across media, for example, it can also be used to track a certain group of targets, associate voice or video at the same time, and associate multiple video devices with similar regions at the same time. Streams are not restricted.

There are multiple tags in the tag stream, the tags include at least one associated tag, and the associated tag includes a uniform resource locator for locating the source of the tag stream.

Referring to Figure 4A, the transmission of the label stream is based on RTP. The label stream can have multiple labels, and each label is encapsulated as a label RTP packet for transmission. The encapsulation format of the label stream is defined as the TLV format (Type-Length-Value, TLV). message), including the tag field (Type/Tag), the length field (Length), and the content field (Value). The tag data is packaged and constructed into a series of TLV structural entities.

There are multiple label RTP packets in the label stream, and each label RTP packet may include at least one associated label, and may also include multiple associated data information labels, and the multiple associated data information labels include at least the relevant data information of the associated label. One of the content label, location label, or control label.

The label RTP packet may include at least one associated label, and the associated label contains a uniform resource locator for locating the source of the media stream, and the URL contains the mode (or protocol), server name (or IP address), path and file name. Each file has a unique URL, which contains information indicating the location of the file and what the browser should do with it, locating the source of the media stream, and identifying the file by its URL. Optionally, the associated tag may include a timestamp and associated URL. The time stamp and URL can be associated with the media stream, that is, the tag stream can be associated with the media stream through the associated tag. There is a timestamp in the header structure (RTP header) of the RTP tag packet, and the timestamp can be associated with the media stream. The timestamp in the associated tag can be ignored, and the timestamp in the RTP header in the RTP tag packet can be taken. The associated URL may express the associated transmission port, for example, the URL may be file://e:/abc.rtp, or may be a port, a web page URL, or the like. The URL in this application includes the source IP and port, destination IP and port, synchronization source identifier, and type (payload) of the media stream, and one media stream associated with the tag stream can be determined through the URL. Because of the correspondence between source IP and port, and destination IP and port, URL can express the source information of RTP packets, which can be the source information of media RTP packets in media streams, or the source information of label RTP packets in label streams.

The tag RTP packet includes a plurality of associated data information tags, and the plurality of associated data information tags include at least one of a content tag, a location tag, or a control tag of the related data information of the associated tag.

Please refer to FIG. 4A , in addition to at least one associated label, the label RTP packet may also include multiple associated data information labels, and the plurality of associated data information labels include at least a content label, a location label or a control label of the relevant data information of the associated label. One of the associated data information tags may also include content tags, location tags, and control tags of related data information of the associated tags, and may also include other related data tags. The multiple associated data information tags can be tags related to the associated tags, or tags unrelated to the associated tags, and can be content information, location information, environmental information, etc. Face recognition and license plate behavior recognition tags, voiceprint recognition tags and other identification tags, geographically related equipment tags (which can be nearby people or equipment), environmental information related tags, business related tags, etc. The multiple associated data information tags in the present application may be set according to specific conditions, and the present application does not limit the multiple associated data information tags.

The content tag may be tag content information, and the content tag may include content type, text description, and structured data, where the structured data is information data of the tag. The position tag can represent the position information of the tag, and the position tag can include absolute position and relative position, wherein the absolute position can be the position information obtained by GPS (Global Positioning System, global positioning system); the relative position can represent the horizontal angle of the tag or The vertical angle information is the position information relative to the camera or the sphere, and the camera can be a camera or the like. The control tag may include control information on the tag, such as call control and interface invocation. For example, when the tag is a mobile device, call control can be implemented on the tag, and the calling interface can implement call control.

S12: Send the label stream to the media server, so that the media server sends the media stream and the label stream to the display terminal, and the display terminal displays the media stream and the label stream, and the media stream and the label stream share the transmission port.

The tag server sends the associated tag stream to the media server, so that the media server can send the media stream and tag stream to the display terminal for display. When the media server sends the media stream and tag stream to the display terminal, the transmission path is the same, and the media stream and The label stream is the same RTP stream, the media stream and the label stream share the transmission port and can share the communication channel. This transmission mode can be referred to as the accompanying transmission mode. The transmission of the media stream and the label stream is based on RTP, and the label stream can have a separate communication channel or can share a communication channel with the media stream. In this application, the label stream is associated with the media stream, and does not involve the modification of the media stream. The label stream has good timeliness and strong independence. The label stream can be transmitted together with the media stream. The same transmission port will not conflict, which enhances the label stream. adaptability. The tag stream and the media stream may be real-time streams, and when sent to a display terminal for display, a media augmented reality (Augmented Reality, AR for short) tag can be implemented.

This application provides a method for media augmented reality labeling, and this application also provides a second embodiment, please refer to FIG. 1B and FIG. 2B , the method includes the following steps:

S21: Associate the tag stream with at least one media stream through a uniform resource locator and a timestamp, the time when the tag stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream. For details of the process in this step, reference may be made to the implementation process in the method for media augmented reality labeling in the foregoing embodiment, which will not be repeated here.

S22: Send the label stream to the display terminal, so that the display terminal displays the media stream and the label stream, and the transmission ports of the media stream and the label stream are different.

The tag server sends the tag stream associated with the media stream to the display terminal, the media server sends the media stream to the display terminal, and the display terminal displays the media stream and the tag stream according to the associated timestamp, wherein the media stream and the tag stream are different The transmission ports of the RTP stream, the media stream and the label stream are different, and their transmission paths are also different. The transmission of media stream and tag stream is based on RTP. The media stream can have a separate communication channel, and the tag stream has a separate communication channel and is transmitted separately from the media stream. This transmission mode can be called an independent stream transmission mode. This application associates the label stream with the media stream, does not involve the modification of the media stream, the label stream has good timeliness and strong independence, and the label stream and the media stream are transmitted separately, which prevents the conflict between the media stream and the label stream transmission, and also enhances the label flow. The adaptability of the stream directly transmits the label stream to the display terminal, which increases the transmission efficiency. When the media server has problems such as receiving damage, the display terminal can also receive the label stream and display the label stream.

This application provides a method for media augmented reality labeling, and this application also provides another implementation manner. Please refer to FIG. 3A and FIG. 3B . After the above step S12 or step S22, the following steps are further included:

S13: Receive the control information sent from the display terminal, and transmit the control information in the reverse direction along the transmission path of the label stream, and the control information includes a control command for controlling the operation of the label stream.

The display terminal may send control information to the tag server, where the control information may be control information about tags, the tags may be tags stored in the tag server, or may be control commands to control tag stream operations. The display terminal may also send a control message to the media server, and the media server forwards the control message to the label server. The label server receives the control message sent from the display terminal.

Please refer to Figure 5. The transmission of control messages is based on RTCP (Real-time Transport Control Protocol or RTP Control Protocol, real-time transmission control protocol). There can be multiple control tags in the control information, and the control tags contain control commands and uniform resource locators. , the uniform resource locator can locate the source of the media stream or the label stream, and the control command can be the control of the source label stream or media stream indicated by the uniform resource locator, or the operation of the label stream. The control information is transmitted in the reverse direction along the transmission path of the label stream, that is, it is transmitted to the source direction of the label stream or the media stream.

8A and 8B, the label server can be a plurality of node label servers, and the current node label server determines whether the control information contains the control label of the current node label server; wherein, the control label includes the control command for the operation of the control label . If the control information includes the control label of the current node label server, the control label is taken out. The current node label server takes out the control label and can execute the control command on the label. If the control information does not include the control label of the current node server, the control information is forwarded to the node label server preceding the current node label server, and the back-to-source control is continued to the source of the uniform resource locator.

The label server stores at least one of a synchronization source identifier (Synchronization Source, SSRC for short) and a uniform resource locator of the label stream. The current node label server stores at least one of the synchronization source identifier and the uniform resource locator of the label stream. When the current node label server receives the control information, it can judge whether the control message contains the control label of the current node label server according to the synchronization source identifier or the uniform resource locator, or it can judge the control information according to the synchronization source identifier or the uniform resource locator. Whether the label is the control label of the current node, if the control information does not contain the control label of the current node server, the current node can forward the control information according to the synchronization source identifier or the uniform resource locator. When the synchronization source identifier and the transmission port corresponding to the synchronization source identifier are stored in the label server, the transmission port can be an IP address and a port, and the current node can control the message according to the synchronization source identifier and the corresponding transmission port. Forward.

In this embodiment, the present application provides a method for a media augmented reality label. The method may further include inserting a label RTP packet, updating a label RTP packet, etc. The insertion of the label RTP packet may be before sending the original label stream, or may Insert and so on after sending the original tag stream. This application provides a method for media augmented reality labeling, and this application also provides a third embodiment, please refer to FIG. 6A and FIG. 6B , before the above step S12 or step S22, the following steps are further included:

Referring to FIG. 6A, before the above step S12, the following steps may also be included:

S110: Insert a label RTP packet into the RTP stream, and combine the label RTP packet with the RTP stream. After the label RTP packet is inserted, its sequence number adopts the sequence number of the original RTP packet at the insertion position, and the RTP packet at the insertion position adopts the sequence number of the original RTP packet at the insertion position. and the sequence numbers of all original RTP packets that follow plus the number of RTP packets inserted with corresponding labels.

Referring to FIG. 7A , the tag server can insert the tag stream into the media stream, and the tag stream and the media stream can be transmitted in the same RTP stream. Multiple tagged RTP packets can be inserted, or one tagged RTP packet can be inserted and merged with the RTP stream. After the label RTP packet is inserted into the RTP stream, its sequence number adopts the sequence number of the original RTP packet at the insertion position, the sequence number of the RTP packet at the insertion position and all the original RTP packets behind it plus the number of inserted corresponding label RTP packets . For example, in the accompanying stream transmission mode, insert a label RTP packet into the original media RTP stream, and insert a label RTP packet into the media RTP packet with sequence number 3. At this time, the sequence number of the inserted label RTP packet is 3, before the insertion position. The serial number of the media RTP packet does not change, and the serial number of the media RTP packet after the inserted position is correspondingly increased by one. In this application, the sequence number can be rearranged after inserting the label RTP packet into the RTP stream, and the sequence number can be used to detect whether there is any packet loss, and the display terminal can detect the label RTP packet through the sequence number after receiving the label stream or media stream. or loss of media RTP packets.

Please refer to FIG. 6B , before the above step S22, the following steps may also be included:

S120: Insert a new label RTP packet into the label stream, and combine the new label RTP packet with the label stream, wherein, after the new label RTP packet is inserted, its sequence number adopts the sequence number of the original label RTP packet at the insertion position, and the insertion position The sequence number of the original label RTP packet at and all the original label RTP packets following it plus the number of inserted corresponding new label RTP packets.

A new label stream can be added to the label stream, and the label stream and the media stream are transmitted as different RTP streams. A new label RTP packet can be added to the label stream, and the sequence number (Sequence number) can be rearranged after inserting the new label RTP packet. A new label RTP packet can be inserted into the label stream. After the new label RTP packet is inserted, the sequence number of the new label RTP packet adopts the sequence number of the original label RTP packet at the insertion position, and the sequence number of the label RTP packet before the insertion position remains unchanged. , the sequence number of the label RTP packet after the insertion position is correspondingly increased by one. Referring to FIG. 7B, for example, in the original label stream, a new label RTP packet is added at the position of the sequence number 3, after adding the new label RTP packet, the sequence number of the new label RTP packet is 3, and the label RTP packet after the insertion position Correspondingly add 1, the sequence number of the label RTP packet with the original sequence number 3 is 4, and the sequence number of the label RTP packet with the original sequence number 4 is 5.

It is also possible to insert multiple new label RTP packets at the same time. The sequence number of the label RTP packet before the insertion position of the multiple label RTP packets does not change, and the sequence number of the label RTP packet after the insertion position of the multiple label RTP packets is added accordingly. The number of multiple RTP packets inserted into the new label. The sequence numbers of the plurality of new label RTP packets are randomly assigned, and the randomly assigned sequence numbers are in the range from the sequence number of the original label RTP packet at the insertion position to the sequence number of the original label RTP packet plus the corresponding insertion quantity. For example, two new label RTP packets are added at the same time at the position where the sequence number is 3 in the original label stream, the inserted new label RTP packets are randomly allocated from the sequence number 3, and the sequence numbers of the newly inserted label RTP packets can be 3, 4. Add 2 to the serial number of the original label RTP packet after the insertion position. In this application, the sequence number can be rearranged by inserting the label RTP packet into the label stream, and the sequence number can be used to detect whether there is a packet loss situation. It shows that the terminal can use the sequence number to detect the loss of the label RTP packet after receiving the label stream. .

In the above step S110 or step S120, for the insertion of the label RTP packet of the same channel of media stream, usually not one device, one node, or even in different regions will process the label RTP packet insertion of the same channel of media stream. In this application, the insertion of the label stream may be distributed in a distributed insertion manner. Multiple label servers respectively insert label RTP packets, serially insert label RTP packets, parallel transmission between multiple servers, and parallel transmission of label streams between multiple servers. , media stream or control message, etc., when inserting the label RTP packet, the label server stores the source information, type, etc. of the inserted label, and records the source path of the inserted label RTP packet. For example, the tag server may store the synchronization source identifier or uniform resource locator of the inserted tag. When the tag RTP is inserted, it can also be associated with the media stream through the uniform resource locator and timestamp. The tag server can also store the synchronization source identifier or uniform resource locator corresponding to the media stream inserted into the tag stream or the tag RTP packet for association. .

The label server may have multiple label servers, and the multiple label servers may be label servers of the same type of label, or may be label servers of different types, and multiple label servers may be inserted into the label stream respectively. Please refer to FIG. 8A and FIG. 8B, for example, tag server 1 can insert tags such as behavior, scene recognition, alarm, etc., tag server 2 can insert moving target associated tags, etc., tag server 3 can insert static location or device-associated tags, etc., multiple tags Parallel transmission between servers. The tag stream can be sent to a media server or display terminal.

The label server may have multiple label servers, and may insert label streams in a distributed manner among the multiple label servers, and may insert label streams in parallel or serially. Please refer to FIG. 8A and FIG. 8B , the tag servers are transmitted in parallel, and tags can be inserted serially in multiple tag servers transmitted in parallel, and tag server 4 can insert the tag stream into the original parallel transmission tag stream, for example, at tag server 2 When inserting a label stream, the label server 2 combines the inserted label stream with the original label stream and transmits it to the label server 3 of the next node, and finally sends it to the media server or display terminal.

When a new label stream is inserted, the bandwidth is reserved in advance according to the transmission rate of the label stream before the new label stream is inserted, and the label stream is transmitted at a predetermined rate, and the transmission amount of the label stream is controlled by the flow. The label stream is transmitted at a predetermined rate, and the width is reserved during the transmission. After inserting the label stream, when encapsulating and packaging, the width will be reserved according to the transmission rate of the original label stream. The transmission of the label stream is controlled by the flow, and smoothing measures are adopted to transmit the label stream at a predetermined rate, which can prevent the instantaneous traffic of the label stream. If it is too large, it can also prevent packet loss in the transmission traffic of the excessively large label flow.

In other embodiments, the new label stream can be added to the original label stream, please refer to FIG. 4B , when the new label stream and the original label stream are encapsulated and packaged, the header structure extension mode of RTP can be used to add the new label stream to the original label stream. The label stream is inserted into the original label stream, and some data can be added according to the characteristics of the code stream. The padding data can be considered by the unrecognized processing process as the data of the padding packet of the code stream. The RTP packet can be composed of two parts: the header structure (RTP head) and the message body (RTP body). The padding flag can be marked in the P bit of the RTP header structure, and the extension flag can be marked in the X bit. For example, if X=1 is set, an extended header can be followed by the RTP header. The format of a header extension can be referred to in FIG. 4B. The header extension can be an inserted new tag stream, and the new tag stream can be inserted into the original tag stream through the header structure extension mode.

Referring to FIG. 6A and FIG. 6B , after step S110 or step S120 above, step S130 may be further included, as shown below.

S130: Send the label stream to the media server or the display terminal according to the posture change of the network camera or the state change of the label stream.

The label server stores multiple labels, and can send labels in the field of view of the network camera to the media server according to changes of the network camera, and send the labels in the field of view of the network camera to the display terminal. It is also possible to send a stream of labels with state changes to the display terminal. Please refer to FIG. 9A , FIG. 9B and FIG. 10 , step S130 may include the following steps:

S131: Screen the label with the position of the network camera as the center, and the maximum field of view of the network camera as the radius.

The location of the IP Camera (IPC) can be obtained through GPS (Global Positioning System, Global Positioning System). The IP camera has a predetermined field of view. The GPS position of the network camera is the center of the circle, and the maximum field of view of the network camera is Radius selects the label, the label is the label stored in the label server. The associated data information tag in the tag can have a position tag, and the position tag has an absolute position and a relative position, where the absolute position can be the position information obtained by GPS, and the relative position can represent the horizontal angle or vertical angle information of the tag. Relative position information relative to the network camera. The tags are filtered by the GPS location information of the location tags.

For example, referring to Figure 10, in any area A there may be an IP camera IPC, and a label C. The absolute position of the IPC and the label C can be obtained through GPS, and the relative position of the label C relative to the IPC can be obtained, and the horizontal angle or vertical angle information of the label C relative to the IPC can be obtained. The maximum field of view radius of the IPC can be R. Taking the maximum field of view R as the radius and the GPS position of the IPC as the center of the circle, a circular area B can be obtained. Due to the change of the IPC state, the current field of view of the IPC can be B1. In the area A, the label C is filtered according to the circular area B, the label C in the area B can be selected, and the label C outside the area B is not selected.

S132: Determine whether the posture of the network camera changes.

The posture change of the network camera may be a change in the view direction of the network camera or a change in the orientation of the network camera. The attitude change of the network camera may be the angle change of the network camera, and the angle may be any angle change in space, for example, the angle change in three-dimensional space. The posture change of the network camera may be the change of the focal length of the network camera, and the like.

If the posture of the network camera changes, step S133 is performed, and if the posture of the network camera does not change, step S134 is performed.

S133: Calculate the positions of all labels within the first predetermined time, and send the labels within the field of view after the posture of the network camera is changed.

If the posture of the network camera changes, and if any of the direction, angle, and focal length of the IPC changes, the positions of all tags are calculated within the first predetermined time. All tags can be all tags screened by IPC, or are all the tags sent by the tag server. For example, if the attitude of the IPC changes, the position calculation is performed for the tags screened by the IPC within 3 seconds. When the direction, angle or focal length of the IPC changes, the position of the filtered tags can be calculated to obtain the tags in the current field of view after the IPC changes, and the tags in the current field of view can be sent to the display terminal for display, and only the tags in the field of view can be sent. Labels are beneficial to reduce the amount of data transmitted by labels and increase transmission efficiency.

S134: Perform position calculation of all tags within a second predetermined time, and perform label rotation within a second predetermined time, where the second predetermined time is greater than the first predetermined time.

If the posture of the network camera does not change, and if there is no change in the direction, angle, and focal length of the IPC, the position calculation of all labels is performed within the second predetermined time, and the label rotation is performed within the second predetermined time. Wherein, the second predetermined time is greater than the first predetermined time. All labels can be all labels filtered by IPC, or all labels sent by the label server. The carousel can display all labels filtered by IPC in sequence, and can be clockwise along the current view of IPC. It can be displayed sequentially in a counterclockwise direction, or the tags in the current field of view of the IPC can be displayed in sequence according to the time stamp, or they can be displayed in sequence according to the serial number of the tags, and the tag server will display the tags that need to be displayed. terminal to display. The carousel may also be to send all the filtered tags to the display terminal for display according to a predetermined time, or to sequentially send all the filtered tags to the tag server. It is also possible to send all the filtered tags sequentially and cyclically within a predetermined time. For example, if the posture of the IPC does not change, the position of the tags screened by the IPC will be calculated within 3 minutes, and all the tags in the current field of view can be obtained through the position calculation. The sent tags are displayed in sequence within minutes, which is beneficial to reduce the data volume of the transmitted tags and increase the transmission efficiency.

The above step S131 takes the position of the network camera as the center of the circle, and takes the maximum field of view of the network camera as the radius to filter the labels, and then the following steps may be included:

S142: Determine whether the state of any label has changed in all label streams.

The change of the label stream may be the addition of the label to the label stream, the deletion of the label, etc., or the change of the shape of the label stream, the change of the position, and the like. It can also be the state change of the tags in the tag stream. This application does not limit the changes of labels. If the state of the tag changes, step S143 is executed, and if the state of no tag in all the tags changes, then step S144 is executed. Step S142 and step S132 may be performed simultaneously, or may be determined separately. The present application does not limit the sequence of step S142 and step S132, and the simultaneous or separate execution of step S142 or step S132.

S143: Perform position calculation on the changed label.

If the status of the tag changes, the position of the tag with the status change can be calculated, and the tag with the position change or status change can be sent to the media server to the display terminal, and the tag with the position change or status change can also be sent to the display terminal. . All tags can be all tags screened by IPC. For example, if the status of the tag in the current field of view of IPC changes, the position of the changed tag is calculated. If it is not in the current field of view, the changed tag is not sent. If the position of the tag whose state changes is in the current field of view, the change is sent. The label is displayed on the display terminal. Sending only the tags whose state changes in the current view of the IPC is beneficial to reduce the data amount of the transmitted tags and increase the transmission efficiency.

The user can subscribe and listen to the specified tag type on the display terminal according to their own preferences. After the user specifies the tag type on the display terminal, the display terminal can send the control message of the specified type of tag to the media and forward it to the tag server. Alternatively, the display terminal can send the control message of the specified type of label to the label server, and the label server can send the specified type of label to the display terminal for display according to the received control message according to the subscribed label type of the display terminal.

When the above label server filters the labels to be sent, the label server packages the storable labels into label RTP packets. If it is in the accompanying stream transmission mode, the label RTP packet can be inserted into the original media RTP stream and transmitted to the display terminal for display. The inserted label RTP packet and the original RTP stream are the same RTP stream, and the label server will insert the media stream of the label RTP stream. Send to the media server, and the media server can send the received media stream inserted into the tag RTP stream to the display terminal for display. If it is in the independent stream transmission mode, the label server can compose the label RTP packets into independent label RTP streams and send them to the display terminal for display. The media RTP stream and the label RTP stream are different RTP streams.

S144: Step S132 is executed.

If the state of no label in all labels changes, the above step S132 is performed, and specifically, the step of judging whether the posture of the network camera changes is performed. For details, reference may be made to the foregoing step S132, which will not be repeated here.

A method for media augmented reality labeling provided by the present application also provides a fifth embodiment, please refer to FIG. 11A , in this embodiment, the method may include the following steps:

S510: The display terminal receives the tag stream from the tag server and forwarded by the media server, and the media stream sent by the media server. Determined by the timestamp, the Uniform Resource Locator is used to locate the source of the media stream, the tag stream and the media stream share the transport port.

S511: The display terminal displays the content of the tag stream and the media stream.

In this embodiment, for details of the processes in the steps, reference may be made to the processes in the foregoing embodiments, and details are not described herein again.

For this embodiment, the present application also provides another implementation manner, please refer to FIG. 11B , the method includes the following steps:

S520: The display terminal receives the tag stream from the tag server and the media stream sent by the media server, wherein the tag stream is associated with at least one media stream through a uniform resource locator and a timestamp, and the time when the tag stream is displayed is determined by the timestamp, The uniform resource locator is used to locate the source of the media stream, and the transmission port of the label stream and the media stream is different.

S521: The display terminal displays the content of the tag stream and the media stream.

In this embodiment, for details of the process in the steps, reference may be made to the implementation process of displaying a terminal in the method for media augmented reality label in the foregoing embodiment, and details are not described herein again.

In other embodiments, referring to FIGS. 11C and 11D , after the above step S511 or step S521, the method may further include the following steps:

S512: Display the control information sent by the terminal to the tag server, the control information is transmitted in the reverse direction along the transmission path of the tag stream, and the control information includes a control command for controlling the operation of the tag.

In this embodiment, for details of the process in the steps, reference may be made to the process of displaying the terminal in the foregoing embodiment, which will not be repeated here.

Referring to FIG. 12 , the present application provides a computer device. In this embodiment, the computer device 800 includes a processor 810 , a memory 820 , and a computer program stored in the memory 820 and executable on the processor 810 . When the processor 810 executes the computer program, a method for media augmented reality labeling in the above embodiments is implemented.

In this embodiment, the processor 810 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 810 may be an integrated circuit chip with signal processing capability. Processor 810 may also be a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components . A general purpose processor may be a microprocessor or the processor 810 may be any conventional processor or the like.

For the method of the above-mentioned embodiment, it can exist in the form of a computer program, so this application proposes a computer-readable storage medium, please refer to FIG. 13 , in this embodiment, the computer-readable storage medium 900 stores a computer program 910, which Executed by a processor to implement a method for media augmented reality labeling in the above embodiments.

The computer-readable storage medium 900 in this embodiment may be a U disk, a removable hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc., which can store program instructions or a server storing the program instructions, the server can send the stored program instructions to other devices for running, or can also run the stored program instructions by itself.

In the several embodiments provided in this application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the apparatus implementations described above are only illustrative, for example, the division of modules or units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some ports, devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this implementation manner.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general-purpose computing device, and they can be centralized on a single computing device, or distributed in a network composed of multiple computing devices Alternatively, they can be implemented with program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or they can be integrated into The multiple modules or steps are fabricated into a single integrated circuit module. As such, the present application is not limited to any particular combination of hardware and software.

The above description is only an embodiment of the present application, and is not intended to limit the scope of the patent of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present application, or directly or indirectly applied to other related technologies Fields are similarly included within the scope of patent protection of this application.

Claims

A method for media augmented reality labeling, characterized in that the method comprises:

associating the tag stream with at least one media stream through a uniform resource locator and a timestamp, the time when the tag stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream;

Send the label stream to the media server, so that the media server sends the media stream and the label stream to the display terminal, and the display terminal displays the media stream and the label stream, the media stream and the label stream. the tag stream shares the transport port; or

The label stream is sent to the display terminal, so that the display terminal displays the media stream and the label stream, and the transmission ports of the media stream and the label stream are different.
The method of claim 1, wherein:

The transmission of the tag stream is based on a real-time transmission protocol, and there are multiple tag real-time transmission protocol packets in the tag stream; each of the tag real-time transmission protocol packets includes multiple associated data information tags, the multiple associated data The information tag includes at least one of a content tag, a location tag or a control tag of the relevant data information of the associated tag.
The method according to claim 1, wherein the method further comprises:

Control information sent from the display terminal is received, the control information is transmitted in the reverse direction along the transmission path of the label stream, and the control information includes a control command for controlling the operation of the label stream.
The method of claim 3, wherein:

The label server includes a plurality of node label servers, and the current node label server determines whether the control information contains the control label of the current node label server; wherein, the control label includes the control of the operation of controlling the label flow Order;

If the control information contains the control label of the current node server, then take out the control label; if the control information does not contain the control label of the current node server, send the current node label server to the uploader. A node label server forwards the control information.
The method of claim 4, wherein:

The current node label server stores at least one of the synchronization source identifier of the label stream and the uniform resource locator;

The current node label server receives the control information, and forwards the control information according to the synchronization source identifier or the uniform resource locator.
The method of claim 1, wherein:

The sending of the label stream to the media server or the display terminal includes:

Insert a new-label RTTP packet into the label stream, and combine the new-label RTTP packet with the label stream, wherein, after the new-label RTTP packet is inserted, its serial number adopts the The serial number of the original tag real-time transmission protocol package at the insertion position, the serial number of the original tag real-time transmission protocol package at the insertion position and all original tag real-time transmission protocol packets behind it plus the corresponding insertion of the new tag real-time transmission protocol package the number of packages.
The method according to claim 1, wherein the method further comprises:

The sending the label stream to the media server or the display terminal further includes:

According to the posture change of the network camera or the state change of the label stream, sending the label stream to the media server or the display terminal includes:

Taking the position of the network camera as the center of the circle, and taking the maximum field of view of the network camera as the radius to filter labels;

Determine whether the posture of the network camera has changed;

If the posture of the network camera changes, calculate the positions of all the labels within the first predetermined time, and send the labels in the field of view after the posture of the network camera changes;

If the posture of the network camera does not change, the position calculation of all the tags is performed within a second predetermined time, and the rotation of the tags is performed within the second predetermined time. greater than the first predetermined time.
The method of claim 7, wherein:

Taking the position of the network camera as the center of the circle, and taking the maximum field of view of the network camera as the radius to filter the labels, and then further including:

Determine whether the state of the label has changed in all the labels;

If the state of the tag in all the tags is changed, the position calculation is performed on the changed tag.
A method for media augmented reality labeling, characterized in that the method comprises:

Receive a label stream from a label server and forwarded by the media server, and a media stream sent by the media server, wherein the label stream is associated with at least one of the media streams through a uniform resource locator and a timestamp, the label stream The time at which the stream is displayed is determined by the timestamp, and the uniform resource locator is used to locate the source of the media stream, and the tag stream and the media stream share a transmission port; or

receiving a tag stream from the tag server and a media stream sent by the media server, wherein the tag stream is associated with at least one of the media streams through the uniform resource locator and the timestamp, the The displayed time of the label stream is determined by the timestamp, the uniform resource locator is used to locate the source of the media stream, and the transmission ports of the label stream and the media stream are different;

The display terminal displays the content of the tag stream and the media stream.
A computer device, characterized by comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the method according to claim 1- The method of any one of 9.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1-9.