WO2023051739A1

WO2023051739A1 - Image display method and device, storage medium, and electronic device

Info

Publication number: WO2023051739A1
Application number: PCT/CN2022/122882
Authority: WO
Inventors: 王宙
Original assignee: 中兴通讯股份有限公司
Priority date: 2021-09-30
Filing date: 2022-09-29
Publication date: 2023-04-06
Also published as: CN115914716A

Abstract

The present disclosure provides an image display method and device, a storage medium, and an electronic device. The method comprises: sending a collected target image to a target server; obtaining an identifier and target position of a first target object sent by the target server after processing the target image, the first target object being located in the target image; sending the identifier of the first target object to a content management platform, and obtaining target content sent by the content management platform according to the identifier; and superimposing the target content on the target image according to the target position of the first target object to obtain an augmented reality (AR) image. The described technical solution solves problems such as a traditional AR device independently achieving an augmented reality function, which results in the hardware requirements of the AR device being too high.

Description

Image display method and device, storage medium and electronic device

This disclosure claims the priority of the Chinese patent application with the application number 202111166756.X and the title of the invention "image display method and device, storage medium and electronic device" filed with the China Patent Office on September 30, 2021, the entire contents of which are incorporated by reference incorporated in this disclosure.

technical field

The present disclosure relates to the communication field, and in particular, to an image display method and device, a storage medium, and an electronic device.

Background technique

With the rapid development of society, AR/VR equipment has been further developed, and more and more people pay attention to and use it. However, the current AR applications usually use the application program of the AR equipment to extract the video collected by the camera. Drawing virtual objects after frame, recognition, and calculation will lead to excessive calculation pressure on AR devices, which makes the hardware requirements of AR devices higher, and because AR devices are used alone, the screen cannot be sent to the remote end for sharing and collaboration.

In the related technologies, the traditional augmented reality AR device implements the augmented reality function independently, which makes the hardware requirements of the AR device too high, and no effective solution has been proposed yet.

Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology.

Contents of the invention

Embodiments of the present disclosure provide an image display method and device, a storage medium, and an electronic device, so as to at least solve the problem that traditional augmented reality AR devices implement augmented reality functions independently, which makes the hardware requirements of the AR devices too high.

According to an aspect of an embodiment of the present disclosure, an image display method is provided, including: sending the collected target image to a target server; acquiring the identification of the first target object sent by the target server after processing the target image and a target location, wherein the first target object is located in the target image; sending the identification of the first target object to the content management platform, and obtaining the target content sent by the content management platform according to the identification; According to the target position of the first target object, the target content is superimposed on the target image to obtain an augmented reality AR image.

According to another aspect of the embodiments of the present disclosure, there is also provided an image display method, including: acquiring the target image sent by the target client; processing the target image to obtain the identification and target position of the first target object, wherein, The first target object is located in the target image; sending the identification of the first target object and the target position to the target client to instruct the target client to The logo and target location are identified in the augmented reality AR image.

According to another aspect of the embodiments of the present disclosure, there is also provided an image display method, including: acquiring the identifier of the first target object sent by the target client; determining the target content according to the identifier of the first target object; The content is sent to the target client to instruct the target client to determine an augmented reality AR image according to the target content.

According to another aspect of the embodiments of the present disclosure, there is also provided an image display system, including: a target server configured to acquire the target image collected by the target client, and process the target image to obtain the first The identification and target position of the target object are sent to the target client, wherein the first target object is located in the target image; the content management platform is configured to obtain the first target object sent by the target client identification, and send the target content determined according to the identification of the first target object to the target client; the target client is configured to send the target content according to the target location of the first target object superimposed on the target image to obtain an augmented reality AR image.

According to another aspect of the embodiments of the present disclosure, there is also provided an image display device, including: a first sending module configured to send the collected target image to a target server; a first obtaining module configured to obtain the The target server processes the target image and sends the identification and target position of the first target object, wherein the first target object is located in the target image; the second acquisition module is configured to take the first target object The identification of the target object is sent to the content management platform, and the target content sent by the content management platform according to the identification is obtained; the first display module is configured to display the target content according to the target position of the first target object superimposed on the target image to obtain an augmented reality AR image.

According to another aspect of the embodiments of the present disclosure, there is also provided an image display device, including: a third acquisition module configured to acquire the target image sent by the target client; a processing module configured to process the target image , to obtain the identification and target position of the first target object, wherein the first target object is located in the target image; the second display module is configured to send the identification of the first target object and the target position to the target client, so as to instruct the target client to determine an augmented reality AR image according to the identifier of the first target object and the target position.

According to another aspect of the embodiments of the present disclosure, there is also provided an image display device, which includes: a fourth acquisition module configured to acquire the identifier of the first target object sent by the target client; a determination module configured to The identification of the first target object determines the target content; the third display module is configured to send the target content to the target client to instruct the target client to determine the augmented reality AR image according to the target content .

According to yet another aspect of the embodiments of the present disclosure, there is also provided a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the above-mentioned image display method when running .

According to yet another aspect of the embodiments of the present disclosure, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor executes the above image through the computer program Show method.

Through this disclosure, the target client on the AR device sends the target image collected by the AR device to the target server, and obtains the identification and target position of the first target object sent by the target server after processing the target image, so that the target client The terminal can obtain the target content sent by the content management platform according to the identifier of the first target object, and superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the target server and the content management platform, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

Description of drawings

The drawings described here are used to provide a further understanding of the present disclosure, and constitute a part of the present application. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure, and do not constitute improper limitations to the present disclosure. In the attached picture:

FIG. 1 is a block diagram of the hardware structure of a computer terminal of an image display method according to an embodiment of the present disclosure;

FIG. 2 is a flowchart (1) of an image display method according to an embodiment of the present disclosure;

FIG. 3 is a flowchart (2) of an image display method according to an embodiment of the present disclosure;

FIG. 4 is a flowchart (3) of an image display method according to an embodiment of the present disclosure;

5 is a frame diagram of an image display method according to an embodiment of the present disclosure;

FIG. 6 is a timing diagram of an image display method according to an embodiment of the disclosure;

FIG. 7 is a flowchart of content overlay according to an embodiment of the disclosure;

Fig. 8 is a schematic diagram of content superposition according to an embodiment of the present disclosure;

FIG. 9 is an overall flowchart of an image display method according to an embodiment of the present disclosure;

FIG. 10 is a block diagram of an image display system according to an embodiment of the present disclosure;

FIG. 11 is a frame diagram (1) of an image display device according to an embodiment of the present disclosure;

FIG. 12 is a frame diagram (2) of an image display device according to an embodiment of the present disclosure;

FIG. 13 is a frame diagram (3) of an image display device according to an embodiment of the present disclosure.

Detailed ways

In order to enable those skilled in the art to better understand the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is an embodiment of a part of the present disclosure, but not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present disclosure.

It should be noted that the terms "first" and "second" in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The method embodiments provided in the embodiments of the present application may be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, FIG. 1 is a block diagram of a hardware structure of a computer terminal according to an image display method according to an embodiment of the present disclosure. As shown in Figure 1, the computer terminal can include one or more (only one is shown in Figure 1) processor 102 (processor 102 can include but not limited to microprocessor (Microprocessor Unit, MPU for short) or programmable logic A device (Programmable logic device, PLD for short)) and a memory 104 for storing data. In an exemplary embodiment, the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. Those skilled in the art can understand that the structure shown in FIG. 1 is only for illustration, and it does not limit the structure of the above computer terminal. For example, the computer terminal may also include more or less components than those shown in FIG. 1 , or have a different configuration with functions equivalent to those shown in FIG. 1 or more functions than those shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the image display method in the embodiments of the present disclosure, and the processor 102 executes various functions by running the computer programs stored in the memory 104. A functional application and data processing, that is, to realize the above-mentioned method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory that is remotely located relative to the processor 102, and these remote memories may be connected to a computer terminal through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 106 is used to receive or transmit data via a network. The specific example of the above-mentioned network may include a wireless network provided by the communication provider of the computer terminal. In one example, the transmission device 106 includes a network interface controller (NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet in a wireless manner.

In this embodiment, an image display method is provided, which is applied to a target client. FIG. 2 is a flow chart (1) of an image display method according to an embodiment of the present disclosure. The process includes the following steps:

Step S202, sending the collected target image to the target server;

Step S204, acquiring the identification and target position of the first target object sent by the target server after processing the target image, wherein the first target object is located in the target image;

It should be noted that the first target object is an object in the target image or a two-dimensional code.

Step S206, sending the identifier of the first target object to the content management platform, and obtaining the target content sent by the content management platform according to the identifier;

Step S208, according to the target position of the first target object, superimpose the target content on the target image to obtain an augmented reality (Augmented Reality, AR) image.

It should be noted that, after obtaining the AR image, the target client may display the AR image through the AR device.

It should be noted that AR devices include: AR glasses, AR helmets, etc.

It should be noted that the target client in the embodiment of the present application includes: a video conference client; the target server includes: a video conference server.

Through the above steps, the target client on the AR device sends the target image collected by the AR device to the target server, and obtains the identification and target position of the first target object sent by the target server after processing the target image, so that the target client The terminal can obtain the target content sent by the content management platform according to the identifier of the first target object, and superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the target server and the content management platform, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

Optionally, the aforementioned sending of the collected target image to the target server may be achieved in the following manner: the target area is captured by the image capture device on the AR device to obtain the target image; the target image is transmitted through the real-time transmission protocol (Real-time Transport protocol, referred to as RTP) code stream form and send to the target server.

That is to say, the AR device will collect images of the identifiable target area through an image acquisition device such as a camera, and then the AR device will send the target image to the video conference server through the video conference client installed by itself. Specifically, the video conference The client needs to convert the target image into a real-time transport protocol RTP code stream, and send the RTP code stream to the video conference server.

Further, the above-mentioned step S208 is realized by the following methods: obtaining the preset first position of the target content in the target image; The first target object overlaps; if the target content does not overlap with the first target object, the target content is superimposed on the first position in the target image to obtain the AR image; in the case where the target content overlaps with the first target object, adjusting the preset position of the target content in the target image from the first position to a second position, wherein The target content at the second position does not overlap with the target position of the first target object at the target position; superimposing the target content at the second position in the target image, Obtain the AR image.

That is to say, after the video conferencing client obtains the identifier of the first target object sent by the video conferencing server and the target position of the first target object in the target image, it then obtains the corresponding identifier of the first target object from the content management platform. Since the user will initially set the target content at the initial position, the video conferencing client needs to obtain the preset first position of the target content in the target image, and then need to determine whether the target content at the first position is consistent with the target position The first target object overlaps; if there is no overlap, you can directly superimpose the target content on the first position of the target image to obtain an AR image. If there is an overlap, you need to adjust the position of the target content so that the target content is in The target image does not overlap with the first target object, and when it is adjusted so that no overlap occurs, the second position is determined, and the target content is superimposed on the second position in the target image.

It should be noted that during the process of adjusting the target content from the first position to the second position by the video conferencing client, the size of the target content may not change, only the vertex coordinates of the target content change, or the target content may The vertex coordinates of and the size of the target content changed at the same time. Furthermore, in order to realize the sharing of the screen in the AR device so that other people can see the screen in the AR device, the AR device can send the superimposed AR image to the video conference server through its own video conference client so that multiple video conferencing clients on other AR devices obtain AR images from the video conferencing server, and then other users can see the screens in the AR device in this embodiment through other AR devices, realizing the sharing of screens. It should be noted that multiple video conference clients participate in the target video conference through the video conference server.

It should be noted that since the AR screen can be shared, the user using the AR device can mark the AR image, and then other users can also see the mark. Specifically, it can be realized through a data conference. After the data conference is passed, the data Conference client and data conference server. It should be noted that the video conference client and data conference client are installed on the AR device. The video conference client can communicate with the video conference server, and the data conference client can communicate with the data conference client. The server communicates, and the video conferencing client and the data conferencing client can also communicate.

Specifically, the video conference client on the AR device invokes the data conference client on the AR device to obtain the first transparent layer sent by the data conference client; and sends the second transparent layer to the data conference client to indicate The data conferencing client sends the second transparent layer to the data conferencing server for storage, wherein the second transparent layer is a transparent layer obtained by marking the AR image on the first transparent layer , the second transparent layer is used to be overlaid and displayed on the AR image.

That is to say, if the user using the AR device wants to perform some marking operations on the screen, he can call the data conference client of the AR device, and the data conference client sends a first transparent layer. The first transparent layer does not have marking information, and then The video conferencing client can superimpose the first transparent layer on the AR screen, and the user can make some marks on the first transparent layer. For better illustration, the marked first transparent layer is defined as the second transparent layer, and then the video conference client sends the second transparent layer to the data conference client, so that the data conference client can send the second transparent layer Stored in the data conferencing server. Other AR devices can obtain the second transparent layer from the data conference server through their own data conference client, send the second transparent layer to their own video conference client, and then superimpose the second transparent layer on the AR screen. Make other users can also see the mark, realize the sharing of mark.

It should be noted that after other users join the video conference with their own AR devices and see the shared screen, they can also mark the screen, and the marked information is saved in the data conference server, and the users currently using AR devices You can call the data conference client through the video conference client on your own AR device. The data conference client obtains the second transparent layer from the data conference server, and then sends the obtained second transparent layer to the video client, so that the current user The video conferencing client on the AR device can superimpose the second transparent layer on the AR image to obtain the marked AR image, and display the marked AR image through the AR device, and the current user can also see other users' views on the AR image. Image tagging enables sharing.

It should be noted that the execution subject of the above technical solution is the video conferencing client on the AR device. In order to better understand the present disclosure, the embodiment of the present disclosure also provides an image display method, which is applied to the target server. FIG. 3 is According to the flow chart (2) of the image display method of the embodiment of the present disclosure, the process includes the following steps:

Step S302, acquiring the target image sent by the target client;

Step S304, processing the target image to obtain the identification and target position of the first target object, wherein the first target object is located in the target image;

Step S306, sending the identification of the first target object and the target position to the target client, so as to instruct the target client to determine an augmented reality AR image according to the identification of the first target object and the target position.

Through the above steps, the target server acquires the target image sent by the target client, processes the target image, and sends the processed first target object identifier and target position to the target client to instruct the target client to The object's identification and target location determine the augmented reality AR image. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the target server, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

It should be noted that, in order to better understand how the video conferencing server processes the target image to obtain the identification and target position of the first target object, the following is a description of how the video conferencing server implements the internal details:

In an optional embodiment, the multi-point control unit MCU sends the target image to the identification and positioning module of the target server in the following manner; the identification and positioning module performs identification and positioning processing on the target image, The identifier of the first target object and the target position are obtained.

For a better understanding, the target server is used as a video conference server for description below, and the video conference server is composed of a multi-point control unit MCU, an identification and positioning module, and a video conference management system.

That is to say, the video conference server sends the target image to the identification and positioning module in the video conference server through the multi-point control unit (MulTI-point Control Unit, referred to as MCU), and then performs identification and positioning processing on the target image through the identification and positioning module. , to obtain the identifier of the first target object and the target position. It should be noted that before the video conferencing server sends the target image to the identification and positioning module through the MCU, it needs to send an access request to the identification and positioning module through the MCU; Send the Internet Protocol IP address and port number to the MCU.

It should be noted that when the target client sends the target image to the video conference server, it will send a Session Initiation Protocol (SIP) control command to the video conference management module of the video conference server, and then the video conference management module Instructing the MCU of the video conference server to send an access request to the identification and positioning module of the video conference server, and the identification and positioning module of the video conference server will send the Internet protocol IP address and port number to the MCU according to the request.

Further, the video conferencing server sends the target image to the identification and positioning module through the MCU, which can be achieved in the following manner: using the IP address, the port number and the User Datagram Protocol (User Datagram Protocol, referred to as UDP) by the MCU to The target image is sent to the identification and positioning module in the form of a real-time transport protocol (RTP) code stream.

That is to say, the MCU of the video conference server will send the target image to the identification and positioning module of the video conference server in the form of the real-time transport protocol RTP code stream according to the IP address, port number and user datagram protocol sent by the identification and positioning module. Further, the recognition and positioning module of the video conferencing server processes the target image to obtain the identification and target position of the first target object, which can be realized in the following manner: the recognition and positioning module acquires For the frame image, the identification and positioning module performs identification and positioning processing on the frame image to obtain the identification of the first target object and the target position.

Optionally, the identification and positioning module performs identification and positioning processing on the frame image to obtain the identification of the first target object, which may be achieved in the following manner: the identification and positioning module indicates that the first target object is When the target object is a two-dimensional code, the recognition and positioning module analyzes the two-dimensional code to obtain the identification of the first target object; the recognition and positioning module indicates that the first target object is an object when the result of the frame image recognition is In this case, the identification and positioning module determines the identifier of the first target object from a preset object identifier set. It should be noted that the object identifier set includes one or more object identifiers.

Further, the video conferencing server can send the identification of the first target object and the target position of the first target object to the target client in the following manner: the video conference server first sends the identification and target position of the first target object through the identification and positioning module To its own multipoint control unit MCU, and then the MCU of the video conference server sends the identifier of the first target object and the target location to the video conference client through the video conference management module.

It should be noted that the video conference server includes an MCU, an identification and location module, and a video conference management module.

It should be noted that the video conferencing server can not only determine the identity and target position of the first target object in the target image through the target image, but also realize screen sharing between different AR devices.

Specifically, it may be implemented in the following manner: acquiring the AR image sent by the first video conferencing client, wherein the target client includes the first video conferencing client; In the case of the above AR image, the AR image is sent to the second video conferencing client, wherein the multiple video conferencing clients participate in the target video conference through the video conferencing server, and the multiple video conferencing clients include the The first video conference client and the second video conference client. Through the above method, AR images can be shared between different video conferencing clients. Since different AR devices are installed with different video conferencing clients, picture sharing between different AR devices can be realized.

It should be noted that the execution subject of the above steps is the video conferencing server. In order to better understand the technical solution of the present disclosure, the present application also provides an image display method, which is applied to the content management platform. FIG. 4 is a video conference server according to the present disclosure. The flow chart (3) of the image display method of the embodiment, the process includes the following steps:

Step S402, acquiring the identifier of the first target object sent by the target client;

Step S404, determining the target content according to the identifier of the first target object;

Step S406, sending the target content to the target client to instruct the target client to determine an augmented reality AR image according to the target content.

It should be noted that the target client in the embodiment of the present application includes: a video conference client.

It should be noted that the content management platform has correspondences between identifiers of different first objects and target content, and the correspondences are preset by the second target object.

Through the above steps, the content management platform obtains the identifier of the first target object sent by the target client, determines the target content according to the identifier of the first target object, and sends the target content to the target client, so that the target client can determine the Augmented reality AR imagery. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the content management platform, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

In an optional embodiment, the content management platform determines the target content according to the identifier of the first target object, which may be implemented in the following manner: determine a content list according to the identifier of the first target object, and send the content list to The target client: acquiring operation information of the second target object on the content list sent by the video conferencing client; determining target content according to the operation information.

That is to say, the content management platform determines the content list according to the identification of the first target object, and the content list has different contents, and then the content list is sent to the video conferencing client, and the user (second target object) selects the target content from the content list , and then the content management platform sends the target content to the video conference client.

Apparently, the embodiments described above are only some of the embodiments of the present disclosure, not all of them. In order to better understand the above-mentioned image display method, the above-mentioned process is described below in conjunction with the embodiments, but it is not used to limit the technical solutions of the embodiments of the present disclosure, specifically:

This disclosure provides a system that uses data conferencing, video conferencing, and AR glasses to display different related content according to specified QR codes or objects in the front camera screen of AR glasses, and at the same time, other terminals can also mark. The whole system includes content management platform, video conferencing client, data conferencing client, data conferencing server, data conferencing client, AR glasses and other systems and equipment. It should be noted that the video conference server is composed of an identification and positioning module, an MCU, and a video conference management system.

The video conferencing client on the AR glasses sends the picture of the front camera to the MCU through the video conferencing system. The two-dimensional code or the object identifier (ObjID) of the object is returned to the MCU, and returned to the video conference client on the glasses through the video conference management system. The video conference client obtains the pre-associated content (document, video , link, etc.) superimposed on the glasses screen. At the same time, other terminals in the venue can also mark the content seen by the glasses in real time through the marking function of the data conference.

In this disclosure, AR glasses require wireless networking (such as: 5G, Wifi). The video conferencing client (including the video conferencing SDK) needs to be installed on the AR glasses, and the connection between the client and the server is normal.

In order to better understand the above process, FIG. 5 is a frame diagram of an image display method according to an embodiment of the present disclosure, and FIG. 6 is a timing diagram of an image display method according to an embodiment of the present disclosure. FIG. 5 and FIG. 6 are specifically described below. :

The video conference client is the soft client of the video conference; the MCU is the multi-point control unit in the video conference; the video conference management system is the conference management system in the video conference;

(1) First, create the content in the content management platform, and create the association between the identifier ID and the content.

(2) The video conferencing client on the AR glasses will send the image of the camera to the MCU as a video stream. When the video conferencing client needs to initiate a video recognition AR service, it will notify the video conference management system through a Sip message, and then notify the MCU to send The identifying and positioning module initiates a video access request.

(3) After identifying the location module and verifying the source of the request, return the IP and port of the video access to the MCU, and establish the association between the code stream and the port. According to this address, the MCU forwards the RTP code stream to the identification and positioning module through UDP.

(4) The identification and positioning module decodes the RTP code stream, identifies and positions the acquired frame images at a certain frequency (such as 1s), and returns the identification result.

(5) The recognition and positioning module recognizes the frame image in the code stream. For example, if a two-dimensional code appears in the frame picture, the text information parsed after the two-dimensional code is scanned is: zobjid-95da65f8214211eaae330221860e9b7e. At this time, the identification and positioning module will obtain and return the ObjectID (equivalent to the identification of the first target object in the above embodiment) and the area coordinates of the two-dimensional code (equivalent to the target position of the first target object in the above embodiment).

(6) After receiving the identified ObjectID and coordinates, the MCU returns them to the video conference management system, and the video conference management system returns them to the AR glasses (video conference client), the initiator of the video stream, through the Sip protocol.

(7) The video conferencing client queries the associated content list on the content management platform according to the ObjectID, and displays the content list. After the user selects the specific content, he downloads the specific content from the content management platform according to the URL. Because the sequence number is included in the content description file, it can support sequential display of multiple contents.

(8) The video conference client performs AR content display according to the content downloaded from the content management platform and the area coordinate information transparently transmitted from the video conference management system.

It should be noted that when the video recognition needs to be terminated, the video conference soft client informs the MCU through the video conference management system to stop the forwarding of the code stream.

Furthermore, in order to better understand how the video conferencing client on the AR device superimposes the content downloaded by the content management platform on the screen of the AR device, the following is a specific description:

The video conference client receives the four vertex coordinates (X1Y1, X1Y2, X2Y1, X2Y2) of the object sent back by the video conference server in real time and the HTTP URL of the content to be superimposed, and calculates the content to be superimposed in real time. Assuming the screen size (screen resolution), the X axis is set to X, and the Y axis is set to Y, Fig. 7 is a flow chart of content overlay according to an embodiment of the present disclosure, and Fig. 8 is a schematic diagram of content overlay according to an embodiment of the present disclosure , the specific steps in Figure 7 are as follows:

Step S702: obtain screen size (screen resolution);

Step S704: Obtain the four vertex coordinates of the identified object, for example: X1Y1, X1Y2, X2Y1, X2Y2 respectively;

Step S706: judge whether it is X2>X or Y2>Y, in the case of yes, execute step S708, otherwise execute step S709;

Step S708: a prompt pops up to adjust the position of the glasses;

Step S709: set the initial size of the superimposed content, such as 100*50;

Step S710: set the minimum threshold of superimposed content, such as 80*40;

Step S712: Set the initial position of the superimposed content, such as the origin of the lower left corner (0,0), (0,50), (100,50), (100,0);

Step S714: Detect whether the superimposed content collides with the identified object, and obtain the detection result;

Step S716: determine whether the result collides (overlaps), in the case of overlapping, execute step S720, otherwise execute step S718;

Step S718: the superimposed content window is displayed, and the process ends;

Step S720: reduce the size of the overlay window (for example: XY minus 5 pixels);

Step S722: compared with the minimum display threshold, in the case of greater than, execute step S724, otherwise execute step S726;

Step S724: detect collision, execute step S716;

Step S726: adjust the position of the glasses, and reacquire the vertex coordinates;

Step S728: Acquire the initial size of the overlay content;

Step S730: Detect collision, execute step S716.

It should be noted that, in order to better understand the present disclosure, FIG. 9 is an overall flowchart of an image display method according to an embodiment of the present disclosure, and the specific steps are as follows:

S1: The client on the AR glasses joins the video conference/data conference;

It should be noted that the clients on the AR glasses include video conferencing clients and data conferencing clients.

S2: Share the screen of the front camera of the AR glasses through the MCU of the video conferencing server;

S3: The identification and positioning module extracts frames, identifies and locates the video stream transferred from the MCU;

It should be noted that the identifying and positioning module is located in the video conference server.

S4: The identification and positioning module returns the identified object ObjID and coordinates to the client through the MCU;

S5: The client acquires associated content (documents, pictures, videos, etc.) from the content management platform according to the ObjID;

S6: The client performs superimposed display according to the content and coordinates;

S7: Other clients in the venue mark the content and video.

It should be noted that, the data conference client is invoked by the video conference client, so as to realize the marking of the content.

In addition, the above-mentioned technical solution of the embodiment of the present application adopts the technology of video conferencing and data conferencing, which not only reduces the hardware requirements of AR devices, but also solves the shortcomings of the original technology that cannot be shared, marked, and collaborated.

It should be noted that the present application also provides an image display system. FIG. 10 is a frame diagram of an image display system according to an embodiment of the present disclosure, including:

The target server 12 is configured to acquire the target image collected by the target client 16, and after processing the target image, obtain the identification of the first target object and the target position and send it to the target client 16, wherein, the first target object is located in the target image;

The content management platform 14 is configured to acquire the identifier of the first target object sent by the target client 16, and send the target content determined according to the identifier of the first target object to the target client 16;

The target client 16 is configured to superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image.

Through the above system, the target client 16 on the AR device sends the target image collected by the AR device to the target server 12, and obtains the identification and target position of the first target object sent by the target server 12 after processing the target image, The target client 16 can obtain the target content sent by the content management platform according to the identifier of the first target object, and superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the target server and the content management platform, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present disclosure can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods of various embodiments of the present disclosure.

In this embodiment, an image display device is also provided, which is used to implement the above embodiments and preferred implementation modes, and what has already been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

FIG. 11 is a frame diagram (1) of an image display device according to an embodiment of the present disclosure, which includes:

The first sending module 22 is configured to send the collected target image to the target server;

The first obtaining module 24 is configured to obtain the identification and target position of the first target object sent by the target server after processing the target image, wherein the first target object is located in the target image;

The second acquiring module 26 is configured to send the identifier of the first target object to the content management platform, and acquire the target content sent by the content management platform according to the identifier;

The first display module 28 is configured to superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image.

Through the above modules, the target client on the AR device sends the target image collected by the AR device to the target server, and obtains the identification and target position of the first target object sent by the target server after processing the target image, so that the target client The terminal can obtain the target content sent by the content management platform according to the identifier of the first target object, and superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the target server and the content management platform, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

Optionally, the first sending module 22 is also configured to collect an image of the target area through the image collection device on the AR device to obtain the target image; and transmit the target image in the form of a real-time transport protocol (RTP) stream sent to the target server.

Optionally, the first display module 28 is also configured to acquire a preset first position of the target content in the target image; The position of the first target object overlaps; in the case that the target content does not overlap with the first target object, the target content is superimposed on the first position in the target image, and the obtained the AR image; in the case where the target content overlaps with the first target object, adjusting the preset position of the target content in the target image from the first position to a second position, wherein, The target content located at the second position does not overlap with the target position of the first target object located at the target position; superimposing the target content at the second position in the target image above, get the AR image.

Optionally, the first display module 28 is further configured to send the AR image to a video conferencing server, so that multiple video conferencing clients acquire the AR image from the video conferencing server, wherein the target The server includes the video conference server, and the multiple video conference clients participate in the target video conference through the video conference server.

Optionally, the first display module 28 is also configured to call the data conference client corresponding to the AR device to obtain the first transparent layer sent by the data conference client; send the second transparent layer to the data The conference client is configured to instruct the data conference client to send the second transparent layer to the data conference server for storage, where the second transparent layer is to perform the AR image on the first transparent layer The transparent layer obtained after marking, the second transparent layer is used to be superimposed on the AR image.

Optionally, the first display module 28 is further configured to call the data conference client corresponding to the AR device to obtain the second transparent layer sent by the data conference client, wherein the second transparent layer has a mark information, the second transparent layer is the transparent layer obtained by the data conferencing client from the data conferencing server; superimposing the second transparent layer on the AR image to obtain the marked AR image, and passing The AR device displays the marked AR image.

An image display device is also provided in this embodiment, and Fig. 12 is a frame diagram (2) of an image display device according to an embodiment of the disclosure, the device includes:

The third acquiring module 32 is configured to acquire the target image sent by the target client;

The processing module 34 is configured to process the target image to obtain the identification and target position of the first target object, wherein the first target object is located in the target image;

The second display module 36 is configured to send the identification of the first target object and the target position to the target client, so as to instruct the target client to determine the augmented reality according to the identification of the first target object AR image.

Through the above modules, the target server obtains the target image sent by the target client, and processes the target image, and sends the processed first target object identification and target position to the target client to instruct the target client to follow the first target The object's identification and target location determine the augmented reality AR image. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the target server, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

Optionally, the processing module 34 is also configured to send the target image to the identification and positioning module of the target server through the multi-point control unit MCU; through the identification and positioning module, the identification and positioning process is performed on the target image to obtain the An identification of the first target object and the target location.

Optionally, the processing module 34 is also configured to send an access request to the identification and positioning module through the MCU; when the identification and positioning module responds to the access request, send an Internet Protocol IP address through the identification and positioning module. address and port number to the MCU.

Optionally, the processing module 34 is also configured to use the IP address, the port number, and the User Datagram Protocol UDP to send the target image to the recognition system in the form of a real-time transport protocol (RTP) stream through the MCU. positioning module.

Optionally, the processing module 34 is further configured to obtain a frame image from the target image according to a preset frequency through the identification and positioning module; identify the frame image through the identification and positioning module to obtain the first An identification of the target object and the target location.

Optionally, the processing module 34 is further configured to parse the two-dimensional code through the recognition and positioning module to obtain the second An identification of a target object; when the result of the frame image recognition indicates that the first target object is an object, the identification and positioning module is used to determine the identity of the first target object from a preset object identification set IDs, wherein the set of object IDs includes IDs of one or more objects.

Optionally, the second display module 36 is also configured to send the identification of the first target object and the target position to the multipoint control unit MCU of the target server through the identification and positioning module; The identifier of the first target object and the target location are sent to a video conference client, wherein the target client includes the video conference client.

Optionally, the second display module 36 is further configured to acquire the AR image sent by the first video conferencing client, wherein the target client includes the first video conferencing client; When the terminal requests to obtain the AR image, the AR image is sent to the second video conferencing client, wherein the multiple video conferencing clients participate in the target video conference through the video conferencing server, and the multiple video conferencing The clients include the first video conference client and the second video conference client.

An image display device is also provided in this embodiment. FIG. 13 is a frame diagram (3) of an image display device according to an embodiment of the present disclosure. The device includes:

The fourth obtaining module 42 is configured to obtain the identifier of the first target object sent by the target client;

A determination module 44 configured to determine target content according to the identifier of the first target object;

The third display module 46 is configured to send the target content to the target client to instruct the target client to determine an augmented reality AR image according to the target content.

Through the above modules, the content management platform obtains the identifier of the first target object sent by the target client, determines the target content according to the identifier of the first target object, and sends the target content to the target client, so that the target client can determine the Augmented reality AR imagery. By adopting the above technical solution, the traditional augmented reality AR device realizes the augmented reality function independently, which makes the hardware requirements of the AR device too high and the like. Furthermore, by adopting the above technical solution, the AR device communicates with the content management platform, and then cooperates to realize the augmented reality function of the AR device, which reduces the hardware requirements of the AR device.

Optionally, the determining module 44 is further configured to determine a content list according to the identifier of the first target object, and send the content list to the target client; obtain the second target object sent by the video conferencing client The operation information of the object on the content list; determine the target content according to the operation information.

Embodiments of the present disclosure also provide a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.

Optionally, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for performing the following steps:

S1, sending the collected target image to the target server;

S2. Obtain the identification and target position of the first target object sent by the target server after processing the target image, where the first target object is located in the target image;

S3. Send the identifier of the first target object to the content management platform, and acquire the target content sent by the content management platform according to the identifier;

S4. According to the target position of the first target object, superimpose the target content on the target image to obtain an augmented reality AR image.

In an exemplary embodiment, the above-mentioned computer-readable storage medium may include but not limited to: U disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM) , mobile hard disk, magnetic disk or optical disk and other media that can store computer programs.

For specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and exemplary implementation manners, and details will not be repeated here in this embodiment.

Embodiments of the present disclosure also provide an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

S1, sending the collected target image to the target server;

In an exemplary embodiment, the electronic device may further include a transmission device and an input and output device, wherein the transmission device is connected to the processor, and the input and output device is connected to the processor.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned disclosure can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network composed of multiple computing devices In fact, they can be implemented in program code executable by a computing device, and thus, they can be stored in a storage device to be executed by a computing device, and in some cases, can be executed in an order different from that shown here. Or described steps, or they are fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present disclosure is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

An image display method, comprising:

Send the collected target image to the target server;

Obtaining the identification and target position of the first target object sent by the target server after processing the target image, wherein the first target object is located in the target image;

Send the identifier of the first target object to the content management platform, and acquire the target content sent by the content management platform according to the identifier;

According to the target position of the first target object, the target content is superimposed on the target image to obtain an augmented reality AR image.
The method according to claim 1, wherein said sending the collected target image to the target server comprises:

performing image acquisition on the target area through the image acquisition device on the AR device to obtain the target image;

Sending the target image to the target server in the form of a real-time transport protocol RTP code stream.
The method according to claim 1, wherein, according to the target position of the first target object, superimposing the target content on the target image to obtain an augmented reality (AR) image comprises:

Acquiring a preset first position of the target content in the target image;

determining whether the targeted content at the first location overlaps with the first targeted object at the targeted location;

In the case that the target content does not overlap with the first target object, superimposing the target content on a first position in the target image to obtain the AR image;

In the case that the target content overlaps with the first target object, adjusting a preset position of the target content in the target image from the first position to a second position, wherein the target content is located at the second position The target content at the second position does not overlap with the target position of the first target object at the target position; superimposing the target content at the second position in the target image, the obtained AR image described above.
The method according to claim 1, wherein, after the augmented reality AR image is obtained, the method further comprises:

sending the AR image to a video conferencing server, so that multiple video conferencing clients acquire the AR image from the video conferencing server, wherein the target server includes the video conferencing server, and the multiple video conferencing The conference client participates in the target video conference through the video conference server.
The method according to claim 1, wherein, after the augmented reality AR image is obtained, the method further comprises:

calling the data conference client corresponding to the AR device to obtain the first transparent layer sent by the data conference client;

sending the second transparent layer to the data conferencing client to instruct the data conferencing client to send the second transparent layer to the data conferencing server for storage, wherein the second transparent layer is the A transparent layer obtained by marking the AR image on the first transparent layer, and the second transparent layer is used to be superimposed on the AR image.
The method according to claim 1 or 5, wherein after the AR image is displayed by the AR device, the method further comprises:

calling the data conference client corresponding to the AR device to obtain the second transparent layer sent by the data conference client, wherein the second transparent layer has tag information, and the second transparent layer is the data conference The transparent layer obtained by the client from the data conference server;

superimposing the second transparent layer on the AR image to obtain a marked AR image, and displaying the marked AR image through the AR device.
An image display method, comprising:

Obtain the target image sent by the target client;

Processing the target image to obtain the identification and target position of the first target object, wherein the first target object is located in the target image;

Sending the identification of the first target object and the target position to the target client, so as to instruct the target client to determine an augmented reality AR image according to the identification of the first target object and the target position.
The method according to claim 7, wherein said processing said target image to obtain the identification and target position of the first target object comprises:

Send the target image to the identification and positioning module of the target server through the multi-point control unit MCU;

The identification and positioning module performs identification and positioning processing on the target image to obtain the identification of the first target object and the target position.
The method according to claim 8, wherein, before the multipoint control unit MCU sends the target image to the identification and positioning module of the target server, the method further comprises:

sending an access request to the identification and positioning module through the MCU;

When the identification and positioning module responds to the access request, the identification and positioning module sends an Internet Protocol IP address and port number to the MCU.
The method according to claim 9, wherein the sending the target image to the identification and positioning module of the target server through the multi-point control unit MCU includes:

The MCU uses the IP address, the port number and the User Datagram Protocol (UDP) to send the target image to the identification and positioning module in the form of a real-time transport protocol (RTP) code stream.
The method according to claim 8, wherein, performing identification and positioning processing on the target image by the identification and positioning module to obtain the identification and target position of the first target object comprises:

Acquiring frame images from the target image according to a preset frequency through the identification and positioning module;

The identification and positioning module performs identification and positioning processing on the frame image to obtain the identification of the first target object and the target position.
The method according to claim 11, wherein, performing identification and positioning processing on the frame image through the identification and positioning module to obtain the identification of the first target object comprises:

When the result of the frame image recognition indicates that the first target object is a two-dimensional code, the two-dimensional code is analyzed by the identification and positioning module to obtain the identification of the first target object;

When the result of the frame image recognition indicates that the first target object is an object, the identity of the first target object is determined from a preset object identity set by the recognition and positioning module, wherein the The set of object identifiers includes the identifiers of one or more objects.
The method according to claim 7, wherein, after sending the identification of the first target object and the target location to the target client, the method further comprises:

Acquiring the AR image sent by the first video conferencing client, wherein the target client includes the first video conferencing client;

When the second video conferencing client requests to acquire the AR image, sending the AR image to the second video conferencing client, wherein multiple video conferencing clients participate in the target video conference through the video conferencing server, The plurality of video conference clients include the first video conference client and the second video conference client.
An image display method, comprising:

Obtain the identifier of the first target object sent by the target client;

determining target content according to the identifier of the first target object;

sending the target content to the target client to instruct the target client to determine an augmented reality (AR) image according to the target content.
The method according to claim 14, wherein determining the target content according to the identity of the first target object comprises:

determining a content list according to the identifier of the first target object, and sending the content list to the target client;

Obtaining operation information of the second target object on the content list sent by the video conference client;

Target content is determined according to the operation information.
An image display system comprising:

The target server is configured to obtain the target image collected by the target client, and after processing the target image, obtain the identification of the first target object and the target position and send it to the target client, wherein the first a target object is located in the target image;

The content management platform is configured to acquire the identifier of the first target object sent by the target client, and send the target content determined according to the identifier of the first target object to the target client;

The target client is configured to superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image.
An image display device, comprising:

The first sending module is configured to send the collected target image to the target server;

The first obtaining module is configured to obtain the identification and target position of the first target object sent by the target server after processing the target image, wherein the first target object is located in the target image;

The second acquiring module is configured to send the identifier of the first target object to a content management platform, and acquire the target content sent by the content management platform according to the identifier;

The first display module is configured to superimpose the target content on the target image according to the target position of the first target object to obtain an augmented reality AR image.
An image display device, comprising:

The third obtaining module is configured to obtain the target image sent by the target client;

A processing module configured to process the target image to obtain the identification and target position of the first target object, wherein the first target object is located in the target image;

The second display module is configured to send the identification of the first target object and the target position to the target client, so as to instruct the target client to determine according to the identification of the first target object and the target position Augmented reality AR imagery.
An image display device, comprising:

The fourth obtaining module is configured to obtain the identifier of the first target object sent by the target client;

a determining module configured to determine target content according to the identifier of the first target object;

The third display module is configured to send the target content to the target client to instruct the target client to determine an augmented reality AR image according to the target content.
A computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to perform the method described in any one of claims 1 to 15 when running.
An electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 15 through the computer program.