CN115454247A

CN115454247A - Virtual content interaction method and device, electronic equipment and storage medium

Info

Publication number: CN115454247A
Application number: CN202211125010.9A
Authority: CN
Inventors: 朱延武; 童伟峰
Original assignee: Bestechnic Shanghai Co Ltd
Current assignee: Bestechnic Shanghai Co Ltd
Priority date: 2022-09-15
Filing date: 2022-09-15
Publication date: 2022-12-09

Abstract

The application belongs to the technical field of image processing, and discloses a method, a device, electronic equipment and a storage medium for virtual content interaction, wherein the method comprises the steps of displaying a first original image by adopting a display device; shooting the surrounding environment of the display device to obtain an environment image; identifying an environment image to obtain a first image indication position of a target indication object in the surrounding environment aiming at a first original image; and sending a first image rendering indication containing a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering indication. Therefore, the virtual content position pointed by the user can be rendered and then presented to other users, so that other users can be guided to pay attention to the rendering position in the virtual content, and interaction between the user and the virtual content is realized.

Description

Virtual content interaction method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method and an apparatus for virtual content interaction, an electronic device, and a storage medium.

Background

Virtual Reality (VR) head-mounted display devices (e.g., smart eyewear devices) and Augmented Reality (AR) head-mounted display devices utilize VR or AR technology to block human vision and hearing from the outside world and guide a user to create a sensation of being in a Virtual environment. The display principle is that the left and right eye screens respectively display images of the left and right eyes, and the human eyes generate stereoscopic impression in the brain after acquiring the information with the difference.

In some application scenarios (e.g., teaching, meeting, surgery, and presentation scenarios), a listener (e.g., a student) typically wears a VR head-mounted display device and the listener (e.g., a teacher) interprets images displayed in the VR head-mounted display device.

However, in the course of the explanation, the main speaker can only guide a specific position in the image that the listener needs to pay attention to by voice, and it is often difficult for the listener to determine the corresponding position in the image according to the position indication words of the main speaker, that is, it is difficult for the interaction (for example, pointing to a certain position of the image by a finger) between the main speaker and the virtual content (i.e., the image) to be presented to the listener.

Disclosure of Invention

The embodiment of the application aims to provide a method, a device, electronic equipment and a storage medium for virtual content interaction, and is used for solving the problem that a user is difficult to interact with virtual content in an application scene.

In one aspect, a method for virtual content interaction is provided, including:

the first target equipment displays a first original image by adopting a display device;

shooting the surrounding environment of the display device to obtain an environment image;

identifying an environment image, and obtaining a first image indication position of a target indication object in the surrounding environment aiming at a first original image, wherein the first image indication position is a position of a target element in the first original image pointed by the target indication object in the first original image;

a first image rendering indication including a first image indication location is sent to at least one virtual content display device, such that the virtual content display device renders and displays for the first image indication location of the first original image based on the first image rendering indication.

In the implementation process, the virtual content position (i.e. the first original image displayed by the display device) pointed by the user through the target indication object (e.g. a finger) can be rendered and then presented to other users, so as to guide other users who use the VR head-mounted display device to focus on the rendering position in the virtual content, and thus, the interaction between the users and the virtual content is realized.

In one embodiment, the target indicating object includes at least one of: a user limb and a pointing device;

the first image rendering instruction further comprises at least one of the following rendering modes: adjusting resolution, highlighting, framing, displaying a pointing icon, and overlaying a virtual image;

the first original image in the virtual content display device is acquired from the image distribution device or the first target device.

In one embodiment, before displaying the first original image with the display device, the method further comprises:

receiving a first original image sent by image distribution equipment;

acquiring a first original image stored locally;

or, the first image acquisition device acquires an image to obtain a first original image.

In the implementation process, the first original image may be acquired in various ways.

In one embodiment, an environment surrounding a display device is photographed to obtain an environment image, and the method includes:

and when the image acquisition triggering condition is met, shooting the surrounding environment by adopting a second image acquisition device to obtain an environment image.

In the implementation process, the environment image can be acquired according to the application requirements.

In one embodiment, the method further comprises:

according to the rendering mode in the first image rendering instruction, performing rendering operation at the first image instruction position of the first original image;

and displaying the rendered first original image.

In the implementation process, the highlight rendering and display can be performed locally.

In one embodiment, the method further comprises:

receiving a second image rendering indication for a second original image sent by a second target device;

acquiring a second image indication position contained in the second image rendering indication;

rendering at a second image indication position in a second original image according to a rendering mode in a second image rendering indication, wherein the second original image is acquired from a second target device, an image distribution device or locally;

and displaying the rendered second original image.

In the implementation process, the image can be rendered and displayed according to rendering instructions of other devices.

In one aspect, a method for virtual content interaction is provided, which is applied to a virtual content display device, and includes:

receiving a first image rendering indication which is sent by a first target device and contains a first image indication position, wherein the first image indication position is the position of a target element in a first original image presented in a display device pointed by a target indication object in the surrounding environment of the display device in the first original image;

rendering for a first image indication position of the first original image based on the first image rendering indication;

and displaying the rendered first original image.

In one aspect, the target indicating object comprises at least one of: a user limb and a pointing device;

In one aspect, an apparatus for virtual content interaction is provided, including:

a display unit for displaying a first original image by a first target device using a display device;

the shooting unit is used for shooting the surrounding environment of the display device to obtain an environment image;

the identification unit is used for identifying the environment image and obtaining a first image indication position of a target indication object in the surrounding environment aiming at the first original image, wherein the first image indication position is the position of a target element in the first original image pointed by the target indication object in the first original image;

and the rendering unit is used for sending a first image rendering instruction containing a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction.

In one embodiment, the display unit further:

receiving a first original image sent by image distribution equipment;

acquiring a first original image stored locally;

In one embodiment, the camera unit is configured to:

In one embodiment, the rendering unit is further configured to:

and displaying the rendered first original image.

In one embodiment, the rendering unit is further configured to:

rendering at a second image indication position in a second original image according to a rendering mode in a second image rendering indication, wherein the second original image is obtained from a second target device, an image distribution device or locally;

and displaying the rendered second original image.

the display device comprises a receiving unit, a processing unit and a display unit, wherein the receiving unit is used for receiving a first image rendering indication containing a first image indication position sent by a first target device, and the first image indication position is the position of a target element in a first original image presented in a display device pointed by a target indication object in the surrounding environment of the display device in the first original image;

a rendering unit, configured to render a first image indication position of a first original image based on a first image rendering indication;

and the display unit is used for displaying the rendered first original image.

the first original image is acquired from the image distribution apparatus or the first target apparatus.

In one aspect, an electronic device is provided, comprising a processor and a memory, the memory storing computer readable instructions which, when executed by the processor, perform the steps of the method provided in any of the various alternative implementations of virtual content interaction described above.

In one aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method as provided in any of the various alternative implementations of virtual content interaction described above.

In one aspect, a computer program product is provided which, when run on a computer, causes the computer to perform the steps of the method as provided in any of the various alternative implementations of virtual content interaction described above.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a schematic architecture diagram of a virtual content interaction system according to an embodiment of the present application;

fig. 2 is a flowchart of a method for virtual content interaction according to an embodiment of the present disclosure;

fig. 3 is a flowchart of a method for virtual content interaction of a first application scenario according to an embodiment of the present application;

fig. 4 is a flowchart of a method for virtual content interaction of a second application scenario according to an embodiment of the present application;

fig. 5 is a flowchart of a method for virtual content interaction of a third application scenario provided in an embodiment of the present application;

fig. 6 is a block diagram illustrating a structure of an apparatus for virtual content interaction according to an embodiment of the present disclosure;

fig. 7 is a block diagram illustrating a structure of another virtual content interaction apparatus according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.

Some terms referred to in the embodiments of the present application will be described first to facilitate understanding by those skilled in the art.

The terminal equipment: may be a mobile terminal, a fixed terminal, or a portable terminal such as a mobile handset, station, unit, device, multimedia computer, multimedia tablet, internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system device, personal navigation device, personal digital assistant, audio/video player, digital camera/camcorder, positioning device, television receiver, radio broadcast receiver, electronic book device, gaming device, or any combination thereof, including the accessories and peripherals of these devices, or any combination thereof. It is also contemplated that the terminal device can support any type of interface to the user (e.g., wearable device), and the like.

A server: the cloud server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and can also be a cloud server for providing basic cloud computing services such as cloud service, a cloud database, cloud computing, cloud functions, cloud storage, network service, cloud communication, middleware service, domain name service, security service, big data and artificial intelligence platform and the like.

In order to solve the problem that a user is difficult to interact with virtual content in a VR application scenario or an AR application scenario, embodiments of the present application provide a method and an apparatus for virtual content interaction, an electronic device, and a storage medium.

Fig. 1 is a schematic structural diagram of a virtual content interaction system according to an embodiment of the present disclosure. The virtual content interaction system comprises a first target device (such as VR/AR head-mounted display device of a main speaker) and a virtual content display device (such as VR/AR head-mounted display device of a speaker), and can also comprise a second target device (such as VR/AR head-mounted display device of another main speaker) and an image distribution device (such as cloud).

Wherein, virtual content display device is VR head mounted display device or AR head mounted display device. The first target device and the second target device are electronic devices. The electronic device can be a server or a terminal device. Optionally, the first target device and the second target device may also be VR head-mounted display devices or AR head-mounted display devices.

In one embodiment, a first target device: for recognizing a specific image position (i.e., a first image indication position) in a first original image displayed in a display device pointed to by a user through a target indication object (e.g., a finger, a pointer, and a pointing stick), and transmitting a first image rendering indication including the first image indication position and a rendering manner (the rendering manner may be a default or a user indication) to a virtual content display apparatus.

Further, the first target device may also locally render and display the first original image according to the indicated position and rendering manner of the first image.

Further, the first target device may also perform image rendering and display on the second original image that is locally or received according to the image rendering instruction sent by the other device (e.g., the second target device). The functions of the second target device and the first target device may be the same, which is not described herein again.

A virtual content display device: and the rendering module is used for rendering and displaying the image at the indicated position of the first image in the first original image according to the received rendering mode.

Optionally, the first original image and the second original image may be the same or different. If the first original image and the second original image are the same image, the same position or different positions of the same original image may be rendered according to image rendering instructions (e.g., the first image rendering instruction and the second image rendering instruction) sent by multiple devices.

It should be noted that the second target device and the virtual content display device may be the same device or different devices, and are not limited herein. The first target device, the second target device, and the virtual content display device may all send an image rendering instruction including an image indication position and a rendering manner to other devices, and may also perform image rendering and display at the image indication position in the original image according to the received rendering manner. Referring to fig. 2, a flowchart of a virtual content interaction method provided in this embodiment of the present application is applied to a first target device, and a method for virtual content interaction in the virtual content interaction system in fig. 1 is described with reference to fig. 2, where a specific implementation flow of the method is as follows:

step 200: the first original image is displayed using a display device.

In one embodiment, the implementation of step 200 further includes, but is not limited to, at least one of the following ways of obtaining the first raw image:

mode 1: a first original image sent by an image distribution device (such as a cloud) is received.

Mode 2: a first original image locally stored by a first target device is acquired.

Mode 3: the first original image is obtained by image acquisition by a first image acquisition device (e.g., a camera device).

The first target device may further include a first image capturing device. The first image capturing device may also be a different device than the first target device. The display device may be a display screen, a projection device, and the like.

It should be noted that the image (e.g., the first original image) may be a single picture or a document that can be displayed as an image, or may also be a video frame in a video, and may be displayed on the first target device and the plurality of virtual display devices.

Step 201: the surrounding environment of the display device is photographed to obtain an environment image.

In one embodiment, when it is determined that the image capture trigger condition is satisfied, a second image capture device is used to capture the surrounding environment to obtain an environment image. Optionally, the environment image may or may not include the first original image presented in the display device.

As an example, the second image capturing device is used to capture the surrounding environment in real time or periodically (e.g., at an interval of 1 s) to obtain an environment image.

Optionally, the first target device may further include a second image capturing device. The second image capturing device may also be a different device than the first target device. The first image capturing device and the second image capturing device may be the same device or different devices, and are not limited herein.

Step 202: and identifying the environment image to obtain a first image indication position of a target indication object in the surrounding environment aiming at the first original image.

In one embodiment, the first image indicating position is a position of a target element in the first original image (e.g., a coordinate of an element in the first original image) pointed to by the target indicating object in the first original image. Further, the first image indication position may include spatial information in addition to the position information.

Optionally, the target indicating object is an object for indicating any position in the first original image, and may include, but is not limited to, at least one of the following: a user appendage (e.g., a user finger) and a pointing device. The indicating device can be a teaching rod and an indicating rod.

In practical applications, the target indication object may be set according to a practical application scenario, which is not limited herein.

As an example, a second image capture device (e.g., a second camera) is positioned 2 meters directly in front of the display screen (or projection screen). The indication rod of the main speaker is positioned between the display screen and the second image acquisition device. The presenter points to the center position of the image displayed in the display screen through the pointing stick. The second image acquisition device shoots the display screen to acquire an environment image containing the indication rod of the speaker and the first original image displayed in the display screen. And performing pointer recognition on the environment image, and determining the position of the pointer contacted with the first original image (namely the first image indication position).

Therefore, the specific position in the first original image indicated by the user through the finger, the indication rod and the like can be obtained, and the interaction between the real user and the virtual content is realized.

Step 203: and sending a first image rendering indication containing a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering indication.

In one embodiment, a first image rendering indication including a first image indication location and rendering mode is sent to at least one virtual content display device. And the virtual content display equipment acquires a rendering mode in the first image rendering instruction, executes rendering operation at the first image instruction position of the first original image according to the rendering mode and displays the rendered first original image.

The first image rendering instruction further comprises at least one of the following rendering modes: resolution adjustment, highlighting, framing, pointing icon display, and overlaying virtual images (e.g., virtual animated characters); the first original image in the virtual content display device is acquired from the image distribution device or the first target device.

As an example, the virtual content display device receives a first original image sent by an image distribution device (e.g., a cloud), or receives a first original image sent by a first target device, or obtains a locally existing first original image, and performs a rendering operation at a first image indication position of the first original image according to a rendering manner in a first image rendering indication, and displays the rendered first original image.

As another example, the virtual content display apparatus acquires a first original image further included in the first image rendering instruction, performs a rendering operation at a first image indication position of the first original image according to a rendering manner in the first image rendering instruction, and displays the rendered first original image.

Further, the first target device may also render and display the first original image of itself.

In one embodiment, the first target device performs a rendering operation at the first image indication position of the first original image according to the rendering mode in the first image rendering indication, and displays the rendered first original image.

Further, the first target device may also perform image rendering and display according to rendering instructions of other devices.

In one embodiment, a second image rendering indication for a second original image sent by a second target device is received; acquiring a second image indication position contained in the second image rendering indication; rendering at a second image indication position in the second original image according to a rendering mode in the second image rendering indication; and displaying the rendered second original image. The second original image is obtained from a second target device, an image distribution device, or locally.

The specific steps of the second target device sending the second image rendering instruction for the second original image are as described in steps 200 to 203, which are not described herein again.

The method for virtual content interaction of fig. 2 is illustrated below with reference to several application scenarios.

In the first application scenario, the virtual content interaction system includes a first target device, a virtual content display device, and a cloud (i.e., an image distribution device). The first target device comprises a second image acquisition device. Referring to fig. 3, a flowchart of a method for virtual content interaction of a first application scenario is shown, and a specific implementation flow of the method is as follows:

step 300: the first target device receives a first original image of the cloud and displays the first original image through the display screen.

Step 301: the first target device periodically shoots the surrounding environment of the display screen through the second image acquisition device to obtain an environment image.

Alternatively, the surroundings of the display screen or the user of the first target device may be photographed. The second image acquisition device can be fixedly shot or automatically shot in a moving mode, for example, a visual tracking technology can be adopted to track a shot target indicating object.

It should be noted that, when the first target device is a VR/AR head-mounted display device, the second image capturing device may not capture the first original image in the display screen of the VR/AR head-mounted display device, and if the first original image is displayed as a virtual image, the second image capturing device may not capture the virtual image.

Step 302: the first target device identifies the environment image, obtains the position of a first original image in the display screen pointed by the user through the indicating rod, and obtains a first image indicating position.

Optionally, the first image pointing position may further include an orientation of the first original image in the display screen to which the pointing stick is pointing.

Further, if it is determined that the environmental image does not include the pointing stick or the pointing stick does not point to the first original image in the display screen according to the recognition result, the environmental image is discarded, and step 300 is performed.

Step 303: the first target device sends a first image rendering instruction including a first image instruction position and a rendering manner to the virtual content display device.

As one example, the virtual content display device is a VR head mounted display device, including two left and right eye screens (i.e., display screens). The rendering mode may be user-indicated or may be default.

Step 304: the virtual content display device receives a first original image of the cloud and a first image rendering indication of a first target device.

Step 305: and rendering and displaying the first image indication position in the first original image by the virtual content display equipment according to the rendering mode in the first image rendering indication.

Specifically, the virtual content display device displays a first original image, and highlights a unit image area (e.g., an area of 10 × 10 pixels) where a first image indication position of the first original image is located, in a rendering manner (e.g., highlighting) in the first image rendering indication.

By adopting the method, the first image rendering indication and the first original image are transmitted independently, rather than rendering and then transmitting the rendered image, on one hand, the first image rendering indication is often small in data quantity, such as position signals and rendering modes, and the data quantity of the image is increased by transmitting the first original image after rendering, so that the transmitted data quantity is reduced, and particularly, the reliability of transmission is improved for wireless transmission. On the other hand, since the data amount of the first image rendering instruction is small, the time delay is small, and since the first original image may be transmitted first and then the first image rendering instruction may be transmitted, the virtual content display device may display the first original image simultaneously with the first target device, reducing the time delay for the virtual content display device to display the first original image. Moreover, the virtual content display device can select various rendering modes in a personalized manner according to the actual requirements of the user, and whether the rendering modes are adopted or not, so that the use experience of the user is improved.

The virtual content display device and the first target device can receive a first original image sent by the cloud in a wired or wireless mode. The wireless means may be cellular communication, wireless fidelity (WiFi), and Ultra Wide Band (UWB).

Further, the first target device may further acquire and display a first original image stored locally, and send the first original image to the virtual content display device.

In the second application scenario, the virtual content interaction system includes a first target device, a virtual content display device, and a cloud (i.e., an image distribution device). The first target device comprises a first image acquisition device and a second image acquisition device. Referring to fig. 4, a flowchart of a method for virtual content interaction of a second application scenario is shown, and the specific implementation flow of the method is as follows:

step 400: the first target device carries out image acquisition through the first image acquisition device to obtain a first original image.

Alternatively, the first target device may be provided with a first image capturing device and a second image capturing device.

The first image acquisition device is used for acquiring a first original image. The second image acquisition device is used for acquiring an environment image containing a first original image displayed by the display screen, so that in the subsequent step, a target indication object at the periphery of the display screen is identified, and the position of the first image of the first original image displayed in the display screen is indicated. The first image acquisition device and the second image acquisition device can be the same device or different devices.

Step 401: the first target device displays the first original image through the display screen.

Step 402: the first target device periodically shoots the surrounding environment of the display screen through the second image acquisition device to obtain an environment image containing a first original image presented by the display screen.

Step 403: the first target device identifies the environment image, obtains the position of a first original image in the display screen indicated by the user through the teaching rod, and obtains a first image indication position.

Step 404: the first target device sends a first image rendering instruction containing a first image indication position, a rendering mode and a first original image to the virtual content display device.

Step 405: and rendering and displaying the first image indication position in the received first original image by the virtual content display equipment according to the rendering mode in the first image rendering indication.

In a third application scenario, the virtual content interaction system includes a first target device and a plurality of virtual content display devices. The first target device comprises a first image acquisition device, a second image acquisition device and a display screen. The first target device is a VR head mounted display device or an AR head mounted display device.

Referring to fig. 5, a flowchart of a method for virtual content interaction in a third application scenario is shown, where the method is implemented by the following steps:

step 500: the first target device acquires an image through the first image acquisition device to obtain a first original image, and displays the first original image through the display screen.

Further, the first target device may also receive a first original image of the cloud.

Step 501: the first target device shoots the surrounding environment of a user of the first target device in real time through the second image acquisition device to obtain an environment image containing a first original image presented by the display screen.

Step 502: the first target device identifies the environment image, obtains the position of a first original image in a display screen pointed by the finger of the user, and obtains the indication position of the first image.

Step 503: and rendering and displaying the first image indication position in the first original image by the first target device according to the rendering mode indicated by the user.

Step 504: and the first target equipment sends the rendered first original image to each virtual content display equipment.

Step 505: each virtual content display device displays the received rendered first original image.

As one example, in a teaching scenario, a teacher and a student both wear VR/AR head-mounted display devices (e.g., smart eyewear devices) and obtain video (i.e., a first raw image) from the cloud. When a teacher points to a certain area of a video played in a display screen by fingers, the VR/AR head-mounted display equipment of the teacher shoots the area to obtain an environment image, and the fingers of a user in the environment image are identified by an identification module to obtain the position (namely the first image indication position) of a video frame pointed by the teacher. The teacher's VR/AR head mounted display device renders the first image of the position and the rendering mode that contain the video frame and instructs, shares VR/AR head mounted display device for each student. VR/AR head mounted display device of each student instructs the first image of above-mentioned video frame to instruct the position to add frame etc. to render based on first image rendering to show the video frame after rendering, make the student can pay close attention to the rendering region of video frame more.

As another example, in a meeting scenario (which may also be a teaching, surgery, and presentation scenario), participants all wear VR/AR head mounted display devices. The conference speaker (also can be a lecturer, a person supporting an operation, an instrument repairer and the like) acquires an image through a first camera in the VR/AR head-mounted display device (the image acquired by the camera can be a chopping board of the lecturer, an operation process of an operator, a repair process of a repairman, a product of a product introducer and the like), obtains a first original image, and shares the first original image with the VR/AR head-mounted display device of other participants. The VR/AR head mounted display devices of all the participants display the first original image. When the speaker points to a certain area of the first original image through the indicating rod, the VR/AR head-mounted display device of the speaker shoots a scene that the indicating rod points to the certain area of the first original image through the second camera to obtain an environment image, identifies the indicating rod in the environment image to obtain a first image indicating position, and renders the first image containing the first image indicating position to indicate to the VR/AR head-mounted display devices of other participants. And performing framing and other rendering on the first image indication position of the first original image based on the first image rendering indication by the VR/AR head-mounted display equipment of other participants, and displaying the rendered first original image so as to guide other participants to pay attention to the rendering area.

In the embodiment of the application, in a VR/AR application scene, users can watch the same image or video through a VR/AR head-mounted display device, and can recognize and render any position in the image pointed by a user, so that the users can watch the rendered image through the worn VR/AR head-mounted display device to guide the users to pay attention to a certain position in the image, and interaction between the users and virtual content in the VR/AR application scene is realized.

Based on the same inventive concept, the embodiment of the present application further provides a virtual content interaction apparatus, and as the principle of the apparatus and the device for solving the problem is similar to that of a virtual content interaction method, the implementation of the apparatus can refer to the implementation of the method, and repeated details are not repeated.

As shown in fig. 6, which is a schematic structural diagram of an apparatus for virtual content interaction provided in an embodiment of the present application, including:

a display unit 601 configured to display a first original image by a first target device using a display apparatus;

a shooting unit 602, configured to shoot a surrounding environment of the display device to obtain an environment image;

the identifying unit 603 is configured to identify the environment image, and obtain a first image indication position of the target indication object in the surrounding environment with respect to the first original image, where the first image indication position is a position of a target element in the first original image pointed by the target indication object in the first original image;

a rendering unit 604, configured to send a first image rendering instruction including a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays a first image indication position of a first original image based on the first image rendering instruction.

In one embodiment, the display unit 601 further:

receiving a first original image sent by image distribution equipment;

acquiring a first original image stored locally;

In one embodiment, the capturing unit 602 is configured to:

and when the image acquisition triggering condition is determined to be met, shooting the surrounding environment by adopting a second image acquisition device to obtain an environment image.

In one embodiment, the rendering unit 604 is further configured to:

and displaying the rendered first original image.

In one embodiment, the rendering unit 604 is further configured to:

and displaying the rendered second original image.

Fig. 7 is a schematic structural diagram of another virtual content interaction apparatus provided in the embodiment of the present application, including:

a receiving unit 701, configured to receive a first image rendering instruction including a first image indication position sent by a first target device, where the first image indication position is a position, in a first original image, of a target element in the first original image presented in a display apparatus pointed by a target indication object in a surrounding environment of the display apparatus;

a rendering unit 702, configured to render a first image indication position of a first original image based on a first image rendering indication;

a display unit 703 is configured to display the rendered first original image.

In the method, the device, the electronic equipment and the storage medium for virtual content interaction provided by the embodiment of the application, a display device is adopted to display a first original image; shooting the surrounding environment of the display device to obtain an environment image; identifying an environment image, and obtaining a first image indication position of a target indication object in the surrounding environment aiming at a first original image, wherein the first image indication position is a position of a target element in the first original image pointed by the target indication object in the first original image; and sending a first image rendering indication containing a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering indication, wherein the virtual content display device is a virtual reality head-mounted display device or an augmented reality head-mounted display device. In this way, the position of the virtual content pointed by the user through the target pointing object (e.g., a finger) (i.e., the first original image displayed by the display device) can be rendered and presented to other users, so as to guide other users who use the VR head-mounted display device to pay attention to the rendered position in the virtual content, and thus, the interaction between the users and the virtual content is realized.

Fig. 8 shows a schematic structural diagram of an electronic device 8000. Referring to fig. 8, the electronic device 8000 includes: the processor 8010 and the memory 8020 may further include a power supply 8030, a display unit 8040, and an input unit 8050.

The processor 8010 is the control center of the electronic device 8000, and it is to be understood that various functions of the electronic device 8000 may be performed by operating or executing software programs and/or data stored in the memory 8020 by connecting various components using various interfaces and lines, thereby performing overall monitoring of the electronic device 8000.

In the embodiment of the present application, the processor 8010 executes the steps of the above embodiments when calling the computer program stored in the memory 8020.

Alternatively, the processor 8010 may comprise one or more processing units; preferably, the processor 8010 may integrate an application processor, which handles primarily the operating system, user interface, applications, etc., with a modem processor, which handles primarily wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 8010. In some embodiments, the processor, memory, and/or memory may be implemented on a single chip, or in some embodiments, they may be implemented separately on separate chips.

The memory 8020 may mainly include a program storage area and a data storage area, in which the program storage area may store an operating system, various applications, and the like; the stored data area may store data created according to the use of the electronic device 8000, and the like. Further, the memory 8020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The electronic device 8000 may also include a power supply 8030 (e.g., a battery) that may be used to provide power to the various components, which may be logically coupled to the processor 8010 via a power management system, which may be used to manage charging, discharging, and power consumption.

The display unit 8040 may be used to display information input by a user or information provided to the user, various menus of the electronic device 8000, and the like, and in the embodiment of the present invention, the display unit is mainly used to display a display interface of each application in the electronic device 8000 and objects such as texts and pictures displayed in the display interface. The display unit 8040 may include a display panel 8041. The Display panel 8041 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The input unit 8050 can be used to receive information such as numbers or characters input by a user. The input unit 8050 may include a touch panel 8051 and other input devices 8052. Among other things, the touch panel 8051, also referred to as a touch screen, can collect touch operations by a user on or near the touch panel 8051 (e.g., operations by a user on or near the touch panel 8051 using any suitable object or accessory such as a finger, a stylus, etc.).

Specifically, the touch panel 8051 can detect a touch operation of a user, detect signals caused by the touch operation, convert the signals into touch point coordinates, send the touch point coordinates to the processor 8010, receive a command sent by the processor 8010, and execute the command. In addition, the touch panel 8051 can be implemented by various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. Other input devices 8052 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on/off keys, etc.), a trackball, a mouse, a joystick, and the like.

Of course, the touch panel 8051 can cover the display panel 8041, and when the touch panel 8051 detects a touch operation thereon or nearby, the touch panel 8051 is transmitted to the processor 8010 to determine the type of the touch event, and then the processor 8010 provides a corresponding visual output on the display panel 8041 according to the type of the touch event. Although in FIG. 8, the touch panel 8051 and the display panel 8041 are shown as two separate components to implement the input and output functions of the electronic device 8000, in some embodiments, the touch panel 8051 and the display panel 8041 can be integrated to implement the input and output functions of the electronic device 8000.

The electronic device 8000 may also include one or more sensors, such as pressure sensors, gravitational acceleration sensors, proximity light sensors, and the like. Of course, the electronic device 8000 may also include other components such as a camera, as required in a particular application, and these components are not shown in fig. 8 and will not be described in detail since they are not components that are used in the embodiments of the present application.

Those skilled in the art will appreciate that fig. 8 is merely an example of an electronic device and is not limiting of electronic devices and may include more or fewer components than those shown, or some components may be combined, or different components.

In an embodiment of the present application, a computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the communication device may perform the steps in the above embodiments.

For convenience of description, the above parts are separately described as modules (or units) according to functional division. Of course, the functionality of the various modules (or units) may be implemented in the same one or more pieces of software or hardware when implementing the present application.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method of virtual content interaction, comprising:

identifying the environment image, and obtaining a first image indication position of a target indication object in the surrounding environment for the first original image, wherein the first image indication position is a position of a target element in the first original image, which is pointed by the target indication object, in the first original image;

sending a first image rendering indication containing the first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering indication.

2. The method of claim 1, wherein the target indicating object comprises at least one of: a user limb and a pointing device;

the first original image in the virtual content display device is obtained from an image distribution device or the first target device.

3. The method of claim 1, wherein prior to said displaying the first original image with the display device, the method further comprises:

receiving the first original image sent by the image distribution equipment;

acquiring the first original image stored locally;

or, acquiring an image through a first image acquisition device to obtain the first original image.

4. The method of claim 1, wherein said capturing an environment surrounding said display device to obtain an environment image comprises:

and when the image acquisition triggering condition is determined to be met, shooting the surrounding environment by adopting a second image acquisition device to obtain the environment image.

5. The method of any one of claims 1-4, further comprising:

and displaying the rendered first original image.

6. The method of claim 5, wherein the method further comprises:

acquiring a second image rendering indication position contained in the second image rendering indication;

rendering at the second image indication position in the second original image according to the rendering mode in the second image rendering indication, wherein the second original image is obtained from the second target device, the image distribution device or locally;

and displaying the rendered second original image.

7. A method for virtual content interaction is applied to a virtual content display device and comprises the following steps:

receiving a first image rendering indication sent by a first target device and containing a first image indication position, wherein the first image indication position is the position of a target element in a first original image presented in a display device pointed by a target indication object in the surrounding environment of the display device in the first original image;

rendering for a first image indication location of the first original image based on the first image rendering indication;

and displaying the rendered first original image.

8. The method of claim 7, wherein the target indicating object comprises at least one of: a user limb and a pointing device;

9. An apparatus for virtual content interaction, comprising:

the identification unit is used for identifying the environment image to obtain a first image indication position of a target indication object in the surrounding environment relative to the first original image, wherein the first image indication position is a position of a target element in the first original image pointed by the target indication object in the first original image;

a rendering unit, configured to send a first image rendering instruction including the first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays a first image indication position of the first original image based on the first image rendering instruction.

10. The apparatus of claim 9, wherein the target indication object comprises at least one of: a user limb and a pointing device;

11. The apparatus of claim 9, wherein the display unit is further to:

receiving the first original image sent by the image distribution equipment;

acquiring the first original image stored locally;

12. The apparatus of claim 9, wherein the capture unit is to:

13. The apparatus of any of claims 9-12, wherein the rendering unit is further to:

and displaying the rendered first original image.

14. The apparatus of claim 13, wherein the rendering unit is further to:

and displaying the rendered second original image.

15. An apparatus for virtual content interaction, comprising:

a receiving unit, configured to receive a first image rendering indication sent by a first target device and including a first image indication position, where the first image indication position is a position, in a first original image, of a target element in the first original image presented in a display apparatus pointed to by a target indication object in a surrounding environment of the display apparatus;

a rendering unit configured to render for a first image indication position of the first original image based on the first image rendering indication;

and the display unit is used for displaying the rendered first original image.

16. The apparatus of claim 15, wherein the target pointing object comprises at least one of: a user limb and a pointing device;

the first original image is acquired from an image distribution device or the first target device.

17. An electronic device comprising a processor and a memory, the memory storing computer readable instructions that, when executed by the processor, perform the method of any of claims 1-6 or 7-8.

18. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6 or 7-8.