CN114115527B - Augmented reality AR information display method, device, system and storage medium - Google Patents
Augmented reality AR information display method, device, system and storage medium Download PDFInfo
- Publication number
- CN114115527B CN114115527B CN202111280002.7A CN202111280002A CN114115527B CN 114115527 B CN114115527 B CN 114115527B CN 202111280002 A CN202111280002 A CN 202111280002A CN 114115527 B CN114115527 B CN 114115527B
- Authority
- CN
- China
- Prior art keywords
- information
- target
- display
- acquiring
- target user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Abstract
The application discloses an augmented reality AR information display method, device, system and storage medium, and relates to the technical fields of computer vision, augmented/virtual reality and the like. The specific implementation scheme is as follows: acquiring multi-frame video images of a target user, which are acquired by a camera array; acquiring eyeball position information and visual direction information of a target user according to the multi-frame video image; and acquiring target AR information according to the eyeball position information and the visual direction information, and projecting the target AR information onto a display screen for AR display. According to target user's eyeball positional information and visual direction information, trigger and acquire target AR information to project target AR information and go on showing on the display screen, provide more information display opportunities, increase the breadth and the interest of information of intaking simultaneously for target user.
Description
Technical Field
The application discloses an augmented reality AR information display method, device, system and storage medium, and relates to the technical field of image processing, in particular to the technical fields of computer vision, augmented/virtual reality and the like.
Background
A large shopping mall usually reserves a space in the middle position of the mall to serve as a central hall, and people can enjoy luxurious and magnificent scenery in the mall by leaning on the glass guardrail. But the empty hall also presents a problem with the display density of the merchant. In a shopping mall without a hall, when customers stroll, the left side and the right side of the corridor are provided with merchants. However, in a large shopping mall, the sight of customers is difficult to see opposite merchants through the large shopping mall in an open hall, and half of merchant display opportunities are wasted. Besides indoor scenes such as shopping malls, the problem of lacking of merchant display opportunities also exists for outdoor scene scenes such as outer beaches.
Augmented Reality (AR) performs virtual-real fusion by superimposing information of a virtual world on a picture of a real world in an enhanced manner.
Disclosure of Invention
The application provides an augmented reality AR information display method, device and system and a storage medium, so that more information display opportunities are provided, and the breadth and interestingness of information shooting are increased.
According to a first aspect of the present application, there is provided an augmented reality AR information display method, including:
acquiring multi-frame video images of a target user, which are acquired by a camera array;
acquiring eyeball position information and visual direction information of a target user according to the multi-frame video image;
and acquiring target AR information according to the eyeball position information and the visual direction information, and projecting the target AR information to a display screen for AR display.
According to a second aspect of the present application, there is provided an augmented reality AR information display apparatus including:
the image acquisition module is used for acquiring multi-frame video images of the target user, which are acquired by the camera array;
the information acquisition module is used for acquiring eyeball position information and visual direction information of a target user according to the multi-frame video image;
and the trigger display module is used for acquiring target AR information according to the eyeball position information and the visual direction information and projecting the target AR information to a display screen for AR display.
According to a third aspect of the present application, there is provided an augmented reality AR information display system including:
the camera array is used for acquiring multi-frame video images of a target user;
the display screen is used for setting the camera array and performing AR display;
and the augmented reality AR information display device is in signal connection with the camera array and the display screen and is used for realizing the steps of the method in the first aspect.
According to a fourth aspect of the present application, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of the first aspect.
According to a fifth aspect of the present application, there is provided a computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the method of the first aspect.
According to the technical scheme of the embodiment of the application, more information display opportunities can be provided, and the breadth and interestingness of information taking are increased for the target user.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
fig. 1 is a schematic flowchart of an augmented reality AR information display method according to a first embodiment of the present application;
fig. 2 is a schematic flowchart of an augmented reality AR information display method according to a second embodiment of the present application;
fig. 3 is a schematic flowchart of an augmented reality AR information display method according to a third embodiment of the present application;
fig. 4 is a block diagram of an augmented reality AR information display apparatus according to a fourth embodiment of the present application;
fig. 5 is a block diagram of an augmented reality AR information display apparatus according to a fifth embodiment of the present application;
FIG. 6 is an architectural diagram of an augmented reality AR information display system according to an embodiment of the application;
fig. 7 is a scene diagram of augmented reality AR information display in which the embodiment of the present application may be implemented.
FIG. 8 is a block diagram of an electronic device used to implement embodiments of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application to assist in understanding, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that, in the technical solution of the present application, the acquisition, storage, application, and the like of the personal information of the related user all conform to the regulations of the relevant laws and regulations, and do not violate the customs of the public order.
Large-scale shopping malls usually reserve a space in the middle position of the large-scale shopping mall as a central hall, and people can enjoy luxurious and magnificent scenery in the shopping mall by leaning on the glass guardrail. But the empty hall also presents a problem for the merchant to display density. In a shopping mall without a hall, when customers stroll, the left side and the right side of the corridor are provided with merchants. However, in a large shopping mall, the clear hall makes it difficult for the sight of customers to see opposite merchants through the hall, which wastes half of merchant display opportunities. Besides indoor scenes such as shopping malls, the problem of lacking of merchant display opportunities also exists for outdoor scene scenes such as beaches.
Augmented Reality (AR) technology superimposes information of a virtual world on a picture of a real world in an enhanced manner to achieve virtual-real fusion.
In order to solve the technical problems, the application provides an augmented reality AR information display method, an augmented reality AR information display device, an electronic device and a computer-readable storage medium, so that more information display opportunities can be provided, and the breadth and interestingness of information pickup can be increased for a target user.
An augmented reality AR information display method, an apparatus, an electronic device, and a computer-readable storage medium according to embodiments of the present application are described below with reference to the accompanying drawings.
Fig. 1 is a flowchart of an augmented reality AR information display method according to a first embodiment of the present application. It should be noted that the augmented reality AR information display method according to the embodiment of the present application may be applied to the augmented reality AR information display device according to the embodiment of the present application. The augmented reality AR information display apparatus may be configured on an electronic device. As shown in fig. 1, the augmented reality AR information display method may include the following steps.
In step S101, a plurality of frames of video images of a target user collected by a camera array are acquired.
Optionally, the display screen may be a large-area transparent screen, or a semi-transparent screen, or an opaque common display screen, and a camera with human eye sight following capability is configured on the common display screen.
It can be understood that the scheme of the application enables the transparent screen or the semitransparent screen to directly perform AR display. For a common display screen, a camera with human eye sight following capability needs to be configured to realize enhanced display.
Optionally, the small cameras are arranged in an array on the display screen.
For example, in a practical application scenario, such as a mall hall, the translucent screen is installed on a guardrail of the mall hall. When a target user (i.e., an observer or so-called customer) in a mall moves in front of the transparent screen, the camera array disposed on the transparent screen becomes capable of capturing multiple frames of video images of the target user. Wherein, the image may be an RGB image.
In step S102, eyeball position information and visual direction information of the target user are acquired according to the multiple frames of video images.
In the embodiment of the application, eyeball position information and visual direction information of a target user need to be acquired according to a plurality of frames of video images, and AR display is triggered through the eyeball position information and the visual direction information of the target user. Therefore, the eyeball position information and the visual direction information of the target user need to be acquired based on the multi-frame video images.
Because the distance and the orientation of the camera array are fixed, the position information and the observation orientation information (namely the visual direction information) of the eyeballs of the target users (namely the observers of the display screen) can be accurately positioned through the computer vision face recognition capability and the triangulation principle. The implementation mode of obtaining the eyeball position information and the visual direction information of the target user can be realized by an eyeball tracking algorithm.
In step S103, target AR information is acquired according to the eyeball position information and the visual direction information, and the target AR information is projected onto a display screen for AR display.
In the embodiment of the application, the target user does not need to take out the mobile phone or the tablet computer, and only needs to be close to a large-area transparent display screen, so that the augmented reality AR experience can be obtained by naked eyes.
The AR display is triggered through eyeball position information and visual direction information of the target user, and information of the virtual world is overlaid on a picture of the real world in an enhanced mode to achieve virtual-real fusion and increase the breadth and interestingness of information shooting of the target user.
By taking the application scene of the market hall as an example, after the target user observes the transparent screen, AR display is carried out on the shop information opposite to the target user on the transparent screen, the target user can purposefully select the interested shop according to the AR information, the time of the customer is saved, and the user experience is improved.
The augmented reality AR information display method provided by the embodiment of the application acquires multi-frame video images of a target user through the acquisition camera array, and acquires eyeball position information and visual direction information of the target user according to the multi-frame video images; according to eyeball positional information and visual direction information, triggering and obtaining target AR information, projecting the target AR information onto a display screen for displaying, providing more information display opportunities through AR display, and increasing the breadth and interest of information pickup for a target user.
Fig. 2 is a flowchart of an augmented reality AR information display method according to a second embodiment of the present application. As shown in fig. 2, the augmented reality AR information display method may include the following steps.
In step S201, a plurality of frames of video images of a target user collected by a camera array are acquired.
It should be noted that, the specific implementation process and principle of step S201 are the same as step S101 in the foregoing embodiment, and are not described herein again.
In step S202, a three-dimensional reconstruction model of a face of a target user is obtained by a Motion estimation shape SFM (Structure From Motion) algorithm From a plurality of frames of video images.
SFM algorithm is a technique for estimating three-dimensional structures from a series of two-dimensional image sequences containing visual motion information. And a three-dimensional reconstruction model of the face of the target user is constructed through an SFM algorithm, so that reliable data basis is provided for obtaining accurate eyeball information of the target user.
In step S203, eyeball position information and visual direction information of the target user are acquired according to the three-dimensional reconstruction model.
Optionally, according to the three-dimensional reconstruction model, eyeball position information and visual direction information of the target user are obtained by a face feature point detection method (Facial landmark detection).
In the embodiment, by combining the SFM algorithm and the face feature point detection method, better and more accurate eyeball position information and visual direction information can be obtained.
In step S204, in response to the eyeball position information satisfying the distance threshold, target AR information is acquired according to the visual direction information.
In the embodiment of the application, the target user does not need to take out the mobile phone or the tablet personal computer, and only needs to be close to a large-area transparent screen, so that the augmented reality AR experience can be obtained by naked eyes.
When a target user (an observer of the display screen) observes outward through the display screen and the distance between the observer and the display screen meets the distance threshold requirement of augmented reality display, augmented reality can be triggered.
It will be appreciated that there is a requirement for the distance between the viewer and the display screen, primarily in view of the viewer's parallax, and that generally the further the distance, the smaller the AR parallax, but at the same time the weaker the AR viewing range. For example, when the display screen is a translucent screen, the AR display is triggered when the observer is generally 1 meter away from the translucent screen.
Optionally, obtaining target AR information according to the visual direction information includes:
acquiring a visual range to which the visual direction information belongs according to the visual direction information;
acquiring target AR information from pre-stored AR materials according to the visual range; wherein the content of the first and second substances,
the AR material includes a plurality of AR information stored according to different visual ranges.
Optionally, the obtaining mode of the AR material is:
through supplementary collection equipment, confirm the inside structure information of market and the positional information of display screen for the market hall. Wherein, supplementary collection equipment optional fisheye panoramic lens.
And carrying out three-dimensional drawing of the market according to the internal structure information of the market and the position information of the display screen relative to the market hall, and designing corresponding AR content according to the three-dimensional drawing of the market.
In step S205, projecting the target AR information to the display screen, and enabling the target AR information to perform AR display in a target display area of the display screen; wherein the target display area is acquired based on the visual direction and the visual field range threshold.
When a target user (an observer of the display screen) observes outward through the display screen and the distance between the observer and the display screen meets the distance threshold requirement of augmented reality display, augmented reality within a certain visual angle can be triggered.
Augmented reality within a certain viewing angle can be understood as: when an observer meets the distance requirement from the display screen and looks ahead of the display screen, AR display may occur on the display screen in the viewing cone within a certain range of the field of view, for example, on the display screen in the viewing cone within a 90 degree range. When display conflicts occur among multiple observers, only the observer with the first trigger or the observer with no movement in position is satisfied. Thus, a large area transparent display will satisfy AR enhancement to mall halls at a customer viewing angle within a small range.
The target AR information is projected to a target display area on the display screen to be displayed, and the breadth and interestingness of information shooting are increased in a visual range convenient for a target user to observe.
The augmented reality AR information display method provided by the embodiment of the application acquires multi-frame video images of a target user through the acquisition camera array, and acquires eyeball position information and visual direction information of the target user according to the multi-frame video images; and triggering to acquire target AR information according to the eyeball position information and the visual direction information, and projecting the target AR information to a target display area on a display screen for display. More information display opportunities are provided through AR display, and meanwhile, the breadth and interestingness of information pickup are increased for target users in the visual range convenient for observation.
Fig. 3 is a flowchart of an augmented reality AR information display method according to a third embodiment of the present application. As shown in fig. 3, the augmented reality AR information display method may include the following steps.
In step S301, multiple frames of video images of the target user collected by the camera array are acquired.
In step S302, eyeball position information and visual direction information of the target user are acquired according to the multiple frames of video images.
It should be noted that the specific implementation process and principle of steps S301 and S302 are the same as those of steps S101 and S102 in the above embodiments, and are not described herein again.
In step S303, consumption preferences of the target user are acquired, and target advertisement information is acquired according to the consumption preferences.
Optionally, obtaining consumption preferences of the target users, and obtaining target advertisement information according to the consumption preferences includes:
acquiring a face image when a user transacts a membership card;
acquiring identity information of a target user based on the face image and the multi-frame video image;
acquiring member card consumption information of a target user according to the identity information, and acquiring consumption preference of the target user according to the member card consumption information;
and acquiring target advertisement information according to the consumption preference of the target user.
In the embodiment of the application, consumption preference of the target user is obtained according to the membership card information and the collected image, targeted advertisement display is formulated according to the consumption preference of the target user, more information obtaining opportunities are provided for the target user, and user experience is improved.
In step S304, target AR information is acquired and projected onto a display screen for AR display, and target advertisement information is projected onto the display screen for display, according to the eyeball position information and the visual direction information.
The augmented reality AR information display method provided by the embodiment of the application projects target AR information to a target display area on a display screen for display. More information display opportunities are provided through AR display, and the breadth and interestingness of information pickup are increased for target users in the visual range convenient for observation. Meanwhile, the advertisement information which is possibly interested by the target user is displayed, the information is pushed to the user, and the interest of the user, the visual impact and the development of new advertisement statuses are effectively improved.
As an implementation of the methods shown in the above figures, the present application provides an embodiment of a virtual device for implementing an augmented reality AR information display method, and further refer to fig. 4, which shows a schematic structural diagram of an augmented reality AR information display device provided in a fourth embodiment of the present application. As shown in fig. 4, the augmented reality AR information display apparatus may include an image acquisition module 410, an information acquisition module 420, and a trigger display module 430.
Specifically, the image obtaining module 410 is configured to obtain a multi-frame video image of a target user, which is collected by a camera array;
the information acquisition module 420 is configured to acquire eyeball position information and visual direction information of a target user according to the multi-frame video image;
and the triggering display module 430 is configured to acquire target AR information according to the eyeball position information and the visual direction information, and project the target AR information onto a display screen for AR display.
In an embodiment of the present application, the information obtaining module 420 is specifically configured to:
acquiring a three-dimensional reconstruction model of the face of a target user through a motion estimation shape SFM algorithm according to a plurality of frames of video images;
and acquiring eyeball position information and visual direction information of the target user according to the three-dimensional reconstruction model.
In an embodiment of the present application, when acquiring the eyeball position information and the visual direction information of the target user according to the three-dimensional reconstruction model, the information acquiring module 420 is configured to:
and acquiring eyeball position information and visual direction information of the target user by a face characteristic point detection method according to the three-dimensional reconstruction model.
In an embodiment of the present application, the triggering display module 430 is specifically configured to:
responding to the eyeball position information meeting the distance threshold, and acquiring target AR information according to the visual direction information;
projecting the target AR information to a display screen, and enabling the target AR information to be subjected to AR display in a target display area of the display screen; wherein the target display area is acquired based on the visual direction and the visual field range threshold.
In an embodiment of the present application, when the trigger display module 430 acquires the target AR information according to the visual direction information, it is specifically configured to:
acquiring a visual range to which the visual direction information belongs according to the visual direction information;
acquiring target AR information from pre-stored AR materials according to the visual range; wherein the AR material includes a plurality of AR information stored according to different visual ranges.
With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.
The augmented reality AR information display device provided by the embodiment of the application acquires multi-frame video images of a target user through the acquisition camera array, and acquires eyeball position information and visual direction information of the target user according to the multi-frame video images; according to eyeball position information and visual direction information, triggering and acquiring target AR information, projecting the target AR information onto a display screen for displaying, providing more information display opportunities through AR display, and increasing the breadth and the interestingness of information pickup for a target user.
Fig. 5 is a schematic structural diagram of an augmented reality AR information display device according to a fifth embodiment of the present application, and as shown in fig. 5, the augmented reality AR information display device includes: an image acquisition module 510, an information acquisition module 520, a trigger display module 530, and an advertisement acquisition module 540.
The image obtaining module 510 and the information obtaining module 520 have the same structure and function as the image obtaining module 410 and the information obtaining module 420.
In an embodiment of the present application, the advertisement obtaining module 540 is specifically configured to:
acquiring a face image when a user transacts a membership card;
acquiring identity information of a target user based on the face image and the multi-frame video image;
acquiring member card consumption information of a target user according to the identity information, and acquiring consumption preference of the target user according to the member card consumption information;
and acquiring target advertisement information according to the consumption preference of the target user.
In an embodiment of the present application, the triggering display module 530 is specifically configured to: and according to the eyeball position information and the visual direction information, acquiring target AR information, projecting the target AR information onto a display screen for AR display, and projecting the target advertisement information onto the display screen for display.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The augmented reality AR information display device provided by the embodiment of the application acquires multi-frame video images of a target user through the acquisition camera array, and acquires eyeball position information and visual direction information of the target user according to the multi-frame video images; according to eyeball position information and visual direction information, triggering and acquiring target AR information, projecting the target AR information onto a display screen for displaying, providing more information display opportunities through AR display, and increasing the breadth and the interestingness of information pickup for a target user. And further displaying advertisement information which is possibly interested by the target user, pushing the information for the user, and effectively improving the interest of the user, the visual impact and the development of new advertisement statuses.
Fig. 6 is an architecture diagram of an augmented reality AR information display system according to one embodiment of the present application. As shown in fig. 6, the augmented reality AR information display system includes:
the camera array 610 is used for collecting multi-frame video images of a target user;
a display screen 630 for setting the camera array 610 and performing AR display;
and the augmented reality AR information display device 620 is in signal connection with both the camera array 610 and the display screen 630 and is used for realizing the steps of the augmented reality AR information display method.
In one embodiment of the present application, the display screen is installed on a guardrail of an indoor scene, or the display screen is installed on a guardrail or a bracket of an outdoor scene.
During the application, to indoor scene, for example the market hall, the optional display screen 630 of installing on indoor guardrail, the user of being convenient for observes, and does not occupy more than space. For outdoor scenes, the display screen 630 may be optionally mounted on a guardrail or stand of the outdoor scene, or other location convenient for installation and viewing by the user, depending on the environment.
In one embodiment of the present application, the display screen is mounted on a railing in a mall hall.
Optionally, the augmented reality AR information display device 620 is a host, the display screen 630 is connected to a power supply to supply power to the display screen 630, and the host is disposed between the power supply and the display screen 630.
Optionally, the display screen may be a large-area transparent screen, or a semi-transparent screen, or an opaque common display screen, and a camera with human eye sight following capability is configured on the common display screen.
It can be understood that the scheme of the application enables the transparent screen or the semitransparent screen to directly perform AR display. For a common display screen, a camera with human eye sight following capability needs to be configured to realize enhanced display.
The augmented reality AR information display system provided by the embodiment of the application acquires multi-frame video images of a target user through the camera array 610, and acquires eyeball position information and visual direction information of the target user according to the multi-frame video images through the augmented reality AR information display device 620; according to eyeball position information and visual direction information, triggering and acquiring target AR information, projecting the target AR information onto the display screen 630 for displaying, providing more information display opportunities through AR display, and increasing the breadth and interestingness of information pickup for a target user. And further displaying advertisement information which is possibly interested by the target user, pushing information for the user, and effectively improving the interest of the user, the visual impact and the development of new advertisement statuses.
Fig. 7 is a scene diagram of augmented reality AR information display in which the embodiment of the present application may be implemented. For a mall hall, as shown in fig. 7, a display screen 720 is installed on a guardrail 730 in a room, and an array camera array 710 is arranged on the display screen 720. When the observer of display screen sees through display screen 720 and outwards observes to when the distance of observer apart from the display screen satisfies the distance threshold value requirement that the augmented reality shows, can trigger the augmented reality in the certain visual angle.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 8, the embodiment of the present application is a block diagram of an electronic device according to a method for displaying augmented reality AR information. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 8, the electronic apparatus includes: one or more processors 801, memory 802, and interfaces for connecting the various components, including a high speed interface and a low speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, if desired. Also, multiple electronic devices may be connected, with each device providing some of the necessary operations (e.g., as an array of servers, a group of blade servers, or a multi-processor system). Fig. 8 illustrates an example of a processor 801.
The memory 802 is a non-transitory computer readable storage medium as provided herein. Wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform a method of Augmented Reality (AR) information display as provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the method of augmented reality AR information display provided herein.
The memory 802, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the method for augmented reality AR information display in the embodiments of the present application (e.g., the image acquisition module 410, the information acquisition module 420, and the trigger display module 430 shown in fig. 4). The processor 801 executes various functional applications of the server and data processing by running non-transitory software programs, instructions, and modules stored in the memory 802, that is, implements the method of augmented reality AR information display in the above-described method embodiment.
The memory 802 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the electronic device displayed with the augmented reality AR information, and the like. Further, the memory 802 may include high speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 802 optionally includes memory located remotely from the processor 801, which may be connected over a network to an electronic device for augmented reality AR information display. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the method of augmented reality AR information display may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803, and the output device 804 may be connected by a bus or other means, and are exemplified by a bus in fig. 8.
The input device 803 may receive input numeric or character information and generate key signal inputs related to target user settings and function controls of the electronic device for display of augmented reality AR information, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick, or like input device. The output devices 804 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a target user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a target user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a target user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a target user; for example, feedback provided to the target user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the target user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a target user computer having a graphical target user interface or a web browser through which a target user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, a camera array is obtained to collect multi-frame video images of a target user, and eyeball position information and visual direction information of the target user are obtained according to the multi-frame video images; and triggering to acquire target AR information according to the eyeball position information and the visual direction information, projecting the target AR information onto a display screen for displaying, providing more information display opportunities through AR display, and increasing the breadth and the interestingness of information pickup for a target user. 8230
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present application can be achieved, and the present invention is not limited herein.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (16)
1. An Augmented Reality (AR) information display method includes:
acquiring multi-frame video images of a target user, which are acquired by a camera array;
acquiring eyeball position information and visual direction information of a target user according to the multi-frame video image;
acquiring target AR information according to the eyeball position information and the visual direction information, and projecting the target AR information onto a display screen for AR display, wherein the display screen is used for setting the camera array and performing AR display, and is installed on a guardrail of an indoor scene, or is installed on a guardrail or a support of an outdoor scene;
wherein, according to eyeball positional information and visual direction information, obtain target AR information to with target AR information projects carry out AR and show on the display screen, include:
responding to the eyeball position information meeting a distance threshold, and acquiring target AR information according to the visual direction information;
projecting the target AR information to the display screen, and enabling the target AR information to carry out AR display in a target display area of the display screen; wherein the target display area is acquired based on the visual direction and a field of view range threshold;
and triggering AR display through the eyeball position information and the visual direction information of the target user.
2. The method according to claim 1, wherein the obtaining of the eyeball position information and the visual direction information of the target user according to the plurality of frames of video images comprises:
acquiring a three-dimensional reconstruction model of the face of a target user through a motion estimation shape SFM algorithm according to the multi-frame video images;
and acquiring eyeball position information and visual direction information of the target user according to the three-dimensional reconstruction model.
3. The method according to claim 2, wherein the obtaining of the eyeball position information and the visual direction information of the target user according to the three-dimensional reconstruction model comprises:
and acquiring eyeball position information and visual direction information of the target user by a face characteristic point detection method according to the three-dimensional reconstruction model.
4. The method of claim 1, wherein the obtaining target AR information according to the visual direction information comprises:
acquiring a visual range to which the visual direction information belongs according to the visual direction information;
acquiring target AR information from pre-stored AR materials according to the visual range; wherein the AR material includes a plurality of AR information stored according to different visual ranges.
5. The method of claim 1, wherein the method further comprises:
acquiring a face image when a user transacts a membership card;
acquiring identity information of a target user based on the face image and the multi-frame video image;
acquiring member card consumption information of a target user according to the identity information, and acquiring consumption preference of the target user according to the member card consumption information;
and acquiring target advertisement information according to the consumption preference of the target user.
6. The method according to claim 5, wherein the obtaining target AR information according to the eyeball position information and the visual direction information and projecting the target AR information onto a display screen for AR display comprises:
and according to the eyeball position information and the visual direction information, acquiring target AR information, projecting the target AR information onto a display screen for AR display, and projecting the target advertisement information onto the display screen for display.
7. An Augmented Reality (AR) information display device comprising:
the image acquisition module is used for acquiring multi-frame video images of the target user, which are acquired by the camera array;
the information acquisition module is used for acquiring eyeball position information and visual direction information of a target user according to the multi-frame video image;
the trigger display module is used for acquiring target AR information according to the eyeball position information and the visual direction information, and projecting the target AR information to a display screen for AR display, wherein the display screen is used for setting the camera array and performing AR display, and is installed on a guardrail of an indoor scene, or is installed on a guardrail or a support of an outdoor scene;
the trigger display module is specifically configured to:
responding to the eyeball position information meeting a distance threshold, and acquiring target AR information according to the visual direction information;
projecting the target AR information to a display screen, and enabling the target AR information to carry out AR display in a target display area of the display screen; wherein the target display area is acquired based on the visual direction and a field of view range threshold;
and triggering AR display through the eyeball position information and the visual direction information of the target user.
8. The apparatus according to claim 7, wherein the information obtaining module is specifically configured to:
acquiring a three-dimensional reconstruction model of the face of a target user through a motion estimation shape SFM algorithm according to the multi-frame video images;
and acquiring eyeball position information and visual direction information of the target user according to the three-dimensional reconstruction model.
9. The apparatus according to claim 8, wherein the information obtaining module, when obtaining the eyeball position information and the visual direction information of the target user according to the three-dimensional reconstruction model, is configured to:
and acquiring eyeball position information and visual direction information of the target user by a face characteristic point detection method according to the three-dimensional reconstruction model.
10. The apparatus according to claim 7, wherein the trigger display module, when acquiring the target AR information according to the visual direction information, is specifically configured to:
acquiring a visual range to which the visual direction information belongs according to the visual direction information;
acquiring target AR information from pre-stored AR materials according to the visual range; wherein the AR material includes a plurality of AR information stored according to different visual ranges.
11. The apparatus of claim 7, wherein the apparatus further comprises an advertisement acquisition module, the advertisement acquisition module being specifically configured to:
acquiring a face image when a user transacts a membership card;
acquiring identity information of a target user based on the face image and the multi-frame video image;
acquiring membership card consumption information of a target user according to the identity information, and acquiring consumption preference of the target user according to the membership card consumption information;
and acquiring target advertisement information according to the consumption preference of the target user.
12. The apparatus according to claim 11, wherein the trigger display module is further specifically configured to: and according to the eyeball position information and the visual direction information, acquiring target AR information, projecting the target AR information onto the display screen for AR display, and projecting the target advertisement information onto the display screen for display.
13. An Augmented Reality (AR) information display system comprising:
the camera array is used for acquiring multi-frame video images of a target user;
the display screen is used for setting the camera array and performing AR display;
an Augmented Reality (AR) information display device, in signal connection with both the camera array and the display screen, for implementing the steps of the method of any one of claims 1 to 6.
14. The system of claim 13, wherein the display screen is mounted on a guardrail of an indoor scene or mounted on a guardrail or a support of an outdoor scene.
15. The system of claim 13, wherein the display screen is mounted on a railing of a mall hall.
16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111280002.7A CN114115527B (en) | 2021-10-29 | 2021-10-29 | Augmented reality AR information display method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111280002.7A CN114115527B (en) | 2021-10-29 | 2021-10-29 | Augmented reality AR information display method, device, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114115527A CN114115527A (en) | 2022-03-01 |
| CN114115527B true CN114115527B (en) | 2022-11-29 |
Family
ID=80379837
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111280002.7A Active CN114115527B (en) | 2021-10-29 | 2021-10-29 | Augmented reality AR information display method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114115527B (en) |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101950550B (en) * | 2010-09-28 | 2013-05-29 | 冠捷显示科技（厦门）有限公司 | Display device for displaying pictures at different angles based on visual angle of viewer |
| WO2014056000A1 (en) * | 2012-10-01 | 2014-04-10 | Coggins Guy | Augmented reality biofeedback display |
| CN105955456B (en) * | 2016-04-15 | 2018-09-04 | 深圳超多维科技有限公司 | The method, apparatus and intelligent wearable device that virtual reality is merged with augmented reality |
| CN107463608A (en) * | 2017-06-20 | 2017-12-12 | 上海汇尔通信息技术有限公司 | A kind of information-pushing method and system based on recognition of face |
| JP6855350B2 (en) * | 2017-08-08 | 2021-04-07 | アルパイン株式会社 | Head-up display device, navigation device, display method |
| CN108362279B (en) * | 2018-01-08 | 2022-02-15 | 北京小米移动软件有限公司 | Shopping navigation method, device and system based on AR (augmented reality) technology |
| US10948983B2 (en) * | 2018-03-21 | 2021-03-16 | Samsung Electronics Co., Ltd. | System and method for utilizing gaze tracking and focal point tracking |
| CN112262361A (en) * | 2018-04-20 | 2021-01-22 | Pcms控股公司 | Method and system for gaze-based control of mixed reality content |
| CN109298780A (en) * | 2018-08-24 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Information processing method, device, AR equipment and storage medium based on AR |
| CN109885169B (en) * | 2019-02-25 | 2020-04-24 | 清华大学 | Eyeball parameter calibration and sight direction tracking method based on three-dimensional eyeball model |
| US11107291B2 (en) * | 2019-07-11 | 2021-08-31 | Google Llc | Traversing photo-augmented information through depth using gesture and UI controlled occlusion planes |
| CN110458122B (en) * | 2019-08-15 | 2022-04-22 | 京东方科技集团股份有限公司 | Sight line calibration method, display device playing method and sight line calibration system |
| CN110942521B (en) * | 2019-11-27 | 2024-03-08 | 北京新势界科技有限公司 | AR information point display method and device |
| CN112379812B (en) * | 2021-01-07 | 2021-04-23 | 深圳追一科技有限公司 | Simulation 3D digital human interaction method and device, electronic equipment and storage medium |
-
2021
- 2021-10-29 CN CN202111280002.7A patent/CN114115527B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114115527A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11017603B2 (en) | Method and system for user interaction | |
| CN107920734B (en) | Sight line detection method and device | |
| JP6028351B2 (en) | Control device, electronic device, control method, and program | |
| CN111586319B (en) | Video processing method and device | |
| US20170195664A1 (en) | Three-dimensional viewing angle selecting method and apparatus | |
| US11468643B2 (en) | Methods and systems for tailoring an extended reality overlay object | |
| CN104915979A (en) | System capable of realizing immersive virtual reality across mobile platforms | |
| US11720996B2 (en) | Camera-based transparent display | |
| CN109815409A (en) | A kind of method for pushing of information, device, wearable device and storage medium | |
| JP2013196158A (en) | Control apparatus, electronic apparatus, control method, and program | |
| CN111833458A (en) | Image display method and device, equipment and computer readable storage medium | |
| CN112105983A (en) | Enhanced visual ability | |
| CN111918114A (en) | Image display method, image display device, display equipment and computer readable storage medium | |
| CN114115527B (en) | Augmented reality AR information display method, device, system and storage medium | |
| JP2022515608A (en) | Systems and / or methods for parallax correction in large area transparent touch interfaces | |
| Narducci et al. | Enabling consistent hand-based interaction in mixed reality by occlusions handling | |
| JP7293362B2 (en) | Imaging method, device, electronic equipment and storage medium | |
| CN106127858B (en) | Information processing method and electronic equipment | |
| US10409464B2 (en) | Providing a context related view with a wearable apparatus | |
| CN116917844A (en) | Adjustable personal user interface in cross-application virtual reality scenarios | |
| CN110599602A (en) | AR model training method and device, electronic equipment and storage medium | |
| CN111193858A (en) | Method and system for shooting and displaying augmented reality | |
| US20230405475A1 (en) | Shooting method, apparatus, device and medium based on virtual reality space | |
| CN114173109A (en) | Watching user tracking method and device, electronic equipment and storage medium | |
| CN113129358A (en) | Method and system for presenting virtual objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |