TW202311817A - Method, processing device, and display system for information display - Google Patents

Abstract

A method, a processing device, and a system for information display are proposed, and the system includes a light transmissive display. A first information extraction device extracts spatial position information of a user, and a second information extraction device extracts spatial position information of a target object. The processing device performs the following steps. Display position information of virtual information of the target object on the display is determined according to the spatial position information of the user and the spatial position information of the target object. The display position information includes a first display reference position corresponding to a previous time and a second display position corresponding to a current time. An actual display position of the virtual information on the display corresponding to the current time is determined according to the distance between the first display reference position and the second display reference position. The virtual information is displayed on the display according to the actual display position.

Description

Information display method, processing device and information display system

The invention relates to an information display technology.

With the development of image processing technology and spatial positioning technology, the application of transparent displays has gradually attracted attention. This type of technology allows the display to be matched with physical objects, supplemented with virtual related information, and generates an interactive experience according to the user's needs, so that the information can be presented in a more intuitive way.

Furthermore, the virtual information associated with the physical object can be displayed on a specific position of the transparent display, so that the user can watch the physical object and the virtual information superimposed on or on one side of the physical object through the transparent display at the same time. However, when the physical object moves dynamically, the virtual information may deviate from the physical object, so that the virtual information displayed on the transparent display cannot follow the movement of the physical object in real time. In addition, since the movement of physical objects may not be predictable or may be changeable, it may cause shaking or stacking of virtual information displayed based on real-time object recognition, resulting in difficulty and discomfort for viewers to read virtual information.

The disclosure provides an information display method, a processing device thereof, and an information display system.

In an exemplary embodiment of the present disclosure, the above-mentioned information display system includes a light-transmissible display, a first information capture device, a second information capture device, and a processing device, wherein the processing device is connected to the display, the first information capture device fetching device and the second information fetching device. The first information capturing device is used for capturing the user's spatial position information, and the second information capturing device is used for capturing the target object's spatial position information. The processing device is configured to perform the following steps. The display position information of the virtual information of the target on the display is determined according to the spatial position information of the user and the spatial position information of the target. The display location information includes a first display reference location corresponding to a previous time and a second display reference location corresponding to a current time. The actual display position of the virtual information on the display corresponding to the current time is determined according to the distance between the first display reference position and the second display reference position. Display virtual information on the monitor according to the actual display position.

In an exemplary embodiment of the present disclosure, the above information display method is applicable to an information display system having a light-transmissive display, a first information capture device, a second information capture device, and a processing device, and includes the following steps. The spatial position information of the user is retrieved by using the first information retrieval device. The spatial position information of the target is captured by the second information capturing device. The display position information of the virtual information of the target on the display is determined according to the spatial position information of the user and the spatial position information of the target. The display location information includes a first display reference location corresponding to a previous time and a second display reference location corresponding to a current time. The actual display position of the virtual information on the display corresponding to the current time is determined according to the distance between the first display reference position and the second display reference position. Display virtual information on the monitor according to the actual display position.

In an exemplary embodiment of the present disclosure, the above-mentioned processing device is connected to a light-transmittable display, a first information capturing device, and a second information capturing device. The first information capturing device is used for capturing the user's spatial position information, and the second information capturing device is used for capturing the target object's spatial position information. The processing device includes a memory and a processor connected to the memory. The memory is used to store data and the processor is configured to perform the following steps. The spatial position information of the user is retrieved by using the first information retrieval device. The spatial position information of the target is captured by the second information capturing device. The display position information of the virtual information of the target on the display is determined according to the spatial position information of the user and the spatial position information of the target. The display location information includes a first display reference location corresponding to a previous time and a second display reference location corresponding to a current time. The actual display position of the virtual information on the display corresponding to the current time is determined according to the distance between the first display reference position and the second display reference position. Display virtual information on the monitor according to the actual display position.

In order to make the present disclosure more comprehensible, the following specific embodiments are described in detail together with the accompanying drawings.

Part of the exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. For the referenced component symbols in the following description, when the same component symbols appear in different drawings, they will be regarded as the same or similar components. These exemplary embodiments are only part of the present disclosure, and do not disclose all possible implementations of the present disclosure. Rather, these exemplary embodiments are merely examples of methods, apparatuses, and systems within the scope of the present disclosure.

FIG. 1A is a block diagram of an information display system according to an exemplary embodiment of the present disclosure. Firstly, FIG. 1A firstly introduces various components and configuration relationships in the system, and the detailed functions will be disclosed together with the flow charts of subsequent exemplary embodiments.

Please refer to FIG. 1A, the information display system 100 in this exemplary embodiment may include a display 110, a first information capture device 120, a second information capture device 130, and a processing device 140, wherein the processing device 140 may be wireless, wired Or electrically connected to the display 110 , the first information capturing device 120 and the second information capturing device 130 .

The display 110 can be used to display information, which includes, for example, a liquid crystal display (Liquid crystal display, LCD), a field sequential color (Field sequential color) liquid crystal display, a light emitting diode (Light emitting diode, LED) display, an electrowetting display, etc. Transmissive light-transmissive display, or projection-type light-transmissive display.

The first information capturing device 120 is used for capturing the spatial position information of the user, and the second information capturing device 130 is used for capturing the spatial position information of the object in the physical scene. The first information capture device 120 may be, for example, at least one image sensor or at least one image sensor combined with at least one depth sensor to identify and locate the user's image, wherein the image sensor may include a visible light sensor. Sensors or non-visible light sensors such as infrared sensors, etc. In addition, the first information capturing device 120 can be, for example, an optical locator to perform optical spatial positioning of the user. As long as it is a device capable of locating the user's location information or a combination thereof, it all belongs to the category of the first information capturing device 120 .

The second information capture device 120 can be, for example, at least one image sensor or at least one image sensor combined with at least one depth sensor to perform image recognition and positioning of the target object, wherein the image sensor can include a visible light sensor. detector or non-visible light sensor such as infrared sensor, etc. As long as it is a device capable of locating the position information of the target object or a combination thereof, it all belongs to the category of the second information capturing device 130 .

In the embodiment of the present disclosure, the above-mentioned image sensor can be used to capture images and includes a camera lens having a lens and a photosensitive element. The above-mentioned depth sensor can be used to detect depth information, which can be realized by using active depth sensing technology and passive depth sensing technology. Active depth sensing technology can calculate depth information by actively emitting light sources, infrared rays, ultrasonic waves, lasers, etc. as signals with time-of-flight ranging technology. The passive depth sensing technology can use two image sensors to capture two images in front of it from different angles of view, and use the parallax of the two images to calculate depth information.

The processing device 140 is used to control the operation of the information display system 100 , and may include a memory 141 and a processor 142 . The memory 141 can be, for example, any type of fixed or removable random access memory (random access memory, RAM), read-only memory (read-only memory, ROM), flash memory (flash memory), Hard disk or other similar devices, integrated circuits and combinations thereof. The processor 142 can be, for example, a central processing unit (central processing unit, CPU), an application processor (application processor, AP), or other programmable general purpose or special purpose microprocessor (microprocessor), digital signal processing Digital signal processor (DSP), image signal processor (image signal processor, ISP), graphics processing unit (graphics processing unit, GPU) or other similar devices, integrated circuits and combinations thereof. In the following exemplary embodiment, each component of the information display system 100 will be used to illustrate the details of the process of the information display method performed by the processing device 140 .

In this exemplary embodiment, the processing device 140 may be a computer device built into the display 110 or connected to the display 110 . The first information capture device 120 and the second information capture device 130 can be respectively installed on the opposite sides of the field where the information display system 100 belongs to the display 110, etc., to locate the user and the target, and through The respective communication interfaces transmit information to the processing device 140 in a wired or wireless manner. In some embodiments, the first information capturing device 120 and the second information capturing device 130 may each have a processor and a memory, and have computing capabilities for object recognition and object tracking based on image data. In another exemplary embodiment, the information display system 100 can be a single integrated system, which can be implemented as a head-mounted display device, a smart phone, a tablet computer, etc., and the disclosure is not limited here. In the following exemplary embodiments, the implementation details of the information display method will be illustrated with each component of the information display system 100 .

FIG. 1B is a schematic diagram of an information display system according to an exemplary embodiment of the present disclosure. Referring to FIG. 1B , the user U1 and the object Obj1 are located on different sides of the display 110 . The user U1 can watch the physical scene superimposed with the virtual information Vf1 of the object Obj1 through the display 110 . The virtual information Vf1 can be regarded as augmented reality content amplified based on the object Obj1. In some embodiments, the first information capture device 120 may include a first image sensor for photographing the user U1, and the second information capture device 130 may include a second image sensor for photographing the target Obj1 device. The first information capturing device 120 can obtain the spatial position information of the user U1 based on the captured user image, and the second information capturing device 130 can obtain the spatial position information of the object Obj1 based on the captured object image. .

The processing device 140 can determine the display position information of the virtual information Vf1 on the display 110 according to the spatial position information of the user U1 and the spatial position information of the object Obj1. In the example shown in FIG. 1B , the above display location information is implemented as a reference display object frame RF1 on the display plane. In other embodiments, the display location information can be implemented as a point on the display plane of the display 110 . Specifically, the display position information can be regarded as the point or area where the user's line of sight is projected on the display plane when viewing the object Obj1. Thereby, the processing device 140 can display the virtual information Vf1 by using the display position information (for example, referring to the display object frame RF1 ) as a reference. More specifically, based on various needs or different applications, the processing device 140 can determine the actual display position of the virtual information Vf1 according to the display position information, so that the user U1 can see the virtual information Vf1 near the object Obj1 through the display 110 Or see the virtual information Vf1 superimposed on the object Obj1.

It is worth mentioning that, in some embodiments, both the user U1 and/or the object Obj1 may move dynamically. For example, the target object Obj1 can be a fish kept in an aquarium, and the virtual information Vf1 can be the species name or introduction information of the fish. Therefore, the first information capture device 120 can continuously shoot the user U1 to generate a video sequence including a plurality of user images corresponding to different time points, and the first information capture device 120 can use the The spatial location information of the user U1 is tracked by a video sequence of the user's images. Similarly, the second information capture device 130 can continuously shoot the target object Obj1 to generate a video sequence including a plurality of target object images respectively corresponding to different time points. The video sequence of the object image is used to track the spatial position information of the object Obj1. Since the processing device 140 can continuously update the display position information of the virtual information Vf1 based on the tracking results of the object Obj1 and the user U1, the virtual information Vf1 displayed on the display 110 can reflect the movement of the user U1 and/or the object Obj1. The corresponding movement of the virtual information Vf1 to achieve the visual effect of following the target Obj1.

FIG. 2 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to FIG. 1A, FIG. 1B and FIG. 2 at the same time, and the method flow in FIG. to fulfill. Here, the user U1 can view the target object Obj1 and its virtual information Vf1 through the display 110 of the information display system 100 .

In step S210, the information display system 100 utilizes the first information capturing device 120 to capture the spatial location information of the user U1. In step S220 , the information display system 100 uses the second information capturing device 130 to capture the spatial position information of the object Obj1 . As mentioned above, the first information capturing device 120 and the second information capturing device 130 are, for example, an image sensor, a depth sensor or a combination thereof that can locate the positions of the user U1 and the object Obj1.

In step S230 , the processing device 140 of the information display system 100 determines the display position information of the virtual information Vf1 of the object Obj1 on the display 110 according to the spatial position information of the user U1 and the spatial position information of the object Obj1 . The display location information includes a first display reference location corresponding to a previous time and a second display reference location corresponding to a current time. Specifically, the processing device 140 can continuously calculate a plurality of display reference positions respectively corresponding to a plurality of consecutive times according to the spatial position information of the user U1 and the spatial position information of the object Obj1 . In other words, the first information capturing device 120 and the second information capturing device 130 can continuously update the spatial position information of the object Obj1 and the user U1, and the processing device 140 can continuously update the display position information of the virtual information Vf1 accordingly.

In step S240, the processing device 140 of the information display system 100 determines the actual display position of the virtual information Vf1 corresponding to the current time on the display 110 according to the distance between the first display reference position and the second display reference position. Next, in step S250 , the processing device 140 of the information display system 100 displays the virtual information Vf1 on the display 110 according to the actual display position.

That is to say, when the processing device 140 determines that the virtual information Vf1 corresponds to the actual display position at the current time, the processing device 140 will calculate the distance between the first display reference position at the previous time and the second display reference position at the current time . Next, the processing device 140 decides to use the second display reference position at the current time to update the actual display position of the virtual information Vf1 on the display 110 according to the above distance, or decides not to update the actual display position of the virtual information Vf1.

FIG. 3 is a flow chart of determining an actual display position corresponding to the current time according to an exemplary embodiment of the present disclosure. Referring to FIG. 3 , step S240 in FIG. 2 may be implemented as steps S241 to S243 . In step S241 , the processing device 140 determines whether the distance between the first display reference position corresponding to the previous time and the second display reference position corresponding to the current time is greater than a preset threshold.

If the distance between the first displayed reference position and the second displayed reference position is greater than a preset threshold, it means that the displacement of the object Obj1 seen by the user U1 is obvious. Therefore, in response to the fact that the distance between the first display reference position and the second display reference position is greater than the preset threshold (step S241 judges yes), in step S242, the processing device 140 uses the second display reference position corresponding to the current time Updating the actual display position of the virtual information Vf1 corresponding to the current time on the display 110 .

On the other hand, if the distance between the first displayed reference position and the second displayed reference position is not greater than the preset threshold, it means that the displacement of the object Obj1 seen by the user U1 is relatively small. Therefore, in response to the fact that the distance between the first display reference position and the second display reference position is not greater than the preset threshold (step S241 judged as NO), in step S243, the processing device 140 does not update the virtual information Vf1 on the display 110 to correspond to The actual display position at the current time, that is, the actual display position corresponding to the current time will be the same as the actual display position corresponding to the previous time. In one embodiment, the virtual information Vf1 can be displayed according to the actual display position determined based on the first display reference position at a previous time. That is to say, if the distance between the first display reference position and the second display reference position is not greater than the preset threshold, the display position of the virtual information Vf1 on the display 110 may not be changed. In this way, the actual display position of the virtual information Vf1 will not change due to the slight shaking or movement of the object Obj1, which greatly improves the comfort of the user U1 viewing the virtual information Vf1. That is to say, the virtual information Vf1 can be displayed on the display 110 stably without shaking or stacking of afterimages.

For the sake of convenience and clarity, the following examples will be combined with the display system 100 to illustrate the implementation of calculating and displaying position information in the present disclosure.

FIG. 4 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to FIG. 4 , in step S402 , the first information capture device 120 captures images of users, where the number of users may be more than one. In step S404, the first information capture device 120 performs face detection on multiple user images to obtain the face positions in each user image. The first information capturing device 120 can detect face blocks and facial features in the face blocks from each user image through various face detection technologies and face feature detection technologies. The face detection technology is, for example, a traditional face detection algorithm using pixel value analysis or a deep learning algorithm using machine learning, etc., which is not limited in this disclosure. The aforementioned facial features are, for example, both eyes. In some embodiments, the first information capturing device 120 may use the center point of the face block as the face position or the center point of both eyes as the face position, but this disclosure is not limited thereto.

In step S406, the first information capturing device 120 performs coordinate conversion according to the position of the face to obtain the spatial position information of the user U1. Specifically, the face position acquired based on the user image is a two-dimensional pixel coordinate. The first information capturing device 120 can perform coordinate conversion according to the depth information of the user U1 and the face position in the user's image, so as to generate the spatial position information of the user U1. In the embodiment of FIG. 4 , the spatial position information of the user U1 may include the user's camera coordinates in the camera coordinate system.

公式(1) 於一實施例中，相對於顯示器110的使用者位置座標(t _x1, t _y1, t _z1)所屬的參考座標系的三座標軸分別為顯示器110的顯示X軸、顯示Y軸以及顯示平面法線。拍攝俯仰角θ1為顯示器110的顯示平面法線與第一影像感測器之鏡頭光軸CX1之間的夾角。 Next, in step S408 , the processing device 140 converts the user camera coordinates into user position coordinates relative to the display 110 according to the shooting direction of the first image sensor of the first information capturing device 120 . In one embodiment, the shooting direction of the first image sensor may include a shooting elevation angle in the vertical direction. For example, FIG. 5A is a schematic diagram illustrating shooting directions of the first image sensor and the second image sensor according to an exemplary embodiment of the present disclosure. Referring to FIG. 5A , the first image sensor for photographing the user U1 has a photographing elevation angle θ1 . The shooting pitch angle θ1 is the angle between the lens optical axis CX1 of the first image sensor and the display plane normal HX1 of the display 110 . In one embodiment, assuming that the user camera coordinates are (cx ₁ , cy ₁ , cz ₁ ), the processing device 140 can transform the user camera coordinates (cx ₁ , cy ₁ , cz ₁ ) according to the following formula (1): are the user position coordinates (t _x1 , _ty1 , t _z1 ) relative to the display 110 .

Formula (1) In one embodiment, the three coordinate axes of the reference coordinate system to which the user position coordinates (t _x1 , _ty1 , t _z1 ) of the display 110 belong are respectively the display X axis, the display Y axis and the display 110 of the display 110. Displays plane normals. The shooting pitch angle θ1 is the angle between the normal of the display plane of the display 110 and the lens optical axis CX1 of the first image sensor.

On the other hand, in step S410, the second information capturing device 130 captures the image of the target object, wherein the number of the target object may be more than one. In step S412 , the second information capturing device 130 performs object detection on multiple target images to obtain object bounding boxes of the target in each target image. The second information capturing device 130 can utilize a convolutional neural network (Convolution Neural Network, CNN) model in a deep learning algorithm to perform object detection. The above-mentioned convolutional neural network model is for example used for object detection R-CNN, Fast R-CNN, Faster R-CNN, YOLO or SSD, etc., but this disclosure is not limited thereto. The object detection model used by the second information acquisition device 130 can output the position and size of the object's bounding box (Bonding box) of the object Obj1 and the classification result of the object Obj1 . In one embodiment, the processing device 140 can select the corresponding virtual information Vf1 from the database according to the classification result of the object Obj1 for display.

In step S414, the second information capturing device 130 performs coordinate conversion according to the object bounding box to obtain the spatial position information of the object Obj1. Specifically, the position of the object bounding box acquired based on the target image belongs to two-dimensional pixel coordinates, and the reference point of the above object bounding box can be the vertex position, center point position or frame boundary of the object bounding box. not limited. The second information capturing device 130 can perform coordinate conversion according to the depth information of the object Obj1 and the object bounding box in the image of the object, so as to generate the spatial position information of the object Obj1. In the embodiment shown in FIG. 4 , the spatial position information of the object Obj1 includes the camera coordinates of the object in the camera coordinate system.

公式(2) 於一實施例中，相對於顯示器110的目標物位置座標(t _x2, t _y2, t _z2)所屬的參考座標系相同於使用者位置座標(t _x1, t _y1, t _z1)所屬的參考座標系。拍攝俯仰角θ2為顯示器110的顯示平面法線與第二影像感測器之鏡頭光軸CX2之間的夾角。 Next, in step S416 , the processing device 140 converts the object camera coordinates into object position coordinates relative to the display 110 according to the shooting direction of the second image sensor. In one embodiment, the shooting direction of the second image sensor may include a shooting pitch angle in the vertical direction. For example, please refer to FIG. 5A again, the second image sensor for photographing the object Obj1 has a photographing pitch angle θ2. The shooting pitch angle θ2 is the angle between the optical axis CX2 of the lens of the second image sensor and the normal line HX1 of the display plane. Assuming that the object camera coordinates are (cx ₂ , cy ₂ , cz ₂ ), the processing device 140 can convert the object camera coordinates (cx ₂ , cy ₂ , cz ₂ ) into relative to the display 110 according to the following formula (2): Target position coordinates (t _x2 , _ty2 , t _z2 ).

Equation (2) In one embodiment, the reference coordinate system to which the target position coordinates (t _x2 , _ty2 , t _z2 ) relative to the display 110 belongs is the same as the user position coordinates (t _x1 , _ty1 , t _z1 ) The reference coordinate system to which it belongs. The shooting pitch angle θ2 is the angle between the normal of the display plane of the display 110 and the optical axis CX2 of the lens of the second image sensor.

In some embodiments, the shooting directions of the first image sensor and the second image sensor can be obtained through inertial sensors disposed on the first image sensor and the second image sensor. Taking the example of FIG. 5A as an example, the sensing value of the inertial sensor can be used to calculate the shooting pitch angles θ1 and θ2 in FIG. 5A . For example, the above-mentioned inertial sensor is, for example, an acceleration sensor or the like.

公式(3) 之後，處理裝置140可根據顯示器110的尺寸與解析度獲取顯示位置資訊，而顯示位置資訊為顯示器110的畫素座標。更詳細來說，處理裝置140可根據顯示器110的尺寸與解析度將交點位置IP1的空間座標轉換為顯示器110上的畫素座標，以獲取虛擬資訊Vf1於顯示器110上的顯示位置資訊，亦即由畫素座標來表示的交點位置IP1。 In step S418 , the processing device 140 determines the display position information of the virtual information Vf1 on the display 110 according to the connection line between the target position coordinates and the user position coordinates. FIG. 5B is a schematic diagram of determining to display location information according to an exemplary embodiment of the present disclosure. Please refer to FIG. 5B , the displayed position information may include the display on the display 110 where the connection line between the target position coordinates (t _x2 , _ty2 , t _z2 ) and the user position coordinates (t _x1 , t _y1 , t _z1 ) intersects. The intersection point of the plane is IP1. The user position coordinates (t _x1 , t _y1 , t _z1 ) and target object position coordinates (t _x2 , t _y2 , t _z2 ) calculated according to formula (1) and formula (2) are located on the display plane (ie Z= The spatial coordinates (X, Y, Z) of the intersection point IP1 on 0) can be calculated according to the following formula (3).

After the formula (3), the processing device 140 can obtain display position information according to the size and resolution of the display 110 , and the display position information is the pixel coordinates of the display 110 . More specifically, the processing device 140 can convert the spatial coordinates of the intersection point IP1 into pixel coordinates on the display 110 according to the size and resolution of the display 110, so as to obtain the display position information of the virtual information Vf1 on the display 110, that is, The intersection position IP1 represented by pixel coordinates.

As mentioned above, the display location information includes a first display reference location corresponding to a previous time and a second display reference location corresponding to a current time. In step S420 , the processing device 140 determines the actual display position of the virtual information Vf1 corresponding to the current time on the display 110 according to the distance between the first display reference position and the second display reference position. In step S422, the processing device 140 displays the virtual information Vf1 on the display 110 according to the actual display position.

In detail, FIG. 6A is a schematic diagram of not updating the actual display position according to an exemplary embodiment of the present disclosure. FIG. 6B is a schematic diagram of updating the actual display position according to an exemplary embodiment of the present disclosure. Referring to FIGS. 6A and 6B , in this embodiment, the display position information includes a first display reference position IP1_p corresponding to a previous time and a second display reference position IP1_c corresponding to a current time. The first display reference position IP1_p and the second display reference position IP1_c may be the intersection positions determined according to the calculation method in the embodiment of FIG. 4 . More specifically, based on the previous user image and the previous object image corresponding to the previous time, the processing device 140 can obtain the first display reference position IP1_p on the display plane according to the connection line between the two spatial coordinates. Then, for example, the processing device 140 can set the actual display position corresponding to the previous time as the position M pixels above the first display reference position IP1_p, and the M value can be adjusted according to actual needs, for example, referring to the user's field of view, Make the actual display position fall within the user's field of vision. Afterwards, based on the current user image and the current object image corresponding to the current time, the processing device 140 can obtain the second display reference position IP1_c on the display plane according to the connection line between the two spatial coordinates. The processing device 140 determines whether the distance Δd1 between the first display reference position IP1_p and the second display reference position IP1_c is greater than a preset threshold.

If the distance Δd1 is not greater than the preset threshold, please refer to FIG. 6A , the processing device 140 may not update the actual display position of the virtual information Vf1 corresponding to the current time on the display 110, so that the virtual information Vf1 will remain displayed at the first display reference position. The position of M pixels directly above IP1_p. Conversely, if the distance Δd1 is greater than the preset threshold, please refer to FIG. 6B , the processing device 140 updates the actual display position of the virtual information Vf1 corresponding to the current time on the display 110 according to the second display reference position IP1_c corresponding to the current time, as shown in FIG. Yes, the processing device 140 may set the actual display position corresponding to the current time as the position M pixels above the second display reference position IP1_c.

FIG. 7 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to FIG. 7 , in step S702 , the first information capture device 120 captures images of users, where the number of users may be more than one. In step S704, the first information capture device 120 performs face detection on multiple user images to obtain the face positions in each user image. The details of step S702 and step S704 can refer to similar related descriptions of step S402 and step S404 in FIG. 4 , which will not be repeated here.

In step S706, the first information capturing device 120 performs coordinate transformation according to the face position to obtain the spatial position information of the user U1. Here, the first information capturing device 120 can convert the face position into pixel coordinates corresponding to the default image resolution. In the embodiment of FIG. 7 , the spatial position information of the user U1 may include a horizontal offset angle and a vertical offset angle of the user U1 relative to the first image sensor. Specifically, according to the two-dimensional pixel coordinates of the face position and the horizontal field of view (Field of View, FOV) of the first image sensor, the first information capture device 120 can calculate the relative distance between the user U1 and the optical axis of the lens. the horizontal offset angle. According to the two-dimensional pixel coordinates of the face position and the vertical FOV of the first image sensor, the first information capturing device 120 can calculate the vertical offset angle of the user U1 relative to the optical axis of the lens.

In step S708, the second information capturing device 130 captures the target image. In step S710 , the second information capturing device 130 performs object detection on multiple target images to obtain the object bounding box of the target in each target image. The details of step S708 and step S710 can refer to similar related descriptions of step S410 and step S412 in FIG. 4 , and will not be repeated here.

In step S712, the second information capturing device 130 performs coordinate transformation according to the object bounding box to obtain the spatial position information of the object Obj1. Here, the second information capturing device 130 can convert the object bounding box into pixel coordinates corresponding to the default image resolution. In the embodiment of FIG. 7 , the spatial position information of the object Obj1 may include a horizontal offset angle and a vertical offset angle of the object Obj1 relative to the first image sensor. Specifically, according to the pixel coordinates of the vertical frame boundary of the object's bounding box in the target image and the horizontal FOV of the first image sensor, the first information capturing device 120 can calculate the position of the target Obj1 relative to the optical axis of the lens. Horizontal offset angle, that is, the horizontal offset angle of the left and right borders of the object's bounding box relative to the optical axis of the lens. According to the pixel coordinates of the horizontal border of the object's bounding box in the target image and the vertical FOV of the first image sensor, the first information capture device 120 can calculate the vertical offset angle of the target Obj1 relative to the optical axis of the lens , that is, the vertical offset angles of the upper and lower borders of the object's bounding box relative to the optical axis of the lens.

In step S714, the processing device 140 according to the horizontal offset angle and the vertical offset angle of the target object Obj1 relative to the second image sensor, the horizontal offset angle and the vertical offset angle of the user U1 relative to the first image sensor The angle, the shooting direction of the first image sensor and the shooting direction of the second image sensor determine the display position information of the virtual information Vf1 on the display 110 .

In detail, FIG. 8A and FIG. 8B are schematic diagrams of determining display location information according to an exemplary embodiment of the present disclosure. In one embodiment, the relative positional relationship among the first image sensor, the second image sensor and the display 110 can be defined in advance. For example, if the center point of the upper edge of the display 110 is taken as the origin and the display plane is the XY plane, the position of the lens of the first image sensor and the position of the lens of the second image sensor can be determined according to the actual installation. position is defined.

Please refer to FIG. 8A first, and the following will take the left boundary BL of the object bounding box Roi1 of the object Obj1 as an example for illustration. The first information capturing device 120 can provide the processing device 140 with the horizontal offset angle θ3 of the user U1 relative to the first image sensor and the depth information d1 of the user U1 from the first image sensor. The second information capturing device 130 can provide the processing device 140 with the horizontal offset angle θ4 of the object Obj1 relative to the second image sensor and the depth information d2 of the object Obj1 from the second image sensor.

Therefore, the processing device 140 can obtain the angle θ5 (θ5=180−θ3−θ4), and calculate the distance d3 between the user U1 and the left boundary BL according to the depth information d1 and depth information d2 based on the law of cosines. Then, the processing device 140 can calculate the angle θ6 according to the angle θ5 , the depth information d1 , the depth information d2 and the distance d3 based on the law of sine, and calculate the angle θ7 according to the angle θ6 and the horizontal offset angle θ3 . Therefore, the processing device 140 can calculate the lateral offset distance between the user U1 and the central line L1 of the display 110 as d1*sinθ3, and calculate the lateral offset distance between the left border BL and the central line L1 of the display 110 as d2 *sin θ4. In addition, the processing device 140 can also calculate the lateral offset distance Qx1 of the intersection point Q of the left border BL on the display plane equal to d1*sinθ3+d1*cosθ3*tanθ7. Similarly, the lateral offset distance of the intersection point Q of the right boundary of the object bounding box Roi1 can also be calculated in a similar manner.

Next, please refer to FIG. 8B , and the following will take the upper boundary UL of the object bounding box Roi1 of the object Obj1 as an example for illustration. The first information capturing device 120 can provide the shooting elevation angle θ1 of the first image sensor and the vertical offset angle θ9 of the user U1 relative to the first image sensor to the processing device 140 . The second information capturing device 130 can provide the shooting elevation angle θ2 of the second image sensor and the vertical offset angle θ8 of the object Obj1 relative to the second image sensor to the processing device 140 .

公式(4) 其中，使用者U1與顯示器110的上緣之間的實際距離A1可表示為

，

1為第一影像感測器之鏡頭與顯示器110的上緣之間的距離。目標物Obj與顯示器110的上緣之間的實際距離A2可表示為

，

為第二影像感測器之鏡頭與顯示器110的上緣之間的距離。同理，物件邊界框Roi1下邊界的交點的直向偏移距離亦可以類似的方式計算出。 Therefore, according to the depth information d2, the shooting pitch angle θ2 and the vertical offset angle θ8, the processing device 140 can calculate the distance between the upper boundary UL and the upper edge of the display 110 as d2sin(θ2+θ8). According to the depth information d1, the shooting pitch angle θ1 and the vertical offset angle θ9, the processing device 140 can calculate the distance between the user U1 and the upper edge of the display 110 as d1sin(θ1+θ9). In one embodiment, the processing device 140 can further correct and calculate the aforementioned calculation result according to the distance between the lenses of the two image sensors and the upper edge of the display 110 , so as to obtain the distance between the user U1, the object Obj and the display 110 The actual distance between the upper edges. Afterwards, the processing device 140 can calculate the vertical offset distance Eu1 of the intersection point Q of the upper boundary UL on the display plane based on the following formula (4).

Formula (4) where, the actual distance A1 between the user U1 and the upper edge of the display 110 can be expressed as

,

1 is the distance between the lens of the first image sensor and the upper edge of the display 110 . The actual distance A2 between the object Obj and the upper edge of the display 110 can be expressed as

,

is the distance between the lens of the second image sensor and the upper edge of the display 110 . Similarly, the vertical offset distance of the intersection point of the lower boundary of the object bounding box Roi1 can also be calculated in a similar manner.

Based on the above description, the processing device 140 may acquire a reference display object box corresponding to the object bounding box Roi1 on the display plane. In detail, based on the calculation method shown in FIG. 8A , the processing device 140 can obtain the lateral offset distances between the left and right borders of the reference display object frame on the display plane and the center line L1 of the display 110 . Based on the calculation method shown in FIG. 8B , the processing device 140 can obtain the vertical offset distances between the upper and lower boundaries of the reference display object frame on the display plane and the upper edge of the display 110 respectively. In other words, the reference display object box corresponding to the object bounding box Roi1 on the display plane can be represented by two straight line equations parallel to the display X axis and two straight line equations parallel to the display Y axis. Among them, the four frame vertices of the reference display object frame can be obtained from the intersection points of these straight line equations.

Afterwards, the processing device 140 can obtain display position information according to the size and resolution of the display 110 , and the display position information can be the pixel coordinates of the display 110 . In more detail, the processing device 140 can convert the reference display object frame into pixel coordinates on the display 110 according to the size and resolution of the display 110 to obtain display position information of the virtual information Vf1 on the display 110 . That is, the reference display object frame on the display plane is represented by the pixel coordinate system.

Returning to FIG. 7, in step S716, the processing device 140 compares the distance between the first reference display object frame corresponding to the previous time and the second reference display object frame corresponding to the current time with a preset threshold value to determine the virtual information Vf1 corresponds to the actual display position of the current time on the display 110 . In step S718 , the processing device 140 displays the virtual information Vf1 on the display 110 according to the actual display position.

In detail, FIG. 9A is a schematic diagram of not updating the actual display position according to an exemplary embodiment of the present disclosure. FIG. 9B is a schematic diagram of updating the actual display position according to an exemplary embodiment of the present disclosure. Referring to FIGS. 9A and 9B , in this embodiment, the display location information is included in the reference display object box corresponding to the object bounding box in the target image. The reference display object frame may include a first reference display object frame Roi_p corresponding to a previous time and a second reference display object frame Roi_c corresponding to a current time. The first display reference position includes a first reference display object frame Roi_p, and the second display reference position includes a second reference display object frame Roi_c.

The first reference display object frame Roi_p corresponding to the previous time and the second reference display object frame Roi_c corresponding to the current time may be determined according to the calculation methods shown in the embodiments shown in FIG. 7 , FIG. 8A and FIG. 8B . More specifically, based on the previous user image and the previous object image corresponding to the previous time, the processing device 140 may acquire the first reference display object frame Roi_p on the display plane. Next, for example, assuming that the pixel coordinates of the upper left vertex of the first reference display object frame Roi_p are (x1, y1), the processing device 140 can set the actual display position corresponding to the previous time as the pixel coordinates of (x1+ αw1, y1+βh1). Wherein, w1 is the width of the first reference display object frame Roi_p, and h1 is the height of the first reference display object frame Roi_p. α and β can be set according to the actual situation, which is not limited in this disclosure.

Afterwards, based on the current user image and the current object image corresponding to the current time, the processing device 140 may acquire the second reference display object frame Roi_c on the display plane. The processing device 140 compares the distance Δd2 between the first reference display object frame Roi_p corresponding to the previous time and the second reference display object frame Roi_c corresponding to the current time with a preset threshold.

The processing device 140 calculates the distance between the first reference point of the first reference display object frame Roi_p and the second reference point of the second reference display object frame Roi_c. The first reference point and the second reference point may be, for example, a frame center point or a frame vertex. In the examples of FIGS. 9A and 9B , the first reference point and the second reference point are respectively the upper left vertex of the first reference display object frame Roi_p and the upper left vertex of the second reference display object frame Roi_c.

In an embodiment, the preset threshold may be determined according to the size of the second reference display object frame Roi_c. For example, the preset threshold may be equal to min(γ*w2, γ*h2). Wherein, w2 is the width of the second reference display object frame Roi_c, and h2 is the height of the second reference display object frame Roi_c. γ can be a value between 0 and 1, which can be set according to the actual situation, which is not limited in the present disclosure. min(․) represents the function of taking the minimum value. It can be seen that the preset threshold can be flexibly set in response to the size of the reference object frame. The preset critical value may be directly proportional to the size of the reference object frame.

If the distance Δd2 is not greater than the preset threshold, please refer to FIG. 9A , the processing device 140 does not update the actual display position of the virtual information Vf1 on the display 110 corresponding to the current time, and the virtual information Vf1 can remain displayed in the first reference display object frame. Above Roi_p. Conversely, if the distance Δd2 is greater than the preset threshold, please refer to FIG. 9B , the processing device 140 updates the actual display position of the virtual information Vf1 corresponding to the current time on the display 110 according to the second reference display object frame Roi_c corresponding to the current time. For example, assuming that the pixel coordinates of the upper left vertex of the second reference display object frame Roi_c are (x2, y2), the processing device 140 may set the actual display position corresponding to the current time as the pixel coordinates of (x2+αw2, y2+βh2). Wherein, w2 is the width of the second reference display object frame Roi_c, and h2 is the height of the second reference display object frame Roi_c. α and β can be set according to the actual situation, which is not limited in this disclosure.

In one embodiment, the processing device 140 adjusts the display position information in a direction away from the edge of the display 110 in response to the fact that the display position information is located in the display edge area of the display 110 . In detail, after the processing device 140 obtains the display position information at a certain time point, if the display position information is located in the display edge area of the display 110, it means that the virtual information may be displayed in the display edge area of the display 110 or even displayed incompletely. , resulting in difficulty for the user U1 to watch. Therefore, the processing device 140 will further adjust the display position information, for example, the processing device 140 can adjust the display position information in a direction away from the edge of the display 110 .

For example, when the intersection position calculated based on the calculation method in FIG. 5B is located in the display edge area of the display 110, the processing device 140 can adjust the intersection position away from the edge of the display 110, and then according to the adjusted intersection position Determines the actual display position. Alternatively, when the reference display object frame calculated based on the calculation methods in FIGS. 8A and 8B is located in the display edge area of the display 110, the processing device 140 may adjust the reference display object frame to a direction away from the edge of the display 110, and then adjust The following reference display object box determines the actual display position.

公式(5) 其中，

為顯示位置資訊的原始Y座標、F為調整權重、c為常數。F例如是介於1至100之間的數值，其可視實際而設置，本揭露對此不限制。常數c可根據原始Y座標所在顯示區域而決定。舉例而言，假設顯示位置資訊的原始Y座標位於顯示範圍的中心區域，則常數c可設置為等於原始Y座標，即不調整原始Y座標。假設顯示位置資訊的原始Y座標位於顯示範圍的邊緣區域，則常數c可設置為不等於原始Y座標，即往遠離顯示器110之邊緣的方向調整原始Y座標。同理，顯示位置資訊於水平軸向上的座標分量亦可以類似的方式進行調整。 In one embodiment, the processing device 140 can adjust the coordinate component of the display position information on the vertical axis according to the following formula (5).

Formula (5) where,

is the original Y coordinate of the displayed location information, F is the adjustment weight, and c is a constant. F is, for example, a value between 1 and 100, which can be set according to actual conditions, and the present disclosure is not limited thereto. The constant c can be determined according to the display area where the original Y coordinate is located. For example, assuming that the original Y coordinate of the displayed position information is located in the central area of the display range, the constant c can be set equal to the original Y coordinate, that is, the original Y coordinate is not adjusted. Assuming that the original Y coordinate of the display location information is located at the edge of the display range, the constant c can be set to be different from the original Y coordinate, that is, the original Y coordinate is adjusted away from the edge of the display 110 . Similarly, the coordinate components of the displayed location information on the horizontal axis can also be adjusted in a similar manner.

In some embodiments, the processing device 140 can also determine the display position information of the virtual information on the display 110 according to the gesture information of the user. The first information capturing device 120 can be used to capture the gesture information of the user. Specifically, the first information capturing device 120 can recognize the gestures of the user's limbs, torso and/or head through various human gesture recognition technologies. For example, the first information capturing device 120 can identify the human body skeleton or human body feature points according to the image data, so as to identify the posture of the user. In some embodiments, the first information capturing device 120 can perform face detection on user images to obtain the face positions and facial feature points in each user image. The present invention does not limit the number of facial feature points, which may include, for example, mouth corner feature points, eyebrow center feature points, eye corner feature points, and the like. In some embodiments, the first information capturing device 120 or the processing device 140 can identify the displacement of the user's face center and the head rotation angle according to the facial feature points in multiple frames of user images. The posture information of the user includes the displacement of the center of the face and the rotation angle of the head.

In some embodiments, the processing device 140 can determine the head movement or rotation of the user according to the posture information of the user. In response to determining the user's head movement based on the user's posture information, the processing device 140 can determine the display position information of the virtual information of the target on the display 110 according to the user's spatial position information and the target object's spatial position information. That is, when the processing device 140 determines that the user's head moves, the processing device 140 may determine the actual display position of the virtual information according to the flow shown in FIG. 2 , FIG. 4 or FIG. 7 . On the other hand, in response to determining the user's head rotation based on the user's posture information, the processing device 140 can determine the virtual information of the target object on the display 110 according to the user's spatial position information and posture information and the target object's spatial position information. Show location information on . When the processing device 140 determines that the user's head is turning, the processing device 140 may update the display position information of the virtual information of the target on the display 110 according to the rotation angle of the user's head. For example, when the user's head does not move, the processing device 140 can use the user's head rotation angle to update the display position information determined at the previous time point.

In some embodiments, the processing device 140 can estimate a head behavior index according to the face center displacement in the pose information. The above-mentioned head behavior index may be the face center displacement divided by the head radius reference value. For example, the base radius of the head may be 80mm. By comparing the head behavior index and the variation of the head rotation angle, the processing device 140 can determine the movement or rotation of the user's head, and determine the virtual information on the display 110 in response to the movement or rotation of the user's head. Show location information for .

In some embodiments, if the head behavior index is less than or equal to the variation of the head rotation angle, the processing device 140 determines that the user's head moves, that is, the user's head moves from the first spatial position to the second spatial position. Therefore, the process shown in FIG. 2 , FIG. 4 or FIG. 7 can be executed to determine the actual display position of the virtual information. If the head behavior index is greater than the variation of the head rotation angle, the processing device 140 determines that the user's head is rotating, that is, the direction of the user's head changes. Therefore, the processing device 140 can determine the display position information of the virtual information of the target on the display 110 according to the spatial position information of the user, the head rotation angle and the spatial position information of the target object. For example, the processing device 140 can use the rotation angle of the user's head to update the display position information determined at the previous time point, and determine the actual display position of the virtual information accordingly. Alternatively, the processing device 140 may use the user's head rotation angle and the user's spatial position information to identify another target, so as to determine another target according to the spatial position information of the other target and the user's spatial position information. The location information of another virtual information is displayed on the display 110 .

The information display method and its processing device and information display system proposed in the exemplary embodiments of the present disclosure can calculate the display position information for projecting the line of sight on the display plane according to the user position and the target object position in real time, and generate The actual display position of the virtual information. In this way, even if the movement of the target is variable or unpredictable, the virtual information displayed according to the actual display position can follow the movement of the target in real time. In addition, the exemplary embodiment of the present disclosure may determine whether to update the virtual information on the display at the actual display position corresponding to the current time according to the distance between the first display reference position at the previous time and the second display reference position at the current time. Thereby, the virtual information displayed based on real-time object recognition can be displayed stably and smoothly, greatly improving the user's viewing experience.

Although the present disclosure has been disclosed above with exemplary embodiments, it is not intended to limit the present disclosure. Anyone with ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of this disclosure should be defined by the appended scope of patent application and its equivalent scope.

100:Information display system 110: Display 120: The first information retrieval device 130: the second information retrieval device 140: processing device 141: memory 142: Processor Vf1: Virtual Information RF1: Reference display object frame U1: User Obj1: target object CX1, CX2: Optical axis of lens HX1: display plane normal IP1: Intersection position IP1_p: first display reference position IP1_c: Second display reference position BL: left border UL: upper boundary Q: Intersection L1: midline Roi1: object bounding box Roi_p: first reference display object box Roi_c: Second reference display object frame S210～S250, S241～S243, S402～S422, S702～S718: steps

FIG. 1A is a block diagram of an information display system according to an exemplary embodiment of the present disclosure. FIG. 1B is a schematic diagram of an information display system according to an exemplary embodiment of the present disclosure. FIG. 2 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 3 is a flow chart of determining an actual display position corresponding to the current time according to an exemplary embodiment of the present disclosure. FIG. 4 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 5A is a schematic diagram illustrating shooting directions of the first image sensor and the second image sensor according to an exemplary embodiment of the present disclosure. FIG. 5B is a schematic diagram of determining to display location information according to an exemplary embodiment of the present disclosure. FIG. 6A is a schematic diagram of not updating the actual display position according to an exemplary embodiment of the present disclosure. FIG. 6B is a schematic diagram of updating the actual display position according to an exemplary embodiment of the present disclosure. FIG. 7 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 8A and FIG. 8B are schematic diagrams of determining display location information according to an exemplary embodiment of the present disclosure. FIG. 9A is a schematic diagram of not updating the actual display position according to an exemplary embodiment of the present disclosure. FIG. 9B is a schematic diagram of updating the actual display position according to an exemplary embodiment of the present disclosure.

S210~S250: steps

Claims (20)

An information display system, comprising: Light-transmitting displays; a first information retrieval device, configured to retrieve spatial location information of a user; The second information capturing device is used for capturing spatial position information of a target; a processing device connected to the display, the first information capture device and the second information capture device, configured to: Determining display position information of virtual information of the target on the display according to the spatial position information of the user and the spatial position information of the target, wherein the display position information includes the first time corresponding to the previous time a display reference position and a second display reference position corresponding to the current time; determining an actual display position of the virtual information corresponding to the current time on the display according to a distance between the first display reference position and the second display reference position; and and displaying the virtual information on the display according to the actual display position. The information display system as claimed in claim 1, wherein the processing device is configured to: In response to a distance between the first display reference position and the second display reference position being greater than a predetermined threshold, updating the virtual information at the second display reference position corresponding to the current time said actual display position on said display corresponding to said current time; and In response to the fact that the distance between the first display reference position and the second display reference position is not greater than a predetermined threshold, not updating the virtual information on the display corresponding to the actual time at the current time Show location. The information display system as described in claim 1, wherein the first information capturing device is used to capture the user's posture information, and the processing device is further configured to: determining head movement or rotation of the user based on posture information of the user; and In response to determining the user's head movement according to the user's posture information, determining the virtual information of the target object based on the user's spatial position information and the target object's spatial position information The display position information on the display, or in response to determining the user’s head rotation based on the user’s posture information, according to the user’s spatial position information and posture information and the space of the target The position information determines the display position information of the virtual information of the target on the display. The information display system according to claim 1, wherein the first information capture device includes a first image sensor for photographing the user, and the second information capture device includes a sensor for photographing the user The second image sensor of the target object. The information display system as described in claim 4, wherein the second information capturing device performs object detection on a plurality of target images to obtain the object bounding box of the target in each of the target images, and The second information acquisition device performs coordinate conversion according to the object bounding box to obtain the spatial position information of the target object, Wherein the first information capturing device performs face detection on a plurality of user images to obtain the face position in each of the user images, and the first information capturing device performs face detection according to the face position coordinate conversion to obtain the spatial location information of the user. The information display system as described in claim 4, wherein the spatial position information of the target includes the target camera coordinates in the camera coordinate system, and the user spatial position information includes the user camera coordinates in the camera coordinate system, Wherein the processing device converts the user camera coordinates into user position coordinates relative to the display according to the shooting direction of the first image sensor, and converting the object camera coordinates to object position coordinates relative to the display, Wherein the processing device determines the display position information of the virtual information on the display according to the connection line between the target position coordinates and the user position coordinates. The information display system according to claim 6, wherein the display position information includes a position of an intersection point where the connection line intersects the display plane of the display. The information display system according to claim 4, wherein the spatial position information of the target includes a horizontal offset angle and a vertical offset angle of the target relative to the second image sensor, and the user The spatial position information includes the horizontal offset angle and the vertical offset angle of the user relative to the first image sensor, Wherein the processing device is based on the horizontal deviation angle and the vertical deviation angle of the target object relative to the second image sensor, the horizontal deviation angle of the user relative to the first image sensor And the vertical offset angle, the shooting direction of the first image sensor and the shooting direction of the second image sensor determine the display position information of the virtual information on the display. The information display system as described in claim 8, wherein the second information capturing device performs object detection on a plurality of target images to obtain an object bounding box of the target in each of the target images, the display position information includes a reference display object frame corresponding to each of the object bounding boxes on the display plane of the display, Wherein the reference display object frame includes a first reference display object frame corresponding to the previous time and a second reference display object frame corresponding to the current time, and the first display reference position includes the first reference display object frame an object frame, and the second display reference location includes the second reference display object frame, Wherein the processing device compares the distance between the first reference display object frame corresponding to the previous time and the second reference display object frame corresponding to the current time with a preset threshold value to determine The virtual information corresponds to the actual display position of the current time on the display. The information display system according to claim 9, wherein the preset threshold is determined according to the size of the second reference display object frame. The information display system according to claim 9, wherein the processing means calculates the distance between the first reference point of the first reference display object frame and the second reference point of the second reference display object frame , the first reference point and the second reference point are frame center points or frame vertices. The information display system according to claim 1, wherein the processing device adjusts the display position information in a direction away from the edge of the display in response to the display position information being located in a display edge area of the display. The information display system according to claim 1, wherein the actual display position and the display position information are pixel coordinates of the display, and the processing device obtains the display position according to the size and resolution of the display Information. An information display method is applicable to an information display system having a light-transmissive display, a first information capture device, a second information capture device, and a processing device, and the method includes: using the first information capturing device to capture spatial location information of a user; using the second information capturing device to capture spatial position information of a target; Determining display position information of virtual information of the target on the display according to the spatial position information of the user and the spatial position information of the target, wherein the display position information includes the first time corresponding to the previous time a display reference position and a second display reference position corresponding to the current time; determining an actual display position of the virtual information corresponding to the current time on the display according to a distance between the first display reference position and the second display reference position; and and displaying the virtual information on the display according to the actual display position. The information display method according to claim 14, wherein the position of the virtual information on the display at the current time is determined according to the distance between the first display reference position and the second display reference position The step of the actual display position comprises: Responding to the fact that the distance between the first display reference position and the second display reference position is greater than a preset threshold, determining the virtual information at the second display reference position corresponding to the current time said actual display position on said display corresponding to said current time; and In response to the fact that the distance between the first display reference position and the second display reference position is not greater than the predetermined threshold, the virtual information is determined according to the first display reference position corresponding to the previous time The actual display position corresponding to the current time on the display. The information display method as described in claim 14, wherein the method further includes: using the first information capturing device to capture a user's posture information, The step of determining the display position information of the virtual information of the target on the display according to the spatial position information of the user and the spatial position information of the target includes: determining head movement or rotation of the user based on posture information of the user; and In response to determining the user's head movement according to the user's posture information, determining the virtual information of the target object based on the user's spatial position information and the target object's spatial position information The display position information on the display, or in response to determining the user’s head rotation based on the user’s posture information, according to the user’s spatial position information and posture information and the space of the target The position information determines the display position information of the virtual information of the target on the display. The information display method as described in claim 14, wherein the first information capturing device includes a first image sensor for photographing the user, and the second information capturing device includes a sensor for photographing the user the second image sensor of the target object, Wherein, the step of using the second information capturing device to capture the spatial position information of the target includes: Using the second information acquisition device to perform object detection on multiple target images to obtain the object bounding box of the target in each of the target images, and perform coordinate transformation according to the object bounding box to obtain spatial location information of the target, Wherein, the step of using the first information retrieval device to retrieve the user's spatial location information includes: Use the first information capture device to perform face detection on multiple user images to obtain the face position in each user image, and perform coordinate transformation according to the face position to obtain the user The spatial location information of . The information display method according to claim 17, wherein the second information capturing device performs object detection on multiple target images to obtain an object bounding box of each target in the target image, the display position information includes a reference display object frame corresponding to each of the object bounding boxes on the display plane of the display, Wherein the reference display object frame includes a first reference display object frame corresponding to the previous time and a second reference display object frame corresponding to the current time, and the first display reference position includes the first reference display object frame an object frame, and the second display reference location includes the second reference display object frame, The step of determining the actual display position of the virtual information on the display at the current time according to the distance between the first display reference position and the second display reference position comprises: comparing the distance between the first reference display object frame corresponding to the previous time and the second reference display object frame corresponding to the current time with a preset threshold value, so as to determine when the virtual information is The actual display position corresponding to the current time on the display. The information display method according to claim 14, wherein the processing device adjusts the display position information in a direction away from the edge of the display in response to the fact that the display position information is located in a display edge area of the display. A processing device connected to a light-transmitting display, a first information capturing device, and a second information capturing device, wherein the first information capturing device is used to capture spatial position information of a user, and the second information capturing device Two information retrieval devices are used to retrieve the spatial position information of a target, and the processing device includes: memory for storing data; and a processor coupled to said memory and configured to: obtaining spatial location information of the user from the first information retrieval device; Obtaining spatial position information of the target from the second information capturing device; Determining display position information of virtual information of the target on the display according to the spatial position information of the user and the spatial position information of the target, wherein the display position information includes the first time corresponding to the previous time a display reference position and a second display reference position corresponding to the current time; determining an actual display position of the virtual information corresponding to the current time on the display according to a distance between the first display reference position and the second display reference position; and and displaying the virtual information on the display according to the actual display position.

Images