TWI818665B - Method, processing device, and display system for information display - Google Patents

Abstract

A method, a processing device, and a display system for information display are proposed, and the system includes a plurality of light transmissive displays and a plurality of processing devices. The processing devices are connected and communicate with each other via gateways. A first processing device is selected from the processing devices according to position information of a user, and the first processing device determines sight line information of the user according to the position information and posture information of the user. A second processing device different from the first processing device calculates an object coordinate of the target object. The first processing device selects a third processing device from the processing devices according to the sight line information of the user, and the third processing device determines display position information of a virtual object according to a user coordinate and the object coordinate and controls one of the displays to display the virtual object according to the display position information.

Description

Information display method and information display system and processing device thereof

The invention relates to an information display technology.

With the development of image processing technology and spatial positioning technology, the application of transparent displays has gradually attracted attention. This type of technology allows the display to be matched with physical objects, supplemented by virtual objects, and generate an interactive experience according to the user's needs, allowing information to be presented in a more intuitive way.

Furthermore, virtual objects associated with physical objects can be displayed at specific positions on the transparent display, allowing users to simultaneously view the physical objects and the virtual objects superimposed on or on one side of the physical objects through the transparent display. For example, by setting up a transparent display on an observation deck, viewers can view the landscape and the landscape information provided by the transparent display at the same time. However, in some large-scale application scenarios, it may be necessary to provide virtual and real information display services through a combination of multiple transparent displays, and the number of physical objects and users is also larger. Therefore, if a single central computing device is used to be responsible for all computing tasks, computing delays may occur due to excessive computing workload or other factors, resulting in the inability to provide real-time virtual and real integrated display services to viewers.

The present disclosure provides an information display method and its information display system and processing device, which can allocate computing tasks to selected processing devices based on the user's position and the user's line of sight information.

In an exemplary embodiment of the present disclosure, the above-mentioned information display system includes a plurality of light-transmissive displays, a plurality of sensory information acquisition devices, and a plurality of processing devices. The perceptual information acquisition device is used to acquire the user's position information and attitude information as well as the position information of the target object. The processing devices respectively correspond to the displays and are connected and communicated with each other through multiple gateways. The first processing device is selected from the processing device based on the user's position information, and the first processing device determines the user's line of sight information based on the user's position information and posture information provided by the sensing information acquisition device. The second processing device, which is different from the first processing device, performs coordinate conversion based on the position information of the target object provided by the sensing information acquisition device to calculate the target coordinates of the target object. The first processing device selects the third processing device from the processing devices according to the user's line of sight information. The third processing device determines the display position information of the virtual object based on the user coordinates and the target object coordinates, and controls one of the displays to display the virtual object based on the display position information of the virtual object.

In an exemplary embodiment of the present disclosure, the above information display method is applicable to an information display system having multiple light-transmissive displays, multiple sensory information acquisition devices, and multiple processing devices, and includes the following steps. The perceptual information acquisition device is used to acquire the user's position information and posture information as well as the target object's position information. The first processing device is selected from the processing devices based on the user's location information. The first processing device determines the user's line of sight information based on the user's position information and posture information provided by the sensing information acquisition device. A target coordinate of the target is calculated by a second processing device that is different from the first processing device performing coordinate conversion based on the position information of the target provided by the sensing information acquisition device. The third processing device is selected from the processing devices according to the user's sight information. The third processing device determines the display position information of the virtual object based on the user coordinates and the target object coordinates, and controls one of the displays to display the virtual object based on the display position information of the virtual object.

An exemplary embodiment of the present disclosure provides a processing device that is connected to a light-transmissive display and a sensory information acquisition device, and is connected to a plurality of other processing devices through a plurality of gateways. The perceptual information acquisition device is used to acquire the user's position information and posture information, and the processing device includes a memory and a processor. The memory is used to store data, and the processor is configured to perform the following steps. The sensing information acquisition device determines that the distance between the processing device and the user is smaller than the distance between each other processing device and the user. The user's line of sight information is determined based on the user's position information and posture information provided by the perceptual information acquisition device. One of the plurality of processing devices is selected according to the user's line of sight information, and the user's line of sight information is transmitted to one of the plurality of processing devices through the gateway. Among them, one of the plurality of processing devices determines the display position information of the virtual object based on the line of sight information, the user coordinates and the target object coordinates, and controls the display or another display connected to multiple other processing devices to display the virtual object according to Location information displays virtual objects.

In order to make the present disclosure more obvious and understandable, embodiments are given below and described in detail with reference to the attached drawings.

Some exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The component symbols cited in the following description will be regarded as the same or similar components when the same component symbols appear in different drawings. These exemplary embodiments are only part of the disclosure and do not disclose all possible implementations of the disclosure. Rather, these example embodiments are merely examples of methods and systems within the scope of the present disclosure.

FIG. 1A is a block diagram of an information display system according to an exemplary embodiment of the present disclosure. First, Figure 1A first introduces the various components and configuration relationships in the system. Detailed functions will be disclosed together with the flow charts of subsequent exemplary embodiments.

Referring to FIG. 1A , the information display system 10 in this exemplary embodiment may include multiple displays 110_1, 110_2, 110_3,..., 110_N, multiple sensing information acquisition devices 120_1, 120_2, 120_3,..., 120_N, and multiple Processing devices 130_1, 130_2, 130_3, ..., 130_N. The processing devices 130_1 to 130_N may be connected to the displays 110_1 to 110_N and the sensory information acquisition devices 120_1 to 120_N respectively through wireless, wired or electrical connections. It should be noted that in the example of FIG. 1A , a processing device is connected to a display and a perceptual information acquisition device as an example. For example, the processing device 130_1 is connected to the display 110_1 and the perceptual information acquisition device 120_1. However, this The disclosure is not limited to this. In other examples, a processing device may be connected to multiple sensory information capture devices or multiple displays.

The displays 110_1 to 110_N may be used to display information, which may be composed of a single or multiple display devices. The display device may be, for example, a liquid crystal display (LCD), a field sequential color liquid crystal display, Transmissive light-transmitting displays such as light emitting diode (LED) displays, electrowetting displays, or projection-type light-transmitting displays.

The sensing information acquisition devices 120_1˜120_N can be used to acquire the user's position information and posture information. The sensing information capturing devices 120_1˜120_N include sensing devices used to capture user information. In some embodiments, the sensory information capturing devices 120_1 to 120_N may include at least one image sensor or at least one image sensor and at least one depth sensor to capture the user located in front of the displays 110_1 to 110_N. Obtain image data to identify and locate the user. The aforementioned image sensor may be a visible light sensor or a non-visible light sensor such as an infrared sensor. In addition, the sensory information acquisition devices 120_1 to 120_N may also include optical positioners to perform optical spatial positioning of the user. In some embodiments, the perceptual information acquisition devices 120_1˜120_N can also identify the postures of the user's limbs, torso, and head through various human posture recognition technologies. For example, the perceptual information acquisition devices 120_1 to 120_N can identify the human skeleton, human body feature points, etc. based on the image data, thereby identifying the user's posture. As long as it is a device or a combination thereof that can locate the user's location information and identify the user's posture information, it belongs to the category of perceptual information acquisition devices 120_1 to 120_N.

On the other hand, the sensing information acquisition devices 120_1˜120_N can be used to acquire position information of target objects in the physical scene. The sensing information acquisition devices 120_1˜120_N include sensing devices used to acquire target object information. In some embodiments, the sensory information capturing devices 120_1 to 120_N may include at least one image sensor or at least one image sensor and at least one depth sensor to face the target object located behind the displays 110_1 to 110_N. Capture image data to identify and locate the target object. The aforementioned image sensor may be a visible light sensor or a non-visible light sensor such as an infrared sensor. As long as it is a device or a combination thereof that can locate the location information of the target object, it belongs to the category of sensing information acquisition devices 120_1 to 120_N.

In the embodiment of the present disclosure, the above-mentioned image sensor can be used to capture images and includes a camera lens having a lens and a photosensitive element. The above-mentioned depth sensor can be used to detect depth information, which can be implemented using active depth sensing technology and passive depth sensing technology. Active depth sensing technology can calculate depth information by actively emitting light sources, infrared, ultrasound, laser, etc. as signals together with time-lag ranging technology. Passive depth sensing technology can use two image sensors to capture two images in front of them from different viewing angles to calculate depth information using the parallax of the two images.

In some embodiments, the sensing information acquisition devices 120_1 ~ 120_N can transmit information to the processing devices 130_1 ~ 130_N through their respective communication interfaces in a wired or wireless manner. The processing devices 130_1 to 130_N are computer devices with computing capabilities. The processing devices 130_1 to 130_N can be deployed in the field to which the information display system 10 belongs. They can be computer devices built into the displays 110_1 to 110_N respectively or connected to the displays 110_1 to 110_N respectively. The processing devices 130_1 to 130_N correspond to the displays 110_1 to 110_N respectively, and can be used to control the displays 110_1 to 110_N connected thereto. For example, the processing device 130_1 can be used to control the display 110_1 to display and display content.

For example, FIG. 1B is a schematic diagram of an information display system according to an exemplary embodiment of the present disclosure. For convenience and clear explanation, FIG. 1B uses three displays 110_1 to 110_3 and three sensory information acquisition devices 120_1 to 120_3 as an example for illustration, but the present disclosure is not limited thereto. Please refer to FIG. 1B , the user U1 and the target Obj1 are located on the front side and the back side of the displays 110_1˜110_3 respectively. In this example, the user U1 can view the physical scene of the virtual object Vf1 including the target object Obj1 through the display 110_2. The virtual object Vf1 can be regarded as augmented reality content amplified based on the target object Obj1.

It should be noted that the processing devices 130_1 to 130_N are connected and communicate with each other through a plurality of gateways G1, G2,..., Gk. Each gateway G1 ~ Gk supports a wireless transmission protocol or a wired transmission protocol, and can establish a link with a nearby gateway or processing device 130_1 ~ 130_N. This disclosure does not limit the types of wireless transmission protocols and wired transmission protocols, which can be WiFi standards, ZigBee standards, mobile communication standards, or Ethernet standards, etc. In some embodiments, the gateways G1˜Gk may form a network topology N1. However, the disclosure does not limit the number of gateways G1 to Gk and the type of network topology. Each processing device 130_1˜130_N will be connected to at least one of the gateways G1˜Gk. Through the links between the gateways G1 ~ Gk, the processing devices 130_1 ~ 130_N can transmit information and communicate with each other through the gateways G1 ~ Gk.

It should be noted that by setting up links between multiple processing devices 130_1 to 130_N and gateways G1 to Gk, the virtual object Vf1 is displayed based on the position information and posture information of the user U1 and the position information of the target Obj1. The computing tasks can be distributed to some of the processing devices 130_1˜130_N for execution. In this way, computing efficiency can be improved through a distributed processing architecture to avoid delayed display of virtual objects.

FIG. 2 is a flow chart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to FIGS. 1A, 1B and 2 at the same time. The method flow of FIG. 2 can be implemented by the information display system 10 of FIGS. 1A and 1B. to achieve.

In step S210, the sensing information acquisition devices 120_1˜120_N are used to acquire the position information and attitude information of the user U1 and the position information of the target Obj1. As mentioned above, the perceptual information acquisition devices 120_1 to 120_N are, for example, image sensors, depth sensors or combinations thereof that can locate the positions of the user U1 and the target Obj1.

In step S220, the first processing device is selected from the processing devices 130_1˜130_N according to the location information of the user U1. In some embodiments, the first processing device is the one closest to the location information of the user U1 among the processing devices 130_1˜130_N. That is, the distance between the first processing device and the user U1 is smaller than the distance between each other processing device and the user U1. Specifically, at least one of the sensing information acquisition devices 120_1˜120_N can locate the location information of the user U1. Moreover, when the processing devices 130_1 to 130_N have been fixedly installed in the field to which the information display system 10 belongs, the location information of the processing devices 130_1 to 130_N is known. Therefore, at least one of the processing devices 130_1 to 130_N can obtain the distance between each of the processing devices 130_1 to 130_N and the user U1 based on the location information of the user U1 and the known location information of each of the processing devices 130_1 to 130_N. In this way, the first processing device that is closest to the location information of the user U1 among the plurality of processing devices can be selected. It can be seen that, in response to the dynamic movement of the user U1, the first processing device closest to the user U1 may change accordingly.

In step S230, the first processing device determines the user's line of sight information E1 based on the position information and posture information of the user U1 provided by the sensing information acquisition devices 120_1˜120_N. Specifically, after selecting the first processing device, the first processing device can directly obtain the user's position information and posture information from one of the sensory information acquisition devices 120_1 to 120_N connected thereto, or from the gateway G1 to Gk obtains the user's location information and posture information. Therefore, the first processing device can identify the line of sight information E1 according to the position information and posture information of the user U1, and the line of sight information E1 includes the line of sight vector.

In step S240, the target coordinates of the target Obj1 are calculated by performing coordinate conversion based on the position information of the target Obj1 provided by the sensing information acquisition devices 120_1˜120_N through a second processing device that is different from the first processing device. In other words, the second processing device will perform coordinate conversion on the position information (such as camera coordinates or image coordinates) of the target Obj1 provided by at least one of the sensing information acquisition devices 120_1˜120_N, and obtain the target in the three-dimensional display coordinate system. object coordinates.

In step S250, the first processing device selects a third processing device from the processing devices 130_1˜130_N according to the line of sight information E1 of the user U1. Specifically, after the first processing device obtains the line of sight information E1 of the user U1, the first processing device identifies one of the displays 110_1 to 110_N according to the line of sight information E1 of the user U1, so as to identify one of the displays 110_1 to 110_N based on the line of sight information E1 of the user U1. The corresponding third processing device is selected from the processing devices 130_1 to 130_N. In some embodiments, the first processing device may calculate the line of sight angle range corresponding to one of the displays 110_1˜110_N according to the location information of the user U1. In response to the line of sight information of the user U1 falling within the line of sight angle range, the first processing device identifies one of the displays 110_1 to 110_N from the displays 110_1 to 110_3. Taking FIG. 1B as an example, the first processing device can calculate the line of sight angle range corresponding to the display 110_2 based on the location information of the user U1. Since the line of sight information of the user U1 falls within the line of sight angle range of the display 110_2, it can be determined that the line of sight position of the user U1 falls on the display 110_2.

That is to say, the first processing device can identify the display that the user is looking at based on the sight information E1 of the user U1. Since the displays 110_1 to 110_N will be controlled by the corresponding processing devices 130_1 to 130_N respectively, the first processing device can select the processing device corresponding to the display that the user is looking at as the third processing device. It should be noted that the first processing device may be the same as or different from the third processing device. More specifically, in a situation where the displays 110_1 to 110_N can be arranged in parallel and the processing devices 130_1 to 130_N are respectively disposed adjacent to the corresponding displays 110_1 to 110_N, when the user looks at the display directly in front, the third one closest to the user U1 One processing device will be the same as the third processing device that the user U1 is looking at; when the user looks at the monitors on the left and right sides, the first processing device closest to the user U1 will be different from the third processing device that the user U1 is looking at. processing device.

In some embodiments, the third processing device performs coordinate conversion based on the location information of user U1 provided by the sensing information acquisition devices 120_1˜120_N to calculate the user coordinates of the user. In other words, the third processing device will perform coordinate conversion on the location information (such as camera coordinates or image coordinates) of the user U1 provided by at least one of the sensory information acquisition devices 120_1 to 120_N, and obtain the usage information in the three-dimensional display coordinate system. or coordinates.

In step S260, the third processing device determines the display position information of the virtual object Vf1 according to the user coordinates and the target object coordinates, and controls one of the displays 110_1 to 110_N to display the virtual object Vf1 according to the display position information of the virtual object Vf1. In some embodiments, the second processing device may transmit the object coordinates of the object Obj1 to the third processing device via the gateways G1˜Gk. Similarly, if the first processing device is different from the third processing device, the first processing device can also transmit the line of sight information E1 of the user U1 to the third processing device via the gateways G1 to Gk. Based on this, the third processing device can determine the display position information of the virtual object Vf1 based on the user coordinates, the line of sight information E1 and the target object coordinates. Specifically, the display position information can be regarded as the point or area where the user's line of sight is projected on the display plane when viewing the target object Obj1. Based on various needs or different applications, the third processing device can determine the actual display position of the virtual object Vf1 based on the display position information, so that the user U1 can see the virtual object Vf1 displayed near the target object Obj1 or see it superimposed on the target object Obj1. The virtual object Vf1 on the target Obj1.

It can be seen from this that by linking multiple processing devices 130_1 ~ 130_N through the gateways G1 ~ Gk, the present disclosure can allocate the calculation amount required to display the virtual object Vf1 to multiple processing devices, thereby greatly improving the calculation efficiency. Avoid display delays of virtual objects.

The following is an example of the display system 100 to illustrate the present disclosure's implementation of determining the third processing device and single-user and multi-user based on line of sight information. For convenience and clear explanation, subsequent embodiments will be described using an example in which three processing devices 130_1 to 130_3 are connected to three displays 110_1 to 110_3 and three sensing information acquisition devices 120_1 to 120_3 respectively, but the present disclosure is not limited thereto. The processing devices 130_1 to 130_3 may be respectively disposed adjacent to the corresponding displays 110_1 to 110_3.

FIG. 3A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 3B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to Figure 3A and Figure 3B at the same time. In this embodiment, the user U1 is located in front of the display 110_2 and looks at the display 110_2 located directly in front of the user U1.

The sensing information acquisition device 120_2 can acquire the position information and attitude information of the user U1 (step S302), and transmit the position information and attitude information of the user U1 to, for example, the processing device 130_2. In response to receiving the location information of the user U1, the processing device 130_2 can calculate the distance between each processing device 130_1˜130_3 and the location information of the user U1. Furthermore, the processing device 130_2 may select the first processing device according to the distance between each processing device 130_1 to 130_3 and the location information of the user U1 (step S304). Here, the first processing device is the one closest to the location information of the user U1 among the processing devices 130_1 to 130_3. In this example, it is assumed that the processing device 130_2 is the first processing device closest to the user U1. That is, in one embodiment, the processing device 130_2 can determine through the sensing information acquisition device 120_2 that the distance between the processing device 130_2 and the user U1 is smaller than the distance between each of the other processing devices 130_1, 130_3 and the user U1.

Next, the processing device 130_2 can identify the line of sight information E1 of the user U1 according to the position information and posture information of the user U1 (step S306). To be more specific, the processing device 130_2 is selected to calculate the line of sight information E1 of the user U1 and determine which display the line of sight information E1 of the user U1 falls on. In this example, the processing device 130_2 can determine that the line of sight information E1 of the user U1 falls on the display 110_2 based on the line of sight information E1 of the user U1, and select the third processing device according to the display 110_2 projected by the line of sight information E1 (step S308). In this example, the processing device 130_2 used to control the display 110_2 is the third processing device. That is, the first processing device and the third processing device in this example are the same and both are the processing device 130_2. Therefore, the processing device 130_2 performs coordinate conversion based on the location information of the user U1 provided by the sensing information acquisition device 120_2 to calculate the user coordinates of the user (step S310).

On the other hand, the sensing information capturing devices 120_1 to 120_3 can capture the position information of the target object Obj1 (step S312). Since the processing device 130_2 has been selected as the first processing device, the processing device 130_1 or the processing device 130_3 may be selected as the second processing device (step S314). The following description will continue taking the processing device 130_3 as the second processing device as an example. The processing device 130_3 can receive the position information and other related information of the target Obj1 to further process the target recognition related to the target Obj1 (step S316). Next, the processing device 130_3 performs coordinate conversion based on the position information of the target Obj1 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target coordinates of the target Obj1 (step S318), so as to compare the position information of the user U1 with the target The position information of object Obj1 is converted to the same coordinate system. The processing device 130_3 may transmit the target object coordinates of the target Obj1 to the processing device 130_2 as the third processing device via at least one of the gateways G1 to Gk (step S320).

Finally, the processing device 130_2 determines the display position information of the virtual object Vf1 according to the user coordinates and the target object coordinates (step S322), and controls the display 110_2 to display the virtual object according to the display position information of the virtual object (step S324). Thereby, the processing device 130_2 can display the virtual object Vf1 based on the displayed position information.

FIG. 4A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 4B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to Figure 4A and Figure 4B at the same time. In this embodiment, the user U1 is located in front of the display 110_3 and looks at the display 110_1 located on the left side of the user U1.

The sensing information acquisition device 120_3 can acquire the position information and attitude information of the user U1 (step S402), and transmit the position information and attitude information of the user U1 to, for example, the processing device 130_3. The processing device 130_3 may select the first processing device according to the distance between each processing device 130_1 to 130_3 and the location information of the user U1 (step S404). In this example, the first processing device is the processing device 130_3 that is closest to the location information of the user U1 among the processing devices 130_1 to 130_3. Next, the processing device 130_3 as the first processing device can identify the line of sight information E2 of the user U1 based on the position information and posture information of the user U1 (step S406). In this example, the processing device 130_3 can determine that the line of sight information E2 of the user U1 falls on the display 110_1 based on the line of sight information E2 of the user U1, and select the third processing device according to the display 110_1 projected by the line of sight information E2 (step S408). In this example, the processing device 130_1 used to control the display 110_1 is the third processing device. That is, the first processing device and the third processing device in this example are different. The processing device 130_3 may transmit the line of sight information E2 of the user U1 to the processing device 130_1 as the third processing device via at least one of the gateways G1 to Gk (step S410).

On the other hand, the sensing information capturing devices 120_1 to 120_3 can capture the position information of the target object Obj2 (step S412). Since the processing device 130_3 has been selected as the first processing device and the processing device 130_1 has been selected as the third processing device, the processing device 130_2 may be selected as the second processing device (step S414). The processing device 130_2 can receive the position information and other related information of the target Obj2 to further process the target recognition related to the target Obj2 (step S416). Next, the processing device 130_2 performs coordinate conversion based on the position information of the target Obj2 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target coordinates of the target Obj2 (step S418). The processing device 130_2 may transmit the target object coordinates of the target Obj2 to the processing device 130_1 as the third processing device via at least one of the gateways G1 to Gk (step S420).

The processing device 130_1 may receive the location information of the user U1 through the gateways G1˜Gk or directly from the sensing information acquisition device 120_1. Therefore, the processing device 130_1 performs coordinate conversion based on the location information of the user U1 to calculate the user coordinates of the user (step S422). The processing device 130_1 determines the display position information of the virtual object Vf2 based on the user coordinates, the target object coordinates, and the line of sight information E2 (step S424), and controls the display 110_1 to display the virtual object Vf2 based on the display position information of the virtual object Vf2 (step S426). In this example, the processing device 130_3 (ie, the first processing device) can analyze the line of sight information E2 of the user U1. The processing device 130_2 (ie, the second processing device) can process object recognition and coordinate conversion of the target Obj2. The processing device 130_1 (ie, the third processing device) determines the display position information of the virtual object Vf2 based on the user coordinates and the target object coordinates.

In some embodiments, the first processing device can calculate the line of sight angle range corresponding to a certain display based on the location information of the user U1. In response to the sight line information of the user U1 falling within the sight angle range, the first processing device can identify the display that the user U1 is looking at from the displays 110_1 to 110_3.

In detail, FIG. 5A and FIG. 5B are schematic diagrams of estimating a line of sight position according to an embodiment of the present disclosure. Please refer to FIG. 5A and FIG. 5B , the width of the display 110_2 is dw. By setting reference points P1 to P4 with known positions in front of the display 110_2, the processing device 130_2 can estimate the distance between the user U1 and the display based on the pixel positions of the reference points P1 to P4 on the image captured by the sensory information acquisition device 120_2. The lateral offset distance X of the left boundary of 110_2. The reference points P1 to P4 can be any markers, and this disclosure is not limited thereto.

In more detail, the depth information of reference points P1 to P2 is D1, and the depth information of reference points P3 to P4 is D2. The ratio of the subtraction result of the X-axis pixel coordinate D1 _R of the reference point P1 minus the X-axis pixel coordinate D1 _L of the reference point P2 to the depth information D1 will be equal to the X-axis pixel coordinate Du _R minus the X-axis pixel coordinate Du _L The subtraction result is proportional to the depth information D. In the same way, the ratio of the subtraction result of the X-axis pixel coordinate D2 _R of the reference point P3 minus the X-axis pixel coordinate D2 _L of the reference point P4 to the depth information D2 will be equal to the X-axis pixel coordinate Du _R minus the X-axis pixel. The ratio of the subtraction result of coordinates Du _L to the depth information D. Based on this, when the X-axis pixel coordinate Du _R and the X-axis pixel coordinate Du _L can be known through calculation, the distance to the user U1 can be known through interpolation calculation and user-based depth information D and width dw, for example. The lateral offset distance X of the left border of the display 110_2.

In this way, the line of sight angle range θ can be calculated based on the lateral offset distance X, the depth information D and the tangent function. As shown in FIG. 5B , if the line of sight information E3 of the user U1 does not fall within the line of sight angle range θ, it means that the user U1 is looking at the display 110_1 on the left. On the contrary, if the line of sight information of the user U1 falls within the line of sight angle range θ, it means that the user U1 is looking at the display 110_2.

FIG. 6A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 6B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to Figure 6A and Figure 6B at the same time. In this embodiment, the user U1 is located in front of the display 110_2, and switches from looking at the display 110_2 located directly in front of the user U1 to looking at the display 110_1 located to the left of the user U1.

The sensing information acquisition device 120_2 can acquire the position information and attitude information of the user U1 (step S602), and transmit the position information and attitude information of the user U1 to, for example, the processing device 130_2. The processing device 130_2 may select the first processing device according to the distance between each processing device 130_1 to 130_3 and the location information of the user U1 (step S604). In this example, the first processing device may be the processing device 130_2 that is closest to the location information of the user U1 among the processing devices 130_1 to 130_3. Next, the processing device 130_2 as the first processing device can identify the line of sight information E1 of the user U1 based on the position information and posture information of the user U1 (step S606). In this example, the processing device 130_2 can determine that the line of sight information E1 of the user U1 falls on the display 110_2 based on the line of sight information E1 of the user U1, and select the third processing device according to the display 110_1 projected by the line of sight information E1 (step S608). In this example, before the line of sight information of the user U1 changes, the processing device 130_2 used to control the display 110_2 is also a third processing device. Therefore, the processing device 130_2 calculates the user coordinates of the user (step S610). The processing device 130_2 determines the display position information of the virtual object Vf1 (step S612). The processing device 130_2 controls the display 110_2 to display the virtual object Vf1 (step S614). The detailed operation details of steps S602 to S614 have been described in detail in the previous embodiments and will not be described again here.

It should be noted that in response to the turning or turning of the head of the user U1, the processing device 130_2 detects changes in the line of sight information of the user U1 (step S616). In this example, the user's gaze information E1 changes to gaze information E3. After the line of sight information of the user U1 changes, the processing device 130_2 determines whether the line of sight information E3 of the user U1 still falls within the line of sight angle range of one of the displays 110_1 to 110_3 (ie, the display 110_2) (step S618) . In response to the fact that the line of sight information E3 of the user U1 does not fall within the line of sight angle range of the display 110_2 (step S618 determines as NO), the processing device 130_2 identifies another one of the displays 110_1 to 110_3 (i.e., the display) based on the user's line of sight information E3. 110_1) to select another third processing device from the processing devices 130_1 to 130_3 according to another one of the displays 110_1 to 110_3 (ie, the display 110_1) (step S620). In this example, after the line of sight information of the user U1 changes, the processing device 130_1 used to control the display 110_1 is identified as another third processing device. Therefore, the processing device 130_2 transmits the line of sight information E3 to the subsequent processing device 130_1 responsible for display control (step S622).

On the other hand, the sensing information capturing devices 120_1 to 120_3 can capture the position information of the target objects Obj1 and Obj2 (step S624). Since the processing device 130_2 has been selected as the first processing device, the processing device 130_3 may be selected as the second processing device (step S626). The processing device 130_3 can receive the position information and other related information of the target objects Obj1 and Obj2 to further process the target object recognition related to the target objects Obj1 and Obj2 (step S628). Next, the processing device 130_3 performs coordinate conversion based on the position information of the target objects Obj1 and Obj2 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target coordinates of the target objects Obj1 and Obj2 (step S630). The processing device 130_3 may transmit the target object coordinates of the target objects Obj1 and Obj2 to the processing device 130_1 and the processing device 130_2 as the third processing device through at least one of the gateways G1 to Gk (step S632).

Similar to the aforementioned operating principles, the processing device 130_1 performs coordinate conversion based on the location information of the user U1 to calculate the user coordinates of the user (step S634). The processing device 130_1 determines the display position information of the virtual object Vf2 based on the user coordinates, the target object coordinates, and the line of sight information E3 (step S636), and controls the display 110_1 to display the virtual object Vf2 based on the display position information of the virtual object Vf2 (step S638). In this example, the third processing device responsible for display control will switch from the processing device 130_2 to the processing device 130_1 in response to the line of sight change.

FIG. 7 is a schematic diagram of an application scenario of the information display system according to an exemplary embodiment of the present disclosure. Please refer to FIG. 7 . When the number of users exceeds one, two of the processing devices 130_1 to 130_3 can be used as the first processing device for calculating the line of sight information E2 and E4. In the example of FIG. 7 , since the sensing information acquisition device 120_1 detects the user U2, the processing device 130_1 closest to the user U2 will be selected as the first processing device to calculate the line of sight information E4 of the user U2. . On the other hand, since the sensing information acquisition device 120_1 detects the user U1, the processing device 130_3 closest to the user U1 will be selected as the first processing device to calculate the line of sight information E2 of the user U1. In addition, based on the line of sight information E2 and E4 of the users U1 and U2 projected on the displays 110_1 and 110_2, the processing devices 130_1 and 130_2 can be used to calculate the display position information of the virtual objects Obj1 and Obj2 respectively.

FIG. 8A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 8B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. Please refer to Figure 8A and Figure 8B at the same time. In this embodiment, the user U1 and multiple other users U3 and U4 are located in front of the display 110_3, and the user U1 and multiple other users U3 and U4 are all looking at the display 110_1 located on the left.

The sensing information capturing device 120_3 can capture the location information and attitude information of the user U1 and multiple other users U3 and U4 (step S802), and combine the location information of the user U1 and multiple other users U3 and U4 with The posture information is transmitted to, for example, the processing device 130_3. In this embodiment, the processing device 130_3 can select the first processing device according to the distance between each processing device 130_1˜130_3 and the location information of the user U1 and multiple other users U3 and U4 (step S804). Specifically, the processing device 130_3 can calculate the distance between the processing devices 130_1˜130_3 and the user U1. Similarly, the processing device 130_3 can calculate the distances between the processing devices 130_1˜130_3 and multiple other users U3 and U4 respectively. The processing device 130_3 can find the minimum distance from all the above distances, and select the processing device associated with the minimum distance as the first processing device. In this example, there is a minimum distance between the user U1 and the processing device 130_3, so the processing device 130_3 is selected as the first processing device.

Then, the processing device 130_3 as the first processing device can identify the line of sight information E3, E5, E6 (step S806).

The processing device 130_3 determines whether one of the sensing information capturing devices 120_1 to 120_3 (ie, the sensing information capturing device 120_3) detects the user U1 and multiple other users U3 and U4 at the same time (step S808). In response to the sensing information acquisition device 120_3 detecting the user U1 and the other users U3 and U4 at the same time (step S808 determines as yes), the processing device 130_3 based on the line of sight information E3 of the user U1 and the line of sight of the other users U3 and U4 The information E5 and E6 calculate a common line of sight direction (step S810), and select one of the third processing device and the display 110_1 to 110_3 from the processing devices 130_1 to 130_3 according to the common line of sight direction (step S812). In some embodiments, the processing device 130_3 can calculate the average value of the components in each axis direction of the line of sight information E3 of the user U1 and the line of sight information E5 and E6 of the other users U3 and U4 to obtain a common line of sight direction.

It should be noted that in some embodiments, before calculating a common gaze direction, the processing device 130_3 can also determine the gaze direction between the gaze information E3 of the user U1 and the gaze information E5 and E6 of other users U3 and U4. Whether the difference meets the preset conditions. Specifically, the processing device 130_3 may determine whether the angle difference between the gaze vector of the user U1 and the gaze vectors of the other users U3 and U4 is less than a threshold value. If so, it can be determined that the gaze direction difference between the gaze information E3 of user U1 and the gaze information E5 and E6 of other users U3 and U4 meets a preset condition, which means that the gaze direction of user U1 and other users U3 and U4 are similar. Location.

In addition, in this example, since the common line of sight direction falls on the display 110_1, the processing device 130_3 selects the third processing device according to the display 110_1 projected by the common line of sight direction. In this example, the processing device 130_1 used to control the display 110_1 is the third processing device. The processing device 130_3 may transmit the common line of sight direction to the processing device 130_1 as the third processing device via at least one of the gateways G1 to Gk (step S814).

On the other hand, the sensing information capturing devices 120_1 to 120_3 can capture the position information of the target object Obj2 (step S816). Since the processing device 130_3 has been selected as the first processing device and the processing device 130_1 has been selected as the third processing device, the processing device 130_2 may be selected as the second processing device (step S818). The processing device 130_2 can receive the position information and other related information of the target Obj2 to further process the target recognition related to the target Obj2 (step S820). Next, the processing device 130_2 performs coordinate conversion based on the position information of the target Obj2 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target coordinates of the target Obj2 (step S822). The processing device 130_2 may transmit the target object coordinates of the target Obj2 to the processing device 130_1 as the third processing device via at least one of the gateways G1 to Gk (step S824).

The processing device 130_1 may receive the location information of the user U1 through the gateways G1˜Gk or directly from the sensing information acquisition device 120_1. Therefore, the processing device 130_1 performs coordinate conversion based on the location information of the user U1 to calculate the user coordinates of the user (step S826). The processing device 130_1 determines the display position information of the virtual object Vf2 according to the user coordinates, the target object coordinates and the common sight direction (step S828), and controls the display 110_1 to display the virtual object Vf2 according to the display position information of the virtual object Vf2 (step S830).

FIG. 9 is a block diagram of a processing device according to an embodiment of the present disclosure. The processing device 900 may be the processing devices 130_1˜130_N of the aforementioned embodiments. Referring to FIG. 9 , the processing device 900 may include a memory 901 , a processor 902 and a transmission element 903 . The memory 901 can be, for example, any type of fixed or removable random access memory (random access memory, RAM), read-only memory (read-only memory, ROM), flash memory (flash memory), Hard disk or other similar device, integrated circuit, or combination thereof. The processor 902 may be, for example, a central processing unit (CPU), an application processor (AP), or other programmable general-purpose or special-purpose microprocessor, digital signal processing digital signal processor (DSP), image signal processor (ISP), graphics processing unit (GPU) or other similar devices, integrated circuits or combinations thereof. The transmission element 903 is a communication device that supports wired/wireless transmission protocols, such as a combination of a transceiver and an antenna. The processor 902 can execute instructions, program codes or software modules recorded in the memory 901 to implement the information display method of the present disclosure.

The information display method and its processing device and information display system proposed in the exemplary embodiments of the present disclosure can allocate operations to multiple processing devices according to the user's position and line of sight, thereby improving computing efficiency and avoiding the failure of virtual and real integrated display services. Delayed display. In this way, virtual objects can be displayed in real time and smoothly, greatly improving the user's viewing experience.

Although the disclosure has been disclosed above in the form of exemplary embodiments, this is not intended to limit the disclosure. Anyone with ordinary knowledge in the relevant technical field may make slight changes and modifications without departing from the spirit and scope of the disclosure. Therefore, the protection scope of this disclosure shall be determined by the appended patent application scope and its equivalent scope.

10:Information display system 110_1～110_N: Display 120_1～120_N: Perceptual information acquisition device 130_1～130_N: Processing device G1～Gk:Gateway N1: Network topology Vf1, Vf2: virtual objects RF1: Reference display object frame U1, U2, U3, U4: User Obj1, Obj2: Target object E1, E2, E3, E4, E5, E6: line of sight information P1～P4: reference point 901:Memory 902: Processor 903:Transmission element S210～S260, S302～S324, S402～S426, S602～S638, S802～S830: steps

The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the disclosure and, together with the description, serve to explain principles of the disclosure. FIG. 1A is a block diagram of an information display system according to an exemplary embodiment of the present disclosure. FIG. 1B is a schematic diagram of an information display system according to an exemplary embodiment of the present disclosure. FIG. 2 is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 3A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 3B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 4A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 4B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. 5A and 5B are schematic diagrams of estimating the line of sight position according to an embodiment of the present disclosure. FIG. 6A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 6B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 7 is a schematic diagram of an application scenario of the information display system according to an exemplary embodiment of the present disclosure. FIG. 8A is a schematic diagram of an application scenario of an information display system according to an exemplary embodiment of the present disclosure. FIG. 8B is a flowchart of an information display method according to an exemplary embodiment of the present disclosure. FIG. 9 is a block diagram of a processing device according to an embodiment of the present disclosure.

S210~S260: steps

Claims (17)

An information display system, including: multiple light-transmissive displays; multiple sensing information acquisition devices, used to acquire position information and attitude information of a user and position information of a target object; multiple processing devices , respectively corresponding to the display, and connected and communicated with each other through multiple gateways, wherein a first processing device is selected from the processing device according to the user's location information, and the first processing device The processing device determines the user's line of sight information based on the user's position information and posture information provided by the sensory information acquisition device, wherein a second processing device that is different from the first processing device determines the user's line of sight information based on the user's position information and posture information. The position information of the target object provided by the second sensing information acquisition device performs coordinate conversion to calculate a target coordinate of the target object, wherein the first processing device performs coordinate conversion according to the user's line of sight information A third processing device is selected from the processing device. The third processing device determines the display position information of a virtual object according to a user coordinate and the target object coordinate, and controls one of the displays according to The display location information of the virtual object displays the virtual object, wherein the first processing device is the one closest to the location information of the user among the processing devices. The information display system as claimed in claim 1, wherein the first processing device is the same as or different from the third processing device, and the third processing device is based on the information provided by the sensing information acquisition device. The user's location information is subjected to coordinate conversion to calculate the user coordinates of the user. The information display system of claim 1, wherein the second processing device transmits the target coordinates of the target to the third processing device via the gateway. The information display system as claimed in claim 1, wherein the first processing device identifies the one of the displays according to the user's line of sight information, so as to obtain the information from the display based on the one of the displays. The processing device selects the third processing device. The information display system of claim 4, wherein the first processing device calculates a line of sight angle range corresponding to the one of the displays based on the user's location information, and responds to the user's If the line of sight information falls within the line of sight angle range, the first processing device identifies the one of the displays from the display. The information display system of claim 4, wherein in response to changes in the user's sight information, the first processing device determines whether the user's sight information still falls on one of the displays. within the line of sight angle range of the user; and in response to the user's line of sight information not falling within the line of sight angle range of one of the displays, the first processing device determines line of sight information to identify the other one of the displays based on the The other one of the displays selects a further third processing means from the processing means. The information display system of claim 1, wherein the first processing device determines whether one of the sensing information acquisition devices detects the user and multiple other users at the same time, and responds to the sensing One of the information acquisition devices detects the user and the other users at the same time, and the first processing device calculates a common value based on the user's line of sight information and the line of sight information of the other users. The viewing direction, and selecting one of the third processing device and the display from the processing device according to the common viewing direction. The information display system of claim 7, wherein the difference in gaze direction between the user's gaze information and the gaze information of other users meets a preset condition. An information display method is suitable for an information display system including multiple light-transmissive displays, multiple perceptual information acquisition devices, and multiple processing devices, and the method includes: using the perceptual information acquisition device to capture a The user's position information and attitude information and the position information of a target object; selecting a first processing device from the processing device according to the user's position information; extracting a first processing device according to the sensing information through the first processing device Obtaining the user's position information and posture information provided by the device to determine the user's line of sight information; through a second processing device that is different from the first processing device according to the sense Perform coordinate conversion on the position information of the target object provided by the information acquisition device to calculate a target coordinate of the target object; select a third process from the processing device according to the user's line of sight information device; and determine the display position information of a virtual object according to a user coordinate and the target object coordinate through the third processing device, and control one of the displays according to the display position information of the virtual object Displaying the virtual object, wherein the step of selecting the first processing device from the processing device according to the user's location information includes: selecting the closest processing device to the user's location information of the first processing device. The information display method according to claim 9, wherein the first processing device is the same as or different from the third processing device, and the method further includes: using the third processing device to extract the information according to the sensing information. Obtain the user's location information provided by the device and perform coordinate conversion to calculate the user coordinates of the user. The information display method according to claim 9, wherein the method further includes: transmitting the object coordinates of the target object to the third processing device through the gateway through the second processing device. The information display method according to claim 9, wherein the step of selecting the third processing device from the processing device according to the user's line of sight information includes: using the first processing device to select the third processing device according to the user's line of sight information. The line of sight information identifies the one of the displays; and selecting the third processing device from the processing device based on the one of the displays. The information display method according to claim 12, wherein the step of identifying the one of the displays according to the user's line of sight information through the first processing device includes: using the first processing device according to the The user's position information calculates a line of sight angle range corresponding to the one of the displays; and in response to the user's line of sight information falling within the line of sight angle range, through the first processing device Identify the one of the displays from the display. The information display method as described in claim 12, the method further includes: in response to changes in the user's line of sight information, determining whether the user's line of sight information still falls on the Within the sight angle range of the one of the displays; and in response to the user's sight information not falling within the sight angle range of the one of the displays, through the first The processing device uses the The user's gaze information identifies the other one of the displays to select another third processing device from the processing device based on the other one of the displays. The information display method of claim 9, further comprising: determining, through the first processing device, whether one of the sensing information acquisition devices detects the user and multiple other users at the same time. ; In response to the one of the sensing information acquisition devices detecting the user and the other users at the same time, through the first processing device, based on the user's line of sight information and the other usage The user's gaze information calculates a common gaze direction, wherein the step of selecting the third processing device from the processing device according to the user's gaze information includes: selecting the third processing device from the processing device according to the common gaze direction. one of the third processing device and the display. The information display method of claim 9, wherein the difference in gaze direction between the user's gaze information and the gaze information of other users meets a preset condition. A processing device connected to a light-transmissive display and a perceptual information acquisition device, and connected to a plurality of other processing devices through a plurality of gateways, wherein the perceptual information acquisition device is used to acquire a user's For position information and attitude information, the processing device includes: a memory to store data; and a processor connected to the memory and configured: It is determined through the sensing information acquisition device that the distance between the processing device and the user is smaller than the distance between each of the plurality of other processing devices and the user, wherein the processing device is compared with the user. The plurality of other processing devices are closest to the user's position information; determine the user's line of sight information based on the user's position information and posture information provided by the sensory information acquisition device; and based on the user's line of sight information The user's gaze information selects one of the plurality of processing devices, and transmits the user's gaze information to one of the plurality of processing devices via the gateway, wherein the plurality of processing devices One of the processing devices determines the display position information of a virtual object based on the line of sight information, a user coordinate and a target object coordinate, and controls the display or another display connected to the other processing devices according to the virtual object coordinates. The display location information of the object displays the virtual object.

Images