JP7521270B2 - Circuit device, electronic device and mobile device - Google Patents

Description

The present invention relates to circuit devices, electronic devices, mobile objects, etc.

Head-up displays are known that display images on a transparent screen or similar, overlaying information on the user's field of vision. By combining such head-up displays with AR technology, it is possible to display virtual objects on the head-up display so that the virtual objects follow real objects, such as a car ahead or a road. AR stands for Augmented Reality.

In the technology disclosed in Patent Document 1, posture changes in the low frequency band are corrected by image processing, and posture changes in the high frequency band are corrected by a projection optical unit. By correcting posture changes in the high frequency band by the projection optical unit, the position of the projected image is corrected so that the virtual image is superimposed in the correct position on the real image when a posture change occurs in the vehicle.

JP 2019-98756 A

In a head-up display using AR technology, sensor output is acquired, the sensor output is used to render a virtual object, the rendered image is warped, and the warped image is displayed on the head-up display. In such a display system, there is a time lag between the sampling of the sensor output used for rendering and its display on the head-up display, so any movement of a vehicle or the like that occurs during that time is not reflected in the display position of the virtual object. This results in a positional shift between the rendered virtual object and the real object at the time of display, and there is a challenge in that it is desirable to minimize this positional shift due to latency.

One aspect of the present disclosure is a circuit device that performs display control of a head-up display that displays a virtual object corresponding to a real object in a real space in a display area, the circuit device including an interface to which tracking information, which is at least one of first tracking information of a moving body on which the head-up display is mounted, second tracking information of a viewer of the head-up display, and third tracking information of the real object, and a rendering image including the virtual object are input, a parameter calculation unit that calculates latency compensation parameters that compensate for latency including a rendering processing latency of the rendering image based on the tracking information, and a warp processing unit that performs a warp process that converts the coordinates of the rendering image in accordance with the curvature of the display area to generate a display image to be displayed in the display area, and the warp processing unit relates to a circuit device that performs latency compensation processing that compensates for the position of the virtual object in the display area based on the latency compensation parameters in the warp process.

Another aspect of the present disclosure relates to an electronic device that includes the circuit device described above.

Yet another aspect of the present disclosure relates to a moving object including the circuit device described above.

An example of AR display using a HUD. 5A and 5B are explanatory diagrams for explaining a display position shift caused by a pitch displacement or a vertical displacement. 5A and 5B are diagrams for explaining latency compensation according to the present embodiment; 1 shows an example of the configuration of a display system according to the prior art. 3 is a process flow of a display system according to the prior art. 1 shows a first configuration example of a circuit device of the present embodiment and a display system including the circuit device. 4 is a detailed processing flow of a tracking processing unit. 4 is a processing flow of the display system in the first configuration example. 2 shows a second configuration example of the circuit device of the present embodiment and a display system including the circuit device. 13 is a process flow of the display system in the second configuration example. 5A to 5C are explanatory diagrams of warp processing in the present embodiment. 1 is a diagram showing the correspondence between tracking information of a moving object or the like and latency compensation based on the tracking information; Example of latency compensation parameters to realize the compensation method. 1A and 1B are diagrams illustrating a method for calculating latency compensation parameters from tracking information. Example of electronic device configuration. An example of a moving object.

A preferred embodiment of the present disclosure will be described in detail below. Note that the present embodiment described below does not unduly limit the content described in the claims, and not all of the configurations described in the present embodiment are necessarily essential components.

1. First Configuration Example Fig. 1 shows an example of AR display using a HUD. HUD is an abbreviation for head-up display, and AR is an abbreviation for augmented reality. In the following, an example in which a HUD is mounted on an automobile 30 will be mainly described, but the HUD of this embodiment may be mounted on various moving bodies such as an airplane, a ship, or a motorcycle.

Figure 1 shows an AR display seen by the driver through the HUD. Through the windshield, the driver sees the vehicle 10 ahead, the road on which the vehicle 10 ahead is driving, and the scenery around them. The driver also sees a virtual image projected by the HUD into a display area 40 on the windshield, which appears to the driver as a virtual object 20 that overlaps with real space. The display area 40 indicates the range in which the HUD can project virtual images, and the virtual object 20 is displayed within this display area 40. The virtual object 20 in Figure 1 is an AR display object, and is displayed so that its display position follows the vehicle 10 ahead or the road. In other words, the HUD attempts to display the virtual object 20 so that the relative positional relationship between the vehicle 10 ahead or the road and the virtual object 20 as seen by the driver does not change.

The HUD display system uses sensors such as Lidar to track the vehicle 10 ahead, renders the virtual object 20 based on the tracking results, and warps the rendered image to display it on the HUD. At this time, there is a delay due to tracking processing, rendering processing, warping processing, data communication, etc. between the time when the sensor samples and the time when the virtual object 20 is displayed on the HUD. The delay caused by such processing or communication is called latency.

Latency causes a misalignment between the virtual object 20 and the vehicle 10 in front, etc. In other words, a misalignment between the position of the vehicle 10 in front, etc. when the display system renders the virtual object 20 and the position of the vehicle 10 in front when the virtual object 20 is actually displayed on the HUD causes the virtual object 20 and the vehicle 10 in front to be displayed misaligned. For example, even if the position of the vehicle 10 in front at a future display timing is predicted in the rendering process, there is a time lag between the prediction and the display, so there is a possibility that the predicted position of the vehicle 10 in front, etc. will be misaligned from the position of the vehicle 10 in front when the virtual object 20 is actually displayed on the HUD.

As an example, FIG. 2 shows an explanatory diagram of the display position shift due to pitch displacement or vertical displacement. xyz is a coordinate system fixed to the automobile 30. The z direction is the traveling direction of the automobile, and the x and y directions are perpendicular to the z direction and perpendicular to each other. When the y direction coincides with the vertical direction, the x direction is the horizontal direction, and the y direction is the vertical direction or up-down direction. When the automobile is in a position where the y direction does not coincide with the vertical direction, the x and y directions do not coincide horizontally or vertically, but for convenience, the x direction will be referred to as the horizontal direction and the y direction will be referred to as the vertical direction or up-down direction below. The direction of the arrow shown in the figure is the positive direction, and when it is desired to clearly indicate the positive and negative directions, they will be described as +x direction, -x direction, etc.

The first diagram in Figure 2 shows a state where there is no positional misalignment between the virtual object 21 and the real object 11. The real object 11 is an object that exists in real space and is the object that is the tracking target for the AR display. For example, the car in front 10 in Figure 1 is an example of a real object. The projection unit 36 of the HUD projects a virtual image onto the windshield 35, and the virtual image is seen by the driver's eyes 50 as the virtual object 21.

2 shows the positional deviation between the virtual object 21 and the real object 11 when the windshield 35 and the projection unit 36 shift by WSM in the +y direction. The vertical shift WSM occurs due to pitch rotation or vertical shift of the automobile 30. The pitch rotation is a rotation around an axis parallel to the x direction, and the vertical shift is a translation in the y direction. When the display system of the HUD renders the virtual object 21, the state shown in the first diagram is assumed, but since there is a time until it is displayed on the HUD, if a vertical shift WSM occurs during that time, the vertical shift WSM is not reflected in the rendered image. Therefore, the display position of the virtual object 21 is shifted in the +y direction by the vertical shift WSM of the windshield 35, etc., and the virtual object 21 appears to be shifted in the +y direction relative to the real object 11. For example, if the automobile 30 repeats pitch motion due to unevenness in the road, the virtual object 21 appears to be shifted relative to the real object 11, as if it is swaying up and down.

The third diagram in Figure 2 shows the positional deviation between the virtual object 21 and the real object 11 when the driver's eyes 50 shift in the +y direction by the DEM. The windshield 35 does not move, but the eyes 50 shift in the +y direction relative to the windshield 35, so the display position of the virtual object 21 shifts in the -y direction. As a result, the virtual object 21 appears to be shifted in the -y direction relative to the real object 11.

The fourth diagram in Figure 2 shows the positional deviation between the virtual object 21 and the real object 11 when the real object 11 shifts by RJM in the -y direction. The display position of the virtual object 21 does not shift because the windshield 35 and the eyes 50 do not move, but because the real object 11 shifts in the -y direction, the virtual object 21 appears to be shifted in the +y direction relative to the real object 11.

3 is a diagram for explaining latency compensation in this embodiment. FIG. 3 shows a virtual object 21, a real object 11, and a virtual object 21 as seen by the driver. The left diagram in FIG. 3 shows a state in which the display positions of the real object 11 and the virtual object 21 match as seen by the driver. As shown in the middle diagram in FIG. 3, when the driver moves his head to the upper left, the real object 11 shifts to the upper left by MV. Since the display position of the virtual object 21 does not change, the real object 11 and the virtual object 21 shift by MV. As shown in the right diagram in FIG. 3, in this embodiment, the display positions of the real object 11 and the virtual object 21 are matched by compensating for the shift amount MV due to latency. By performing such latency compensation, when the driver moves his head to the upper left, the state transitions from the left diagram to the right diagram, and the position shift state as shown in the middle diagram can be avoided. Details will be described later, but in this embodiment, latency compensation is performed in a warp process as close as possible to the display timing, thereby minimizing position shift due to latency as much as possible.

To consider what kind of latency exists in a HUD display system, we will explain a typical display system in the prior art and the latency that occurs in that display system.

Figure 4 shows an example of the configuration of a display system 90 in the prior art. The display system 90 includes a sensor 65, a processing device 60, a HUD controller 70, and a HUD 80.

Sensor 65 is a sensor that detects the position, posture, or motion of a vehicle, a driver, or a real object.

The processing device 270 includes a sensor interface 61, a tracking processing unit 62, and a rendering image generation unit 63. The sensor interface 61 is an interface circuit that receives the output signal of a sensor 65. The tracking processing unit 62 is configured with a CPU, and tracks the position, attitude, or motion of the automobile, driver, or real object based on the output signal of the sensor 65. The rendering image generation unit 63 is configured with a GPU, and renders a virtual object so that it is displayed following the real object based on the tracking information of the automobile, driver, or real object.

The HUD controller 70 includes a warp processing unit 71 that warps the rendering image. The HUD 80 includes a display 81 that displays the image after warping. The display 81 is a display device such as a liquid crystal display device. The image displayed on the display 81 is projected onto the windshield by an optical system, thereby displaying the virtual object of the rendering image.

Figure 5 shows the processing flow of the display system 90. Here, an example is shown in which an IMU and a camera are used as the sensor 65. The IMU performs sensing at a sampling rate that is sufficiently faster than the frame rate of the camera. Tck, Tck+1, and Tck+2 indicate the exposure timing of the camera. In other words, the image obtained by the camera indicates the position of the real object at the timing Tck, etc. Image signal processing is processing that is performed after the camera exposure ends, and includes signal reading from the image sensor, development processing, etc.

The tracking processing unit 62 tracks the position of the real object from the image after image signal processing by visual inertial odometry. At this stage, the position of the real object at the camera exposure timing tck is detected. The tracking processing unit 62 tracks the position of the real object by inertial odometry based on the IMU output signal at timing tod after the visual inertial odometry ends and the result of the visual inertial odometry. This process compensates for the latency from the camera exposure timing tck to the inertial odometry timing tod, and the position of the real object at timing tod is detected.

The rendering image generation unit 63 renders a virtual object at a position corresponding to the real object detected by inertial odometry, and outputs the rendering image to the warp processing unit 71. At this time, a data communication time occurs from the GPU constituting the rendering image generation unit 63 to the HUD controller 70. The warp processing unit 71 performs warp processing on the rendering image, and the image after warp processing is displayed on the HUD 80. The first line to the last line are horizontal scanning lines of the display 81, and are scanned one line at a time. Therefore, the time required to scan the first line to the last line is required to display one frame of image, but here, the timing of displaying the first line is representatively referred to as display timing tdp.

The virtual object displayed on the HUD 80 corresponds to the position of the real object at the timing tod. For this reason, there is a latency between the timing tod and the display timing tdp, and the motion of the car, driver, or real object during this period may cause a misalignment between the real object and the virtual object.

As described above, by displaying a virtual object rendered using tracking information detected before the display timing tdp, there is an issue that a display position shift occurs due to the latency between the timing at which the tracking information is detected and the display timing tdp.

In the above-mentioned Patent Document 1, the measurement time of the posture data is made shorter than the communication time of the image data, etc., and the projection optical unit uses the posture data to perform correction in the high frequency range, thereby suppressing the positional deviation between the real image and the projected image caused by the communication time. However, because data processing time is required to control the projection optical unit and the operation time of the components is required, the positional deviation caused by these times still remains.

Figure 6 shows a first configuration example of a circuit device 160 in this embodiment and a display system including the circuit device 160. The display system shown in Figure 6 includes a sensor 500, a processing device 150, the circuit device 160, and a HUD 400, and displays a virtual object corresponding to a real object on the HUD 400 based on the output signal of the sensor 500.

The sensor 500 is a sensor that detects the position, posture, or motion of a moving body, a viewer, or a real object. Specifically, the sensor 500 includes at least one of a first sensor that acquires two-dimensional or three-dimensional information of a real object, a second sensor that detects the motion of the moving body, and a third sensor that detects the viewpoint or line of sight of a viewer. The sensor 500 is provided on the moving body and includes, for example, a Lidar, an IMU, a camera, an eye tracking sensor, or a head tracking sensor. In this example, the Lidar or the camera corresponds to the first sensor, the IMU corresponds to the second sensor, and the eye tracking sensor or the head tracking sensor corresponds to the third sensor. Lidar stands for Light Detection and Ranging, and is a sensor that acquires three-dimensional information such as a z map. The IMU stands for Inertial Measurement Unit, and is a sensor that detects motion on one axis or multiple axes. The IMU is composed of, for example, an acceleration sensor, a gyro sensor, or a combination thereof. The camera is a sensor that captures an image that is two-dimensional information. An eye tracking sensor is a sensor that detects the position, gaze direction, or both of the viewer's eyes. A head tracking sensor is a sensor that detects the position, posture, or both of the viewer's head.

The moving body is an object that moves in real space carrying the HUD 400, the viewer, and the sensor 500, such as an automobile, motorcycle, airplane, or ship. The viewer is a user who sees the virtual image projected on the HUD 400, and is the operator or passenger of the moving body. The real object is an object that exists in real space. The real object may be any object in which the position or orientation of the real object changes in the HUD display area as seen by the viewer when the position or orientation of the moving body, the viewer, or the real object changes.

The sensor 500 includes a rendering sensor 510 for obtaining tracking information for rendering, and a latency compensation sensor 520 for obtaining latency compensation parameters. The rendering sensor 510 includes, for example, a Lidar, an IMU, and a camera. The latency compensation sensor 520 includes, for example, an IMU and a head tracking sensor. The types of sensors included in the rendering sensor 510 and the latency compensation sensor 520 are not limited to those described above, and may be any sensor that detects the position, posture, or motion of a moving body, a viewer, or a real object. The latency compensation sensor 520 and the rendering sensor 510 are different sensors, and even if they include the same type of sensor, the same type of sensor is provided separately in the latency compensation sensor 520 and the rendering sensor 510.

The processing device 150 is configured, for example, by a SoC, and includes an interface 151, a tracking processing unit 110, and a rendering image generating unit 120. SoC is an abbreviation for System on Chip.

The interface 151 is an interface for inputting the output signal of the rendering sensor 510 to the processing device 150. The interface 151 is, for example, a receiving circuit that receives the output signal of the rendering sensor 510. Alternatively, the interface 151 may be a terminal to which the output signal of the rendering sensor 510 is input. In this case, the output signal of the rendering sensor 510 input to the terminal is input to a processor constituting the tracking processing unit 110, and the processor receives and processes the output signal of the rendering sensor 510.

The tracking processing unit 110 tracks the position, posture, or motion of a moving body, viewer, or real object based on the output signal of the sensor 500, and outputs the result as tracking information. For example, the tracking processing unit 110 tracks a real object based on two-dimensional distance measurement information from a Lidar or two-dimensional images from a camera. The tracking processing unit 110 also tracks a car based on acceleration or angular velocity information from an IMU. The tracking processing unit 110 also tracks the driver's eyes based on eye position or gaze direction information from an eye tracking sensor.

The tracking information may be in any form as long as it indicates the position, attitude, or motion of a moving body, a viewer, or a real object. For example, the tracking information may be coordinates indicating a position in real space, an angle indicating an attitude, a vector indicating a parallel translation, or an angular velocity indicating a rotation. Alternatively, the tracking information may be information obtained by converting the coordinates in the real space into coordinates, angles, vectors, or angular velocities on an image. The tracking information includes first tracking information of a moving body, second tracking information of a viewer, and third tracking information of a real object. However, it is sufficient that the tracking information includes at least one of the first to third tracking information, and for example, the second tracking information of the viewer may be omitted.

The rendering image generation unit 120 renders a virtual object based on tracking information of a moving body, a viewer, or a real object, and outputs a rendering image including the virtual object. Specifically, the rendering image generation unit 120 determines the position at which the real object is visible in the display area of the HUD 400, and renders the virtual object at a position corresponding to the position of the real object. The rendering image generation unit 120 outputs the tracking information acquired by the tracking processing unit 110 together with the rendering image to the circuit device 160.

The tracking processing unit 110 and the rendering image generation unit 120 are configured with a processor such as a CPU or a GPU. CPU stands for Central Processing Unit. GPU stands for Graphical Processing Unit. The tracking processing unit 110 and the rendering image generation unit 120 may be realized by a single processor, or each may be realized by a separate processor.

The circuit device 160 is a so-called HUD controller, and performs distortion correction of the image displayed on the HUD 400, display timing control, etc. The circuit device 160 is configured, for example, by an integrated circuit device, and includes an interface 161, a warp processing unit 130, a parameter calculation unit 140, and a memory unit 145.

The interface 161 is an interface for inputting the rendering image and tracking information to the circuit device 160. Specifically, the rendering image and tracking information transmitted from the processing device 150 are input to the interface 161. In addition, the output signal of the latency compensation sensor 520 is input to the interface 161 as tracking information. The interface 161 is, for example, a receiving circuit that receives the rendering image, tracking information, and the output signal of the latency compensation sensor 520. Alternatively, the interface 161 may be a receiving circuit that receives the rendering image and tracking information, and a terminal to which the output signal of the latency compensation sensor 520 is input. In that case, the output signal of the latency compensation sensor 520 input to the terminal is input to the parameter calculation unit 140, and the parameter calculation unit 140 receives and processes the output signal of the latency compensation sensor 520.

The storage unit 145 is a semiconductor memory such as a RAM or a non-volatile memory, and stores the warp parameters before rendering compensation. RAM is an abbreviation for Random Access Memory. As described above, the warp parameters are a matrix or a table. The warp parameters may be either forward map parameters or inverse map parameters. The forward map parameters are parameters that associate each pixel of the input image with the destination coordinates corresponding to each pixel, or parameters that associate each pixel of the input image with the relative movement amount to the destination coordinates corresponding to each pixel. The inverse map parameters are parameters that associate each pixel of the output image with the reference source coordinates corresponding to each pixel, or parameters that associate each pixel of the output image with the relative movement amount from the reference source coordinates corresponding to each pixel.

The parameter calculation unit 140 determines latency compensation parameters based on the tracking information transmitted from the processing unit 150 together with the rendering image and the output signal of the latency compensation sensor 520. The parameter calculation unit 140 corrects the warp parameters read from the storage unit 145 based on the latency compensation parameters, and outputs the corrected warp parameters. The tracking information used here is sampled later than the tracking information used in the rendering process. From the viewpoint of minimizing latency, it is desirable to use tracking information obtained immediately before or as close as possible to the parameter calculation.

The latency compensation parameter is a parameter that compensates for the difference between the position of a virtual object in a rendered image and the position of the virtual object at the display timing. The latency compensation parameter indicates the amount of shift or the angle of rotation on the image data. More specifically, the latency compensation parameter indicates the amount of shift or the angle of rotation in an undistorted image before warping processing.

Warp parameters are parameters for coordinate transformation in warping processing, and are parameters that associate the coordinates of the input image with the coordinates of the output image in warping processing. Warp parameters are, for example, a matrix or table that represents the coordinate transformation between the input image and the output image.

The warp processing unit 130 warps the rendering image so that the rendering image is not distorted when it is projected onto a curved screen. Specifically, the warp processing unit 130 generates a display image by performing coordinate transformation on the rendering image using the corrected warp parameters output by the parameter calculation unit 140. This coordinate transformation is a coordinate transformation that cancels image distortion caused by the curvature of the screen. The display image is an image displayed on the display panel of the HUD 400, and is displayed as an undistorted virtual image by projecting the display image displayed on the display panel onto the screen by the projection unit.

The warp processing unit 130 and the parameter calculation unit 140 are composed of logic circuits, for example, an automatically placed and wired gate array or an automatically wired standard cell array.

HUD 400 includes a display 410, and displays a display image on display 410. Display 410 is a display device such as a liquid crystal display device. HUD 400 includes an optical system (not shown) for projecting a virtual image onto a screen, and the optical system projects the display image displayed on the display panel onto the screen as a virtual image, thereby superimposing a virtual object onto real space. The screen is a windscreen of a moving object, or the like. Alternatively, HUD 400 may include a dedicated screen onto which a virtual image is projected.

Figure 7 shows a detailed processing flow of the tracking processing unit 110, and Figure 8 shows a processing flow of the display system in the first configuration example. Note that the tracking process shown in Figure 7 is just one example, and the method for tracking real objects, etc. is not limited to this. In addition, the sensor used in the tracking process may be selected depending on the method to be adopted.

The basic operation of the sensor shown in FIG. 8 is as described in FIG. 5. However, in FIG. 8, IMU1 and the camera are the rendering sensor 510, and IMU2 is the latency compensation sensor 520.

The tracking processing unit 110 performs visual inertial odometry processing 111 based on the image IMG(tck) captured by the camera at the timing tck and the motion information MTI(tck) of the moving object sampled by the IMU1 at the timing tck, and estimates the position and orientation Pr(tck) of the real object and the moving object at the timing tck.

The tracking processing unit 110 performs inertial odometry processing 112 based on the position and orientation Pr(tck) and the motion information MTI(tnow1) of the moving body sampled by the IMU1 at the timing tnow1, and estimates the position and orientation Pc(tck, tnow1) of the real object and the moving body at the timing tnow1. tnow1 is after tck and before the start of the inertial odometry processing 112. More preferably, tnow1 is after the end of the visual inertial odometry processing 111 and before the start of the inertial odometry processing 112.

The tracking processing unit 110 performs prediction processing 113 based on the position and orientation Pc (tck, tnow1) to predict the position and orientation Pp (tck, tnow1, tfuture) of the real object and the moving body at a timing tfuture after tnow1. Figure 8 shows an example where tfuture = tdp and the position and orientation Pp (tck, tnow1, tdp) is predicted. tdp is the display timing of the HUD. The prediction processing 113 predicts the future position and orientation Pp, for example, based on the differential value of the position and orientation Pc obtained in time series.

The rendering image generation unit 120 renders a virtual object based on the position and orientation Pp (tck, tnow1, tdp) and outputs a rendering image. If this rendering image were displayed as is on the HUD 400, the virtual object would be displayed corresponding to the position of the real object indicated by the position and orientation Pp.

Here, the predicted position and orientation Pp (tck, tnow1, tdp) is predicted using information at tck and tnow1, and is not a position based on the actual sensor output at tdp. In other words, if the actual movement that occurs between tnow1 and tdp differs from the movement predicted by the prediction process 113, a position shift due to latency may occur between the virtual object and the real object. In other words, the latency to be compensated for in the example of FIG. 8 is the latency due to the processing time between tnow1 and tdp. Note that the latency to be compensated for is not limited to this. For example, if the inertial odometry and prediction process are omitted and the rendering process is performed using Pr(tck), the latency due to the processing time between tck and tdp is compensated for.

In this embodiment, the position shift due to the above latency is compensated for as follows. The tracking processing unit 110 performs visual inertial odometry processing 111 based on the image IMG(tck+1) captured by the camera at timing tck+1 and the motion information MTI(tck+1) of the moving body sampled by the IMU at timing tck+1, and estimates the position and orientation Pr(tck+1) of the real object and the moving body at timing tck+1. tck+1 is the next camera exposure timing after tck.

The rendering image generation unit 120 outputs the position and orientation Pp (tck, tnow1, tdp) and the position and orientation Pr (tck+1) to the circuit device 160 along with the rendering image rendered based on the position and orientation Pp (tck, tnow1, tdp).

The parameter calculation unit 140 performs inertial odometry processing based on the position and orientation Pr(tck+1) input via the interface 161 and the motion information MTI'(tnow2) of the moving body sampled by the IMU2 at the timing tnow2, and estimates the position and orientation Pc'(tck+1, tnow2) of the real object and the moving body at the timing tnow2.

The parameter calculation unit 140 performs prediction processing based on the position and orientation Pc'(tck+1, tnow2) and predicts the position and orientation Pp'(tck+1, tnow2, tdp) of the real object and the moving body at tdp.

The parameter calculation unit 140 calculates a latency compensation parameter based on the position and orientation Pp (tck, tnow1, tdp) and the position and orientation Pp' (tck+1, tnow2, tdp). Specifically, the parameter calculation unit 140 calculates a latency compensation parameter that cancels the position shift of the virtual object due to the difference between the position and orientation Pp (tck, tnow1, tdp) and the position and orientation Pp' (tck+1, tnow2, tdp). The parameter calculation unit 140 corrects the warp parameter using the latency compensation parameter and outputs the corrected warp parameter.

The warp processing unit 130 warps the rendering image using the corrected warp parameters to generate a display image. The warp processing unit 130 sequentially performs warp processing on the first line to the final line according to the scanning timing of the first line to the final line on the display panel of the HUD 400, and outputs the data of each line after the warp processing to the HUD 400 sequentially in accordance with the scanning timing. In FIG. 8, the display timing of the first line is set to tdp, but tdp may be any timing during the frame period from displaying the first line to the final line.

According to this embodiment, the warp processing unit 130 performs latency compensation processing in the warp processing, which compensates for the position of a virtual object in the display area based on a latency compensation parameter.

In this way, the positional deviation of the virtual object caused by the latency due to the rendering process between tnow1 and tdp is compensated for based on the latency compensation parameters. By performing latency compensation in the warp process performed immediately before display, the remaining latency can be minimized, making it possible to realize AR display in which the virtual object has high tracking ability with respect to the real object. Furthermore, in this embodiment, latency compensation is performed by image data processing rather than using an optical compensation mechanism as in Patent Document 1, so latency compensation can be realized without changing the configuration of the projection unit, etc. Furthermore, since latency compensation is performed as an integrated part in the warp process, accurate distortion correction and latency compensation can be realized.

In this embodiment, the interface 161 receives a rendering image generated based on the tracking information at the first timing and the tracking information at the first timing. The parameter calculation unit 140 calculates a latency compensation parameter based on the tracking information at the first timing and the tracking information at the second timing, which is after the first timing and before the image display timing tdp. In the example of FIG. 8, tnow1 corresponds to the first timing, and tnow2 corresponds to the second timing.

In this way, the latency between the first timing and the display timing is shortened to the latency between the second timing and the display timing, reducing positional deviation due to latency. To explain using the example of Figure 8, the latency compensation parameter that corrects the warp parameter reflects the motion information sampled by IMU2 at tnow2, which is after tnow1. By correcting the display position of the virtual object using this latency compensation parameter, the latency between tnow1 and tdp is shortened to the latency between tnow2 and tdp. Since the warp process is performed immediately before the HUD display, tnow2 is set close to tdp, making the latency as small as possible and reducing the display position deviation of the virtual object due to latency.

In this embodiment, the parameter calculation unit 140 corrects the warp parameters of the coordinate transformation corresponding to the curvature of the display area based on the latency compensation parameters, thereby outputting the corrected warp parameters. The warp processing unit 130 performs the latency compensation process by performing the warp process using the corrected warp parameters.

In this way, by performing warp processing using corrected warp parameters based on the latency compensation parameters, latency compensation can be performed in the warp processing. For example, a configuration in which image processing for latency compensation is performed separately from warp processing is conceivable, but the processing load increases by the amount of image processing for latency compensation, and performing warp processing after latency compensation results in the latency of the warp processing remaining. In this embodiment, latency compensation and warp processing are integrated, making it possible to minimize latency while suppressing an increase in processing load.

In this embodiment, the tracking information is information indicating a displacement including at least one of a first axis displacement and a second axis displacement perpendicular to the moving direction of the moving body. The moving direction corresponds to the z direction, the first axis corresponds to the x direction, and the second axis corresponds to the y direction. The parameter calculation unit 140 outputs the corrected warp parameters by correcting the error in the position of the virtual object caused by the above displacement.

Latency compensation compensates for changes in position or orientation that occur over a short period of time, but it is believed that the shaking of the moving body or the viewer has a large impact on changes in position or orientation that occur over a short period of time. When the moving body or the viewer shakes, it is believed that the real object that the viewer sees through the display area shifts. According to this embodiment, the error in the position of the virtual object caused by the first axis displacement and second axis displacement perpendicular to the moving direction of the moving body is corrected, so that the display position deviation caused by the shaking of the moving body or the viewer can be compensated for.

In this embodiment, of the first axis displacement and the second axis displacement perpendicular to the moving body's direction of travel, the displacement in the up-down direction of the moving body is the second axis displacement. At this time, the tracking information includes information indicating the second axis displacement of the HUD 400 or the viewer caused by the pitch displacement of the moving body. The parameter calculation unit 140 outputs the corrected warp parameters by correcting the error in the position of the virtual object caused by the second axis displacement.

The above-mentioned shaking of the moving body is assumed to be a pitch displacement, and the pitch displacement of the moving body causes a second-axis displacement of the HUD screen or the viewer. This second-axis displacement of the HUD screen or the viewer causes a shift displacement in the second axis direction in the real object that the viewer sees through the display area. According to this embodiment, the error in the position of the virtual object caused by the second-axis displacement of the HUD screen or the viewer is corrected, so that the display position deviation caused by the pitch displacement of the moving body can be compensated for.

In addition, in this embodiment, the warp processing unit 130 performs latency compensation processing in the warp processing, which compensates for at least one of the rotation and scale of the virtual object in the display area based on the latency compensation parameters.

In this way, latency compensation may be performed not only for shift displacement, but also for rotation or scale. Scaling refers to shrinking or zooming a virtual object. According to this embodiment, rotation error, scale error, or both of the virtual object due to latency are compensated for, making it possible to realize AR display with higher tracking performance.

In this embodiment, the interface 161 receives the rendering tracking information generated by the tracking process based on the output signal of the rendering sensor 510, the rendering image generated based on the rendering tracking information, and the latency compensation tracking information, which is the output signal of the latency compensation sensor 520, which is different from the rendering sensor 510. The parameter calculation unit 140 calculates the latency compensation parameter based on the rendering tracking information and the latency compensation tracking information. In the example of FIG. 8, Pp (tck, tnow1, tdp) corresponds to the rendering tracking information. The parameter calculation unit 140 also obtains Pp' (tck+1, tnow2, tdp) based on the output signal of the latency compensation sensor 520, which is the latency compensation tracking information, and calculates the latency compensation parameter from Pp (tck, tnow1, tdp) and Pp' (tck+1, tnow2, tdp).

In this way, the parameter calculation unit 140 can obtain the output signal of the latency compensation sensor 520 at any timing, and can obtain the output signal of the latency compensation sensor 520 at a timing as close as possible to the display timing tdp, and calculate the latency compensation parameters.

9 shows a second configuration example of the circuit device 160 according to the present embodiment and a display system including the circuit device 160. Note that components already described are given the same reference numerals, and descriptions of those components will be omitted as appropriate.

In FIG. 9, the tracking processing unit 110 outputs tracking information to the circuit device 160, and the rendering image generation unit 120 outputs a rendering image to the circuit device 160. The interface 161 of the circuit device 160 receives the rendering image from the rendering image generation unit 120 and the tracking information from the tracking processing unit 110.

Figure 10 shows the processing flow of the display system in the second configuration example. Note that the processing flow shown in Figure 10 is just an example, and the tracking method is not limited to that shown in Figure 10. In addition, the sensor used for tracking may be selected according to the tracking method to be adopted.

The process in which the tracking processing unit 110 predicts the position and orientation Pp (tck, tnow1, tdp), the process in which the rendering image generating unit 120 generates the rendering image, and the process in which the tracking processing unit 110 determines the position and orientation Pr (tck+1) are the same as in the first configuration example.

In the second configuration example, the tracking processing unit 110 performs inertial odometry processing 112 based on the position and orientation Pr(tck+1) and the motion information MTI(tnow2) of the moving body sampled by the IMU at timing tnow2, and estimates the position and orientation Pc(tck+1, tnow2) of the real object and the moving body at timing tnow2.

The tracking processing unit 110 performs prediction processing 113 based on the position and orientation Pc(tck+1, tnow2) and predicts the position and orientation Pp(tck+1, tnow2, tdp) of the real object and the moving body at tdp.

The parameter calculation unit 140 calculates a latency compensation parameter based on the position and orientation Pp(tck, tnow1, tdp) and the position and orientation Pp(tck+1, tnow2, tdp). Specifically, the parameter calculation unit 140 calculates a latency compensation parameter that cancels the position shift of the virtual object due to the difference between the position and orientation Pp(tck, tnow1, tdp) and the position and orientation Pp(tck+1, tnow2, tdp). The parameter calculation unit 140 corrects the warp parameter using the latency compensation parameter and outputs the corrected warp parameter.

The warp processing unit 130 warps the rendering image using the corrected warp parameters to generate a display image.

3. Warp Processing Warp processing will be described in detail below, taking inverse warp as an example. However, the warp processing may be either forward warp or inverse warp, and the warp processing unit 130 may be either a forward warp engine or an inverse warp engine. Forward warp is a transformation that moves each pixel of an input image to a warp engine to an arbitrary position in an output image. A forward warp engine is a warp engine that has a forward warp function. Inverse warp is a transformation that obtains each pixel of an output image of a warp engine from a pixel at an arbitrary position in an input image. An inverse warp engine is a warp engine that has an inverse warp function.

Figure 11 is an explanatory diagram of the warp process in this embodiment. The input image is a rendering image. The output image is a display image that is displayed on the liquid crystal display panel of the HUD or the like. (xsrc, ysrc) indicate the horizontal and vertical positions in the input image. (xdst, ydst) indicate the horizontal and vertical positions in the output image.

If the correction of the warp parameters using the latency compensation parameters is represented as g( ), the position in the latency compensation image is expressed as the following equation (1).
(xlat,ylat)=g(xsrc,ysrc)...(1)

If the distortion correction using the warp parameters according to the curvature of the screen is expressed as f( ), the position in the output image is expressed by the following equation (2).
(xdst,ydst)=f(xlat,ylat)...(2)

From the above equations (1) and (2), the coordinate transformation between the input image and the output image is expressed as the following equation (3).
(xdst, ydst) = f・g(xsrc, ysrc) (3)

Inverse warping uses the inverse transformation of equation (3) above, resulting in equation (4) below.
(xsrc, ysrc) = g ^-1・f ^-1 (xdst, ydst) ... (4)

In this embodiment, the parameter calculation unit 140 obtains a latency compensation parameter g ⁻¹ , corrects the warp parameter f ⁻¹ by calculating g ⁻¹ ·f ⁻¹ , and outputs the corrected warp parameter g ⁻¹ ·f ⁻¹ . The warp processing unit 130 performs the warp processing of the above formula (4) using the corrected warp parameter g ⁻¹ ·f ⁻¹ to generate an output image.

Figure 12 shows the correspondence between tracking information of a moving object, etc., and latency compensation based on that tracking information. Here, it is assumed that six-axis motion information is obtained as tracking information.

The yaw displacement Δα is a rotational displacement about an axis parallel to the vertical y direction. The yaw displacement Δα causes a horizontal shift in the real object seen by the viewer through the screen. The horizontal shift is a shift in the horizontal x direction. For this reason, the latency compensation technique is a horizontal shift.

The pitch displacement Δβ is a rotational displacement about an axis parallel to the horizontal x-direction. The pitch displacement Δβ causes a vertical shift in the real object seen by the viewer through the screen. The vertical shift is a shift in the vertical y-direction. For this reason, the latency compensation technique is a vertical shift.

The roll displacement Δγ is a rotational displacement with an axis parallel to the z-direction, which is the forward and backward direction of the moving object, as the axis of rotation. The roll displacement Δγ causes a rotation in the real object seen by the viewer through the screen. For this reason, the latency compensation method is rotation.

The horizontal shift Δx is a shift in the x direction, which causes a horizontal shift in the real object that the viewer sees through the screen. Therefore, the latency compensation technique is a horizontal shift.

The vertical shift Δy is a shift in the y direction, which causes a vertical shift in the real object that the viewer sees through the screen. Therefore, the latency compensation technique is a vertical shift.

The fore-aft shift Δz is a shift in the z-direction, which causes a shrinking or enlargement of the real object seen by the viewer through the screen. Hence, the latency compensation technique is a shrinking or zooming.

FIG. 13 shows an example of latency compensation parameters that realize the compensation method. The matrix on the right side of the equation shown in FIG. 13 corresponds to g ⁻¹ described in FIG. 11. m13 indicates a horizontal shift, and m23 indicates a vertical shift. m12 indicates a horizontal skew. The horizontal skew is a transformation that distorts a rectangle into a parallelogram, but is used here as an approximation of rotation. m11 indicates a horizontal scale, and m22 indicates a vertical scale. m11 and m22 correspond to reduction or zoom.

For example, when only a vertical shift occurs, m11=m22=1, and m12=m13=0. In this case, the equation in FIG.
(xsrc, ysrc) = (xlat, ylat+m23) ... (5)

11, the inverse transformation f ⁻¹ ( ) of the distortion correction moves (xdst, ydst) to (xlat, ylat). The above equation (5) corresponds to shifting this amount by m23 in the vertical direction, and latency compensation is realized by this shift.

Figure 14 illustrates a method for calculating latency compensation parameters from tracking information.

The displacement of the moving body, the viewer, or the real object can be converted into a shift amount or rotation angle on the screen based on the geometric positional relationship between the centers of rotation of the yaw, roll, and pitch, the viewer's head, the HUD screen, and the real object. Then, the matrix on the right side of Figure 13 can be determined from the shift amount or rotation angle.

In FIG. 14, a method for obtaining latency compensation parameters from tracking information is explained using m23, which indicates a vertical shift in FIG. 13, as an example. Here, an example is explained in which the pitch displacement Δβ and vertical shift Δym of the moving object 32 and the vertical shift Δyp of the viewer 52 are used as tracking information.

If the distance between the pitch rotation center PTC and the screen 34 is DCF, then the pitch displacement Δβ of the moving body 32 shifts the screen 34 vertically by DCF×Δβ. Also, the vertical shift Δym of the moving body 32 shifts the screen 34 vertically by Δym. Adding these together, the vertical shift amount of the screen 34 is DCF×Δβ+Δym. If we assume that the head of the viewer 52 and the real object 12 do not move, for example, when the screen 34 is raised by +(DCF×Δβ+Δym), the position of the real object 12 on the screen 34 as seen by the viewer appears to be lowered by -(DCF×Δβ+Δym) relative to the screen 34. Therefore, the latency compensation parameter in this case is m23=-(DCF×Δβ+Δym).

Let DPT be the distance between the head of the viewer 52 and the real object 12, and DFT be the distance between the screen 34 and the real object 12. Assuming that the screen 34 and the real object 12 do not move, for example, when the viewer moves up by +Δyp, the position of the real object 12 on the screen 34 as seen by the viewer 52 appears to move up by +(DFT/DPT)×Δyp relative to the screen 34. If we assume that the real object 12 is sufficiently far away, we can approximate DFT/DPT=1, so the vertical shift amount of the real object 12 is Δyp. Therefore, the latency compensation parameter in this case is m23=Δyp.

According to this embodiment, the circuit device 160 includes a storage unit 145, which stores a table that associates coordinates in a display image with a movement amount indicating the movement destination of the coordinates as a warp parameter. The parameter calculation unit 140 shifts the movement amount of the table based on the latency compensation parameter, and outputs the table with the shifted movement amount as a compensated warp parameter.

In the example of Fig. 11, f ^-1 () is stored as a table in the storage unit 145. In the example described in Fig. 13 and the above formula (5), the amount of movement of the table that moves (xdst, ydst) to (xlat, ylat) is shifted vertically by, for example, m23. That is, the table is corrected to move (xdst, ydst) to (xlat, ylat + m23).

Alternatively, the storage unit 145 may store, as warp parameters, parameters of a transformation equation that converts coordinates in the display image into destination coordinates. The parameter calculation unit 140 shifts the destination coordinates based on the latency compensation parameters, and outputs, as compensated warp parameters, the parameters of the transformation equation in which the destination coordinates have been shifted.

The conversion equation is, for example, a polynomial, and the coefficients of the polynomial are stored in the storage unit 145 as parameters of the conversion equation. In the example of Fig. 11, the parameters of the conversion equation indicating f ^-1 () are stored in the storage unit 145. In the example described in Fig. 13 and the above formula (5), the amount of movement of the conversion equation that moves (xdst, ydst) to (xlat, ylat) is shifted vertically by, for example, m23. That is, the conversion equation is corrected to one that moves (xdst, ydst) to (xlat, ylat+m23).

15 shows an example of the configuration of an electronic device 300 including the circuit device 160 of the present embodiment. The electronic device 300 includes a processing device 150, a circuit device 160, a storage device 350, an operation device 360, a communication device 370, a head-up display 400, and a sensor 500.

The processing device 150 outputs a rendering image obtained by rendering a virtual object to the circuit device 160. The processing device 150 may also output a rendering image obtained by combining image data stored in the storage device 350 or image data received by the communication device 370 with a virtual object to the circuit device 160. The circuit device 160 performs latency compensation along with warping processing on the rendering image. The circuit device 160 also performs display timing control, etc. The head-up display 400 drives the display panel based on the image data transferred from the circuit device 160 and the display timing control by the circuit device 160, and displays an image on the display panel. The display panel is, for example, a liquid crystal display panel or an EL display panel. The head-up display 400 projects the image displayed on the display panel onto a screen by a projection optical unit. The storage device 350 is, for example, a memory, a hard disk drive, an optical disk drive, etc. The operation device 360 is a device for a user to operate the electronic device 300, and is, for example, a button, a touch panel, or a keyboard. The communication device 370 is, for example, a device that performs wired communication or wireless communication. Examples of wired communication include a LAN or a USB. Examples of wireless communication include a wireless LAN or wireless proximity communication.

Electronic devices including the circuit device of this embodiment can be various devices such as electronic devices for vehicles, display terminals for factory equipment, display devices mounted on robots, and information processing devices. An example of an electronic device for vehicles is a meter panel. An example of an information processing device is a PC.

Figure 16 is an example of a moving body including the circuit device 160 of this embodiment. The moving body includes a control device 208, a head-up display 400, and a sensor 500. The control device 208 includes a processing device 150 and a circuit device 160. The control device 208 is an ECU (Electronic Control Unit), and the processing device 150 and the circuit device 160 are incorporated into the ECU. The circuit device 160 may be incorporated into the head-up display 400. The circuit device 160 of this embodiment can be incorporated into various moving bodies such as cars, airplanes, motorcycles, bicycles, or ships. The moving body is, for example, a device or equipment that is equipped with a driving mechanism such as an engine or a motor, a steering mechanism such as a handle or a rudder, and various electronic devices and moves on the ground, in the air, or on the sea. Figure 16 shows a schematic diagram of an automobile 206 as a specific example of a moving body. The automobile 206 has a body 207 and wheels 209. The ECU controls the running of the automobile 206 by controlling the engine, brakes, steering, etc. based on the operation by the driver. The head-up display 400 has a transparent screen, and the transparent screen is installed between the driver's seat and the windshield. Alternatively, the head-up display may use the windshield as a transparent screen and project an image onto the windshield. The head-up display 400 may perform AR display and may also function as, for example, a meter panel of the automobile 206.

The circuit device of the present embodiment described above performs display control of a head-up display that displays a virtual object corresponding to a real object in a real space in a display area. The circuit device includes an interface, a parameter calculation unit, and a warp processing unit. The interface receives tracking information, which is at least one of first tracking information of a moving object on which the head-up display is mounted, second tracking information of a viewer of the head-up display, and third tracking information of a real object, and a rendered image including a virtual object. The parameter calculation unit calculates a latency compensation parameter that compensates for latency including a rendering processing latency of the rendered image based on the tracking information. The warp processing unit performs a warp process that converts the coordinates of the rendered image in accordance with the curvature of the display area, and generates a display image that is displayed in the display area. In the warp process, the warp processing unit performs a latency compensation process that compensates for the position of the virtual object in the display area based on the latency compensation parameter.

In this way, positional deviations of virtual objects caused by latency, including rendering process latency, are compensated for based on the latency compensation parameters. By performing latency compensation in the warp process performed immediately before display, the remaining latency can be minimized, making it possible to realize AR display in which virtual objects closely track real objects.

In the present embodiment, a rendering image generated based on the tracking information at the first timing and the tracking information at the first timing may be input to the interface. The parameter calculation unit may calculate a latency compensation parameter based on the tracking information at the first timing and the tracking information at a second timing that is after the first timing and before the display timing of the display image.

In this way, the latency between the first timing and the display timing is reduced to the latency between the second timing, which is after the first timing, and the display timing, thereby reducing positional deviations due to latency.

In addition, in this embodiment, the parameter calculation unit may output corrected warp parameters by correcting warp parameters of coordinate transformation according to the curvature of the display area based on the latency compensation parameters. The warp processing unit may perform latency compensation processing by performing warp processing using the corrected warp parameters.

In this way, by performing warp processing using corrected warp parameters based on the latency compensation parameters, latency compensation can be performed in the warp processing.

In the present embodiment, the tracking information may be information indicating a displacement including at least one of a first axis displacement and a second axis displacement perpendicular to the traveling direction of the moving body. The parameter calculation unit may output a corrected warp parameter by correcting an error in the position of the virtual object caused by the displacement.

Latency compensation compensates for changes in position or orientation that occur over a short period of time, but it is believed that the shaking of the moving body or the viewer has a large impact on changes in position or orientation that occur over a short period of time. When the moving body or the viewer shakes, it is believed that the real object that the viewer sees through the display area shifts. According to this embodiment, the error in the position of the virtual object caused by the first axis displacement and second axis displacement perpendicular to the moving direction of the moving body is corrected, so that the display position deviation caused by the shaking of the moving body or the viewer can be compensated for.

In the present embodiment, of the first axis displacement and the second axis displacement perpendicular to the traveling direction of the moving body, the displacement in the up-down direction of the moving body may be the second axis displacement. In this case, the tracking information may be information indicating the second axis displacement of the head-up display or the viewer caused by the pitch displacement of the moving body. The parameter calculation unit may output the corrected warp parameters by correcting the error in the position of the virtual object caused by the second axis displacement.

The above-mentioned shaking of the moving body is assumed to be a pitch displacement, and the pitch displacement of the moving body causes a second-axis displacement of the screen of the head-up display or the viewer. This second-axis displacement of the screen of the head-up display or the viewer causes a shift displacement in the second axis direction in the real object that the viewer sees through the display area. According to this embodiment, the error in the position of the virtual object caused by the second-axis displacement of the screen of the head-up display or the viewer is corrected, so that the display position deviation caused by the pitch displacement of the moving body can be compensated for.

In this embodiment, the circuit device may also include a storage unit that stores, as warp parameters, a table that associates coordinates in a display image with a movement amount indicating the destination of the coordinates. The parameter calculation unit may shift the movement amount of the table based on the latency compensation parameter, and output the table with the shifted movement amount as a corrected warp parameter.

In this way, the amount of movement of the table is shifted based on the latency compensation parameters, and the warp process is performed using the corrected table, thereby achieving latency compensation.

In this embodiment, the circuit device may also include a storage unit that stores, as warp parameters, parameters of a transformation equation that converts coordinates in the display image into destination coordinates. The parameter calculation unit may shift the destination coordinates based on the latency compensation parameters, and output, as corrected warp parameters, the parameters of the transformation equation in which the destination coordinates have been shifted.

In this way, the destination coordinates of the transformation formula are shifted based on the latency compensation parameters, and the warp process is performed using the corrected transformation formula, thereby achieving latency compensation.

In addition, in this embodiment, the warp processing unit may perform latency compensation processing in the warp processing, which compensates for at least one of the rotation and scale of the virtual object in the display area based on a latency compensation parameter.

In this way, latency compensation may be performed not only for shift displacement, but also for rotation or scale. Scaling refers to shrinking or zooming a virtual object. According to this embodiment, rotation error, scale error, or both of the virtual object due to latency are compensated for, making it possible to realize AR display with higher tracking performance.

In addition, in this embodiment, the interface may receive rendering tracking information generated by a tracking process based on the output signal of the rendering sensor, a rendering image generated based on the rendering tracking information, and latency compensation tracking information, which is an output signal of a latency compensation sensor separate from the rendering sensor. The parameter calculation unit may calculate a latency compensation parameter based on the rendering tracking information and the latency compensation tracking information.

In this way, the parameter calculation unit can obtain the output signal of the latency compensation sensor at any timing, so it can obtain the output signal of the latency compensation sensor at a timing as close as possible to the display timing and calculate the latency compensation parameters.

The electronic device of this embodiment also includes any of the circuit devices described above.

The moving body of this embodiment also includes any of the circuit devices described above.

Although the present embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible that do not substantially deviate from the novel matters and effects of the present disclosure. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader or similar meaning may be replaced with that different term anywhere in the specification or drawings. All combinations of the present embodiment and modifications are also included in the scope of the present disclosure. The configurations and operations of the sensor, head-up display, processing device, head-up display controller, display system, electronic device, and mobile object are not limited to those described in the present embodiment, and various modifications are possible.

10...vehicle in front, 11, 12...real object, 20, 21...virtual object, 30...car, 32...moving object, 34...screen, 35...windshield, 36...projection unit, 40...display area, 50...eye, 52...viewer, 110...tracking processing unit, 111...visual inertial odometry processing, 112...inertial odometry processing, 113...prediction processing, 120...rendering image generation unit, 130...warp processing unit, 140...parameter calculation Calculation unit, 145...Memory unit, 150...Processing device, 151...Interface, 160...HUD controller, 161...Interface, 206...Automobile, 208...Control device, 270...Processing device, 300...Electronic device, 350...Storage device, 360...Operation device, 370...Communication device, 400...Head-up display, 410...Display, 500...Sensor, 510...Rendering sensor, 520...Latency compensation sensor

Claims (10)

A circuit device for controlling a head-up display that displays a virtual object corresponding to a real object in a real space in a display area,
an interface into which tracking information, which is at least one of first tracking information of a moving object on which the head-up display is mounted, second tracking information of a viewer of the head-up display, and third tracking information of the real object, and a rendering image including the virtual object are input;
a parameter calculation unit that calculates a latency compensation parameter that compensates for a latency including a rendering process latency of the rendering image based on the tracking information;
a warp processing unit that performs a warp process on the rendering image by performing a coordinate transformation to correct image distortion due to curvature of the display area, and generates a display image to be displayed in the display area;
Including,
The parameter calculation unit is
correcting a warp parameter of the coordinate transformation based on the latency compensation parameter to output a corrected warp parameter;
The warp processing unit includes:
A circuit device comprising: a circuit device that performs a latency compensation process for compensating for a position of the virtual object in the display area by performing the coordinate transformation using the corrected warp parameters . 2. The circuit device according to claim 1 ,
a storage unit that stores a table that associates coordinates in the display image with amounts of movement that indicate destinations of the coordinates as warp parameters;
The parameter calculation unit includes:
A circuit device comprising: a circuit for shifting the movement amount of the table based on the latency compensation parameter; and outputting the table with the shifted movement amount as the corrected warp parameter. 2. The circuit device according to claim 1 ,
a storage unit that stores parameters of a transformation equation for transforming coordinates in the display image into coordinates of a destination as warp parameters;
The parameter calculation unit is
A circuit device comprising: a circuit for shifting the destination coordinates based on the latency compensation parameters; and outputting the transformation equation parameters with the destination coordinates shifted as the corrected warp parameters. 4. The circuit device according to claim 1,
the rendering image generated based on the tracking information at a first timing and the tracking information at the first timing are input to the interface;
The parameter calculation unit is
A circuit device characterized in that the latency compensation parameter is calculated based on the tracking information at a second timing that is later than the first timing and earlier than the display timing of the display image, and the tracking information at the first timing. 5. The circuit device according to claim 1 ,
The tracking information is
information indicating a displacement including at least one of a first axial displacement and a second axial displacement perpendicular to a traveling direction of the moving body,
The parameter calculation unit is
A circuit device that outputs the corrected warp parameters by correcting an error in the position of the virtual object caused by the displacement. 5. The circuit device according to claim 1 ,
When the displacement in the up-down direction of the moving body is the second axis displacement, out of the first axis displacement and the second axis displacement perpendicular to the traveling direction of the moving body,
The tracking information is
information indicating a second axis displacement of the head-up display or the viewer due to a pitch displacement of the moving object;
The parameter calculation unit is
A circuit device comprising: a first circuit portion for correcting a position error of the virtual object caused by the second axis displacement, and outputting the corrected warp parameters. 7. The circuit device according to claim 1,
The warp processing unit includes:
The circuit device is characterized in that, in the warp process, a latency compensation process is performed in which at least one of a rotation and a scale of the virtual object in the display area is compensated for based on the latency compensation parameter. 8. The circuit device according to claim 1,
the interface receives, as input, rendering tracking information generated by a tracking process based on an output signal of a rendering sensor, the rendering image generated based on the rendering tracking information, and latency compensation tracking information which is an output signal of a latency compensation sensor separate from the rendering sensor;
The parameter calculation unit is
A circuit device for calculating the latency compensation parameter based on the rendering tracking information and the latency compensation tracking information. An electronic device comprising the circuit device according to claim 1 . A moving object comprising the circuit device according to any one of claims 1 to 8 .

Images