JP2021145287A - Display control device and display control method - Google Patents

Abstract

To display an image that takes into consideration a human sense.SOLUTION: A display control device disclosed herein includes a determination unit that determines whether it is necessary to transform an image using vection control, a control amount calculation unit that calculates a vection control amount for transforming the image using vection control when the determination unit determines that it is necessary to transform the image using vection control, and an image generation unit that generates an image in which a predetermined area is deformed on the basis of the vection control amount calculated by the control amount calculation unit.SELECTED DRAWING: Figure 5

Description

The present disclosure relates to a display control device and a display control method.

Systems have been developed to remotely control moving objects such as robots and drones while watching the images displayed on the display.

JP-A-2019-89512

Yasuaki Tamada, “Recent Trends in Vection Research”, Science of Vision, 2017, Vol. 38, No. 1, p.5-10 "Vection", [online], NEXCO Central Japan, [Search on February 25, 2nd year of Reiwa], Internet <URL: https://www.c-nexco.co.jp/jam/cause/measure05.html> "Slow down on the road with polka dots? First on a general road, introduced as an accident countermeasure on a mountain pass What is the effect?", [Online], 2018.08.27, Vehicle News Editorial Department, [Search on February 25, 2nd year of Reiwa ], Internet <URL: https://trafficnews.jp/post/81300> “VR gameplay“ Eagle Flight ”Become an eagle and fly around the city of Paris! vol.1 ”, [online], 2016/10/30, Kuo's VR and games, [Search on February 25, 2nd year of Reiwa], Internet <URL: https://www.youtube.com/watch?v = Ob # SgQytsWg ＞ Paul Bourke, “Converting dual fisheye images into a spherical (equirectangular) projection”, [online], August 2016, [Searched on February 25, 2nd year of Reiwa], Internet <URL: http://paulbourke.net/dome/ dualfish2sphere />

When remote control or the like is performed while viewing an image displayed on a display or the like, so-called VR (Virtual Reality) sickness may occur and work efficiency may decrease.

It is desirable to provide a display control device and a display control method capable of displaying an image in consideration of human senses.

The display control device according to the embodiment of the present disclosure has a determination unit that determines whether or not it is necessary to transform an image using vection control, and a determination unit that transforms an image using vection control. When it is determined that it is necessary, it is determined based on the control amount calculation unit that calculates the vection control amount for deforming the image using the vection control and the vection control amount calculated by the control amount calculation unit. It is provided with an image generation unit that generates an image obtained by transforming the area of.

According to the display control method according to the embodiment of the present disclosure, it is necessary to determine whether or not it is necessary to perform image transformation using vection control and to perform image transformation using vection control. When it is determined, the vection control amount for deforming the image using the vection control is calculated, and the image in which a predetermined area is deformed is generated based on the calculated vection control amount. including.

In the display control device or the display control method according to the embodiment of the present disclosure, when it is determined that it is necessary to transform the image using the vection control, the image is transformed using the vection control. The vection control amount is calculated, and an image in which a predetermined area is deformed is generated based on the calculated vection control amount.

It is a block diagram which shows the 1st example of a remote control system. It is a block diagram which shows the 2nd example of a remote control system. It is explanatory drawing which shows an example of the technique using vection. It is a block diagram which shows the outline of the display system including the display control device which concerns on 1st Embodiment of this disclosure. It is a block diagram which shows the detailed configuration example of the display system which concerns on 1st Embodiment. It is explanatory drawing which shows an example of the image in a normal state without distortion, which is displayed on the display part in the display system which concerns on 1st Embodiment. It is explanatory drawing which shows an example of the image after transformation by the vection control, which is displayed on the display part in the display system which concerns on 1st Embodiment. It is explanatory drawing about the angle of view of the peripheral visual field region in the image after transformation by the vection control, which is displayed on the display part in the display system which concerns on 1st Embodiment. It is explanatory drawing about the angle of view of the peripheral visual field region, etc. when the line-of-sight direction changes with respect to FIG. It is a flow chart which shows typically an example of the operation flow of the display control using vection in the display system which concerns on 1st Embodiment. FIG. 5 is a flow chart schematically showing an example of a determination operation of the necessity of vection control by a vection control determination unit in the display control device according to the first embodiment. It is explanatory drawing which shows the 1st calculation example of the field of view enlargement ratio according to the speed by the vection control amount calculation unit in the display control apparatus which concerns on 1st Embodiment. It is explanatory drawing which shows the 2nd calculation example of the field of view enlargement ratio according to the speed by the vection control amount calculation unit in the display control apparatus which concerns on 1st Embodiment. It is explanatory drawing which shows an example of the coordinate transformation of the fisheye image based on the field of view magnification. It is explanatory drawing which shows typically an example of the situation where the operation target is moving a place away from the ground. It is explanatory drawing which shows typically an example of the situation where the operation target is moving a place close to the ground. It is explanatory drawing which shows typically an example of the situation where the operation target is moving the place near a wall. It is a flow chart which shows typically one modification of the operation of determining the necessity of vection control by the vection control determination unit in the display control device which concerns on 1st Embodiment. It is explanatory drawing which shows the 1st transformation example of the image after transformation by the vection control. It is explanatory drawing which shows the 2nd transformation example of the image after transformation by the vection control. It is explanatory drawing which shows the modification of the region which performs vection control.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The explanation will be given in the following order.
0. Comparative example (Figs. 1 to 3)
1. 1. First Embodiment (FIGS. 4 to 21)
1.1 Configuration 1.2 Operation 1.3 Modification example 1.4 Effect 2. Other embodiments

<0. Comparative example>
FIG. 1 shows a first example of a remote control system. FIG. 2 shows a second example of a remote control system.

With the development of communication infrastructure represented by 5G, the development of technology that enables remote real-time operation of devices such as robots, robot arms, cars, and drones using displays and remote control devices has been developed. It is desired. In addition, it is desired to develop a technology that enables remote real-time work instructions to people in remote areas and real-time observation of remote areas using robots and drones. There is. For example, as shown in FIG. 1, it is desired to develop a technique for the operator 100 to remotely control the robot 101 or the like using the HMD (head-mounted display) 102. Further, as shown in FIG. 2, it is desired to develop a technique for the operator 110 to display an image of a remote place by using the large screen display 111 and perform work or the like.

When such remote control is performed, as typified by VR sickness, sickness due to sensory disagreement may occur, making it difficult to continue the work or the like, and reducing work efficiency. This sensory disagreement is a phenomenon in which, for example, the operator (observer) himself is not actually moving, but is shown a moving image, and the contradiction causes the discomfort of sickness such as VR sickness. be. Therefore, it is possible to reduce the influence of sickness due to sensory inconsistency by reducing the sensation that the operator is moving.

In addition, while an immersive image such as when using an HMD has a sense of presence, the immersive feeling is so strong that when an image that seems to be moving at a high speed is presented to the operator, The operator feels a sense of fear and it becomes difficult to work efficiently. On the contrary, when a remote operator operates a moving object such as a drone at an unnecessarily high speed, it may cause danger or anxiety to people around the moving object. In this case, it is desirable to guide the operator to slow down the moving object without reducing the operator's attention.

On the other hand, regarding human kinesthetic sensation, there is a technique focusing on vection (visually induced self-motor sensation). Here, vection is a phenomenon in which one is not actually moving, but one feels as if one is moving. For example, when the next train starts to move while riding on a stationary train, it gives the illusion that the stationary train is moving. Vection can also give the illusion of speed when moving as well as when stationary.

FIG. 3 shows an example of a technique using vection. For example, as shown in FIG. 3, the vehicle moves faster than it actually is by continuously painting a plurality of round dots 121 on the road surface or continuously displaying a plurality of light emitting points on the side of the road. It is used as an example of inducing the driver to slow down by giving the driver the illusion of doing so (see Non-Patent Documents 2 and 3).

Patent Document 1 proposes a technique for preventing overspeed by superimposing light spots in the field of view while driving a vehicle by using a head-up display. Further, in the technique of Non-Patent Document 4, by darkening a part of the image of the peripheral visual field in the visual field and blocking the optical flow of the peripheral visual field, the influence of sensory mismatch due to vection is reduced and VR sickness is prevented. ing. A common problem with these technologies using vection is that the obstruction of the visual field by superimposing a light spot on the visual field or blocking a part of the visual field hinders the operation. Further, for example, in a VR game, the AR (Augmented Reality) display of light spots and dots is unnecessary for the world view originally presented by the content, and reduces the immersive feeling in the content. Also, in VR games, there is a technology to deal with the problem of sensory disagreement due to vection by instantaneously warping from a first point to a second point at a distance without moving continuously. However, it is not easy to warp a real-world image in the case of a normal camera image.

Further, as a human visual perception characteristic, it is well known that although the perception characteristic for a moving object is high in the peripheral visual field, the image cannot be seen at a high resolution. As an example of applying this, there is a method called fobeated rendering. In this method, the resolution of the peripheral visual field is reduced, and the image is rendered at a high resolution only in the vicinity of the area of gaze, thereby reducing the rendering processing cost without giving a sense of discomfort to humans.

Therefore, it is desired to develop a technique for appropriately controlling a human sense, particularly a sense of speed that a person feels, by utilizing vection without unnecessarily obstructing the field of view of an operator (image observer) or the like.

<1. First Embodiment>
[1.1 Configuration]
(Overview of display system)
FIG. 4 shows an outline of a display system including the display control device 1 according to the first embodiment of the present disclosure. FIG. 5 shows a detailed configuration example of the display system according to the first embodiment.

The display system according to the first embodiment can be applied to, for example, a remote control system for a moving body (operation target) such as a robot or a drone described in the above comparative example. Further, it can be applied not only to a real remote control system or the like but also to a virtual reality system such as a VR game.

The display system according to the first embodiment includes an input unit 10, a display control device 1, and a display unit 40. The display control device 1 includes a control unit 20 and a processing unit 30.

The input unit 10 includes, for example, an image pickup unit 11, a sensor unit 12, and a transmission unit 13. The image pickup unit 11 is composed of, for example, a wide-angle camera. The sensor unit 12 includes, for example, a speed sensor, an acceleration sensor, a distance sensor, and the like. Further, the sensor unit 12 may include a line-of-sight detection sensor capable of eye tracking, a detection sensor capable of head tracking, and the like. When applied to a remote control system, for example, the image pickup unit 11 and a part of the sensor unit 12 (speed sensor, acceleration sensor, distance sensor, etc.) are provided on the moving body to be operated.

The input unit 10 captures, for example, a wide viewing angle image in the image pickup unit 11, and transmits the image data to the control unit 20 by the transmission unit 13 by wire or wirelessly. Further, the input unit 10 may detect the speed or angular velocity of the operation target by the sensor unit 12 and transmit the detection data to the control unit 20. Further, the input unit 10 may detect the distance to the surrounding obstacle by the sensor unit 12 and transmit the distance information to the control unit 20. Further, the input unit 10 may detect the line of sight of the operator (observer of the image), the posture of the head, and the like by the sensor unit 12, and transmit the tracking information to the control unit 20.

The control unit 20 includes, for example, a reception unit 21, a vection control determination unit 22, and a vection control amount calculation unit 23.
The vection control determination unit 22 corresponds to a specific example of the “determination unit” in the technique of the present disclosure. The vection control amount calculation unit 23 corresponds to a specific example of the “control amount calculation unit” in the technique of the present disclosure.

The control unit 20 receives video data, detection data, and the like transmitted from the transmission unit 13 of the input unit 10 by wire or wirelessly by the reception unit 21. The control unit 20 determines whether or not it is necessary to perform vection control by the vection control determination unit 22. When the control unit 20 determines that it is necessary to perform the vection control by the vection control determination unit 22, the control unit 20 calculates the vection control amount of the video according to the situation by the vection control amount calculation unit 23. The vection control amount includes, for example, information on the visual field enlargement ratio M and the gazing point 51, which will be described later. As will be described later, the vection control amount calculation unit discriminates between the stable gaze area 50 and the peripheral visual field area 60 based on the information of the gaze point 51 with respect to the image, and sets the peripheral visual field area 60 as a predetermined area. Calculate the vection control amount. The control unit 20 outputs the vection control amount calculated by the vection control amount calculation unit 23 to the processing unit 30.

The processing unit 30 has, for example, a video generation unit 31. The processing unit 30 generates an image by the image generation unit 31 so as to realize a peripheral visual field and a gaze area according to the vection control amount from the vection control amount calculation unit 23 of the control unit 20.

The display unit 40 is composed of, for example, an HMD. The display unit 40 presents the video generated by the video generation unit 31 of the processing unit 30 to the operator.

[1.2 Operation]
(Specific example of vection control)
Here, the imaging unit 11 of the input unit 10 is, for example, a wide-angle fisheye camera having a shooting angle of view of 260 °. Further, it is assumed that the display unit 40 is an HMD and the drone is moved and controlled.

FIG. 6 shows an example of an image in a normal state without distortion displayed on the display unit 40.
FIG. 7 shows an example of the image displayed on the display unit 40 after being deformed by the vection control.

As explained in the above comparative example, human beings have an illusion of the amount of movement due to the influence of vection. In the display system according to the first embodiment, when it is determined by the vection control determination unit 22 that it is necessary to perform vection control, the display unit 40 displays an image after deformation by vection control as shown in FIG. indicate. The image is deformed in a predetermined area. The predetermined region is, for example, the peripheral visual field region 60. As the image after the deformation, an image in a state in which the stable visual field region 50 near the gazing point 51 and the peripheral visual field region 60 have different visual field enlargement ratios M, which will be described later, is displayed. For example, as shown in FIG. 7, in the stable gaze area 50 near the gaze point 51, an image in a normal state without distortion is displayed. Further, in the peripheral visual field region 60, a compressed image (an enlarged image as a visual field) is displayed instead of the image in a normal state without distortion as shown in FIG. In the deformed image shown in FIG. 7, it can be seen that the field of view of the peripheral visual field region 60 is enlarged as compared with the image in the normal state of FIG. The expansion of the field of view of the peripheral visual field area 60 reduces the speed of the moving amount (optical flow) of the image generated when the drone moves, so that even if the drone moves at the same moving speed, it is shown in FIG. In the deformed image, the perceived speed is slower than that in the image in the normal state of FIG. As an adverse effect, the image is distorted, but since the resolution of the peripheral visual field region 60 of the human eye is low, it is difficult to perceive the distortion due to the characteristics of human visual perception. On the contrary, by making the peripheral visual field area 60 look narrow, it is possible to make the drone feel faster than the actual moving speed due to the effect of vection.

With reference to FIGS. 8 and 9, the angle of view of the peripheral visual field region 60 in the image after deformation by vection control, which is displayed on the display unit in the display system according to the first embodiment, will be described. FIG. 9 shows the angle of view and the like of the peripheral visual field region 60 when the line-of-sight direction changes with respect to FIG.

As shown in FIGS. 8 and 9, the angle of view of the stable visual field region 50 that does not distort the image centered on the gazing point 51 is R1, and the angle of view of the peripheral visual field region 60 after deformation is R3. Let R2 be the angle of view obtained by excluding the angle of view R3 of the stable viewing area 50 from the angle of view taken by 11. The angle of view R3 of the peripheral visual field region 60 is determined by the visual field enlargement ratio M of the peripheral visual field region 60, and M is obtained by the following formula. * Indicates a multiplication symbol. The unit of each angle of view is degree.
M = (R2 / R3) * 100 (%)

For example, when the angle of view R1 of the stable viewing area 50 is 60 ° and the shooting angle of view is 260 °, the angle of view R2 is 100 ° (260 / 2-60/2). When the visual field magnification M is 200%, the angle of view R3 of the peripheral visual field region 60 is 50 °. That is, the image having an angle of view of 100 ° is compressed twice and displayed in the peripheral visual field region 60 of the angle of view R3, and the field of view is enlarged twice. By changing the field of view enlargement ratio M, it is possible to continuously control the feeling of movement speed due to vection. By increasing the visual field enlargement ratio M to more than 100%, the speed of the optical flow in the peripheral visual field region 60 becomes slower, and the perceived speed decreases. By making the visual field enlargement ratio M smaller than 100%, the visual field is compressed, the optical flow speed is increased, and the perceived speed is improved. If the sensor unit 12 of the input unit 10 can detect the line of sight by eye tracking and the posture of the head by head tracking with low delay and with high accuracy, the stable gaze area 50 may be narrowed, or vice versa. If this is not the case, the stable gaze area 50 may be widened to make it difficult to perceive the distorted area.

Here, the gazing point 51 is determined by using the tracking information of the line-of-sight detection and the posture detection of the head by the sensor unit 12 of the input unit 10. That is, when the line-of-sight direction of the operator wearing the HMD is toward the front of the image as shown in FIG. 8, when the head is rotated, the stable gaze area 50 is always applied to the center of the image. Will be done. Further, when the line-of-sight direction changes as shown in FIG. 9, the stable gaze area 50 is determined centering on the gaze point 51. As a result, when the observation target is viewed only from the line of sight without moving the head, the stable gaze area 50 follows the gaze point 51, so that the distortion of the peripheral visual field area 60 is less likely to be perceived.

(Display control operation flow using vection)
FIG. 10 schematically shows an example of an operation flow of display control using vection in the display system according to the first embodiment.

The vection control determination unit 22 of the control unit 20 detects the gaze point 51 of the operator based on the tracking information of the line-of-sight detection and the posture detection of the head by the sensor unit 12 of the input unit 10 (step S11). Next, the vection control determination unit 22 determines whether or not it is necessary to perform vection control on the image captured by the imaging unit 11 (step S12). The vection control determination unit 22 determines whether or not the drone is moving or turning based on the detection data of the speed or the angular velocity by the sensor unit 12 of the input unit 10, and if the drone is moving or turning, for example, if the drone is moving or turning. It is determined that the velocity control needs to be performed (step S12: Y). Further, the vection control determination unit 22 does not need to perform vection control when it is determined that the drone is in a stationary state, for example, based on the detection data of the speed and the angular velocity by the sensor unit 12 of the input unit 10. (Step S12: N). The determination of whether or not the vehicle is in a stationary state can be determined by, for example, whether or not the speed is equal to or less than a threshold value. Further, it may be determined whether or not the norm of the angular velocity is equal to or less than the threshold value.

Further, the vection control determination unit 22 may determine whether or not it is necessary to perform vection control based on the moving image captured by the imaging unit 11. FIG. 11 schematically shows an example of a determination operation (step S12) of the necessity of vection control by the vection control determination unit 22.

The vection control determination unit 22 calculates the optical flow based on the moving image captured by the imaging unit 11 (step S21). Next, the vection control determination unit 22 determines whether or not there is an region in which the optical flow is equal to or greater than the threshold value in the peripheral visual field region 60 (step S22). When it is determined that the optical flow has a region equal to or larger than the threshold value (step S22: Y), the vection control determination unit 22 determines that it is necessary to perform vection control (step S23). When it is determined that there is no region where the optical flow is equal to or greater than the threshold value (step S22: N), the vection control determination unit 22 determines that it is not necessary to perform vection control (step S24).

When the vection control determination unit 22 determines that it is not necessary to perform vection control (step S12: N, step S24), the image generation unit 31 of the processing unit 30 generates an image in a normal state and displays the image. It is displayed on 40 (step S16). When the display unit 40 is an HMD, the image generation unit 31 texture-maps the fisheye image captured by the wide-angle fisheye camera, which is the imaging unit 11, to the Sphere model, for example. As a technique for texture mapping a fisheye image to a spherical image, for example, the technique described in Non-Patent Document 5 can be used.

When the vection control determination unit 22 determines that it is necessary to perform vection control (step S12: Y, step S23), the vection control amount calculation unit 23 of the control unit 20 calculates the field of view enlargement ratio M. (Step S13). The field of view magnification M may be given as a fixed value, or may be set according to the speed of the drone or the speed of the optical flow. This is because the higher the speed, the greater the discomfort caused by vection.

FIG. 12 shows a first calculation example of the field of view enlargement ratio M according to the speed by the vection control amount calculation unit 23. FIG. 13 shows a second calculation example of the field of view enlargement ratio M according to the speed by the vection control amount calculation unit 23. In FIGS. 12 and 13, the vertical axis represents the field of view magnification M, and the horizontal axis represents the speed of the drone or the speed of the optical flow. When the visual field enlargement ratio M is changed according to the speed, the vection control amount calculation unit 23 may set, for example, an application speed threshold value Vmin that starts changing the visual field enlargement ratio M from 100 (%). Further, the magnifying power upper limit value Mmax and the magnifying power lower limit value Mmin of the field of view enlargement ratio M may be set. When the field of view expansion rate M is made larger than 100, the vection control amount calculation unit 23 linearly changes the field of view expansion rate M according to the speed when the speed becomes equal to or higher than the application speed threshold value Vmin, but is more than the magnification rate upper limit value Mmax. Does not change to a large value. This is because there is a limit to the range in which the image in the peripheral visual field can be compressed due to the shooting angle of view of the imaging unit 11. Similarly, when the field of view enlargement ratio M is made smaller than 100, the vection control amount calculation unit 23 linearly changes the field of view enlargement ratio M according to the speed when the speed becomes equal to or higher than the application speed threshold value Vmin, but the lower limit of the enlargement ratio. It is not changed to a value smaller than the value Mmin.

Although FIGS. 12 and 13 show an example in which the visual field enlargement ratio M is linearly changed according to the speed, the visual field enlargement ratio M may be changed non-linearly. Further, the visual field magnification M may be changed linearly or non-linearly depending on the angular velocity. The speed of the drone may be obtained by using a value obtained by measuring the actual speed of the drone, or by estimating it from the input value of the controller of the drone.

The image generation unit 31 of the processing unit 30 calculates the texture coordinates to be mapped to the Sphere model based on the vection control amount including the field of view enlargement ratio M calculated by the vection control amount calculation unit 23 (step S14). As a technique for texture mapping a fisheye image to a spherical image, for example, the technique described in Non-Patent Document 5 can be used.

FIG. 14 shows an example of coordinate conversion of a fisheye image based on the field of view magnification M. Here, as an example of coordinate conversion, an example of coordinate conversion of the coordinates (r, θ) of the radius of the fisheye image is shown. For the sake of simplification of the explanation, a situation in which the center of the fisheye image is the gazing point 51 will be described as an example. As shown in FIG. 14, the radius of the stable visual field region 50 is r1, and the visual field enlargement ratio M (%) is m = M / 100. Assuming that the radius of the coordinate system of the fisheye image is r, the radius r'after the coordinate conversion can be described as follows.
-Stable gaze area 50
r <r1
r'= r
Peripheral visual field region 60 (compresses or enlarges the fisheye image toward the center (m> 1))
r> r1
r'= r1 + (r-r1) * m

By sampling the coordinates (r, θ) to the coordinates (r', θ) as shown in FIG. 14, when m> 1, the image of the peripheral visual field region 60 is compressed with respect to the gazing point 51 (as a visual field). It becomes possible to expand). On the contrary, when m <1, the image of the peripheral visual field region 60 can be enlarged (compressed as a visual field) with respect to the gazing point 51. As described above, the gazing point 51 is obtained based on the posture of the head and the direction of the line of sight.

The image generation unit 31 generates, for example, a deformed image by performing the above coordinate conversion and displays it on the display unit 40 (step S15). When the display unit 40 is an HMD, the image generation unit 31 texture-maps the fisheye image captured by the wide-angle fisheye camera, which is the imaging unit 11, to the Sphere model, for example. As a technique for texture mapping a fisheye image to a spherical image, for example, the technique described in Non-Patent Document 5 can be used.

[1.3 Modification example]
In the above description, a binary control in which the visual field enlargement ratio is 100% in the stable gaze area 50 and the visual field enlargement ratio is M in the peripheral visual field region 60 has been described as an example, but the technique of the present disclosure is binary. It is not limited to control. Specifically, when the image is deformed in the peripheral visual field region 60, if the angle of view R1 of the stable visual field region 50 is narrow, a boundary line with or without distortion is generated and it is easy to feel a sense of discomfort. The visual field enlargement ratio M may be changed stepwise as the distance from 50 is increased. In particular, when the display unit 40 is mounted on a large screen display, it is possible to reduce the sense of discomfort at the boundary between the non-distorted region and the distorted region.

In the above description, an example of performing vection control according to the moving state and speed of the operation target has been given, but vection control may be performed in consideration of the surrounding environment. For example, the field of view magnification M may be changed depending on whether there is a structure such as a floor or a wall around the operation target, or the determination as to whether or not it is necessary to perform vection control may be changed. ..

FIG. 15 schematically shows an example of a situation in which an operation target such as a drone 70 is moving away from the ground 71. FIG. 16 schematically shows an example of a situation in which an operation target such as the drone 70 is moving near the ground 71. FIG. 17 schematically shows an example of a situation in which an operation target such as a drone 70 is moving near a wall 72.

For example, as shown in FIG. 15, the operation target such as the drone 70 is moving far away from the ground 71 (distance d), and as shown in FIG. 16, the operation target is moving near the ground 71. The way of applying vection is different from the situation where the vehicle is moving or the situation where the vehicle is moving near the wall 72 as shown in FIG. For example, in a situation where the operation target is moving far away from the ground 71 as shown in FIG. 15, the optical flow becomes slow. In this situation, the vection control may be performed so as to reduce the field of view enlargement ratio M, or the vection control may not be performed. On the contrary, in the situation shown in FIGS. 16 and 17, the optical flow becomes faster. Therefore, the vection control may be performed so as to increase the field of view enlargement ratio M.

FIG. 18 schematically shows a modified example of the determination operation of the necessity of vection control by the vection control determination unit 22 in the display control device 1. FIG. 18 shows an example of determining whether or not it is necessary to perform vection control in consideration of the surrounding environment.

The vection control determination unit 22 determines whether or not the operator's viewpoint movement has occurred based on the tracking information by the sensor unit 12 of the input unit 10 (step S31). This determination is made, for example, by determining whether or not the line of sight itself is moving by a threshold value or more without changing the posture of the head. It is also performed by determining whether or not the head is rotated by a threshold value or more. When it is determined that the operator's viewpoint movement has not occurred (step S31: N), the vection control determination unit 22 determines that it is not necessary to perform vection control (step S34).

When it is determined that the operator's viewpoint movement has occurred (step S31: Y), the vection control determination unit 22 then uses the distance information from the sensor unit 12 of the input unit 10 or the imaging unit 11. Based on the imaging result, it is determined whether or not there is an object around the operation target (step S32). This determination is performed, for example, by determining whether or not the operation target is within a threshold distance from the ground, wall, floor, ceiling, or the like. When it is determined that there is an object around the operation target (step S32: Y), the vection control determination unit 22 determines that it is necessary to perform vection control (step S33). When it is determined that there is no object around the operation target (step S32: N), the vection control determination unit 22 determines that it is not necessary to perform vection control (step S34).

Further, in the above description, the imaging unit 11 of the input unit 10 uses a monocular wide-angle fisheye camera, but a 360 ° camera capable of capturing a spherical image may be used.

Further, in the above description, the image generation unit 31 of the processing unit 30 has given an example of performing processing based on the spherical image, but a three-dimensionally reconstructed model may be used.

Further, in the above description, an example in which the image is uniformly distorted from the gazing point 51 has been given, but an area in the visual field may be identified by using the Presence Segmentation, and the image may be deformed for each identified area. ..

FIG. 19 shows a first deformation example of the image after deformation by vection control. FIG. 20 shows a second deformation example of the image after deformation by vection control. As shown in FIGS. 19 and 20, the width w of the passage and the height h of the wall may be deformed so as to be enlarged or reduced. The same may be applied to a VR game to which a 3D model is applied.

Further, in the above description, an example in which the entire image of the peripheral visual field area 60 is deformed by vection control is given as a predetermined area for deforming the image, but only a part of the image of the peripheral visual field area 60 is deformed. You may do so.

FIG. 21 shows a modified example of the region where the vection control is performed. For example, as shown in FIG. 21, the image may be deformed by the vection control only in the region 80 where the optical flow is equal to or higher than the threshold value as a predetermined region.

Further, in the above description, an example of performing eye tracking has been given, but eye tracking does not necessarily have to be performed. In that case, only head tracking related to the movement of the head may be performed. In addition, head tracking does not necessarily have to be performed. For example, in the case of high-speed eye movement such as saccade, eye tracking of the gazing point 51 may not be performed.

Further, in the above description, an example in which the angle of view R1 of the stable viewing area is set to a fixed value has been given, but the angle of view R1 of the stable viewing area may be changed depending on the situation. For example, driving a car on a highway is known to have a narrow field of vision. The angle of view R1 may be widened when the vehicle is stationary or the speed is low, and the angle of view R1 may be narrowed according to the speed. This is because when the angle of view R1 is narrow, the visual field enlargement ratio M of the peripheral visual field can be further improved.

[1.4 Effect]
As described above, according to the display system according to the first embodiment, when it is determined that it is necessary to transform the image using the velocity control, the image is transformed using the velocity control. The vection control amount for this purpose is calculated, and an image in which a predetermined area is deformed is generated based on the calculated vection control amount. It becomes possible to display.

According to the display system according to the first embodiment, it is possible to appropriately control the sense of speed felt by a person by utilizing vection without unnecessarily obstructing the field of view of the operator or the like. For example, by controlling the optical flow of the peripheral visual field region 60 by image processing, it is possible to control the image display so as to reduce sickness and anxiety without causing discomfort or obstruction to the operator or the like. Further, according to the display system according to the first embodiment, since the image presented on the HMD or the large screen display can be directly processed, only in the situation where it is necessary to change the image. It is possible to adaptively control the image display. In addition, since it also has the effect of expanding the viewing angle presented in the peripheral visual field, it is possible to improve the detection rate of obstacles and moving objects approaching the operation target such as a drone, and it is possible to improve the detection rate between the operation target and surrounding objects. It can prevent contact and collision, which is effective in improving work efficiency and safety.

Further, according to the display system according to the first embodiment, it is possible to appropriately control the sense of speed perceived by the operator by deforming the image of the peripheral visual field region 60. As a result, when performing remote control or the like, it is possible to reduce work efficiency and operation load without obstructing the field of view. For example, it is possible to reduce the feeling of fear when moving at high speed, and to reduce sickness due to sensory disagreement. In addition, since the viewing angle can be expanded, the ability to judge the situation of a nearby object is improved, and safety and work efficiency can be improved. In addition, when the speed is exceeded, the operator can be guided to naturally reduce the speed without losing the concentration of the operator.

Further, according to the display system according to the first embodiment, even if the influence of vection changes according to the speed of the operation target, the field of view enlargement ratio M of the image is adjusted according to the speed of the operation target. It is possible to display an appropriate image.

Further, according to the display system according to the first embodiment, even if the intensity of the vection changes according to the distance to the surrounding object or the like, the visual field enlargement ratio M of the image is adjusted to obtain the peripheral image. It is possible to display an appropriate image according to the distance to the object.

Further, according to the display system according to the first embodiment, image distortion is perceived by performing image deformation in the peripheral visual field relatively viewed from the gazing point 51 according to the line of sight, the posture of the head, and the like. Visibility is improved because it is difficult to do.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

<2. Other embodiments>
The technique according to the present disclosure is not limited to the description of each of the above embodiments, and various modifications can be implemented.

For example, the present technology may have the following configuration.
According to the present technology having the following configuration, when it is determined that it is necessary to deform the image using the vection control, the vection control amount for deforming the image using the vection control is calculated and calculated. Since an image obtained by deforming a predetermined area is generated based on the obtained vection control amount, it is possible to display an image in consideration of human senses.

(1)
A determination unit that determines whether or not it is necessary to transform the image using vection control,
When the determination unit determines that it is necessary to deform the image using the vection control, the control amount calculation unit that calculates the vection control amount for deforming the image using the vection control, and the control amount calculation unit.
A display control device including an image generation unit that generates an image obtained by deforming a predetermined area based on the vection control amount calculated by the control amount calculation unit.
(2)
The display control device according to (1) above, wherein the determination unit determines whether or not it is necessary to perform deformation of the image using the vection control based on the optical flow of the image.
(3)
The display control device according to (1) or (2) above, wherein the control amount calculation unit calculates the vection control amount based on the optical flow of the image.
(4)
The display control device according to any one of (1) to (3) above, wherein the control amount calculation unit calculates the enlargement ratio of the visual field in the predetermined region as the vection control amount.
(5)
The determination unit determines whether or not it is necessary to deform the image taken by the moving body including the imaging unit using the vection control. Any of the above (1) to (4). The display control device according to one.
(6)
The display control device according to (5) above, wherein the determination unit determines whether or not it is necessary to perform deformation of an image using the vection control based on the speed of the moving body.
(7)
The display control device according to (5) above, wherein the determination unit determines whether or not it is necessary to perform deformation of an image using the vection control based on the surrounding environment of the moving body.
(8)
The display control device according to any one of (5) to (7) above, wherein the control amount calculation unit calculates the vection control amount based on the speed of the moving body.
(9)
The display control device according to any one of (5) to (7) above, wherein the control amount calculation unit calculates the vection control amount based on the surrounding environment of the moving body.
(10)
The control amount calculation unit discriminates between a stable gaze area and a peripheral visual field area based on the information of the gaze point of the observer of the image, and calculates the vection control amount with the peripheral visual field area as the predetermined area. The display control device according to any one of (1) to (9) above.
(11)
The image generation unit sets a region obtained based on at least one of the line of sight of the observer of the image and the posture of the head as the predetermined region in any one of (1) to (10) above. The display control device described.
(12)
The display control device according to any one of (1) to (11) above, wherein the image generation unit sets a region in which the optical flow of the video is equal to or greater than a threshold value as the predetermined region.
(13)
Determining whether it is necessary to transform the image using vection control, and
When it is determined that it is necessary to transform the image using the vection control, the vection control amount for transforming the image using the vection control is calculated.
A display control method including generating an image in which a predetermined area is deformed based on the calculated vection control amount.

1 ... Display control device, 10 ... Input unit, 11 ... Imaging unit, 12 ... Sensor unit, 13 ... Transmission unit, 20 ... Control unit, 21 ... Reception unit, 22 ... Vection control judgment unit (judgment unit), 23 ... Vection Control amount calculation unit (control amount calculation unit), 30 ... Processing unit, 31 ... Image generation unit, 40 ... Display unit, 50 ... Stable viewing area, 51 ... Gaze point, 60 ... Peripheral visual field area, 70 ... Drone, 71 ... ground, 72 ... wall, 80 ... area where optical flow is above the threshold, 100 ... operator, 101 ... robot, 102 ... HMD (head mount display), 110 ... operator, 111 ... large screen display, 121 ... dot, d ... Distance to the ground, R1 ... Stable viewing angle of view, R2 ... Shooting angle of view minus stable viewing area, R3 ... Peripheral field of view, M ... Field magnification , W ... width, h ... height.

Claims (13)

A determination unit that determines whether or not it is necessary to transform the image using vection control,
When the determination unit determines that it is necessary to deform the image using the vection control, the control amount calculation unit that calculates the vection control amount for deforming the image using the vection control, and the control amount calculation unit.
A display control device including an image generation unit that generates an image obtained by deforming a predetermined area based on the vection control amount calculated by the control amount calculation unit. The display control device according to claim 1, wherein the determination unit determines whether or not it is necessary to perform deformation of the image using the vection control based on the optical flow of the image. The display control device according to claim 1, wherein the control amount calculation unit calculates the vection control amount based on an optical flow of video. The display control device according to claim 1, wherein the control amount calculation unit calculates the enlargement ratio of the visual field in the predetermined region as the vection control amount. The display control device according to claim 1, wherein the determination unit determines whether or not it is necessary to deform an image using the vection control with respect to an image captured by a moving body including an imaging unit. The display control device according to claim 5, wherein the determination unit determines whether or not it is necessary to perform deformation of an image using the vection control based on the speed of the moving body. The display control device according to claim 5, wherein the determination unit determines whether or not it is necessary to perform deformation of an image using the vection control based on the surrounding environment of the moving body. The display control device according to claim 5, wherein the control amount calculation unit calculates the vection control amount based on the speed of the moving body. The display control device according to claim 5, wherein the control amount calculation unit calculates the vection control amount based on the surrounding environment of the moving body. The control amount calculation unit discriminates between a stable gaze area and a peripheral visual field area based on the information of the gaze point of the observer of the image, and calculates the vection control amount with the peripheral visual field area as the predetermined area. The display control device according to claim 1. The display control device according to claim 1, wherein the image generation unit sets a region obtained based on at least one of the line of sight of the observer of the image and the posture of the head as the predetermined region. The display control device according to claim 1, wherein the image generation unit has a region in which the optical flow of the video is equal to or greater than a threshold value as the predetermined region. Determining whether it is necessary to transform the image using vection control, and
When it is determined that it is necessary to transform the image using the vection control, the vection control amount for transforming the image using the vection control is calculated.
A display control method including generating an image in which a predetermined area is deformed based on the calculated vection control amount.

Images