JP7363454B2 - Image adjustment device, virtual reality image display system, and image adjustment method - Google Patents

Description

The present invention relates to an image adjustment device, a virtual reality image display system, and an image adjustment method.

Virtual reality image display systems that display omnidirectional images taken with an omnidirectional camera (360-degree camera) in all directions, top, bottom, left, and right on a head-mounted display are rapidly becoming popular. Hereinafter, virtual reality may be abbreviated as VR.

Japanese Patent Application Publication No. 2005-56295

Patent Document 1 describes detecting the horizontal plane of an image taken with an omnidirectional camera and correcting the tilt of the image. An omnidirectional camera may detect a horizontal plane in a captured image, add auxiliary information indicating the horizontal plane to an image signal, and output the resultant image signal.

However, depending on the image taken, the horizontal plane of the image may not be detected correctly, and incorrect auxiliary information may be added to the image signal. Even when a horizontal plane of an image is detected by a three-axis sensor, auxiliary information indicating an erroneously detected horizontal plane may be generated. When a VR image display system displays an omnidirectional image taken by an omnidirectional camera, if incorrect auxiliary information is added to the image signal, the direction of gravity felt by the head-mounted display user and the zenith of the omnidirectional image may be incorrect. The user feels a sense of discomfort because the direction in which the user is positioned is shifted.

Additionally, when the omnidirectional camera generates omnidirectional images while moving, it is necessary to correlate the front of the subject photographed by the omnidirectional camera with the image displayed on the head-mounted display when the user is facing forward. .

The present invention can easily correct the tilt of the horizontal plane of an omnidirectional image taken with an omnidirectional camera and displayed on a head-mounted display. An object of the present invention is to provide an image adjustment device, a virtual reality image display system, and an image adjustment method that can automatically associate area images displayed on a head-mounted display when the user is in the room.

The present invention includes an image generation unit that generates a spherical image, and a superimposed image obtained by superimposing the spherical image on an omnidirectional image obtained by photographing a subject by an omnidirectional camera disposed on a moving body, or a head of the omnidirectional image. a region image extracting unit that supplies a region image extracted corresponding to the direction in which a user wearing a mounted display faces to the head mounted display; and a region image of the superimposed image extracted by the region image extracting portion to the head mounted display. an image rotation unit that rotates the omnidirectional image by an operation of rotating the spherical image to correct the inclination of the horizontal plane of the omnidirectional image when displayed on a mount display; a vanishing point detection unit that detects a vanishing point of the omnidirectional image while the azimuth image is changing; and a vanishing point detection unit that identifies the front of the omnidirectional image based on the vanishing point, and a forward setting unit that rotates the omnidirectional image while maintaining the horizontal plane corrected by the image rotation unit so as to associate the area image with the area image extracted when the user is facing forward; When the moving body moving forward accelerates, the extracting unit extracts a region image rotated upward by a predetermined angle and supplies it to the head mounted display, and when the moving body moving forward accelerates. Provided is an image adjustment device that extracts a region image rotated downward by a predetermined angle and supplies it to the head mounted display when the vehicle decelerates.

The present invention provides image data of an omnidirectional image taken of a subject by an omnidirectional camera disposed on a moving object, and an acceleration detection signal detected by an acceleration sensor attached to the moving object or the omnidirectional camera. a communication unit that receives information from an image transmission server; a head-mounted display that is attached to a user's head to show the omnidirectional image to the user; a controller that is operated by the user; a chair on which the user sits; an image generation unit that generates an image; an image superposition unit that generates a superimposed image by superimposing the spherical image on the omnidirectional image; a region image extracting unit that supplies a region image extracted by the user to the head mounted display; and a region image extraction unit that rotates the spherical image by operating the controller while the user is sitting on the chair, thereby generating the superimposed image. an image rotation unit configured to rotate the omnidirectional image to correct the inclination of the horizontal plane of the omnidirectional image; and detecting a vanishing point of the omnidirectional image when the omnidirectional camera moves and the omnidirectional image changes. a vanishing point detection unit, identifying the front of the omnidirectional image based on the vanishing point, and associating the front of the omnidirectional image with a region image extracted when the user is facing forward; a forward setting unit that rotates the omnidirectional image while maintaining a horizontal plane corrected by the image rotation unit; and when the acceleration detection signal indicates that the moving body moving forward is accelerating; a chair control unit that tilts the chair backward by a predetermined angle and tilts the chair forward by a predetermined angle when the acceleration detection signal indicates that the moving body moving forward is decelerating; A virtual reality image display system is provided.

The present invention generates a spherical image, and superimposes the spherical image on an omnidirectional image taken of a subject by an omnidirectional camera placed on a moving body, or a superimposed image of the omnidirectional image, which is attached to a head-mounted display. supplying the extracted regional image corresponding to the direction in which the user faces to the head mounted display, and rotating the spherical image while displaying the extracted regional image of the superimposed image on the head mounted display. rotate the omnidirectional image to correct the inclination of the horizontal plane of the omnidirectional image, and detect the vanishing point of the omnidirectional image while the moving object is moving and the omnidirectional image is changing. The omnidirectional image is configured to identify the front of the omnidirectional image based on the vanishing point, and to associate the front of the omnidirectional image with a region image extracted when the user is facing forward. is rotated while maintaining the corrected horizontal plane, and when the moving object moving forward accelerates, an area image rotated upward by a predetermined angle is extracted and supplied to the head mounted display, and the image is rotated while maintaining the corrected horizontal plane. The present invention provides an image adjustment method for extracting a region image rotated downward by a predetermined angle and supplying the extracted area image to the head mounted display when the moving object decelerates.

According to the image adjustment device, virtual reality image display system, and image adjustment method of the present invention, it is possible to easily correct the tilt of the horizontal plane of an image taken with an omnidirectional camera and displayed on a head-mounted display. It is possible to automatically associate the front of the subject photographed by the omnidirectional camera with the image displayed on the head-mounted display when the user is facing forward.

FIG. 1 is a block diagram showing an omnidirectional image transmission/reception system including an image adjustment device and a virtual reality image display system according to each embodiment. FIG. 1 is a partial perspective view showing a vehicle in which an omnidirectional camera is arranged inside the vehicle. FIG. 2 is a perspective view showing an example of the external configuration of an omnidirectional camera. FIG. 2 is a block diagram showing a specific configuration example of an image processing unit included in the image adjustment device and virtual reality image display system of the first embodiment. FIG. 2 is a perspective view showing a user sitting on a VR chair and viewing an omnidirectional image taken by an omnidirectional camera. FIG. 2 is a conceptual diagram showing a spherical image in which the user is virtually positioned, which is generated by the image generation unit of the image adjustment device and the virtual reality image display system of each embodiment to adjust the horizontal plane of the omnidirectional image. be. FIG. 6 is a diagram for explaining an operation in which the front setting unit included in the image adjustment device and the virtual reality image display system of each embodiment sets the front of the omnidirectional image based on the vanishing point of the omnidirectional image. 7 is a flowchart showing processing executed by the image adjustment device of the first embodiment. FIG. 3 is a block diagram illustrating a specific configuration example of a control unit included in an image adjustment device and a virtual reality image display system according to second and third embodiments. FIG. 3 is a diagram conceptually showing a state in which a partial region image of an omnidirectional image photographed while a vehicle is traveling straight is displayed on a head-mounted display. Conceptually shows a state in which the VR chair and the area image are tilted to the right while a partial area image of the omnidirectional image taken while the vehicle is turning left is displayed on the head-mounted display. It is a diagram. 7 is a flowchart showing processing executed by the image adjustment device and virtual reality image display system of the second embodiment. FIG. 3 is a diagram conceptually showing a state in which the VR chair and the area image are tilted backward while a vehicle traveling ahead is accelerating. FIG. 3 is a diagram conceptually showing a state in which the VR chair and the area image are tilted forward while a vehicle running ahead is decelerating. 12 is a flowchart showing processing executed by an image adjustment device and a virtual reality image display system according to a third embodiment. FIG. 7 is a block diagram showing a specific configuration example of a control unit included in the image adjustment device and the virtual reality image display system of the fourth and fifth embodiments. FIG. 7 is a diagram showing how to control the VR chair while the vehicle is moving along a ballistic trajectory in the fourth embodiment. It is a flowchart which shows the process performed by the virtual reality image display system of 4th embodiment. FIG. 7 is a diagram showing how to control the VR chair while the vehicle is moving along a ballistic trajectory in the fifth embodiment. It is a flowchart which shows the process performed by the virtual reality image display system of 5th Embodiment. It is a flowchart which shows the process performed by the image adjustment device and virtual reality image display system of 6th Embodiment. It is a flowchart which shows the preferable process performed by the image adjustment apparatus and virtual reality image display system of 6th Embodiment.

Hereinafter, an image adjustment device, a virtual reality image display system, an image adjustment method, and a control method of the virtual reality image display system according to each embodiment will be described with reference to the accompanying drawings.

<First embodiment>
In FIG. 1, an omnidirectional camera 12 and a three-axis acceleration sensor 13 are connected to a communication unit 11. As shown in FIG. 2, the omnidirectional camera 12 is placed on the dashboard of the vehicle 10, as an example. As shown in FIG. 3, the omnidirectional camera 12, in the arrangement state shown in FIG. and a right-eye fisheye lens 12RR.

The omnidirectional camera 12 may be placed in front of the driver. The omnidirectional camera 12 is not limited to being placed inside the vehicle 10, but may be placed outside the vehicle 10, for example, on the roof. The omnidirectional camera 12 is placed at an arbitrary position on an arbitrary moving object such as the vehicle 10, and can photograph a relatively moving subject.

As shown in FIG. 3, an acceleration sensor 13 is attached to the housing of the omnidirectional camera 12. The acceleration sensor 13 may be placed inside the housing. The acceleration sensor 13 may be attached to a moving body to which the omnidirectional camera 12 is attached. The omnidirectional camera 12 includes a configuration such as an image sensor and a video signal processing circuit inside the housing. The omnidirectional camera 12 generates a left-eye image signal and a right-eye image signal. Thereby, the omnidirectional camera 12 generates omnidirectional image data for three-dimensional (3D) display.

The omnidirectional camera 12 detects the horizontal plane of the image based on the photographed image, adds auxiliary information indicating the horizontal plane to the omnidirectional image data, and outputs the data. The omnidirectional camera 12 may detect the horizontal plane of the image using a three-axis sensor. The omnidirectional camera 12 does not need to generate auxiliary information indicating the horizontal plane.

Returning to FIG. 1, the communication unit 11 supplies omnidirectional image data generated by the omnidirectional camera 12 and an acceleration detection signal obtained by detecting acceleration by the acceleration sensor 13 to the image transmission server 31 via the network 20. Although auxiliary information is added to the omnidirectional image data, it will simply be referred to as omnidirectional image data. Typically, network 20 is the Internet.

The memory 32 temporarily stores omnidirectional image data and acceleration detection signals supplied to the image transmission server 31. The image transmission server 31 transmits the omnidirectional image data and acceleration detection signal via the network 20 to the VR image display system 40 located on the client side that receives the omnidirectional image data generated by the omnidirectional camera 12. do.

The VR image display system 40 includes a communication section 41 , a control section 42 , an image generation section 43 , a head-mounted display 44 , a glove-type controller 45 , a VR chair 46 , and an operation section 47 . The control section 42 includes an image processing section 420. At least the image generation section 43 and the image processing section 420 constitute an image adjustment device. As shown in FIG. 4, the image processing section 420 in the first embodiment includes an image superimposition section 421, an image rotation section 422, a vanishing point detection section 423, a front setting section 424, and a region image extraction section 425.

The control unit 42 may be composed of a microcomputer or a microprocessor, or may be a central processing unit (CPU) included in the microcomputer. The image processing unit 420 configured as shown in FIG. 4 may be realized by a CPU executing a computer program. At least a portion of the image processing unit 420 may be configured by a hardware circuit. The use of hardware and software is arbitrary.

As shown in FIG. 5, the user Us who views the omnidirectional image indicated by the omnidirectional image data transmitted from the image transmission server 31 wears the head mounted display 44 on his head while sitting on the VR chair 46. A glove-type controller 45 is worn on both hands.

In the standard state, the VR chair 46 has a seat surface adjusted to be horizontal at a predetermined height. This state is set as the reference height and reference angle of the VR chair 46. The VR chair 46 is equipped with a seat belt 461, and the user Us is fastening the seat belt 461. When the user Us is fastening the seat belt 461, a signal indicating that the seat belt 461 is fastened is supplied to the control unit 42. Seat belt 461 is an example of a safety device.

The communication unit 41 communicates with the image transmission server 31 via the network 20 and receives omnidirectional image data and acceleration detection signals transmitted from the image transmission server 31. The communication unit 41 supplies omnidirectional image data and acceleration detection signals to the control unit 42 . When the image generation section 43 is instructed to output spherical image data by the operation section 47, the image generation section 43 generates spherical image data made of computer graphics and supplies it to the control section 42.

In FIG. 4, omnidirectional image data transmitted from the image transmission server 31 and spherical image data generated by the image generation section 43 are input to the image superimposition section 421. The image superimposition unit 421 superimposes the omnidirectional image data on the spherical image data to generate superimposed image data.

The superimposed image data is supplied to a region image extraction section 425 via an image rotation section 422 and a front setting section 424. The region image extraction unit 425 receives direction information from the head mounted display 44 indicating the direction in which the head mounted display 44 (the face of the user Us) is facing. The region image extraction unit 425 extracts region image data corresponding to the direction in which the face of the user Us is facing from among the superimposed image data or the omnidirectional image data based on the input direction information, and extracts the extracted region image. The data is supplied to the head mounted display 44.

The image superimposition unit 421 sets the horizontal plane of the omnidirectional image data based on the auxiliary information added to the omnidirectional image data, and superimposes the spherical image data on the omnidirectional image data. If auxiliary information is not added to the omnidirectional image data, the image processing unit 420 may detect the horizontal plane of the image and set the horizontal plane of the omnidirectional image data.

FIG. 6 conceptually shows the spherical image VSS indicated by the spherical image data. As an example, the spherical image VSS is formed by a line image indicating a plurality of latitudes and a line image indicating a plurality of longitudes. Almost the upper body of the user Us is virtually located inside the spherical image VSS. The spherical image VSS is displayed so as to be within reach of the user Us. When the user Us views the omnidirectional image (not shown in FIG. 6), if the auxiliary information is incorrect or the image processing unit 420 incorrectly detects a horizontal plane, the zenith of the omnidirectional image and the spherical image VSS does not match the zenith ZE, and the user Us feels uncomfortable.

Therefore, the image processing unit 420 is configured to correct the inclination of the horizontal plane of the omnidirectional image so that the zenith of the omnidirectional image coincides with the zenith ZE of the spherical image VSS. In FIG. 6, the spherical image VSS is displayed so as to be within reach of the user Us. The user Us who wears the glove-shaped controller 45 on his hand can feel as if he is touching the spherical image VSS by extending his hand.

The glove-type controller 45 preferably includes an actuator on the inner surface that comes in contact with the hand. If the control unit 42 controls the actuator to operate when the glove-type controller 45 reaches the position where it touches the spherical image VSS, it is possible to give the user Us the feeling of touching the spherical image VSS.

When the user Us rotates the spherical image VSS in an arbitrary direction using the glove-shaped controller 45 while touching the spherical image VSS, the rotation operation information output from the glove-shaped controller 45 is input to the image rotation unit 422 . The image rotation unit 422 rotates the omnidirectional image in response to rotation operation information. This allows the user Us to easily correct the inclination of the horizontal plane of the omnidirectional image. As a result, the zenith of the omnidirectional image and the zenith ZE of the spherical image VSS match, and the discomfort felt by the user Us can be eliminated. The image rotation unit 422 holds correction values for the tilt of the horizontal plane.

After correcting the horizontal plane, it is preferable that the user Us operate the operation unit 47 to hide the spherical image VSS. After correcting the horizontal plane, the user Us may remove the glove-type controller 45.

The above correction of the horizontal plane of the omnidirectional image is possible while the vehicle 10 is stopped. By only correcting the horizontal plane, the user Us cannot recognize which direction of the omnidirectional image is in front of the subject being photographed by the omnidirectional camera 12.

When the vehicle 10 starts moving, assuming that the user Us is facing forward and the front area image data of the omnidirectional image is supplied to the head mounted display 44, the user Us will be able to see the vanishing point Vp as shown in FIG. The user views an area image 44i in which the scenery expands radially around . In a state where the front of the omnidirectional image is not specified, the vanishing point Vp is not necessarily located within the area image 44i.

The vanishing point detection unit 423 detects a motion vector MV between frames based on at least two frames of images. The vanishing point detection unit 423 detects a point where a plurality of motion vectors MV intersect by extending them in the negative direction of motion as a vanishing point Vp. The vanishing point detection unit 423 may use either the left-eye image signal or the right-eye image signal when detecting the vanishing point Vp.

The front setting unit 424 identifies the front of the omnidirectional image corresponding to the front of the subject photographed by the omnidirectional camera 12 based on the vanishing point Vp detected by the vanishing point detection unit 423. The front setting unit 424 rotates the omnidirectional image while maintaining the corrected horizontal plane so that the front of the omnidirectional image is associated with the region image 44i extracted when the user Us is facing forward. The front setting unit 424 preferably rotates the omnidirectional image so that the vanishing point Vp is located in front of the user Us's face.

In this way, the front of the omnidirectional image can be automatically associated with the area image 44i displayed on the head-mounted display 44 when the user Us is facing forward. Note that it is also possible to manually set the front by rotating the spherical image VSS using the glove-type controller 45.

The VR chair 46 is configured to be rotatable in a horizontal plane, tiltable in the left-right direction and front-back direction, and capable of changing its height. The control unit 42 is supplied with the rotation angle in the horizontal plane of the VR chair 46, the tilt angle in the left-right direction, the tilt angle in the front-rear direction, and the position information in the vertical direction.

The front setting unit 424 may rotate the omnidirectional image so that the vanishing point Vp is located in the direction of the rotation angle of the VR chair 46 in the horizontal plane. The direction of the rotation angle of the VR chair 46 is equivalent to the front direction that the user Us is facing. Therefore, even if the omnidirectional image is rotated so that the vanishing point Vp is located in the direction of the rotation angle of the VR chair 46, the front of the omnidirectional image and the area image 44i when the user Us is facing forward cannot be changed. Can be associated.

Processing executed in the first embodiment will be described using the flowchart shown in FIG. 8. In FIG. 7, when the process is started, the image rotation unit 422 rotates the spherical image VSS using the glove-shaped controller 45 while the user Us is sitting on the horizontal seat surface of the VR chair 46 in step S11. Correct the tilt of the horizontal plane of the omnidirectional image by rotating it. The image processing unit 420 (vanishing point detection unit 423) determines whether the image is changing in step S12. If the image has not changed (NO), the image processing unit 420 repeats the process of step S12.

If the image is changing in step S12 (YES), it means that the vehicle 10 is moving, and the vanishing point detection unit 423 detects the vanishing point Vp in step S13. In step S14, the forward setting unit 424 rotates the omnidirectional image while maintaining the horizontal plane so that the vanishing point Vp is located within the area image 44i extracted when the user Us is facing forward. and terminate the process.

According to the first embodiment described above, the inclination of the horizontal plane of the omnidirectional image captured by the omnidirectional camera 12 and displayed on the head-mounted display 44 can be easily corrected. Further, according to the first embodiment, it is possible to automatically associate the front of the omnidirectional image with the area image 44i displayed on the head-mounted display 44 when the user Us is facing forward.

<Second embodiment>
The second embodiment, like the first embodiment, corrects the inclination of the horizontal plane of the omnidirectional image, sets the front of the omnidirectional image, and then gives the user Us a sense of realism corresponding to the movement of the vehicle 10. This is an image adjustment device and a VR image display system 40 configured as follows.

As shown in FIG. 9, the control unit 42 includes a chair control unit and a mode setting unit 4202 in the sixth embodiment. The image processing unit 420 included in the control unit 42 includes an image tilting unit 426 to which the area image data output from the area image extraction unit 425 is supplied. The acceleration detection signal is input to the region image extraction unit 425, the image tilting unit 426, and the chair control unit 4201. Note that in the second embodiment, it is not necessary to input the acceleration detection signal to the regional image extraction unit 425.

FIG. 10A conceptually shows a state in which the user Us is viewing the area image 44i while the vehicle 10 is traveling straight and the acceleration sensor 13 is detecting the angle θ0, which is the direction of gravitational acceleration. There is. Here, the backrest of the VR chair 46 is omitted and only the seat surface adjusted horizontally is shown.

As shown in FIG. 10B, when the vehicle 10 turns left and the acceleration sensor 13 detects a predetermined leftward angle θ1, the chair control unit 4201 controls the VR chair 46 to tilt the VR chair 46 rightward by a predetermined angle θ2. control chair 46. In accordance with this, the image tilt section 426 tilts the region image 44i output from the region image extraction section 425 by a predetermined angle θ3 in the right direction.

When the vehicle 10 turns right, the acceleration sensor 13 detects a predetermined angle θ1 to the right, so the chair control unit 4201 controls the VR chair 46 to tilt the VR chair 46 to the left by an angle θ2. However, the image tilting unit 426 only has to tilt the area image 44i to the left by a predetermined angle θ3.

The angle θ2 may be the same angle as the angle θ1, or may be a different angle. The angle θ3 may be the same angle as the angle θ1, or may be a different angle. The angle θ2 and the angle θ3 may be the same angle or may be different angles.

If it is only desired to give the user Us a sense of presence as if the user Us is riding in the vehicle 10, the angles θ2 and θ3 may be less than or equal to the angle θ1. A mode that gives a sense of presence as if the user Us is riding in the vehicle 10 is defined as a normal mode. If it is desired to give the user Us a sense of realism that emphasizes the movement of the vehicle 10, it is preferable to make the angle θ2 and the angle θ3 larger than the angle θ1. A mode that gives the user Us a sense of realism by emphasizing the movement of the vehicle 10 is defined as an enhancement mode.

The mode setting section 4202 has a normal mode or an enhanced mode set by the user Us using the operation section 47 in advance.

Processing executed in the second embodiment will be described using the flowchart shown in FIG. 11. In FIG. 11, when the process is started, the control unit 42 determines in step S21 whether the acceleration sensor 13 has detected the angle θ1. If the acceleration sensor 13 does not detect the angle θ1 (NO), the control unit 42 repeats the process of step S21. If the acceleration sensor 13 detects the angle θ1 (YES), the control unit 42 (mode setting unit 4202) determines whether or not the normal mode is set in step S22.

If the normal mode is set in step S22 (YES), in step S23, the chair control unit 4201 tilts the VR chair 46 to the right or left by an angle θ2 satisfying θ1≧θ2, and the image tilting unit 4201 tilts the area image 44i to the right or left by an angle θ3 such that θ1≧θ3.

If the normal mode is not set in step S22 (NO), it means that the enhanced mode is set. In step S24, the chair control unit 4201 tilts the VR chair 46 rightward or leftward by an angle θ2 where θ1<θ2, and the image tilting unit 426 tilts the area image 44i rightward or leftward at an angle θ1<θ3. Tilt by θ3.

In step S25 following step S23 or S24, the control unit 42 determines whether the acceleration sensor 13 has detected the angle θ0. If the acceleration sensor 13 does not detect the angle θ0 (NO), the control unit 42 or the image processing unit 420 repeats the processing of steps S22 to S25. If the acceleration sensor 13 detects the angle θ0 (YES), the chair control unit 4201 returns the angle of the VR chair 46 to 0, and the image tilting unit 426 returns the angle of the area image 44i to 0, in step S26.

In step S27, the control unit 42 determines whether or not to end receiving the image data from the image transmission server 31. If the reception of the image data is not completed (NO), the control unit 42 or the image processing unit 420 repeats the processing of steps S21 to S27. If the reception of the image data is completed (YES), the control unit 42 ends the process.

According to the second embodiment described above, in addition to the effects achieved by the first embodiment, it is possible to provide the user Us with a sense of presence corresponding to the feeling that the occupant of the vehicle 10 feels when the vehicle 10 turns right or left. I can do it. By making it possible to select between two modes, the normal mode and the enhanced mode, the user Us may want to feel as if he or she is simply riding in the vehicle 10, or may want to obtain a stronger sense of presence that emphasizes the movement of the vehicle 10. can be set according to the user's preference.

Note that in the second embodiment, only the VR chair 46 may be tilted without tilting the area image 44i. Of course, it is preferable to tilt the area image 44i in correspondence with tilting the VR chair 46.

<Third embodiment>
The third embodiment corrects the inclination of the horizontal plane of the omnidirectional image and sets the front of the omnidirectional image as in the first embodiment, and then uses a method different from the second embodiment to tell the user Us to the vehicle 10. This is an image adjustment device and a VR image display system 40 configured to provide a sense of realism corresponding to the movement of the user. The configuration of the control unit 42 in the third embodiment may be the same as that in FIG. 9, but the mode setting unit 4202 and the image tilting unit 426 can be omitted.

As shown in FIG. 12A, when the vehicle 10 traveling forward accelerates and the acceleration sensor 13 detects the forward angle θ4, the chair control unit 4201 tilts the VR chair 46 backward by a predetermined angle θ5. The VR chair 46 is controlled accordingly. In accordance with this, the region image extracting unit 425 extracts a region image 44i that is rotated upward by a predetermined angle θ7 from the region image 44i shown by the two-dot chain line that was extracted before tilting by the angle θ5, and mounts the region image 44i. The signal is supplied to the display 44.

As shown in FIG. 12B, when the vehicle 10 traveling forward decelerates and the acceleration sensor 13 detects the backward angle θ4, the chair control unit 4201 tilts the VR chair 46 forward by a predetermined angle θ6. The VR chair 46 is controlled accordingly. In accordance with this, the region image extracting unit 425 extracts a region image 44i that is rotated downward by a predetermined angle θ8 from the region image 44i shown by the two-dot chain line that was extracted before tilting by the angle θ6, and mounts the region image 44i by a predetermined angle θ8. The signal is supplied to the display 44.

The angle θ5 may be the same angle as the angle θ4, or may be a different angle from the angle θ4. The angle θ7 may be the same angle as the angle θ5, or may be a different angle from the angle θ5. The angle θ6 may be the same angle as the angle θ4, or may be a different angle from the angle θ4. The angle θ8 may be the same angle as the angle θ6, or may be a different angle from the angle θ6.

Even if the front and rear angles θ4 are the same, it is preferable to make the angle θ6 smaller than the angle θ5. The user Us is more likely to feel fear when tilting the VR chair 46 forward than when tilting it backward. Therefore, the angle θ6 is preferably an angle obtained by multiplying the angle θ5 by a value less than 1. For example, let angle θ6 be 0.8×angle θ5.

Processing executed in the third embodiment will be described using the flowchart shown in FIG. 13. In FIG. 13, when the process is started, the control unit 42 determines in step S31 whether the acceleration sensor 13 has detected the forward angle θ4. If the acceleration sensor 13 does not detect the forward angle θ4 (NO), the control unit 42 determines in step S32 whether the acceleration sensor 13 detects the backward angle θ4. Note that the front angle θ4 and the rear angle θ4 are not necessarily the same angle, but are each arbitrary angles.

If the acceleration sensor 13 does not detect the rear angle θ4 (NO), the control unit 42 repeats the processing of steps S31 and S32.

If the acceleration sensor 13 detects the forward angle θ4 in step S31 (YES), the chair control unit 4201 tilts the VR chair 46 backward by the angle θ5, and the area image extraction unit 425 tilts the VR chair 46 upward. A region image 44i rotated by an angle θ7 is extracted. If the acceleration sensor 13 detects the backward angle θ4 in step S32 (YES), the chair control unit 4201 tilts the VR chair 46 forward by an angle θ6 in step S34, and the area image extraction unit 425 tilts the VR chair 46 downward. A region image 44i rotated by an angle θ8 is extracted.

In step S35 following step S33 or S34, the control unit 42 determines whether the forward or backward angle of the acceleration sensor 13 has become 0. If the acceleration sensor 13 does not detect an angle of 0 (NO), the control unit 42 or the image processing unit 420 repeats the processing of steps S31 to S35. If the acceleration sensor 13 detects an angle of 0 (YES), in step S36, the chair control unit 4201 returns the front or rear angle of the VR chair 46 to 0, and the area image extraction unit 425 returns the area at the original angle. Extract the image 44i.

In step S37, the control unit 42 determines whether or not to end receiving the image data from the image transmission server 31. If the reception of the image data is not completed (NO), the control unit 42 or the image processing unit 420 repeats the processing of steps S31 to S37. If the reception of the image data is completed (YES), the control unit 42 ends the process.

According to the third embodiment described above, in addition to the effects achieved by the first embodiment, it is possible to provide the user Us with a sense of presence corresponding to the feeling that the occupant of the vehicle 10 feels when the vehicle 10 accelerates or decelerates. I can do it. Note that in the third embodiment, only the VR chair 46 may be tilted without rotating the area image 44i. Of course, it is preferable to rotate the area image 44i in response to tilting the VR chair 46.

<Fourth embodiment>
The fourth embodiment corrects the inclination of the horizontal plane of the omnidirectional image in the same manner as the first embodiment, sets the front of the subject, and then uses a method different from the second and third embodiments to tell the user Us to drive the vehicle. This is an image adjustment device and a VR image display system 40 configured to provide a sense of realism corresponding to the movements of 10. As shown in FIG. 14, the control unit 42 includes a chair control unit 4201. The image processing section 420 included in the control section 42 has the same configuration as in FIG. 3.

As shown in FIG. 15, in the fourth embodiment, a case is assumed in which a vehicle 10 traveling on a road R0 travels on an uphill slope R1 and jumps out from a step R12 at the boundary between the uphill slope R1 and the road R2. ing. At this time, the vehicle 10 that jumped out from the step R12 traces a ballistic trajectory Bt, lands on the road R2, and continues traveling. Assuming that the vehicle 10 traveling on the road R0 is accelerating at an acceleration 10a, the acceleration detected by the acceleration sensor 13 is the square root of the sum of the square of the acceleration 10a and the square of the gravitational acceleration G. The gravitational acceleration is greater than G.

While the vehicle 10 is jumping out of the step R12 and drawing the ballistic trajectory Bt, the acceleration detected by the acceleration sensor 13 is 0 or an extremely small value. Therefore, the timing at which the acceleration detected by the acceleration sensor 13 suddenly decreases from a predetermined value equal to or higher than the gravitational acceleration G can be determined to be the timing at which the vehicle 10 starts traveling on the ballistic trajectory Bt. Further, when the vehicle 10 lands on the road R2, the acceleration sensor 13 detects an acceleration equal to or higher than the gravitational acceleration G. Therefore, the timing at which the acceleration detected by the acceleration sensor 13 suddenly increases from 0 or an extremely small value can be determined to be the timing at which the vehicle 10 has finished traveling on the ballistic trajectory Bt.

When the vehicle 10 is traveling on the road R0 and the uphill slope R1, the height of the VR chair 46 is at the reference height. When the input acceleration detection signal suddenly decreases from a predetermined value, the chair control unit 4201 lowers the VR chair 46 by a predetermined height in a short period of time, and then gradually returns it to the reference height. Control the chair 46. Furthermore, when the input acceleration detection signal suddenly increases from 0 or an extremely small value, the chair control unit 4201 raises the VR chair 46 by a predetermined height in a short time, and then gradually returns it to the reference height. The VR chair 46 is controlled to return.

Processing executed in the fourth embodiment will be described using the flowchart shown in FIG. 16. In FIG. 16, when the process is started, the control unit 42 determines in step S41 whether or not the start of the ballistic trajectory Bt has been detected. If the start of the ballistic trajectory Bt is not detected (NO), the control unit 42 repeats the process of step S41. If the start of the ballistic trajectory Bt is detected (YES), the chair control unit 4201 lowers the VR chair 46 for a first time in step S42, and lowers the VR chair 46 for a longer time than the first time in step S43. Raise it in the second time.

Subsequently, in step S44, the control unit 42 determines whether or not the end of the ballistic trajectory Bt has been detected. If the end of the ballistic trajectory Bt is not detected (NO), the control unit 42 repeats the process of step S44. If the end of the ballistic trajectory Bt is detected (YES), the chair control unit 4201 raises the VR chair 46 for the first time in step S45, and lowers the VR chair 46 for the second time in step S46. .

The first time in step S45 may not be the same time as the first time in step S42. The second time in step S46 may not be the same time as the second time in step S43.

In step S47, the control unit 42 determines whether or not to end receiving the image data from the image transmission server 31. If the reception of the image data is not completed (NO), the control unit 42 (chair control unit 4201) repeats the processing of steps S41 to S47. If the reception of the image data is completed (YES), the control unit 42 ends the process.

According to the fourth embodiment described above, in addition to the effects of the first embodiment, the user Us is provided with a feeling corresponding to the feeling that the occupant of the vehicle 10 feels when the vehicle 10 moves along the trajectory Bt. It can give a sense of presence.

<Fifth embodiment>
The fifth embodiment corrects the inclination of the horizontal plane of the omnidirectional image as in the first embodiment, sets the front of the subject, and then uses a method different from the fourth embodiment to tell the user Us that the vehicle 10 is This is an image adjustment device and a VR image display system 40 configured to provide a sense of realism corresponding to the movement of the vehicle 10 when the vehicle 10 travels along a trajectory Bt. The configuration of the control section 42 in the fifth embodiment is the same as that in FIG. 14.

In FIG. 17, when the vehicle 10 is traveling on the road R0 and the uphill slope R1, the VR chair 46 is at the reference angle. When the input acceleration detection signal rapidly decreases from a predetermined value, the chair control unit 4201 controls the VR chair 46 to tilt the VR chair 46 backward by an angle θ9.

The acceleration detected by the acceleration sensor 13 is minimum at the peak Btp of the ballistic trajectory Bt. The chair control unit 4201 controls the VR chair 46 to tilt the VR chair 46 forward by an angle θ10 when the acceleration detected by the acceleration sensor 13 becomes the minimum. Since the peak Btp cannot be detected until the trajectory Bt passes the peak Btp, the VR chair 46 tilted backward starts rotating forward after the vehicle 10 passes the peak Btp. Become.

When the vehicle 10 lands on the road R2 and the acceleration detection signal rapidly increases, the chair control unit 4201 controls the VR chair 46 to return the longitudinal angle of the VR chair 46 to the reference angle.

Processing executed in the fifth embodiment will be described using the flowchart shown in FIG. 18. In FIG. 18, when the process is started, the control unit 42 determines in step S51 whether or not the start of the ballistic trajectory Bt has been detected. If the start of the ballistic trajectory Bt is not detected (NO), the control unit 42 repeats the process of step S51. If the start of the ballistic trajectory Bt is detected (YES), the chair control unit 4201 tilts the VR chair 46 backward by an angle θ9 in step S52.

In step S53, the control unit 42 determines whether the vehicle 10 has reached the peak Btp of the ballistic trajectory Bt. If the vehicle 10 has not reached the peak Btp (NO), the chair control unit 4201 repeats the process of step S52. If the vehicle 10 has reached the peak Btp (YES), the chair control unit 4201 tilts the VR chair 46 forward by an angle θ10 in step S54.

Subsequently, in step S55, the control unit 42 determines whether or not the end of the ballistic trajectory Bt has been detected. If the end of the ballistic trajectory Bt is not detected (NO), the control unit 42 (chair control unit 4201) repeats the processing of steps S54 and S55. If the end of the ballistic trajectory Bt is detected (YES), the chair control unit 4201 returns the longitudinal angle of the VR chair 46 to the reference angle in step S56.

In step S57, the control unit 42 determines whether or not to end receiving the image data from the image transmission server 31. If the reception of the image data is not completed (NO), the control unit 42 (chair control unit 4201) repeats the processing of steps S51 to S57. If the reception of the image data is completed (YES), the control unit 42 ends the process.

According to the fifth embodiment described above, in addition to the effects of the first embodiment, when the vehicle 10 moves along the ballistic trajectory Bt, It is possible to provide a sense of realism that corresponds to the sensation experienced by passengers.

<Sixth embodiment>
In the second, third, and fifth embodiments, the angles θ2, θ3, and θ5 to θ10 are set according to the acceleration detected by the acceleration sensor 13. When the vehicle 10 drives abnormally or an accident occurs, the acceleration sensor 13 may detect abnormal acceleration. In this case, it is not preferable to set the angles θ2, θ3, θ5 to θ10 according to the acceleration detected by the acceleration sensor 13.

Therefore, in the sixth embodiment, the processing shown in the flowchart of FIG. 19 is executed in the configurations of the second, third, and fifth embodiments. In FIG. 19, the control unit 42 determines in step S61 whether any of the angles θ2, θ3, and θ5 to θ10 has been calculated. Upper limit values of angles θ2, θ3, and θ5 to θ10 are set in advance in the control unit 42. In step S62, the control unit 42 determines whether the angle calculated in step S61 is within an upper limit value.

If the angle calculated in step S61 is within the upper limit value (YES), the control unit 42 adopts the calculated value and ends the process in step S63. If the angle calculated in step S61 is not within the upper limit value (NO), the control unit 42 limits the angle to the upper limit value in step S64 and ends the process.

Here, upper limit values of angles θ2, θ3, θ5 to θ10 are set, but in addition to the upper limit values of angles, an upper limit value of angular velocity may be set to limit the angles within the upper limit value of angular velocity. In particular, it is preferable that the angular velocity when tilting the VR chair 46 in the left-right direction or the front-back direction is within the upper limit value.

The upper limit value used in step S62 in FIG. 19 may vary depending on whether or not the user Us is wearing a safety device such as the seat belt 461. As shown in the flowchart of FIG. 20, the control unit 42 determines in step S65 whether the user Us is wearing a safety device. If the user Us is wearing the safety device (YES), the control unit 42 sets the first upper limit value in step S66 and ends the process.

If the user Us is not wearing the safety device in step S65 (NO), the control unit 42 sets a second upper limit value smaller than the first upper limit value in step S67, and ends the process.

As described above, in the sixth embodiment, the chair control unit 4201 controls the VR chair 46 to tilt in the left-right direction or the front-back direction according to the acceleration detection signal. If the calculated value of the angle at which the VR chair 46 is tilted according to the acceleration detection signal is within a predetermined upper limit value, the chair control unit 4201 tilts the chair at the calculated value, and if the calculated value exceeds the upper limit value, The VR chair 46 is controlled to be tilted at the upper limit value.

Specifically, when the acceleration detection signal indicates that the moving object is turning left, the chair control unit 4201 tilts the VR chair 46 to the right by a predetermined angle, and the acceleration detection signal indicates that the moving object is turning left. When indicating a right turn, the VR chair 46 is controlled to be tilted to the left by a predetermined angle.

In accordance with this, the image tilting unit 426 preferably tilts the area image 44i to be supplied to the head mounted display 44 by a predetermined angle to the right when the acceleration detection signal indicates that the moving object is turning left. . The image tilting unit 426 preferably tilts the area image 44i supplied to the head mounted display 44 by a predetermined angle to the left when the acceleration detection signal indicates that the moving object is turning right.

At this time, if the calculated value of the angle at which the area image 44i is tilted according to the acceleration detection signal is within a predetermined upper limit value, the area image 44i is tilted by the calculated value, and if the calculated value exceeds the upper limit value, the area image 44i is tilted. It is best to control the angle so that it is tilted at an upper limit value.

The chair control unit 4201 preferably tilts the VR chair 46 backward by a predetermined angle when the acceleration detection signal indicates that the moving object moving forward is accelerating. The chair control unit 4201 preferably tilts the VR chair 46 forward by a predetermined angle when the acceleration detection signal indicates that the moving object moving forward is decelerating.

In accordance with this, when the acceleration detection signal indicates that the moving body moving forward has accelerated, the region image extraction unit 425 extracts the region image 44i rotated upward by a predetermined angle and It is preferable to supply it to the mounted display 44. It is preferable that the region image extraction unit 425 extracts a region image 44i rotated downward by a predetermined angle and supplies it to the head-mounted display 44 when indicating that the moving object moving forward has decelerated.

At this time, if the calculated value of the angle at which the area image 44i is rotated upward or downward calculated according to the acceleration detection signal is within a predetermined upper limit value, the area image 44i is rotated by the calculated value, and the calculated value is the upper limit value. If it exceeds the maximum value, it is preferable to control the area image 44i to rotate by the upper limit value.

The chair control unit 4201 controls the VR chair 46 to tilt the VR chair 46 backward, which is at the reference angle, when the acceleration detection signal indicates the start of the movement of the moving object on the ballistic trajectory Bt. Good. The chair control unit 4201 preferably controls the VR chair 46 so as to tilt the VR chair 46 forward when the acceleration detection signal passes the peak Btp of the ballistic trajectory Bt. The chair control unit 4201 preferably controls the VR chair 46 to return the VR chair 46 to the reference angle when the acceleration detection signal indicates the end of the movement of the moving body on the ballistic trajectory Bt.

According to the sixth embodiment, in addition to the effects of the second, third, and fifth embodiments, the safety of the VR image display system 40 can be improved.

The present invention is not limited to the first to sixth embodiments described above, and various modifications can be made without departing from the gist of the present invention.

10 Vehicle (mobile object)
11, 41 Communication section 12 Omnidirectional camera 13 Acceleration sensor 20 Network 31 Image transmission server 32 Memory 40 Virtual reality image display system 42 Control section 43 Image generation section 44 Head-mounted display 45 Glove-type controller 46 VR chair 47 Operation section 420 Image processing section 421 Image superimposition section 422 Image rotation section 423 Vanishing point detection section 424 Front setting section 425 Region image extraction section 426 Image tilting section
4201 Chair control section
4202 Mode setting section

Claims (5)

an image generation unit that generates a spherical image;
A superimposed image in which the spherical image is superimposed on an omnidirectional image taken of the subject by an omnidirectional camera disposed on a moving object, or a superimposed image in which the omnidirectional image is extracted in accordance with the direction in which the user wearing the head-mounted display faces. a region image extraction unit that supplies the obtained region image to the head mounted display;
With the regional image of the superimposed image extracted by the regional image extraction unit displayed on the head-mounted display, the omnidirectional image is rotated by an operation of rotating the spherical image, and the horizontal plane of the omnidirectional image is rotated. an image rotation unit that corrects the tilt;
a vanishing point detection unit that detects a vanishing point of the omnidirectional image while the moving object is moving and the omnidirectional image is changing;
The omnidirectional image is configured such that the front of the omnidirectional image is specified based on the vanishing point, and the front of the omnidirectional image is associated with a region image extracted when the user is facing forward. a front setting section that rotates while maintaining the horizontal plane corrected by the image rotation section;
Equipped with
When the mobile object moving forward accelerates, the region image extraction unit extracts a region image rotated upward by a predetermined angle and supplies it to the head mounted display, so that the moving object moves forward. When the moving body decelerates, an image adjustment device extracts a region image rotated downward by a predetermined angle and supplies the extracted area image to the head mounted display. Image data of an omnidirectional image taken of a subject by an omnidirectional camera placed on a moving object and an acceleration detection signal detected by an acceleration sensor attached to the moving object or the omnidirectional camera are sent from an image transmission server. a communication section for receiving;
a head-mounted display mounted on a user's head to show the omnidirectional image to the user;
a controller operated by the user;
a chair on which the user sits;
an image generation unit that generates a spherical image;
an image superimposition unit that superimposes the spherical image on the omnidirectional image to generate a superimposed image;
a region image extracting unit that supplies, to the head mounted display, a region image extracted from the superimposed image or the omnidirectional image corresponding to the direction in which the user faces;
an image rotation unit that rotates the superimposed image by operating the controller to rotate the spherical image while the user is sitting on the chair, and corrects the inclination of the horizontal plane of the omnidirectional image; and,
a vanishing point detection unit that detects a vanishing point of the omnidirectional image when the omnidirectional camera moves and the omnidirectional image changes;
The omnidirectional image is configured such that the front of the omnidirectional image is specified based on the vanishing point, and the front of the omnidirectional image is associated with a region image extracted when the user is facing forward. a front setting section that rotates while maintaining the horizontal plane corrected by the image rotation section;
When the acceleration detection signal indicates that the moving body moving forward is accelerating, the chair is tilted backward by a predetermined angle, and the acceleration detection signal indicates that the moving body moving forward is accelerating. a chair control unit that tilts the chair forward by a predetermined angle when indicating that the body is decelerating;
A virtual reality image display system comprising: When the mobile object moving forward accelerates, the region image extraction unit extracts a region image rotated upward by a predetermined angle and supplies it to the head mounted display, so that the moving object moves forward. 3. The virtual reality image display system according to claim 2, wherein when the moving object decelerates, a region image rotated downward by a predetermined angle is extracted and supplied to the head-mounted display. The controller is a glove-type controller worn on the user's hand,
The image rotation unit rotates the superimposed image in response to an operation by the user who is virtually located inside the spherical image to rotate the spherical image using the glove-type controller. Virtual reality image display system. Generate a spherical image,
A superimposed image in which the spherical image is superimposed on an omnidirectional image taken of the subject by an omnidirectional camera disposed on a moving object, or a superimposed image in which the omnidirectional image is extracted in accordance with the direction in which the user wearing the head-mounted display faces. supplying the area image to the head-mounted display;
Correcting the inclination of the horizontal plane of the omnidirectional image by rotating the omnidirectional image by an operation of rotating the spherical image while displaying the extracted area image of the superimposed image on the head mounted display;
detecting a vanishing point of the omnidirectional image while the moving object is moving and the omnidirectional image is changing;
Identifying the front of the omnidirectional image based on the vanishing point, and correcting the omnidirectional image so as to associate the front of the omnidirectional image with a region image extracted when the user is facing forward. Rotate while maintaining the horizontal plane,
When the moving body moving forward accelerates, a region image rotated upward by a predetermined angle is extracted and supplied to the head mounted display, and when the moving body moving forward decelerates, the area image is extracted and supplied to the head mounted display. , extracting a regional image rotated downward by a predetermined angle and supplying it to the head-mounted display.

Images