JP2020134616A - Image adjustment system, image adjustment device, and image adjustment method - Google Patents

Abstract

To provide an image adjustment system capable of easily correcting the horizontality or zenith of an image, when the horizontal direction of the image is erroneously detected or the specification of the horizontality or zenith is shifted.SOLUTION: An image adjustment system 201 includes a camera 202, an image adjustment device 210, an image display device 204, a controller 205, and a server 206. The server 206 specifies the image display device 204 or its user US. The image adjustment device 210 includes an image generation part 212 and an image processing part 211. The image processing part 211 acquires a spherical image VSS2 from the image generation part 212 on the basis of instruction information NN2 outputted from the controller 205, displays it on the image display device 204, and rotates the spherical image VSS2, to adjust a photographed image IM2 displayed on the image display device 204 according to the rotation of the spherical image VSS2.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image adjustment system, an image adjustment device, and an image adjustment method.

In recent years, a head-mounted display has attracted attention as an image display device. By displaying an image while the head-mounted display is attached to the user's head, the user can obtain a feeling of entering a virtual space (immersive feeling). As described in Patent Document 1, the head-mounted display can display an image taken by an external camera via a neckwork.

Japanese Unexamined Patent Publication No. 2005-56295

However, with a conventional head-mounted display, it is difficult to detect the horizontal direction of an image depending on the displayed image, and the horizontal direction may be detected incorrectly. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the image is detected incorrectly, or if the horizontal or zenith designation is incorrect, the displayed image does not match the user's sense of gravity, and the user feels uncomfortable.

The present invention provides an image adjustment system, an image adjustment device, and an image that can easily correct the horizontal or zenith of an image when the horizontal direction of the image is incorrectly detected or the horizontal or zenith is not specified. The purpose is to provide an adjustment method.

The present invention outputs instruction information to a camera, an image adjusting device that adjusts a captured image captured by the camera, an image display device that displays the captured image adjusted by the image adjusting device, and the image adjusting device. The image adjusting device includes an image generating unit for generating a spherical image and the image generating unit based on the instruction information. The spherical image is acquired from the unit and displayed on the image display device, the spherical image is rotated based on the instruction information, and the spherical image is displayed on the image display device in correspondence with the rotation of the spherical image. Provided is an image adjustment system including an image processing unit for adjusting the captured image.

The present invention obtains the spherical image from the image generation unit based on an image generation unit that generates a spherical image and instruction information acquired from the controller, displays the spherical image on an image display device specified by a server, and displays the spherical image. An image including an image processing unit that rotates the spherical image based on the instruction information and adjusts the captured image captured by the camera and displayed on the image display device in accordance with the rotation of the spherical image. Provide an adjusting device.

In the present invention, the server identifies the image display device, the image processing unit acquires instruction information from the controller, the image processing unit acquires a spherical image from the image generation unit based on the instruction information, and the server. The image display device specified by the above displays the spherical image, the image processing unit rotates the spherical image based on the instruction information, and the image processing unit is photographed by a camera to display the image. Provided is an image adjustment method for adjusting the captured image displayed in the above in accordance with the rotation of the spherical image.

According to the image adjustment system, the image adjustment device, and the image adjustment method of the present invention, the horizontal or zenith of the image can be easily adjusted when the horizontal direction of the image is incorrectly detected or the horizontal or zenith is not specified. It can be corrected.

It is a block diagram which shows the image adjustment system of 1st Embodiment. It is a figure which shows the relationship between a spherical image and a user. It is a flowchart which shows an example of the image adjustment method of 1st Embodiment. It is a block diagram which shows the image adjustment system of 2nd Embodiment. It is a figure which shows an example of the distribution of the correction value for each user. It is a flowchart which shows an example of the image adjustment method of 2nd Embodiment. It is a flowchart which shows an example of the image adjustment method of 2nd Embodiment. It is a figure which shows typically the relationship between a user and a horizontal plane in a state where a camera is moving. It is a figure which shows an example of a wall surface. It is a flowchart which shows an example of the image adjustment method of 3rd Embodiment. It is a flowchart which shows an example of the image adjustment method of 3rd Embodiment. It is a flowchart which shows an example of the image adjustment method of 4th Embodiment. It is a figure which shows an example of the image displayed on the image display device. It is a figure which shows the state which the image shown in FIG. 12A is horizontally adjusted.

[First Embodiment]
A configuration example of the image adjustment system of the first embodiment will be described with reference to FIG. The image adjustment system 101 includes a camera 102, a communication unit 103, an image display device 104, a controller 105, and an image adjustment device 110. The image adjusting device 110 includes an image processing unit 111, an image generation unit 112, and a communication unit 113.

The camera 102 is an omnidirectional camera (360 degree camera) capable of photographing a range of 360 degrees. The communication unit 103 and the communication unit 113 are connected via a network. The image adjusting device 110 can acquire the captured image IM1 captured by the camera 102 via the communication units 103 and 113 and the network.

A computer device may be used as the image adjusting device 110. A CPU may be used as the image processing unit 111 and the image generation unit 112. The image generation unit 112 and the communication unit 113 may be provided outside the image adjusting device 110.

The captured image IM1 acquired by the image adjusting device 110 is input to the image processing unit 111. The image processing unit 111 recognizes the horizontal direction in the captured image IM1 by analyzing the captured image IM1. The image processing unit 111 may recognize the vertical direction in the captured image IM1 or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM1. The image adjusting device 110 corrects the distortion of the captured image IM1, executes image processing such as adjusting the horizontal direction of the captured image IM1, and outputs the image-processed captured image IM1 to the image display device 104.

The image display device 104 displays the captured image IM1 image-processed by the image adjustment device 110. The image display device 104 is, for example, a head-mounted display. The controller 105 is a globe-type controller used for, for example, VR (Virtual Reality).

FIG. 2 schematically shows a state in which the image display device 104 is attached to the head of the user US and the controller 105 is attached to the hand of the user US. The symbol ZE shown in FIG. 2 indicates the zenith. It is desirable that the zenith of the camera 102 and the zenith of the user US coincide with each other. By attaching the image display device 104 to the head of the user US, the user US can see the captured image IM1 image-processed by the image adjusting device 110.

The image display device 104 generates posture information PN1 based on the direction in which the user US is facing, the posture of the user US, and the like when the image display device 104 is attached to the head of the user US. The image processing unit 111 acquires the posture information PN1 from the image display device 104. That is, the image processing unit 111 acquires the posture information PN1 based on the posture of the image display device 104. Based on the posture information PN1, the image processing unit 111 displays an image of a region corresponding to a state such as the direction in which the user US is facing and the posture of the user US from the captured image IM1 taken by the camera 102. Displayed on the device 104.

The controller 105 generates instruction information NN1 based on the movement or posture of the hand of the user US or the movement or posture of the finger of the user US when the controller 105 is attached to the hand of the user US. .. Hereinafter, the hand or finger is simply abbreviated as a hand. The image processing unit 111 acquires the instruction information NN1 from the controller 105. The image processing unit 111 can change or adjust the captured image IM1 displayed on the image display device 104 based on the instruction information NN1.

The image generation unit 112 generates a spherical image VSS1 which is a virtual image composed of spherical surfaces which are CG (Computer Graphics). The image generation unit 112 stores the spherical image VSS1 in the built-in memory or an external memory.

The image processing unit 111 acquires the spherical image VSS1 from the image generation unit 112 based on the instruction information NN1 and displays it on the image display device 104. FIG. 2 shows an image of the user US when the user US sees the spherical image VSS1 displayed on the image display device 104 in a state where the image display device 104 is attached to the head of the user US. It is shown schematically.

When the user US sees the spherical image VSS1 displayed on the image display device 104 in a state where the image display device 104 is attached to the head of the user US, the spherical image VSS1 is the user. It is arranged around the US and the image display device 104, and is set so that the hand of the user US is displayed within the reach of the spherical image VSS1. The user US moves the hand on which the controller 105 is attached to a position corresponding to the spherical image VSS1 displayed on the image display device 104, so that the hand of the user US comes into contact with the spherical image VSS1. I feel like I'm doing it.

The controller 105 may have an actuator arranged at a portion that comes into contact with the hand of the user US. The image processing unit 111 operates the actuator when it is determined that the hand of the user US has moved to the position corresponding to the spherical image VSS1 based on the instruction information NN1. When the actuator applies pressure to the hand of the user US, the user US can actually feel the hand touching the spherical image VSS1.

When the user US moves the hand on which the controller 105 is attached in an arbitrary direction while the spherical image VSS1 is displayed on the image display device 104, the image processing unit 111 displays the instruction information NN1. Based on this, image processing is performed so that the spherical image VSS1 and the captured image IM1 displayed on the image display device 104 move according to the moving direction, moving speed, and moving destination position of the hand of the user US. To execute.

The user US can rotate the spherical image VSS1 in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed to an arbitrary position. it can. That is, the user US can rotate the spherical image VSS1 by the movement of his / her hand. The image processing unit 111 moves the captured image IM1 in correspondence with the rotation of the spherical image VSS1.

The image processing unit 111 indicates to which position on the coordinates of the spherical image VSS1 the zenith ZE before the user US rotates the spherical image VSS1 moves by rotating the spherical image VSS1 by the user US. Can be determined. The image processing unit 111 is a spherical image before and after the user US rotates the spherical image VSS1 based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS1. The amount of change in VSS1 is calculated.

The amount of change in the spherical image VSS1 is the amount of rotation (rotation angle) with the X-axis as the rotation axis, the amount of rotation (rotation angle) with the Y-axis as the rotation axis, and the rotation of the Z-axis in the spherical image VSS1. The amount of rotation (rotation angle) as the axis corresponds to the amount of rotation (rotation angle) of the spherical image VSS1 synthesized. The image processing unit 111 stores the amount of change in the spherical image VSS1 as the correction value CV1. That is, the correction value CV1 is calculated based on the rotation direction of the spherical image VSS1 and the movement amount or movement angle of the zenith ZE (rotation angle of the spherical image VSS1).

The image processing unit 111 may save the coordinates on the spherical image VSS1 of the zenith ZE after the user US rotates the spherical image VSS1 as the correction value CV1. The image processing unit 111 stores the correction value CV1 in the internal memory or an external memory.

The image processing unit 111 detects the horizontal direction of the captured image IM1. However, it is difficult for the image processing unit 111 to detect the horizontal direction depending on the captured image IM1, and the horizontal direction may be erroneously detected. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the captured image IM1 is incorrectly detected, or if the horizontal or zenith designation is deviated, the captured image IM1 displayed on the image display device 104 does not match the gravitational sensation of the user US, so that the user US I feel uncomfortable.

An example of the image adjustment method of the first embodiment will be described with reference to the flowchart shown in FIG. Specifically, an example of a method of adjusting the horizontal of the captured image IM1 will be described. The image display device 104 is attached to the head of the user US, and the controller 105 is attached to the hand of the user US. The captured image IM1 is displayed on the image display device 104.

When the user US determines that the captured image IM1 displayed on the image display device 104 is not horizontal, in FIG. 3, when the user US operates the controller 105, the image processing unit 111 moves to step S101. , The image display device 104 is made to display the setting screen. When the user US operates the controller 105 to select a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 111 selects the item selected in step S102. The process shifts to the corresponding predetermined process mode. When the horizontal adjustment item is selected, the image processing unit 111 shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM1.

In step S103, the image processing unit 111 acquires the spherical image VSS1 from the image generation unit 112 and displays it on the image display device 104. The captured image IM1 and the spherical image VSS1 are mixed and displayed on the image display device 104. When the user US rotates the spherical image VSS1 so that the captured image IM1 is horizontal, the image processing unit 111 causes the image display device 104 to correspond to the rotation of the spherical image VSS1 in step S104. The displayed captured image IM1 is moved. The user US may perform the operation of rotating the spherical image VSS1 a plurality of times until the captured image IM1 becomes horizontal.

When the user US determines that the captured image IM1 is horizontal, the image processing unit 111 causes the image display device 104 to display the setting screen in step S105 by operating the controller 105. By leveling the captured image IM1 displayed on the image display device 104, the zenith of the camera 102 and the zenith of the user US can be matched.

When the user US operates the controller 105 and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 111 corresponds to the selected item in step S106. The processing is shifted to the predetermined processing mode. When the end item is selected, the image processing unit 111 shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S107, the image processing unit 111 acquires the amount of rotation (rotation angle) before and after the spherical image VSS1 rotates as the amount of change in the spherical image VSS1. In step S108, the image processing unit 111 saves the amount of change in the spherical image VSS1 as the correction value CV1 and ends the processing.

In the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, the spherical image VSS1 is displayed on the image display device 104. According to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, it is used when the horizontal direction of the captured image IM1 is erroneously detected or the horizontal or zenith ZE designation is deviated. The person US can adjust the captured image IM1 displayed on the image display device 104 so as to be horizontal by operating the controller 105 to rotate the spherical image VSS1.

Therefore, according to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, when the horizontal direction of the captured image IM1 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM1.

According to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, when the correction value CV1 is stored, the image processing unit 111 reads out the correction value CV1 and the camera 102 takes a picture. The captured image IM1 can be image-adjusted based on the correction value CV1 and displayed on the image display device 104.

[Second Embodiment]
A configuration example of the image adjustment system of the second embodiment will be described with reference to FIG. The image adjustment system 201 includes a camera 202, a communication unit 203, an image display device 204, a controller 205, an image adjustment device 210, and a server 206. The image adjusting device 210 includes an image processing unit 211, an image generation unit 212, and a communication unit 213.

The camera 202, the communication unit 203, the image display device 204, the controller 205, and the image adjustment device 210 are attached to the camera 102, the communication unit 103, the image display device 104, the controller 105, and the image adjustment device 110 of the first embodiment, respectively. Equivalent to. The image processing unit 211, the image generation unit 212, and the communication unit 213 correspond to the image processing unit 111, the image generation unit 112, and the communication unit 113 of the first embodiment, respectively.

The image adjusting device 210 can acquire the captured image IM2 captured by the camera 202 via the communication units 203 and 213 and the network. The server 206 connects to the camera 202 via the network and the communication unit 203, and connects to the image adjusting device 210 via the network and the communication unit 213.

The server 206 may acquire the captured image IM2 captured by the camera 202 via the communication unit 203 and the network, and the image adjusting device 210 may acquire the captured image IM2 from the server 206 via the network and the communication unit 213.

The captured image IM2 acquired by the image adjusting device 210 is input to the image processing unit 211. The image processing unit 211 recognizes the horizontal direction in the captured image IM2 by analyzing the captured image IM2. The image processing unit 211 may recognize the vertical direction in the captured image IM2, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM2. The image adjusting device 210 corrects the distortion of the captured image IM2, executes image processing such as adjusting the horizontal of the captured image IM2, and outputs the image-processed captured image IM2 to the image display device 204. The image display device 204 displays the captured image IM2 image-processed by the image adjustment device 210.

The server 206 corrects the distortion of the captured image IM2 captured by the camera 202, executes image processing such as adjusting the level of the captured image IM2, and outputs the image-processed captured image IM2 to the image adjusting device 210. You may. By attaching the image display device 204 to the head of the user US, the user US can see the captured image IM2 image-processed by the image adjusting device 210 or the server 206.

The image display device 204 generates posture information PN2 based on the direction in which the user US is facing, the posture of the user US, and the like when the image display device 204 is attached to the head of the user US. The image processing unit 211 acquires the posture information PN2 from the image display device 204. That is, the image processing unit 211 acquires the posture information PN2 based on the posture of the image display device 204. Based on the posture information PN2, the image processing unit 211 displays an image of a region corresponding to a state such as the direction in which the user US is facing and the posture of the user US from the captured image IM2 taken by the camera 202. Displayed on device 204.

The controller 205 generates instruction information NN2 based on a state such as the movement or posture of the user US's hand while it is attached to the user US's hand. The image processing unit 211 acquires the instruction information NN2 from the controller 205. The image processing unit 211 can change or adjust the image displayed on the image display device 204 based on the instruction information NN2.

The image generation unit 212 generates a spherical image VSS2 which is a virtual image composed of a spherical surface which is CG. The image generation unit 212 stores the spherical image VSS2 in the built-in memory or an external memory.

The image processing unit 211 acquires the spherical image VSS2 from the image generation unit 212 based on the instruction information NN2 and displays it on the image display device 204. The spherical image VSS2 corresponds to the spherical image VSS1 of the first embodiment. The user US moves the hand on which the controller 205 is attached to a position corresponding to the spherical image VSS2 displayed on the image display device 204, so that the hand of the user US comes into contact with the spherical image VSS2. I feel like I'm doing it.

In the controller 205, the actuator may be arranged at a portion that comes into contact with the hand of the user US. The image processing unit 211 operates the actuator when it is determined that the hand of the user US has moved to the position corresponding to the spherical image VSS2 based on the instruction information NN2. When the actuator applies pressure to the hand of the user US, the user US can actually feel the hand touching the spherical image VSS2.

When the user US moves the hand on which the controller 205 is mounted in an arbitrary direction while the spherical image VSS2 is displayed on the image display device 204, the image processing unit 211 displays the instruction information NN2. Based on this, image processing is performed so that the spherical image VSS2 and the captured image IM2 displayed on the image display device 204 move according to the moving direction, moving speed, and moving destination position of the hand of the user US. To execute.

The user US can rotate the spherical image VSS2 in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed to an arbitrary position. it can. That is, the user US can rotate the spherical image VSS2 by the movement of his / her hand. The image processing unit 211 moves the captured image IM2 in correspondence with the rotation of the spherical image VSS2.

The image processing unit 211 indicates to which position on the coordinates of the spherical image VSS2 the zenith ZE before the user US rotates the spherical image VSS2 has moved by rotating the spherical image VSS2 by the user US. Can be determined. The image processing unit 211 is a spherical image before and after the user US rotates the spherical image VSS2 based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS2. Calculate the amount of change in VSS2. The amount of change in the spherical image VSS2 corresponds to the amount of change in the spherical image VSS1 of the first embodiment.

The image processing unit 211 outputs the amount of change in the spherical image VSS2 as the correction value CV2 to the server 206 via the communication unit 213 and the network. That is, the correction value CV2 is calculated based on the rotation direction of the spherical image VSS2 and the movement amount or movement angle of the zenith ZE (rotation angle of the spherical image VSS2). The image processing unit 211 may output the coordinates on the spherical image VSS2 of the zenith ZE after the user US rotates the spherical image VSS2 to the server 206 as the correction value CV2.

The server 206 may acquire the correction value CV2 from the image adjusting device 210 via the network and the communication unit 213. The server 206 associates the correction value CV2 with the user US or the image display device 204 and stores it in the internal memory or an external memory.

The image processing unit 211 detects the horizontal direction of the captured image IM2. However, it is difficult for the image processing unit 211 to detect the horizontal direction depending on the captured image IM2, and the horizontal direction may be erroneously detected. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the captured image IM2 is incorrectly detected, or if the horizontal or zenith designation is deviated, the captured image IM2 displayed on the image display device 204 does not match the gravitational sensation of the user US, so that the user US I feel uncomfortable.

When the user US wears the image display device 204 on the head, the shape of the head and the wearing state of the image display device 204 differ depending on the user US. Therefore, the horizontal direction of the captured image IM2 displayed on the image display device 204 may differ depending on the user US. The horizontal direction of the captured image IM2 displayed on the image display device 204 may differ depending on the dominant eye of the user US. Therefore, the correction value CV2 may differ depending on the user US.

FIG. 5 schematically shows the distribution of the correction value CV2 for each user US. FIG. 5 schematically shows a state in which the spherical image VSS2 is viewed from above the user US. The front shown in FIG. 5 indicates the front of the user US, and the rear indicates the rear of the user US. The right side shown in FIG. 5 indicates the right side of the user US, and the left side indicates the left side of the user US.

The round symbol shown in FIG. 5 indicates the correction value CV2 in the first user US or the first image display device 204, and the square symbol indicates the correction value CV2 in the second user US or the second image display device 204. The correction value CV2 in 204 is shown, and the triangular symbol indicates the correction value CV2 in the third user US or the third image display device 204. In FIG. 5, the correction value CV2 is distributed to the front right side in the first user US, to the front left side in the second user US, and to the rear side in the third user US. Shown.

The server 206 can identify the user US or the image display device 204 when the user US executes a login operation or the image display device 204 is connected to the network via the communication unit 213. The server 206 stores the correction value CV2 in association with the specified user US or the image display device 204.

An example of the image adjustment method of the second embodiment will be described with reference to the flowcharts shown in FIGS. 6 and 7. Specifically, an example of a method of adjusting the horizontal of the captured image IM2 will be described for each user US or the image display device 204. The image display device 204 is attached to the head of the user US, and the controller 205 is attached to the hand of the user US. The captured image IM2 is displayed on the image display device 204.

In FIG. 6, in step S201, the server 206 displays the user US or the image when the user US executes a login operation or the image display device 204 is connected to the network via the communication unit 213. Identify device 204.

When the user US determines that the captured image IM2 displayed on the image display device 204 is not horizontal, the user US operates the controller 205, and the image processing unit 211 causes the image display device 211 in step S202. Display the setting screen on 204. When the user US operates the controller 205 and selects a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 211 selects the selected item in step S203. The process shifts to the corresponding predetermined process mode. When the horizontal adjustment item is selected, the image processing unit 211 shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM2.

In step S204, the image processing unit 211 acquires the spherical image VSS2 from the image generation unit 212 and displays it on the image display device 204. The captured image IM2 and the spherical image VSS2 are mixed and displayed on the image display device 204. When the user US rotates the spherical image VSS2 so that the captured image IM2 is horizontal, the image processing unit 211 causes the image display device 204 to correspond to the rotation of the spherical image VSS2 in step S205. The displayed captured image IM2 is moved. The user US may perform the operation of rotating the spherical image VSS2 a plurality of times until the captured image IM2 becomes horizontal.

When the user US determines that the captured image IM2 is horizontal, the image processing unit 211 causes the image display device 204 to display the setting screen in step S206 by operating the controller 205. By leveling the captured image IM2 displayed on the image display device 204, the zenith of the camera 202 and the zenith of the user US can be matched.

When the user US operates the controller 205 and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 111 corresponds to the selected item in step S207. The processing is shifted to the predetermined processing mode. When the end item is selected, the image processing unit 211 shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S208, the image processing unit 211 acquires the amount of rotation (rotation angle) before and after the spherical image VSS2 rotates as the amount of change in the spherical image VSS2. Further, the image processing unit 211 outputs the change amount of the spherical image VSS2 as the correction value CV2 to the server 206 via the communication unit 213 and the network. In step S209, the server 206 stores the correction value CV2 in association with the user US or the image display device 204. The server 206 may acquire the correction value CV2 from the image adjusting device 210 via the network and the communication unit 213.

Using the flowchart shown in FIG. 7, an example of a method of adjusting the horizontal of the captured image IM2 when the correction value CV2 is stored in the server 206 in association with the user US or the image display device 204 will be described. The image display device 204 is attached to the head of the user US, and the controller 205 is attached to the hand of the user US. The captured image IM2 is displayed on the image display device 204.

In FIG. 7, the server 206 displays the user US or the image when the user US executes a login operation in step S211 or the image display device 204 is connected to the network via the communication unit 213. Identify device 204.

In step S212, the server 206 reads out the correction value CV2 corresponding to the user US or the image display device 204 specified in step S211 and outputs the correction value CV2 to the image adjusting device 210 via the network and the communication unit 213.

When a plurality of correction values CV2 corresponding to the user US or the image display device 204 are stored in the server 206, the server 206 calculates an average value from the plurality of correction values CV2 and uses this average value as the correction value CV2. It may be output to the image adjusting device 210, or the latest correction value CV2 of the plurality of correction values CV2 may be output to the image adjusting device 210. The image adjusting device 210 may read the correction value CV2 corresponding to the user US or the image display device 204 from the server 206 via the communication unit 213 and the network.

The correction value CV2 output from the server 206 to the image adjusting device 210 is input to the image processing unit 211. In step S213, the image processing unit 211 horizontally adjusts the captured image IM2 captured by the camera 202 based on the correction value CV2, and outputs the captured image IM2 to the image display device 204. The image display device 204 displays the captured image IM2 horizontally adjusted based on the correction value CV2 in step S214. Therefore, the image display device 204 can display the horizontally adjusted captured image IM2 corresponding to the user US.

When the user US determines that the captured image IM2 displayed on the image display device 204 is not horizontal, the user US operates the controller 205 to process the image adjustment system 201 in steps S202 to S209 and S211 to S214. May be executed.

In the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, the spherical image VSS2 is displayed on the image display device 204. According to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, it is used when the horizontal direction of the captured image IM2 is incorrectly detected or the horizontal or zenith ZE designation is deviated. By operating the controller 205 to rotate the spherical image VSS2, the person US can adjust the captured image IM2 displayed on the image display device 204 so as to be horizontal.

Therefore, according to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, when the horizontal direction of the captured image IM2 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM2.

In the image adjusting system 201, the image adjusting device 210, and the image adjusting method of the second embodiment, the correction value CV2 is associated with the user US or the image display device 204 and stored in the server 206. Based on the correction value CV2, the image adjusting device 210 can horizontally adjust the captured image IM2 captured by the camera 202 for each user US or the image display device 204 and display it on the image display device 204. According to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, since the correction value CV2 is associated with the user US or the image display device 204, the captured image IM2 is horizontal. It can be adjusted for each user US.

[Third Embodiment]
A configuration example of the image adjustment system of the third embodiment will be described with reference to FIG. The image adjustment system 301 includes a camera 302, a communication unit 303, an image display device 304, a controller 305, an image adjustment device 310, and a server 306. The image adjusting device 310 includes an image processing unit 311, an image generation unit 312, and a communication unit 313.

The camera 302, the communication unit 303, the image display device 304, the controller 305, the image adjustment device 310, and the server 306 are the camera 202, the communication unit 203, the image display device 204, the controller 205, and the image adjustment device of the second embodiment, respectively. Corresponds to 210 and server 206. The image processing unit 311, the image generation unit 312, and the communication unit 313 correspond to the image processing unit 211, the image generation unit 212, and the communication unit 213 of the second embodiment, respectively.

The image adjusting device 310 can acquire the captured image IM3 captured by the camera 302 via the communication units 303 and 313 and the network. The server 306 connects to the camera 302 via the network and the communication unit 303, and connects to the image adjusting device 310 via the network and the communication unit 313.

The server 306 may acquire the captured image IM3 captured by the camera 302 via the communication unit 303 and the network, and the image adjusting device 310 may acquire the captured image IM3 from the server 306 via the network and the communication unit 313.

The captured image IM3 acquired by the image adjusting device 310 is input to the image processing unit 311. The image processing unit 311 recognizes the horizontal direction in the captured image IM3 by analyzing the captured image IM3. The image processing unit 311 may recognize the vertical direction in the captured image IM3, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM3. The image adjusting device 310 corrects the distortion of the captured image IM3, executes image processing such as adjusting the horizontal of the captured image IM3, and outputs the image-processed captured image IM3 to the image display device 304. The image display device 304 displays the captured image IM3 image-processed by the image adjusting device 310.

The server 306 corrects the distortion of the captured image IM3 captured by the camera 302, executes image processing such as adjusting the level of the captured image IM3, and outputs the image-processed captured image IM2 to the image adjusting device 210. You may. By attaching the image display device 204 to the head of the user US, the user US can see the captured image IM2 image-processed by the image adjusting device 210 or the server 206.

The image display device 304 generates posture information PN3 based on the direction in which the user US is facing, the posture of the user US, and the like when the image display device 304 is attached to the head of the user US. The image processing unit 311 acquires the posture information PN3 from the image display device 304. That is, the image processing unit 311 acquires the posture information PN3 based on the posture of the image display device 304. Based on the posture information PN3, the image processing unit 311 displays an image of a region corresponding to a state such as the direction in which the user US is facing and the posture of the user US from the captured image IM3 taken by the camera 302. Displayed on device 304.

The controller 305 generates instruction information NN3 based on a state such as the movement or posture of the hand of the user US in a state of being attached to the hand of the user US. The image processing unit 311 acquires the instruction information NN3 from the controller 305. The image processing unit 311 can change or adjust the image displayed on the image display device 304 based on the instruction information NN3.

The image generation unit 312 generates a spherical image VSS3 which is a virtual image composed of a spherical surface which is CG. The image generation unit 312 stores the spherical image VSS3 in the built-in memory or an external memory.

The image processing unit 311 acquires the spherical image VSS3 from the image generation unit 312 based on the instruction information NN3 and displays it on the image display device 304. The spherical image VSS3 corresponds to the spherical images VSS1 and VSS2 of the first and second embodiments. The user US moves the hand on which the controller 305 is attached to a position corresponding to the spherical image VSS3 displayed on the image display device 304, so that the hand of the user US comes into contact with the spherical image VSS3. I feel like doing it.

The controller 305 may have an actuator arranged at a portion that comes into contact with the hand of the user US. The image processing unit 311 operates the actuator when it is determined that the hand of the user US has moved to the position corresponding to the spherical image VSS3 based on the instruction information NN3. When the actuator applies pressure to the hand of the user US, the user US can actually feel the hand touching the spherical image VSS3.

When the user US moves the hand on which the controller 305 is attached in an arbitrary direction while the spherical image VSS3 is displayed on the image display device 304, the image processing unit 311 sends the instruction information NN3. Based on this, image processing is performed so that the spherical image VSS3 and the captured image IM3 displayed on the image display device 304 move according to the moving direction, moving speed, and moving destination position of the user US's hand. To execute.

The user US can rotate the spherical image VSS3 in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed to an arbitrary position. it can. That is, the user US can rotate the spherical image VSS3 by the movement of his / her hand. The image processing unit 311 moves the captured image IM3 in correspondence with the rotation of the spherical image VSS3.

The image processing unit 311 indicates to which position on the coordinates of the spherical image VSS3 the zenith ZE before the user US rotates the spherical image VSS3 moves by rotating the spherical image VSS3 by the user US. Can be determined. The image processing unit 311 is a spherical image before and after the user US rotates the spherical image VSS3 based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS3. Calculate the amount of change in VSS3. The amount of change in the spherical image VSS3 corresponds to the amount of change in the spherical images VSS1 and VSS2 of the first and second embodiments.

The image processing unit 311 outputs the amount of change in the spherical image VSS3 as the correction value CV3 to the server 306 via the communication unit 313 and the network. That is, the correction value CV3 is calculated based on the rotation direction of the spherical image VSS3 and the movement amount or movement angle of the zenith ZE (rotation angle of the spherical image VSS3). The image processing unit 311 may output the coordinates on the spherical image VSS3 of the zenith ZE after the user US rotates the spherical image VSS3 to the server 306 as the correction value CV3.

The server 306 may acquire the correction value CV3 from the image adjusting device 310 via the network and the communication unit 313. The server 306 associates the correction value CV3 with the user US or the image display device 304 and stores it in the internal memory or an external memory.

The image processing unit 311 detects the horizontal direction of the captured image IM3. However, it is difficult for the image processing unit 311 to detect the horizontal direction depending on the captured image IM3, and the horizontal direction may be erroneously detected. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the captured image IM3 is incorrectly detected, or if the horizontal or zenith designation is deviated, the captured image IM3 displayed on the image display device 304 does not match the gravitational sensation of the user US, so that the user US I feel uncomfortable.

When the user US operates the controller 305, the image adjustment system 301 performs the same processing as steps S101 to S108 of the flowchart shown in FIG. 3 or the same processing as steps S201 to S209 of the flowchart shown in FIG. Execute. Therefore, the user US can adjust the captured image IM3 displayed on the image display device 304 so as to be horizontal by operating the controller 305 to rotate the spherical image VSS3.

FIG. 8 shows a state in which the camera 302 moves in the order of point A, point B, and point C, and a wall orthogonal to the ground is arranged on the left side of the camera 302. Let the ground be the first surface PL1 and the wall surface be the second surface PL2. In the initial state, the first surface PL1 is a horizontal plane and the second surface PL2 is a vertical plane. By moving the camera 302 in the order of point A, point B, and point C, the user US can get the feeling that the camera 302 is moving in the order of point A, point B, and point C.

At point A, the captured image IM3 is adjusted horizontally. When the captured image IM3 is not adjusted horizontally, the user US operates the controller 305 to rotate the spherical image VSS3 so that the captured image IM3 displayed on the image display device 304 becomes horizontal. It will be adjusted. The image processing unit 311 acquires the normal vector NV1 on the first surface PL1. The image processing unit 311 may also acquire the normal vector NV1 at the point B or C.

The image processing unit 311 detects a surface other than the first surface PL1 from the captured image IM3. The image processing unit 311 detects, for example, the second surface PL2. The image processing unit 311 determines whether or not the detected surface is located within a predetermined range. The image processing unit 311 determines, for example, whether or not the second surface PL2 is located within a predetermined range. The predetermined range will be described later.

When it is determined that the second surface PL2 is located within a predetermined range, the image processing unit 311 estimates the normal vector NV2 on the second surface PL2. As shown in FIG. 9, when the second surface PL2 has a plurality of linear patterns extending in the horizontal direction and the vertical direction, the image processing unit 311 is perpendicular to the plurality of linear patterns extending in the horizontal direction. The normal vector NV2 may be estimated based on the distortion of the quadrangle formed by the plurality of extending linear patterns.

When the second surface PL2 has an uneven shape and a shadow is formed corresponding to the uneven shape, the image processing unit 311 may estimate the normal vector NV2 based on the shape of the shadow. If the second surface PL2 has no features and it is difficult to estimate the normal vector NV2, the normal vector is a vector parallel to the first surface PL1 and in the direction from the second surface PL2 to the camera 302. It may be set as NV2. The image processing unit 311 acquires or acquires the normal vector NV2 on the second surface PL2 at the point A. The image processing unit 311 may acquire the normal vector NV2 also at the point B or C.

When a plurality of surfaces other than the first surface PL1 are detected from the captured image IM3, the image processing unit 311 assigns a number for identifying each surface to each surface. The image processing unit 311 superimposes at least one of the CG (Computer Graphics) and the number corresponding to each surface on the captured image IM3 displayed on the image display device 304 at the position corresponding to each surface. Can be done.

An example of the image adjustment method of the third embodiment will be described with reference to the flowcharts shown in FIGS. 10A and 10B. Specifically, an example of a method of changing the horizontal plane of the captured image IM3 displayed on the image display device 304 will be described. A case where the first surface PL1 is a horizontal plane at the point A, the second surface PL2 is changed to the horizontal plane at the point B, and the first surface PL1 returns to the horizontal plane at the point C will be described.

The image display device 304 is attached to the head of the user US, and the controller 305 is attached to the hand of the user US. The captured image IM3 is displayed on the image display device 304. The camera 302 moves in the order of point A, point B, and point C. At point A, it is assumed that the captured image IM3 is adjusted horizontally.

In FIG. 10A, when the camera 302 moves from the point A to the point B, or when the camera 302 moves, the user US operates the controller 305, so that the image processing unit 311 moves the image display device 304 in step S301. Display the setting screen. When the user US operates the controller 305 and selects a predetermined item (for example, an item for changing the horizontal plane) displayed on the setting screen, the image processing unit 311 selects the item selected in step S302. The process shifts to the corresponding predetermined process mode. When the item for changing the horizontal plane is selected, the image processing unit 111 shifts the processing to the processing mode for changing the horizontal plane (horizontal plane change mode).

In step S303, the image processing unit 311 detects a surface other than the first surface PL1 from the captured image IM3 at the point B. The image processing unit 311 detects, for example, the second surface PL2. In step S304, the image processing unit 311 determines whether or not the detected surface is located within a predetermined range. The image processing unit 311 determines, for example, whether or not the second surface PL2 is located within a predetermined range.

When it is determined that the detected surface is located within a predetermined range (YES), the image processing unit 311 detects the CG corresponding to the detected surface in the captured image IM3 in step S305. It is synthesized at a position corresponding to the surface and displayed on the image display device 304. When it is determined that the plurality of faces are located within a predetermined range, the plurality of CGs corresponding to the plurality of faces are combined with the positions corresponding to the plurality of faces in the captured image IM3, and each of them is further combined. A number for identifying the surface is assigned to each surface and displayed on the image display device 304. If it is determined that the detected surface is not located within the predetermined range (NO), the image processing unit 311 does not synthesize the CG and the number with the captured image IM3 in step S306, that is, the CG and the number are combined. It is hidden and the process returns to step S303.

When the user US operates the controller 305 and, for example, a CG image or a number corresponding to the second surface PL2 is selected, the image processing unit 311 performs the normal vector on the second surface PL2 in step S307. Estimate NV2. In step S308, the image processing unit 311 processes the captured image IM3 based on the normal vector NV2 so that the second surface PL2 becomes a horizontal plane. In the state shown in FIG. 8, the image processing unit 311 rotates the captured image IM3 counterclockwise by 90 degrees. By this image processing, the user US can get the feeling of standing on the wall surface, which is the second surface PL2, and moving.

As shown in FIG. 8, in order to obtain the feeling that the user US is standing on the wall surface (second surface PL2) and moving, the distance L2 between the camera 302 and the wall surface is important. If the distance L2 is too long, the user US feels like it is floating from the wall surface, and if the distance L2 is too short, the user US feels like it is buried in the wall surface. The distance L2 is preferably in the range of 0.7 × L1 <L2 <1.5 × L1 with respect to the distance L1 from the ground (first surface PL1) to the camera 302. Therefore, the predetermined range in step S304 is, for example, the range of 0.7 × L1 <L2 <1.5 × L1. As a result, the user US can feel as if he / she is standing on the wall surface and moving.

In FIG. 10B, when the camera 302 moves from the point B to the point C, or when the camera 302 moves, the user US operates the controller 305, so that the image processing unit 311 moves the image display device 304 in step S309. Display the setting screen. When the user US operates the controller 305 and selects a predetermined item (for example, an item for changing the horizontal plane) displayed on the setting screen, the image processing unit 311 selects the item selected in step S310. The process shifts to the corresponding predetermined process mode. When the item for changing the horizontal plane is selected, the image processing unit 111 shifts the processing to the processing mode for changing the horizontal plane (horizontal plane change mode).

In step S311 the image processing unit 311 detects, for example, the first surface PL1 and the second surface PL2 from the captured image IM3 at the point C. When the user US operates the controller 305 and, for example, a CG or a number corresponding to the first surface PL1 is selected, the image processing unit 311 first operates the controller 305 in step S312 based on the normal vector NV1. The captured image IM3 is processed so that the surface PL1 is a horizontal plane. In the state shown in FIG. 8, the image processing unit 311 rotates the captured image IM3 clockwise by 90 degrees. By this image processing, the user US can obtain the feeling of standing on the ground, which is the first surface PL1, and moving.

In steps S301 and S302, and steps S309 and S310, the user US operates the controller 305 to select a predetermined surface PL, but the server 306 or the image adjusting device 310 selects an arbitrary surface PL at an arbitrary time. May be selected and the captured image IM3 may be processed in correspondence with the selected surface PL. When a plurality of user USs are viewing the captured image IM3 via the network, one user US may select any surface PL at any time with respect to the other user US. When the user US uses the image display device 304 for an attraction or the like, the attraction operator may select any surface PL at any time.

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, the image display device 304 displays the spherical image VSS3. According to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, it is used when the horizontal direction of the captured image IM3 is erroneously detected or the horizontal or zenith ZE designation is deviated. By operating the controller 305 to rotate the spherical image VSS3, the person US can adjust the captured image IM3 displayed on the image display device 304 so as to be horizontal.

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, the correction value CV3 is associated with the user US or the image display device 304 and stored in the server 306. The image adjusting device 310 can horizontally adjust the captured image IM3 captured by the camera 302 based on the correction value CV3 and display it on the image display device 304.

Therefore, according to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, when the horizontal direction of the captured image IM3 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM3.

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, a surface other than the ground is detected, it is determined whether or not the detected surface is located within a predetermined range, and a predetermined surface is determined. The normal vector NV on the plane located within the range of is estimated. According to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, the captured image IM is rotated based on the normal vector NV so that the designated surface PL becomes a horizontal plane. Can be done. When the designated surface PL is a wall surface, the user US can get the feeling of standing on the wall surface and moving.

[Fourth Embodiment]
A configuration example of the image adjustment system of the fourth embodiment will be described with reference to FIG. The image adjustment system 401 includes a camera 402, a communication unit 403, an image display device 404, a controller 405, an image adjustment device 410, and a server 406. The image adjusting device 410 includes an image processing unit 411, an image generation unit 412, and a communication unit 413.

The camera 402, communication unit 403, image display device 404, controller 405, image adjustment device 410, and server 406 are the camera 202, communication unit 203, image display device 204, controller 205, and image adjustment device of the second embodiment, respectively. It corresponds to 210 and server 206. The image processing unit 411, the image generation unit 412, and the communication unit 413 correspond to the image processing unit 211, the image generation unit 212, and the communication unit 213 of the second embodiment, respectively.

The image adjusting device 410 can acquire the captured image IM4 captured by the camera 402 via the communication units 403 and 413 and the network. The server 406 connects to the camera 402 via the network and the communication unit 403, and connects to the image adjustment device 410 via the network and the communication unit 413.

The server 406 may acquire the captured image IM4 via the communication unit 403 and the network, and the image adjusting device 410 may acquire the captured image IM4 from the server 406 via the network and the communication unit 413.

The captured image IM4 acquired by the image adjusting device 410 is input to the image processing unit 411. The image processing unit 411 recognizes the horizontal direction in the captured image IM4 by analyzing the captured image IM4. The image adjusting device 410 may recognize the vertical direction in the captured image IM4, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM4. The image adjusting device 410 corrects the distortion of the captured image IM4, executes image processing such as adjusting the horizontal of the captured image IM4, and outputs the image-processed captured image IM4 to the image display device 304. The image display device 404 displays the captured image IM4.

The server 406 acquires the captured image IM4 from the camera 402 via the network and the communication unit 403, corrects the distortion of the captured image IM4, executes image processing such as adjusting the level of the captured image IM4, and performs image processing. The captured image IM4 may be output to the image adjusting device 410. By attaching the image display device 404 to its head, the user US can see the captured image IM4 image-processed by the image adjustment device 410 or the server 406.

The image display device 404 generates posture information PN4 based on the direction in which the user US is facing, the posture of the user US, and the like when the image display device 404 is attached to the head of the user US. The image processing unit 411 acquires the posture information PN4 from the image display device 404. That is, the image processing unit 411 acquires the posture information PN4 based on the posture of the image display device 404. Based on the posture information PN4, the image processing unit 411 displays an image of a region corresponding to a state such as the direction in which the user US is facing and the posture of the user US from the captured image IM4 taken by the camera 402. Displayed on device 404.

The controller 405 generates instruction information NN4 based on a state such as the movement or posture of the hand of the user US in a state of being attached to the hand of the user US. The image processing unit 411 acquires the instruction information NN4 from the controller 405. The image processing unit 411 can change or adjust the image displayed on the image display device 404 based on the instruction information NN4.

The image generation unit 412 generates a spherical image VSS4 which is a virtual image composed of a spherical surface which is CG. The image generation unit 412 stores the spherical image VSS4 in the built-in memory or an external memory.

The image processing unit 411 acquires the spherical image VSS4 from the image generation unit 412 based on the instruction information NN4 and displays it on the image display device 404. The spherical image VSS4 corresponds to the spherical image VSS2 of the second embodiment. The user US moves the hand on which the controller 405 is attached to a position corresponding to the spherical image VSS4 displayed on the image display device 404, so that the hand of the user US comes into contact with the spherical image VSS4. I feel like doing it.

The controller 405 may have an actuator arranged at a portion that comes into contact with the hand of the user US. The image processing unit 411 operates the actuator when it is determined that the hand of the user US has moved to the position corresponding to the spherical image VSS4 based on the instruction information NN4. When the actuator applies pressure to the hand of the user US, the user US can actually feel the hand touching the spherical image VSS4.

When the user US moves the hand on which the controller 405 is attached in an arbitrary direction while the spherical image VSS4 is displayed on the image display device 404, the image processing unit 411 sends the instruction information NN4. Based on this, image processing is performed so that the spherical image VSS4 and the captured image IM4 displayed on the image display device 404 move according to the moving direction, moving speed, and moving destination position of the hand of the user US. To execute.

The user US can rotate the spherical image VSS4 in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed to an arbitrary position. it can. That is, the user US can rotate the spherical image VSS4 by the movement of his / her hand. The image processing unit 411 moves the captured image IM4 in correspondence with the rotation of the spherical image VSS4.

When the horizon or a building is reflected in the captured image IM4 captured by the camera 402, the image adjusting device 410 and the server 406 can accurately recognize the horizontal direction of the captured image IM4. However, when the captured image IM4 captured by the camera 402 is an image composed of only curves such as a natural landscape, the image adjusting device 410 and the server 406 cannot accurately recognize the horizontal direction of the captured image IM4. There is.

When the horizontal direction of the captured image IM4 cannot be recognized accurately, the captured image IM4 is displayed on the image display device 404 in an inclined state. The user US can adjust the captured image IM4 to be horizontal by rotating the spherical image VSS4.

An example of the image adjustment method of the fourth embodiment will be described with reference to the flowchart shown in FIG. Specifically, an example of a method of adjusting the horizontal of the captured image IM4 captured by the camera 402 will be described.

The server 406 executes the same process as in step S201 shown in FIG. As shown in FIG. 12A, when the user US determines that the captured image IM4 displayed on the image display device 404 is not horizontal and the user US operates the controller 205, the image adjustment system 401 is shown in FIG. The same process as in steps S202 to S207 shown in 6 is executed. As a result, as shown in FIG. 12B, the captured image IM4 displayed on the image display device 404 can be adjusted horizontally.

In the image processing unit 411, the position of the zenith ZE before the user US rotates the spherical image VSS4 on the coordinates of the spherical image VSS4 is moved by the user US rotating the spherical image VSS4. Can be determined. The image processing unit 411 is a spherical image before and after the user US rotates the spherical image VSS4 based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS4. Calculate the amount of change in VSS4. The amount of change in the spherical image VSS4 corresponds to the amount of change in the spherical image VSS2 of the second embodiment.

The image processing unit 411 calculates the rotation angle RA as the amount of change in the spherical image VSS4. The rotation angle RA is the rotation angle of the spherical image VSS4 in the state after the spherical image VSS4 is rotated with respect to the state before the user US rotates the spherical image VSS4. The rotation angle RA may be a rotation angle with the X-axis, Y-axis, or Z-axis as the rotation axis, or may be a rotation angle synthesized with the X-axis, Y-axis, and Z-axis as the rotation axis. ..

The image adjustment system 401 executes the same processing as in steps S208 and S209 shown in FIG. The image processing unit 411 outputs the amount of change in the spherical image VSS4 as the correction value CV4 to the server 406 via the communication unit 413 and the network. The correction value CV4 includes information on the rotation angle RA. The server 406 may acquire the correction value CV4 from the image adjusting device 410 via the network and the communication unit 413.

In FIG. 11, the server 406 acquires the rotation angle RA in step S401. In step S402, the server 406 determines whether or not the rotation angle RA is equal to or greater than a predetermined angle. That is, whether or not the server 406 horizontally adjusts the captured image IM4 displayed on the image display device 404 to a predetermined angle or more (for example, 10 degrees or more) by rotating the spherical image VSS4. Will be determined.

If it is determined that the rotation angle RA is equal to or greater than a predetermined angle (YES), the server 406 fails in the horizontal adjustment by the image adjusting device 410 or the server 406 with respect to the captured image IM4 taken by the camera 402 in step S403. It is determined that there is, and the captured image IM4b before the horizontal adjustment by the user US and the captured image IM4a after the horizontal adjustment are associated and saved. When it is determined that the rotation angle RA is not equal to or greater than a predetermined angle (NO), the image adjustment system 401 returns the process to step S401.

In step S404, the server 406 determines whether or not the number of stored captured images IM4b and IM4a is equal to or greater than a predetermined value. If it is determined that the number of the stored captured images IM4b and IM4a is equal to or greater than a predetermined value (YES), the server 406 refers to the plurality of captured images IM4b and IM4a stored in step S405. Execute learning processing such as deep learning. The server 406 executes the learning process using, for example, the captured image IM4b as input data and the captured image IM4a as correct answer data. When it is determined that the number of the stored captured images IM4b and IM4a is not equal to or greater than a predetermined value (NO), the image adjustment system 401 returns the process to step S401.

In step S406, the server 406 horizontally adjusts the captured image IM4 captured by the camera 402 based on the learning result in step S405. In step S406, the server 406 may output the learning result to the image adjusting device 410, and the image processing unit 411 may horizontally adjust the captured image IM4 captured by the camera 402 based on the learning result. The image processing unit 411 may execute the processing of steps S401 to S406.

By repeatedly executing the processes of steps S401 to S406 by the server 406 or the image processing unit 411, the horizontal adjustment accuracy of the captured image IM4 displayed on the image display device 404 can be improved.

In the image adjustment system 401, the image adjustment device 410, and the image adjustment method of the fourth embodiment, the spherical image VSS4 is displayed on the image display device 404. According to the image adjustment system 401, the image adjustment device 410, and the image adjustment method of the fourth embodiment, it is used when the horizontal direction of the captured image IM4 is incorrectly detected or the horizontal or zenith ZE designation is deviated. The person US can adjust the captured image IM4 displayed on the image display device 404 so as to be horizontal by operating the controller 405 to rotate the spherical image VSS4.

In the image adjustment system 401, the image adjustment device 410, and the image adjustment method of the fourth embodiment, the correction value CV4 is stored in the server 406 or its external memory in association with the user US or the image display device 404. The image adjusting device 410 can adjust the captured image IM4 captured by the camera 402 based on the correction value CV4 and display it on the image display device 404.

Therefore, according to the image adjustment system 401, the image adjustment device 410, and the image adjustment method of the fourth embodiment, when the horizontal direction of the captured image IM4 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM4.

In the image adjustment system 401, the image adjustment device 410, and the image adjustment method of the fourth embodiment, when the rotation angle RA of the spherical image VSS4 is equal to or larger than a predetermined angle, it is determined that the horizontal adjustment has failed, and the determination result is obtained. The learning process is executed for the captured images IM4b and IM4a for which the horizontal adjustment has failed based on. According to the image adjustment system 401, the image adjustment device 410, and the image adjustment method of the fourth embodiment, the horizontal adjustment accuracy of the captured image IM4 displayed on the image display device 404 is improved by executing the learning process. be able to.

The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

For example, the virtual image VSS generated by CG may be an ellipsoidal surface (elliptical surface), and may be an arbitrary closed surface (closed curved surface) within the reach of the user US. All you need is. That is, it suffices if the photographed image IM can be horizontally adjusted with the feeling that the user US touches the closed curved surface from the inside. Since the user US rotates the virtual image VSS, it is desirable that the virtual image VSS has a spherical shape close to a spherical surface such as a spherical surface or an ellipsoidal surface.

101, 201, 301, 401 Image adjustment system 102, 202, 302, 402 Camera 104, 204, 304, 404 Image display device 105, 205, 305, 405 Controller 110, 210, 310, 410 Image adjustment device 111, 211, 311,411 Image processing unit 112,212,212,412 Image generation unit 206,306,406 Server IM1, IM2, IM3, IM4 Captured image NN1, NN2, NN3, NN4 Instruction information PL1 Ground (first surface)
PL2 wall surface (second surface)
RA rotation angle US user VSS1, VSS2, VSS3, VSS4 Spherical image (virtual image)

Claims (8)

With the camera
An image adjustment device that adjusts the captured image taken by the camera, and
An image display device that displays the captured image adjusted by the image adjustment device, and
A controller that outputs instruction information to the image adjustment device,
A server that identifies the image display device or the user of the image display device,
With
The image adjusting device is
An image generator that generates a spherical image and
The spherical image is acquired from the image generation unit based on the instruction information and displayed on the image display device, and the spherical image is rotated based on the instruction information to correspond to the rotation of the spherical image. An image processing unit that adjusts the captured image displayed on the image display device, and
Image adjustment system with. The camera is an omnidirectional camera that captures a 360-degree range.
The image display device is a head-mounted display mounted on the user's head.
The controller is a glove-type controller that is worn on the user's hand.
When the user sees the spherical image displayed on the image display device while the image display device is attached to the head of the user, the spherical image is used. It is a virtual image that is arranged around a person and the image display device and is set so that the user's hand or finger is displayed within a range that can reach the spherical image.
The image adjustment system according to claim 1. When the user moves his / her hand or finger in an arbitrary direction while the controller is attached to the user's hand, the image processing unit uses the instruction information of the user. The spherical image is rotated in response to the movement of the hand or finger, and the captured image displayed on the image display device is adjusted in response to the rotation of the spherical image.
The image adjustment system according to claim 2. The image processing unit acquires the amount of change between before and after the spherical image is rotated, and adjusts the captured image based on the amount of change.
The image adjustment system according to claim 3. The server stores the amount of change in association with the image display device or the user specified by the server.
The image adjustment system according to claim 4. The image processing unit adjusts the horizontality of the captured image displayed on the image display device in correspondence with the rotation of the spherical image.
The image adjustment system according to any one of claims 1 to 5. An image generator that generates a spherical image and
The spherical image is acquired from the image generation unit based on the instruction information acquired from the controller and displayed on the image display device specified by the server, and the spherical image is rotated based on the instruction information by the camera. An image processing unit that adjusts the captured image that has been captured and displayed on the image display device in accordance with the rotation of the spherical image, and
An image adjustment device comprising. The server identifies the image display device
The image processing unit acquires the instruction information from the controller and
The image processing unit acquires a spherical image from the image generation unit based on the instruction information, and obtains a spherical image.
The image display device specified by the server displays the spherical image, and the spherical image is displayed.
The image processing unit rotates the spherical image based on the instruction information, and the spherical image is rotated.
The image processing unit adjusts the captured image captured by the camera and displayed on the image display device in accordance with the rotation of the spherical image.
Image adjustment method.

Images