JP2020134632A - Image adjustment system, image adjustment device, and image adjustment method - Google Patents

Abstract

To provide an image adjustment system capable of easily correcting the horizontality or zenith of an image.SOLUTION: An image adjustment system 401 includes a camera 402, an image adjustment device 410, an image display device 404, and a controller 405. The image adjustment device 410 includes an image generation part 412 and an image processing part 411. The image processing part 411 acquires a spherical image VSS4 from the image generation part 412 by the instruction information NN4 of the controller 405, displays it on the image display device 404, and adjusts a photographed image IM41 displayed on the image display device 404 according to the rotation of the spherical image VSS4, to adjust a synthetic image CM4b according to the adjusted photographed image IM41 and synthesize the adjusted synthetic image CM4b in the position of the second camera 4022 of the photographed image IM41 by the positional information of first and second cameras 4021 and 4022, when the second camera 4022 among the plurality of cameras 402 is included in the photographed image IM41.SELECTED DRAWING: Figure 14

Description

The present invention relates to an image adjustment system, an image adjustment device, and an image adjustment method.

In recent years, a head-mounted display has attracted attention as an image display device. By displaying an image while the head-mounted display is attached to the user's head, the user can obtain a feeling of entering a virtual space (immersive feeling). As described in Patent Document 1, the head-mounted display can display an image taken by an external camera via a neckwork.

Japanese Unexamined Patent Publication No. 2005-56295

However, with a conventional head-mounted display, it is difficult to detect the horizontal direction of an image depending on the displayed image, and the horizontal direction may be detected incorrectly. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the image is detected incorrectly, or if the horizontal or zenith designation is incorrect, the displayed image does not match the user's sense of gravity, and the user feels uncomfortable.

The present invention provides an image adjustment system, an image adjustment device, and an image that can easily correct the horizontal or zenith of an image when the horizontal direction of the image is incorrectly detected or the horizontal or zenith is not specified. The purpose is to provide an adjustment method.

The present invention includes a plurality of cameras, an image adjusting device that adjusts a captured image captured by the first camera among the plurality of cameras, and an image display device that displays the captured image adjusted by the image adjusting device. The image adjusting device includes a controller that outputs instruction information to the image adjusting device, and the image adjusting device has an image generation unit that generates a spherical image and a composite image that is combined with the captured image, and the instruction information. Based on this, the spherical image generated by the image generation unit is acquired and displayed on the image display device, and the spherical image is rotated based on the instruction information so as to correspond to the rotation of the spherical image. The captured image displayed on the image display device is adjusted, the composite image is adjusted in correspondence with the adjusted captured image, and the second camera of the plurality of cameras is reflected in the captured image. If so, the composite image adjusted based on the position information of the first camera and the position information of the second camera is placed at the position where the second camera of the captured image is projected. Provided is an image adjustment system having an image processing unit for associating and synthesizing.

The present invention is based on an image generator that generates a spherical image and a composite image that is combined with a captured image captured by a camera and displayed on an image display device, and instruction information acquired from a controller. The spherical image generated by the generation unit is acquired and displayed on an image display device, the spherical image is rotated based on the instruction information, and the spherical image is photographed by the first camera among the plurality of cameras. The captured image displayed on the image display device is adjusted in correspondence with the rotation of the spherical image, the composite image is adjusted in correspondence with the adjusted captured image, and the captured image is combined with the plurality of cameras. When the second camera is shown, the composite image adjusted based on the position information of the first camera and the position information of the second camera is used as the second image of the captured image. Provided is an image adjusting device including an image processing unit that synthesizes images corresponding to the position in which the camera of 1 is projected.

In the present invention, the image processing unit acquires instruction information from the controller, the image processing unit acquires a spherical image generated by the image generation unit based on the instruction information, and the image display device acquires the spherical image. Is displayed, the image processing unit rotates the spherical image based on the instruction information, and the image processing unit is photographed by the first camera among the plurality of cameras and displayed on the image display device. The captured image is adjusted in correspondence with the rotation of the spherical image, and the composite image generated by the image processing unit and combined with the captured image displayed on the image display device is generated. When the image processing unit adjusts the composite image in correspondence with the adjusted captured image and the second camera among the plurality of cameras is reflected in the captured image, the above-mentioned The image processing unit corresponds the adjusted composite image based on the position information of the first camera and the position information of the second camera to the position where the second camera is reflected in the captured image. To provide an image adjustment method for compositing.

According to the image adjustment system, the image adjustment device, and the image adjustment method of the present invention, the horizontal or zenith of the image can be easily adjusted when the horizontal direction of the image is incorrectly detected or the horizontal or zenith is not specified. It can be corrected.

It is a block diagram which shows the image adjustment system of 1st and 2nd Embodiment. It is a figure which shows the relationship between a spherical image and a user. It is a flowchart which shows an example of the image adjustment method of 1st Embodiment. It is a flowchart which shows an example of the image adjustment method of 1st Embodiment. It is a figure which shows the relationship between the horizontal plane of a photographed image, and the horizontal plane of a composite image. It is a flowchart which shows an example of the image adjustment method of 2nd Embodiment. It is a flowchart which shows an example of the image adjustment method of 2nd Embodiment. It is a flowchart which shows an example of the image adjustment method of 2nd Embodiment. It is a figure which shows the relationship between the horizontal plane of a photographed image, and the horizontal plane of a composite image. It is a block diagram which shows the image adjustment system of 3rd Embodiment. It is a figure which shows an example of the positional relationship between a user and a composite image corresponding to another user when the user is looking at an image display device. It is a figure which shows an example of the positional relationship between a user and a composite image corresponding to another user when the user is looking at an image display device. It is a flowchart which shows an example of the image adjustment method of 3rd Embodiment. It is a flowchart which shows an example of the image adjustment method of 3rd Embodiment. It is a block diagram which shows the image adjustment system of 4th Embodiment. It is a flowchart which shows an example of the image adjustment method of 3rd Embodiment. It is a flowchart which shows an example of the image adjustment method of 3rd Embodiment. It is a figure which shows an example of the positional relationship between a plurality of cameras and a composite image corresponding to another user.

[First Embodiment]
A configuration example of the image adjustment system of the first embodiment will be described with reference to FIG. The image adjustment system 101 includes a camera 102, a communication unit 103, an image display device 104, a controller 105, an image adjustment device 110, and a server 106. The image adjusting device 110 includes an image processing unit 111, an image generation unit 112, and a communication unit 113.

The camera 102 is an omnidirectional camera (360 degree camera) capable of photographing a range of 360 degrees. The communication unit 103 and the communication unit 113 are connected via a network. The image adjusting device 110 can acquire the captured image IM1 captured by the camera 102 via the communication units 103 and 113 and the network.

A computer device may be used as the image adjusting device 110. A CPU may be used as the image processing unit 111 and the image generation unit 112. The image generation unit 112 and the communication unit 113 may be provided outside the image adjusting device 110.

The server 106 connects to the camera 102 via the network and the communication unit 103, and connects to the image adjusting device 110 via the network and the communication unit 113. The server 106 may acquire the captured image IM1 captured by the camera 102 via the communication unit 103 and the network, and the image adjusting device 110 may acquire the captured image IM1 from the server 106 via the network and the communication unit 113.

The captured image IM1 acquired by the image adjusting device 110 is input to the image processing unit 111. The image processing unit 111 recognizes the horizontal direction in the captured image IM1 by analyzing the captured image IM1. The image processing unit 111 may recognize the vertical direction in the captured image IM1 or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM1. The image adjusting device 110 corrects the distortion of the captured image IM1, executes image processing such as adjusting the horizontal of the captured image IM1, and outputs the image-processed captured image IM1 to the image display device 104.

The server 106 corrects the distortion of the captured image IM1 captured by the camera 102, executes image processing such as adjusting the level of the captured image IM1, and outputs the image-processed captured image IM1 to the image adjusting device 110. You may. The image display device 104 displays the captured image IM1 image-processed by the image adjustment device 110 or the server 106.

The image display device 104 is, for example, a head-mounted display. The controller 105 is a globe-type controller used for, for example, VR (Virtual Reality).

FIG. 2 schematically shows a state in which the image display device 104 is attached to the head of the user US and the controller 105 is attached to the hand of the user US. The symbol ZE shown in FIG. 2 indicates the zenith. It is desirable that the zenith of the camera 102 and the zenith of the user US coincide with each other. By attaching the image display device 104 to its head, the user US can see the captured image IM1 image-processed by the image adjustment device 110 or the server 106.

The image display device 104 generates posture information PN1 based on the direction in which the user US is facing, the posture of the user US, and the like when the image display device 104 is attached to the head of the user US. The image processing unit 111 acquires the posture information PN1 from the image display device 104. That is, the image processing unit 111 acquires the posture information PN1 based on the posture of the image display device 104. Based on the posture information PN1, the image processing unit 111 displays an image of a region corresponding to a state such as the direction in which the user US is facing and the posture of the user US from the captured image IM1 taken by the camera 102. Displayed on the device 104.

The controller 105 generates instruction information NN1 based on the movement or posture of the hand of the user US or the movement or posture of the finger of the user US when the controller 105 is attached to the hand of the user US. .. Hereinafter, the hand or finger is simply abbreviated as a hand. The image processing unit 111 acquires the instruction information NN1 from the controller 105. The image processing unit 111 can change or adjust the captured image IM1 displayed on the image display device 104 based on the instruction information NN1.

The image generation unit 112 generates a composite image CM1 which is CG (Computer Graphics) combined with the captured image IM1. The composite image CM1 is an image of a character such as an avatar. The image generation unit 112 stores the composite image CM1 in the internal memory or an external memory. The image processing unit 111 may acquire the composite image CM1 generated by the image generation unit 112 and output it to the server 106 via the communication unit 113 and the network.

The server 106 stores the composite image CM1 in the built-in memory or an external memory in association with the image display device 104 or the user US. The image processing unit 111 acquires the composite image CM1 from the image generation unit 112 or the server 106 based on the instruction information NN1, and synthesizes the composite image CM1 with the captured image IM1 displayed on the image display device 104. Therefore, the image display device 104 can display the captured image IM1 captured by the camera 102 and the composite image CM1 acquired from the image generation unit 112 or the server 106 and combined with the captured image IM1.

The image generation unit 112 generates a spherical image VSS1 which is a virtual image composed of a spherical surface which is CG. The image generation unit 112 stores the spherical image VSS1 in the built-in memory or an external memory. The image processing unit 111 acquires the spherical image VSS1 generated by the image generation unit 112 based on the instruction information NN1 and displays it on the image display device 104.

FIG. 2 shows an image of the user US when the user US sees the spherical image VSS1 displayed on the image display device 104 in a state where the image display device 104 is attached to the head of the user US. It is shown schematically.

When the user US sees the spherical image VSS1 displayed on the image display device 104 in a state where the image display device 104 is attached to the head of the user US, the spherical image VSS1 is the user. It is arranged around the US and the image display device 104, and is set so that the hand of the user US is displayed within the reach of the spherical image VSS1. The user US moves the hand on which the controller 105 is attached to a position corresponding to the spherical image VSS1 displayed on the image display device 104, so that the hand of the user US comes into contact with the spherical image VSS1. I feel like I'm doing it.

The controller 105 may have an actuator arranged at a portion that comes into contact with the hand of the user US. The image processing unit 111 operates the actuator when it is determined that the hand of the user US has moved to the position corresponding to the spherical image VSS1 based on the instruction information NN1. When the actuator applies pressure to the hand of the user US, the user US can actually feel the hand touching the spherical image VSS1.

When the user US moves the hand on which the controller 105 is attached in an arbitrary direction while the spherical image VSS1 is displayed on the image display device 104, the image processing unit 111 displays the instruction information NN1. Based on this, image processing is performed so that the spherical image VSS1 and the captured image IM1 displayed on the image display device 104 move according to the moving direction, moving speed, and moving destination position of the hand of the user US. To execute.

The user US can rotate the spherical image VSS1 in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed to an arbitrary position. it can. That is, the user US can rotate the spherical image VSS1 by the movement of his / her hand. The image processing unit 111 moves the captured image IM1 in correspondence with the rotation of the spherical image VSS1.

The image processing unit 111 indicates to which position on the coordinates of the spherical image VSS1 the zenith ZE before the user US rotates the spherical image VSS1 moves by rotating the spherical image VSS1 by the user US. Can be determined. The image processing unit 111 is a spherical image before and after the user US rotates the spherical image VSS1 based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS1. The amount of change in VSS1 is calculated.

The amount of change in the spherical image VSS1 is the amount of rotation (rotation angle) with the X-axis as the rotation axis, the amount of rotation (rotation angle) with the Y-axis as the rotation axis, and the rotation of the Z-axis in the spherical image VSS1. The amount of rotation (rotation angle) as the axis corresponds to the amount of rotation (rotation angle) of the spherical image VSS1 synthesized. The image processing unit 111 stores the amount of change in the spherical image VSS1 as the correction value CV1. That is, the correction value CV1 is calculated based on the rotation direction of the spherical image VSS1 and the movement amount or movement angle of the zenith ZE (rotation angle of the spherical image VSS1).

The image processing unit 111 may save the coordinates on the spherical image VSS1 of the zenith ZE after the user US rotates the spherical image VSS1 as the correction value CV1. The image processing unit 111 may store the correction value CV1 in the built-in memory or an external memory, or may output the correction value CV1 to the server 106 via the communication unit 113 and the network. The server 106 associates the correction value CV1 with the image display device 104 or the user US and stores it in the internal memory or an external memory.

The image processing unit 111 detects the horizontal direction of the captured image IM1. However, it is difficult for the image processing unit 111 to detect the horizontal direction depending on the captured image IM1, and the horizontal direction may be erroneously detected. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the captured image IM1 is incorrectly detected, or if the horizontal or zenith designation is deviated, the captured image IM1 displayed on the image display device 104 does not match the gravitational sensation of the user US, so that the user US I feel uncomfortable.

An example of the image adjustment method of the first embodiment will be described with reference to the flowcharts shown in FIGS. 3 and 4. Specifically, an example of a method of adjusting the horizontal of the captured image IM1 and the composite image CM1 will be described. The image display device 104 is attached to the head of the user US, and the controller 105 is attached to the hand of the user US. The captured image IM1 and the composite image CM1 are displayed on the image display device 104. At this point, the composite image CM1 may not be displayed.

When the user US determines that the captured image IM1 displayed on the image display device 104 is not horizontal, in FIG. 3, when the user US operates the controller 105, the image processing unit 111 moves to step S101. , The image display device 104 is made to display the setting screen.

When the user US operates the controller 105 to select a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 111 selects the item selected in step S102. The process is shifted to the corresponding predetermined process mode. When the horizontal adjustment item is selected, the image processing unit 111 shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM1.

In step S103, the image processing unit 111 acquires the spherical image VSS1 generated by the image generation unit 112 and displays it on the image display device 104. The captured image IM1, the spherical image VSS1, and the composite image CM1 are mixed and displayed on the image display device 104. When the user US rotates the spherical image VSS1 so that the captured image IM1 is horizontal, the image processing unit 111 causes the image display device 104 to correspond to the rotation of the spherical image VSS1 in step S104. The displayed captured image IM1 is moved. The user US may perform the operation of rotating the spherical image VSS1 a plurality of times until the captured image IM1 becomes horizontal.

When the user US determines that the captured image IM1 is horizontal, the image processing unit 111 ends the display of the spherical image VSS1 in step S105 by operating the controller 105, and causes the image display device 104 to display the spherical image VSS1. Display the setting screen. By leveling the captured image IM1 displayed on the image display device 104, the zenith of the camera 102 and the zenith of the user US can be matched.

When the user US operates the controller 105 and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 111 corresponds to the selected item in step S106. The processing is shifted to the predetermined processing mode. When the end item is selected, the image processing unit 111 shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S107, the image processing unit 111 acquires the amount of rotation (rotation angle) before and after the spherical image VSS1 rotates as the amount of change in the spherical image VSS1. In step S108, the image processing unit 111 saves the amount of change in the spherical image VSS1 as the correction value CV1 and ends the processing.

When the composite image CM1 is not displayed in steps S101 to S108 of FIG. 3, when the user US operates the controller 105, the image adjusting device 110 (specifically, the image processing unit 111) causes the image generation unit 112. The composite image CM1 generated in the above can be acquired, combined with the captured image IM1 displayed on the image display device 104, and displayed.

FIG. 5 schematically shows the relationship between the horizontal plane PHP1 of the captured image IM1 horizontally adjusted by the user US and the horizontal plane SHP1 of the composite image CM1 acquired by the image processing unit 111 from the image generation unit 112. If the horizontal plane PHP1 of the captured image IM1 and the horizontal plane SHP1 of the composite image CM1 are out of alignment, the user US feels uncomfortable.

In FIG. 4, the image processing unit 111 determines in step S111 whether or not the captured image IM1 has been horizontally adjusted. The image processing unit 111 can determine whether or not the captured image IM1 has been horizontally adjusted based on the history of rotation of the spherical image VSS1 or the correction value CV1 or the like.

When it is determined in step S111 that the captured image IM1 is horizontally adjusted (YES), the image processing unit 111 acquires the stored correction value CV1 in step S112. In step S113, the image processing unit 111 rotates the coordinate axes of the composite image CM1 based on the correction value CV1 to match the horizontal plane SHP1 of the composite image CM1 with the horizontal plane PHP1 of the captured image IM1. In step S114, the image processing unit 111 performs arithmetic processing on the composite image CM1 based on the corrected horizontal plane SHP1 to adjust the image.

In step S115, the image processing unit 111 combines the image-adjusted composite image CM1 with the captured image IM1 and displays it on the image display device 104. When it is determined in step S111 that the captured image IM1 is not horizontally adjusted (NO), the image processing unit 111 shifts the processing to step S115.

In the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, the spherical image VSS1 is displayed on the image display device 104. According to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, it is used when the horizontal direction of the captured image IM1 is erroneously detected or the horizontal or zenith ZE designation is deviated. The person US can adjust the captured image IM1 displayed on the image display device 104 so as to be horizontal by operating the controller 105 to rotate the spherical image VSS1.

Therefore, according to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, when the horizontal direction of the captured image IM1 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM1.

According to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, when the correction value CV1 is stored, the image processing unit 111 reads out the correction value CV1 and the camera 102 takes a picture. The captured image IM1 can be image-adjusted based on the correction value CV1 and displayed on the image display device 104.

In the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, when the captured image IM1 is horizontally adjusted, the coordinate axes of the composite image CM1 are rotated based on the correction value CV1 to rotate the composite image CM1. The horizontal plane SHP1 is made to match the horizontal plane PHP1 of the captured image IM1. According to the image adjustment system 101, the image adjustment device 110, and the image adjustment method of the first embodiment, the captured image IM1 and the composite image CM1 are displayed on the image display device 104 without the user US feeling uncomfortable. be able to.

[Second Embodiment]
A configuration example of the image adjustment system of the second embodiment will be described with reference to FIG. The image adjustment system 201 includes a camera 202, a communication unit 203, an image display device 204, a controller 205, an image adjustment device 210, and a server 206. The image adjusting device 210 includes an image processing unit 211, an image generation unit 212, and a communication unit 213.

The camera 202, the communication unit 203, the image display device 204, the controller 205, the image adjustment device 210, and the server 206 are the camera 102, the communication unit 103, the image display device 104, the controller 105, and the image adjustment device 110 of the first embodiment. And the server 106. The image processing unit 211, the image generation unit 212, and the communication unit 213 correspond to the image processing unit 111, the image generation unit 112, and the communication unit 113 of the first embodiment.

The image adjusting device 210 can acquire the captured image IM2 captured by the camera 202 via the communication units 203 and 213 and the network. The server 206 may acquire the captured image IM2 captured by the camera 202 via the communication unit 203 and the network, and the image adjusting device 210 may acquire the captured image IM2 from the server 206 via the network and the communication unit 213.

The captured image IM2 acquired by the image adjusting device 210 is input to the image processing unit 211. The image processing unit 211 recognizes the horizontal direction in the captured image IM2 by analyzing the captured image IM2. The image processing unit 211 may recognize the vertical direction in the captured image IM2, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM2. The image adjusting device 210 corrects the distortion of the captured image IM2, executes image processing such as adjusting the horizontal of the captured image IM2, and outputs the image-processed captured image IM2 to the image display device 204.

The server 206 corrects the distortion of the captured image IM2 captured by the camera 202, executes image processing such as adjusting the level of the captured image IM2, and outputs the image-processed captured image IM2 to the image adjusting device 210. You may. The image display device 204 displays the captured image IM2 image-processed by the image adjustment device 210 or the server 206. By attaching the image display device 204 to the head of the user US, the user US can see the captured image IM2 image-processed by the image adjusting device 210 or the server 206.

The image display device 204 generates posture information PN2 based on the direction in which the user US is facing, the posture of the user US, and the like when the image display device 204 is attached to the head of the user US. The image processing unit 211 acquires the posture information PN2 from the image display device 204. That is, the image processing unit 211 acquires the posture information PN2 based on the posture of the image display device 204. Based on the posture information PN2, the image processing unit 211 displays an image of a region corresponding to a state such as the direction in which the user US is facing and the posture of the user US from the captured image IM2 taken by the camera 202. Displayed on device 204.

The controller 205 generates instruction information NN2 based on a state such as the movement or posture of the user US's hand while it is attached to the user US's hand. The image processing unit 211 acquires the instruction information NN2 from the controller 205. The image processing unit 211 can change or adjust the captured image IM2 displayed on the image display device 204 based on the instruction information NN2.

The image generation unit 212 generates a composite image CM2 which is a CG to be combined with the captured image IM2. The composite image CM2 is an image of a character such as an avatar. The image generation unit 212 stores the composite image CM2 in the internal memory or an external memory. The image processing unit 211 may acquire the composite image CM2 generated by the image generation unit 212 and output it to the server 206 via the communication unit 213 and the network. The server 206 stores the composite image CM2 in the built-in memory or an external memory in association with the image display device 204 or the user US.

The image processing unit 211 acquires the composite image CM2 from the image generation unit 212 or the server 206 based on the instruction information NN2, and synthesizes the composite image CM2 with the captured image IM2 displayed on the image display device 204. Therefore, the image display device 204 can display the captured image IM2 captured by the camera 202 and the composite image CM2 acquired from the image generation unit 212 or the server 206 and combined with the captured image IM2.

The image generation unit 212 generates a spherical image VSS2 which is a virtual image composed of a spherical surface which is CG. The image generation unit 212 stores the spherical image VSS2 in the built-in memory or an external memory. The image processing unit 211 acquires the spherical image VSS2 generated by the image generation unit 212 based on the instruction information NN2 and displays it on the image display device 204.

The spherical image VSS2 corresponds to the spherical image VSS1 of the first embodiment. The user US moves the hand on which the controller 205 is attached to a position corresponding to the spherical image VSS2 displayed on the image display device 204, so that the hand of the user US comes into contact with the spherical image VSS2. I feel like I'm doing it.

In the controller 205, the actuator may be arranged at a portion that comes into contact with the hand of the user US. The image processing unit 211 operates the actuator when it is determined that the hand of the user US has moved to the position corresponding to the spherical image VSS2 based on the instruction information NN2. When the actuator applies pressure to the hand of the user US, the user US can actually feel the hand touching the spherical image VSS2.

When the user US moves the hand on which the controller 205 is mounted in an arbitrary direction while the spherical image VSS2 is displayed on the image display device 204, the image processing unit 211 displays the instruction information NN2. Based on this, image processing is performed so that the spherical image VSS2 and the captured image IM2 displayed on the image display device 204 move according to the moving direction, moving speed, and moving destination position of the hand of the user US. To execute.

The user US can rotate the spherical image VSS2 in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed to an arbitrary position. it can. That is, the user US can rotate the spherical image VSS2 by the movement of his / her hand. The image processing unit 211 moves the captured image IM2 in correspondence with the rotation of the spherical image VSS2.

The image processing unit 211 indicates to which position on the coordinates of the spherical image VSS2 the zenith ZE before the user US rotates the spherical image VSS2 has moved by rotating the spherical image VSS2 by the user US. Can be determined. The image processing unit 211 is a spherical image before and after the user US rotates the spherical image VSS2 based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS2. Calculate the amount of change in VSS2. The amount of change in the spherical image VSS2 corresponds to the amount of change in the spherical image VSS1 of the first embodiment.

The image processing unit 211 saves the amount of change in the spherical image VSS2 as the correction value CV2. The correction value CV2 corresponds to the correction value CV1 of the first embodiment. The image processing unit 211 may save the coordinates on the spherical image VSS2 of the zenith ZE after the user US rotates the spherical image VSS2 as the correction value CV2. The image processing unit 211 may store the correction value CV2 in the built-in memory or an external memory, or may output the correction value CV2 to the server 206 via the communication unit 213 and the network. The server 206 associates the correction value CV2 with the image display device 204 or the user US and stores it in the internal memory or an external memory.

The image processing unit 211 detects the horizontal direction of the captured image IM2. However, it is difficult for the image processing unit 211 to detect the horizontal direction depending on the captured image IM2, and the horizontal direction may be erroneously detected. In addition, the horizontal or zenith designation may shift due to problems such as system synchronization. If the horizontal direction of the captured image IM2 is incorrectly detected, or if the horizontal or zenith designation is deviated, the captured image IM2 displayed on the image display device 204 does not match the gravitational sensation of the user US, so that the user US I feel uncomfortable.

An example of the image adjustment method of the second embodiment will be described with reference to the flowcharts shown in FIGS. 6A, 6B, and 7. Specifically, an example of a method of adjusting the horizontal of the captured image IM2 and the composite image CM2 will be described. The image display device 204 is attached to the head of the user US, and the controller 205 is attached to the hand of the user US. The captured image IM2 and the composite image CM2 are displayed on the image display device 204. The composite image CM2 may not be displayed.

When the user US determines that the captured image IM2 displayed on the image display device 204 is not horizontal, in FIG. 6A, when the user US operates the controller 205, the image processing unit 211 moves to step S201. , Display the setting screen on the image display device 204.

When the user US operates the controller 205 and selects a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 211 selects the selected item in step S202. The process is shifted to the corresponding predetermined process mode. When the horizontal adjustment item is selected, the image processing unit 211 shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM2.

In step S203, the image processing unit 211 acquires the spherical image VSS2 generated by the image generation unit 212 and displays it on the image display device 204. The captured image IM2 and the spherical image VSS2 are mixed and displayed on the image display device 204. In step S204, the image processing unit 211 determines whether or not the composite image CM2 is combined with the captured image IM2 and displayed on the image display device 204.

When it is determined in step S204 that the composite image CM2 is combined with the captured image IM2 and displayed on the image display device 204 (YES), the image processing unit 211 and the horizontal plane SHP2 of the composite image CM2 in step S205. A process of separating the captured image IM2 from the horizontal plane PHP2 is executed. FIG. 8 schematically shows a state in which the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 are separated.

If it is determined in step S204 that the composite image CM2 is combined with the captured image IM2 and is not displayed on the image display device 204 (NO), the image processing unit 211 advances the process to step S206.

The user US rotates the spherical image VSS2 so that the captured image IM2 is horizontal. Since the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM1 are separated, the composite image CM2 does not rotate even if the spherical image VSS2 is rotated. Therefore, the user US can easily correct the horizontality of the captured image IM2 without being confused by the rotation of the composite image CM2.

In step S205, the image processing unit 211 may hide the composite image CM2. By hiding the composite image CM2, the composite image CM2 does not get in the way, so that the user US can easily correct the horizontality of the captured image IM2.

In step S206, the image processing unit 211 moves the captured image IM2 displayed on the image display device 204 in correspondence with the rotation of the spherical image VSS2. The user US may perform the operation of rotating the spherical image VSS2 a plurality of times until the captured image IM2 becomes horizontal.

In FIG. 6B, when the user US determines that the captured image IM2 is horizontal, the image processing unit 211 ends the display of the spherical image VSS2 in step S207 by operating the controller 205, and the image Display the setting screen on the display device 204. By leveling the captured image IM2 displayed on the image display device 204, the zenith of the camera 202 and the zenith of the user US can be matched.

When the user US operates the controller 205 and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 211 corresponds to the selected item in step S208. The process is shifted to the predetermined process mode. When the end item is selected, the image processing unit 211 shifts the processing to the processing mode for ending the horizontal adjustment (horizontal adjustment end mode).

In step S209, the image processing unit 211 acquires the amount of rotation (rotation angle) before and after the spherical image VSS2 rotates as the amount of change in the spherical image VSS2. In step S210, the image processing unit 211 saves the amount of change in the spherical image VSS2 as the correction value CV2.

When the composite image CM2 is not displayed in steps S201 to S210 of FIGS. 6A and 6B, the user US operates the controller 205 to cause the image adjusting device 210 (specifically, the image processing unit 211) to display the image. The composite image CM2 generated by the generation unit 212 can be acquired and displayed on the image display device 204.

The image adjusting device 210 (specifically, the image processing unit 211) recognizes the horizontal direction in the captured image IM2 by analyzing the captured image IM2 captured by the camera 202. After the user US rotates the spherical image VSS2 to adjust the level of the captured image IM2, the image adjusting device 210 acquires the composite image CM2 generated by the image generation unit 212 and displays it on the image display device 204. , The horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 do not match. If the horizontal plane PHP2 of the captured image IM2 and the horizontal plane SHP2 of the composite image CM2 are deviated from each other, the user US feels uncomfortable.

In FIG. 7, the image processing unit 211 determines in step S211 whether or not the composite image CM2 is combined with the captured image IM2 and displayed on the image display device 204. When it is determined that the composite image CM2 is combined with the captured image IM2 and displayed on the image display device 204 (YES), the image processing unit 211 acquires the stored correction value CV2 in step S212. ..

In step S213, the image processing unit 211 rotates the coordinate axes of the composite image CM2 based on the correction value CV2, and makes the horizontal plane SHP2 of the composite image CM2 coincide with the horizontal plane PHP2 of the captured image IM2 horizontally adjusted by the user US. Let me. In step S214, the image processing unit 211 performs arithmetic processing on the composite image CM2 based on the corrected horizontal plane SHP2 to adjust the image. FIG. 8 schematically shows a state in which the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM1 coincide with each other.

When it is determined in step S211 that the captured image IM2 is not horizontally adjusted (NO) by the user US, the image processing unit 211 shifts the processing to step S215. In step S215, the image processing unit 211 combines the image-adjusted composite image CM2 with the captured image IM2 and displays it on the image display device 204. The user US can see the captured image IM2 and the composite image CM2 that are displayed on the image display device 204 and are horizontally aligned with each other.

In the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, the spherical image VSS2 is displayed on the image display device 204. According to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, it is used when the horizontal direction of the captured image IM2 is incorrectly detected or the horizontal or zenith ZE designation is deviated. By operating the controller 205 to rotate the spherical image VSS2, the person US can adjust the captured image IM2 displayed on the image display device 204 so as to be horizontal.

Therefore, according to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, when the horizontal direction of the captured image IM2 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM2.

According to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, when the correction value CV2 is stored, the image processing unit 211 reads out the correction value CV2 and the camera 202 takes a picture. The captured image IM2 can be image-adjusted based on the correction value CV2 and displayed on the image display device 204.

In the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM1 are separated, and the horizontality of the captured image IM1 is adjusted and adjusted. The horizontal of the composite image CM1 is adjusted based on the result (correction value CV2). Since the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM1 are separated, even if the user US rotates the spherical image VSS2, the composite image CM2 does not rotate. Therefore, the user US can easily correct the horizontality of the captured image IM2.

According to the image adjustment system 201, the image adjustment device 210, and the image adjustment method of the second embodiment, when the captured image IM2 is horizontally adjusted, the coordinate axes of the composite image CM2 are rotated based on the correction value CV2 to rotate the composite image. The horizontal plane SHP2 of CM2 is made to match the horizontal plane PHP2 of the captured image IM2. Therefore, the user US can see the captured image IM2 and the composite image CM2 displayed on the image display device 204 without feeling any discomfort.

[Third Embodiment]
A configuration example of the image adjustment system of the third embodiment will be described with reference to FIG. The image adjustment system 301 includes a camera 302, a communication unit 303, a plurality of image display devices 304, a plurality of controllers 305, a plurality of image adjustment devices 310, and a server 306. The image adjusting device 310 includes an image processing unit 311, an image generation unit 312, and a communication unit 313.

The camera 302, the communication unit 303, the image display device 304, the controller 305, the image adjustment device 310, and the server 306 are the cameras 102 and 202 of the first and second embodiments, the communication units 103 and 203, and the image display device 104, respectively. And 204, controllers 105 and 205, image adjusters 110 and 210, and servers 106 and 206. The image processing unit 311, the image generation unit 312, and the communication unit 313 correspond to the image processing units 111 and 211, the image generation units 112 and 212, and the communication units 113 and 213 of the first and second embodiments, respectively. ..

In FIG. 9, two image display devices 304, a controller 305, and an image adjustment device 310 are shown, respectively, for the sake of clarity. The number of the image display device 304, the controller 305, and the image adjustment device 310 may be three or more. The image display device 304, the controller 305, and the image adjustment device 310 used by the user USa (first user) are combined with the image display device 304a (first image display device in the third embodiment) and the controller 305a (first). 3 The first controller in the embodiment) and the image adjusting device 310a (the first image adjusting device in the third embodiment). The image display device 304, the controller 305, and the image adjustment device 310 used by the user USb (second user) are the image display device 304b (second image display device in the third embodiment) and the controller 305b (third). The second controller in the embodiment) and the image adjusting device 310b (the second image adjusting device in the third embodiment).

The image processing unit 311, the image generation unit 312, and the communication unit 313 in the image adjustment device 310a are the image processing unit 311a, the image generation unit 312a, and the communication unit 313a, and the image processing unit 311 and the image generation unit in the image adjustment device 310b. The 312 and the communication unit 313 are the image processing unit 311b, the image generation unit 312b, and the communication unit 313b.

The image adjusting device 310a can acquire the captured image IM3 captured by the camera 302 via the communication units 303 and 313a and the network. The image adjusting device 310b can acquire the captured image IM3 captured by the camera 302 via the communication units 303 and 313b and the network.

The server 306 acquires the captured image IM3 captured by the camera 302 via the communication unit 303 and the network, and the image adjusting devices 310a and 310b acquire the captured image IM3 from the server 306 via the network and the communication units 313a and 313b. You may.

The server 306 can identify the user USa or the image display device 304a when the user USa executes a login operation or the image display device 304a is connected to the network via the communication unit 313a. The server 306 can identify the user USb or the image display device 304b when the user USb executes a login operation or the image display device 304b is connected to the network via the communication unit 313b.

Hereinafter, the processing in the image adjusting device 310a, the image display device 304a, and the controller 305a will be described. Reference numerals in parentheses indicate processing in the image adjusting device 310b, the image display device 304b, and the controller 305b.

The captured image IM3 acquired by the image adjusting device 310a (310b) is input to the image processing unit 311a (311b). The image processing unit 311a (311b) recognizes the horizontal direction in the captured image IM3 by analyzing the captured image IM3. The image processing unit 311a (311b) may recognize the vertical direction in the captured image IM3, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM3.

The image adjusting device 310a (310b) corrects the distortion of the captured image IM3, executes image processing such as adjusting the level of the captured image IM3, and transfers the image-processed captured image IM3 to the image display devices 304a and 304b. Output.

The server 306 corrects the distortion of the captured image IM3 captured by the camera 302, executes image processing such as adjusting the level of the captured image IM3, and transfers the image-processed captured image IM3 to the image adjusting device 310a (310b). It may be output to. The image display device 304a (304b) displays the captured image IM3 image-processed by the image adjustment device 310a (310b) or the server 306.

The user USa (USb) can see the captured image IM3 image-processed by the image adjusting device 310a (310b) or the server 306 by attaching the image display device 304a (304b) to its head.

When the image display device 304a (304b) is attached to the head of the user USa (USb), the direction in which the user USa (USb) is facing, the posture of the user USa (USb), and the like. Attitude information PN3a (PN3b) is generated based on. The image processing unit 311a (311b) acquires the posture information PN3a (PN3b) from the image display device 304a (304b). That is, the image processing unit 311a (311b) acquires the posture information PN3a (PN3b) based on the posture of the image display device 304a (304b).

Based on the posture information PN3a (PN3b), the image processing unit 311a (311b) refers to the direction in which the user USa (USb) is facing and the posture of the user USa (USb) from the captured image IM3 taken by the camera 302. The image of the region corresponding to the above state is displayed on the image display device 304a (304b).

The controller 305a (305b) generates instruction information NN3a (NN3b) based on a state such as the movement or posture of the hand of the user USa (USb) in a state of being attached to the hand of the user USa (USb). .. The image processing unit 311a (311b) acquires instruction information NN3a (NN3b) from the controller 305a (305b). The image processing unit 311a (311b) can change or adjust the captured image IM3 displayed on the image display device 304a (304b) based on the instruction information NN3a (NN3b).

The image generation unit 312a (312b) generates a composite image CM3b (CM3a) which is a CG image to be combined with the captured image IM3. The composite image CM3a (CM3b) is an image of a character such as an avatar. The composite image CM3a is the avatar of the user USa, and the composite image CM3b is the avatar of the user USb. When the user USa (USb) operates the controller 305a (305b), the image generation unit 312a (312b) generates a composite image CM3b (CM3a) which is an avatar of the user USb (USa), and the built-in memory or Save to external memory.

The image processing unit 311a (311b) may acquire the composite image CM3b (CM3a) generated by the image generation unit 312a (312b) and output it to the server 306 via the communication unit 313a (313b) and the network. ..

When the user USa (USb) operates the controller 305a (305b), the image generation unit 312a (312b) may generate the composite image CM3a (CM3b) which is the avatar of the user USa (USb). The image processing unit 311a (311b) may acquire the composite image CM3a (CM3b) generated by the image generation unit 312a (312b) and output it to the server 306 via the communication unit 313a (313b) and the network. .. That is, the user USa may set the avatar of another user USb, may set its own avatar, and the user USb may set the avatar of another user USa. You may also set your own avatar.

The server 306 stores the composite image CM3a (CM3b) in the internal memory or an external memory in association with the user USa (USb). The server 306 may automatically set the composite image CM3a (CM3b) in association with the user USa (USb) and save it in the internal memory or an external memory.

The image processing unit 311a (311b) acquires the composite image CM3b (CM3a) from the image generation unit 312a (312b) or the server 306 based on the instruction information NN3a (NN3b), and displays it on the image display device 304a (304b). The composite image CM3a (CM3b) is synthesized with the captured image IM3. Therefore, the image display device 304a (304b) combines the captured image IM3 captured by the camera 302 with the composite image CM3a (CM3b) acquired from the image generation unit 312a (312b) or the server 306 and combined with the captured image IM3. Can be displayed.

The image generation unit 312a (312b) generates a spherical image VSS3a (VSS3b) which is a virtual image composed of a spherical surface which is CG. The image generation unit 312a (312b) stores the spherical image VSS3a (VSS3b) in the internal memory or an external memory. The image processing unit 311a (311b) acquires the spherical image VSS3a (VSS3b) generated by the image generation unit 312a (312b) based on the instruction information NN3a (NN3b) and displays it on the image display device 304a (304b). To do.

The spherical images VSS3a (VSS3b) correspond to the spherical images VSS1 and VSS2 of the first and second embodiments. The user USa (USb) moves the hand on which the controller 305a (305b) is attached to a position corresponding to the spherical image VSS3a (VSS3b) displayed on the image display device 304a (304b). It feels like the hand of the user USa (USb) is in contact with the spherical image VSS3a (VSS3b).

In the controller 305a (305b), the actuator may be arranged at a portion where the user USa (USb) comes into contact with the hand. The image processing unit 311a (311b) operates the actuator when it is determined that the hand of the user USa (USb) has moved to the position corresponding to the spherical image VSS3a (VSS3b) based on the instruction information NN3a (NN3b). Let me. When the actuator applies pressure to the hand of the user USa (USb), the user USa (USb) can actually feel the hand touching the spherical image VSS3a (VSS3b).

In a state where the spherical image VSS3a (VSS3b) is displayed on the image display device 304a (304b), the user USa (USb) moves the hand on which the controller 305a (305b) is attached in an arbitrary direction. In the case, the image processing unit 311a (311b) uses the spherical image VSS3a (VSS3b) displayed on the image display device 304a (304b) and the captured image IM3 based on the instruction information NN3a (NN3b), and the user USa Image processing is performed so as to move according to the moving direction, moving speed, and moving destination position of the hand (USb).

The user USa (USb) moves the spherical image VSS3a (VSS3b) in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed. Can be rotated to position. That is, the user USa (USb) can rotate the spherical image VSS3a (VSS3b) by the movement of his / her hand. The image processing unit 311a (311b) moves the captured image IM3 in correspondence with the rotation of the spherical image VSS3a (VSS3b).

The image processing unit 311a (311b) is such that the zenith ZE before the user USa (USb) rotates the spherical image VSS3a (VSS3b) and the user USa (USb) rotate the spherical image VSS3a (VSS3b). It is possible to determine to which position on the coordinates of the spherical image VSS3a (VSS3b) it has moved.

In the image processing unit 311a (311b), the user USa (USb) rotates the spherical image VSS3a (VSS3b) based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS3a (VSS3b). The amount of change in the spherical image VSS3a (VSS3b) before and after the rotation is calculated. The amount of change in the spherical images VSS3a (VSS3b) corresponds to the amount of change in the spherical images VSS1 and VSS2 of the first and second embodiments.

The image processing unit 311a (311b) saves the amount of change in the spherical image VSS3a (VSS3b) as a correction value CV3a (CV3b). The correction values CV3a (CV3b) correspond to the correction values CV1 and CV2 of the first and second embodiments. The image processing unit 311a (311b) uses the coordinates on the spherical image VSS3a (VSS3b) of the zenith ZE after the user USa (USb) rotates the spherical image VSS3a (VSS3b) as the correction value CV3a (CV3b). You may save it.

The image processing unit 311a (311b) may save the correction value CV3a (CV3b) in the internal memory or an external memory, or the correction value CV3a (CV3b) is stored in the server via the communication unit 313a (313b) and the network. It may be output to 306. The server 306 associates the correction value CV3a (CV3b) with the image display device 304a (304b) or the user USa (USb) and saves it in the internal memory or an external memory.

The image adjustment system 301 combines the composite image CM3b (CM3a), which is an avatar of another user USb (USa), with the captured image IM3 on the image display device 304a (304b) worn by the user USa (USb). Can be displayed.

When the users USa and USb see the captured image IM3 taken by the camera 302, the users USa and USb feel as if they are looking at the surroundings from the position of the camera 302. In such a case, the image display device 304a (304b) viewed by the user USa (USb) has a composite image CM3b (composite image CM3b) which is an avatar of another user USb (USa) at the position of the user USa (USb). Since CM3a) is displayed, the user USa (USb) feels uncomfortable. When there are a plurality of other user USs, the avatars of the plurality of users USb, USc ,,, are displayed overlapping at the position of the user USa.

In the server 306, the positional relationship of a plurality of users USa, USb ,,, is preset. In other words, the server 306 stores position setting information indicating the positional relationship between the plurality of users USa, USb ,,. The positional relationship between the plurality of users USa, USb ,,, may be automatically set by the server 306, or may be set by the users USa, USb ,,.

FIG. 10 shows an example of the positional relationship between the user USa when the user USa is looking at the image display device 304a and the composite image CM3b which is an avatar of another user USb. FIG. 11 shows an example of the positional relationship between the user USb and the composite image CM3a which is an avatar of another user USa when the user USb is looking at the image display device 304b.

The black dots shown in FIGS. 10 and 11 indicate the position of the camera 302, and the arrows indicate the moving direction of the camera 302. That is, FIGS. 10 and 11 show a state in which the users USa and USb are facing the moving direction of the camera 302. Therefore, in FIGS. 10 and 11, the user USa and the user USb are looking at the same captured image IM3.

In such a case, if the composite image CM3b is displayed at a position away from the user USa or the composite image CM3a is displayed at a position away from the user USb, the users USa and USb feel uncomfortable. Therefore, it is desirable that the composite images CM3a and CM3b are displayed at positions within a predetermined range RG from the positions of the users USa and USb (camera 302).

As shown in FIG. 10, on the server 306, for example, in the user USa or the image display device 304a, the user USa and the user USb are arranged so that the composite image CM3b is arranged side by side on the right side of the user USa. The positional relationship with is set. As shown in FIG. 11, on the server 306, for example, in the user USb or the image display device 304b, the user USa and the user USb are arranged so that the composite image CM3a is arranged side by side on the left side of the user USb. The positional relationship with is set.

That is, FIGS. 10 and 11 show the user USa and the user USb so that the user USb is arranged side by side on the right side of the user USa when the moving direction of the camera 302 is the front direction of the user USa and the USb. The positional relationship with the user USb is set. Therefore, the user USa always sees the user USb on the right side, and the user USb always sees the user USa on the left side. When the camera 302, which is a 360-degree camera, is stationary, there is no front / rear / left / right reference, so it is desirable to determine the positional relationship between the user USa and the user USb at the time of initial setting.

An example of the image adjustment method of the third embodiment will be described with reference to the flowcharts shown in FIGS. 12 and 13. Specifically, an example of a method of synthesizing the composite image CM3 with the captured image IM3 will be described. Hereinafter, the processing in the image adjusting device 310a, the image display device 304a, and the controller 305a will be described. Reference numerals in parentheses indicate processing in the image adjusting device 310b, the image display device 304b, and the controller 305b.

The image display device 304a (304b) is attached to the head of the user USa (USb), and the controller 305a (305b) is attached to the hand of the user USa (USb). The captured image IM3 is displayed on the image display device 304a (304b).

When the user USa (USb) determines that the captured image IM3 displayed on the image display device 304a (304b) is not horizontal, the user USa (USb) operates the controller 305a (305b) in FIG. As a result, the image processing unit 311a (311b) causes the image display device 304a (304b) to display the setting screen in step S301.

When the user USa (USb) operates the controller 305a (305b) and selects a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 311a (311b) steps. In S302, the process is shifted to the predetermined process mode corresponding to the selected item. When the horizontal adjustment item is selected, the image processing unit 311a (311b) shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM3.

In step S303, the image processing unit 311a (311b) acquires the spherical image VSS3a (VSS3b) generated by the image generation unit 312a (312b) and displays it on the image display device 304a (304b). The captured image IM3 and the spherical image VSS3a (VSS3b) are mixed and displayed on the image display device 304a (304b).

The user USa (USb) rotates the spherical image VSS3a (VSS3b) so that the captured image IM3 is horizontal, so that the image processing unit 311a (311b) performs the spherical image VSS3a (VSS3b) in step S304. The captured image IM3 displayed on the image display device 304a (304b) is moved in correspondence with the rotation of. The user USa (USb) may perform the operation of rotating the spherical image VSS3a (VSS3b) a plurality of times until the captured image IM3 becomes horizontal.

When the user USa (USb) determines that the captured image IM3 is horizontal, the controller 305a (305b) is operated, so that the image processing unit 311a (311b) performs the spherical image VSS3a (VSS3b) in step S305. ) Is terminated, and the setting screen is displayed on the image display device 304a (304b). By leveling the captured image IM3 displayed on the image display device 304a (304b), the zenith of the camera 302 and the zenith of the user USa (USb) can be aligned.

When the user USa (USb) operates the controller 305a (305b) and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 311a (311b) moves to step S306. In, the process shifts to the predetermined processing mode corresponding to the selected item. When the end item is selected, the image processing unit 311a (311b) shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S307, the image processing unit 311a (311b) acquires the amount of rotation (rotation angle) before and after the spherical image VSS3a (VSS3b) rotates as the amount of change in the spherical image VSS3a (VSS3b). .. In step S308, the image processing unit 311a (311b) saves the amount of change in the spherical image VSS3a (VSS3b) as a correction value CV3a (CV3b), and ends the process.

The case where the composite image CM3b, which is the avatar of the user USb, is combined with the captured image IM3 and displayed on the image display device 304a viewed by the user USa will be described with reference to the flowchart shown in FIG. The reference numerals in parentheses indicate a case where the composite image CM3a, which is the avatar of the user USa, is combined with the captured image IM3 and displayed on the image display device 304b viewed by the user USb.

In FIG. 13, when the user USa (USb) operates the controller 305a (305b), the image processing unit 311a (311b) causes the image display device 304a (304b) to display the setting screen in step S311. On the setting screen, for example, an avatar selection item for selecting an avatar of another user US is displayed.

The user USa (USb) selects, for example, another user, the user USb (USa). In step S312, the controller 305a (305b) generates instruction information NN3a (NN3b) according to the selection contents of the user USb (USa) and outputs it to the image processing unit 311a (311b).

In step S313, the image processing unit 311a (311b) acquires the composite image CM3b (CM3a) based on the instruction information NN3a (NN3b). When the composite image CM3b (CM3a) is stored in the server 306, the image processing unit 311a (311b) acquires the composite image CM3b (CM3a) from the server 306 via the communication unit 313a (313b) and the network.

The image processing unit 311a (311b) determines in step S314 whether or not the camera 302 is moving. If it is determined in step S314 that the camera 302 is moving (YES), the image processing unit 311a (311b) recognizes the moving direction of the camera 302 in step S315. The image processing unit 311a (311b) can determine whether or not the camera 302 is moving based on the captured image IM3, and can recognize the moving direction of the camera 302.

In step S316, the image processing unit 311a (311b) synthesizes the composite image CM3b (CM3a) so as to be arranged on the right side (left side) of the user USa (USb), for example, with respect to the moving direction of the camera 302. The image CM3b (CM3a) is combined with the captured image IM3 and displayed on the image display device 304a (304b).

If it is determined in step S314 that the camera 302 is not moving (NO), the image processing unit 311a (311b) determines that the camera 302 is in a stationary state, and in step S316, at the time of initial setting. Based on the determined positional relationship between the user USa and the user USb, the composite image CM3b (CM3a) is combined with the captured image IM3 and displayed on the image display device 304a (304b).

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, the image display device 304 displays the spherical image VSS3. According to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, it is used when the horizontal direction of the captured image IM3 is erroneously detected or the horizontal or zenith ZE designation is deviated. By operating the controller 305 to rotate the spherical image VSS3, the person US can adjust the captured image IM3 displayed on the image display device 304 so as to be horizontal.

Therefore, according to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, when the horizontal direction of the captured image IM3 is erroneously detected, or when the horizontal or zenith designation is deviated, The user US can easily correct the horizontal or zenith of the captured image IM3.

According to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, when the correction value CV3 is stored, the image processing unit 311 reads out the correction value CV3 and the camera 302 takes a picture. The captured image IM3 can be image-adjusted based on the correction value CV3 and displayed on the image display device 304.

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, the positional relationship between the plurality of users US is preset. As a result, the composite image CM3b (CM3a), which is an avatar of another user US, is combined with the captured image IM3 displayed on the image display device 304a (304b) at a position based on a preset positional relationship. Can be displayed.

According to the image adjustment system 301, the image adjustment device 310, and the image adjustment method of the third embodiment, when the user USa and USb see the captured image IM3 captured by the camera 302, the user USa (USb) Since the composite image CM3b (CM3a), which is an avatar of another user USb (USa), is displayed in the vicinity of, the user USa (USb) does not feel uncomfortable with the other user USb (USa). You can feel as if you are looking at the same captured image IM3.

[Fourth Embodiment]
A configuration example of the image adjustment system of the fourth embodiment will be described with reference to FIG. The image adjustment system 401 includes a plurality of cameras 402, a plurality of communication units 403, a plurality of image display devices 404, a plurality of controllers 405, a plurality of image adjustment devices 410, and a server 406. The image adjusting device 410 includes an image processing unit 411, an image generation unit 412, and a communication unit 413.

The camera 402, the communication unit 403, and the image display device 404 are the cameras 102, 202, and 302 of the first to third embodiments, the communication units 103, 203, and 303, and the image display devices 104, 204, respectively. And 304. The controller 405, the image adjustment device 410, and the server 406 are the controllers 105, 205, and 305 of the first to third embodiments, the image adjustment devices 110, 210, and 310, and the servers 106, 206, and 306, respectively. Corresponds to. The image processing unit 411, the image generation unit 412, and the communication unit 413 are the image processing units 111, 211, and 311 of the first to third embodiments, the image generation units 112, 212, and 312, and the communication unit, respectively. Corresponds to 113, 213, and 313.

In FIG. 14, two cameras 402, a communication unit 403, an image display device 404, a controller 405, and an image adjustment device 410 are shown, respectively, for the sake of clarity. The number of the camera 402, the communication unit 403, the image display device 404, the controller 405, and the image adjusting device 410 may be three or more. The image display device 404, controller 405, and image adjustment device 410 used by the user USa (first user) are combined with the image display device 404a (first image display device in the fourth embodiment) and controller 405a (first). 4 The first controller in the embodiment) and the image adjusting device 410a (the first image adjusting device in the fourth embodiment). The image display device 404, the controller 405, and the image adjustment device 410 used by the user USb (second user) are the image display device 404b (second image display device in the fourth embodiment) and the controller 405b (second user). 4 The second controller in the embodiment) and the image adjusting device 410b (the second image adjusting device in the fourth embodiment).

The image processing unit 411, the image generation unit 412, and the communication unit 413 in the image adjustment device 410a are the image processing unit 411a, the image generation unit 412a, and the communication unit 413a, and the image processing unit 411 and the image generation unit in the image adjustment device 410b. The 412 and the communication unit 413 are the image processing unit 411b, the image generation unit 412b, and the communication unit 413b. The image adjusting device 410a is connected to the network via the communication unit 413a, and the image adjusting device 410b is connected to the network via the communication unit 413b.

One of the two cameras 402 is the camera 4021 (first camera) and the other is the camera 4022 (second camera). One of the two communication units 403 is the communication unit 4031 and the other is the communication unit 4032. The camera 4021 is connected to the network via the communication unit 4031, and the camera 4022 is connected to the network via the communication unit 4032. The cameras 4021 and 4022 may be connected to the network via a common communication unit 4031 or 4032. Server 406 is connected to the network.

The cameras 4021 and 4022 can move independently of each other. The cameras 4021 and 4022 have a GPS function. The server 406 can continuously acquire the positions (current positions) of the cameras 4021 and 4022 via the network and the communication units 4031 and 4032.

The image adjusting device 410a can acquire the positions of the cameras 4021 and 4022 from the server 406 via the communication unit 413a and the network. The image adjusting device 410b can acquire the positions of the cameras 4021 and 4022 from the server 406 via the communication unit 413b and the network.

The image adjusting device 410a can acquire the captured image IM41 captured by the camera 4021 via the communication units 4031 and 413a and the network. The image adjusting device 410b can acquire the captured image IM41 captured by the camera 4021 via the communication units 4031 and 413b and the network.

The image adjusting device 410a can acquire the captured image IM42 captured by the camera 4022 via the communication units 4032 and 413a and the network. The image adjusting device 410b can acquire the captured image IM42 captured by the camera 4022 via the communication units 4032 and 413b and the network. FIG. 14 shows a case where the image adjusting device 410a acquires the captured image IM41 and the image adjusting device 410b acquires the captured image IM42.

The server 406 may acquire the captured images IM41 and IM42 taken by the cameras 4021 and 4022 via the communication units 4031 and 4032 and the network. In this case, the image adjusting device 410a acquires the captured images IM41 and IM42 from the server 406 via the network and the communication unit 413a. The image adjusting device 410b acquires the captured images IM41 and IM42 from the server 406 via the network and communication unit 413b.

The server 406 can identify the user USa or the image display device 404a when the user USa executes a login operation or the image display device 404a is connected to the network via the communication unit 413a. The server 406 can identify the user USb or the image display device 404b when the user USb executes a login operation or the image display device 404b is connected to the network via the communication unit 413b.

The server 406 can recognize which captured image is displayed on which image display device. For example, as shown in FIG. 14, when the captured image IM41 is displayed on the image display device 404a and the captured image IM42 is displayed on the image display device 404b, the server 406 displays the captured image IM41 on the image display device 404a. It recognizes that the captured image IM42 is displayed on the image display device 404b. In other words, the server 406 may recognize that the user USa is looking at the captured image IM41 taken by the camera 4021 and that the user USb is looking at the captured image IM42 taken by the camera 4022. it can.

The image adjusting device 410a can recognize from the server 406 that the current positions of the cameras 4021 and 4022 that are capturing the captured images IM41 and IM42 and that the image adjusting device 410b has acquired the captured image IM42. .. The image adjusting device 410b can recognize from the server 406 that the current positions of the cameras 4021 and 4022 that are capturing the captured images IM41 and IM42 and that the image adjusting device 410a has acquired the captured image IM41. ..

The captured image IM41 (IM42) acquired by the image adjusting device 410a is input to the image processing unit 411a. The image processing unit 411a recognizes the horizontal direction in the captured image IM41 (IM42) by analyzing the captured image IM41 (IM42). The image processing unit 411a may recognize the vertical direction in the captured image IM41 (IM42), or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM41 (IM42).

The image adjusting device 410a corrects the distortion of the captured image IM41 (IM42), executes image processing such as adjusting the level of the captured image IM41 (IM42), and images the image-processed captured image IM41 (IM42). Output to the display device 404a.

The captured image IM42 (IM41) acquired by the image adjusting device 410b is input to the image processing unit 411b. The image processing unit 411b recognizes the horizontal direction in the captured image IM42 (IM41) by analyzing the captured image IM42 (IM41). The image processing unit 411b may recognize the vertical direction in the captured image IM42 (IM41), or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM42 (IM41).

The image adjusting device 410b corrects the distortion of the captured image IM42 (IM41), executes image processing such as adjusting the level of the captured image IM42 (IM41), and images the image-processed captured image IM42 (IM41). Output to display device 404b.

The server 406 corrects the distortion of the captured images IM41 and IM42 captured by the cameras 4021 and 4022, executes image processing such as adjusting the horizontality of the captured images IM41 and IM42, and performs the image-processed captured images IM41 and IM42. , The image may be output to the image adjusting device 310a, or may be output to the image adjusting device 310b.

The image display device 404a displays the captured image IM41 (IM42) image-processed by the image adjustment device 410a or the server 406. By attaching the image display device 404a to its head, the user USa can see the captured image IM41 (IM42) image-processed by the image adjustment device 410a or the server 406.

The image display device 404b displays the captured image IM42 (IM41) image-processed by the image adjustment device 410b or the server 406. By attaching the image display device 404b to its head, the user USb can see the captured image IM42 (IM41) image-processed by the image adjustment device 410b or the server 406.

When the image display device 404a (404b) is attached to the head of the user USa (USb), the direction in which the user USa (USb) is facing, the posture of the user USa (USb), and the like. Attitude information PN4a (PN4b) is generated based on. The image processing unit 411a (411b) acquires the posture information PN4a (PN4b) from the image display device 404a (404b). That is, the image processing unit 411a (411b) acquires the posture information PN4a (PN4b) based on the posture of the image display device 404a (404b).

The image processing unit 411a is a region corresponding to the direction in which the user USa is facing and the state such as the posture of the user USa from the captured image IM41 or IM42 taken by the camera 4021 or 4022 based on the posture information PN4a. The image is displayed on the image display device 404a. The image processing unit 411b is a region corresponding to the direction in which the user USb is facing and the state such as the posture of the user USb from the captured image IM41 or IM42 taken by the camera 4021 or 4022 based on the posture information PN4b. The image is displayed on the image display device 404b.

The controller 405a (405b) generates instruction information NN4a (NN4b) based on a state such as the movement or posture of the hand of the user USa (USb) in a state of being attached to the hand of the user USa (USb). .. The image processing unit 411a (411b) acquires the instruction information NN4a (NN4b) from the controller 405a (405b).

The image processing unit 411a can change or adjust the captured image IM41 or IM42 displayed on the image display device 404a based on the instruction information NN4a. The image processing unit 411b can change or adjust the captured image IM41 or IM42 displayed on the image display device 404b based on the instruction information NN4b.

The image generation unit 412a generates a composite image CM4b which is a CG to be synthesized with the captured image IM41 (IM42). The image generation unit 412b generates a composite image CM4a which is a CG to be synthesized with the captured image IM42 (IM41). The composite images CM4a and CM4b are images of characters such as avatars. The composite image CM4a is the avatar of the user USa, and the composite image CM4b is the avatar of the user USb.

When the user USa operates the controller 405a, the image generation unit 412a generates a composite image CM4b which is an avatar of the user USb and stores it in the internal memory or an external memory. When the user USb operates the controller 405b, the image generation unit 412b generates a composite image CM4a which is an avatar of the user USa and stores it in the internal memory or an external memory.

The image processing units 411a and 411b may acquire the composite images CM4b and CM4a generated by the image generation units 412a and 412b and output them to the server 406 via the communication units 413a and 413b and the network.

When the user USa (USb) operates the controller 405a (405b), the image generation unit 412a (412b) may generate a composite image CM4a (CM4b) which is an avatar of the user USa (USb). The image processing unit 411a (411b) may acquire the composite image CM4a (CM4b) generated by the image generation unit 412a (412b) and output it to the server 406 via the communication unit 413a (413b) and the network. .. That is, the user USa may set the avatar of another user USb, may set its own avatar, and the user USb may set the avatar of another user USa. You may also set your own avatar.

The server 406 associates the composite image CM4a (CM4b) with the user USa (USb) and stores it in the internal memory or an external memory. The server 406 may automatically set the composite image CM4a (CM4b) in association with the user USa (USb) and save it in the internal memory or an external memory. The image processing unit 411a (411b) acquires the composite image CM4b (CM4a) from the image generation unit 412a (412b) or the server 306 based on the instruction information NN4a (NN4b) and displays it on the image display device 404a (404b). be able to.

The image generation unit 412a (412b) generates a spherical image VSS4a (VSS4b) which is a virtual image composed of a spherical surface which is CG. The image generation unit 412a (412b) stores the spherical image VSS4a (VSS4b) in the internal memory or an external memory. The image processing unit 411a (411b) acquires the spherical image VSS4a (VSS4b) generated by the image generation unit 412a (412b) based on the instruction information NN4a (NN4b) and displays it on the image display device 404a (404b). To do.

The spherical image VSS4a (VSS4b) corresponds to the spherical images VSS1, VSS2, VSS3a, and VSS3b of the first to third embodiments. The user USa (USb) moves the hand on which the controller 405a (405b) is attached to a position corresponding to the spherical image VSS4a (VSS4b) displayed on the image display device 404a (404b). It feels like the hand of the user USa (USb) is in contact with the spherical image VSS4a (VSS4b).

The controller 405a (405b) may have an actuator arranged at a portion in contact with the hand of the user USa (USb). The image processing unit 411a (411b) operates the actuator when it is determined that the hand of the user USa (USb) has moved to the position corresponding to the spherical image VSS4a (VSS4b) based on the instruction information NN4a (NN4b). Let me. When the actuator applies pressure to the hand of the user USa (USb), the user USa (USb) can actually feel the hand touching the spherical image VSS4a (VSS4b).

When the user USa moves the hand on which the controller 405a is attached in an arbitrary direction while the spherical image VSS4a is displayed on the image display device 404a, the image processing unit 411a sends the instruction information NN4a to the instruction information NN4a. Based on this, the spherical image VSS4a displayed on the image display device 404a and the captured image IM41 or IM42 move according to the moving direction, moving speed, and moving destination position of the user USa's hand. Perform image processing.

When the user USb moves the hand on which the controller 405b is attached in an arbitrary direction while the spherical image VSS4b is displayed on the image display device 404b, the image processing unit 411b sends the instruction information NN4b. Based on this, the spherical image VSS4b displayed on the image display device 404b and the captured image IM41 or IM42 move according to the moving direction, moving speed, and moving destination position of the user USb's hand. Perform image processing.

The user USa (USb) moves the spherical image VSS3a (VSS3b) in an arbitrary direction and at an arbitrary speed by moving the hand in an arbitrary direction and at an arbitrary speed. Can be rotated to position. That is, the user USa (USb) can rotate the spherical image VSS3a (VSS3b) by the movement of his / her hand. The image processing unit 311a (311b) moves the captured image IM3 in correspondence with the rotation of the spherical image VSS3a (VSS3b).

The image processing unit 411a (411b) is such that the zenith ZE before the user USa (USb) rotates the spherical image VSS4a (VSS4b) is the zenith ZE, and the user USa (USb) rotates the spherical image VSS4a (VSS4b). It is possible to determine to which position on the coordinates of the spherical image VSS4a (VSS4b) it has moved.

In the image processing unit 411a (411b), the user USa (USb) rotates the spherical image VSS4a (VSS4b) based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS4a (VSS4b). The amount of change in the spherical image VSS4a (VSS4b) before and after the rotation is calculated. The amount of change in the spherical image VSS4a (VSS4b) corresponds to the amount of change in the spherical images VSS1, VSS2, VSS3a, and VSS3b of the first to third embodiments.

The image processing unit 411a (411b) saves the amount of change in the spherical image VSS4a (VSS4b) as a correction value CV4a (CV4b). The correction values CV4a (CV4b) correspond to the correction values CV1, CV2, CV3a, and CV3b of the first to third embodiments. The image processing unit 411a (411b) uses the coordinates on the spherical image VSS4a (VSS4b) of the zenith ZE after the user USa (USb) rotates the spherical image VSS4a (VSS4b) as the correction value CV4a (CV4b). You may save it.

The image processing unit 411a (411b) may save the correction value CV4a (CV4b) in the internal memory or an external memory, or the correction value CV4a (CV4b) is stored in the server via the communication unit 413a (413b) and the network. It may be output to 406. The server 406 associates the correction value CV4a (CV4b) with the image display device 404a (404b) or the user USa (USb) and stores it in the internal memory or an external memory.

The image adjustment system 401 can display the composite image CM4b, which is an avatar of another user USb, by combining it with the captured image IM41 (IM42) on the image display device 404a worn by the user USa. The image adjustment system 401 can display the composite image CM4a, which is an avatar of another user USa, by combining it with the captured image IM42 (IM41) on the image display device 404b worn by the user USb.

An example of the image adjustment method of the fourth embodiment will be described with reference to the flowcharts shown in FIGS. 15 and 16. Specifically, an example of a method of synthesizing the composite image CM4b with the captured image IM41 will be described. The image display device 404a is attached to the head of the user USa, and the controller 405a is attached to the hand of the user USa. The image display device 404b is attached to the head of the user USb, and the controller 405b is attached to the hand of the user USb. The image display device 404a displays the captured image IM41 captured by the camera 4021, and the image display device 404b displays the captured image IM42 captured by the camera 4022.

When the user USa (USb) determines that the captured image IM41 (IM42) displayed on the image display device 404a (404b) is not horizontal, the user USa (USb) uses the controller 405a (405b) in FIG. By operating, the image processing unit 411a (411b) causes the image display device 404a (404b) to display the setting screen in step S401.

When the user USa (USb) operates the controller 405a (405b) and selects a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 411a (411b) steps. In S402, the process is shifted to the predetermined process mode corresponding to the selected item. When the horizontal adjustment item is selected, the image processing unit 411a (411b) shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM41 (IM42).

In step S403, the image processing unit 411a (411b) acquires the spherical image VSS4a (VSS4b) generated by the image generation unit 412a (412b) and displays it on the image display device 404a (404b). The captured image IM41 (IM42) and the spherical image VSS4a (VSS4b) are mixed and displayed on the image display device 404a (404b).

The user USa (USb) rotates the spherical image VSS4a (VSS4b) so that the captured image IM41 (IM42) is horizontal, so that the image processing unit 411a (411b) moves the spherical image VSS4a in step S404. The captured image IM41 (IM42) displayed on the image display device 404a (404b) is moved in correspondence with the rotation of (VSS4b). The user USa (USb) may perform the operation of rotating the spherical image VSS4a (VSS4b) a plurality of times until the captured image IM41 (IM42) becomes horizontal.

When the user USa (USb) determines that the captured image IM41 (IM42) is horizontal, the user USa (USb) operates the controller 405a (405b), so that the image processing unit 411a (411b) moves the spherical image in step S405. The display of the VSS4a (VSS4b) is terminated, and the setting screen is displayed on the image display device 404a (404b). By leveling the captured image IM41 (IM42) displayed on the image display device 404a (404b), the zenith of the camera 4021 (4022) and the zenith of the user USa (USb) can be aligned.

The user USa (USb) operates the controller 405a (405b) to select a predetermined item (for example, an end item) displayed on the setting screen, so that the image processing unit 411a (411b) moves to step S406. In, the process shifts to the predetermined processing mode corresponding to the selected item. When the end item is selected, the image processing unit 411a (411b) shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S407, the image processing unit 411a (411b) acquires the amount of rotation (rotation angle) before and after the spherical image VSS4a (VSS4b) rotates as the amount of change in the spherical image VSS4a (VSS4b). .. In step S408, the image processing unit 411a (411b) saves the amount of change in the spherical image VSS4a (VSS4b) as a correction value CV4a (CV4b), and ends the process.

Using the flowchart shown in FIG. 16, a case where the composite image CM4b, which is the avatar of the user USb, is combined with the captured image IM41 and displayed on the image display device 404a viewed by the user USa will be described. In FIG. 16, the image processing unit 411a identifies the camera 4021 that is capturing the acquired captured image IM41 in step S411.

In step S412, the image processing unit 411a acquires the position information of the camera 4021 from the camera 4021 or the server 406. The position information of the camera 4021 includes the current position of the camera 4021 and the information in the direction in which the camera 4021 is facing or moving.

In step S413, the image processing unit 411a acquires information on another camera (a camera other than the camera 4021) from the server 406. For example, the image processing unit 411a acquires the information of the camera 4022 from the server 406 in step S413. The information of the camera 4022 includes information on the user USb looking at the captured image IM42 taken by the camera 4022, the current position of the camera 4022, and the direction in which the camera 4022 is facing or moving. It has been. The direction in which the camera 4022 is facing or moving corresponds to the direction in which the user USb is facing, and can be specified based on the captured image IM42 displayed on the image display device 404b.

In step S414, the image processing unit 411a determines whether or not another camera (a camera other than the camera 4021) is reflected in the captured image IM41, in other words, whether or not another camera is photographing, for example, by image recognition. judge. If it is determined in step S414 that no other camera is visible in the captured image IM41 (NO), the image processing unit 411a returns the process to step S414.

If it is determined in step S414 that another camera is reflected in the captured image IM41 (YES), the image processing unit 411a determines the captured image in step S415 based on the information of the other camera acquired in step S413. Identify other cameras in the IM41.

FIG. 17 shows a state in which the camera 4021 is located at the point A and the camera 4022 is located at the point B in front of the camera 4021. In the state shown in FIG. 17, the captured image IM41 shows the camera 4022, which is another camera. The image processing unit 411a identifies that the camera shown in the captured image IM 41 is the camera 4022 based on the information of the camera 4022 acquired in step S413.

In step S416, the image processing unit 411a determines the position of the camera 4022 with respect to the camera 4021 and the camera 4022 based on the position information of the camera 4021 acquired in step S412 and the information of the camera 4022 acquired in step S413. Identify the direction you are facing or moving.

In step S417, the image processing unit 411a identifies the user US who is viewing the captured image IM captured by the camera identified in step S415. For example, the image processing unit 411a identifies the user USb who is looking at the captured image IM42 captured by the camera 4022. In step S418, the image processing unit 411a acquires the composite image CM4b which is the avatar of the user USb. When the composite image CM4b is stored in the server 406, the image processing unit 411a acquires the composite image CM4b from the server 406.

In step S419, the image processing unit 411a synthesizes the composite image CM4b at the position corresponding to the camera 4022 shown in the captured image IM41 and displays it on the image display device 404a. When the captured image IM41 is horizontally adjusted in steps S401 to S408, the image processing unit 411a horizontally adjusts the composite image CM4b in the same manner as the captured image IM41 and displays it on the image display device 404a. When the correction value CV4a is stored, the image processing unit 411a horizontally adjusts the composite image CM4b based on the correction value CV4a and displays it on the image display device 404a.

The image processing unit 411a may change the size of the composite image CM4b according to the position of the camera 4022 with respect to the camera 4021. For example, in the image processing unit 411a, when the camera 4022 is located far away from the camera 4021, the size of the composite image CM4b becomes smaller according to the distance between the camera 4021 and the camera 4022. The composite image CM4b is image-processed and composited with the captured image IM41.

The image processing unit 411a may change the direction of the composite image CM4b according to the direction in which the camera 4022 is facing or moving. The image processing unit 411a may acquire the composite image CM4b in the direction corresponding to the direction in which the camera 4022 is facing or moving.

In the image adjusting system 401, the image adjusting devices 410a and 410b, and the image adjusting method of the fourth embodiment, the spherical images VSS4a and VSS4b are displayed on the image display devices 404a and 404b. According to the image adjustment system 401, the image adjustment devices 410a and 410b, and the image adjustment method of the fourth embodiment, the horizontal directions of the captured images IM41 and IM42 are incorrectly detected, or the designation of the horizontal or zenith ZE is deviated. In this case, the users USa and USb operate the controllers 405a and 405b to rotate the spherical images VSS4a and VSS4b, so that the captured images IM41 and IM42 displayed on the image display devices 404a and 404b become horizontal. Can be adjusted as follows.

Therefore, according to the image adjustment system 401, the image adjustment devices 410a and 410b, and the image adjustment method of the fourth embodiment, the horizontal directions of the captured images IM41 and IM42 are incorrectly detected, or the horizontal or zenith designation is deviated. In this case, the user US can easily correct the horizontal or zenith of the captured images IM41 and IM42.

According to the image adjustment system 401, the image adjustment devices 410a and 410b, and the image adjustment method of the fourth embodiment, when the correction values CV4a and CV4b are stored, the image processing units 411a and 411b perform the correction values CV4a and CV4b. The captured images IM41 and IM42 captured by the cameras 4021 and 4022 can be adjusted based on the correction values CV4a and CV4b and displayed on the image display devices 404a and 404b.

In the image adjustment system 401, the image adjustment devices 410a and 410b, and the image adjustment method of the fourth embodiment, the position of the camera 4022 with respect to the camera 4021 is specified, and the camera 4022 takes a picture at the position of the camera 4022 in the captured image IM. The composite image CM4b, which is the avatar of the user USb looking at the captured image IM42, is synthesized. As a result, the user USa can obtain the feeling of sharing the same space as the user USb.

The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

For example, the virtual image VSS generated by CG may be an ellipsoidal surface (elliptical surface), and may be an arbitrary closed surface (closed curved surface) within the reach of the user US. All you need is. That is, it suffices if the photographed image IM can be horizontally adjusted with the feeling that the user US touches the closed curved surface from the inside. Since the user US rotates the virtual image VSS, it is desirable that the virtual image VSS has a spherical shape close to a spherical surface such as a spherical surface or an ellipsoidal surface.

101, 201, 301, 401 Image adjustment system 102, 202, 302, 402 (4021, 4022) Camera 104, 204, 304 (304a, 304b), 404 (404a, 404b) Image display device 105, 205, 305 (305a) , 305b), 405 (405a, 405b) Controller 106, 206, 306, 406 Server 110, 210, 310 (310a, 310b), 410 (410a, 410b) Image adjuster 111, 211, 311 (311a, 311b), 411 (411a, 411b) Image processing unit 112, 212, 312 (312a, 312b), 412 (412a, 412b) Image generation unit CM1, CM2, CM3 (CM3a, CM3b), CM4 (CM4a, CM4b) Composite image IM1, IM2, IM3, IM4 (IM41, IM42) Captured image NN1, NN2, NN3 (NN3a, NN3b), NN4 (NN4a, NN4b) Instruction information PHP1, PHP2, SHP1, SHP2 Horizontal plane US, USa, USb User VSS1, VSS2 VSS3 (VSS3a, VSS3b), VSS4 (VSS4a, VSS4b) Spherical image (virtual image)

Claims (5)

With multiple cameras
An image adjusting device that adjusts a captured image taken by the first camera among the plurality of cameras, and an image adjusting device.
An image display device that displays the captured image adjusted by the image adjustment device, and
A controller that outputs instruction information to the image adjustment device,
With
The image adjusting device is
An image generation unit that generates a spherical image and a composite image that is combined with the captured image.
The spherical image generated by the image generation unit is acquired based on the instruction information and displayed on the image display device, the spherical image is rotated based on the instruction information, and the spherical image is rotated. The captured image displayed on the image display device is adjusted in accordance with the above, the composite image is adjusted in correspondence with the adjusted captured image, and the captured image is the second of the plurality of cameras. When the camera is shown, the second camera of the captured image shows the composite image adjusted based on the position information of the first camera and the position information of the second camera. An image processing unit that synthesizes images according to the position
Image adjustment system with. The camera is an omnidirectional camera that captures a 360-degree range.
The image display device is a head-mounted display mounted on the user's head.
The controller is a glove-type controller that is worn on the user's hand.
When the user sees the spherical image displayed on the image display device while the image display device is attached to the head of the user, the spherical image is used. It is a virtual image that is arranged around a person and the image display device and is set so that the user's hand or finger is displayed within a range that can reach the spherical image.
The image adjustment system according to claim 1. When the user moves his / her hand or finger in an arbitrary direction while the controller is attached to the user's hand, the image processing unit uses the instruction information of the user. The spherical image is rotated in response to the movement of the hand or finger, and the captured image displayed on the image display device is adjusted in response to the rotation of the spherical image.
The image adjustment system according to claim 2. An image generator that generates a spherical image and a composite image that is combined with the captured image captured by the camera and displayed on the image display device.
The spherical image generated by the image generation unit is acquired based on the instruction information acquired from the controller and displayed on the image display device, and the spherical image is rotated based on the instruction information of a plurality of cameras. The captured image captured by the first camera and displayed on the image display device is adjusted in correspondence with the rotation of the spherical image, and the composite image is adjusted in correspondence with the adjusted captured image. When the second camera of the plurality of cameras is reflected in the captured image, the adjustment is made based on the position information of the first camera and the position information of the second camera. An image processing unit that synthesizes the composite image corresponding to the position where the first camera is projected on the captured image, and an image processing unit.
An image adjustment device comprising. The image processing unit acquires the instruction information from the controller and
The image processing unit acquires a spherical image generated by the image generation unit based on the instruction information, and obtains a spherical image.
The image display device displays the spherical image,
The image processing unit rotates the spherical image based on the instruction information, and the spherical image is rotated.
The image processing unit adjusts the captured image captured by the first camera among the plurality of cameras and displayed on the image display device in accordance with the rotation of the spherical image.
The image processing unit acquires a composite image generated by the image generation unit and combined with the captured image displayed on the image display device.
The image processing unit adjusts the composite image in correspondence with the adjusted captured image.
When the second camera of the plurality of cameras is shown in the captured image, the image processing unit is based on the position information of the first camera and the position information of the second camera. , The adjusted composite image is composited in correspondence with the position where the second camera is projected in the captured image.
Image adjustment method.

Images