WO2020166492A1

WO2020166492A1 - Image adjustment system, image adjustment device, and image adjustment method

Info

Publication number: WO2020166492A1
Application number: PCT/JP2020/004700
Authority: WO
Inventors: 崇日昔
Original assignee: 株式会社Ｊｖｃケンウッド
Priority date: 2019-02-15
Filing date: 2020-02-07
Publication date: 2020-08-20
Also published as: US20210329223A1

Abstract

An image adjustment system (101) comprises a camera (102), an image adjustment device (110), an image display device (104), and a controller (105). The image adjustment device (110) is equipped with an image generation unit (112) and an image processing unit (111). The image generation unit (112) generates a spherical image (VSS1) and a synthetic image (CM1) that is to be combined with a captured image (IM1) captured by the camera (102). The image processing unit (111) acquires the spherical image (VSS1) generated by the image generation unit (112), and displays said image on the image display device (104). The image processing unit (111) rotates the spherical image (VSS1) on the basis of instruction information (NN1) output from the controller (105), adjusts the captured image (IM1), which is displayed on the image display device (104), in accordance with the rotation of the spherical image (VSS1), adjusts the synthetic image (CM1) in accordance with the adjusted captured image (IM1), and combines the synthetic image (CM1) with the captured image (IM1).

Description

Image adjusting system, image adjusting apparatus, and image adjusting method

The present invention relates to an image adjusting system, an image adjusting device, and an image adjusting method.

In recent years, head-mounted displays have been attracting attention as image display devices. The head mounted display displays an image in a state where the head mounted display is mounted on the user's head, so that the user can feel as if he/she entered the virtual space (immersive feeling). The conventional head-mounted display described in Patent Document 1 can display an image captured by an external camera via a neckwork.

JP-A-2005-56295 (JP2005-056295A)

However, with the conventional head-mounted display, it is difficult to detect the horizontal direction of the image depending on the displayed image, and the horizontal direction may be detected incorrectly. Also, due to problems such as system synchronization, the horizontal or zenith designation may shift. If the horizontal direction of the image is erroneously detected, or the horizontal or zenith designation is deviated, the displayed image does not match the user's sense of gravity, and the user feels uncomfortable.

The present invention provides an image adjusting system, an image adjusting device, and an image adjusting system capable of easily correcting the horizontal or zenith of an image when the horizontal direction of the image is erroneously detected or the designation of the horizontal or zenith is deviated. The purpose is to provide a method of adjustment.

An image adjustment system according to a first aspect of the present invention is for displaying a camera, an image adjustment device for adjusting a captured image captured by the camera, and the captured image adjusted by the image adjustment device. An image display device and a controller for outputting instruction information to the image adjustment device, wherein the image adjustment device generates an image for generating a spherical image and a composite image to be combined with the captured image. Section and the spherical image generated by the image generation unit based on the instruction information is acquired and displayed on the image display device, and the spherical image is rotated based on the instruction information to generate the spherical image. Image processing for adjusting the captured image displayed on the image display device in correspondence with the rotation of the image, adjusting the composite image in association with the adjusted captured image, and combining with the captured image And a section.

An image adjusting apparatus according to a second aspect of the present invention includes an image generating unit for generating a spherical image and a composite image that is combined with a captured image captured by a camera and displayed on an image display device. The spherical image generated by the image generation unit based on the instruction information acquired from the controller is acquired and displayed on the image display device, the spherical image is rotated based on the instruction information, and the image is displayed. An image for adjusting the captured image displayed on the display device in correspondence with the rotation of the spherical image, adjusting the composite image in association with the adjusted captured image, and combining the captured image with the captured image. And a processing unit.

In the image adjusting method according to the third aspect of the present invention, the image processing unit acquires the instruction information from the controller, and the image processing unit acquires the spherical image generated by the image generation unit based on the instruction information. An image display device displays the spherical image, the image processing unit rotates the spherical image based on the instruction information, and the image processing unit is photographed by a camera and displayed on the image display device. A captured image that is adjusted in accordance with the rotation of the spherical image, and the image processing unit generates a composite image that is generated by the image generation unit and that is combined with the captured image displayed on the image display device. Then, the image processing unit adjusts the composite image corresponding to the adjusted captured image and combines the adjusted image with the captured image.

According to the image adjustment system of the first aspect, the image adjustment device of the second aspect, and the image adjustment method of the third aspect, the horizontal direction of the image is erroneously detected, When the designation of the zenith shifts, the horizontal or zenith of the image can be easily corrected.

FIG. 1 is a configuration diagram showing an image adjustment system according to the first embodiment and the second embodiment. FIG. 2 is a diagram showing the relationship between the spherical image and the user. FIG. 3 is a flowchart showing an example of the image adjusting method according to the first embodiment. FIG. 4 is a flowchart showing an example of the image adjusting method according to the first embodiment. FIG. 5 is a diagram showing the relationship between the horizontal plane of the captured image and the horizontal plane of the composite image. FIG. 6A is a flowchart showing an example of the image adjusting method according to the second embodiment. FIG. 6B is a flowchart showing an example of the image adjusting method according to the second embodiment. FIG. 7 is a flowchart showing an example of the image adjusting method according to the second embodiment. FIG. 8 is a diagram showing the relationship between the horizontal plane of the captured image and the horizontal plane of the composite image. FIG. 9 is a configuration diagram showing an image adjustment system according to the third embodiment. FIG. 10 is a diagram showing an example of the positional relationship between the user when the user is looking at the image display device and a composite image corresponding to another user. FIG. 11 is a diagram showing an example of a positional relationship between the user and a composite image corresponding to another user when the user is looking at the image display device. FIG. 12 is a flowchart showing an example of the image adjusting method according to the third embodiment. FIG. 13 is a flowchart showing an example of the image adjusting method according to the third embodiment. FIG. 14 is a configuration diagram showing an image adjustment system according to the fourth embodiment. FIG. 15 is a flowchart showing an example of the image adjusting method according to the third embodiment. FIG. 16 is a flowchart showing an example of the image adjusting method according to the third embodiment. FIG. 17 is a diagram showing an example of the positional relationship between a plurality of cameras and a composite image corresponding to another user.

[First Embodiment]
A configuration example of the image adjustment system 101 according to the first embodiment will be described with reference to FIG. 1. The image adjustment system 101 according to the first embodiment includes a camera 102, a first communication unit 103, an image display device 104, a controller 105, an image adjustment device 110, and a server 106. The image adjustment device 110 includes an image processing unit 111, an image generation unit 112, and a second communication unit 113.

The camera 102 is an omnidirectional camera (360 degree camera) capable of shooting a 360 degree range. The first communication unit 103 and the second communication unit 113 are connected via a network. The image adjustment apparatus 110 can acquire the captured image IM1 captured by the camera 102 via the first communication unit 103, the second communication unit 113, and the network.

A computer device may be used as the image adjustment device 110. A CPU may be used as the image processing unit 111 and the image generation unit 112. The image generation unit 112 and the second communication unit 113 may be provided outside the image adjustment device 110.

The server 106 is connected to the camera 102 via the network and the first communication unit 103, and is connected to the image adjustment apparatus 110 via the network and the second communication unit 113. The server 106 acquires the captured image IM1 captured by the camera 102 via the first communication unit 103 and the network, and the image adjustment apparatus 110 acquires the captured image IM1 from the server 106 via the network and the second communication unit 113. Good.

The captured image IM1 acquired by the image adjusting apparatus 110 is input to the image processing unit 111. The image processing unit 111 analyzes the captured image IM1 to recognize the horizontal direction in the captured image IM1. The image processing unit 111 may recognize the vertical direction in the captured image IM1, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM1. The image adjustment device 110 corrects the distortion of the captured image IM1, executes image processing such as adjusting the horizontal of the captured image IM1, and outputs the image-processed captured image IM1 to the image display device 104.

The server 106 corrects the distortion of the captured image IM1 captured by the camera 102, executes image processing such as adjusting the horizontal of the captured image IM1, and outputs the image-processed captured image IM1 to the image adjustment apparatus 110. May be. The image display device 104 displays the captured image IM1 image-processed by the image adjustment device 110 or the server 106.

The image display device 104 is, for example, a head mounted display. The controller 105 is, for example, a globe-type controller used for VR (Virtual Reality) or the like.

FIG. 2 schematically shows a state in which the image display device 104 is mounted on the head of the user US and the controller 105 is mounted on the hand of the user US. Reference numeral ZE shown in FIG. 2 indicates the zenith. It is desirable that the zenith of the camera 102 and the zenith of the user US coincide. By mounting the image display device 104 on the head of the user US, the user US can see the captured image IM1 image-processed by the image adjustment device 110 or the server 106.

The image display device 104 generates the posture information PN1 based on the direction in which the user US is facing, the posture of the user US, and the like while being mounted on the head of the user US. The image processing unit 111 acquires the posture information PN1 from the image display device 104. That is, the image processing unit 111 acquires the posture information PN1 based on the posture of the image display device 104. Based on the posture information PN1, the image processing unit 111 displays an image of a region corresponding to the direction in which the user US is facing and the state such as the posture of the user US from the captured image IM1 captured by the camera 102. Display on the device 104.

The controller 105 generates the instruction information NN1 based on the movement or posture of the hand of the user US, the movement or posture of the finger of the user US, or the like in a state where the controller 105 is attached to the hand of the user US. .. Hereinafter, the hand or the finger is simply abbreviated as the hand. The image processing unit 111 acquires the instruction information NN1 from the controller 105. The image processing unit 111 can change or adjust the captured image IM1 displayed on the image display device 104 based on the instruction information NN1.

The image generation unit 112 generates a composite image CM1 which is a CG (Computer Graphics) composited with the captured image IM1. The composite image CM1 is an image of a character such as an avatar. The image generation unit 112 stores the composite image CM1 in a built-in memory or an external memory. The image processing unit 111 may acquire the composite image CM1 generated by the image generation unit 112 and output it to the server 106 via the second communication unit 113 and the network.

The server 106 stores the composite image CM1 in the built-in memory or an external memory in association with the image display device 104 or the user US. The image processing unit 111 acquires the combined image CM1 from the image generation unit 112 or the server 106 based on the instruction information NN1 and combines the combined image CM1 with the captured image IM1 displayed on the image display device 104. Therefore, the image display device 104 can display the captured image IM1 captured by the camera 102 and the composite image CM1 acquired from the image generation unit 112 or the server 106 and combined with the captured image IM1.

The image generation unit 112 generates a spherical image VSS1 which is a virtual image formed by a spherical surface which is CG. The image generation unit 112 stores the spherical image VSS1 in a built-in memory or an external memory. The image processing unit 111 acquires the spherical image VSS1 generated by the image generation unit 112 based on the instruction information NN1 and displays it on the image display device 104.

FIG. 2 shows an image of the user US when the user US views the spherical image VSS1 displayed on the image display device 104 with the image display device 104 mounted on the head of the user US. It is shown schematically.

When the user US views the spherical image VSS1 displayed on the image display device 104 while the image display device 104 is mounted on the head of the user US, the spherical image VSS1 is It is arranged around the US and the image display device 104, and is set so as to be displayed within a range where the hand of the user US reaches the spherical image VSS1. The user US moves the hand on which the controller 105 is attached to the position corresponding to the spherical image VSS1 displayed on the image display device 104, so that the hand of the user US contacts the spherical image VSS1. I feel like I'm doing it.

The controller 105 may have an actuator arranged in a portion that comes into contact with the hand of the user US. When the image processing unit 111 determines that the hand of the user US has moved to the position corresponding to the spherical image VSS1 based on the instruction information NN1, the image processing unit 111 operates the actuator. When the actuator applies pressure to the hand of the user US, the user US can actually feel the touch of the hand with the spherical image VSS1.

When the user US moves the hand on which the controller 105 is attached in any direction while the spherical image VSS1 is displayed on the image display device 104, the image processing unit 111 displays the instruction information NN1. Based on this, image processing is performed so that the spherical image VSS1 and the captured image IM1 displayed on the image display device 104 move in correspondence with the moving direction, moving speed, and position of the moving destination of the hand of the user US. To execute.

The user US can rotate the spherical image VSS1 in any direction at any speed to any position by moving the hand to any position in any direction at any speed. it can. That is, the user US can rotate the spherical image VSS1 by the movement of the hand. The image processing unit 111 moves the captured image IM1 in correspondence with the rotation of the spherical image VSS1.

The image processing unit 111 determines to which position on the coordinates of the spherical image VSS1 the zenith ZE before the user US rotates the spherical image VSS1 moves as the user US rotates the spherical image VSS1. Can be determined. The image processing unit 111, based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS1, the spherical image before and after the user US rotates the spherical image VSS1. The amount of change in VSS1 is calculated.

The amount of change of the spherical image VSS1 is the amount of rotation (rotation angle) with the X axis as the rotation axis, the amount of rotation (rotation angle) with the Y axis as the rotation axis, and the Z axis with respect to the spherical image VSS1. The rotation amount (rotation angle) about the axis corresponds to the rotation amount (rotation angle) of the spherical image VSS1 combined. The image processing unit 111 stores the change amount of the spherical image VSS1 as the correction value CV1. That is, the correction value CV1 is calculated based on the rotation direction of the spherical image VSS1 and the movement amount or movement angle of the zenith ZE (the rotation angle of the spherical image VSS1).

The image processing unit 111 may store the coordinates of the zenith ZE on the spherical image VSS1 after the user US rotates the spherical image VSS1 as the correction value CV1. The image processing unit 111 may store the correction value CV1 in an internal memory or an external memory, or may output the correction value CV1 to the server 106 via the second communication unit 113 and the network. The server 106 associates the correction value CV1 with the image display device 104 or the user US and stores the correction value CV1 in an internal memory or an external memory.

The image processing unit 111 detects the horizontal direction of the captured image IM1. However, it is difficult for the image processing unit 111 to detect the horizontal direction depending on the captured image IM1, and the horizontal direction may be erroneously detected. Also, due to problems such as system synchronization, the horizontal or zenith designation may shift. If the horizontal direction of the captured image IM1 is incorrectly detected or the horizontal or zenith designation is deviated, the captured image IM1 displayed on the image display device 104 does not match the sense of gravity of the user US. I feel uncomfortable.

An example of the image adjustment method according to the first embodiment will be described using the flowcharts shown in FIGS. Specifically, an example of a method of adjusting the horizontalness of the captured image IM1 and the composite image CM1 will be described. The image display device 104 is mounted on the head of the user US, and the controller 105 is mounted on the hand of the user US. The captured image IM1 and the composite image CM1 are displayed on the image display device 104. At this point, the composite image CM1 may not be displayed.

When the user US determines that the captured image IM1 displayed on the image display device 104 is not horizontal, the user US operates the controller 105 in FIG. 3 to cause the image processing unit 111 to operate in step S101. , The setting screen is displayed on the image display device 104.

When the user US operates the controller 105 to select a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 111 causes the selected item to be displayed in step S102. The processing is shifted to the corresponding predetermined processing mode. When the horizontal adjustment item is selected, the image processing unit 111 shifts the processing to a processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM1.

The image processing unit 111 acquires the spherical image VSS1 generated by the image generation unit 112 and displays it on the image display device 104 in step S103. On the image display device 104, the captured image IM1, the spherical image VSS1, and the composite image CM1 are mixed and displayed. When the user US rotates the spherical image VSS1 so that the captured image IM1 is horizontal, the image processing unit 111 causes the image display device 104 to correspond to the rotation of the spherical image VSS1 in step S104. The displayed captured image IM1 is moved. The user US may perform the operation of rotating the spherical image VSS1 a plurality of times until the captured image IM1 becomes horizontal.

When the user US determines that the captured image IM1 has become horizontal, the image processing unit 111 ends the display of the spherical image VSS1 and causes the image display device 104 to display the spherical image VSS1 in step S105. Display the setting screen. By making the captured image IM1 displayed on the image display device 104 horizontal, the zenith of the camera 102 and the zenith of the user US can be matched.

When the user US operates the controller 105 and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 111 corresponds to the selected item in step S106. The processing shifts to a predetermined processing mode. When the end item is selected, the image processing unit 111 shifts the processing to a processing mode (end mode) for ending the horizontal adjustment.

In step S107, the image processing unit 111 acquires the rotation amount (rotation angle) before and after the spherical image VSS1 rotates as the change amount of the spherical image VSS1. In step S108, the image processing unit 111 stores the amount of change in the spherical image VSS1 as the correction value CV1 and ends the process.

If the composite image CM1 is not displayed in steps S101 to S108 of FIG. 3, the user US operates the controller 105 to cause the image adjusting device 110 (specifically, the image processing unit 111) to move to the image generating unit 112. It is possible to obtain the composite image CM1 generated in step (1), combine it with the captured image IM1 displayed on the image display device 104, and display the composite image CM1.

FIG. 5 schematically shows the relationship between the horizontal plane PHP1 of the captured image IM1 horizontally adjusted by the user US and the horizontal plane SHP1 of the composite image CM1 acquired by the image processing unit 111 from the image generation unit 112. If the horizontal plane PHP1 of the captured image IM1 and the horizontal plane SHP1 of the composite image CM1 are misaligned, the user US feels strange.

In FIG. 4, the image processing unit 111 determines in step S111 whether the captured image IM1 has been horizontally adjusted. The image processing unit 111 can determine whether or not the captured image IM1 is horizontally adjusted based on the history of rotation of the spherical image VSS1 or the correction value CV1.

When it is determined in step S111 that the captured image IM1 has been horizontally adjusted (YES), the image processing unit 111 acquires the stored correction value CV1 in step S112. In step S113, the image processing unit 111 rotates the coordinate axes of the composite image CM1 based on the correction value CV1 so that the horizontal plane SHP1 of the composite image CM1 matches the horizontal plane PHP1 of the captured image IM1. In step S114, the image processing unit 111 performs arithmetic processing on the composite image CM1 based on the corrected horizontal plane SHP1 to adjust the image.

In step S115, the image processing unit 111 combines the image-adjusted composite image CM1 with the captured image IM1 and displays the composite image CM1 on the image display device 104. When it is determined in step S111 that the captured image IM1 is not horizontally adjusted (NO), the image processing unit 111 shifts the processing to step S115.

In the image adjusting system 101, the image adjusting device 110, and the image adjusting method according to the first embodiment, the spherical image VSS1 is displayed on the image display device 104. According to the image adjusting system 101, the image adjusting device 110, and the image adjusting method according to the first embodiment, when the horizontal direction of the captured image IM1 is erroneously detected or the designation of the horizontal or the zenith ZE is deviated, The user US can adjust the captured image IM1 displayed on the image display device 104 to be horizontal by operating the controller 105 to rotate the spherical image VSS1.

Therefore, according to the image adjustment system 101, the image adjustment apparatus 110, and the image adjustment method according to the first embodiment, when the horizontal direction of the captured image IM1 is erroneously detected or the horizontal or zenith designation is deviated. The user US can easily correct the horizontal or zenith of the captured image IM1.

According to the image adjustment system 101, the image adjustment apparatus 110, and the image adjustment method according to the first embodiment, when the correction value CV1 is stored, the image processing unit 111 reads the correction value CV1 and the camera 102 captures the image. The captured image IM1 can be image-adjusted based on the correction value CV1 and displayed on the image display device 104.

In the image adjustment system 101, the image adjustment apparatus 110, and the image adjustment method according to the first embodiment, when the captured image IM1 is horizontally adjusted, the coordinate axes of the composite image CM1 are rotated based on the correction value CV1, and the composite image CM1 is rotated. The horizontal plane SHP1 of is matched with the horizontal plane PHP1 of the captured image IM1. According to the image adjustment system 101, the image adjustment device 110, and the image adjustment method according to the first embodiment, the captured image IM1 and the composite image CM1 are displayed on the image display device 104 without the user US feeling uncomfortable. can do.

[Second Embodiment]
A configuration example of the image adjustment system 201 according to the second embodiment will be described with reference to FIG. The image adjustment system 201 according to the second embodiment includes a camera 202, a first communication unit 203, an image display device 204, a controller 205, an image adjustment device 210, and a server 206. The image adjustment device 210 includes an image processing unit 211, an image generation unit 212, and a second communication unit 213.

The camera 202, the first communication unit 203, the image display device 204, the controller 205, the image adjustment device 210, and the server 206 of the second embodiment are the camera 102, the first communication unit 103, and the image display device 104 of the first embodiment. , The controller 105, the image adjustment apparatus 110, and the server 106. The image processing unit 211, the image generation unit 212, and the second communication unit 213 of the second embodiment correspond to the image processing unit 111, the image generation unit 112, and the second communication unit 113 of the first embodiment.

The image adjustment device 210 can acquire the captured image IM2 captured by the camera 202 via the first communication unit 203, the second communication unit 213, and the network. The server 206 acquires the captured image IM2 captured by the camera 202 via the first communication unit 203 and the network, and the image adjustment apparatus 210 acquires the captured image IM2 from the server 206 via the network and the second communication unit 213. Good.

The captured image IM2 acquired by the image adjustment device 210 is input to the image processing unit 211. The image processing unit 211 recognizes the horizontal direction in the captured image IM2 by analyzing the captured image IM2. The image processing unit 211 may recognize the vertical direction in the captured image IM2, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM2. The image adjustment device 210 corrects the distortion of the captured image IM2, executes image processing such as adjusting the horizontal of the captured image IM2, and outputs the image-processed captured image IM2 to the image display device 204.

The server 206 corrects the distortion of the captured image IM2 captured by the camera 202, performs image processing such as adjusting the horizontal of the captured image IM2, and outputs the image-processed captured image IM2 to the image adjustment apparatus 210. May be. The image display device 204 displays the captured image IM2 image-processed by the image adjustment device 210 or the server 206. The user US can mount the image display device 204 on his/her head to view the captured image IM2 image-processed by the image adjustment device 210 or the server 206.

The image display device 204 generates the posture information PN2 based on the direction in which the user US is facing, the posture of the user US, and the like while being mounted on the head of the user US. The image processing unit 211 acquires the posture information PN2 from the image display device 204. That is, the image processing unit 211 acquires the posture information PN2 based on the posture of the image display device 204. Based on the posture information PN2, the image processing unit 211 displays an image of a region corresponding to the direction in which the user US faces and the state of the user US such as the posture from the captured image IM2 captured by the camera 202. Display on the device 204.

The controller 205 generates the instruction information NN2 based on the state of the movement or posture of the hand of the user US in the state of being attached to the hand of the user US. The image processing unit 211 acquires the instruction information NN2 from the controller 205. The image processing unit 211 can change or adjust the captured image IM2 displayed on the image display device 204 based on the instruction information NN2.

The image generation unit 212 generates a composite image CM2 that is a CG that is composited with the captured image IM2. The composite image CM2 is an image of a character such as an avatar. The image generation unit 212 saves the composite image CM2 in an internal memory or an external memory. The image processing unit 211 may acquire the composite image CM2 generated by the image generation unit 212 and output it to the server 206 via the second communication unit 213 and the network. The server 206 stores the composite image CM2 in the built-in memory or an external memory in association with the image display device 204 or the user US.

The image processing unit 211 acquires the combined image CM2 from the image generation unit 212 or the server 206 based on the instruction information NN2, and combines the combined image CM2 with the captured image IM2 displayed on the image display device 204. Therefore, the image display device 204 can display the captured image IM2 captured by the camera 202 and the composite image CM2 acquired from the image generation unit 212 or the server 206 and combined with the captured image IM2.

The image generation unit 212 generates a spherical image VSS2 that is a virtual image formed by a spherical surface that is CG. The image generator 212 stores the spherical image VSS2 in the internal memory or an external memory. The image processing unit 211 acquires the spherical image VSS2 generated by the image generation unit 212 based on the instruction information NN2, and displays it on the image display device 204.

The spherical image VSS2 of the second embodiment corresponds to the spherical image VSS1 of the first embodiment. The user US moves the hand on which the controller 205 is attached to the position corresponding to the spherical image VSS2 displayed on the image display device 204, so that the hand of the user US contacts the spherical image VSS2. I feel like I'm doing it.

The controller 205 may have an actuator arranged in a portion that comes into contact with the hand of the user US. When the image processing unit 211 determines based on the instruction information NN2 that the hand of the user US has moved to the position corresponding to the spherical image VSS2, the image processing unit 211 operates the actuator. By applying pressure to the hand of the user US by the actuator, the user US can actually feel the touch of the hand with the spherical image VSS2.

When the user US moves the hand on which the controller 205 is attached in any direction while the spherical image VSS2 is displayed on the image display device 204, the image processing unit 211 displays the instruction information NN2. Based on this, image processing is performed so that the spherical image VSS2 and the captured image IM2 displayed on the image display device 204 move corresponding to the moving direction, moving speed, and position of the moving destination of the hand of the user US. To execute.

The user US can rotate the spherical image VSS2 in any direction at any speed to any position by moving the hand to any position in any direction at any speed. it can. That is, the user US can rotate the spherical image VSS2 by the movement of the hand. The image processing unit 211 moves the captured image IM2 in correspondence with the rotation of the spherical image VSS2.

The image processing unit 211 determines to which position on the coordinates of the spherical image VSS2 the zenith ZE before the user US rotates the spherical image VSS2 is moved by the rotation of the spherical image VSS2 by the user US. Can be determined. The image processing unit 211, based on the moving direction of the zenith ZE and the position of the moving destination on the coordinates of the spherical image VSS2, the spherical image before and after the user US rotates the spherical image VSS2. The amount of change in VSS2 is calculated. The amount of change of the spherical image VSS2 of the second embodiment corresponds to the amount of change of the spherical image VSS1 of the first embodiment.

The image processing unit 211 stores the variation amount of the spherical image VSS2 as the correction value CV2. The correction value CV2 of the second embodiment corresponds to the correction value CV1 of the first embodiment. The image processing unit 211 may store the coordinates on the spherical image VSS2 of the zenith ZE after the user US rotates the spherical image VSS2 as the correction value CV2. The image processing unit 211 may store the correction value CV2 in a built-in memory or an external memory, or may output the correction value CV2 to the server 206 via the second communication unit 213 and the network. The server 206 stores the correction value CV2 in the built-in memory or an external memory in association with the image display device 204 or the user US.

The image processing unit 211 detects the horizontal direction of the captured image IM2. However, it is difficult for the image processing unit 211 to detect the horizontal direction depending on the captured image IM2, and the horizontal direction may be erroneously detected. Also, due to problems such as system synchronization, the horizontal or zenith designation may shift. If the horizontal direction of the captured image IM2 is incorrectly detected or the horizontal or zenith designation is deviated, the captured image IM2 displayed on the image display device 204 does not match the sense of gravity of the user US. I feel uncomfortable.

An example of the image adjustment method according to the second embodiment will be described using the flowcharts shown in FIGS. 6A, 6B, and 7. Specifically, an example of a method of adjusting the horizontalness of the captured image IM2 and the composite image CM2 will be described. The image display device 204 is mounted on the head of the user US, and the controller 205 is mounted on the hand of the user US. The image display device 204 displays the captured image IM2 and the composite image CM2. The composite image CM2 does not have to be displayed.

When the user US determines that the captured image IM2 displayed on the image display device 204 is not horizontal, the user US operates the controller 205 in FIG. 6A to cause the image processing unit 211 to operate in step S201. The setting screen is displayed on the image display device 204.

When the user US operates the controller 205 and selects a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 211 sets the selected item in step S202. The processing is shifted to the corresponding predetermined processing mode. When the horizontal adjustment item is selected, the image processing unit 211 shifts the processing to a processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM2.

The image processing unit 211 acquires the spherical image VSS2 generated by the image generation unit 212 and displays it on the image display device 204 in step S203. On the image display device 204, the captured image IM2 and the spherical image VSS2 are mixed and displayed. In step S204, the image processing unit 211 determines whether the combined image CM2 is combined with the captured image IM2 and is displayed on the image display device 204.

When it is determined in step S204 that the composite image CM2 is combined with the captured image IM2 and is displayed on the image display device 204 (YES), the image processing unit 211 determines in step S205 that the composite image CM2 is the horizontal plane SHP2. A process of separating the captured image IM2 from the horizontal plane PHP2 is executed. FIG. 8 schematically shows a state in which the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 are separated.

If it is determined in step S204 that the combined image CM2 is combined with the captured image IM2 and is not displayed on the image display device 204 (NO), the image processing unit 211 advances the process to step S206.

The user US rotates the spherical image VSS2 so that the captured image IM2 becomes horizontal. Since the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 are separated, the composite image CM2 does not rotate even if the spherical image VSS2 is rotated. Therefore, the user US can easily correct the horizontal of the captured image IM2 without being confused by the rotation of the composite image CM2.

In step S205, the image processing unit 211 may hide the composite image CM2. By hiding the composite image CM2, the composite image CM2 does not get in the way, so that the user US can easily correct the horizontal of the captured image IM2.

In step S206, the image processing unit 211 moves the photographed image IM2 displayed on the image display device 204 in response to the rotation of the spherical image VSS2. The user US may perform the operation of rotating the spherical image VSS2 a plurality of times until the captured image IM2 becomes horizontal.

In FIG. 6B, when the user US determines that the captured image IM2 has become horizontal, the image processing unit 211 ends the display of the spherical image VSS2 in step S207 by operating the controller 205, and the image is displayed. The setting screen is displayed on the display device 204. By making the captured image IM2 displayed on the image display device 204 horizontal, the zenith of the camera 202 and the zenith of the user US can be matched.

When the user US operates the controller 205 and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 211 corresponds to the selected item in step S208. The processing shifts to a predetermined processing mode. When the end item is selected, the image processing unit 211 shifts the processing to a processing mode for ending the horizontal adjustment (horizontal adjustment end mode).

In step S209, the image processing unit 211 acquires the rotation amount (rotation angle) before and after the spherical image VSS2 rotates as the change amount of the spherical image VSS2. In step S210, the image processing unit 211 stores the variation amount of the spherical image VSS2 as the correction value CV2.

When the composite image CM2 is not displayed in steps S201 to S210 of FIGS. 6A and 6B, the user US operates the controller 205 to cause the image adjusting apparatus 210 (specifically, the image processing unit 211) to generate an image. The composite image CM2 generated by the unit 212 can be acquired and displayed on the image display device 204.

The image adjustment device 210 (specifically, the image processing unit 211) recognizes the horizontal direction in the captured image IM2 by analyzing the captured image IM2 captured by the camera 202. When the user US rotates the spherical image VSS2 to adjust the horizontalness of the captured image IM2, the image adjustment device 210 acquires the composite image CM2 generated by the image generation unit 212 and displays it on the image display device 204. The horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 do not match. If the horizontal plane PHP2 of the captured image IM2 and the horizontal plane SHP2 of the composite image CM2 are misaligned, the user US feels uncomfortable.

7, in step S211, the image processing unit 211 determines whether the composite image CM2 is combined with the captured image IM2 and is displayed on the image display device 204. When it is determined that the combined image CM2 is combined with the captured image IM2 and displayed on the image display device 204 (YES), the image processing unit 211 acquires the stored correction value CV2 in step S212. ..

In step S213, the image processing unit 211 rotates the coordinate axes of the composite image CM2 based on the correction value CV2 so that the horizontal plane SHP2 of the composite image CM2 matches the horizontal plane PHP2 of the captured image IM2 horizontally adjusted by the user US. Let In step S214, the image processing unit 211 performs arithmetic processing on the composite image CM2 based on the corrected horizontal plane SHP2 to adjust the image. FIG. 8 schematically shows a state in which the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 match each other.

When it is determined in step S211 that the captured image IM2 has not been horizontally adjusted by the user US (NO), the image processing unit 211 shifts the processing to step S215. In step S215, the image processing unit 211 combines the image-adjusted composite image CM2 with the captured image IM2 and displays the composite image CM2 on the image display device 204. The user US can view the captured image IM2 and the composite image CM2 which are displayed on the image display device 204 and whose horizontal levels match each other.

In the image adjustment system 201, the image adjustment device 210, and the image adjustment method according to the second embodiment, the spherical image VSS2 is displayed on the image display device 204. According to the image adjusting system 201, the image adjusting device 210, and the image adjusting method according to the second embodiment, when the horizontal direction of the captured image IM2 is erroneously detected or the designation of the horizontal or zenith ZE is deviated, The user US can adjust the captured image IM2 displayed on the image display device 204 to be horizontal by operating the controller 205 to rotate the spherical image VSS2.

Therefore, according to the image adjusting system 201, the image adjusting device 210, and the image adjusting method according to the second embodiment, when the horizontal direction of the captured image IM2 is erroneously detected or the horizontal or zenith designation is deviated. The user US can easily correct the horizontal or zenith of the captured image IM2.

According to the image adjusting system 201, the image adjusting apparatus 210, and the image adjusting method according to the second embodiment, when the correction value CV2 is stored, the image processing unit 211 reads the correction value CV2 and the camera 202 captures the image. The captured image IM2 can be image-adjusted based on the correction value CV2 and displayed on the image display device 204.

In the image adjusting system 201, the image adjusting apparatus 210, and the image adjusting method according to the second embodiment, the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 are separated, and the horizontal of the captured image IM2 is adjusted, The horizontal of the composite image CM2 is adjusted based on the adjustment result (correction value CV2). Since the horizontal plane SHP2 of the composite image CM2 and the horizontal plane PHP2 of the captured image IM2 are separated, the composite image CM2 does not rotate even if the user US rotates the spherical image VSS2. Therefore, the user US can easily correct the horizontal of the captured image IM2.

According to the image adjusting system 201, the image adjusting apparatus 210, and the image adjusting method according to the second embodiment, when the captured image IM2 is horizontally adjusted, the coordinate axes of the combined image CM2 are rotated based on the correction value CV2 to combine the images. The horizontal plane SHP2 of the image CM2 is matched with the horizontal plane PHP2 of the captured image IM2. Therefore, the user US can see the captured image IM2 and the composite image CM2 displayed on the image display device 204 without feeling a sense of discomfort.

[Third Embodiment]
A configuration example of the image adjustment system 301 according to the third embodiment will be described with reference to FIG. 9. The image adjustment system 301 according to the third embodiment includes a camera 302, a first communication unit 303, a plurality of image display devices 304, a plurality of controllers 305, a plurality of image adjustment devices 310, and a server 306. .. The image adjustment device 310 includes an image processing unit 311, an image generation unit 312, and a second communication unit 313.

The camera 302, the first communication unit 303, the image display device 304, the controller 305, the image adjustment device 310, and the server 306 of the third embodiment are the camera 102, the first communication unit 103, and the image display of the first embodiment, respectively. It corresponds to the device 104, the controller 105, the image adjustment device 110, and the server 106. The image processing unit 311, the image generation unit 312, and the second communication unit 313 of the third embodiment correspond to the image processing unit 111, the image generation unit 112, and the second communication unit 113 of the first embodiment, respectively. ..

In FIG. 9, two image display devices 304, a controller 305, and an image adjustment device 310 are shown for the sake of clarity. There may be three or more image display devices 304, controllers 305, and image adjustment devices 310. The image display device 304 used by the first user USa is the first image display device 304a, the controller 305 is the first controller 305a, and the image adjustment device 310 is the first image adjustment device 310a. The image display device 304 used by the second user USb is the second image display device 304b, the controller 305 is the second controller 305b, and the image adjustment device 310 is the second image adjustment device 310b.

The image processing unit 311 in the first image adjustment apparatus 310a is the image processing unit 311a, the image generation unit 312 is the image generation unit 312a, and the second communication unit 313 is the second communication unit 313a. In the second image adjusting device 310b, the image processing unit 311 is an image processing unit 311b, the image generation unit 312 is an image generation unit 312b, and the second communication unit 313 is a second communication unit 313b.

The first image adjustment device 310a can acquire the captured image IM3 captured by the camera 302 via the first communication unit 303, the second communication unit 313a of the first image adjustment device 310a, and the network. The second image adjustment device 310b can acquire the captured image IM3 captured by the camera 302 via the first communication unit 303, the second communication unit 313b of the second image adjustment device 310b, and the network.

The server 306 acquires the captured image IM3 captured by the camera 302 via the first communication unit 303 and the network, and the first image adjustment device 310a transmits the captured image IM3 from the server 306 to the network and the first image adjustment device 310a. The second image adjustment device 310b may obtain the captured image IM3 from the server 306 via the network and the second communication unit 313b of the second image adjustment device 310b.

In the server 306, the first user USa performs the login operation, or the first image display device 304a is connected to the network via the second communication unit 313a of the first image adjustment device 310a, and thus the first The user USa or the first image display device 304a can be specified. In the server 306, the second user USb performs the login operation, or the second image display device 304b is connected to the network via the second communication unit 313b of the second image adjustment device 310b, and thus the second The user USb or the second image display device 304b can be specified.

Hereinafter, processing in the first image adjustment device 310a, the first image display device 304a, and the first controller 305a will be described. The reference numerals in parentheses indicate the processes performed by the second image adjustment device 310b, the second image display device 304b, and the second controller 305b.

The captured image IM3 acquired by the image adjusting apparatus 310a (310b) is input to the image processing unit 311a (311b). The image processing unit 311a (311b) recognizes the horizontal direction in the captured image IM3 by analyzing the captured image IM3. The image processing unit 311a (311b) may recognize the vertical direction in the captured image IM3, or may recognize the horizontal direction and the vertical direction by analyzing the captured image IM3.

The image adjustment device 310a (310b) corrects the distortion of the captured image IM3 and executes image processing such as adjusting the horizontal of the captured image IM3, and the image processed image IM3 is displayed on the image display device 304a (304b). Output to.

The server 306 corrects the distortion of the captured image IM3 captured by the camera 302, executes image processing such as adjusting the horizontal of the captured image IM3, and the image-processed captured image IM3 is adjusted by the image adjusting device 310a (310b). May be output to. The image display device 304a (304b) displays the captured image IM3 image-processed by the image adjustment device 310a (310b) or the server 306.

The user USa (USb) can view the captured image IM3 image-processed by the image adjustment device 310a (310b) or the server 306 by mounting the image display device 304a (304b) on its head.

The image display device 304a (304b) is in a state in which the user USa (USb) is facing, the posture of the user USa (USb), and the like in a state where the image display device 304a (304b) is mounted on the head of the user USa (USb). The posture information PN3a (PN3b) is generated based on The image processing unit 311a (311b) acquires the posture information PN3a (PN3b) from the image display device 304a (304b). That is, the image processing unit 311a (311b) acquires the posture information PN3a (PN3b) based on the posture of the image display device 304a (304b).

The image processing unit 311a (311b), based on the posture information PN3a (PN3b), from the captured image IM3 captured by the camera 302, the direction in which the user USa (USb) is facing and the posture of the user USa (USb). The image of the area corresponding to the above state is displayed on the image display device 304a (304b).

The controller 305a (305b) generates the instruction information NN3a (NN3b) based on the state of the movement or posture of the hand of the user USa (USb) while being attached to the hand of the user USa (USb). .. The image processing unit 311a (311b) acquires the instruction information NN3a (NN3b) from the controller 305a (305b). The image processing unit 311a (311b) can change or adjust the captured image IM3 displayed on the image display device 304a (304b) based on the instruction information NN3a (NN3b).

The image generation unit 312a (312b) generates a composite image CM3b (CM3a) that is a CG to be combined with the captured image IM3. The composite image CM3a (CM3b) is an image of a character such as an avatar. The synthetic image CM3a generated by the second image generating unit 312b is the avatar of the first user USa, and the synthetic image CM3b generated by the first image generating unit 312a is the avatar of the second user USb. When the user USa (USb) operates the controller 305a (305b), the image generation unit 312a (312b) generates a composite image CM3b (CM3a) which is an avatar of the user USb (USa), and the built-in memory or Save to external memory.

The image processing unit 311a (311b) acquires the composite image CM3b (CM3a) generated by the image generation unit 312a (312b) and outputs it to the server 306 via the second communication unit 313a (313b) and the network. Good.

The image generation unit 312a (312b) may generate a composite image CM3a (CM3b) that is an avatar of the user USa (USb) by operating the controller 305a (305b) by the user USa (USb). The image processing unit 311a (311b) acquires the composite image CM3a (CM3b) generated by the image generation unit 312a (312b) and outputs it to the server 306 via the second communication unit 313a (313b) and the network. Good. That is, the first user USa may set the avatar of the second user USb or may set his own avatar. Further, the second user USb may set the avatar of the first user USa or may set his own avatar.

The server 306 stores the composite image CM3a (CM3b) in the internal memory or the external memory in association with the user USa (USb). The server 306 may automatically set the composite image CM3a (CM3b) in association with the user USa (USb), and save the composite image CM3a (CM3b) in the built-in memory or an external memory.

The image processing unit 311a (311b) acquires the composite image CM3b (CM3a) from the image generation unit 312a (312b) or the server 306 based on the instruction information NN3a (NN3b) and displays it on the image display device 304a (304b). The synthesized image CM3b (CM3a) is synthesized with the captured image IM3. Therefore, the image display device 304a (304b) displays the captured image IM3 captured by the camera 302 and the composite image CM3b (CM3a) that is obtained from the image generation unit 312a (312b) or the server 306 and is combined with the captured image IM3. Can be displayed.

The image generation unit 312a (312b) generates a spherical image VSS3a (VSS3b) that is a virtual image formed by a spherical surface that is CG. The image generation unit 312a (312b) stores the spherical image VSS3a (VSS3b) in a built-in memory or an external memory. The image processing unit 311a (311b) acquires the spherical image VSS3a (VSS3b) generated by the image generation unit 312a (312b) based on the instruction information NN3a (NN3b) and displays it on the image display device 304a (304b). To do.

The spherical image VSS3a (VSS3b) of the third embodiment corresponds to the spherical image VSS1 of the first embodiment. The user USa (USb) moves the hand on which the controller 305a (305b) is attached to a position corresponding to the spherical image VSS3a (VSS3b) displayed on the image display device 304a (304b), It is felt that the hand of the user USa (USb) is in contact with the spherical image VSS3a (VSS3b).

In the controller 305a (305b), an actuator may be arranged in a portion that comes into contact with the hand of the user USa (USb). When the image processing unit 311a (311b) determines that the hand of the user USa (USb) has moved to the position corresponding to the spherical image VSS3a (VSS3b) based on the instruction information NN3a (NN3b), the actuator is activated. Let When the actuator applies pressure to the hand of the user USa (USb), the user USa (USb) can actually feel the touch of the hand with the spherical image VSS3a (VSS3b).

In the state where the spherical image VSS3a (VSS3b) is displayed on the image display device 304a (304b), the user USa (USb) moves the hand on which the controller 305a (305b) is attached in any direction. In this case, the image processing unit 311a (311b) determines that the spherical image VSS3a (VSS3b) and the captured image IM3 displayed on the image display device 304a (304b) are the user USa based on the instruction information NN3a (NN3b). The image processing is executed so as to move in accordance with the moving direction, the moving speed, and the position of the moving destination of (USb).

The user USa (USb) moves his or her hand to any position in any direction at any speed, thereby moving the spherical image VSS3a (VSS3b) in any direction at any speed at any speed. Can be rotated into position. That is, the user USa (USb) can rotate the spherical image VSS3a (VSS3b) by the movement of the hand. The image processing unit 311a (311b) moves the captured image IM3 in correspondence with the rotation of the spherical image VSS3a (VSS3b).

The image processing unit 311a (311b) causes the zenith ZE before the user USa (USb) rotates the spherical image VSS3a (VSS3b) and the user USa (USb) rotates the spherical image VSS3a (VSS3b). The position on the coordinate of the spherical image VSS3a (VSS3b) can be determined by the.

The image processing unit 311a (311b) rotates the spherical image VSS3a (VSS3b) by the user USa (USb) based on the movement direction and the position of the movement destination of the zenith ZE on the coordinates of the spherical image VSS3a (VSS3b). The amount of change in the spherical image VSS3a (VSS3b) before and after rotation is calculated. The change amount of the spherical image VSS3a (VSS3b) of the third embodiment corresponds to the change amount of the spherical image VSS1 of the first embodiment.

The image processing unit 311a (311b) stores the amount of change in the spherical image VSS3a (VSS3b) as a correction value CV3a (CV3b). The correction value CV3a (CV3b) of the third embodiment corresponds to the correction value CV1 of the first embodiment. The image processing unit 311a (311b) uses the coordinates on the spherical image VSS3a (VSS3b) of the zenith ZE after the user USa (USb) rotates the spherical image VSS3a (VSS3b) as the correction value CV3a (CV3b). You may save.

The image processing unit 311a (311b) may store the correction value CV3a (CV3b) in an internal memory or an external memory, or may store the correction value CV3a (CV3b) via the second communication unit 313a (313b) and a network. May be output to the server 306. The server 306 stores the correction value CV3a (CV3b) in the built-in memory or an external memory in association with the image display device 304a (304b) or the user USa (USb).

The image adjustment system 301 synthesizes a composite image CM3b (CM3a), which is an avatar of another user USb (USa), with the captured image IM3 on the image display device 304a (304b) worn by the user USa (USb). Can be displayed.

When the user USa (USb) looks at the captured image IM3 taken by the camera 302, the user USa (USb) feels as if he or she is looking at the surroundings from the position of the camera 302. In such a case, on the image display device 304a (304b) viewed by the user USa (USb), a composite image CM3b (an avatar of another user USb (USa) is displayed at the position of the user USa (USb). Since CM3a) is displayed, the user USa (USb) feels strange. When there are a plurality of other users US, the avatars of the plurality of other users USb, USc,... Are overlapped and displayed at the position of the user USa.

The positional relationship among a plurality of users USa, USb,... Is preset in the server 306. In other words, the server 306 stores position setting information indicating the positional relationship among the plurality of users USa, USb,.... The positional relationship among the plurality of users USa, USb,... May be automatically set by the server 306, or may be set by the plurality of users USa, USb,.

FIG. 10 shows an example of the positional relationship between the first user USa and the synthetic image CM3b which is the avatar of the second user USb when the first user USa is looking at the first image display device 304a. .. FIG. 11 shows an example of the positional relationship between the second user USb and the composite image CM3a which is the avatar of the first user USa when the second user USb is looking at the second image display device 304b. ..

Black dots shown in each of FIGS. 10 and 11 indicate the position of the camera 302. The arrow shown in each of FIGS. 10 and 11 indicates the moving direction of the camera 302. That is, FIG. 10 shows a state in which the first user USa faces the moving direction of the camera 302, and FIG. 11 shows a state in which the second user USb faces the moving direction of the camera 302. There is. Therefore, in each of FIGS. 10 and 11, the first user USa and the second user USb see the same captured image IM3.

In such a case, if the composite image CM3b is displayed at a position away from the first user USa or the composite image CM3a is displayed at a position away from the second user USb, the first user USa and the second user Both users USb feel discomfort. Therefore, it is desirable to display the composite image CM3b (CM3a) at a position within a predetermined range RG from the position of the user USa (USb) (camera 302).

As shown in FIG. 10, in the server 306, for example, in the first user USa or the first image display device 304a, the composite image CM3b is arranged side by side on the right side of the first user USa. The positional relationship between the first user USa and the second user USb is set. Further, as shown in FIG. 11, in the server 306, for example, in the second user USb or the second image display device 304b, the composite image CM3a is arranged side by side on the left side of the second user USb. , The positional relationship between the first user USa and the second user USb is set.

That is, in each of FIGS. 10 and 11, when the moving direction of the camera 302 is the front direction of the first user USa and the second user USb, the second user USb is arranged on the right side of the first user USa. The positional relationship between the first user USa and the second user USb is set so as to be arranged in. Therefore, the first user USa always sees the second user USb on its right side, and the second user USb always sees the first user USa on its left side. When the camera 302, which is a 360-degree camera, is stationary, there is no front-back or left-right reference, so it is desirable to determine the positional relationship between the first user USa and the second user USb at the time of initial setting.

An example of the image adjusting method according to the third embodiment will be described with reference to the flowcharts shown in FIGS. Specifically, an example of a method of compositing the composite image CM3 with the captured image IM3 will be described. Hereinafter, processing in the first image adjustment device 310a, the first image display device 304a, and the first controller 305a will be described. The reference numerals in parentheses indicate the processes performed by the second image adjustment device 310b, the second image display device 304b, and the second controller 305b.

The image display device 304a (304b) is attached to the head of the user USa (USb), and the controller 305a (305b) is attached to the hand of the user USa (USb). The captured image IM3 is displayed on the image display device 304a (304b).

When the user USa (USb) determines that the captured image IM3 displayed on the image display device 304a (304b) is not horizontal, the user USa (USb) operates the controller 305a (305b) in FIG. Accordingly, the image processing unit 311a (311b) causes the image display device 304a (304b) to display the setting screen in step S301.

When the user USa (USb) operates the controller 305a (305b) to select a predetermined item (for example, a horizontal adjustment item) displayed on the setting screen, the image processing unit 311a (311b) causes the step In S302, the processing is shifted to a predetermined processing mode corresponding to the selected item. When the horizontal adjustment item is selected, the image processing unit 311a (311b) shifts the processing to a processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM3.

In step S303, the image processing unit 311a (311b) acquires the spherical image VSS3a (VSS3b) generated by the image generation unit 312a (312b) and displays it on the image display device 304a (304b). The captured image IM3 and the spherical image VSS3a (VSS3b) are mixed and displayed on the image display device 304a (304b).

The user USa (USb) rotates the spherical image VSS3a (VSS3b) so that the captured image IM3 becomes horizontal, so that the image processing unit 311a (311b) causes the spherical image VSS3a (VSS3b) in step S304. The captured image IM3 displayed on the image display device 304a (304b) is moved in correspondence with the rotation of. The user USa (USb) may perform the operation of rotating the spherical image VSS3a (VSS3b) multiple times until the captured image IM3 becomes horizontal.

The user USa (USb) operates the controller 305a (305b) when determining that the captured image IM3 has become horizontal, so that the image processing unit 311a (311b) causes the spherical image VSS3a (VSS3b) in step S305. ) Is ended and the setting screen is displayed on the image display device 304a (304b). By making the captured image IM3 displayed on the image display device 304a (304b) horizontal, the zenith of the camera 302 and the zenith of the user USa (USb) can be matched.

When the user USa (USb) operates the controller 305a (305b) and selects a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 311a (311b) causes the image processing unit 311a (311b) to perform step S306. At, the processing shifts to a predetermined processing mode corresponding to the selected item. When the end item is selected, the image processing unit 311a (311b) shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S307, the image processing unit 311a (311b) acquires a rotation amount (rotation angle) before and after the spherical image VSS3a (VSS3b) rotates as a variation amount of the spherical image VSS3a (VSS3b). .. In step S308, the image processing unit 311a (311b) saves the amount of change in the spherical image VSS3a (VSS3b) as the correction value CV3a (CV3b), and ends the process.

A case where the combined image CM3b, which is the avatar of the second user USb, is combined with the captured image IM3 and displayed on the first image display device 304a viewed by the first user USa using the flowchart shown in FIG. explain. It should be noted that the reference numerals in parentheses indicate the case where the second image display device 304b viewed by the second user USb displays the combined image CM3a, which is the avatar of the first user USa, in combination with the captured image IM3. ing.

In FIG. 13, when the user USa (USb) operates the controller 305a (305b), the image processing unit 311a (311b) causes the image display device 304a (304b) to display the setting screen in step S311. For example, an avatar selection item for selecting an avatar of another user US is displayed on the setting screen.

The user USa (USb) selects the user USb (USa) as another user US, for example. In step S312, the controller 305a (305b) generates the instruction information NN3a (NN3b) according to the selection content of the other user USb (USa) and outputs it to the image processing unit 311a (311b).

The image processing unit 311a (311b) acquires the composite image CM3b (CM3a) based on the instruction information NN3a (NN3b) in step S313. When the composite image CM3b (CM3a) is stored in the server 306, the image processing unit 311a (311b) acquires the composite image CM3b (CM3a) from the server 306 via the second communication unit 313a (313b) and the network. To do.

The image processing unit 311a (311b) determines in step S314 whether or not the camera 302 is moving. When it is determined that the camera 302 is moving (YES) in step S314, the image processing unit 311a (311b) recognizes the moving direction of the camera 302 in step S315. The image processing unit 311a (311b) can determine whether the camera 302 is moving based on the captured image IM3 and recognize the moving direction of the camera 302.

In step S316, the image processing unit 311a (311b) synthesizes the synthesized image CM3b (CM3a) with respect to the moving direction of the camera 302, for example, on the right side (left side) of the user USa (USb). The image CM3b (CM3a) is combined with the captured image IM3 and displayed on the image display device 304a (304b).

When it is determined in step S314 that the camera 302 is not moving (NO), the image processing unit 311a (311b) determines that the camera 302 is stationary, and in step S316, the initial setting is performed. Based on the determined positional relationship between the first user USa and the second user USb, the composite image CM3b (CM3a) is composited with the captured image IM3 and displayed on the image display device 304a (304b).

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method according to the third embodiment, the spherical image VSS3 is displayed on the image display device 304. According to the image adjusting system 301, the image adjusting apparatus 310, and the image adjusting method according to the third embodiment, when the horizontal direction of the captured image IM3 is erroneously detected or the designation of the horizontal or zenith ZE is deviated, The user US can adjust the captured image IM3 displayed on the image display device 304 to be horizontal by operating the controller 305 to rotate the spherical image VSS3.

Therefore, according to the image adjusting system 301, the image adjusting apparatus 310, and the image adjusting method according to the third embodiment, when the horizontal direction of the captured image IM3 is erroneously detected or the horizontal or zenith designation is deviated. The user US can easily correct the horizontal or zenith of the captured image IM3.

According to the image adjustment system 301, the image adjustment apparatus 310, and the image adjustment method according to the third embodiment, when the correction value CV3 is stored, the image processing unit 311 reads the correction value CV3 and the camera 302 captures the image. The captured image IM3 can be image-adjusted based on the correction value CV3 and displayed on the image display device 304.

In the image adjustment system 301, the image adjustment device 310, and the image adjustment method according to the third embodiment, the positional relationship among a plurality of users US is preset. As a result, the composite image CM3b (CM3a), which is the avatar of another user USb (USa), is added to the captured image IM3 displayed on the image display device 304a (304b) based on the preset positional relationship. Can be combined and displayed.

According to the image adjustment system 301, the image adjustment device 310, and the image adjustment method according to the third embodiment, when the user USa (USb) views the captured image IM3 captured by the camera 302, the user USa( Since the composite image CM3b (CM3a) that is the avatar of the other user USb (USa) is displayed in the vicinity of (USb), the user USa (USb) does not feel uncomfortable and the other user USb (USa) does not feel uncomfortable. It is possible to feel as if the user is looking at the same captured image IM3 as shown in FIG.

[Fourth Embodiment]
A configuration example of the image adjustment system 401 according to the fourth embodiment will be described with reference to FIG. The image adjustment system 401 according to the fourth embodiment includes a plurality of cameras 402, a plurality of first communication units 403, a plurality of image display devices 404, a plurality of controllers 405, a plurality of image adjustment devices 410, and a server. And 406. The image adjustment device 410 includes an image processing unit 411, an image generation unit 412, and a second communication unit 413.

The camera 402, the communication unit 403, and the image display device 404 of the fourth embodiment correspond to the camera 102, the first communication unit 103, and the image display device 104 of the first embodiment, respectively. The controller 405, the image adjustment device 410, and the server 406 of the fourth embodiment correspond to the controller 105, the image adjustment device 110, and the server 106 of the first embodiment, respectively. The image processing unit 411, the image generation unit 412, and the second communication unit 413 of the fourth embodiment correspond to the image processing unit 111, the image generation unit 112, and the second communication unit 113 of the first embodiment, respectively. ..

In FIG. 14, two cameras 402, a first communication unit 403, an image display device 404, a controller 405, and an image adjustment device 410 are shown for the sake of clarity. The camera 402, the first communication unit 403, the image display device 404, the controller 405, and the image adjustment device 410 may be three or more. The image display device 404, the controller 405, and the image adjustment device 410 used by the first user USa are referred to as a first image display device 404a, a first controller 405a, and a first image adjustment device 410a. The image display device 404, the controller 405, and the image adjustment device 410 used by the second user USb are referred to as the second image display device 404b, the second controller 405b, and the second image adjustment device 410b.

The image processing unit 411, the image generation unit 412, and the second communication unit 413 in the second image adjustment device 410a are the image processing unit 411a, the image generation unit 412a, and the second communication unit 413a. Further, the image processing unit 411, the image generation unit 412, and the second communication unit 413 in the image adjustment device 410b are referred to as the image processing unit 411b, the image generation unit 412b, and the second communication unit 413b. The first image adjustment device 410a is connected to the network via the second communication unit 413a of the first image adjustment device 410a. The second image adjustment device 410b is connected to the network via the second communication unit 413b of the second image adjustment device 410b.

One of the two cameras 402 is the first camera 4021 and the other is the second camera 4022. One of the two first communication units 403 is the first communication unit 4031 and the other is the first communication unit 4032. The first camera 4021 is connected to the network via the first communication unit 4031, and the second camera 4022 is connected to the network via the first communication unit 4032. The first camera 4021 and the second camera 4022 may be connected to the network via the common first communication unit 4031 or first communication unit 4032. The server 406 is connected to the network.

The first camera 4021 and the second camera 4022 can move independently. The first camera 4021 and the second camera 4022 have a GPS function. The server 406 can continuously acquire the positions (current positions) of the first camera 4021 and the second camera 4022 via the network, the first communication unit 4031 on one side, and the first communication unit 4032 on the other side. ..

The first image adjustment device 410a can acquire the positions of the first camera 4021 and the second camera 4022 from the server 406 via the second communication unit 413a of the first image adjustment device 410a and the network. The second image adjustment device 410b can acquire the positions of the first camera 4021 and the second camera 4022 from the server 406 via the second communication unit 413b of the second image adjustment device 410b and the network.

The first image adjustment device 410a can acquire the captured image IM41 taken by the first camera 4021 via the one first communication unit 4031, the second communication unit 413a of the first image adjustment device 410a, and the network. it can. The second image adjustment device 410b can acquire the captured image IM41 captured by the first camera 4021 via the one first communication unit 4031, the second communication unit 413b of the second image adjustment device 410b, and the network. it can.

The first image adjustment device 410a can acquire the captured image IM42 captured by the second camera 4022 via the other first communication unit 4032, the second communication unit 413a of the first image adjustment device 410a, and the network. it can. The second image adjustment device 410b can acquire the captured image IM42 captured by the second camera 4022 via the other first communication unit 4032, the second communication unit 413b of the second image adjustment device 410b, and the network. it can. FIG. 14 illustrates a case where the first image adjustment device 410a acquires the captured image IM41 captured by the first camera 4021 and the second image adjustment device 410b acquires the captured image IM42 captured by the second camera 4022. ..

The server 406 acquires the captured image IM41 captured by the first camera 4021 and the captured image IM42 captured by the second camera 4022 via the first communication unit 4031 on one side and the first communication unit 4032 on the other side and the network. Good. In this case, the first image adjustment apparatus 410a uses the captured image IM41 captured by the first camera 4021 and the captured image IM42 captured by the second camera 4022 from the server 406 to the network and the second communication unit 413a of the first image adjustment apparatus 410a. To get through. The second image adjustment device 410b receives the captured image IM41 captured by the first camera 4021 and the captured image IM42 captured by the second camera 4022 from the server 406 via the network and the second communication unit 413b of the second image adjustment device 410b. get.

In the server 406, the first user USa performs the login operation, or the first image display device 404a is connected to the network via the second communication unit 413a, so that the first user USa or the first image is displayed. The display device 404a can be specified. In the server 406, the second user USb performs the login operation, or the second image display device 404b is connected to the network via the second communication unit 413b, so that the second user USb or the second image is displayed. The display device 404b can be specified.

The server 406 can recognize which captured image is displayed on which image display device. For example, as shown in FIG. 14, the captured image IM41 captured by the first camera 4021 is displayed on the first image display device 404a, and the captured image IM42 captured by the second camera 4022 is displayed on the second image display device 404b. In this case, the server 406 displays the captured image IM41 captured by the first camera 4021 on the first image display device 404a and displays the captured image IM42 captured by the second camera 4022 on the second image display device 404b. Recognize that In other words, the server 406 recognizes that the first user USa views the captured image IM41 captured by the first camera 4021, and the second user USb views the captured image IM42 captured by the second camera 4022. Can recognize that.

The first image adjustment device 410a detects the current positions of the first camera 4021 that is capturing the captured image IM41 and the second camera 4022 that is capturing the captured image IM42, and the second image adjustment device 410b from the server 406. It can be recognized that the captured image IM42 captured by the second camera 4022 is acquired. The second image adjustment device 410b detects the current positions of the first camera 4021 that is capturing the captured image IM41 and the second camera 4022 that is capturing the captured image IM42, and the first image adjustment device 410a from the server 406. It can be recognized that the captured image IM41 captured by the first camera 4021 is acquired.

The captured image IM41 (IM42) acquired by the first image adjustment device 410a is input to the image processing unit 411a. The image processing unit 411a analyzes the captured image IM41 (IM42) to recognize the horizontal direction in the captured image IM41 (IM42). The image processing unit 411a may recognize the vertical direction in the captured image IM41 (IM42) by analyzing the captured image IM41 (IM42), or may recognize the horizontal direction and the vertical direction.

The first image adjustment device 410a corrects the distortion of the captured image IM41 (IM42), performs image processing such as adjusting the horizontal of the captured image IM41 (IM42), and the like, and the image-processed captured image IM41 (IM42). Is output to the first image display device 404a.

The captured image IM42 (IM41) acquired by the second image adjustment device 410b is input to the image processing unit 411b. The image processing unit 411b recognizes the horizontal direction in the captured image IM42 (IM41) by analyzing the captured image IM42 (IM41). The image processing unit 411b may recognize the vertical direction in the captured image IM42 (IM41) by analyzing the captured image IM42 (IM41), or may recognize the horizontal direction and the vertical direction.

The second image adjustment device 410b corrects the distortion of the captured image IM42 (IM41), executes image processing such as adjusting the horizontal of the captured image IM42 (IM41), and the image processed captured image IM42 (IM41). Is output to the second image display device 404b.

The server 406 corrects the distortion between the captured image IM41 captured by the first camera 4021 and the captured image IM42 captured by the second camera 4022, and the captured image IM41 captured by the first camera 4021 and the captured image captured by the second camera 4022. Image processing such as horizontal adjustment of IM42 is executed, and the image-processed captured image IM41 and captured image IM42 may be output to the first image adjustment device 410a or the second image adjustment device 410b. You may.

The first image display device 404a displays a captured image IM41 (IM42) that has been image-processed by the first image adjustment device 410a or the server 406. The first user USa can view the captured image IM41 (IM42) image-processed by the first image adjusting device 410a or the server 406 by mounting the first image display device 404a on the head.

The second image display device 404b displays the captured image IM42 (IM41) that has been image-processed by the second image adjustment device 410b or the server 406. The second user USb can view the captured image IM42 (IM41) image-processed by the second image adjustment device 410b or the server 406 by mounting the second image display device 404b on its head.

The image display device 404a (404b) is in a state in which the user USa (USb) is facing, the posture of the user USa (USb), and the like in a state where the image display device 404a (404b) is mounted on the head of the user USa (USb). The posture information PN4a (PN4b) is generated based on The image processing unit 411a (411b) acquires the posture information PN4a (PN4b) from the image display device 404a (404b). That is, the image processing unit 411a (411b) acquires the posture information PN4a (PN4b) based on the posture of the image display device 404a (404b).

The image processing unit 411a of the first image adjustment device 410a is directed toward the first user USa from the captured image IM41 captured by the first camera 4021 or the captured image IM42 captured by the second camera 4022 based on the posture information PN4a. The image of the area corresponding to the direction in which the user is moving and the state such as the posture of the first user USa is displayed on the first image display device 404a. The image processing unit 411b of the second image adjustment device 410b faces the second user USb from the captured image IM41 captured by the first camera 4021 or the captured image IM42 captured by the second camera 4022 based on the posture information PN4b. The second image display device 404b displays an image of a region corresponding to the direction in which the user is moving and the posture of the second user USb.

The controller 405a (405b) generates the instruction information NN4a (NN4b) based on the state of the movement or posture of the hand of the user USa (USb) in a state where the controller 405a (405b) is attached to the hand of the user USa (USb). .. The image processing unit 411a (411b) acquires the instruction information NN4a (NN4b) from the controller 405a (405b).

The image processing unit 411a of the first image adjustment device 410a can change or adjust the captured image IM41 or the captured image IM42 displayed on the first image display device 404a based on the instruction information NN4a. The image processing unit 411b of the second image adjustment device 410b can change or adjust the captured image IM41 or the captured image IM42 displayed on the second image display device 404b based on the instruction information NN4b.

The image generation unit 412a of the first image adjustment device 410a generates a combined image CM4b that is a CG combined with the captured image IM41 (IM42). The image generation unit 412b of the second image adjustment device 410b generates a combined image CM4a that is a CG combined with the captured image IM42 (IM41). The composite image CM4a (CM4b) is an image of a character such as an avatar. The composite image CM4a is the avatar of the first user USa, and the composite image CM4b is the avatar of the second user USb.

When the first user USa operates the first controller 405a, the image generation unit 412a of the first image adjustment device 410a generates the composite image CM4b that is the avatar of the second user USb, and then the built-in memory or the external device. Save to memory. When the second user USb operates the second controller 405b, the image generation unit 412b of the second image adjustment device 410b generates the composite image CM4a which is the avatar of the first user USa, and is stored in the internal memory or the external device. Save to memory.

The image processing unit 411a (411b) acquires the composite image CM4b (CM4a) generated by the image generation unit 412a (412b) and outputs it to the server 406 via the second communication unit 413a (413b) and the network. Good.

The user USa (USb) may operate the controller 405a (405b) to cause the image generation unit 412a (412b) to generate a composite image CM4a (CM4b) that is an avatar of the user USa (USb). The image processing unit 411a (411b) acquires the composite image CM4a (CM4b) generated by the image generation unit 412a (412b) and outputs it to the server 406 via the second communication unit 413a (413b) and the network. Good. That is, the first user USa may set the avatar of the second user USb or may set his own avatar. Further, the second user USb may set the avatar of the first user USa or may set his own avatar.

The server 406 stores the composite image CM4a (CM4b) in the internal memory or the external memory in association with the user USa (USb). The server 406 may automatically set the composite image CM4a (CM4b) in association with the user USa (USb), and save the composite image CM4a (CM4b) in the internal memory or an external memory. The image processing unit 411a (411b) acquires the composite image CM4b (CM4a) from the image generation unit 412a (412b) or the server 306 based on the instruction information NN4a (NN4b) and displays it on the image display device 404a (404b). be able to.

The image generation unit 412a (412b) generates a spherical image VSS4a (VSS4b) that is a virtual image formed by a spherical surface that is CG. The image generation unit 412a (412b) stores the spherical image VSS4a (VSS4b) in a built-in memory or an external memory. The image processing unit 411a (411b) acquires the spherical image VSS4a (VSS4b) generated by the image generation unit 412a (412b) based on the instruction information NN4a (NN4b) and displays it on the image display device 404a (404b). To do.

The spherical image VSS4a (VSS4b) of the fourth embodiment corresponds to the spherical image VSS1 of the first embodiment. The user USa (USb) moves the hand on which the controller 405a (405b) is attached to a position corresponding to the spherical image VSS4a (VSS4b) displayed on the image display device 404a (404b), It feels as if the hand of the user USa (USb) is in contact with the spherical image VSS4a (VSS4b).

In the controller 405a (405b), an actuator may be arranged in a portion that comes into contact with the hand of the user USa (USb). When the image processing unit 411a (411b) determines that the hand of the user USa (USb) has moved to the position corresponding to the spherical image VSS4a (VSS4b) based on the instruction information NN4a (NN4b), the actuator is activated. Let When the actuator applies pressure to the hand of the user USa (USb), the user USa (USb) can actually feel the touch of the hand with the spherical image VSS4a (VSS4b).

When the first user USa moves the hand on which the first controller 405a is attached in any direction while the spherical image VSS4a is displayed on the first image display device 404a, the first image adjustment The image processing unit 411a of the device 410a determines that the spherical image VSS4a and the captured image IM41 or the captured image IM42 displayed on the first image display device 404a are the hands of the first user USa based on the instruction information NN4a. The image processing is executed so as to move in accordance with the moving direction, the moving speed, and the position of the moving destination.

When the second user USb moves the hand on which the second controller 405b is attached in any direction while the spherical image VSS4b is displayed on the second image display device 404b, the second image adjustment The image processing unit 411b of the device 410b determines that the spherical image VSS4b and the captured image IM41 or the captured image IM42 displayed on the second image display device 404b are the hands of the second user USb based on the instruction information NN4b. The image processing is executed so as to move in accordance with the moving direction, the moving speed, and the position of the moving destination.

The user USa (USb) moves his or her hand to any position in any direction at any speed, thereby moving the spherical image VSS4a (VSS4b) in any direction at any speed at any speed. Can be rotated into position. That is, the user USa (USb) can rotate the spherical image VSS4a (VSS4b) by the movement of the hand. The image processing unit 411a (411b) moves the captured image IM41 (IM42) in correspondence with the rotation of the spherical image VSS4a (VSS4b).

The image processing unit 411a (411b) causes the zenith ZE before the user USa (USb) rotates the spherical image VSS4a (VSS4b) and the user USa (USb) rotates the spherical image VSS4a (VSS4b). It is possible to determine which position on the coordinates of the spherical image VSS4a (VSS4b) is moved by.

The image processing unit 411a (411b) rotates the spherical image VSS4a (VSS4b) by the user USa (USb) based on the movement direction and the position of the movement destination of the zenith ZE on the coordinates of the spherical image VSS4a (VSS4b). The amount of change in the spherical image VSS4a (VSS4b) before and after rotation is calculated. The amount of change of the spherical image VSS4a (VSS4b) of the fourth embodiment corresponds to the amount of change of the spherical image VSS1 of the first embodiment.

The image processing unit 411a (411b) stores the amount of change in the spherical image VSS4a (VSS4b) as a correction value CV4a (CV4b). The correction value CV4a (CV4b) of the fourth embodiment corresponds to the correction value CV1 of the first embodiment. The image processing unit 411a (411b) uses the coordinates on the spherical image VSS4a (VSS4b) of the zenith ZE after the user USa (USb) rotates the spherical image VSS4a (VSS4b) as the correction value CV4a (CV4b). You may save.

The image processing unit 411a (411b) may store the correction value CV4a (CV4b) in an internal memory or an external memory, or may store the correction value CV4a (CV4b) via the second communication unit 413a (413b) and a network. May be output to the server 406. The server 406 stores the correction value CV4a (CV4b) in the built-in memory or an external memory in association with the image display device 404a (404b) or the user USa (USb).

The image adjustment system 401 may display the combined image CM4b, which is the avatar of the second user USb, with the captured image IM41 (IM42) on the first image display device 404a worn by the first user USa. it can. The image adjustment system 401 may display the combined image CM4a, which is the avatar of the first user USa, on the second image display device 404b worn by the second user USb by combining the captured image IM42 (IM41). it can.

An example of the image adjustment method according to the fourth embodiment will be described using the flowcharts shown in FIGS. Specifically, an example of a method of combining the captured image IM41 with the composite image CM4b will be described. The first image display device 404a is attached to the head of the first user USa, and the first controller 405a is attached to the hand of the first user USa. The second image display device 404b is attached to the head of the second user USb, and the second controller 405b is attached to the hand of the second user USb. A captured image IM41 captured by the first camera 4021 is displayed on the first image display device 404a, and a captured image IM42 captured by the second camera 4022 is displayed on the second image display device 404b.

When the user USa (USb) determines that the captured image IM41 (IM42) displayed on the image display device 404a (404b) is not horizontal, the user USa (USb) controls the controller 405a (405b) in FIG. By operating, the image processing unit 411a (411b) causes the image display device 404a (404b) to display the setting screen in step S401.

When the user USa (USb) operates the controller 405a (405b) to select a predetermined item (for example, an item for horizontal adjustment) displayed on the setting screen, the image processing unit 411a (411b) causes the step In S402, the processing is shifted to a predetermined processing mode corresponding to the selected item. When the horizontal adjustment item is selected, the image processing unit 411a (411b) shifts the processing to the processing mode (horizontal adjustment mode) for adjusting the horizontal of the captured image IM41 (IM42).

In step S403, the image processing unit 411a (411b) acquires the spherical image VSS4a (VSS4b) generated by the image generation unit 412a (412b) and displays it on the image display device 404a (404b). The photographed image IM41 (IM42) and the spherical image VSS4a (VSS4b) are mixed and displayed on the image display device 404a (404b).

The user USa (USb) rotates the spherical image VSS4a (VSS4b) so that the captured image IM41 (IM42) becomes horizontal, so that the image processing unit 411a (411b) causes the spherical image VSS4a in step S404. The captured image IM41 (IM42) displayed on the image display device 404a (404b) is moved in correspondence with the rotation of (VSS4b). The user USa (USb) may perform the operation of rotating the spherical image VSS4a (VSS4b) multiple times until the captured image IM41 (IM42) becomes horizontal.

The user USa (USb) operates the controller 405a (405b) when it is determined that the captured image IM41 (IM42) has become horizontal, so that the image processing unit 411a (411b) causes the spherical image to be obtained in step S405. The display of the VSS 4a (VSS 4b) is ended, and the setting screen is displayed on the image display device 404a (404b). By horizontally setting the captured image IM41 (IM42) displayed on the image display device 404a (404b), the zenith of the camera 4021 (4022) and the zenith of the user USa (USb) can be matched.

When the user USa (USb) operates the controller 405a (405b) to select a predetermined item (for example, an end item) displayed on the setting screen, the image processing unit 411a (411b) causes the image processing unit 411a (411b) to perform step S406. At, the processing shifts to a predetermined processing mode corresponding to the selected item. When the end item is selected, the image processing unit 411a (411b) shifts the processing to the processing mode (end mode) for ending the horizontal adjustment.

In step S407, the image processing unit 411a (411b) acquires the rotation amount (rotation angle) before and after the spherical image VSS4a (VSS4b) rotates as the change amount of the spherical image VSS4a (VSS4b). .. In step S408, the image processing unit 411a (411b) stores the amount of change in the spherical image VSS4a (VSS4b) as the correction value CV4a (CV4b), and ends the processing.

A case where the combined image CM4b, which is the avatar of the second user USb, is combined with the captured image IM41 and displayed on the first image display device 404a viewed by the first user USa using the flowchart shown in FIG. explain. In FIG. 16, the image processing unit 411a identifies the first camera 4021 that is capturing the captured image IM41 that has been acquired in step S411.

The image processing unit 411a acquires the position information of the first camera 4021 from the first camera 4021 or the server 406 in step S412. The position information of the first camera 4021 includes information on the current position of the first camera 4021 and the direction in which the first camera 4021 faces or is moving.

The image processing unit 411a acquires information of another camera (camera other than the first camera 4021) from the server 406 in step S413. For example, the image processing unit 411a acquires information on the second camera 4022 from the server 406 in step S413. The information of the second camera 4022 includes the second user USb watching the captured image IM42 captured by the second camera 4022, the current position of the second camera 4022, and the direction in which the second camera 4022 faces. Or it contains information about the direction you are moving. The direction in which the second camera 4022 is facing or the direction in which it is moving corresponds to the direction in which the second user USb is facing, and is specified based on the captured image IM42 displayed on the second image display device 404b. be able to.

In step S414, the image processing unit 411a determines whether or not another camera (camera other than the first camera 4021) is captured in the captured image IM41, in other words, whether or not another camera is captured, for example, an image. Judge by recognition. When it is determined in step S414 that another camera is not included in the captured image IM41 (NO), the image processing unit 411a returns the process to step S414.

When it is determined in step S414 that another camera is included in the captured image IM41 (YES), the image processing unit 411a determines in step S415 based on the information of the other camera acquired in step S413. Identify the other camera shown in IM41.

FIG. 17 shows a state in which the first camera 4021 is located at the point A and the second camera 4022 is located at the point B in front of the first camera 4021. In the state shown in FIG. 17, the second camera 4022, which is another camera, is shown in the captured image IM41 of the first camera 4021. The image processing unit 411a identifies that the camera shown in the captured image IM41 of the first camera 4021 is the second camera 4022 based on the information of the second camera 4022 acquired in step S413.

In step S416, the image processing unit 411a, based on the position information of the first camera 4021 acquired in step S412 and the information of the second camera 4022 acquired in step S413, the second camera 4022 for the first camera 4021. And the direction in which the second camera 4022 is facing or moving.

In step S417, the image processing unit 411a identifies the user US who is viewing the captured image IM captured by the camera identified in step S415. For example, the image processing unit 411a identifies the second user USb who is looking at the captured image IM42 captured by the second camera 4022. In step S418, the image processing unit 411a acquires the composite image CM4b that is the avatar of the second user USb. When the composite image CM4b is stored in the server 406, the image processing unit 411a acquires the composite image CM4b from the server 406.

In step S419, the image processing unit 411a combines the composite image CM4b at the position corresponding to the second camera 4022 shown in the captured image IM41, and displays the composite image CM4b on the first image display device 404a. When the captured image IM41 is horizontally adjusted in steps S401 to S408, the image processing unit 411a horizontally adjusts the composite image CM4b in the same manner as the captured image IM41, and displays it on the first image display device 404a. When the correction value CV4a is stored, the image processing unit 411a horizontally adjusts the composite image CM4b based on the correction value CV4a and displays it on the first image display device 404a.

The image processing unit 411a may change the size of the composite image CM4b in correspondence with the position of the second camera 4022 with respect to the first camera 4021. For example, when the camera 4022 is located far away from the camera 4021, the image processing unit 411a determines that the size of the composite image CM4b depends on the distance between the first camera 4021 and the second camera 4022. The combined image CM4b is subjected to image processing so as to be small and combined with the captured image IM41.

The image processing unit 411a may change the orientation of the composite image CM4b in accordance with the direction in which the second camera 4022 is facing or the direction in which the second camera 4022 is moving. The image processing unit 411a may acquire the composite image CM4b in the direction corresponding to the direction in which the second camera 4022 is facing or the direction in which the second camera 4022 is moving.

In the image adjustment system 401, the image adjustment device 410a (410b), and the image adjustment method according to the fourth embodiment, the spherical image VSS4a (VSS4b) is displayed on the image display device 404a (404b). According to the image adjustment system 401, the image adjustment device 410a (410b), and the image adjustment method according to the fourth embodiment, the horizontal direction of the captured image IM41 (IM42) is erroneously detected, or the horizontal or zenith ZE is designated. In the case of misalignment, the user USa (USb) operates the controller 405a (405b) to rotate the spherical image VSS4a (VSS4b), so that the captured image displayed on the image display device 404a (404b). IM41 (IM42) can be adjusted to be horizontal.

Therefore, according to the image adjustment system 401, the image adjustment device 410a (410b), and the image adjustment method according to the fourth embodiment, the horizontal direction of the captured image IM41 (IM42) is erroneously detected, or the horizontal or zenith is designated. When there is a deviation, the user US can easily correct the horizontal or zenith of the captured image IM41 (IM42).

According to the image adjustment system 401, the image adjustment device 410a (410b), and the image adjustment method according to the fourth embodiment, when the correction value CV4a (CV4b) is stored, the image processing unit 411a (411b) performs the correction. The value CV4a (CV4b) can be read, the captured image IM41 (IM42) captured by the camera 4021 (4022) can be image-adjusted based on the correction value CV4a (CV4b), and displayed on the image display device 404a (404b).

In the image adjustment system 401, the image adjustment device 410a (410b), and the image adjustment method according to the fourth embodiment, the position of the second camera 4022 with respect to the first camera 4021 is specified, and the position of the second camera 4022 in the captured image IM. Then, the composite image CM4b which is the avatar of the second user USb looking at the captured image IM42 captured by the second camera 4022 is synthesized. As a result, the first user USa can feel as if sharing the same space as the second user USb.

The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

For example, the virtual image VSS generated by the CG may be a surface of an ellipsoid (elliptical surface), and may be an arbitrary closed surface (closed curved surface) within the reach of the user US. I wish I had it. That is, it is sufficient that the user US feels as if he/she touches the closed curved surface from the inside, and that the captured image IM can be horizontally adjusted. Since the user US rotates the virtual image VSS, it is desirable that the virtual image VSS has a spherical shape close to a spherical surface such as a spherical surface or an elliptic surface.

Claims

A camera,
An image adjusting device for adjusting a captured image captured by the camera,
An image display device for displaying the captured image adjusted by the image adjustment device,
A controller for outputting instruction information to the image adjustment device,
The image adjustment device,
A spherical image, and an image generation unit for generating a composite image to be combined with the captured image,
The spherical image generated by the image generation unit based on the instruction information is acquired and displayed on the image display device, the spherical image is rotated based on the instruction information, and the spherical image is rotated. And an image processing unit for adjusting the captured image displayed on the image display device in correspondence with the captured image, adjusting the composite image in association with the adjusted captured image, and combining the captured image with the captured image. Image adjustment system equipped.
The camera is an omnidirectional camera that captures a 360 degree range,
The image display device is a head mounted display that can be mounted on the head of the user,
The controller is a glove type controller that can be worn on the user's hand,
When the user views the spherical image displayed on the image display device in a state where the image display device is mounted on the head of the user, the spherical image is The image according to claim 1, wherein the image is a virtual image that is arranged around a person and the image display device and is set to be displayed within a range where the hand or finger of the user reaches the spherical image. Adjustment system.
In a state where the controller is attached to the user's hand, when the user moves a hand or a finger in an arbitrary direction, the image processing unit, based on the instruction information, The image adjustment according to claim 2, wherein the spherical image is rotated in response to movement of a hand or a finger, and the captured image displayed on the image display device is adjusted in response to rotation of the spherical image. system.
The image adjustment system according to claim 3, wherein the image processing unit acquires a change amount before and after the spherical image is rotated, and adjusts the captured image and the composite image based on the change amount.
5. The image processing unit according to claim 1, wherein the image processing unit adjusts the horizontalness of the captured image and the composite image displayed on the image display device in response to the rotation of the spherical image. Image adjustment system.
An image generation unit for generating a spherical image and a composite image that is captured by the camera and is combined with the captured image displayed on the image display device;
The spherical image generated by the image generation unit is acquired based on the instruction information acquired from the controller, displayed on the image display device, and the spherical image is rotated based on the instruction information to display the image. An image for adjusting the captured image displayed on the device in correspondence with the rotation of the spherical image, adjusting the composite image in association with the adjusted captured image, and combining the captured image with the captured image. An image adjustment apparatus including a processing unit.
The image processing unit acquires the instruction information from the controller,
The image processing unit, based on the instruction information, acquires the spherical image generated by the image generation unit,
The image display device displays the spherical image,
The image processing unit rotates the spherical image based on the instruction information,
The image processing unit adjusts a captured image captured by a camera and displayed on the image display device in correspondence with rotation of the spherical image,
The image processing unit acquires a composite image that is generated by the image generation unit and is combined with the captured image displayed on the image display device,
An image adjusting method comprising: the image processing unit adjusting the composite image in correspondence with the adjusted captured image and combining the composite image with the captured image.
A camera,
An image adjusting device for adjusting a captured image captured by the camera,
An image display device for displaying the captured image adjusted by the image adjustment device,
A controller for outputting instruction information to the image adjustment device,
The image adjustment device,
A spherical image, and an image generation unit for generating a composite image to be combined with the captured image,
The spherical image generated by the image generation unit based on the instruction information is acquired and displayed on the image display device, the horizontal plane of the captured image and the horizontal plane of the composite image are separated, and the instruction information is stored. The spherical image is rotated based on the spherical image, the photographed image displayed on the image display device is adjusted in correspondence with the rotation of the spherical image, and the composite image is adjusted in correspondence with the adjusted photographed image. An image adjustment system comprising an image processing unit for adjusting and combining with the photographed image.
The camera is an omnidirectional camera that captures a 360 degree range,
The image display device is a head mounted display that can be mounted on the head of the user,
The controller is a glove type controller that can be worn on the user's hand,
When the user views the spherical image displayed on the image display device in a state where the image display device is mounted on the head of the user, the spherical image is 9. The image according to claim 8, which is a virtual image that is arranged around a user and the image display device, and is set so as to be displayed within a range where the hand or finger of the user reaches the spherical image. Adjustment system.
In a state where the controller is attached to the user's hand, when the user moves a hand or a finger in an arbitrary direction, the image processing unit, based on the instruction information, The image adjustment according to claim 9, wherein the spherical image is rotated in response to movement of a hand or a finger, and the photographed image displayed on the image display device is adjusted in response to rotation of the spherical image. system.
The image adjustment system according to claim 10, wherein the image processing unit acquires a change amount before and after the spherical image is rotated, and adjusts the captured image based on the change amount.
The image adjustment system according to claim 11, wherein the image processing unit adjusts the composite image by matching a horizontal plane of the composite image with a horizontal plane of the captured image based on the amount of change.
An image generation unit for generating a spherical image and a composite image that is captured by the camera and is combined with the captured image displayed on the image display device;
The spherical image generated by the image generation unit is acquired based on the instruction information acquired from the controller, displayed on the image display device, and the horizontal plane of the captured image displayed on the image display device and the Separate the horizontal plane of the composite image, rotate the spherical image based on the instruction information, adjust the captured image in response to the rotation of the spherical image, in correspondence with the adjusted captured image An image adjusting apparatus, comprising: an image processing unit for adjusting the composite image to combine the captured image.
The image processing unit displays the captured image captured by the camera on the image display device,
The image processing unit acquires the composite image generated by the image generation unit and displays it on the image display device,
The image processing unit acquires instruction information from the controller,
The image processing unit, based on the instruction information, acquires the spherical image generated by the image generation unit and displays it on the image display device,
The image processing unit separates the horizontal plane of the captured image from the horizontal plane of the composite image,
The image processing unit rotates the spherical image based on the instruction information,
The image processing unit adjusts the captured image in correspondence with the rotation of the spherical image,
An image adjusting method comprising: the image processing unit adjusting the composite image in correspondence with the adjusted captured image and combining the composite image with the captured image.
A camera,
An image adjusting device for adjusting a captured image captured by the camera,
A plurality of image display devices corresponding to a plurality of users, for displaying the photographed image adjusted by the image adjusting device;
A plurality of controllers corresponding to the plurality of users, for outputting instruction information to the plurality of image adjusting devices;
A server for identifying a first user of the plurality of users or a first image display device of the plurality of image display devices;
Each of the plurality of image adjustment devices,
A spherical image, and an image generation unit for generating a composite image to be combined with the captured image,
The spherical image generated by the image generation unit based on the instruction information is acquired and displayed on the image display device, the spherical image is rotated based on the instruction information, and the spherical image is rotated. The photographed image displayed on the image display device according to the above, and corresponding to the adjusted photographed image, and based on position setting information indicating the mutual positional relationship of the plurality of users. And an image processing unit configured to combine the adjusted composite image with the captured image.
The camera is an omnidirectional camera that captures a 360 degree range,
The image display device is a head mounted display that can be mounted on the head of the user,
The controller is a glove type controller that can be worn on the user's hand,
When the user views the spherical image displayed on the image display device in a state where the image display device is mounted on the head of the user, the spherical image is 16. The image according to claim 15, which is a virtual image that is arranged around a person and the image display device, and is set so as to be displayed within a range where the hand or finger of the user reaches the spherical image. Adjustment system.
In a state where the controller is attached to the user's hand, when the user moves a hand or a finger in an arbitrary direction, the image processing unit, based on the instruction information, The image adjustment according to claim 16, wherein the spherical image is rotated in response to movement of a hand or a finger, and the captured image displayed on the image display device is adjusted in response to rotation of the spherical image. system.
The image processing unit adjusts the composite image corresponding to a second user other than the first user of the plurality of users based on the position setting information, and the adjusted composite image 18. The image adjusting system according to claim 15, wherein the image is combined with the captured image and displayed on the first image display device.
An image generation unit for generating a spherical image and a composite image that is captured by the camera and is combined with the captured image displayed on the image display device;
The spherical image generated by the image generation unit is acquired based on the instruction information acquired from the controller, displayed on the image display device, and the spherical image is rotated based on the instruction information to display the image. Position setting for adjusting the captured image displayed on the device in correspondence with the rotation of the spherical image, corresponding to the adjusted captured image, and indicating a positional relationship between a plurality of users. An image adjustment apparatus comprising: an image processing unit for adjusting the composite image based on information, and for combining the adjusted composite image with the captured image.
The server identifies a first user of a plurality of users or an image display device used by the first user,
The image processing unit acquires the instruction information from the controller,
The image processing unit, based on the instruction information, acquires the spherical image generated by the image generation unit,
The image display device displays the spherical image,
The image processing unit rotates the spherical image based on the instruction information,
The image processing unit adjusts a captured image captured by a camera and displayed on the image display device in correspondence with rotation of the spherical image,
The image processing unit acquires a composite image that is generated by the image generation unit and is combined with the captured image displayed on the image display device,
The image processing unit adjusts the composite image corresponding to the adjusted captured image and based on position setting information indicating the mutual positional relationship of the plurality of users, and the adjusted composite image And an image adjusting method including synthesizing the same with the captured image.
Multiple cameras,
An image adjusting device for adjusting a captured image captured by a first camera of the plurality of cameras,
An image display device for displaying the captured image adjusted by the image adjustment device,
A controller for outputting instruction information to the image adjustment device,
The image adjustment device,
A spherical image, and an image generation unit for generating a composite image to be combined with the captured image,
The spherical image generated by the image generation unit based on the instruction information is acquired and displayed on the image display device, the spherical image is rotated based on the instruction information, and the spherical image is rotated. The captured image displayed on the image display device, the composite image is adjusted in correspondence with the adjusted captured image, and the captured image is adjusted to the second of the plurality of cameras. When the second camera of the captured image is displayed, the adjusted composite image is displayed based on the position information of the first camera and the position information of the second camera. And an image processing unit for synthesizing the image corresponding to the position.
The camera is an omnidirectional camera that captures a 360 degree range,
The image display device is a head mounted display that can be mounted on the head of the user,
The controller is a glove type controller that can be worn on the user's hand,
When the user views the spherical image displayed on the image display device in a state where the image display device is mounted on the head of the user, the spherical image is 22. The image according to claim 21, which is a virtual image that is arranged around a user and the image display device and is set so as to be displayed within a range where the hand or finger of the user reaches the spherical image. Adjustment system.
In a state where the controller is attached to the user's hand, when the user moves a hand or a finger in an arbitrary direction, the image processing unit, based on the instruction information, 23. The image adjustment according to claim 22, wherein the spherical image is rotated in response to movement of a hand or finger, and the captured image displayed on the image display device is adjusted in response to rotation of the spherical image. system.
An image generation unit for generating a spherical image and a composite image that is captured by the camera and is combined with the captured image displayed on the image display device;
Based on the instruction information acquired from the controller, the spherical image generated by the image generation unit is acquired and displayed on the image display device, and the spherical image is rotated based on the instruction information. The captured image captured by the first camera of the two and displayed on the image display device is adjusted according to the rotation of the spherical image, and the composite image is adjusted according to the adjusted captured image. Is adjusted, and when the second camera of the plurality of cameras is reflected in the captured image, the adjustment is performed based on the position information of the first camera and the position information of the second camera. And an image processing unit for synthesizing the synthesized image corresponding to a position of the photographed image in which the first camera is shown.
The image processing unit acquires the instruction information from the controller,
The image processing unit, based on the instruction information, acquires the spherical image generated by the image generation unit,
The image display device displays the spherical image,
The image processing unit rotates the spherical image based on the instruction information,
The image processing unit adjusts a captured image captured by a first camera of a plurality of cameras and displayed on the image display device in correspondence with rotation of the spherical image,
The image processing unit acquires a composite image that is generated by the image generation unit and is combined with the captured image displayed on the image display device,
The image processing unit adjusts the composite image corresponding to the adjusted captured image,
When the second camera of the plurality of cameras is reflected in the captured image, the image processing unit determines, based on the position information of the first camera and the position information of the second camera. An image adjusting method comprising: combining the adjusted combined image corresponding to a position of the photographed image in which the second camera appears.