WO2019176035A1

WO2019176035A1 - Image generation device, image generation system, and image generation method

Info

Publication number: WO2019176035A1
Application number: PCT/JP2018/010078
Authority: WO
Inventors: 孝範南野; 武中川; 貴一池田
Original assignee: 株式会社ソニー・インタラクティブエンタテインメント
Priority date: 2018-03-14
Filing date: 2018-03-14
Publication date: 2019-09-19
Also published as: JPWO2019176035A1; JP7122372B2

Abstract

A camera image acquisition unit 135 acquires an image captured by a camera in which it is possible to change orientation in coordination with changes in the orientation of a head-mounted display. A camera orientation information acquisition unit 134 acquires orientation information of a camera during photographing. A difference computation unit 136 calculates the difference between the orientation information of the head-mounted display and the orientation information of the camera during photographing. An image correction unit 138 corrects a photographic image on the basis of the calculated difference so as to match the orientation of the head-mounted display.

Description

Image generating apparatus, image generating system, and image generating method

The present invention relates to an apparatus, system, and method for generating an image.

Head mounted display (HMD) is used in various fields. By providing the HMD with a head tracking function and updating the display screen in conjunction with the posture of the user's head, a sense of immersion in the video world can be enhanced.

JP-A-2015-95045

A command is transmitted from the HMD to a control device such as a robot located at a remote place to operate the robot, and an image viewed from a camera mounted on the robot is received from the robot and displayed on the display panel of the HMD. A system in which a user who wears a remote control robot is used.

In such a remote operation system, the posture of the robot camera is controlled in conjunction with the posture of the user wearing the HMD, and the user can view the captured image of the robot camera in real time. Remote operation can be performed with a sense of presence like being in a designated place. However, due to delays and errors in robot motor control, and fluctuating network delays, the camera posture does not match the HMD posture, and the camera image deviated from the HMD posture is viewed on the HMD. You may feel like

The present invention has been made in view of these problems, and an object of the present invention is to provide an image generation technique capable of adapting the camera image displayed on the head mounted display to the attitude of the head mounted display.

In order to solve the above-described problem, an image generation apparatus according to an aspect of the present invention includes a camera image acquisition unit that acquires a captured image obtained by a camera capable of changing a posture in conjunction with a change in posture of a head-mounted display; Based on the calculated difference, a camera posture information acquisition unit that acquires posture information at the time of shooting, a difference calculation unit that calculates a difference between posture information of the head mounted display and posture information at the time of shooting of the camera, and And an image correction unit that corrects the captured image so as to match the posture of the head mounted display.

Another aspect of the present invention is an image generation method. This method includes a camera image acquisition step for acquiring a photographed image obtained by a camera capable of posture change in conjunction with a posture change of the head mounted display, a camera posture information acquisition step for acquiring posture information at the time of shooting of the camera, A difference calculating step for calculating a difference between the posture information of the head mounted display and the posture information at the time of shooting of the camera, and correcting the captured image so as to match the posture of the head mounted display based on the calculated difference. Image correction step.

Note that any combination of the above components, the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded so as to be readable, a data structure, and the like are also included in the present invention. It is effective as an embodiment of

According to the present invention, the camera image displayed on the head mounted display can be adapted to the posture of the head mounted display.

It is a figure which shows the structural example of the information processing system which concerns on this Embodiment. It is a figure which shows the functional block of HMD of FIG. It is a figure which shows the external appearance structure of the robot of FIG. It is a figure which shows the example of the attitude | position of the housing | casing in the robot of FIG. It is a figure which shows the example of the attitude | position of the housing | casing in the robot of FIG. It is a figure which shows the functional block of the robot of FIG. It is a function block diagram of the reprojection part of FIG. It is a figure explaining the conventional reprojection process. It is a figure explaining the reprojection process by the reprojection part of FIG. It is a flowchart explaining the procedure of the reprojection process by the reprojection part of FIG.

FIG. 1 shows a configuration example of an information processing system 1 according to the present embodiment. The information processing system 1 includes a robot 10 and a head mounted display device (HMD) 100 that a user A wears on the head. The HMD 100 is connected to the network 4 via an access point (AP) 2. The AP 2 has functions of a wireless access point and a router, and the HMD 100 is connected to the AP 2 by a known wireless communication protocol, but may be connected by a cable.

The robot 10 includes an actuator device 12 and a housing 20 that is driven by the actuator device 12 so that the posture can be changed. The housing 20 is equipped with a right camera 14a and a left camera 14b. Hereinafter, the right camera 14a and the left camera 14b are referred to as “camera 14” unless otherwise distinguished. In the embodiment, the camera 14 is provided in a housing 20 that is driven by the actuator device 12. The robot 10 is connected to the network 4 via an access point (AP) 3. The robot 10 is connected to the AP 3 by a known wireless communication protocol, but may be connected by a cable.

In the information processing system 1, the HMD 100 and the robot 10 are communicably connected via the network 4. When the HMD 100 and the robot 10 exist in the vicinity, they may be connected to each other so as to be able to communicate directly or wirelessly without using an AP. In the information processing system 1, the robot 10 operates as a so-called alternate user A. The movement of the HMD 100 worn by the user A is transmitted to the robot 10, and the actuator device 12 moves the housing 20 in conjunction with the movement of the HMD 100. For example, when the user A swings his / her neck back and forth, the actuator device 12 moves the casing 20 to swing back and forth, and when the user A swings his / her neck left and right, the actuator device 12 moves the casing 20 to swing left and right. . Thus, a person around the robot 10 can communicate with the user A with a sense that the user A is on the spot.

Instead of AP2, an information processing device such as a game machine or a personal computer having a communication function may be provided as a client device, and the HMD 100 may be connected to the client device wirelessly or by wire. Further, an information processing device such as a game machine or a personal computer may be provided as a server device instead of AP3, and the robot 10 may be connected to the server device wirelessly or by wire.

The right camera 14a and the left camera 14b are arranged at a predetermined interval in the horizontal direction on the front surface of the housing 20. The right camera 14a and the left camera 14b constitute a stereo camera. The right camera 14a captures a right-eye image at a predetermined cycle, and the left camera 14b captures a left-eye image at a predetermined cycle. In this case, the right camera 14 a and the left camera 14 b are configured to be able to change postures in conjunction with the posture change of the HMD 100. The captured right-eye image and left-eye image are transmitted to the user A's HMD 100 in real time. The HMD 100 displays the received right-eye image on the right-eye display panel, and displays the received left-eye image on the left-eye display panel. As a result, the user A can view the video in the direction in which the housing 20 of the robot 10 is facing in real time.

In this way, in the information processing system 1, the robot 10 is remotely operated by the user A to reproduce the movement of the face of the user A, and the user A can view the image around the robot through the HMD 100.

The HMD 100 is not limited to an immersive (non-transmissive) display device that completely covers both eyes, and may be a transmissive display device. The shape may be a hat shape as shown in the figure, but may also be a glasses shape. Note that the HMD 100 may include not only a dedicated head-mounted display device but also a terminal device having a display panel, a microphone, and a speaker, and a casing that fixes the display panel of the terminal device to a position in front of the user. Good. The terminal device may have a relatively small display panel such as a smartphone or a portable game machine.

FIG. 2 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various signals and data such as image signals, audio signals, sensor information, and commands. The storage unit 122 temporarily stores data, commands, and the like that are processed by the control unit 120. The attitude sensor 124 detects attitude information such as the rotation angle and inclination of the HMD 100 at a predetermined cycle. The posture sensor 124 includes at least a triaxial acceleration sensor and a triaxial gyro sensor.

The communication control unit 126 transmits and receives signals and data to and from the robot 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 126 receives posture information detected by the posture sensor 124 from the control unit 120 and transmits the posture information to the robot 10. The communication control unit 126 receives image data from the robot 10 and supplies the image data to the control unit 120. When the control unit 120 receives the image data from the robot 10, the control unit 120 supplies the image data to the display panel 102 for display.

The reprojection unit 130 performs a reprojection process on the image data received from the robot 10. The detailed configuration of the reprojection unit 130 will be described later.

FIG. 3 shows an external configuration of the robot 10. The housing 20 accommodates the camera 14. The camera 14 is provided on the front surface of the housing. The camera 14 operates by being supplied with electric power from a power supply device accommodated in the housing 36 via a power line (not shown).

The housing 20 has a protective cover 19. When the robot 10 is not used, that is, when the power of the robot 10 is turned off, the protective cover 19 is disposed at a closed position that covers the front of the housing, and the camera 14 is Protect.

The housing 20 is supported by the actuator device 12 so that the posture can be changed. The actuator device 12 includes a leg portion 40, a hemispherical housing 36 supported on the upper portion of the leg portion 40, and a drive mechanism 50 for driving the housing 20. The drive mechanism 50 includes a first arcuate arm 32 having a first through hole 32a formed in the longitudinal direction, a second arcuate arm 34 having a second through hole 34a formed in the longitudinal direction, A pedestal 30 that rotatably supports the first arc-shaped arm 32 and the second arc-shaped arm 34 in a state where the first arc-shaped arm 32 and the second arc-shaped arm 34 intersect each other is provided. The upper side of the pedestal 30 is covered with a cover 38, and motors for rotating the first arcuate arm 32 and the second arcuate arm 34 are arranged in the space covered with the cover 38, respectively. The pedestal 30 is rotatably supported with respect to the housing 36, and a motor for rotating the pedestal 30 is disposed in the housing 36.

The first arc-shaped arm 32 and the second arc-shaped arm 34 are formed in a semicircular shape, and both ends are supported by the pedestal 30 so as to have the same center of rotation. The diameter of the semicircular first arc-shaped arm 32 is slightly larger than the diameter of the semicircular second arc-shaped arm 34, and the first arc-shaped arm 32 is located on the outer peripheral side of the second arc-shaped arm 34. Be placed. The first arc-shaped arm 32 and the second arc-shaped arm 34 may be arranged so as to be orthogonal to each other on the pedestal 30. In the embodiment, a line connecting both ends of the first arcuate arm 32 supported by the pedestal 30 and a line connecting both ends of the second arcuate arm 34 supported by the pedestal 30 are orthogonal to each other. The insertion member 42 is inserted into the first through long hole 32a and the second through long hole 34a, and is disposed at the intersection of the first through long hole 32a and the second through long hole 34a. The insertion member 42 slides in the first through long hole 32a and the second through long hole 34a by the rotation of the first arc-shaped arm 32 and the second arc-shaped arm 34.

The first motor is provided for rotating the first arcuate arm 32, and the second motor is provided for rotating the second arcuate arm 34. A 1st motor and a 2nd motor are arrange | positioned on the base 30, and if the base 30 rotates, a 1st motor and a 2nd motor will also rotate with the base 30. FIG. The third motor is provided to rotate the pedestal 30 and is disposed in the housing 36. The first motor, the second motor, and the third motor are rotated by power supplied from a power supply device (not shown).

The first motor rotates the first arc-shaped arm 32, the second motor rotates the second arc-shaped arm 34, and the third motor rotates the pedestal 30, whereby the actuator device 12 is attached to the insertion member 42. The orientation and posture of the housing 20 can be changed.

4 and 5 are diagrams illustrating examples of the posture of the housing 20 in the robot 10. FIG.
4A and 4B show an example in which the housing 20 is tilted in the left-right direction. 5A and 5B show an example in which the housing 20 is tilted in the front-rear direction. Thus, the drive mechanism 50 of the robot 10 can cause the housing 20 to take an arbitrary posture. The attitude of the housing 20 is controlled by adjusting the driving amounts of the first motor and the second motor, and the orientation of the housing 20 is controlled by adjusting the driving amount of the third motor.

FIG. 6 shows functional blocks of the robot 10. The robot 10 includes an input system 22 that receives and processes input from the outside, and an output system 24 that processes output to the outside. The input system 22 includes a reception unit 60, a sensor information acquisition unit 62, a motion detection unit 64, a line-of-sight direction determination unit 66, and an actuator control unit 68. The output system 24 includes an image processing unit 80 and a transmission unit 90.

In FIG. 6, each element described as a functional block for performing various processes can be configured by a circuit block, a memory, and other LSIs in terms of hardware, and loaded in the memory in terms of software. Realized by programs. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

As described above, the HMD 100 transmits the sensor information detected by the attitude sensor 124 to the robot 10, and the receiving unit 60 receives the sensor information.

The sensor information acquisition unit 62 acquires the posture information detected by the posture sensor 124 of the HMD 100. The motion detection unit 64 detects the posture of the HMD 100 attached to the user A's head. The line-of-sight direction determination unit 66 determines the line-of-sight direction of the camera 14 of the housing 20 according to the attitude of the HMD 100 detected by the motion detection unit 64.

The motion detection unit 64 performs a head tracking process for detecting the posture of the head of the user wearing the HMD 100. The head tracking process is performed in order to link the visual field displayed on the display panel 102 of the HMD 100 to the posture of the user's head. In the head tracking process of the embodiment, the rotation angle with respect to the horizontal reference direction of the HMD 100 and the horizontal plane The tilt angle is detected. The horizontal reference direction may be set, for example, as a direction facing when the power of the HMD 100 is turned on.

The line-of-sight direction determination unit 66 determines the line-of-sight direction according to the attitude of the HMD 100 detected by the motion detection unit 64. This line-of-sight direction is the line-of-sight direction of the user A, and by extension, the line-of-sight direction (optical axis direction) of the camera 14 of the robot 10 that is a substitute.

In order to link the viewing direction (optical axis direction) of the camera 14 to the viewing direction of the user A, it is necessary to set the reference posture of the robot 10 in advance. FIG. 3 shows a state in which the first arc-shaped arm 32 and the second arc-shaped arm 34 stand up by 90 degrees with respect to the pedestal 30. This state is set as the horizontal direction and the power supply of the robot 10 The direction in which the front surface of the housing 20 faces when is turned on may be set as the horizontal reference direction. Note that the robot 10 may have a posture sensor as in the HMD 100 so that the horizontal direction can be set autonomously.

With the reference postures of the HMD 100 and the robot 10 set, the line-of-sight direction determination unit 66 determines the rotation angle and tilt angle detected by the motion detection unit 64 as the line-of-sight direction (optical axis direction) of the camera 14 as it is. Good. When the motion detection unit 64 detects the rotation angle and the inclination angle of the HMD 100, the line-of-sight direction determination unit 66 determines the line-of-sight direction of the HMD 100 as a vector (x, y, z) of three-dimensional coordinates. The line-of-sight direction of the camera 14 may be determined as the same (x, y, z), or may be determined as (x ′, y ′, z ′) with some correction.

The actuator control unit 68 controls the orientation of the camera 14 so that the line-of-sight direction determined by the line-of-sight direction determination unit 66 is obtained. Specifically, the actuator control unit 68 adjusts the power supplied to the first motor 52, the second motor 54, and the third motor 56 so that the movement of the housing 20 follows the movement of the HMD 100. The motor drive control by the actuator control unit 68 is performed in real time, and therefore the direction of the housing 20 is moved in the same way as the direction of the line of sight of the user A.

According to the actuator device 12 of the embodiment, the housing 20 is driven with reference to the rotation centers of the first arc-shaped arm 32 and the second arc-shaped arm 34, and this movement shows the same movement as the human neck. . The actuator device 12 reproduces the movement of the neck of the user A with a simple structure in which two semicircular arms are crossed.

Next, the output system 24 will be described.
In the output system 24, the right camera 14a and the left camera 14b are directed in the directions controlled by the actuator device 12 and shoot the respective angles of view. The right camera 14a and the left camera 14b may be arranged apart from each other, for example, so as to be an average distance between eyes of an adult. The right-eye image data captured by the right camera 14a and the left-eye image data captured by the left camera 14b are transmitted from the transmission unit 90 to the HMD 100 and displayed on the right half and the left half of the display panel 102, respectively. These images form parallax images viewed from the right eye and the left eye, and can be displayed stereoscopically by displaying the display panel 102 in an area divided into two. In order for the user A to view the display panel 102 through the optical lens, the image processing unit 80 may generate image data in which optical distortion caused by the lens is corrected in advance and supply the image data to the HMD 100.

The right camera 14a and the left camera 14b perform shooting at a predetermined cycle (for example, 1/60 seconds), and the transmission unit 90 transmits image data to the HMD 100 without delay. As a result, the user A can see the situation around the robot 10 in real time, and can also see the desired direction by changing the direction of the face.

The camera posture information acquisition unit 70 acquires posture information including the posture and orientation of the camera 14 and supplies the posture information to the transmission unit 90. The transmission unit 90 transmits the posture information of the camera 14 to the HMD 100. The posture information of the camera 14 may be obtained using, for example, posture information detected by a motion sensor, or position information of a motor mounted on the actuator device 12 that changes the posture and orientation of the housing 20 of the robot 10. You may ask for.

FIG. 7 is a functional configuration diagram of the reprojection unit 130 of FIG. This figure depicts a block diagram focusing on functions, and these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The reprojection unit 130 includes an HMD posture information acquisition unit 132, a camera posture information acquisition unit 134, a camera image acquisition unit 135, a difference calculation unit 136, a warning unit 137, and an image correction unit 138.

The HMD posture information acquisition unit 132 acquires posture information (referred to as “HMD posture information”) including the posture and orientation of the HMD 100 detected by the posture sensor 124 and supplies the posture information to the difference calculation unit 136. The camera posture information acquisition unit 134 acquires posture information (referred to as “camera posture information”) that the communication control unit 126 received from the robot 10 and includes the posture and orientation of the camera 14 at the time of shooting with the camera 14 and calculates the difference. Supplied to the unit 136. The camera image acquisition unit 135 acquires the captured images (referred to as “camera images”) of the left and right cameras 14 received by the communication control unit 126 from the robot 10 and supplies them to the image correction unit 138.

Here, the camera posture information may be embedded as metadata in the camera image. The HDMI (registered trademark) 2.1 standard may be used as a method for transmitting camera posture information as metadata embedded in a camera image. In the HDMI 2.1 standard, it is studied to insert and transmit dynamic metadata to be applied to each frame in the VBlank signal of each frame data. If the camera posture information is embedded in the dynamic metadata, the camera posture information at the time of imaging by the camera 14 of the robot 10 can be transmitted to the HMD 100 in synchronization with the frame data of the camera image. Here, an example has been described in which dynamic metadata is inserted into an HDMI 2.1-standard VBlank signal. However, this is only an example, and dynamic metadata may be inserted into some synchronization signal synchronized with each frame and transmitted. . The camera posture information acquisition unit 134 extracts the camera posture information from the metadata inserted in the synchronization signal of each frame of the camera image received by the communication control unit 126 from the robot 10.

The difference calculation unit 136 calculates the difference between the HMD posture information and the camera posture information and supplies the difference to the image correction unit 138. The image correction unit 138 performs reprojection processing by correcting the camera image so as to match the latest posture of the HMD 100 based on the calculated posture difference. The reprojected camera image is supplied to the control unit 120 and displayed on the display panel 102.

Based on the HMD posture information, it is determined that the HMD 100 has been moved beyond the movable range of the robot 10, or it is determined that the posture of the HMD 100 has approached a predetermined threshold with respect to the limit value of the movable range of the robot 10. In this case, the difference calculation unit 136 interrupts or continues to instruct the image correction unit 138 to perform the reprojection process, and notifies the warning unit 137 of the instruction. The warning unit 137 displays a message on the display panel 102 that the movable range is exceeded or that the movable limit is approached. At that time, the direction in which the user wearing the HMD 100 should return the head may be displayed together with the warning message.

Specifically, the difference calculation unit 136 may determine whether the attitude of the HMD 100 is within the movable range of the robot 10. For example, when the camera 14 of the robot 10 cannot be rotated 360 degrees and can only be rotated within a certain range, the range is set in the HMD 100 in advance as the movable range, and the HMD posture information is moved to the movable position of the camera 14. Compare with range. Without setting the movable range in advance, it may be determined that the movable range has been exceeded when the posture of the HMD 100 and the posture of the camera 14 are greatly deviated.

Further, since the posture of the camera 14 of the robot 10 is controlled by a motor, the posture cannot be changed as fast as the user wearing the HMD 100. Therefore, if it is determined that the HMD 100 has been moved beyond the speed at which the camera 14 can follow the attitude change of the HMD 100 based on the HMD attitude information, the difference calculation unit 136 instructs the image correction unit 138 to perform a reprojection process. This is interrupted or continued, and the warning unit 137 is notified. The warning unit 137 displays on the display panel 102 a message that prompts the user wearing the HMD 100 to move his / her head slowly.

Specifically, the difference calculation unit 136 determines that the followable speed of the camera 14 has been exceeded when the difference between the HMD posture information and the camera posture information exceeds a threshold or when the change speed of the HMD posture information exceeds the threshold. . When the attitude of the HMD 100 moves faster than a predetermined threshold, the difference between the HMD attitude information and the camera attitude information increases, so it may be determined that the followable speed of the camera 14 has been exceeded.

As described above, when the movement of the HMD 100 approaches the limit of the movable range or followable speed of the camera 14 or exceeds the limit, when the user corrects the camera image by the reprojection process, the user moves the movable range of the camera 14. If you can move your head even after exceeding the speed that can be followed, it may be misunderstood. In the present embodiment, the reprojection process is interrupted in such a case, and the warning unit 137 displays a warning on the display panel 102, thereby appropriately restricting the movement of the head of the user wearing the HMD 100. it can. However, even in such a case, it is possible to display a warning while continuing the reprojection process without interruption.

The operation of the reprojection unit 130 in FIG. 7 will be described. For comparison, the conventional reprojection processing will be described with reference to FIG. 8, and then the reprojection processing of the present embodiment will be described with reference to FIG.

FIG. 8 is a diagram for explaining a conventional reprojection process. Here, the rendering apparatus 300 is connected to the HMD 100 via a network instead of the robot 10.

When the HMD 100 is provided with a head tracking function and the image is rendered by changing the viewpoint or line-of-sight direction in conjunction with the movement of the user's head and displayed on the HMD 100, the rendering is delayed from rendering to display. Sometimes, a deviation occurs between the orientation of the user's head that is assumed at the time and the orientation of the user's head when the image is displayed on the HMD 100, and the user may feel drunk. In the case of passing through a network, a shift also occurs due to delay fluctuations caused by the network. Therefore, a reprojection is performed by applying a correction process for compensating for a deviation between the attitude of the HMD 100 used at the time of rendering and the attitude of the latest HMD 100 to the rendering image, and the user can display the image after the reprojection on the HMD 100. Reduce the sickness of

The HMD 100 performs motion prediction based on the current posture information, and transmits the predicted HMD posture information 202 to the rendering device 300. The rendering apparatus 300 renders an image to be displayed on the HMD 100 using the predicted HMD posture information 202. The rendering apparatus 300 transmits the rendered image 310 to the HMD 100 together with the posture information 204 used for rendering.

Note that the posture information 204 used for rendering here is the same as the predicted HMD posture information 202. The rendering apparatus 300 may perform motion prediction of the HMD 100 and generate the predicted HMD posture information 202.

The difference calculation unit 302 calculates a difference between the latest HMD posture information 200 and the posture information 204 used for rendering, and the reprojection unit 304 performs a reprojection process on the rendered image 310 based on the calculated posture difference. Display on HMD100.

FIG. 9 is a diagram for explaining reprojection processing by the reprojection unit 130 of FIG.

The motion is predicted based on the current posture information of the HMD 100, and the predicted HMD posture information 202 is transmitted to the robot 10. The robot 10 controls the posture and orientation of the camera 14 using the predicted HMD posture information 202 and takes an image with the camera 14. The robot 10 transmits the camera image 210 to the HMD 100 together with the camera posture information 205 at the time of shooting.

Note that the camera posture information 205 at the time of shooting is different from the predicted HMD posture information 202. This is because when the motor controls the posture of the camera 14 in the robot 10, a motor error, an operation delay, and the like occur, and thus the posture of the camera 14 cannot be completely matched with the predicted posture of the HMD 100.

The difference calculation unit 136 calculates the difference between the latest HMD posture information 200 and the camera posture information 205 at the time of shooting. The image correction unit 138 performs reprojection processing on the camera image 210 so as to match the latest attitude of the HMD 100 based on the calculated attitude difference, and displays the image on the HMD 100. Strictly speaking, the latest HMD posture information 200 is predicted in consideration of the time until the image displayed on the HMD 100 after reprojection processing enters the user's eyes, not the posture of the current HMD 100. You may use the attitude | position of HMD100.

The reprojection unit 130 according to the present embodiment is different from the conventional reprojection processing in that reprojection is performed using the camera posture information 205 at the time of shooting instead of the predicted HMD posture information 202.

The reprojection process using the camera attitude information 205 at the time of shooting eliminates the deviation caused by the attitude of the camera 14 not matching the attitude of the HMD 100 due to a motor error or operation delay, and the user wearing the HMD 100 feels uncomfortable. Can be reduced.

Since the movement of the camera 14 controlled by the motor is less accurate than the movement of the human head, the moving image shot by the camera 14 does not move smoothly. By performing the reprojection process, there is also an effect of smoothing the motion of the moving image shot by the camera 14.

Further, when the predicted HMD posture information 202 is transmitted to the robot 10, packet loss may occur depending on the congestion state of the network, and the predicted HMD posture information 202 may be lost without being transmitted to the robot 10. Even in such a case, since the posture information transmitted to the HMD 100 together with the camera image 210 is the camera posture information 205 at the time of shooting, even if the predicted HMD posture information 202 is missing, it is based on the camera posture information 205 at the time of shooting. There is also an advantage that the reprojection processing can be performed reliably.

The camera posture information 205 at the time of shooting is not transmitted together with the camera image 210, and the posture of the camera 14 at the time of shooting is estimated by self-position estimation from the camera image 210 using a technique such as SLAM (Simultaneous Localization and Mapping). You may comprise so that it may estimate.

FIG. 10 is a flowchart for explaining the procedure of the reprojection process performed by the reprojection unit 130 shown in FIG.

The camera image acquisition unit 135 receives a camera image from the robot 10, and the camera posture information acquisition unit 134 acquires camera posture information added as metadata to the camera image (S10).

The HMD posture information acquisition unit 132 acquires the latest HMD posture information detected by the posture sensor 124 (S12).

The difference calculation unit 136 calculates the difference between the latest HMD posture information and the camera posture information (S14).

The difference calculation unit 136 determines whether or not the latest HMD posture information is within the movable range of the camera 14 of the robot 10 (S16), and if within the movable range (Y of S16), the process proceeds to step S20.

When the latest HMD posture information is approaching the limit value of the movable range of the camera 14 of the robot 10 or exceeds the movable range (N in S16), the warning unit 137 moves the movement of the HMD 100 within the movable range of the robot 10. Is displayed on the display panel 102 (S18).

The difference calculation unit 136 determines whether the posture change of the HMD 100 is within the followable speed of the camera 14 of the robot 10 (S20), and if within the followable speed (Y of S20), the process proceeds to step S24.

When the posture change of the HMD 100 approaches the limit value of the followable speed of the robot 10 or exceeds the followable speed (N in S20), the warning unit 137 causes the posture change of the HMD 100 to be within the followable speed of the robot 10. A warning message for guiding the user to fit in the screen is displayed on the display panel 102 (S22).

The image correcting unit 138 executes the reprojection process by correcting the camera image based on the difference between the latest HMD posture information and the camera posture information (S24).

The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. . Such a modification will be described.

In the above embodiment, the reprojection unit 130 is provided in the HMD 100, but the reprojection unit 130 may be provided in a client device to which the HMD 100 is connected.

In the above embodiment, the case where the HMD 100 and the robot 10 are connected via a network and the robot 10 is remotely controlled has been described. However, the same applies to the case where the HMD 100 and the robot 10 are connected via a network. Reprojection processing can be applied. Since there is a motor delay or error, a deviation between the posture of the camera 14 and the posture of the HMD 100 occurs even if there is no delay variation due to the network. Therefore, the reprojection processing for correcting the camera image according to the posture of the HMD 100 is effective. is there. Further, the same reprojection process can be applied to any control device as long as the control device is not limited to the robot 10 and is equipped with the camera 14.

In the above-described embodiment, the case where the robot 10 itself does not move and the camera 14 mounted on the robot 10 changes the posture has been described, but the same reprojection processing is performed even when the robot 10 itself changes the position. Can be applied. In that case, not only the camera posture by the motion sensor but also the absolute position information of the camera is acquired from the position information of the servo mechanism and transmitted together with the camera image. When the camera 14 moves, a self-position estimation technique that estimates the position and orientation of the camera 14 on the environment map, such as SLAM, can be used. Note that the position information of the servo mechanism and the self-position estimation technique such as SLAM can be used to acquire the posture information of the camera 14 even when the robot 10 does not change the position.

1 Information processing system, 10 robot, 12 actuator device, 14a right camera, 14b left camera, 20 housing, 22 input system, 24 output system, 30 pedestal, 32 1st arc arm, 34 2nd arc arm, 36 Housing, 38 cover, 40 legs, 42 insertion member, 50 drive mechanism, 52 1st motor, 54 2nd motor, 56 3rd motor, 60 receiving section, 62 sensor information acquisition section, 64 motion detection section, 66 gaze direction Determination unit, 68 Actuator control unit, 70 Camera posture information acquisition unit, 80 Image processing unit, 90 Transmission unit, 100 HMD, 102 Display panel, 120 Control unit, 122 Storage unit, 124 Posture sensor, 126 Communication Control unit, 130 reprojection unit, 132 HMD orientation obtaining unit, 134 a camera orientation obtaining unit, 135 a camera image acquiring unit, 136 difference calculation unit, 137 a warning unit, 138 image correcting unit.

* Applicable to image generation technology.

Claims

A camera image acquisition unit that acquires a photographed image by a camera capable of posture change in conjunction with the posture change of the head mounted display;
A camera posture information acquisition unit for acquiring posture information at the time of shooting of the camera;
A difference calculation unit for calculating a difference between posture information of the head-mounted display and posture information at the time of shooting of the camera;
An image generation apparatus comprising: an image correction unit configured to correct the captured image so as to match the posture of the head mounted display based on the calculated difference.
The image generation apparatus according to claim 1, wherein the camera posture information acquisition unit extracts posture information at the time of shooting of the camera from metadata inserted in a synchronization signal of each frame of the shot image.
When it is determined that the attitude of the head mounted display approaches a predetermined threshold with respect to the limit value of the movable range of the camera based on the attitude information of the head mounted display, or exceeds the movable range of the camera, The image generation apparatus according to claim 1, further comprising a warning unit that interrupts or continues correction of the captured image by the image correction unit and displays a warning.
When it is determined that the posture change of the head mounted display exceeds the followable speed of the camera based on the posture information of the head mounted display, the correction of the captured image by the image correction unit is interrupted or continued, and a warning is issued. The image generating apparatus according to claim 1, further comprising a warning unit for displaying.
A camera image acquisition step for acquiring a photographed image by a camera capable of changing the posture in conjunction with the posture change of the head mounted display;
Camera posture information acquisition step for acquiring posture information at the time of shooting of the camera;
A difference calculating step for calculating a difference between posture information of the head-mounted display and posture information at the time of shooting of the camera;
An image generation method comprising: an image correction step of correcting the captured image so as to match the posture of the head mounted display based on the calculated difference.
A camera image acquisition function for acquiring a captured image by a camera capable of changing the posture in conjunction with the posture change of the head mounted display;
A camera posture information acquisition function for acquiring posture information at the time of shooting of the camera;
A difference calculation function for calculating a difference between posture information of the head-mounted display and posture information at the time of shooting of the camera;
A program for causing a computer to realize an image correction function for correcting the photographed image so as to match the posture of the head mounted display based on the calculated difference.