JPWO2019176035A1 - Image generator, image generation system, and image generation method - Google Patents

Abstract

The camera image acquisition unit 135 acquires an image captured by the camera that can change the posture in conjunction with the change in the posture of the head-mounted display. The camera posture information acquisition unit 134 acquires posture information at the time of shooting by the camera. The difference calculation unit 136 calculates the difference between the posture information of the head-mounted display and the posture information at the time of shooting by the camera. The image correction unit 138 corrects the captured image so as to match the posture of the head-mounted display based on the calculated difference.

Description

The present invention relates to devices, systems and methods for producing images.

Head-mounted displays (HMDs) are used in various fields. By adding a head tracking function to the HMD and updating the display screen in conjunction with the posture of the user's head, it is possible to enhance the immersive feeling in the video world.

JP-A-2015-95045

By sending a command from the HMD to a control device such as a robot located in a remote location to operate the robot, and receiving the image seen from the camera mounted on the robot from the robot and displaying it on the display panel of the HMD, the HMD A system is used in which a user wearing a robot remotely controls a robot.

In such a remote control system, the posture of the robot camera is controlled in conjunction with the posture of the user wearing the HMD, and the user can see the captured image of the robot camera in real time. Remote control can be performed with a sense of realism as if you were in a designated place. However, due to the delay and error of the motor control of the robot and the fluctuating network delay, the attitude of the camera does not match the attitude of the HMD, and the camera image deviated from the attitude of the HMD is viewed on the HMD, and the user gets drunk. You may feel like you are.

The present invention has been made in view of these problems, and an object of the present invention is to provide an image generation technique capable of adapting a camera image displayed on a head-mounted display to the posture of the head-mounted display.

In order to solve the above problems, the image generation device of an embodiment of the present invention includes a camera image acquisition unit that acquires an image captured by a camera capable of changing the posture in conjunction with the posture change of the head mount display, and a camera image acquisition unit of the camera. The camera attitude information acquisition unit that acquires the attitude information at the time of shooting, the difference calculation unit that calculates the difference between the attitude information of the head mount display and the attitude information at the time of shooting by the camera, and the calculation unit based on the calculated difference. It includes an image correction unit that corrects the captured image so as to match the posture of the head mount display.

Another aspect of the present invention is an image generation method. In this method, a camera image acquisition step of acquiring an image taken by a camera capable of changing the posture in conjunction with a posture change of the head mount display, a camera attitude information acquisition step of acquiring the posture information at the time of shooting by the camera, and a camera attitude information acquisition step. A difference calculation step for calculating the difference between the posture information of the head mount display and the posture information at the time of shooting by the camera, and the photographed image is corrected so as to match the posture of the head mount display based on the calculated difference. Includes image correction steps.

It should be noted that any combination of the above components, the expression of the present invention converted between a method, a device, a system, a computer program, a recording medium in which a computer program is readable, a data structure, and the like are also used in the present invention. It is effective as an aspect of.

According to the present invention, the camera image displayed on the head-mounted display can be adapted to the posture of the head-mounted display.

It is a figure which shows the configuration example of the information processing system which concerns on this embodiment. It is a figure which shows the functional block of the HMD of FIG. It is a figure which shows the appearance structure of the robot of FIG. It is a figure which shows the example of the posture of the housing in the robot of FIG. It is a figure which shows the example of the posture of the housing in the robot of FIG. It is a figure which shows the functional block of the robot of FIG. It is a functional block diagram of the projection part of FIG. It is a figure explaining the conventional reprojection processing. It is a figure explaining the reprojection processing by the reprojection part of FIG. It is a flowchart explaining the procedure of the reprojection processing by the reprojection part of FIG.

FIG. 1 shows a configuration example of the information processing system 1 according to the present embodiment. The information processing system 1 includes a robot 10 and a head-mounted display device (HMD) 100 worn by the user A on the head. The HMD 100 is connected to the network 4 via the access point (AP) 2. The AP2 has the functions of a wireless access point and a router, and the HMD 100 connects to the AP2 using a known wireless communication protocol, but may be connected by a cable.

The robot 10 includes an actuator device 12 and a housing 20 that is driven by the actuator device 12 so that its posture can be changed. A right camera 14a and a left camera 14b are mounted on the housing 20. Hereinafter, when the right camera 14a and the left camera 14b are not particularly distinguished, they are referred to as "camera 14". In the embodiment, the camera 14 is provided in the housing 20 driven by the actuator device 12. The robot 10 is connected to the network 4 via the access point (AP) 3. The robot 10 is connected to the AP3 by a known wireless communication protocol, but may be connected by a cable.

In the information processing system 1, the HMD 100 and the robot 10 are communicably connected to each other via the network 4. When the HMD 100 and the robot 10 are close to each other, they may be directly connected to each other by wireless or wired communication without going through the AP. In the information processing system 1, the robot 10 operates as a so-called alter ego of the user A. The movement of the HMD 100 worn by the user A is transmitted to the robot 10, and the actuator device 12 moves the housing 20 in conjunction with the movement of the HMD 100. For example, when the user A shakes his head back and forth, the actuator device 12 moves the housing 20 back and forth, and when the user A shakes his head left and right, the actuator device 12 moves the housing 20 left and right. .. As a result, the person around the robot 10 can communicate with the user A as if the user A were there.

Instead of AP2, an information processing device such as a game machine or a personal computer having a communication function may be provided as a client device, and the HMD 100 may be connected to the client device wirelessly or by wire. Further, instead of AP3, an information processing device such as a game machine or a personal computer may be provided as a server device, and the robot 10 may be connected to the server device wirelessly or by wire.

The right camera 14a and the left camera 14b are arranged laterally at predetermined intervals on the front surface of the housing 20. The right camera 14a and the left camera 14b constitute a stereo camera, the right camera 14a captures an image for the right eye at a predetermined cycle, and the left camera 14b captures an image for the left eye at a predetermined cycle. In this case, the right camera 14a and the left camera 14b are configured so that the posture can be changed in conjunction with the posture change of the HMD 100. The captured right-eye image and left-eye image are transmitted to the user A's HMD100 in real time. The HMD 100 displays the received right-eye image on the right-eye display panel, and displays the received left-eye image on the left-eye display panel. As a result, the user A can see the image in the direction in which the housing 20 of the robot 10 is facing in real time.

In this way, in the information processing system 1, the robot 10 is remotely controlled by the user A to reproduce the movement of the user A's face, and the user A can see the image around the robot through the HMD 100.

The HMD 100 is not limited to an immersive (non-transmissive) display device that completely covers both eyes, and may be a transmissive display device. The shape may be a hat type as shown in the figure, but may be a spectacle type. The HMD 100 may be composed of not only a dedicated head-mounted display device but also a terminal device having a display panel, a microphone, and a speaker, and a housing for fixing the display panel of the terminal device at a position in front of the user. Good. The terminal device may have a relatively small display panel, such as a smartphone or a portable game machine.

FIG. 2 shows a functional block of the HMD 100. The control unit 120 is a main processor that processes and outputs various signals and data such as image signals, audio signals, and sensor information, and commands. The storage unit 122 temporarily stores data, instructions, and the like processed by the control unit 120. The posture sensor 124 detects posture information such as the rotation angle and tilt of the HMD 100 at a predetermined cycle. The attitude sensor 124 includes at least a 3-axis accelerometer and a 3-axis gyro sensor.

The communication control unit 126 transmits / receives signals and data to / from the robot 10 via a network adapter or an antenna by wired or wireless communication. The communication control unit 126 receives the attitude information detected by the attitude sensor 124 from the control unit 120 and transmits it to the robot 10. Further, the communication control unit 126 receives image data from the robot 10 and supplies it to the control unit 120. When the control unit 120 receives the image data from the robot 10, the control unit 120 supplies the image data to the display panel 102 and displays the image data.

The reprojection unit 130 performs a reprojection process on the image data received from the robot 10. The detailed configuration of the reprojection unit 130 will be described later.

FIG. 3 shows the appearance configuration of the robot 10. The housing 20 houses the camera 14. The camera 14 is provided on the front surface of the housing. The camera 14 operates by being supplied with electric power from a power supply device housed in the housing 36 via a power line (not shown).

The housing 20 has a protective cover 19, and when the robot 10 is not used, that is, when the power of the robot 10 is turned off, the protective cover 19 is arranged in a closed position covering the front surface of the housing, and the camera 14 is mounted. Protect.

The housing 20 is supported by the actuator device 12 so that its posture can be changed. The actuator device 12 includes a leg portion 40, a hemispherical housing 36 supported on the upper portion of the leg portion 40, and a drive mechanism 50 for driving the housing 20. The drive mechanism 50 includes a first arc-shaped arm 32 having a first penetrating elongated hole 32a formed in the elongated direction, a second arc-shaped arm 34 having a second penetrating elongated hole 34a formed in the elongated direction, and a second. A pedestal 30 that rotatably supports the first arc-shaped arm 32 and the second arc-shaped arm 34 in a state where the first arc-shaped arm 32 and the second arc-shaped arm 34 are crossed is provided. The upper side of the pedestal 30 is covered with a cover 38, and in the space covered by the cover 38, motors for rotating the first arc-shaped arm 32 and the second arc-shaped arm 34 are arranged. The pedestal 30 is rotatably supported with respect to the housing 36, and a motor for rotating the pedestal 30 is arranged in the housing 36.

The first arc-shaped arm 32 and the second arc-shaped arm 34 are formed in a semicircular shape, and both ends thereof are supported by the pedestal 30 so as to have the same rotation center. The diameter of the semicircular first arc-shaped arm 32 is slightly larger than the diameter of the semicircular second arc-shaped arm 34, and the first arc-shaped arm 32 is located on the outer peripheral side of the second arc-shaped arm 34. Be placed. The first arc-shaped arm 32 and the second arc-shaped arm 34 may be arranged so as to be orthogonal to each other on the pedestal 30. In the embodiment, the line connecting both ends of the first arcuate arm 32 supported by the pedestal 30 and the line connecting both ends of the second arcuate arm 34 supported by the pedestal 30 are orthogonal to each other. The insertion member 42 is inserted through the first through elongated hole 32a and the second through elongated hole 34a, and is arranged at the intersection position of the first through elongated hole 32a and the second through elongated hole 34a. The insertion member 42 slides in the first through elongated hole 32a and in the second through elongated hole 34a by the rotation of the first arc-shaped arm 32 and the second arc-shaped arm 34.

The first motor is provided to rotate the first arc-shaped arm 32, and the second motor is provided to rotate the second arc-shaped arm 34. The first motor and the second motor are arranged on the pedestal 30, and when the pedestal 30 rotates, the first motor and the second motor also rotate together with the pedestal 30. The third motor is provided to rotate the pedestal 30 and is arranged in the housing 36. The first motor, the second motor, and the third motor are rotated by being supplied with electric power from a power supply device (not shown).

The actuator device 12 is attached to the insertion member 42 by the first motor rotating the first arc-shaped arm 32, the second motor rotating the second arc-shaped arm 34, and the third motor rotating the pedestal 30. The orientation and orientation of the housing 20 can be changed.

4 and 5 are views showing an example of the posture of the housing 20 in the robot 10.
4 (a) and 4 (b) show an example in which the housing 20 is tilted in the left-right direction. 5 (a) and 5 (b) show an example in which the housing 20 is tilted in the front-rear direction. In this way, the drive mechanism 50 of the robot 10 allows the housing 20 to take an arbitrary posture. The posture of the housing 20 is controlled by adjusting the driving amounts of the first and second motors, and the orientation of the housing 20 is controlled by adjusting the driving amounts of the third motor.

FIG. 6 shows a functional block of the robot 10. The robot 10 includes an input system 22 that receives and processes an input from the outside, and an output system 24 that processes an output to the outside. The input system 22 includes a receiving unit 60, a sensor information acquisition unit 62, a motion detecting unit 64, a line-of-sight direction determining unit 66, and an actuator control unit 68. The output system 24 also includes an image processing unit 80 and a transmission unit 90.

In FIG. 6, each element described as a functional block that performs various processes can be composed of a circuit block, a memory, and other LSIs in terms of hardware, and is loaded in the memory in terms of software. It is realized by a program or the like. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any of them.

As described above, the HMD 100 transmits the sensor information detected by the attitude sensor 124 to the robot 10, and the receiving unit 60 receives the sensor information.

The sensor information acquisition unit 62 acquires the attitude information detected by the attitude sensor 124 of the HMD 100. The motion detection unit 64 detects the posture of the HMD 100 mounted on the head of the user A. The line-of-sight direction determination unit 66 determines the line-of-sight direction of the camera 14 of the housing 20 according to the posture of the HMD 100 detected by the motion detection unit 64.

The motion detection unit 64 performs a head tracking process for detecting the posture of the head of the user wearing the HMD 100. The head tracking process is performed to link the posture of the user's head with the field of view displayed on the display panel 102 of the HMD 100, and in the head tracking process of the embodiment, the rotation angle of the HMD 100 with respect to the horizontal reference direction and the horizontal plane. The tilt angle and is detected. The horizontal reference direction may be set as, for example, the direction facing when the power of the HMD 100 is turned on.

The line-of-sight direction determination unit 66 determines the line-of-sight direction according to the posture of the HMD 100 detected by the motion detection unit 64. This line-of-sight direction is the line-of-sight direction of the user A, and by extension, the line-of-sight direction (optical axis direction) of the camera 14 of the robot 10 which is an alter ego.

In order to link the line-of-sight direction (optical axis direction) of the camera 14 with the line-of-sight direction of the user A, it is necessary to set the reference posture of the robot 10 in advance. FIG. 3 shows a state in which the first arc-shaped arm 32 and the second arc-shaped arm 34 are upright 90 degrees with respect to the pedestal 30, but this state is set as the horizontal direction and the power supply of the robot 10 is set. The direction in which the front surface of the housing 20 faces when is turned on may be set as the horizontal reference direction. The robot 10 may have an attitude sensor similar to the HMD 100 so that the horizontal direction can be set autonomously.

With the reference postures of the HMD 100 and the robot 10 set, the line-of-sight direction determination unit 66 directly determines the rotation angle and tilt angle detected by the motion detection unit 64 as the line-of-sight direction (optical axis direction) of the camera 14. Good. When the motion detection unit 64 detects the rotation angle and the tilt angle of the HMD 100, the line-of-sight direction determination unit 66 determines the line-of-sight direction of the HMD 100 as a vector (x, y, z) of three-dimensional coordinates, and at this time, the robot 10 The line-of-sight direction of the camera 14 may be determined to be the same (x, y, z), or may be determined as (x', y', z') with some correction.

The actuator control unit 68 controls the direction of the camera 14 so that the line-of-sight direction is determined by the line-of-sight direction determination unit 66. Specifically, the actuator control unit 68 adjusts the electric power supplied to the first motor 52, the second motor 54, and the third motor 56 to make the movement of the housing 20 follow the movement of the HMD 100. The motor drive control by the actuator control unit 68 is performed in real time, and therefore the orientation of the housing 20 is moved in the same manner as the direction of the line of sight of the user A.

According to the actuator device 12 of the embodiment, the housing 20 is driven with reference to the rotation centers of the first arc-shaped arm 32 and the second arc-shaped arm 34, and this movement shows the same movement as the human neck. .. The actuator device 12 reproduces the movement of the user A's neck with a simple structure in which two semicircular arms are crossed.

Next, the output system 24 will be described.
In the output system 24, the right camera 14a and the left camera 14b are directed in the direction controlled by the actuator device 12 and photograph within their respective angles of view. The right camera 14a and the left camera 14b may be spaced apart, for example, at the average distance between the eyes of an adult. The image data for the right eye taken by the right camera 14a and the image data for the left eye taken by the left camera 14b are transmitted from the transmission unit 90 to the HMD 100 and displayed on the right half and the left half of the display panel 102, respectively. These images form a parallax image viewed from the right eye and the left eye, and the display panel 102 is divided into two and displayed in each of the regions, so that the image can be viewed stereoscopically. In order for the user A to see the display panel 102 through the optical lens, the image processing unit 80 may generate image data in which the optical distortion caused by the lens is corrected in advance and supply the image data to the HMD 100.

The right camera 14a and the left camera 14b take pictures at a predetermined cycle (for example, 1/60 second), and the transmission unit 90 transmits the image data to the HMD 100 without delay. As a result, the user A can see the situation around the robot 10 in real time, and can see the desired direction by changing the direction of the face.

The camera posture information acquisition unit 70 acquires posture information including the posture and orientation of the camera 14 and supplies it to the transmission unit 90. The transmission unit 90 transmits the attitude information of the camera 14 to the HMD 100. The posture information of the camera 14 may be obtained by using, for example, the posture information detected by the motion sensor, or the position information of the motor mounted on the actuator device 12 that changes the posture and orientation of the housing 20 of the robot 10. You may ask from.

FIG. 7 is a functional configuration diagram of the projection unit 130 of FIG. The figure depicts a block diagram focusing on functions, and these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The reprojection unit 130 includes an HMD attitude information acquisition unit 132, a camera attitude information acquisition unit 134, a camera image acquisition unit 135, a difference calculation unit 136, a warning unit 137, and an image correction unit 138.

The HMD attitude information acquisition unit 132 acquires attitude information (referred to as “HMD attitude information”) including the attitude and orientation of the HMD 100 detected by the attitude sensor 124, and supplies it to the difference calculation unit 136. The camera posture information acquisition unit 134 acquires posture information (referred to as “camera posture information”) including the posture and orientation of the camera 14 at the time of shooting by the camera 14 received by the communication control unit 126 from the robot 10, and calculates the difference. Supply to unit 136. The camera image acquisition unit 135 acquires the captured images (referred to as “camera images”) of the left and right cameras 14 received from the robot 10 by the communication control unit 126 and supplies them to the image correction unit 138.

Here, the camera posture information may be embedded in the camera image as metadata. HDMI (registered trademark) 2.1 standard may be used as a method of embedding camera posture information in a camera image as metadata and transmitting the information. In the HDMI 2.1 standard, it is considered to insert the dynamic metadata to be applied to each frame into the VBlank signal of each frame data and transmit it. By embedding the camera attitude information in the dynamic metadata, the camera attitude information at the time of imaging by the camera 14 of the robot 10 can be synchronized with the frame data of the camera image and transmitted to the HMD 100. Here, an example of inserting dynamic metadata into a VBlank signal of the HDMI 2.1 standard has been described, but this is an example, and dynamic metadata may be inserted into some synchronization signal synchronized with each frame and transmitted. .. The camera posture information acquisition unit 134 extracts the camera posture information from the metadata inserted in the synchronization signal of each frame of the camera image received from the robot 10 by the communication control unit 126.

The difference calculation unit 136 calculates the difference between the HMD attitude information and the camera attitude information, and supplies the difference to the image correction unit 138. The image correction unit 138 executes the reprojection process by correcting the camera image so as to match the latest posture of the HMD 100 based on the calculated posture difference. The reprojected camera image is supplied to the control unit 120 and displayed on the display panel 102.

Based on the HMD posture information, it is determined that the HMD 100 has been moved beyond the movable range of the robot 10, or that the posture of the HMD 100 has approached a predetermined threshold value with respect to the limit value of the movable range of the robot 10. In this case, the difference calculation unit 136 interrupts or continues to instruct the image correction unit 138 to perform the reprojection process, and notifies the warning unit 137. The warning unit 137 displays a message on the display panel 102 indicating that the movable range has been exceeded or the movable limit is approaching. At that time, the direction in which the user wearing the HMD 100 should return the head may be displayed together with the warning message.

Specifically, the difference calculation unit 136 may determine whether or not the posture of the HMD 100 is within the movable range of the robot 10. For example, if the camera 14 of the robot 10 cannot be rotated 360 degrees and can be rotated only within a certain range, the range is set in advance in the HMD 100 as a movable range, and the HMD posture information is set to the movable range of the camera 14. Compare with range. If the posture of the HMD 100 and the posture of the camera 14 deviate significantly from each other without setting the movable range in advance, it may be determined that the movable range has been exceeded.

Further, since the posture of the camera 14 of the robot 10 is controlled by the motor, the posture cannot be changed as quickly as the user wearing the HMD 100. Therefore, when it is determined that the HMD 100 has been moved beyond the speed at which the camera 14 can follow the attitude change of the HMD 100 based on the HMD attitude information, the difference calculation unit 136 instructs the image correction unit 138 to perform the reprojection process. This is interrupted or continued, and the warning unit 137 is notified. The warning unit 137 displays a message on the display panel 102 prompting the user wearing the HMD 100 to slowly move his / her head.

Specifically, the difference calculation unit 136 determines that the followable speed of the camera 14 has been exceeded when the difference between the HMD attitude information and the camera attitude information exceeds the threshold value or when the change speed of the HMD attitude information exceeds the threshold value. .. When the posture of the HMD 100 moves faster than a predetermined threshold value, the difference between the HMD posture information and the camera posture information becomes large, so that it may be determined that the followable speed of the camera 14 has been exceeded.

In this way, when the movement of the HMD 100 approaches or exceeds the limit of the movable range and the followable speed of the camera 14, the camera image is corrected by the reprojection processing and the user handles the movable range of the camera 14. It may be misunderstood that the head can be moved even if the speed that can be followed is exceeded. In the present embodiment, the reprojection process is interrupted in such a case, and the warning unit 137 displays a warning on the display panel 102, whereby the movement of the head of the user wearing the HMD 100 can be appropriately regulated. it can. However, even in such a case, it is possible to display a warning while continuing the reprojection process without interrupting it.

The operation of the reprojection unit 130 of FIG. 7 will be described. For comparison, the conventional reprojection process will be described with reference to FIG. 8, and then the reprojection process of the present embodiment will be described with reference to FIG.

FIG. 8 is a diagram illustrating a conventional reprojection process. Here, instead of the robot 10, the rendering device 300 is connected to the HMD 100 via a network.

When the HMD100 has a head tracking function and the image is rendered by changing the viewpoint and the direction of the line of sight in conjunction with the movement of the user's head and displayed on the HMD100, there is a delay from the rendering of the image to the display. Occasionally, there is a discrepancy between the orientation of the user's head, which is assumed, and the orientation of the user's head when the image is displayed on the HMD 100, and the user may feel drunk. In addition, when passing through a network, a deviation occurs due to delay fluctuations due to the network. Therefore, by applying a correction process to the rendered image to compensate for the deviation between the posture of the HMD 100 used at the time of rendering and the latest posture of the HMD 100, the reprojection is executed, and the image after the reprojection is displayed on the HMD 100 by the user. Reduce sickness.

The HMD 100 predicts the movement based on the current posture information, and transmits the predicted HMD posture information 202 to the rendering device 300. The rendering device 300 uses the predicted HMD attitude information 202 to render an image to be displayed on the HMD 100. The rendering device 300 transmits the rendered image 310 to the HMD 100 together with the attitude information 204 used for rendering.

It should be noted that the attitude information 204 used for rendering here is the same as the predicted HMD attitude information 202. The rendering device 300 may predict the movement of the HMD 100 and generate the predicted HMD posture information 202.

The difference calculation unit 302 calculates the difference between the latest HMD posture information 200 and the posture information 204 used for rendering, and the reprojection unit 304 reprojects the rendered image 310 based on the calculated posture difference. Display on HMD100.

FIG. 9 is a diagram illustrating a reprojection process by the reprojection unit 130 of FIG. 7.

Motion prediction is performed based on the current posture information of the HMD 100, and the predicted HMD posture information 202 is transmitted to the robot 10. The robot 10 controls the posture and orientation of the camera 14 using the predicted HMD posture information 202, and captures an image with the camera 14. The robot 10 transmits the camera image 210 to the HMD 100 together with the camera posture information 205 at the time of shooting.

It should be noted here that the camera attitude information 205 at the time of shooting is different from the predicted HMD attitude information 202. This is because when the motor controls the posture of the camera 14 in the robot 10, an error of the motor, an operation delay, and the like occur, so that the posture of the camera 14 cannot be completely matched with the predicted posture of the HMD 100.

The difference calculation unit 136 calculates the difference between the latest HMD posture information 200 and the camera posture information 205 at the time of shooting. The image correction unit 138 reprojects the camera image 210 so as to match the latest posture of the HMD 100 based on the calculated posture difference, and displays the camera image 210 on the HMD 100. Strictly speaking, the latest HMD posture information 200 is not the current posture of the HMD 100, but is predicted by taking into consideration the time until the image displayed on the HMD 100 after the reprojection processing is seen by the user. The posture of the HMD 100 may be used.

The reprojection unit 130 of the present embodiment is different from the conventional reprojection process in that the reprojection is performed using the camera posture information 205 at the time of shooting instead of the predicted HMD posture information 202.

The reprojection process using the camera posture information 205 at the time of shooting eliminates the deviation caused by the posture of the camera 14 not matching the posture of the HMD 100 due to the error of the motor and the operation delay, and the user wearing the HMD 100 feels uncomfortable. Can be reduced.

Since the movement of the camera 14 controlled by the motor is less accurate than the movement of the human head, the moving image captured by the camera 14 is not smooth. The reprojection processing also has the effect of smoothing the movement of the moving image captured by the camera 14.

Further, when the predicted HMD posture information 202 is transmitted to the robot 10, packet loss may occur depending on the congestion state of the network, and the predicted HMD posture information 202 may be lost without being transmitted to the robot 10. Even in such a case, since the posture information transmitted to the HMD 100 together with the camera image 210 is the camera posture information 205 at the time of shooting, even if the predicted HMD posture information 202 is missing, it is based on the camera posture information 205 at the time of shooting. There is also an advantage that the reprojection process can be performed reliably.

It should be noted that the posture of the camera 14 at the time of shooting is estimated by self-position estimation from the camera image 210 by using a technique such as SLAM (Simultaneous Localization and Mapping) without transmitting the camera posture information 205 at the time of shooting together with the camera image 210. It may be configured to estimate.

FIG. 10 is a flowchart illustrating a procedure of the reprojection process by the reprojection unit 130 of FIG. 7.

The camera image acquisition unit 135 receives the camera image from the robot 10, and the camera attitude information acquisition unit 134 acquires the camera attitude information added as metadata to the camera image (S10).

The HMD attitude information acquisition unit 132 acquires the latest HMD attitude information detected by the attitude sensor 124 (S12).

The difference calculation unit 136 calculates the difference between the latest HMD attitude information and the camera attitude information (S14).

The difference calculation unit 136 determines whether or not the latest HMD posture information is within the movable range of the camera 14 of the robot 10 (S16), and if it is within the movable range (Y of S16), proceeds to step S20.

When the latest HMD posture information approaches or exceeds the limit value of the movable range of the camera 14 of the robot 10 (N in S16), the warning unit 137 moves the HMD 100 within the movable range of the robot 10. A warning message for guiding the user to fit in the display panel 102 is displayed on the display panel 102 (S18).

The difference calculation unit 136 determines whether or not the posture change of the HMD 100 is within the followable speed of the camera 14 of the robot 10 (S20), and if it is within the followable speed (Y of S20), proceeds to step S24.

When the attitude change of the HMD 100 approaches the limit value of the followable speed of the robot 10 or exceeds the followable speed (N of S20), the warning unit 137 changes the attitude of the HMD 100 within the followable speed of the robot 10. A warning message for guiding the user to fit in the display panel 102 is displayed on the display panel 102 (S22).

The image correction unit 138 executes the reprojection process by correcting the camera image based on the difference between the latest HMD posture information and the camera posture information (S24).

The present invention has been described above based on the embodiments. Embodiments are examples, and it will be understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. .. An example of such a modification will be described.

In the above embodiment, the reprojection unit 130 is provided in the HMD 100, but the reprojection unit 130 may be provided in the client device to which the HMD 100 is connected.

In the above embodiment, the case where the HMD 100 and the robot 10 are connected via the network and the robot 10 is remotely controlled has been described, but the same applies even when the HMD 100 and the robot 10 are connected without passing through the network. Reprojection processing can be applied. Since there is a motor delay and error, even if there is no delay fluctuation due to the network, the attitude of the camera 14 and the attitude of the HMD 100 will deviate, so the reprojection process that corrects the camera image according to the attitude of the HMD 100 is effective. is there. Further, the same reprojection processing can be applied not only to the robot 10 but also to any control device equipped with the camera 14.

In the above embodiment, the case where the robot 10 itself does not move and the camera 14 mounted on the robot 10 changes its posture has been described, but the same reprojection processing is performed even when the robot 10 itself changes its position. Can be applied. In that case, the absolute position information of the camera is acquired not only from the camera posture by the motion sensor but also from the position information of the servo mechanism and transmitted together with the camera image. When the camera 14 moves, a self-position estimation technique that estimates the position and orientation of the camera 14 on the environment map, such as SLAM, can also be used. The position information of the servo mechanism and the self-position estimation technique such as SLAM can be used to acquire the attitude information of the camera 14 even when the robot 10 does not change the position.

1 Information processing system, 10 Robot, 12 Actuator device, 14a Right camera, 14b Left camera, 20 Housing, 22 Input system, 24 Output system, 30 Pedestal, 32 1st arc arm, 34 2nd arc arm, 36 Housing, 38 cover, 40 legs, 42 insertion member, 50 drive mechanism, 52 1st motor, 54 2nd motor, 56 3rd motor, 60 receiver, 62 sensor information acquisition unit, 64 motion detection unit, 66 line-of-sight direction Decision unit, 68 Actuator control unit, 70 Camera attitude information acquisition unit, 80 Image processing unit, 90 Transmission unit, 100 HMD, 102 Display panel, 120 Control unit, 122 Storage unit, 124 Attitude sensor, 126 Communication control unit, 130 Projection unit, 132 HMD posture information acquisition unit, 134 camera attitude information acquisition unit, 135 camera image acquisition unit, 136 difference calculation unit, 137 warning unit, 138 image correction unit.

It can be applied to the technology to generate images.

Claims (6)

A camera image acquisition unit that acquires images taken by a camera that can change its posture in conjunction with a change in the posture of the head-mounted display.
A camera posture information acquisition unit that acquires posture information during shooting by the camera, and a camera posture information acquisition unit.
A difference calculation unit that calculates the difference between the posture information of the head-mounted display and the posture information at the time of shooting by the camera, and
An image generation device including an image correction unit that corrects the captured image so as to match the posture of the head-mounted display based on the calculated difference. The image generation device according to claim 1, wherein the camera posture information acquisition unit extracts posture information at the time of shooting by the camera from metadata inserted in a synchronization signal of each frame of the shot image. When it is determined that the posture of the head-mounted display approaches a predetermined threshold value with respect to the limit value of the movable range of the camera or exceeds the movable range of the camera based on the posture information of the head-mounted display, the above-mentioned The image generation device according to claim 1 or 2, further comprising a warning unit that interrupts or continues the correction of the captured image by the image correction unit and displays a warning. When it is determined that the posture change of the head-mounted display exceeds the followable speed of the camera based on the posture information of the head-mounted display, the correction of the captured image by the image correction unit is interrupted or continued, and a warning is issued. The image generator according to claim 1 or 2, further comprising a warning unit to be displayed. The camera image acquisition step to acquire the image taken by the camera that can change the attitude in conjunction with the attitude change of the head-mounted display,
The camera posture information acquisition step for acquiring the posture information at the time of shooting by the camera, and
A difference calculation step for calculating the difference between the posture information of the head-mounted display and the posture information at the time of shooting by the camera, and
An image generation method comprising an image correction step of correcting the captured image so as to match the posture of the head-mounted display based on the calculated difference. A camera image acquisition function that acquires images taken by a camera that can change the attitude in conjunction with the attitude change of the head-mounted display,
The camera posture information acquisition function that acquires the posture information at the time of shooting by the camera, and
A difference calculation function that calculates the difference between the posture information of the head-mounted display and the posture information at the time of shooting by the camera, and
A program characterized in that a computer realizes an image correction function that corrects the captured image so as to match the posture of the head-mounted display based on the calculated difference.

Images