JP2019022014A - Image providing method, image providing device and computer program - Google Patents

Abstract

To impart a feeling of presence as if a person were moving although the person is stationary.SOLUTION: An image providing method comprises: a mode determination step of determining, as an operation mode of a device itself, which mode the imaging device operates in between a first mode in which an image picked up by an imaging device installed at a selected spot is provided among images picked up by imaging devices which are installed at respective spots and can pick up images over wide ranges and a second mode in which an image of a route to an imaging device at a movement destination from an imaging device having picked up an image that a user currently views so as to switch an image picked up by an imaging device at another spot is provided; an image processing step of performing image processing on the image of the route according to a movement of the head of the user in the second mode; and an image output step of providing an image after the image processing for a display device that the user puts on.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image providing method, an image providing apparatus, and a computer program.

  In recent years, omnidirectional cameras have been widely used. The omnidirectional camera is a camera that can shoot 360-degree panoramic images / videos in all directions, including up / down / left / right / front / back, or part of the position. In addition, omnidirectional cameras have various forms, such as those with very wide-angle lenses (fisheye lenses, etc.) and those that combine images and videos from multiple cameras to generate a single panoramic video. There is. Images taken with an omnidirectional camera are called omnidirectional images, and images taken with an omnidirectional camera are called omnidirectional images, and the surroundings of the camera are recorded in a highly realistic manner. It is starting to be widely used as a device to operate. In the future, the explanation will proceed as an omnidirectional image under the interpretation that the omnidirectional video is composed of a plurality of frames of omnidirectional images.

  Furthermore, HMD (Head Mounted Display) has started to spread as a tool for viewing omnidirectional images. The HMD is worn so as to cover both eyes of the user and presents the omnidirectional image in the direction that the user wants to see from among the omnidirectional images. It is a display that can give the user a unique feeling.

  When cameras and displays like the one above start to spread, the user changes the viewing position by switching the omnidirectional images taken by the omnidirectional cameras installed at multiple points. The desire to enjoy omnidirectional images at multiple points is created with the sense that the user moves the viewing position. For example, in a sports venue such as a soccer field, a plurality of omnidirectional cameras are installed so as to surround the field, and omnidirectional images captured by each omnidirectional camera are synchronized and distributed. To do. The user can select an omnidirectional image at a desired position from the distributed omnidirectional images, and can view it using the HMD as if he / she was on the spot. By further selecting an omnidirectional image taken with a certain omnidirectional camera, you can see the omnidirectional image from another viewpoint as if you moved to that position. .

  In recent years, research and services that allow users to select and view such omnidirectional images from multiple locations have emerged. For example, in Google Street View of Google Inc. in the United States, people can continuously view omnidirectional images taken at regular intervals along the road, including omnidirectional images of routes accompanying movement. Is a service that allows you to move to a desired point while grasping the direction and distance of movement, and to feel as if you are walking around a remote city, such as looking at a spherical image as if you were looking around the point. Is deployed. In addition, the VR Planet technique described in Non-Patent Document 1 arranges omnidirectional images taken at various points in a virtual space, and moves between multiple omnidirectional images using human gestures or the like. By doing so, it is possible to obtain a feeling as if moving around a plurality of points.

Kevin Fan, Daiya Kato, Liwei Chan, Kouta Minamizawa, Masahiko Inami, “VR Planet: Interface for Meta-View and Feet Interaction of VR Contents,” In Proceedings of SIGGRAPH 2016, 2016.

  However, there is still a problem with the conventional method when using the above method. This will be described using the example of the previous soccer field. While a person is paying attention to one player in a celestial sphere image taken from there, the person is on the other side of the field (all images taken from another position). Suppose you want to see the same object from a celestial sphere. It is assumed that the three-dimensional positional relationship of each omnidirectional camera placed around the field is known by prior calibration. In addition, it is assumed that an omnidirectional image between the omnidirectional cameras to be presented in order for a person to feel a sense of movement is prepared.

  In this case, ideally, it is most intuitive for a person to virtually walk around the path between the omnidirectional cameras and watch the player in the field from a favorite viewpoint position. However, this requires a space equivalent to a soccer field. Of course, as with Google Street View, it is possible to assign the moving means to another operation (for example, a means to move with the arrow keys of the keyboard), but the intuition of watching while moving is greatly reduced. In addition to reducing the sense of realism, if you specify only the direction you want to move by operating the keyboard (the direction you are looking at before moving is not maintained), attention will be focused on after moving or while moving There was a problem of losing sight of the player who was playing.

  In view of the above circumstances, an object of the present invention is to provide a technique capable of giving a sense of reality as if a person is moving without moving.

  According to one aspect of the present invention, as an operation mode of the own device, an image captured by the imaging device installed at a selected point among images captured by an imaging device capable of imaging a wide range installed at each point is displayed. A first mode to be provided and an image of a route from the imaging device that has captured the image currently viewed by the user to the destination imaging device in order to switch to an image captured by the imaging device at another point A mode determining step for determining which mode to operate in the second mode, and an image for performing image processing on the image of the route in accordance with the motion of the user's head in the second mode. An image providing method comprising: a processing step; and an image output step of providing an image after image processing to a display device worn by the user.

  One aspect of the present invention is the above-described image providing method, in which an image of a path from an imaging device that captures an image viewed by a user to a destination imaging device is captured in advance while moving with the imaging device. It is an image.

  One aspect of the present invention is the above-described image providing method, wherein in the image processing step, when operating in the second mode, an image of a route provided to the destination is displayed on the head of the user. Image processing is performed according to the amount of rotation.

  One aspect of the present invention is the image providing method described above, wherein, in the image processing step, the amount of rotation of the head is the same as the amount of rotation performed in real space.

  One aspect of the present invention is the above-described image providing method, wherein in the image processing step, the rotation amount of the head is a rotation amount obtained by multiplying a rotation amount performed in real space by an arbitrary function.

  One aspect of the present invention is the above-described image providing method, wherein in the image processing step, three-dimensional coordinates of a gaze target being gaze based on a convergence angle of the eyeball of the user are detected, and the gaze target is the user Image processing is performed so as to be positioned at the center of the field of view.

  According to one aspect of the present invention, as an operation mode of the own device, an image captured by the imaging device installed at a selected point among images captured by an imaging device capable of imaging a wide range installed at each point is displayed. A first mode to be provided and an image of a route from the imaging device that has captured the image currently viewed by the user to the destination imaging device in order to switch to an image captured by the imaging device at another point An image for performing image processing on the image of the path according to the movement of the user's head in the second mode; An image providing apparatus comprising: a processing unit; and an image output unit that provides an image after image processing to a display device worn by the user.

  According to one aspect of the present invention, as an operation mode of the own device, an image captured by the imaging device installed at a selected point among images captured by an imaging device capable of imaging a wide range installed at each point is displayed. A first mode to be provided and an image of a route from the imaging device that has captured the image currently viewed by the user to the destination imaging device in order to switch to an image captured by the imaging device at another point A mode determining step for determining which mode to operate in the second mode, and an image for performing image processing on the image of the route in accordance with the motion of the user's head in the second mode. A computer program for causing a computer to execute a processing step and an image output step of providing an image after image processing to a display device worn by the user

  According to the present invention, it is possible to give a sense of reality as if a person is moving without moving.

It is a block diagram showing the system configuration | structure of the image provision system 100 in this invention. It is a figure which shows the specific example of a route list. It is a figure for demonstrating the convergence angle demonstrated by (summary). It is a figure for demonstrating the operation | movement at the time of the 2nd mode in 1st Embodiment. It is a figure which shows an example of the omnidirectional image which the user 40 sees according to head rotation operation | movement. It is a flowchart which shows the flow of 10 processes in 1st Embodiment. It is a figure for demonstrating 2nd Embodiment. It is a figure for demonstrating 3rd Embodiment. It is a figure for demonstrating 4th Embodiment. It is a figure which shows the specific example of a path | route list | wrist in case there exist two or more paths like 4th Embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(Overview)
First, an outline of the present invention will be described.
The image providing apparatus in the image providing system according to the present invention is a celestial sphere in a scene where an omnidirectional image (including video) is viewed at a point where an omnidirectional camera capable of imaging a wide range (for example, all directions) is installed. A first mode in which a celestial sphere image can be viewed using a display device such as an HMD while facing a user's preferred direction at a point where the camera is installed, and is captured by an omnidirectional camera at another point. In order to switch to the omnidirectional image, a second omnidirectional image of the path from the omnidirectional camera that captured the omnidirectional image currently being viewed by the person to the omnidirectional camera to be moved can be viewed. There are two modes: mode.

  In the first mode, a person can select and view a omnidirectional image captured by the omnidirectional camera using a display device at a favorite position as in the past. In the second mode, it is possible to move between the omnidirectional cameras. Specifically, in the second mode, an omnidirectional image of a moving path from a certain omnidirectional camera to a destination omnidirectional camera is provided and captured by the destination omnidirectional camera. By providing an omnidirectional image, it is possible to give a person a feeling as if they moved between the omnidirectional cameras. It is assumed that the omnidirectional image of the movement route provided at the time of movement is previously captured by an omnidirectional camera.

  In the second mode, in order to make a person feel a sense of movement while on the spot, the movement of the person looking at the surroundings while moving normally (the movement of rotating the head while moving), except for the actual movement of the place, By providing an omnidirectional image according to the rotational movement of the head, it is possible to give a sense of reality to a person in a small space without a person actually moving in a space such as a wide field. In the second mode, an operation in which a person rotates his / her head while paying attention to a subject of interest (hereinafter referred to as “target of interest”) is assigned to “movement” between omnidirectional cameras.

However, one problem here is that if you always want to keep the subject of interest at the center of the field of view during moving observation (you want to keep an eye on the subject of interest), where the subject of interest is located in the 3D space In addition, after knowing from which position the person is looking at the target object, a part of the omnidirectional image presented to the person in which the target object appears must be brought to the center of the field of view. To solve this problem, measure the part of the person in the three-dimensional space from the gaze direction of the two eyes of the person and the angle of convergence formed by them, and select one point in the three-dimensional space. A technique that moves around the CG space and looks around is used (see Reference 1).
(Reference 1: Yun Suen Pai, Benjamin Outram, Noriyasu Vontin, and Kai Kunze, “Transparent Reality: Using Eye Gaze Focus Depth as Interaction Modality,” In Proceedings of UIST 2016, 2016.)

  By using the technique described in Reference 1, an omnidirectional camera with an intuitive moving action of turning the head (shaking the head) while keeping the subject of interest at the center of the field of view even in a live action such as an omnidirectional image It is possible to see the object of interest that you want to see from your favorite place / angle while feeling as if you were moving in a wide space like a field.

(Details)
FIG. 1 is a configuration diagram showing a system configuration of an image providing system 100 according to the present invention. The image providing system 100 includes a distribution server 10, an image processing device 20, and a display device 30.
The distribution server 10 includes an omnidirectional image captured by an omnidirectional camera installed at each point and an omnidirectional image captured on a moving path between the omnidirectional cameras (hereinafter referred to as “path omnidirectional image”). And a route list. In the route list, the initial position at the start of observation (current location), the movable route (destination), the amount of head rotation required for movement, and the means required for mode switching are described in association with each other. It is a list. The distribution server 10 distributes a corresponding omnidirectional image to the image processing apparatus 20 in response to a request from the image processing apparatus 20.

  Although FIG. 1 shows a case where the distribution server 10 and the image processing apparatus 20 are separate enclosures, the distribution server 10 and the image processing apparatus 20 may be integrated. In this case, the distribution server 10 corresponds to the storage of the image processing apparatus 20. The distribution server 10 distributes the image to the image processing device 20 at the first stage when a person starts to view the image. The first stage when the person starts to see the image is, for example, the timing when the image processing apparatus 20 is turned on. In addition, the omnidirectional images of multiple points and the omnidirectional images of the route are stored and distributed in a state in which they are aligned in a uniquely determined direction (for example, in a state in which they are aligned in advance with respect to east, west, south, and north) Shall be.

The image processing device 20 is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 20 and the display device 30 may be separate or integrated. The image processing device 20 performs image processing on the image distributed from the distribution server 10, and provides the image after image processing to the display device 30. The image processing apparatus 20 is an aspect of an image providing apparatus.
The display device 30 is a display device such as a head mounted display. The display device 30 displays the image provided from the image processing device 20. The display device 30 is attached to the user 40.

(First embodiment)
Next, functional configurations of the image processing device 20 and the display device 30 in the first embodiment will be described. First, the functional configuration of the image processing apparatus 20 will be described. The image processing apparatus 20 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 20 functions as an apparatus including an image request / reception unit 201, an image rendering unit 202, an image output unit 203, and a mode determination unit 204. Note that all or part of the functions of the image processing apparatus 20 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). . The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image request / reception unit 201 requests an image from the distribution server 10 according to the mode determined by the mode determination unit 204, and receives the image transmitted from the distribution server 10. Specifically, the image request / reception unit 201 has one omnidirectional image among a plurality of omnidirectional images in the first mode, and one route in the route in the second mode. The distribution server 10 is requested and received for the route omnidirectional image corresponding to. The image request / reception unit 201 outputs the omnidirectional image obtained from the distribution server 10 and information indicating which mode the omnidirectional image is to the image rendering unit 202. Further, the image request / reception unit 201 receives a route list from the image processing apparatus 20 and holds the route list at the first stage when a person starts viewing an image.

  The image rendering unit 202 performs rendering by changing the rendering method of the image provided via the image output unit 203 based on the input omnidirectional image. When the input omnidirectional image is one of a plurality of omnidirectional images, the image rendering unit 202 is directed to the direction of the head of the user 40 obtained from the head direction detection unit 301 included in the display device 30. Is output to the display device 30 as a result of rendering the image having the size of the image output unit 203 corresponding to the head direction in the omnidirectional images at a plurality of points. When the input omnidirectional image is any one of the path omnidirectional images, the image rendering unit 202 determines the head direction of the user 40 obtained from the head direction detection unit 301 included in the display device 30. And a three-dimensional position of a person's attention target obtained from the eyeball direction detection unit 302 included in the display device 30, and a path corresponding to the amount of the rotation angle of the head calculated from the head direction data. A rendering result of an image of the size of the image output unit 203 corresponding to the direction in which the user 40 is looking at the target object of the user 40 is output to the display device 30 as an omnidirectional image from the advanced position.

The image output unit 203 displays the rendering result of the image rendered by the image rendering unit 202 on the display unit 303 in the display device 30.
The mode determination unit 204 determines whether to operate in the first mode or the second mode based on the route list in accordance with an input from the user 40. The image output unit 203 outputs information indicating the determined operation mode to the image request / reception unit 201. The mode determination unit 204 is configured using an existing input device such as a keyboard, a pointing device (mouse, tablet, etc.), a touch panel, and buttons. The mode determination unit 204 is operated by the user when inputting a user instruction to the image processing apparatus 20. The mode determination unit 204 receives an input of a mode switching instruction. The mode determination unit 204 may be an interface for connecting the input device to the image processing device 20. In this case, the image processing apparatus 20 inputs an input signal generated according to a user input in the input device to the mode determination unit 204.

Next, the functional configuration of the display device 30 will be described. The display device 30 includes a head direction detection unit 301, an eyeball direction detection unit 302, and a display unit 303.
The head direction detection unit 301 is a sensor that detects the direction in which the user 40 faces. The head direction detection unit 301 is, for example, a gyro sensor or an acceleration sensor. The head direction detection unit 301 outputs the direction data of the human head to the image rendering unit 202.
The eyeball direction detection unit 302 calculates the gaze direction of each eyeball of the person and the convergence angle from both eyes, and calculates the three-dimensional position of the person's attention target. The eyeball direction detection unit 302 outputs the three-dimensional position of the person's attention target to the image rendering unit 202.
The display unit 303 displays the image output from the image output unit 203.

  FIG. 2 is a diagram illustrating a specific example of a route list. As shown in FIG. 2, the route list describes the current location, the amount of head rotation required for movement, the destination, and each value of mode switching means. The value at the current point represents the position at the start of observation. Specifically, the value of the current location represents the position of the omnidirectional camera that captured the omnidirectional image viewed by a person. The value of the amount of rotation of the head necessary for movement represents the amount of rotation of the head necessary for movement between the omnidirectional cameras. Specifically, the value of the amount of rotation of the head necessary for movement represents the amount of rotation in the horizontal direction. The movement destination represents the position of the movement destination. Specifically, the value of the movement destination represents the position of the omnidirectional camera set as the movement destination when the head is rotated by the amount of rotation of the head necessary for movement from the current position. The mode switching means represents a method for switching the operation mode of the image processing apparatus 20 from the first mode to the second mode, or from the second mode to the first mode.

  In the example shown in FIG. 2, a plurality of current points are described in the route list. These current points are “A”, “B”, “B”, and “C”. In FIG. 2, the record described at the top of the route list has a current point value “A”, a head rotation amount necessary for movement “θ1”, a destination value “B”, The value of the mode switching means is “space key pressed”. That is, when the position of the omnidirectional camera that captured the omnidirectional image that the person is viewing is the position “A”, the movement destination when the person rotates the head “θ1” is “B”. It shows that the method for switching the mode is “pressing the space key”. In FIG. 2, “space key pressing” is described as an example of a method for switching the operation mode. However, the method for switching the operation mode is not limited to this. For example, the method for switching the operation mode may be a combination of a plurality of buttons, or a time during which the buttons are pressed.

Next, the convergence angle described in (Overview) will be described with reference to FIG.
FIG. 3A is a diagram illustrating an example of the case where the user 40 is gazing at the target object 50 at a distant position, and FIG. 3B is the user 40 gazing at the target object 50 at a close position. It is a figure which shows an example of a case. As shown in FIG. 3, the convergence angle is smaller when the user 40 is gazing at the attention object 50 at a far position than when the user 40 is gazing at the attention object 50 at a near position. Become.

  FIG. 4 is a diagram for explaining the operation in the second mode in the first embodiment. The omnidirectional images distributed in synchronism with the omnidirectional cameras 60-1, 60-2, 60-3 installed at three points (A point, B point, C point) on the road 41 are Consider a scene in which the user 40 sees the virtual omnidirectional images D and E captured while moving the sphere camera in advance while moving virtually. It is assumed that all omnidirectional images are stored in the distribution server 10 so that they can be virtually viewed on the display device 30 worn by the user 40. Further, in FIG. 4, it is assumed that the user 40 moves while paying attention to the attention object 50 beside the road 41. If the point B is in front of the target object 50, the point A exists at a position that forms an angle of θ1, and the point C exists at an angle of θ2. That is, the omnidirectional camera 60-2 is arranged in front of the target object 50, the omnidirectional camera 60-1 is arranged at a position that makes an angle of θ1, and the omnidirectional camera 60-3 makes an angle of θ2. Placed in position.

  If it is a real space, the user 40 performs an operation of rotating his / her head while moving on the road 41 from the point A to the point C and viewing the target object 50. On the other hand, in the image provision in the image provision system 100 according to the present invention, the user 40 does not actually move, but is performed in such a manner that the moving operation in the real space is assigned to the rotation of the user 40's head. FIG. 5 shows a schematic diagram of the head rotation operation performed by the actual user 40 and the omnidirectional image viewed by the user 40 by the method according to the present embodiment.

  The user 40 who has started to view the image at the point A can view the periphery by facing the favorite direction at the point A because the point A is captured as a spherical image (first mode). When the user 40 wants to move while looking at the target 50 at a certain timing, the mode switching means (for example, the space key of the keyboard) is input in the direction of the target 50. At this time, the mode determination unit 204 refers to the route list, and if the mode switching unit specified in the route list at the current location where the mode switching unit is made matches the executed mode switching unit, The operation mode is switched from the first mode to the second mode.

  When the mode is switched to the second mode, the distance and direction from the user 40 to the target 50 are first measured from the convergence angle obtained from the eyeball direction detection unit 302. In the present invention, since the user 40 does not actually move, the head 40 rotates instead of moving. By continuing the same convergence angle calculation process in conjunction with the rotation of the head, the rotation of the head (virtual movement operation) while always capturing the target object 50 within the visual field of the user 40 (here, the center of the display device 30). ) Can continue. Since the route omnidirectional image D is captured as a video, the frame of the omnidirectional image at the corresponding position is actually presented to the user 40 as a still image in accordance with the rotation angle of the head.

  During the second mode, the user 40 can advance or return the head rotation. When returning, the corresponding frame may be displayed in the form of rewinding the route omnidirectional image D. Also, if the user 40 presses the space key again to return from the second mode to the first mode, the omnidirectional video frame of the route omnidirectional image D at that point can be viewed as a still image. In this case, the user 40 can stop at a favorite position and look around the space.

  When the user 40 rotates the head by θ1 and switches the mode to the first mode, the omnidirectional camera 60-2 is installed at the point B, and the omnidirectional image is taken. 40 can enjoy watching the video. In a similar operation, the user 40 can go to the point C by rotating the head, passing the path E, and further rotating the head by θ2. Note that the head rotation amount starts to be measured immediately after the mode is switched to the second mode, and in this embodiment, the movable direction is a direction along the road that can be regarded as a straight line to some extent. The rotation amount of the head to be used is the rotation amount (angle) of the head in the component only in the horizontal direction.

  However, when the movement direction to be handled is a slope (having a certain angle) or the upper direction of the user 40 (the direction of flying in the sky), the head rotation direction may be adjusted to that direction. It is not limited to the horizontal direction. On the other hand, even when the omnidirectional image of the route ahead of the moving direction is not a road that can be regarded as a straight line (in the case of a winding route, etc.), the amount of rotation and direction of the head necessary for movement are uniquely determined. Application is also conceivable (for example, it is possible to use a component only in the horizontal direction and follow a winding road).

  By performing such an operation, the user 40 can watch the subject that he / she wants to see virtually from a remote location while moving the location without going to the actual location. At this time, the user 40 does not actually move the position, but only the head is rotated, so a large space is not required. In addition, although the moving operation is omitted while viewing while moving, the head rotation operation remains, so it is more intuitive than moving by simple key operation, and the subject of interest can always be seen in the field of view even when moving. There is an advantage that.

FIG. 6 is a flowchart showing the flow of 10 processes in the first embodiment.
The image request / reception unit 201 determines the operation mode of the own apparatus (step S101). Specifically, the image request / reception unit 201 determines whether the operation mode of the own device is the first operation mode based on the information indicating the operation mode of the own device output from the mode determination unit 204. It is determined whether it is an operation mode. When the operation mode of the own apparatus is the first operation mode (step S101—first operation mode), the image request / reception unit 201 requests the distribution server 10 for the omnidirectional image at the position where the user is viewing. . Thereafter, the image request / reception unit 201 receives the omnidirectional image distributed from the distribution server 10 (step S102). The image request / reception unit 201 outputs the received omnidirectional image to the image rendering unit 202.

  The image rendering unit 202 acquires information on the user's head direction from the head direction detection unit 301 (step S103). Further, the image rendering unit 202 acquires the omnidirectional image output from the image request / reception unit 201. The image rendering unit 202 renders the acquired omnidirectional image according to the head direction indicated by the acquired information on the head direction of the user. The image rendering unit 202 outputs the rendered omnidirectional image to the image output unit 203. Thereafter, the image output unit 203 outputs the rendered omnidirectional image output from the image rendering unit 202 to the display device 30.

The image processing apparatus 20 determines whether the browsing of the image is complete (step S104). When the image browsing is ended (step S104—YES), the image processing apparatus 20 ends the provision of the omnidirectional image.
On the other hand, when the browsing of the image is not complete (step S104—NO), the image processing apparatus 20 executes the processing after step S101.

  In the process of step S101, when the operation mode of the own apparatus is the second operation mode (step S101-second operation mode), the image request / reception unit 201 distributes the spherical image in the direction in which the user moves. Request to server 10. Thereafter, the image request / reception unit 201 receives the omnidirectional image distributed from the distribution server 10 (step S106). At this time, the image request / reception unit 201 receives the route omnidirectional image from the distribution server 10. The image request / reception unit 201 outputs the received omnidirectional image to the image rendering unit 202. The image rendering unit 202 acquires information on the three-dimensional position of the target object 50 of the user from the eyeball direction detection unit 302 (step S107).

  The image rendering unit 202 acquires information on the user's head direction from the head direction detection unit 301. The image rendering unit 202 calculates the rotation amount of the head from the acquired information on the head direction of the user (step S108). The amount of rotation of the head calculated in the first embodiment is the same as the amount of rotation of the head by the user in real space. The image rendering unit 202 renders an omnidirectional image corresponding to the target object 50 at a position that has traveled a path corresponding to the calculated amount of rotation of the head. The image rendering unit 202 outputs the rendered omnidirectional image to the image output unit 203. Thereafter, the image output unit 203 outputs the rendered omnidirectional image output from the image rendering unit 202 to the display device 30 (step S109).

The image processing apparatus 20 determines whether the browsing of the image is complete (step S110). When the browsing of the image is complete (step S110-YES), the image processing apparatus 20 ends the provision of the omnidirectional image.
On the other hand, when the browsing of the image is not finished (step S110—NO), the image processing apparatus 20 executes the processing after step S101.

  According to the image processing apparatus 20 configured as described above, it is possible to give a sense of reality to a person without the person moving. Specifically, the image processing apparatus 20 has a first mode in which an image of a point desired by the user can be viewed and a second mode in which the user can move between omnidirectional cameras. In the second mode, the image processing device 20 provides an image of a destination position corresponding to the amount of rotation of the user's head. That is, the user can see an image of the place moved from the present time by rotating his head. Therefore, the user does not need to actually move. In addition, it is possible to give the user a feeling of actually moving. Therefore, it is possible to give a sense of reality as if the person is moving without moving.

  In sports watching held in a wide space such as a field, even when omnidirectional images of a plurality of points are virtually moved and viewed, it is not necessary to move in the actual space. Therefore, it is possible to watch sports watching even in a narrow space. In addition, when looking at the target object 50 while moving, it is possible to obtain the three-dimensional position of the target object from the convergence angle of both eyes without losing sight of the object by a natural and intuitive operation of rotating the head. It is possible to move while keeping it at the center of the line of sight.

(Second Embodiment)
In the first embodiment, the user 40 needs to perform the head rotation by an amount necessary for actually continuing to pay attention to the target object 50 in the head rotation operation for movement. However, if the amount of movement is large, the amount of rotation of the head also increases, which is a burden on the person. Therefore, in the second embodiment, a configuration in which the required head rotation amount is made smaller than the actual amount (for example, half the actual amount) will be described.

  FIG. 7 is a diagram for explaining the second embodiment. As shown in FIG. 7, in the second embodiment, when the user moves the omnidirectional images D and E in the second mode virtually by moving the head while viewing the image, If the head is rotated by θ1 / 2, it is possible to move from point A to point B. Similarly, if the head rotation of θ2 / 2 is performed, it is possible to move from point B to point C. Here, an example has been shown in which the actual movement can be performed with an amount that is half the amount of head rotation, but the value that can be set is not limited to this. It is also possible to make it 1/10. Furthermore, it is not always the same set value, for example, the head rotation amount is larger as it is closer to point A, and virtual movement is possible even if the head rotation amount is smaller as it is closer to point B. It is also possible to make the set value to be multiplied non-linear on the route.

  According to the image providing system 100 in the second embodiment configured as described above, the burden on the user can be reduced more than in the first embodiment. For example, when assigning the amount of rotation of the head and the amount of movement in the virtual space, the association is changed and set (for example, it is possible to move a meter with a rotation angle of the head of θ / 2). A person can prevent his / her head from rotating more than necessary, and the user can see the image while moving virtually while reducing fatigue of the user.

(Third embodiment)
In 1st Embodiment and 2nd Embodiment, the user 40 showed the example which moves on the straight road 41 virtually. On the other hand, in the third embodiment, a configuration in the case of applying to a place where the moving route is not a straight line, such as watching a stadium, will be described.
FIG. 8 is a diagram for explaining the third embodiment. An omnidirectional image that is distributed in synchronism by the omnidirectional cameras 60-1, 60-2, 60-3 installed at three points on the road 41 (A point, B point, C point) A scene in which a person sees a virtual moving image is considered by using the path omnidirectional images D and E taken while moving the omnidirectional camera in advance.

  The conditions are the same as those in the first embodiment except that the route is not a straight line. In this embodiment, the state of a person at one point on each of the routes D and E is shown as an example. In the present embodiment, it is assumed that the person moves while paying attention to the target object 50 in the field 62. In addition, since the route omnidirectional images D and E are taken in advance, the current state of the target object 50 is not taken. Here, using another appropriate technique (for example, human area identification, human area extraction, etc.), the image of the subject of interest cut out from the nearest camera among the omnidirectional cameras at both ends of the path. It is assumed that they are displayed superimposed. In addition, although synchronized images can be seen at the points A, B, and C, it is possible to simplify the display of only the pre-captured images by dividing “moving” on the travel paths D and E. .

  According to the image providing system 100 in the third embodiment configured as described above, the situation setting gives the person a sense of realism without moving in the same manner as in the first embodiment. It becomes possible.

(Fourth embodiment)
In the first to third embodiments, a configuration in which a person can move from one point to one or two (only one route to move to another point) is shown. On the other hand, in the fourth embodiment, a configuration when there are two or more routes will be described.
FIG. 9 is a diagram for explaining the fourth embodiment. FIG. 10 is a diagram illustrating a specific example of a route list when there are two or more routes as in the fourth embodiment. FIG. 10 shows only when the current location is “C”. The configuration of each value in FIG. 10 is the same as that in FIG.

  If a person is at point C, the person can select the routes G, I, and J to go to point A, point B, and point D. In this case, for example, when switching to the second mode, a route to be moved is assigned to a plurality of keys on the keyboard (in this case, the → key and the ↑ key, or both of them are pressed simultaneously), and the key is pressed. It is conceivable to select a route so that the mode can be shifted to the second mode, or to perform a key operation for shifting to the second mode after directing the line of sight in the direction of the desired route. In the fourth embodiment, another example of solving a problem that can occur in the present invention is shown. When a person looks at the omnidirectional image at an arbitrary point, it may be difficult to intuitively understand which point (or direction) the user can move to. For this reason, as shown in FIG. 9B, a means of displaying a movable point (a route may be displayed with a line or the like) in the omnidirectional image viewed by a person is conceivable. When this measure is taken, it is considered that a more intuitive implementation is possible, for example, when a point is selected when a person puts his eyes on the point A.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Distribution server, 20 ... Image processing apparatus, 30 ... Display apparatus, 201 ... Image request / reception part, 202 ... Image rendering part, 203 ... Image output part, 204 ... Mode determination part, 301 ... Head direction detection part, 302 ... Eye direction detection unit, 303 ... Display unit

Claims (8)

As the operation mode of the own device, a first mode for providing an image captured by the imaging device installed at a selected point among images captured by an imaging device capable of imaging a wide range installed at each point; Any one of the second mode for providing an image of a route from an imaging device that has captured an image currently viewed by the user to a destination imaging device in order to switch to an image captured by the imaging device at another point A mode determining step for determining whether to operate in the mode of
An image processing step of performing image processing on the image of the route according to the movement of the user's head in the second mode;
An image output step of providing an image after image processing to a display device worn by the user;
An image providing method.   The image providing method according to claim 1, wherein an image of a route from an imaging device that captures an image viewed by a user to a destination imaging device is an image that is captured in advance while moving by the imaging device.   3. The image processing step according to claim 1, wherein when operating in the second mode, an image of a route provided to the destination is subjected to image processing according to a rotation amount of the user's head. Image providing method.   The image providing method according to claim 3, wherein, in the image processing step, the amount of rotation of the head is the same as the amount of rotation performed in real space.   The image providing method according to claim 3, wherein in the image processing step, the rotation amount of the head is a rotation amount obtained by multiplying a rotation amount performed in real space by an arbitrary function.   The image processing step detects three-dimensional coordinates of a gaze target being watched from a convergence angle of the user's eyeball, and performs image processing so that the gaze target is positioned at the center of the user's field of view. The image providing method according to any one of 1 to 5. As the operation mode of the own device, a first mode for providing an image captured by the imaging device installed at a selected point among images captured by an imaging device capable of imaging a wide range installed at each point; Any one of the second mode for providing an image of a route from an imaging device that has captured an image currently viewed by the user to a destination imaging device in order to switch to an image captured by the imaging device at another point A mode determination unit that determines whether to operate in the mode of
An image processing unit that performs image processing on the image of the route according to the movement of the user's head in the second mode;
An image output unit for providing an image after image processing to a display device worn by the user;
An image providing apparatus comprising: As the operation mode of the own device, a first mode for providing an image captured by the imaging device installed at a selected point among images captured by an imaging device capable of imaging a wide range installed at each point; Any one of the second mode for providing an image of a route from an imaging device that has captured an image currently viewed by the user to a destination imaging device in order to switch to an image captured by the imaging device at another point A mode determining step for determining whether to operate in the mode of
An image processing step of performing image processing on the image of the route according to the movement of the user's head in the second mode;
An image output step of providing an image after image processing to a display device worn by the user;
A computer program for causing a computer to execute.

Images