JPWO2011132382A1 - Information providing system, information providing method, and information providing program - Google Patents

Abstract

Provided is an information providing system capable of providing moving image information such as a video image related to a real world scene in association with the real world from a user's viewpoint. The camera is worn by the user to photograph the real world. The position / orientation estimation means estimates the position and orientation of the real world from the video taken by the camera. The video image transformation means transforms a video image, which is an image obtained by capturing a real world scene in advance, in accordance with the estimated position and posture of the real world. The superimposing unit superimposes the video photographed by the camera and the video video deformed by the video video transformation unit.

Description

  The present invention relates to an information providing system, a video display terminal, a video reproduction control device, an information providing method, and an information providing program, and in particular, an information providing system that provides easy-to-understand information associated with an object photographed by a camera, and video display The present invention relates to an information terminal, a video reproduction control device, an information presentation method, and an information presentation program.

  Augmented Reality (Augmented Reality) is a method of providing information related to the real world by photographing the real world from the user's viewpoint with a camera and providing information related to the content of the captured real world on the video. There is a technology called). An example of an apparatus using augmented reality is described in Patent Document 1. The work information providing apparatus described in Patent Literature 1 identifies a spatial arrangement of work objects photographed by a camera, and superimposes and displays information related to work to be performed by the worker in association with the work object. By using such a method, it becomes possible to provide information that can be easily understood by the operator.

  Patent Document 2 describes a captured video processing system that displays captured images superimposed on a map. In the photographic image processing system described in Patent Document 2, the photographic position in the air is specified three-dimensionally, and the photographic image frame is calculated by specifying the posture of the camera and the body relative to the ground surface. The video is transformed to match.

  Patent Document 3 describes an information presentation device that outputs a video obtained by synthesizing a real world video from a user's viewpoint position and a virtual world video to an image display device. The information presentation device described in Patent Literature 3 calculates the user's visual field in the virtual world based on the user's viewpoint position and orientation information. And the said information presentation apparatus produces | generates CG (Computer Graphics) which overlaps with a user's visual field based on the data of the part which enters into a user's visual field.

JP 2001-282349 A JP 2003-316259 A JP-A-2005-174021

  Among the information associated with real-world scenes are video images. However, a general apparatus does not consider a method of displaying video images in association with the real world. Therefore, the work information providing apparatus described in Patent Document 1 has a problem that a video image that can be easily created cannot be displayed in association with the real world.

  In addition, the system described in Patent Document 2 deforms an image captured by the camera based on the posture of the camera with respect to the ground surface, and superimposes the deformed images. In other words, in the system described in Patent Document 2, the video imaged according to the information associated with the real world is transformed. Therefore, it is hard to say that relevant information is provided for the real world from the user's perspective.

  In addition, the information presentation device described in Patent Document 3 merely outputs an image in which the user's field of view (real world) and CG are simply superimposed on the display device, and the superimposed information (CG) and the actual information are displayed. There is no relevance to the world. Therefore, even if the information presentation device described in Patent Document 3 is used, a video image cannot be displayed in association with the real world.

  Therefore, the present invention provides an information providing system, a video display terminal, a video reproduction control device, an information presentation method, and information presentation information that can provide moving picture information related to a real world scene in association with the real world from the viewpoint of the user. The purpose is to provide a program.

  An information providing system according to the present invention includes a camera that is worn by a user and captures the real world, a position and orientation estimation unit that estimates a position and orientation of the real world from an image captured by the camera, and a real world scene in advance. Video image transformation means for transforming a video image, which is a photographed image, according to the estimated position and orientation of the real world, superimposition that superimposes a video image taken by the camera and a video image transformed by the video image transformation means Means.

  The video display terminal according to the present invention includes a camera that is mounted on a user and shoots the real world, and a server device that estimates the position and orientation of the real world from a video of the real world. Transmitting means for transmitting images, and video images that transform video images that are images of real-world scenes in advance to match the position and orientation of the real world estimated by the server device based on the images captured by the camera The image forming apparatus includes a deforming unit, and a superimposing unit that superimposes the video photographed by the camera and the video image deformed by the video image deforming unit.

  The video playback control apparatus according to the present invention pre-captures a position and orientation estimation means for estimating the position and orientation of the real world, and a scene of the real world from the video received from the terminal device that transmits the video of the real world. Separation determination means for comparing division information, which is information predetermined as a division in a video image, which is video, and video received from a terminal device, and determining a division of a user's action, according to a determination result Reproduction control signal generation means for generating a reproduction control signal that is a signal for controlling the reproduction of the video image in the section specified by the delimiter information, information indicating the position and orientation of the real world estimated by the position and orientation estimation means, An information transmission means for transmitting the reproduction control signal to the terminal device is provided.

  The information presentation method according to the present invention is based on a video image that is a video in which a real-world scene is preliminarily estimated by estimating the position and orientation of the real world from a video captured by a camera that is worn by a user and that captures the real world. The segmentation information, which is information defined as segmentation, is compared with the video captured by the camera to determine the segmentation of the user's action, and according to the determination result, the video video in the section specified by the segmentation information The reproduction is controlled, the video image is deformed in accordance with the estimated position and orientation of the real world, and the image captured by the camera is superimposed on the deformed video image.

  An information presentation program according to the present invention is an information presentation program applied to a computer equipped with a camera that is mounted on a user and photographs the real world, and from the video captured by the camera, Position / orientation estimation processing to estimate position and orientation, and use by comparing segment information, which is information defined as segments in video images that are pre-captured images of real-world scenes, and images captured by the camera Delimiter determination process for determining the delimiter of the user's movement, playback control process for controlling the playback of the video image of the section specified by the delimiter information according to the determination result, video image according to the estimated real-world position and orientation Video image transformation processing that transforms the video, and superimposition processing that superimposes the video shot by the camera and the transformed video video. The features.

  According to the present invention, it is possible to provide moving image information such as a video image related to a real world scene in association with the real world from the viewpoint of the user.

It is a block diagram which shows the example of the information provision system in the 1st Embodiment of this invention. It is a flowchart which shows the example of operation | movement in 1st Embodiment. It is a block diagram which shows the example of the information provision system in the 2nd Embodiment of this invention. It is a flowchart which shows the example of operation | movement in 2nd Embodiment. It is explanatory drawing which shows the specific example which performs work assistance in a specific work environment. It is explanatory drawing which shows the example of 1 scene in a video image | video. It is a block diagram which shows the example of the minimum structure of the information provision system by this invention. It is a block diagram which shows the example of the minimum structure of the terminal for video display by this invention. It is a block diagram which shows the example of the minimum structure of the video reproduction control apparatus by this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment 1. FIG.
FIG. 1 is a block diagram showing an example of an information providing system according to the first embodiment of the present invention. The information providing system in the present embodiment includes a camera 1, a display device 2, a position / orientation estimation unit 3, a segment estimation unit 4, a segment information storage unit 5, a video image storage unit 6, and a playback control unit 7. , A position / orientation information storage unit 8, a video image transformation unit 9, and a superimposition unit 10.

  The camera 1 captures the real world as seen from the user's perspective. The camera 1 is installed at a position close to the user's line of sight, and shoots an image similar to that in the real world viewed by the user.

  The display device 2 displays information associated with the real world superimposed on the real-world video. For example, the display device 2 presents to the user a video obtained by superimposing a video associated with the real world on a video captured by the camera 1 by a superimposing unit 10 described later. Such a mechanism is called video see-through. In addition, when the display device 2 includes a half mirror, the display device 2 uses a video in which only associated information is superimposed on an actual video that can be seen through the half mirror, not a video shot by the camera 1. May be presented to the person. In this case, the video taken by the camera 1 is used when a position / orientation estimation unit 3 described later estimates a spatial position in the real world.

  The position / orientation estimation unit 3 estimates a real-world spatial position and orientation taken by the camera 1. Specifically, the position / orientation estimation unit 3 analyzes the real world image captured by the camera 1 and estimates the relative spatial position and orientation of the captured real world. Here, the spatial position of the real world means the position of the real world as viewed from the camera 1, and can be represented by the orientation of the camera 1 and the distance between the camera 1 and the real world. The real world posture means the real world posture viewed from the camera 1 and indicates the degree of rotation of the entire real world.

  Examples of methods for estimating the position and orientation of the real world by the position / orientation estimation unit 3 include a method of using markers arranged in the real world, a method of using image features extracted from real-world video, and the like. Here, the marker means a graphic designed to be easily identified.

For example, figures (hereinafter referred to as markers) designed to be easily identified are arranged in advance in the real world to be photographed, and the position / orientation estimation unit 3 is a marker in an image photographed by the camera 1. The real world position and orientation may be estimated based on the type, position, and mode. Note that such a method using graphic markers can be realized by using software called ARTToolKit, for example. Note that ARTToolKit is introduced in Reference Document 1 below.
[Reference 1] http://www.artoolworks.com/ARToolKit_Professional.html

When the marker is not arranged, the position / orientation estimation unit 3 extracts, for example, an image feature from a real-world video, and compares the extracted image feature with a pre-registered image feature position. The position and orientation may be estimated. PTAM is known as software that realizes such a markerless system. PTAM is described in Reference Document 2 below.
[Reference 2] http://www.robots.ox.ac.uk/~gk/PTAM/

  The video image storage unit 6 stores an image obtained by capturing a real-world scene in advance. The video obtained by photographing a real world scene in advance is a so-called moving image, and examples thereof include a video video obtained by photographing an operation performed in the real world in advance. In addition, as a specific example of the video image, there is a teacher video image as a model of work such as how to assemble furniture and how to handle fish. Hereinafter, a video image will be described as an example of an image obtained by photographing a real-world scene in advance.

  The video image is an image having relevance to the real-world image captured by the camera 1. Specifically, the video image is information that is displayed overlaid on the real-world image captured by the camera 1 so that the user can work according to the image. For example, when a user wearing the camera 1 and the display device 2 performs assembly work of furniture, the display device 2 displays a video showing how the furniture 1 is assembled in a video image of the camera 1 being assembled. Overlays the image. The video image displayed on the display device 2 may be selected in advance according to the work performed by the user.

  The position / orientation information storage unit 8 stores information previously analyzed as a real-world spatial position and orientation captured as a video image. Hereinafter, information indicating the real-world spatial position and orientation in the captured video image will be referred to as position and orientation information.

  The position / orientation information is analyzed using, for example, a method similar to the method in which the position / orientation estimation unit 3 estimates the current position and orientation in the real world taken by the camera 1. However, the analysis method of position and orientation information is not limited to the above method. For example, when shooting a video image, a sensor that can detect the position and orientation of the real world may be attached to the camera, and information detected by the sensor may be used as position and orientation information. As described above, the position and orientation information storage unit 8 stores the position and orientation information analyzed using an arbitrary method.

  As the position / orientation information, for example, a conversion matrix for converting video can be cited. For example, when a video image is shot with the camera fixed, the same transformation matrix may be stored between the position and orientation information storage unit 8 and the state. Further, when a video image is shot with the camera moving, the position / orientation information storage unit 8 may store a conversion matrix that changes with time. However, the position / orientation information is not limited to the transformation matrix.

  The video image deformation unit 9 deforms the video image so as to match the estimated position and orientation of the real world. That is, the video image deformation unit 9 deforms the video image according to the position and posture of the real world estimated by the position / orientation estimation unit 3.

  Specifically, the position and orientation of the photographed object in the real world (that is, position and orientation information) are obtained in advance from the video image, and the position and orientation information is stored in the position and orientation information storage unit 8. Then, the video image transformation unit 9 matches the position / orientation information with the real-world position and orientation taken by the camera 1 (that is, the real-world position and orientation estimated by the position / orientation estimation unit 3). , Transform video footage. The video image deformation unit 9 may deform the video image by perspective projection conversion, for example. However, the video image transformation method is not limited to perspective projection conversion.

  By deforming the video image in this way, the current real world and the video image look the same. Therefore, the user can view a video image that is very intuitive and easy to understand as information associated with the real world.

  The superimposing unit 10 superimposes the video image deformed by the video image deforming unit 9 on the image captured by the camera 1.

  For example, the superimposing unit 10 may superimpose the semi-transparent video image at the same position as in the real world. Note that when the real-world video and the semi-transparent video image are superimposed, in order to prevent any of the images from being difficult to see, the superimposing unit 10 may process the video video so that the video is easy to see. Good. That is, the superimposing unit 10 may process the video image in a manner that can be distinguished from the real-world image captured by the camera 1. For example, the superimposing unit 10 changes the color of the video image, enhances the edge of the video image to make the other portions transparent, changes the degree of semi-transmission according to time, and the like. It may be processed.

  The superimposing unit 10 may superimpose the video image so that the display position is slightly shifted while aligning the direction and size. Further, the superimposing unit 10 may superimpose so that the video image is always displayed on the screen edge.

  The delimiter information storage unit 5 stores delimiter information. Here, the delimiter information is information predetermined as a delimiter in the video image stored in the video image storage unit 6. The delimiter information storage unit 5 may store, for example, an image at the delimiter as the delimiter information. However, the delimiter information is not limited to video. Other information may be used as long as it is information that can determine a break from a video image. For example, instead of the video itself, features obtained from the video may be used as the delimiter information. In addition, when the feature obtained from the video is used as the delimiter information, it is desirable to extract a feature that can be easily used for matching with the video video.

  The delimiter estimation unit 4 determines the delimiter of the action performed by the user by comparing the real-world video with the delimiter information. When the delimiter estimation unit 4 determines that the user's action has reached the delimiter, the delimiter estimation unit 4 sends information to that effect to the reproduction control unit 7.

  Specifically, first, the delimiter estimation unit 4 calculates the real world image captured by the camera 1 from the real world position and orientation estimated by the position / orientation estimation unit 3 and the position / orientation information corresponding to the video image. Transform to fit the video image. After that, the delimiter estimation unit 4 compares the deformed real-world video with the delimiter information stored in the delimiter information storage unit 5 to estimate the progress of the user's operation. In the following description, the delimiter information may be described as a final state.

  For example, it is assumed that the video image is divided in advance for each operation unit, and the position / orientation information storage unit 8 stores an image indicating the final state of the video image divided into operation units as the delimiter information. . At this time, the delimiter estimation unit 4 compares the video showing the final state in the section including the current operation with the real-world video captured by the camera 1. When the delimiter estimation unit 4 determines that these videos match, it is assumed that the operation in the current section has ended, and a signal for proceeding to the next section (hereinafter referred to as a playback control signal). .) Is sent to the reproduction control unit 7.

  In the above description, the case where the delimiter estimation unit 4 compares the video showing the final state with the real-world video captured by the camera 1 has been described. However, the target that the delimiter estimation unit 4 compares with the real-world video is not limited to the video. The delimiter estimation unit 4 may determine the delimiter by comparing information indicating the characteristics of the final state with a real-world video.

  The reproduction control unit 7 controls the reproduction of the video image in the section specified by the division information according to the division determined by the division estimation unit 4. For example, the playback control unit 7 repeatedly plays back the video image corresponding to the section up to the currently determined break.

  Specifically, when the delimiter estimation unit 4 determines that the delimiter information and the video captured by the camera 1 match, the playback control unit 7 controls to play the video video of the next section. On the other hand, when the delimiter estimation unit 4 determines that the delimiter information and the video captured by the camera 1 do not match, the reproduction control unit 7 repeatedly reproduces the video image of the section being reproduced.

  Note that when the video image of the corresponding section is repeatedly reproduced, the reproduction control unit 7 reproduces the video image after a certain period of time that is not reproduced every time rewinding (that is, every time reproduction is completed). Also good. Further, the reproduction control unit 7 may change the reproduction speed each time reproduction is repeated (for example, the speed is decreased and the reproduction is slowly performed). For example, by reducing the playback speed as the number of playback repetitions increases, the video speed can be gradually reduced if the video image is too early. Therefore, the user can easily understand the provided information.

  In this way, the position and orientation estimation unit 3 and the video image transformation unit 9 synchronize the difference in position and orientation between the real world and the video image, and the playback control unit determines the progress of the work operation while the estimation unit 4 determines the progress. 7 performs video image reproduction control. For this reason, information such as work procedures associated with the real world can be displayed in augmented reality using video images that can be easily created.

  The delimiter information storage unit 5, the video image storage unit 6, and the position / orientation information storage unit 7 are each realized by a magnetic disk or the like.

  The position / orientation estimation unit 3, the segment estimation unit 4, the playback control unit 7, and the video image transformation unit 9 are realized by a CPU of a computer that operates according to a program (information providing program). For example, the program is stored in a storage unit (not shown) in the information providing system, and the CPU reads the program, and in accordance with the program, the position / orientation estimation unit 3, the segment estimation unit 4, the playback control unit 7, and the video image The deformation unit 9 may be operated.

  Further, the information providing system may be configured to include a video display terminal that displays video and a video playback control device that controls the video to be played back. In this case, for example, the video display terminal includes the camera 1, the display device 2, the video video deformation unit 9, and the superimposition unit 10, and the video playback control device includes the position / orientation estimation unit 3, the segment estimation unit, and the like. 4, a reproduction control unit 7, a delimiter information storage unit 5, a video image storage unit 6, and a position / orientation information storage unit 8.

  At this time, the control unit (not shown) of the video reproduction control apparatus may transmit the real world video captured by the camera 1 to the video reproduction control apparatus to estimate the position and orientation of the real world. Further, at this time, the playback control unit 7 may generate a signal (that is, a playback control signal) for controlling playback of the video image, and transmit the generated playback control signal to the video display terminal.

  Furthermore, each of the position / orientation estimation unit 3, the segment estimation unit 4, the playback control unit 7, and the video image transformation unit 9 may be realized by dedicated hardware.

  Next, the operation will be described. FIG. 2 is a flowchart showing an example of the operation in the first embodiment.

  First, the position / orientation estimation unit 3 estimates the relative position and orientation of the real world with respect to the user by analyzing the image captured by the camera 1 (step S1). Next, the delimiter estimation unit 4 matches the real world image captured by the camera 1 with the video image from the real world position and orientation estimated by the position / orientation estimation unit 3 and the position / orientation information corresponding to the video image. It deforms as follows. After that, the delimiter estimation unit 4 compares the deformed real-world video and the final state in the current section accumulated in the delimiter information storage unit 5, so that the progress of the user's action is within the section. It is determined whether the end (that is, the break) has been reached (step S2).

  When it is determined that the user's action has reached the end of the segment (Yes in step S2), the playback control unit 7 advances the playback position of the video image to the section up to the next segment (step S3). ). When the reproduction control unit 7 reproduces the video image in the section up to the current segment, the video image deformation unit 9 deforms the video image according to the position and posture of the real world estimated by the position / orientation estimation unit 3 ( Step S4). Finally, the superimposing unit 10 superimposes the deformed video image on the image captured by the camera 1 and causes the display device 2 to display the image (step S5).

  On the other hand, when it is determined in step S2 that the user's action has not reached the end of the break (No in step S2), the playback control unit 7 repeatedly plays back the video image of the section being played back. The process after step S4 is performed.

  Next, the effect of this embodiment will be described. According to the present embodiment, the position / orientation estimation unit 3 estimates the position and orientation of the real world from the video captured by the camera 1. The delimiter determination unit 4 compares the delimiter information with the video captured by the camera to determine the delimiter of the user's action. Then, the playback control unit 7 controls the playback of the video image in the section specified by the delimiter information according to the determination result. The video image deforming unit 9 deforms the video image according to the estimated position and orientation of the real world, and the superimposing unit 10 superimposes the image captured by the camera 1 and the deformed video image. Therefore, it is possible to provide moving image information such as a video image related to a real world scene in association with the real world from the viewpoint of the user.

  That is, in this embodiment, the position / orientation estimation unit 3 estimates the position and orientation of the real world as viewed from the user, and the video image transformation unit 9 and the playback control unit 7 display the video image according to the position and orientation. Play while transforming. Therefore, the user can work while easily comparing the video image as a teacher with the real world.

  Further, in the present embodiment, the video image is segmented for each series of operations, and the segment estimation unit 4 and the playback control unit 7 perform playback for each segment segmented according to the progress of the operations. For this reason, it is possible to synchronize the video image and the actual operation at intervals at intervals obtained by dividing the video image according to the operation. In addition, existing video images can be used as they are for information provided in augmented reality.

Embodiment 2. FIG.
FIG. 3 is a block diagram illustrating an example of an information providing system according to the second embodiment of the present invention. In addition, about the structure similar to 1st Embodiment, the code | symbol same as FIG. 1 is attached | subjected and description is abbreviate | omitted. The information providing system according to the present embodiment includes a first video information storage unit 12 and second video information instead of the delimiter information storage unit 5, the video image storage unit 6, and the position / orientation information storage unit 8 according to the first embodiment. And a storage unit 13. In addition to the information providing system in the first embodiment, the information providing system in the present embodiment includes a video information selection unit 11.

  The first video information storage unit 12 stores information including a set of a video image, delimiter information corresponding to the video image, and position and orientation information. Hereinafter, this information is referred to as video information, and the video information stored in the first video information storage unit 12 is referred to as first video information.

  The second video information storage unit 13 stores information including a set of a video image different from the first video information, delimiter information corresponding to the video image, and position and orientation information. Hereinafter, the video information stored in the second video information storage unit 13 is referred to as second video information.

  The first video information storage unit 12 and the second video information storage unit 13 include the delimiter information storage unit 5, the video image storage unit 6, and the position / orientation information storage unit 8 in the first embodiment, respectively. The video information stored in the first video information storage unit 12 and the second video information storage unit 13 includes information provided in different scenes. The first video information storage unit 12 and the second video information storage unit 13 are each realized by a magnetic disk or the like.

  In the example shown in FIG. 3, each video information is stored in different storage devices (specifically, the first video information storage unit 12 and the second video information storage unit 13). However, each video information may not be stored in different storage devices, and may be stored in one storage device.

  The video information selection unit 11 compares the real-world video and the video video in the plurality of video information, and selects more suitable video information. Specifically, the video information selection unit 11 selects video information including a video image most similar to the image captured by the camera 1 from the plurality of video information. For example, the video information selection unit 11 compares the video images included in the first video information and the second video information with real-world images, and selects video information including more suitable video images from the video images. To do. Note that the video information selection unit 11 is realized by a CPU that operates according to a program, for example.

  Hereinafter, a method in which the video information selection unit 11 selects a video image similar to a real-world image will be described. First, the video information selection unit 11 uses the real-world position and orientation estimated by the position / orientation estimation unit 3, the position / orientation information in the first video information, and the position / orientation information in the second video information. The photographed real-world video is transformed so as to match the video video of each (that is, the first video information and the second video information). Then, the video information selection unit 11 compares the deformed real-world video with each video video, and selects a video image that is more similar.

  In the above description, the case where the video information selection unit 11 compares the deformed real-world image with the video image has been described. However, the object that the video information selection unit 11 compares with the deformed real world image is not limited to the image. The video information selection unit 11 may select more suitable video information by comparing the information indicating the characteristics of the video image with the deformed real-world image.

  In the following description, as exemplified in FIG. 3, the case where the video information compared by the video information selection unit 11 is two types of first video information and second video information will be described. However, the video information is not limited to two types, and may be three or more types. In this case, the video information selection unit 11 may select video information including the most similar video image.

  Next, the operation will be described. FIG. 4 is a flowchart illustrating an example of an operation in the second embodiment. First, as in the first embodiment, the position / orientation estimation unit 3 estimates the relative position and orientation of the real world with respect to the user by analyzing the image captured by the camera 1 (step S1). Thereafter, the video information selection unit 11 selects video information including a video image that is more similar to a real-world image from the first video information and the second video information (step S10).

  Thereafter, the processing until the selected video image is deformed and displayed superimposed on the real-world image is the same as the processing from steps S2 to S5 in FIG.

  Next, the effect of this embodiment will be described. According to the present embodiment, the video information selection unit 11 selects, from among a plurality of video information, video information including a video image most similar to the image captured by the camera 1 (that is, the current real-world image). To do. Therefore, it is possible to automatically display images suitable for the real world simply by preparing various video images.

  Hereinafter, the present invention will be described with reference to specific examples, but the scope of the present invention is not limited to the contents described below.

  FIG. 5 is an explanatory diagram illustrating a specific example of performing work support in a specific work environment. In the environment illustrated in FIG. 5, the user 110 is wearing a wearable camera 111 and a wearable display device 112. Here, the wearable camera 111 corresponds to the camera 1 in FIG. 1, and the wearable display device 112 corresponds to the display device 2 in FIG.

  The information providing system in this embodiment is realized by the wearable camera 111, the wearable display device 112, and a small personal computer equipped with a CPU and a memory. The information providing system according to the present embodiment includes, in addition to the wearable camera 111 and the wearable display device 112, the position / orientation estimation unit 3, the delimiter estimation unit 4, the delimiter information storage unit 5, and the video in the first embodiment. A video storage unit 6, a playback control unit 7, a position / orientation information storage unit 8, a video video transformation unit 9, and a superimposition unit 10 are included. However, in the example shown in FIG. 5, the description of the configuration other than the wearable camera 111 and the wearable display device 112 is omitted.

  In this embodiment, it is assumed that a teacher video image of work is used as a video image, and a method of performing work support using this video image will be described.

  First, the wearable camera 111 attached to the upper part of the wearable display device 112 shoots an image similar to the environment that the user 110 is viewing. The captured video is an image including the markers 101 to 104 and the work target 120.

  The position / orientation estimation unit 3 detects a marker from the captured video. Then, the position / orientation estimation unit 3 integrates information representing the positions and orientations of the markers 101 to 104 obtained from the information in the captured video, and estimates the real-world position and orientation with respect to the user 110. Here, the posture of the marker indicates the degree to which the marker is rotating with respect to the real world.

  Note that by arranging a plurality of markers, for example, by calculating the average of the positions and orientations of the plurality of markers, it is possible to reduce errors with respect to the real-world positions and orientations. Further, the position and orientation of the real world can be estimated even when some markers are not visible.

  FIG. 6 is an explanatory diagram showing an example of one scene in the video image stored in the video image storage unit 6. The video image illustrated in FIG. 6 indicates that the markers 201 to 204 and the work target 220 corresponding to the markers 101 to 104 and the work target 120 illustrated in FIG. This video image is divided in advance for each delimitation of the operation procedure, and information delimiting the video image (that is, delimiter information) is stored in the delimiter information storage unit 5.

  Further, the position and orientation information is analyzed by detecting the markers 201 to 204 illustrated in FIG. 6, and the analyzed information is stored in the position and orientation information storage unit 8 in advance. Further, the position and orientation information of scenes other than those illustrated in FIG. 6 are similarly analyzed, and the analyzed information is stored in the position and orientation information storage unit 8.

  The segment estimation unit 4 uses the markers 201 to 204 in the segment (final state) of the video image illustrated in FIG. 6 based on the position and orientation of the real world estimated by the markers 101 to 104 illustrated in FIG. The real-world video is transformed to match the estimated real-world position and orientation. Then, the delimiter estimation unit 4 compares the video image indicating the delimiter (final state) with the deformed real-world image to determine whether the progress of the user's operation has reached the end in the section. Determine.

  When it is determined that the progress status of the user's operation has reached the end of the section, the playback control unit 7 advances the playback position of the video image to the section up to the next segment. The playback control unit 7 plays back video images in the current section. At that time, the video image deformation unit 9 deforms the video image so as to match the position and posture of the real world estimated by the position / orientation estimation unit 3. The superimposing unit 10 superimposes the deformed video image on the image captured by the camera 1 and causes the wearable display device 112 to display the image.

  In the above description, a method of using a teacher video image of work as a video image and performing work support using the video image has been described. The information presentation system according to the present invention can be applied to scenes where route guidance is performed, for example.

  When using the information presentation system according to the present invention for route guidance, for example, taking a route in advance while walking with a hand-held camera, and preparing a video that explains a landmark building or the like with audio or subtitles as a teacher video image Keep it. In this case, the position / orientation estimation unit 3 estimates the current position / orientation using information (features, signboards, scenery, etc.) of feature points existing in the video. Further, the video image may be divided for each operation such as walking a certain distance or turning a corner. At this time, when the final scene in the section is reached, the video image of the next section is displayed by being transformed according to the scenery being viewed.

  Next, the minimum configuration of the present invention will be described. FIG. 7 is a block diagram showing an example of the minimum configuration of the information providing system according to the present invention. FIG. 8 is a block diagram showing an example of the minimum configuration of the video display terminal according to the present invention. FIG. 9 is a block diagram showing an example of the minimum configuration of the video reproduction control apparatus according to the present invention.

  The information providing system according to the present invention includes a camera 81 (for example, camera 1) that is worn by a user and captures the real world, and an image captured by the camera 81 from the position and orientation of the real world (for example, the camera orientation, Position / orientation estimation means 82 (for example, position / orientation estimation unit 3) for estimating camera distance and degree of rotation, and video images (for example, stored in the video image storage unit 6) that are images of real-world scenes previously captured. Video image deformation means 83 (for example, video image deformation section 9) that deforms the estimated video image in accordance with the estimated position and orientation of the real world, video captured by the camera 81, and video image deformation means 83. Is provided with superimposing means 84 (for example, the superimposing unit 10) for superimposing the deformed video image.

  With such a configuration, it is possible to provide moving image information such as a video image related to a real-world scene in association with the real world from the viewpoint of the user.

  In addition, the information presenting system uses delimiter information (for example, delimiter information stored in the delimiter information storage unit 5 and the final state), which is information predetermined as a delimiter in the video image, and the video captured by the camera 81. In comparison, a delimiter determining unit (for example, delimiter estimation unit 4) that determines the delimiter of the user's action, and a reproduction control unit that controls the reproduction of the video image in the section specified by the delimiter information according to the determination result ( For example, a reproduction control unit 7) may be provided.

  The delimiter determining means determines whether the state in the video image indicated by the delimiter information is the same as the state indicated by the image captured by the camera 81, and determines the delimiter of the user's action, When the playback control means determines that the break determination means is in the same state, it controls to play the video image of the next section of the break specified by the break information, and determines that the break determination means is not in the same state In this case, it may be controlled to repeatedly reproduce the video image in the section up to the break specified by the break information.

  Further, the playback control means controls to repeatedly play back the video image of the section specified by the delimiter information, and changes the playback speed when repeatedly playing back the video image of the section (for example, playback Control may be performed so that the playback speed decreases as the number of repetitions increases. By doing so, the user can easily understand the provided information.

  In addition, the information presentation system includes a video information storage unit (for example, the first video information storage unit 12 and the second video information storage unit 13) that stores a plurality of pieces of video information that is information obtained by combining video images and segment information. The video information selecting means (for example, the video information selecting unit 11) for selecting video information including a video image most similar to the image captured by the camera 81 from a plurality of video information may be provided. With such a configuration, it is possible to automatically display images suitable for the real world simply by preparing various video images.

  Further, the superimposing means 84 may process the video image so that it can be distinguished from the image taken by the camera 81 (for example, changing the color of the video image, emphasizing the edge of the video image, and making other than the edge transparent) The degree of semi-transmission is changed according to time).

  In addition, a video display terminal according to the present invention includes a camera 71 (for example, camera 1) that is worn by a user and captures the real world, and a server device that estimates the position and orientation of the real world from the video captured of the real world. 60 (for example, a video reproduction control device), a video transmission means 72 (for example, a control unit of the video reproduction control device) for transmitting a video captured by the camera 71, and a server device 60 based on the video captured by the camera 71. In accordance with the estimated position and orientation of the real world, the video image transformation means 73 (for example, the video image transformation unit 9) that transforms the video image, which is the image obtained by photographing the real world scene in advance, and the camera 71 photographed. Superimposing means 74 for superimposing the video and the video image deformed by the video image deformation means 73 is provided.

  Also, the video reproduction control apparatus according to the present invention estimates the position and orientation of the real world from the video received from the terminal device 70 (for example, video display terminal) that transmits the video of the real world. Information determined in advance as a break in the means 61 (for example, the position / orientation estimation unit 3) and a video image (for example, a video image stored in the video image storage unit 6) that is an image obtained by capturing a real-world scene in advance. The delimiter determining means 62 (for example, delimiter information stored in the delimiter information storage unit 5) is compared with the video received from the terminal device 70 to determine the delimiter of the user's action. The delimiter estimation unit 4) and a reproduction control signal generation for generating a reproduction control signal that is a signal for controlling the reproduction of the video image in the section specified by the delimiter information according to the determination result Information transmission means 64 (for example, reproduction control) that transmits information indicating the real world position and orientation estimated by the stage 63 (for example, the reproduction control unit 7), the position and orientation estimation means, and a reproduction control signal to the terminal device 70. Part 7).

  Even with these configurations, it is possible to provide moving image information such as a video image related to a real-world scene in association with the real world from the viewpoint of the user.

  In addition, although a part or all of said embodiment can be described also as the following additional remarks, it is not restricted to the following.

(Supplementary note 1) A camera that is mounted on a user and shoots the real world, a position and orientation estimation unit that estimates the position and orientation of the real world from the video captured by the camera, and a video that pre-captures a real world scene A video image deformation means for deforming the video image according to the estimated position and orientation of the real world, and a superimposing means for superimposing the image captured by the camera and the video image deformed by the video image deformation means An information presentation system characterized by comprising:

(Supplementary Note 2) Separation determination means for comparing division information, which is information predetermined as a division in a video image, and video captured by a camera to determine a division of a user's action, and the determination result Therefore, the information presentation system according to appendix 1, further comprising reproduction control means for controlling reproduction of a video image in a section specified by the delimiter information.

(Additional remark 3) The division | segmentation determination means determines whether the state in the video image | video which division | segmentation information shows, and the state which the image | video which was image | photographed with the camera is the same state, and determines the division | segmentation of a user's operation | movement. The playback control means controls to play back the video image of the next section after the break specified by the break information when the break determination means determines that they are in the same state, and the break determination means is the same. The information presentation system according to supplementary note 2, wherein when it is determined that the state is not in a state, control is performed so as to repeatedly reproduce a video image of a section up to a break specified by the break information.

(Supplementary note 4) The reproduction control means controls to repeatedly reproduce the video image of the section specified by the delimiter information, and controls to change the reproduction speed when repeatedly reproducing the video image of the section. 2. Information presentation system according to 2 or appendix 3.

(Supplementary Note 5) Video information storage means for storing a plurality of video information, which is a set of video images and delimiter information, and a video image most similar to a video photographed by a camera among the plurality of video information. 5. The information presentation system according to any one of supplementary notes 2 to 4, further comprising video information selection means for selecting video information to be included.

(Supplementary note 6) The information presentation system according to any one of supplementary note 1 to supplementary note 5, wherein the superimposing unit processes the video image into a mode distinguishable from the image captured by the camera.

(Additional remark 7) A video image deformation | transformation means deform | transforms a video image so that the position and attitude | position information which is the information which shows the position and attitude | position of the real world in a video image, and the estimated real world position and attitude | position match. The information presentation system according to any one of supplementary notes 1 to 6.

(Supplementary note 8) The information presentation system according to any one of supplementary notes 1 to 7, wherein the position / orientation estimation unit extracts a marker position from an image captured by a camera and estimates a real-world position and orientation.

(Supplementary note 9) The information according to any one of supplementary notes 1 to 7, wherein the position / orientation estimation unit estimates a real-world position and orientation based on a feature point extracted from an image captured by a camera. Presentation system.

(Supplementary Note 10) A real-world video captured by the camera is transmitted to a camera that is mounted on the user and that captures the real world, and a server device that estimates the position and orientation of the real world from the video captured of the real world. And a video image transformation that transforms a video image, which is an image obtained by photographing a real-world scene in advance, in accordance with the position and orientation of the real world estimated by the server device based on the image taken by the camera And a superimposing unit for superimposing the video captured by the camera and the video image deformed by the video image modifying unit.

(Supplementary note 11) The video display terminal according to supplementary note 10, wherein the superimposing unit processes the video image into a mode distinguishable from the image captured by the camera.

(Supplementary note 12) Position / orientation estimation means for estimating the position and orientation of the real world from the video received from the terminal device that transmits the video obtained by photographing the real world, and a video image that is a video obtained by photographing the scene of the real world in advance Delimiter information that is predetermined information as a delimiter in the middle and a video received from the terminal device, delimiter determining means for determining a delimiter of user's action, and according to the determination result, the delimiter Reproduction control signal generation means for generating a reproduction control signal that is a signal for controlling reproduction of a video image in a section specified by the information, information indicating the position and orientation of the real world estimated by the position and orientation estimation means, An image reproduction control device comprising: information transmission means for transmitting a reproduction control signal to the terminal device.

(Additional remark 13) A division | segmentation determination means determines whether the state in the video image | video which division | segmentation information shows, and the state which the image | video which was image | photographed with the camera is the same state, and determines the division | segmentation of a user's operation | movement. The reproduction control unit generates a reproduction control signal for performing control to reproduce the video image of the next section of the segment specified by the segment information when it is determined that the segment determination unit is in the same state; 13. The video playback control device according to appendix 12, which generates a playback control signal for performing control to repeatedly play back video images in a section up to a partition specified by the partition information when it is determined that the partition determination means is not in the same state.

(Supplementary Note 14) The position and orientation of the real world are estimated from the video taken by the camera that is mounted on the user and shoots the real world, and is defined as a break in the video video that is a pre-captured video of the real world scene. The segment information, which is the recorded information, is compared with the video captured by the camera to determine the segment of the user's action, and in accordance with the determination result, the video image of the section specified by the segment information is reproduced. The information presentation method is characterized in that the video image is deformed in accordance with the estimated position and orientation of the real world, and the image captured by the camera is superimposed on the deformed video image.

(Supplementary note 15) The information presentation according to supplementary note 14, wherein control is performed so that the video image of the section specified by the delimiter information is repeatedly reproduced, and control is performed to change the reproduction speed when the video image of the section is repeatedly reproduced. Method.

(Supplementary Note 16) An information presentation program applied to a computer equipped with a camera that is mounted on a user and captures the real world, and the position and orientation of the real world from the video captured by the camera The position and orientation estimation processing for estimating the segmentation information, which is information defined as a segment in the video image, which is a video obtained by capturing a real-world scene in advance, and the image captured by the camera are compared with each other. Delimitation determination processing for determining an operation delimiter, playback control processing for controlling playback of a video image in a section specified by the delimiter information according to the determination result, video image in accordance with the estimated real-world position and orientation The video image deformation process for deforming the image and the superimposition process for superimposing the image captured by the camera and the deformed video image are executed. Program for the information presented.

(Supplementary Note 17) In the reproduction control process, control is performed so that the video image in the section specified by the delimiter information is repeatedly reproduced, and control is performed so as to change the reproduction speed when the video image in the section is repeatedly reproduced. 16. Information presentation program according to 16.

  Although the present invention has been described with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  This application claims the priority on the basis of the JP Patent application 2010-96055 for which it applied on April 19, 2010, and takes in those the indications of all here.

  The present invention is suitably applied to an information providing system that provides information related to an object photographed by a camera in an easy-to-understand manner. Specifically, the present invention can be applied to uses such as an information presentation system and an information providing program for transforming a video image for teacher according to an object in the real world and reproducing the image. The present invention can also be applied to uses such as an information presentation system for reproducing a video image divided for each operation according to progress, and an information providing program.

DESCRIPTION OF SYMBOLS 1 Camera 2 Display apparatus 3 Position and orientation estimation part 4 Separation estimation part 5 Separation information storage part 6 Video image storage part 7 Playback control part 8 Position and orientation information storage part 9 Video image transformation part 10 Superposition part 11 Video information selection part 12 1st Video information storage unit 13 Second video information storage unit 101-104, 201-204 Marker 120, 220 Work target

Claims (10)

A camera that is attached to the user and shoots the real world,
Position and orientation estimation means for estimating the position and orientation of the real world from the video taken by the camera;
Video image transformation means for transforming a video image, which is an image obtained by photographing a real world scene in advance, in accordance with the estimated position and posture of the real world,
An information presenting system comprising: a superimposing unit that superimposes a video photographed by the camera and a video video deformed by the video video transforming unit. Delimiter determination means for comparing delimiter information, which is information predetermined as a delimiter in a video image, and a video captured by the camera to determine a delimiter of the user's action;
The information presentation system according to claim 1, further comprising reproduction control means for controlling reproduction of a video image in a section specified by the delimiter information according to the determination result. The delimiter determining means determines whether the state in the video image indicated by the delimiter information is the same as the state indicated by the video captured by the camera, determines a delimiter of the user's action,
The reproduction control unit controls to play back the video image of the next section of the segment specified by the segment information when it is determined that the segment determination unit is in the same state, and the segment determination unit is in the same state The information presentation system according to claim 2, wherein when it is determined that the video image is not, the video image in the section up to the segment specified by the segment information is repeatedly reproduced. The playback control means controls to repeatedly play the video image of the section specified by the delimiter information, and controls to change the playback speed when repeatedly playing the video image of the section. Item 4. The information presentation system according to Item 3. Video information storage means for storing a plurality of video information, which is information obtained by combining video images and delimiter information;
5. The video information selecting means for selecting video information including a video image most similar to the image captured by the camera from the plurality of video information. 5. Information presentation system described. The information presentation system according to any one of claims 1 to 5, wherein the superimposing unit processes the video image so as to be distinguishable from the image captured by the camera. A camera that is attached to the user and shoots the real world,
Transmitting means for transmitting the real-world video captured by the camera to a server device that estimates the position and orientation of the real world from the video captured of the real world;
Video image transformation means for transforming a video image, which is an image obtained by photographing a real-world scene in advance in accordance with the position and orientation of the real world estimated by the server device based on the image taken by the camera;
A video display terminal comprising superimposing means for superimposing video captured by the camera and video video deformed by the video video deforming means. Position and orientation estimation means for estimating the position and orientation of the real world from the video received from the terminal device that transmits the video of the real world,
Determining the user's action delimiter by comparing delimiter information, which is information predetermined as a delimiter in a video image, which is a video of a real-world scene, and the video received from the terminal device Break determination means;
According to the determination result, reproduction control signal generation means for generating a reproduction control signal that is a signal for controlling reproduction of a video image in a section specified by the delimiter information;
An image reproduction control apparatus comprising: information transmission means for transmitting information indicating the position and orientation of the real world estimated by the position / orientation estimation means and the reproduction control signal to the terminal device. Estimate the position and orientation of the real world from the video taken by the camera that is attached to the user and shoots the real world,
Compare the delimiter information, which is information defined as a delimiter in the video image, which is a video that is pre-captured in the real world scene, and the video captured by the camera, determine the delimiter of the user's action,
In accordance with the determination result, control the playback of the video image of the section specified by the delimiter information,
Transform the video image to match the estimated position and orientation of the real world,
An information presentation method comprising superimposing a video image taken by the camera and a deformed video image. An information presentation program applied to a computer equipped with a camera that is mounted on a user and photographs the real world,
In the computer,
A position and orientation estimation process for estimating the position and orientation of the real world from the video captured by the camera;
Separation determination that determines segmentation of user's action by comparing segmentation information, which is information defined as segmentation in video images that are pre-captured images of real world scenes, and video captured by the camera processing,
According to the determination result, a playback control process for controlling playback of a video image in a section specified by the delimiter information;
Video image transformation processing for transforming the video image according to the estimated position and orientation of the real world, and
An information presentation program for executing a superimposition process for superimposing a video photographed by the camera and a deformed video video.

Images