JP2008065684A - Stereoscopic image synthesizing device, shape data generation method, and its program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereoscopic image synthesizing device for outputting stereoscopic image data and shape data by presenting the live stereoscopic image photographed by a video camera by a stereoscopic video display device, and presenting a haptic and tactile sense appropriate to the stereoscopic image by a haptic/tactile sense presenting device. <P>SOLUTION: In this stereoscopic image synthesizing device for synthesizing stereoscopic images from a left image obtained by imaging an object from the view point of the left eye and a right image obtained by imaging the object from the view point of the right eye, a shape calculation part 405 calculates the shape data of the object from the distance image of the input object. A shape video space adjusting part 407 converts the shape data calculated by the shape calculation part 405 into the display space of the stereoscopic images to be synthesized by its own device. A shape data output part 409 outputs the shape data converted by the shape video space adjusting part 407 to a haptic/tactile sense presenting device. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention particularly relates to a stereoscopic video composition apparatus, a shape data generation method, and a program for generating shape data together with stereoscopic video data.

A conventional stereoscopic image display device prepares an image viewed from two viewpoints corresponding to the left and right eyes in advance, and a three-dimensional display using a barrier method (see, for example, Patent Document 1 and Patent Document 2) or a polarizing glass shutter method. By displaying the above, the user can perceive in three dimensions.
Also, a force feedback device that has a pen-type operation unit and can experience force and tactile sensation by operating the pen, and it can be worn on the arm to experience the tactile sense of the entire arm and the tactile sensation of the hand There are also haptic / tactile presentation devices such as haptic devices.
JP-A-8-248355 Special table 2003-521181

  However, the conventional stereoscopic video display device only presents a stereoscopic video, and there is a problem that there is a stereoscopic effect and even if an object appears to protrude, it cannot be touched. Also, based on the shape data such as CAD (Computer Aided Design) data, the haptic and tactile sensation devices were able to present the haptic and tactile sensation while displaying the CG (Computer Graphics). However, there is a problem that even if it can be applied to a CG that requires shape data to generate an image, it cannot be applied to a real image captured by a video camera or the like.

  The present invention has been made in view of such circumstances, and an object of the present invention is to present a stereoscopic image of a real image captured by a video camera or the like on a stereoscopic image display device, and to match a force sense corresponding to the stereoscopic image. It is an object of the present invention to provide a 3D image synthesizing device capable of outputting 3D image data and shape data that can present tactile sensations with a force / tactile sense presentation device.

  The present invention has been made to solve the above-described problems, and the stereoscopic video composition apparatus of the present invention synthesizes a stereoscopic video from a left image obtained by imaging a subject from the viewpoint of the left eye and a right image obtained by imaging from the viewpoint of the right eye. In the stereoscopic image synthesizing apparatus, a shape calculation unit that calculates shape data of the subject from a distance image of the subject input from a distance image photographing device that captures a distance image in which each pixel represents a distance, and the shape calculation unit The shape image space adjustment unit that converts the shape data calculated by the device into the display space of the 3D image synthesized by the device, and the shape data converted by the shape image space adjustment unit is output to the sensation presentation device. And a shape data output unit.

  The stereoscopic video composition apparatus according to the present invention is the stereoscopic video composition apparatus described above, wherein three or more subjects are extracted from the left image and the right image based on respective colors, and the three or more subjects are extracted. The position of each of the three-dimensional images in the display space is calculated, the three or more subjects are extracted from the distance image based on each distance, and the position of each of the three or more subjects in the space based on the distance image is determined. And a conversion generation unit that calculates a conversion from the space based on the distance image to the display space based on the calculated positions, and the shape image space adjustment unit performs the conversion obtained by the conversion generation unit. And applying to the shape data.

  The stereoscopic video composition apparatus according to the present invention is the above-described stereoscopic video composition apparatus, wherein three or more stereoscopic video display space points designated by a user operation are associated with each of the points by a user operation. A conversion generation unit that calculates a conversion from the space based on the distance image to the display space based on the point of the distance image, and the shape video space adjustment unit performs the conversion obtained by the conversion generation unit, It is applied to the shape data.

  The shape data generation method of the present invention is a shape data generation method in a stereoscopic video composition device that synthesizes a stereoscopic video from a left image obtained by imaging a subject from a left eye viewpoint and a right image obtained from a right eye viewpoint, A first process in which the stereoscopic video synthesizing device calculates shape data of the subject from a distance image of the subject input from a distance image capturing device that captures a distance image in which each pixel represents a distance; and the stereoscopic video A second process in which the synthesizing apparatus converts the shape data calculated in the first process into a display space of the stereoscopic video synthesized by the own apparatus, and the stereoscopic video synthesizing apparatus includes the second process. And a third process of outputting the shape data converted in the process to the sensory presentation device.

  Further, the program of the present invention is a program for causing a computer to function as a stereoscopic video composition device that synthesizes a stereoscopic video from a left image obtained by imaging a subject from a left eye viewpoint and a right image obtained from a right eye viewpoint. A shape calculation unit that calculates shape data of the subject from the distance image of the subject that is input from a distance image capturing device that captures a distance image in which a pixel represents a distance, and the shape data calculated by the shape calculation unit, A shape image space adjustment unit that converts a stereoscopic image synthesized by the device into a display space, and a shape data output unit that outputs the shape data converted by the shape image space adjustment unit to a sensation presentation device.

  According to the present invention, a stereoscopic video display can be obtained by causing the present apparatus to input a real image obtained by photographing a subject with a video camera or the like from two left and right viewpoints and a video comprising a distance image obtained by photographing the subject with a distance image photographing device. It is possible to generate shape data that allows the haptic / tactile sensation presentation device to provide a haptic / tactile sensation that matches the stereoscopic image displayed on the device.

  As shown in FIG. 1, the stereoscopic video composition apparatus 400 according to the present embodiment includes a video viewed from the left eye viewpoint captured by the left video imaging apparatus 100 and a video viewed from the right eye viewpoint captured by the right video imaging apparatus 200. Are synthesized and displayed on the stereoscopic video display device 500, and the shape data of the subject is calculated from the distance image photographed by the distance image photographing device 300, and the subject shape data is used as a three-dimensional space based on the distance image. The shape data converted from the 3D image display space to the 3D image display space is calculated and output to the haptic / tactile sense presentation device 600. As a result, the user can view the stereoscopic video displayed on the stereoscopic video display device 500 and simultaneously obtain the haptic / tactile sensation with the haptic / tactile sensation presentation device 600 for the subject in the displayed stereoscopic video.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 is a schematic block diagram showing the configuration of the stereoscopic video composition apparatus 400 according to one embodiment of the present invention. The left image capturing apparatus 100 is a video camera that captures an image corresponding to an image viewed from the viewpoint of the user's left eye. The right image capturing device 200 is a video camera that is installed in parallel to the right side of the left image capturing device 100 and captures an image corresponding to an image viewed from the viewpoint of the user's right eye. The distance image capturing device 300 is installed in parallel in the vicinity of the left image capturing device 100 and the right image capturing device 200, and after the light irradiated by the present apparatus is irradiated, it is reflected by the measurement object and is reflected by the present apparatus. An image in which each pixel represents a distance from the present device is generated in real time by a time of flight (TOF) method in which the time until the light is received is obtained to determine the distance. The stereoscopic video composition apparatus 400 receives the video viewed from the left eye viewpoint and the video viewed from the right eye viewpoint from the left video imaging apparatus 100 and the right video imaging apparatus 200, and synthesizes the stereoscopic video to display the stereoscopic video display apparatus 500. And a distance image is received from the distance image capturing device 300, the shape data of the subject is calculated, the calculated shape data is converted and mapped to the display space of the stereoscopic image, and then the haptic / tactile sensation presentation device Output to 600.
In the present embodiment, the haptic / tactile sensation presentation device 600 functions as a sensation presentation device.

FIG. 3 is a schematic block diagram showing the configuration of the stereoscopic video composition apparatus 400 according to the present embodiment.
Reference numeral 401 denotes a left video data input unit that receives a video input from the left video shooting device 100 and outputs a left image extracted frame by frame from the video. Reference numeral 402 denotes a right video data input unit that receives a video input from the right video shooting device 200 and outputs a right image extracted frame by frame from the video. A distance image data input unit 403 receives a video input from the distance image capturing device 300 and outputs a distance image extracted frame by frame from the video.

  Reference numeral 404 denotes a stereoscopic video composition unit that synthesizes the left image and the right image received from the left video data input unit 401 and the right video data input unit 402 and generates stereoscopic video data in a format suitable for the stereoscopic video display device 500. It is. A shape extraction calculation unit 405 generates shape data of a subject based on the distance image received from the distance image data input unit. Details of the shape extraction calculation unit 405 will be described later with reference to FIG.

  A conversion generation unit 406 obtains a matrix representing affine transformation from a space based on the distance image to a stereoscopic video display space. If the positional relationship among the left image capturing device 100, the right image capturing device 200, and the distance image capturing device 300 does not change, this affine transformation does not change. Therefore, in the present embodiment, three markers having different colors are photographed in advance by the left image capturing device 100, the right image capturing device 200, and the distance image capturing device 300, and the conversion generation unit is based on the captured images. A matrix representing affine transformation is generated at 406, and this generated matrix is used for subsequent transformation in the shape video space adjustment unit 407.

  FIG. 4 is a flowchart for explaining the outline of the affine transformation generation performed in advance. The image conversion generation unit 406 extracts three markers from the left image and the right image based on the colors (S1: details will be described later with reference to FIG. 6), and each of these three markers in the stereoscopic video display space. The position is calculated based on the parallax between the left image and the right image (S2: details will be described later with reference to FIG. 7). Further, the conversion generation unit 406 extracts the three markers described above from the distance images captured simultaneously with the left image and the right image described above, and in the space based on the distance image of each of these three markers. The position is calculated (S3: details will be described later with reference to FIG. 8). Based on the position in the space based on the distance image of these three markers and the position in the display space of the stereoscopic video, conversion from the space based on the above-described distance image, which is affine transformation, to the display space of the stereoscopic video is obtained (S4). .

  Since any affine transformation can be expressed by a combination of rotation around each axis, enlargement / reduction in each axial direction, and parallel movement, the matrix representing the affine transformation is a total of nine unknowns each in three in a three-dimensional space. Consists of. For this reason, if the coordinates before and after the conversion of the three points that are not on the same straight line are known, nine linear expressions substituted with these can be established, so that a matrix representing the affine transformation can be uniquely obtained. In step S4 described above, the conversion generation unit 406 uses the space based on the distance image of the three subjects and the position (coordinates) in the display space of the stereoscopic video as the coordinates before and after the conversion of the three points, and uses the affine transformation. Find the matrix that represents.

  The shape video space adjustment unit 407 uses the matrix representing the affine transformation obtained by the conversion generation unit 406 to convert the subject shape data generated by the shape extraction calculation unit 405 into a stereoscopic video display space. That is, affine transformation is performed by multiplying a matrix representing affine transformation by a vector of each point of shape data. A stereoscopic video output unit 408 outputs the stereoscopic video data generated by the stereoscopic video synthesis unit 404 to the stereoscopic video display device 500. Reference numeral 409 denotes a shape output unit that outputs the shape data converted by the shape image space adjustment unit 407 to the haptic / tactile sense presentation device 600.

FIG. 5 is a flowchart for explaining processing for generating shape data of a subject based on the distance image received from the distance image data input unit 403 in the shape extraction calculation unit 405. First, in order to extract a subject, a distance extraction threshold Dm is set in advance by a user operation, and the shape extraction calculation unit 405 stores the extraction threshold Dm (Sa1).
Next, the shape extraction calculation unit 405 performs the processing of steps Sa3 to Sa6 for all Imax pixels from i = 0 to Imax-1 constituting the distance image received from the distance image data input unit 403 (Sa2). ). In step Sa3, the shape lottery calculation unit 405 acquires the distance Di of the i-th (initially 0th) pixel. Next, the shape extraction calculation unit 405 determines whether the distance Di acquired in step Sa3 is a distance to be extracted as a subject, that is, whether it is smaller than the extraction threshold Dm stored in the storage unit in step Sa1. Is determined (Sa4). If it is determined in step Sa4 that the distance Di is smaller than the extraction threshold value Dm, the process proceeds to step Sa5, and the shape extraction calculation unit 405 determines the position in the distance image and its distance for the i-th pixel at this time. After storing Di, the process proceeds to step Sa6.

  On the other hand, if it is determined in step Sa4 that the distance Di is not smaller (same or larger) than the extraction threshold Dm, the process directly proceeds to step Sa6. In step Sa6, the shape extraction calculation unit 405 increases the value of i by 1, and when the value of i is smaller than Imax, the shape lottery calculation unit 405 returns to step Sa3 and repeats the above-described processing. In this way, until the value of i reaches Imax, that is, for all the pixels of the distance image, the shape extraction calculation unit 405 repeats steps Sa3 to Sa6.

  When the value of i becomes Imax in step Sa6, the process proceeds to step Sa7, and among the pixels whose positions are stored in step Sa6, the shape extraction calculation unit 405 is adjacent to the position in the distance image vertically or horizontally. Create a group of things that match. Here, whether or not any two pixels are adjacent to each other is determined based on whether the position in the horizontal axis direction of the two pixels is the same and the position in the vertical axis direction is shifted by one pixel, or the position in the horizontal axis direction is If one of the conditions of shifting by one pixel and the same position in the vertical axis direction is satisfied, it is possible to try to adjoin each other. Next, the shape extraction calculation unit 405 polygonizes each group generated in step Sa7 with a triangle formed by three adjacent points based on the pixel position (horizontal axis direction and vertical axis direction) and distance. Shape data is generated (Sa8).

FIG. 6 is a flowchart for describing processing for calculating the position of the marker in the left image and the right image in the conversion generation unit 406. The conversion generation unit 406 calculates the positions of the three markers by performing the process of the flowchart shown in FIG. 6 three times for each of the left image and the right image while changing the extraction threshold. In other words, the process of FIG. 6 is performed a total of six times by performing three markers on two images on the left and right. In the present embodiment, the color values for expressing the color of each pixel are represented by red, green, and blue component values.
First, in order to detect three markers, an upper limit value (Rmax, Gmax, Bmax) and a lower limit value (Rmin, Gmin, Bmin) are previously set by user operation as extraction threshold values for red, green, and blue components of each marker. Then, the conversion generation unit 406 stores these values in the storage unit (Sb1).

Next, the conversion generation unit 406 performs steps Sb3 to Sb6 for all the Jmax pixels from j = 0 to Jmax−1 that constitute the image received from the left video data input unit 401 or the right video data input unit 402. Processing is performed (Sb2). In step Sb3, the conversion generation unit 406 acquires the red, green, and blue component values (Rj, Gj, Bj) of the jth (initially 0th) pixel. Next, the conversion generation unit 406 stores in the storage unit whether or not the red, green, and blue component values (Rj, Gj, Bj) acquired in step Sb3 are marker colors, that is, in step Sb1. It is determined whether it is within the range of the upper limit value (Rmax, Gmax, Bmax) and the lower limit value (Rmin, Gmin, Bmin), that is, within the extraction threshold range (Sb4). In this determination, when all of the following expressions (1) to (3) are satisfied, it is determined that the value is within the extraction threshold range.
Rmin <Rj <Rmax (1)
Gmin <Gj <Gmax (2)
Bmin <Bj <Bmax (3)
If it is determined in step Sb4 that it is within the range of the extraction threshold, the process proceeds to step Sb5, and the conversion generation unit 406 stores the position in the image for the j-th pixel at this time, Transition to Sb6.

  On the other hand, if it is determined in step Sb4 that it is not within the extraction threshold range, the process directly proceeds to step Sb6. In step Sb6, the conversion generation unit 406 increases the value of j by 1, and when the value of j is smaller than Jmax, the process proceeds to step Sb3 and repeats the above processing. In this way, the conversion generation unit 406 repeats steps Sb3 to Sb6 until the value of j reaches Jmax, that is, for all the pixels of the image.

  When the value of j reaches Jmax in step Sb6, the process proceeds to step Sb7, and the conversion generation unit 406 has pixels whose positions are stored in step Sb6 whose positions in the image are adjacent vertically or horizontally. Generate a group that summarizes. Here, whether or not any two pixels are adjacent to each other is determined by determining whether the two pixels have the same horizontal axis position and the vertical axis position is shifted by one pixel, or the horizontal axis direction position is one pixel. If any one of the conditions of being shifted and having the same position in the vertical axis direction is satisfied, it is possible to make them adjacent to each other. Next, the conversion generation unit 406 determines that the group having the largest area, that is, the one having the largest number of pixels, is extracted from the group generated in step Sb7, and extracts the center of gravity of the extracted marker. The coordinates are calculated and output by averaging the coordinates of the pixels constituting the marker (Sb8).

  For example, the X and Y coordinates of the pixel whose position is stored by the conversion generation unit 406 in step Sb6 are (10, 10), (10, 11), (11, 11), (25, 60), (24, 61), (25, 61), (26, 61), and (25, 62), assuming that there are 8 pixels, in step Sb7, the conversion generation unit 406 selects (10, 10), (10, 11), Group 1 consisting of three pixels (11, 11) and five pixels (25, 60), (24, 61), (25, 61), (26, 61), (25, 62) Group 2 is generated. Next, in step Sb8, the number of pixels of the conversion generation unit 406, the group 1 of 3 pixels and the group 2 of 5 pixels are compared, and the group 2 having a large number of pixels is determined to be a marker and extracted. Is calculated by obtaining the average of (25, 60), (24, 61), (25, 61), (26, 61), and (25, 62). That is, the position of the center of gravity is ((25 + 24 + 25 + 26 + 25) / 5, (60 + 61 + 61 + 61 + 62) / 5) = (25, 61).

FIG. 7 is a diagram illustrating a method for calculating the position of the marker in the Z-axis direction, that is, the direction perpendicular to the image (depth direction) in the stereoscopic video display space by the conversion generation unit 406. In this figure, only one marker is described for the sake of simplicity, but the conversion generation unit 406 performs the process of FIG. 7 for each of the three markers to calculate the position of each marker. The coordinate XL is a coordinate in the horizontal axis direction among the barycentric positions of the marker image M1 in the left image G1 calculated by the conversion generation unit 406 in the process of FIG. 6, and has the left end of the left image G1 as the origin. The coordinate XR is a coordinate in the horizontal axis direction among the barycentric positions of the marker image M2 in the right image G2 calculated by the conversion generation unit 406 in the process of FIG. 6, and has the left end of the right image G2 as the origin. The direction perpendicular to the image (Z-axis) is based on the viewpoint of the user viewing the stereoscopic video displayed on the stereoscopic video display device 500, and the conversion generator 406 is perpendicular to the image. The coordinate Z of the direction is calculated by equation (4).
Z = 1 / (XL-XR) (4)
Further, the conversion generation unit 406 takes the average of the barycentric positions of the marker images in the left image and the right image calculated in the process of FIG. 6, so that the X-axis direction of the marker in the stereoscopic video display space, that is, the horizontal direction of the image And the Y-axis direction, that is, the vertical coordinate of the image is calculated.

For example, the X coordinate XL = 80 and Y coordinate YL = 42 of the centroid position of the marker image in the left image calculated in the process of FIG. 6, and the X coordinate XR = 50 and Y coordinate of the centroid position of the marker image in the right image When YR = 40, the conversion generation unit 406 calculates the X coordinate in the stereoscopic image display space using Equation (5), the Y coordinate using Equation (6), and the Z coordinate using Equation (7). Y, Z) = (65, 41, 0.033).
X = (XL + XR) / 2 = (80 + 50) / 2 = 65 (5)
Y = (YL + YR) / 2 = (42 + 40) / 2 = 41 (6)
Z = 1 / (XL-XR) = 1 / (80-50) = 0.033 (7)
Here, the value of the Z coordinate is very small compared to the value of the X coordinate and the Y coordinate. This is because the value of the Z coordinate obtained by the equation (4) is the X coordinate and the Y coordinate. This is because the scale is different from that of FIG. 2, and this is adjusted by multiplying the Z coordinate by a predetermined constant C. Further, the magnitude of the predetermined constant C may be adjusted so as to emphasize the position in the Z-axis direction.

FIG. 8 is a flowchart for explaining the process of calculating the position in the space based on the distance image of each of the three markers in the conversion generation unit 406. In this figure, only one marker will be described for the sake of simplicity, but the conversion generation unit 406 performs the process of FIG. 8 for each of the three markers, and in the space based on the distance image of the three markers. Calculate the position. That is, the process of FIG. 8 is performed three times.
First, in order to detect three markers, a lower limit value Dmin and an upper limit value Dmax, which are distance extraction threshold values, are set in advance by a user operation for each marker, and the conversion generation unit 406 sets the lower limit value Dmin and the upper limit value Dmax. Store in the storage unit (Sc1).
Next, the conversion generation unit 406 performs the processes of Steps Sc3 to Sc6 for all Kmax pixels from k = 0 to Kmax−1 constituting the distance image received from the distance image data input unit 403 (Sc2). . In step Sc3, the conversion generation unit 406 acquires the distance Dk of the kth (initially 0th) pixel. Next, the conversion generation unit 406 determines whether or not the distance Dk acquired in step Sc3 is a distance to be extracted as a marker, that is, the lower limit value Dmin and the upper limit value Dmax stored in the storage unit in step Sc1. It is determined whether it is between (Sc4). If it is determined in step Sc4 that the distance Dk is between the lower limit value Dmin and the upper limit value Dmax, the process proceeds to step Sc5, and the conversion generation unit 406 determines that the kth pixel at this time is in the distance image. After storing the position and the distance Dk at, the process proceeds to step Sc6.

  On the other hand, if it is determined in step Sc4 that the distance Dk is not between the lower limit value Dmin and the upper limit value Dmax, the process directly proceeds to step Sc6. In step Sc6, the conversion generation unit 406 increases the value of k by 1, and when the value of k is smaller than Kmax, the conversion generation unit 406 returns to step Sc3 and repeats the above-described processing. In this way, the conversion generation unit 406 repeats Steps Sc3 to Sc6 until the value of k becomes Kmax, that is, for all the pixels of the distance image.

  In step Sc6, when the value of k becomes Kmax, the process proceeds to step Sc7, and the conversion generation unit 406 makes the positions in the distance image adjacent to each other vertically or horizontally among the pixels whose positions are stored in step Sc6. Create a group of things together. Here, whether or not any two pixels are adjacent to each other is determined based on whether the position in the horizontal axis direction of the two pixels is the same and the position in the vertical axis direction is shifted by one pixel, or the position in the horizontal axis direction is If one of the conditions of shifting by one pixel and the same position in the vertical axis direction is satisfied, it is possible to try to adjoin each other. Next, the conversion generation unit 406 determines that the group having the largest area, that is, the one having the largest number of pixels, is extracted from the group generated in Step Sc7, and extracts the center of gravity of the extracted marker. The coordinates are calculated and output by averaging the coordinates of the pixels constituting the marker (Sc8).

As described above, the shape video space adjustment unit 407 uses the affine transformation generated by the conversion generation unit 406 to convert the shape data from the space based on the distance image to the stereoscopic video display space. The video composition device 400 outputs shape data in which the motion and coordinate system coincide with the live-action stereoscopic video, and when the live-action video is displayed on the stereoscopic video display device 500, a request for a tactile sensation that anyone requests intuitively. On the other hand, it can be realized by the force / tactile sense presentation device 600 that has received the shape data. Conventional live-action 3D images were simply viewed as 3D objects, but the addition of tactile sensation makes it possible to grasp the 3D objects more reliably and expand the possibilities of new media and interfaces.
In addition, since the stereoscopic video composition apparatus 400 according to this embodiment can output stereoscopic video data and shape data for a plurality of subjects, it can be used particularly effectively in fields such as remote communication and spatial recording. .

Note that the storage unit of the stereoscopic video composition device 400 includes a hard disk device, a magneto-optical disk device, a nonvolatile memory such as a flash memory, a storage medium that can only be read such as a CR-ROM, and a RAM (Random Access Memory). Such a volatile memory, or a combination thereof.
In addition, it is assumed that an input device, a display device, and the like (none of which are shown) are connected to the stereoscopic video composition device 400 as peripheral devices. Here, the input device refers to an input device such as a keyboard and a mouse. The display device refers to a CRT (Cathode Ray Tube), a liquid crystal display device, or the like.

  Also, the left video data input unit 401, the right video data input unit 402, the distance image data input unit 403, the stereoscopic video synthesis unit 404, the shape extraction calculation unit 405, the conversion generation unit 406, the shape video space adjustment unit 407 in FIG. A program for realizing the functions of the stereoscopic image output unit 408 and the shape output unit 409 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system and executed. Video data input unit 401, right video data input unit 402, distance image data input unit 403, stereoscopic video composition unit 404, shape extraction calculation unit 405, conversion generation unit 406, shape video space adjustment unit 407, stereoscopic video output unit 408, Processing of the shape output unit 409 may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

In the present embodiment, the conversion generation unit 406 has been described as generating an affine transformation from an image obtained by photographing three markers, but an affine transformation is produced from an image obtained by photographing more than three markers. Alternatively, one marker may be photographed three or more times at different locations, and affine transformation may be generated from these images. Further, instead of photographing the marker in advance, the marker may be photographed together with the subject, and the affine transformation may be generated in real time.
In the present embodiment, the conversion generation unit 406 extracts three markers with respect to the color in the left image and the right image, and extracts with reference to the distance in the distance image. A point in the display space of the stereoscopic video that is displayed by the conversion generation unit 406 of the stereoscopic video synthesis apparatus 400 on the left image and the right image using an input device such as a mouse. The conversion generation unit 406 accepts a point specified by the user with respect to the distance image using an input device such as a mouse as a marker, and the conversion generation unit 406 may generate an affine transformation based on the marker. . Also at this time, three or more markers are required to generate the affine transformation. Thus, affine transformation from a space based on a distance image to a stereoscopic video display space can be obtained based on an image obtained by photographing an arbitrary scene without using a marker.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within a scope not departing from the gist of the present invention.

  The stereoscopic video composition apparatus of the present invention is suitable for use in remote communication or spatial recording, but is not limited to this.

It is a figure explaining the outline | summary of one Embodiment of this invention. It is a schematic block diagram which shows the structure of the system using the three-dimensional video composition apparatus 400 in the embodiment. It is a schematic block diagram which shows the structure of the three-dimensional video composition apparatus 400 in the embodiment. It is a flowchart explaining operation | movement of the conversion production | generation part 406 in the embodiment. 6 is a flowchart for describing processing for generating shape data of a subject in the shape extraction calculation unit 405 according to the embodiment. It is a flowchart explaining the process which calculates the position of the marker in a left image and a right image in the conversion production | generation part 406 in the embodiment. It is a figure explaining the calculation method regarding the position of the Z-axis direction of the marker in the display space of a three-dimensional image in the conversion production | generation part 406 in the embodiment. It is a flowchart explaining the process which calculates the position in the space based on the distance image of each of three markers in the conversion production | generation part 406 in the embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Left image imaging device 200 ... Right image imaging device 300 ... Distance image imaging device 400 ... Three-dimensional image composition device 500 ... Three-dimensional image display device 600 ... Haptic / tactile sense presentation device 401 ... Left image data input unit 402 ... Right image data Input unit 403 ... Distance image data input unit 404 ... Stereoscopic image synthesis unit 405 ... Shape extraction calculation unit 406 ... Conversion generation unit 407 ... Shape video space adjustment unit 408 ... Stereoscopic video output unit 409 ... Shape output unit

Claims (5)

In a stereoscopic video composition device that synthesizes a stereoscopic video from a left image captured from the viewpoint of the left eye and a right image captured from the viewpoint of the right eye,
A shape calculating unit that calculates shape data of the subject from a distance image of the subject input from a distance image capturing device that captures a distance image in which each pixel represents a distance;
A shape image space adjustment unit that converts the shape data calculated by the shape calculation unit into a display space of a stereoscopic image synthesized by the device;
A three-dimensional image synthesizing device comprising: a shape data output unit that outputs shape data converted by the shape image space adjustment unit to a sensation presentation device. Three or more subjects are extracted from the left image and the right image based on the respective colors, the positions of the three or more subjects in the display space of the stereoscopic video are calculated, and each distance from the distance image is calculated. The three or more subjects are extracted based on the image, the positions of the three or more subjects in the space based on the distance image are calculated, and the display is performed from the space based on the distance image based on the calculated positions. It has a conversion generator that calculates the conversion to space,
The stereoscopic video composition apparatus according to claim 1, wherein the shape video space adjustment unit performs the conversion obtained by the conversion generation unit on the shape data. From the space based on the distance image to the display space based on three or more points in the display space of the stereoscopic video designated by the user operation and the points of the distance image associated with the respective points by the user operation A conversion generation unit that calculates conversion to
The stereoscopic video composition apparatus according to claim 1, wherein the shape video space adjustment unit performs the conversion obtained by the conversion generation unit on the shape data. A shape data generation method in a stereoscopic video synthesizing apparatus that synthesizes a stereoscopic video from a left image captured from a left eye viewpoint and a right image captured from a right eye viewpoint,
A first process in which the stereoscopic video composition device calculates shape data of the subject from a distance image of the subject input from a distance image photographing device that captures a distance image in which each pixel represents a distance;
A second process in which the stereoscopic video synthesizing device converts the shape data calculated in the first process into a display space of the stereoscopic video synthesized by the own device;
A method of generating shape data, comprising: a third process in which the stereoscopic video composition apparatus outputs the shape data converted in the second process to a sensation presentation apparatus. In a program for causing a computer to function as a stereoscopic video composition device that synthesizes a stereoscopic video from a left image obtained by imaging a subject from a left eye viewpoint and a right image obtained by imaging from a right eye viewpoint,
A shape calculating unit that calculates shape data of the subject from a distance image of the subject input from a distance image capturing device that captures a distance image in which each pixel represents a distance;
A shape image space adjustment unit that converts the shape data calculated by the shape calculation unit into a display space of a stereoscopic image synthesized by the device;
A program for functioning as a shape data output unit for outputting shape data converted by the shape image space adjustment unit to a sensation presentation device.

Images