JP2008067169A - Three-dimensional video composing device, shape data generation method and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional video composing device, a shape data generation method and a program therefor by which three-dimensional video data and shape data which can present a sense of force/tactile sense matched to a three-dimensional video image of actual video image photographed by a video camera, etc. by a sense of force/tactile sense presentation device are output while presenting the three-dimensional video image by a three-dimensional video display device. <P>SOLUTION: In the three-dimensional video composing device which composes the three-dimensional video images from a left image seen from a viewpoint of a left eye and a right image seen from a viewpoint of a right eye, a shape calculation part 33 calculates the shape data of a subject from the left image and the right image. A shape output part 34 outputs the shape data arranged by the shape calculation part 33 to the sense of force/tactile sense presentation device. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a stereoscopic video composition apparatus, a shape data generation method, and a program therefor, in particular, in addition to the synthesis of a stereoscopic video, and the output of shape data to a haptic / tactile sense presentation apparatus.

A conventional stereoscopic image display device prepares an image viewed from two viewpoints corresponding to the left and right eyes in advance, and a three-dimensional display using a barrier method (see, for example, Patent Document 1 and Patent Document 2) or a polarizing glass shutter method. By displaying the above, the user can perceive in three dimensions.
Also, a force feedback device that has a pen-type operation unit and can experience force and tactile sensation by operating the pen, and it can be worn on the arm to experience the tactile sense of the entire arm and the tactile sensation of the hand There are also haptic / tactile presentation devices such as haptic devices.
JP-A-8-248355 Special table 2003-521181

  However, the conventional stereoscopic video display device only presents a stereoscopic video, and there is a problem that there is a stereoscopic effect and even if an object appears to protrude, it cannot be touched. Also, based on the shape data such as CAD (Computer Aided Design) data, the haptic and tactile sensation devices were able to present the haptic and tactile sensation while displaying the CG (Computer Graphics). However, there is a problem that even if it can be applied to a CG that requires shape data to generate an image, it cannot be applied to a real image captured by a video camera or the like.

  The present invention has been made in view of such circumstances, and an object of the present invention is to present a stereoscopic image of a real image captured by a video camera or the like on a stereoscopic image display device, and to match a force sense corresponding to the stereoscopic image. It is an object of the present invention to provide a 3D image synthesizing device capable of outputting 3D image data and shape data that can present tactile sensations with a force / tactile sense presentation device.

  The present invention has been made in order to solve the above-described problems, and the stereoscopic video composition apparatus of the present invention is a stereoscopic video that synthesizes a stereoscopic video from a left image viewed from the left eye viewpoint and a right image viewed from the right eye viewpoint. In the video composition device, a shape calculation unit that calculates shape data of a subject from the left image and the right image, and a shape output unit that outputs the shape data calculated by the shape calculation unit to the haptic / tactile sense presentation device; It is characterized by providing.

  The stereoscopic video composition device of the present invention is the stereoscopic video composition device described above, wherein the shape calculation unit extracts an image of a specific subject from each of the left image and the right image, and For each pixel of the image, the vertical coordinate is given to the image according to the distance from the outline of the image of the subject to generate shape data of the subject, and the stereoscopic image display space synthesized by the device itself A position of a specific subject is calculated based on the parallax of the extracted image, and shape data in which the generated shape data is arranged at the calculated position is calculated.

  The stereoscopic video composition apparatus of the present invention is the above-described stereoscopic video composition apparatus, wherein the shape calculation unit extracts an image of a specific subject from each of the left image and the right image, and The object shape data is calculated by stereo measurement based on parallax.

  The stereoscopic video composition apparatus of the present invention is the stereoscopic video composition apparatus described above, wherein the shape calculation unit calculates the shape data of the subject by stereo measurement based on the parallax between the left image and the right image. It is characterized by that.

  Further, the shape data generation method of the present invention is the shape data generation method in a stereoscopic video composition apparatus for synthesizing a stereoscopic video from a left image viewed from the left eye viewpoint and a right image viewed from the right eye viewpoint. A first process in which the apparatus calculates shape data of a subject from the left image and the right image, and a shape data calculated in the first process by the stereoscopic video composition apparatus is displayed as a haptic / tactile sensation. And a second process of outputting to the apparatus.

  Further, the program of the present invention is a program for causing a computer to function as a stereoscopic video composition device that synthesizes a stereoscopic video from a left image viewed from the left eye viewpoint and a right image viewed from the right eye viewpoint. And the shape calculation unit for calculating the shape data of the subject from the right image, and the shape data calculated by the shape calculation unit also function as a shape output unit for outputting to the haptic / tactile sensation presentation device.

  According to the present invention, the stereoscopic video synthesizing device inputs the real video captured by the video camera or the like from the two left and right viewpoints to the device, thereby matching the haptics and the stereoscopic video displayed on the stereoscopic video display device. Shape data for providing a tactile sensation with a force sense / tactile sense presentation device can be generated.

  An outline of an embodiment of the present invention will be described. As shown in FIG. 1, the stereoscopic video synthesis apparatus 300 synthesizes a stereoscopic video by combining a video from the left eye viewpoint and a video from the right eye viewpoint captured by the left video imaging apparatus 100 and the right video imaging apparatus 200. When displaying on the video display device 400, shape data is calculated from the captured video and output to the haptic / tactile sensation presentation device 500. As a result, the user can view the stereoscopic video displayed on the stereoscopic video display device 400 and simultaneously obtain the haptic / tactile sensation with the haptic / tactile sensation presentation device 500 for a shape that matches the movement of the displayed stereoscopic video. Can do.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 is a schematic block diagram showing a configuration of the stereoscopic video image synthesizing apparatus 300 according to the embodiment of the present invention. The left image capturing device 100 is a video camera that captures an image from the viewpoint of the left eye. The right image capturing device 200 is a video camera that is installed in parallel to the right side of the left image capturing device 100 and captures an image from the viewpoint of the right eye. The stereoscopic video synthesizing device 300 receives the left-eye and right-eye videos from the left video photographing device 100 and the right video photographing device 200, synthesizes the stereoscopic video, and outputs the synthesized video to the stereoscopic video display device 400. Shape data matching the movement is calculated and output to the haptic / tactile sensation presentation device 500.

FIG. 3 is a schematic block diagram illustrating the configuration of the stereoscopic video composition apparatus 300 according to the present embodiment.
A left video data input unit 31 receives a video input from the left video shooting device 100 and outputs a left image extracted frame by frame from the video.
Reference numeral 32 denotes a right video data input unit that receives a video input from the right video shooting device 200 and outputs a right image extracted frame by frame from the video.
33 generates a left subject image and a right subject image obtained by extracting only the subject from the left image and the right image received from the left video data input unit 31 and the right video data input unit 32, and based on these two images, It is a shape calculation unit that calculates shape data synchronized with the stereoscopic video data generated by the stereoscopic video synthesis unit 35 by calculating the shape data. Details of the subject image generation method and the subject shape data calculation method in the shape calculation unit 33 will be described later.

A shape output unit 34 outputs the shape data calculated by the shape calculation unit 33 to the haptic / tactile sensation presentation device 500.
The stereoscopic video composition unit 35 combines the left image and the right image received from the left video data input unit 31 and the right video data input unit 32 to generate stereoscopic video data in a format suitable for the stereoscopic video display device 400. .
A stereoscopic video output unit 36 outputs the stereoscopic video data generated by the stereoscopic video synthesis unit 35 to the stereoscopic video display device 400.

FIG. 4 is a flowchart for explaining a method for generating a subject image in the shape calculation unit 33. The flowchart in FIG. 4 shows processing when a subject is extracted from the left image to generate a left subject image, but the shape calculation unit 33 extracts the subject for the right image in the same manner as for the left image. A right subject image is generated.
The shape calculation unit 33 captures only the background with the left image capturing device 10 and stores in advance the left background image received and extracted by the left image data input unit 31 (Sa1). When the shape calculation unit 33 receives the left image from the left video data input unit 31, the shape calculation unit 33 acquires the left background image stored in step Sa <b> 1, and all the pixels i from i = 0 to Imax−1 constituting the left image. Steps Sa3 to Sa6 are repeated for (Sa2).

  First, the shape calculation unit 33 acquires the red, green, and blue component values of the pixel i in the left image and the red, green, and blue component values of the pixel corresponding to the pixel i in the left background image (Sa3). Then, it is determined whether the red, green, and blue component values of these two pixels match (Sa4). If it is determined in step Sa4 that they match, the process directly proceeds to step Sa6. If it is determined that they do not match, the shape calculation unit 33 extracts the pixel i in the left image as a pixel of the subject image (Sa5 ), And the process proceeds to step Sa6. In step Sa6, the shape calculation unit 33 adds 1 to the value of i, and then determines whether or not i is smaller than Imax by performing the processing of steps Sa3 to Sa5 for all the pixels i constituting the left image. , I is smaller than Imax, the process proceeds to step Sa3 to repeat the above-described processing. When i is not smaller than Imax, the processing is terminated. The image composed of the extracted pixels is the left subject image. That is, in the subject image, the color of the pixels in the subject area is set, but the color of the pixels outside the subject area is not set.

  FIG. 5 is a flowchart for explaining a method for calculating a direction (depth direction) perpendicular to the subject image, out of the subject shape data generation method based on the left subject image or the right subject image in the shape calculation unit 33. FIG. 6 is a diagram showing possible values of the state S used in the flowchart. In the state S, there are a subject area outside and a subject area, and the subject area has a depth added and a depth not added. Pixels for which colors are set are subject areas, and pixels for which colors are not set are outside the subject areas. With this method, the position in the direction perpendicular to the image (depth direction) is calculated for each pixel in one of the left subject image and the right subject image. Regarding the position of the image of each pixel in the horizontal axis direction and the vertical axis direction, the center of gravity of the subject area in the subject image is the average of the center of gravity of the subject area in the left subject image and the center of gravity of the subject area in the right subject image. It is calculated by moving to the position of.

  Before performing this process in advance, the shape calculation unit 33 calculates an initial value D0 of the depth to be added using a method described later with reference to FIG. 8 (Sb1). When the process is started, the shape calculation unit 33 sets an initial value D0 in a variable D that is set as the depth of each pixel in a subsequent step Sb10 (Sb2). Next, the shape calculation unit 33 performs the processing of Steps Sb4 to Sb10 sandwiched between Step Sb3 and Step Sb11 for all pixels from i = 0 to Imax-1 constituting the subject image (Sb3). . In step Sb4, the shape calculation unit 33 acquires the state S of the pixel i in the subject image (Sb4). In acquiring the state S, whether the pixel i is out of the subject area or the subject area can be determined by whether or not the color of the pixel i is set. Whether the depth has been added or not has been determined based on whether or not the depth of the pixel i has been given. However, the pixel that has been given after step Sb14 is not given depth. That is, the loop from 0 to Imax of steps Sb3 to Sb11 is repeated many times so as to give different depth values through step Sb14, but the depth is given between the loops giving the same value of depth. The pixel is regarded as having no depth.

  Next, the shape calculation unit 33 determines whether or not the state S of the pixel i acquired in step Sb4 is the subject area (Sb5). The determination in step Sb5 can be made based on whether or not a color is set for the pixel. If the color is set, it is the subject area, and if it is not set, it is outside the subject area. If it is determined in step Sb5 that it is not the subject area, the process proceeds to step Sb11, and the process proceeds to the next pixel. If it is determined that the color is set and the object area is determined, the process proceeds to step Sb6.

  In step Sb6, the shape calculation unit 33 determines whether or not a depth is not given to the pixel (Sb6). If it is determined that a depth has already been given to the pixel and the pixel has not been given, the process proceeds to step Sb11 to proceed to processing for the next pixel, but the pixel has not yet been given a depth and has not been given. Then, the shape calculation unit 33 acquires the state Sn (n = 1 to 8) of the eight pixels surrounding the pixel i (Sb7). The shape calculation unit 33 determines whether or not there is a state Sn out of the subject area among the eight pixel states Sn (Sb8). If the shape calculation unit 33 determines that at least one of the eight pixels is outside the subject area, the shape calculation unit 33 proceeds to step Sb10 and assigns the value of the depth D as the depth value of the pixel i (Sb10). The process proceeds to step Sb11 to proceed to processing for the next pixel. If it is determined in step Sb8 that there are no objects outside the subject area in the state Sn, the process proceeds to step Sb9, and the shape calculation unit 33 determines that the state Sn has been given a depth. It is determined whether or not there is (Sb9).

If it is determined that there is no assigned one, the process proceeds to step Sb11 and the process for the next pixel is performed. If it is determined in step Sb9 that there is a depth-given one, the process proceeds to step Sb10, and the shape calculation unit 33 assigns a depth D value as the depth value of the pixel i (Sb10). The process proceeds to step Sb11. In step Sb11, the shape calculation unit 33 adds 1 to the value of i, returns to step Sb4 while i is smaller than Imax, and performs the processing of steps Sb4 to Sb10 for all pixels. When i becomes equal to or greater than Imax, the process proceeds to step Sb12, and the shape calculation unit 33 determines whether there is a pixel in the subject area of the subject image that has not been given a depth (Sb12). If it is determined that there are no unassigned ones, the shape calculating unit 33 terminates the process. If it is determined that there are unassigned ones, the shape calculating unit 33 proceeds to step Sb14 and sets a preset addition value ΔD (Sb13). It adds to the variable D, changes to step Sb3, and repeats the above-mentioned process. In this manner, the shape calculation unit 33 can calculate the position of the depth for all the pixels in the subject area according to the distance from the outline of the subject area.
Note that the addition value ΔD added to the variable D in step Sb13 described above may be a constant or a value that varies depending on the number of additions.

FIG. 7 is a diagram illustrating a process of calculating the depth of the subject area A1 by the method of FIG. When there is a subject area A1 shown in FIG. 7A, the shape calculation unit 33 first assigns a depth D0 to the outermost area A2 shown in FIG. 7B. Next, the shape calculation unit 33 gives the depth D0 + ΔD to the second outline region A3 shown in FIG. Next, the shape calculation unit 33 assigns the depth D0 + 2 · ΔD to the third outline region A4 illustrated in FIG. Thus, since the depth is given to the pixels of all the subject areas, the processing of FIG.
FIG. 8 is a diagram illustrating a method for calculating the initial value D0 in step S1 of FIG.
The coordinate XL is a coordinate in the horizontal axis direction among the barycentric positions of the subject area M1 extracted from the left subject image G1 calculated by the shape calculation unit 33 described with reference to FIG. 4, and the origin is the left end of the left subject image G1. . The position of the center of gravity is obtained by averaging the coordinates of all the pixels in the subject area M1. The coordinate XR is a coordinate in the horizontal axis direction among the barycentric positions of the subject area M2 extracted from the right subject image G2 calculated by the shape calculation unit 33 described with reference to FIG. 4, and the left end of the right subject image G2 is the origin. . For the direction perpendicular to the image, the origin of the viewpoint of the user viewing the stereoscopic video displayed on the stereoscopic video display device 400 is the origin, and the shape calculation unit 33 assigns the coordinate Z in the direction perpendicular to the image, that is, gives The initial value D0 of the depth is calculated by the equation (1).
D0 = 1 / (XL-XR) (1)

For example, the X coordinate XL = 80 and Y coordinate YL = 42 of the centroid position of the subject area M1 in the left image calculated by the shape calculation unit 33, and the X coordinate XR = 50 and Y of the centroid position of the subject area M2 in the right image. When the coordinate YR = 40, the shape calculation unit 33 calculates the Z coordinate in the stereoscopic video display space, that is, the initial value D0 of the given depth, using the equation (2).
D0 = 1 / (XL-XR) = 1 / (80-50) = 0.033 (2)
Here, the value of the Z coordinate is very small compared to the value of the X coordinate and the Y coordinate. This is because the value of the Z coordinate obtained by the equation (2) is the X coordinate and the Y coordinate. This is because the scale is different from that of FIG. 2, and this is adjusted by multiplying the Z coordinate by a predetermined constant C. Further, the magnitude of the predetermined constant C may be adjusted so as to emphasize the position in the Z-axis direction.

  Further, the position of the pixel I in the outermost area of the subject area M1 and having the X coordinate X = 78 and the Y coordinate Y = 40 in the stereoscopic image display space is calculated as follows. Since the depth is the outermost region, D0 = 0.333. The average (Xm, Ym) of the centroid position of the subject area M1 and the centroid position of the subject area M2 is Xm = (XL + XR) / 2 = (80 + 50) / 2 = 65, Ym = (YL + YR) / 2 = (42 + 40) / 2 = 41. The movement from (XL, YL) to (Xm, Ym) is Xm−XL = 65−80 = −15 in the X axis direction and Ym−YL = 41−42 = −1 in the Y axis direction. When (X = 78, Y = 40) is translated from (XL, YL) to (Xm, Ym), the X coordinate is 78 = -15 = 63 and the Y coordinate is 40-1 = 39. Become. Accordingly, the position of the pixel I in the stereoscopic image display space is (63, 39, 0.333).

As a result, the stereoscopic video composition device 300 according to the present embodiment calculates and outputs shape data that matches the motion of the live-action stereoscopic video, and displays the real-time stereoscopic video on the stereoscopic video display device 400. The tactile sensation requested by anyone intuitively can be realized by the haptic / tactile sensation presentation device 500 that has received shape data. Conventional live-action 3D images were simply viewed as 3D objects, but the addition of tactile sensation makes it possible to grasp the 3D objects more reliably and expand the possibilities of new media and interfaces.
In addition, since the system using the 3D image synthesizing apparatus 300 of the present invention is simple in configuration, it can be easily used regardless of installation conditions, and therefore can be used particularly effectively in fields such as education and manual presentation. it can.

  In the present embodiment, as a method for generating a subject image in the shape calculation unit 33, a left background image and a right background image prepared in advance are compared with the left image and the right image, and pixels having different colors are extracted. However, instead of the background image, a chroma key process for extracting pixels different from a predetermined color set in advance may be used. This eliminates the need to capture a background image in advance or to make the positional relationship between the left image capturing device 100 and the right image capturing device 200 and the background constant, but it is necessary to set the background to a predetermined color when capturing images. is there.

  In this embodiment, the depth is set according to the distance from the outline of the subject area as a method for calculating the shape data of the subject. However, the depth is calculated based on the parallax between the left subject image and the right subject image. The shape data may be calculated by stereo measurement (JP-A-8-254416, JP-A-11-94527, JP-A-2001-241928, etc.). At this time, the chroma key process described above may be used as a method for generating a subject image in the shape calculation unit 33. Thus, the calculation amount for calculating the shape data is larger than that for setting the depth according to the distance from the outline of the subject area, but shape data close to the actual shape can be calculated.

  In the present embodiment, the shape calculation unit 33 generates a left subject image and a right subject image obtained by extracting a subject from the left image and the right image, and generates shape data based on the generated left subject image and right subject image. Although it is calculated, shape data is calculated by calculating the depth of each pixel by performing stereo measurement processing based on the parallax between the left image and the right image without extracting the subject from the left image and the right image Also good. This increases the amount of calculation, but it is necessary to capture a background image in advance, or to make the positional relationship between the left image capturing device 100 and the right image capturing device 200 and the background constant, and when the image is captured, the background is set to a predetermined color. The shape data close to the actual shape can be calculated.

  In addition, it is assumed that an input device, a display device, and the like (none of which are shown) are connected to the stereoscopic video composition device 300 as peripheral devices. Here, the input device refers to an input device such as a keyboard and a mouse. The display device refers to a CRT (Cathode Ray Tube), a liquid crystal display device, or the like.

  Further, a computer program for realizing the functions of the left video data input unit 31, the right video data input unit 32, the shape calculation unit 33, the shape output unit 34, the stereoscopic video synthesis unit 35, and the stereoscopic video output unit 36 in FIG. A left video data input unit 31, a right video data input unit 32, a shape calculation unit 33, and a shape output are recorded on a readable recording medium, and a program recorded on the recording medium is read and executed by a computer system. The processing of the unit 34, the stereoscopic video composition unit 35, and the stereoscopic video output unit 36 may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within a scope not departing from the gist of the present invention.

  The 3D image synthesizing apparatus of the present invention is suitable for education and manual presentation because it has a simple configuration and does not select installation conditions. However, the present invention is not limited to this.

It is a block diagram which shows the outline of one Embodiment of this invention. It is a schematic block diagram which shows the structure of the system using the three-dimensional video composition apparatus 300 in the embodiment. It is a schematic block diagram which shows the structure of the three-dimensional video composition apparatus 300 in the embodiment. It is a flowchart explaining the production | generation method of the to-be-photographed image in the shape calculation part 33 in the embodiment. 7 is a flowchart illustrating a method for generating shape data of a subject based on a left subject image or a right subject image in a shape calculation unit 33 in the embodiment. It is a figure explaining the contents of state S in the embodiment. It is a figure explaining progress which calculates the depth of subject field A1 in shape calculation part 33 in the embodiment. It is a figure explaining the calculation method of the initial value D0 of step S1 of FIG. 5 in the shape calculation part 33 in the embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Left video imaging device 200 ... Right video imaging device 300 ... stereoscopic image synthesizing device 400 ... stereoscopic video display device 500 ... force / tactile sensation presentation device 31 ... left video data input unit 32 ... right video data input unit 33 ... shape calculation Unit 34 ... shape data output unit 35 ... 3D image composition unit 36 ... 3D image output unit

Claims (6)

In a stereoscopic video composition device that synthesizes a stereoscopic video from a left image viewed from the left eye viewpoint and a right image viewed from the right eye viewpoint,
A shape calculation unit for calculating shape data of a subject from the left image and the right image;
A stereoscopic image synthesizing device comprising: a shape output unit that outputs shape data calculated by the shape calculation unit to a haptic / tactile sensation presentation device.   The shape calculation unit extracts an image of a specific subject from each of the left image and the right image, and for each pixel of the extracted subject image, an image corresponding to a distance from the outline of the subject image The shape data of the subject is generated by giving the coordinates in the vertical direction, the position of the specific subject in the display space of the stereoscopic video synthesized by the device is calculated based on the parallax of the extracted image, and the generated The stereoscopic video composition apparatus according to claim 1, wherein shape data in which shape data is arranged at the calculated position is calculated.   The shape calculation unit extracts an image of a specific subject from each of the left image and the right image, and calculates shape data of the subject by stereo measurement based on the parallax of the extracted image. The stereoscopic video composition apparatus according to 1.   The stereoscopic image synthesis apparatus according to claim 1, wherein the shape calculation unit calculates shape data of a subject by stereo measurement based on a parallax between the left image and the right image. In the shape data generation method in the stereoscopic video composition device that synthesizes the stereoscopic video from the left image viewed from the left eye viewpoint and the right image viewed from the right eye viewpoint,
A first process in which the stereoscopic video composition device calculates shape data of a subject from the left image and the right image;
A shape data generating method comprising: a second step in which the stereoscopic video composition device outputs the shape data calculated in the first step to a haptic / tactile sensation presentation device. In a program for causing a computer to function as a stereoscopic video composition device that synthesizes a stereoscopic video from a left image viewed from the left eye viewpoint and a right image viewed from the right eye viewpoint,
A shape calculation unit for calculating shape data of a subject from the left image and the right image;
A program that causes the shape data calculated by the shape calculation unit to function also as a shape output unit that outputs the shape data to the haptic / tactile sense presentation device.

Images