JP5347987B2 - Video processing device - Google Patents

Description

  The present invention relates to a video processing apparatus, and more particularly to a video processing apparatus that generates a stereo image or a pseudo-stereo image based on images photographed by a plurality of sets of optical systems.

  In recent years, interest in stereoscopic images has increased, and stereo cameras having two sets of optical systems that can acquire stereoscopic images not only for viewing content such as movies but also for general users are on the market. Also, in recent years, compared with the boom of stereoscopic images in the past, interest in the biological effects and safety related to stereoscopic viewing is high, and it is particularly important not to output stereoscopic images that would impose a heavy burden on human vision. There is a need for equipment.

  On the other hand, in order to allow viewing of non-stereoscopic images in pseudo-stereoscopic view, ordinary two-dimensional still images or moving images, that is, depth information is given either explicitly or implicitly as a stereo image. Development of technology to convert a non-stereo 2D image (non-stereoscopic image) into a pseudo 3D image (stereoscopic image; hereinafter also referred to as stereo image) (hereinafter referred to as "2D / 3D conversion") (See, for example, Patent Document 1).

Japanese Patent No. 4214976

  In many cases, the stereo camera has two sets of optical systems in the camera casing at intervals of about the interocular distance. Such a camera imitates the binocular stereopsis of a human (or animal), so it is possible to see the object with a stereoscopic effect because it is possible to see the object with both eyes and with both eyes. The distance is limited so that a suitable parallax can be obtained in the captured scene.

  Although this distance depends on the design and setting of the lens and the optical axis, it is reasonable to estimate the distance to be approximately 1 to 30 meters. If the distance is too short, the images input to the left and right channels of the stereo camera are reasonable, as can be inferred from the fact that humans are looking mainly at one eye instead of binocular stereopsis when looking at the closest object. Rather than an image with parallax, the image is considerably different. As a result, if the distance is too short, it will be impossible for a human to fuse together and make a stereoscopic view as a stereo image, or even if possible, it will give a great deal of fatigue and discomfort to the stereoscopic view. Therefore, it is not preferable in terms of biological effects and safety.

  On the other hand, if the distance is too long, in the case of human beings, the scenes input to the left and right eyes will be the same with no parallax, so that a stereoscopic effect cannot be obtained. However, in the case of a stereo camera, it is difficult to make the optical axes of the left and right channels parallel to each other with high accuracy, and it is difficult to capture the same distant part in the same way when zooming up to near the tele end. Also in this case, the images input to both the left and right channels are considerably different from each other, which is not preferable in terms of biological influence and safety as in the case where the distance is short.

  The present invention has been made in view of the above points. When close-up photography or zooming from the telephoto end is performed, a pseudo-stereo image is generated instead of a stereo image, so that the stereoscopic vision of the viewer is great. It is an object of the present invention to provide a video processing apparatus capable of generating a stereo image that does not give fatigue or discomfort.

  In order to achieve the above object, a video signal processing device according to a first aspect of the present invention is a similarity determination means for determining a similarity between a left eye image and a right eye image constituting a stereo image, and a left eye image and a right eye image. Based on the selected one image, 2D / 3D conversion means for generating a pseudo stereo image is compared with a numerical value indicating similarity and a preset threshold value. When the numerical value indicating similarity is equal to or less than the threshold value, 2D / An image selection unit that selects a pseudo stereo image generated by the 3D conversion unit and selects an input stereo image when a numerical value indicating the degree of similarity is larger than a threshold value.

  In order to achieve the above object, a video signal processing device according to a second aspect of the present invention is a first optical system that captures a subject and outputs a left-eye image, and a second optical system that captures the subject and outputs a right-eye image. Detected by a distance sensor that detects a distance between the optical system and the subject, a 2D / 3D conversion unit that generates a pseudo stereo image based on one of the left-eye image and the right-eye image, and a distance sensor The numerical value indicating the detection result of the distance is compared with a preset threshold value. When the numerical value indicating the detection result of the distance is equal to or less than the threshold value, the pseudo stereo image generated by the 2D / 3D conversion unit is selected, When the numerical value indicating the detection result is larger than the threshold value, an image selection means for selecting a stereo image composed of the input left-eye image and right-eye image is provided.

  Furthermore, in order to achieve the above object, a video signal processing apparatus according to a third aspect of the present invention is a first optical system that captures a subject and outputs a left-eye image, and a second optical system that captures the subject and outputs a right-eye image. Zoom state determination means for performing zoom processing on each of the optical elements and determining the zoom state, and 2D / 3D conversion means for generating a pseudo stereo image based on one of the left-eye image and right-eye image selected And the numerical value indicating the determination result of the zoom state by the zoom state determination unit and a preset threshold value, and when the numerical value indicating the determination result of the zoom state is equal to or less than the threshold value, the 2D / 3D conversion unit generates Select the pseudo stereo image, and select a stereo image composed of the input left eye image and right eye image when the numerical value indicating the zoom state is larger than the threshold value And having an image selection unit that.

  According to the present invention, it is not impossible for a viewer to completely view a stereoscopic image by merging it as a stereo image, and a stereo image that does not give a considerable fatigue or discomfort to the stereoscopic image is always generated. be able to.

[BRIEF DESCRIPTION OF THE DRAWINGS] It is a block diagram of 1st Embodiment of the video processing apparatus of this invention. It is a flowchart for operation | movement description of the apparatus shown in FIG. It is a block diagram of 2nd Embodiment of the video processing apparatus of this invention. It is a flowchart for operation | movement description of the apparatus shown in FIG. It is a block diagram of 3rd Embodiment of the video processing apparatus of this invention. It is a flowchart for operation | movement description of the apparatus shown in FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 shows a block diagram of a first embodiment of a video processing apparatus according to the present invention. As shown in FIG. 1, the video processing apparatus 100 according to the present embodiment includes a left channel (Lch) lens 101, a right channel (Rch) lens 102, an Lch image sensor 103, an Rch image sensor 104, and a similarity. The determination unit 105 includes a channel selector unit 106, a 2D / 3D conversion unit 107, a stereo / pseudo stereo selector unit 108, a stereo image recording unit 109, and a stereo image display unit 110.

  The distance between the optical axes of the lens 101 and the lens 102 is set to be approximately equal to the distance between human eyes. The image sensors 103 and 104 image the same subject through the lenses 101 and 102, respectively. The lens 101 and the image sensor 103 constitute an Lch optical system and output a left-eye video signal. The lens 102 and the image sensor 104 constitute an Rch optical system and output a right-eye video signal. The stereo / pseudo stereo selector unit 108 constitutes an image selection unit.

  Next, the operation of the video processing apparatus 100 of the present embodiment will be described with reference to the flowchart of FIG. The image sensor 103 photoelectrically converts incident light from the subject input through the lens 101, and further performs a known imaging process to generate and output a left-eye video signal (left-eye image). At the same time, the other image sensor 104 photoelectrically converts the incident light from the subject input through the lens 102 and performs a known imaging process to generate and output a right eye video signal (right eye image). The left-eye video signal (left-eye image) and right-eye video signal (right-eye image) generated in this way are respectively input to the similarity determination unit 105, the channel selector unit 106, and the stereo / pseudo stereo selector unit 108 as stereo images. (Step S1).

  The similarity determination unit 105 determines the numerically similarities of the left and right video signals constituting the input stereo image by a known method (step S2). For example, in this similarity determination method, two input images are digitized separately, the digital images are divided into a plurality of cells in each of the horizontal direction and the vertical direction, and the colors in each cell are averaged. First information that has been binarized, and second information that has been binarized and then calculated the density in the cell, the first information extracted for each of the two images, the second information, The error can be obtained by comparing the two, and the smaller the error is, the more similar it is determined to be similar (see Japanese Patent Application Laid-Open No. 11-31153). The similarity determination unit 105 generates a higher degree of similarity as the error value between the left-eye image and the right-eye image constituting the stereo image is smaller (similar) by the above-described known similarity determination method. .

  The stereo / pseudo-stereo selector 108 compares the digitized similarity obtained by the similarity determination unit 105 with a predetermined threshold value, and when the similarity is larger than the threshold, the input stereo Since the left-eye image and the right-eye image constituting the image are similar, it is determined that an appropriate parallax is obtained for the input stereo image, and the input stereo image is selected. On the other hand, when the stereo / pseudo stereo selector 108 has a large error between the left-eye image and the right-eye image constituting the input stereo image and the similarity is low and below the threshold value, the stereo / pseudo stereo selector unit 108 is appropriate for the input stereo image. It is determined that the parallax is not obtained, and the pseudo stereo image generated by the 2D / 3D conversion unit 107 is selected (step S3). This threshold value is determined experimentally.

  When a stereo image from the image sensors 103 and 104 is selected by the stereo / pseudo stereo selector unit 108, this is displayed as it is on the stereo image display unit (viewfinder) 110 and recorded in the stereo image recording unit 109. (Step S4).

  On the other hand, the channel selector unit 106 performs channel setting for determining which of the left and right images constituting the input stereo image is to be used after 2D / 3D conversion (step S5). The channel setting by the channel selector unit 106 may be selectable by the user, or may be determined in one of the designs.

  The 2D / 3D conversion unit 107 receives an image of one channel determined to be used by the channel selector unit 106 (that is, a non-stereo image) as an input image, and converts it into a pseudo stereo image by a known 2D / 3D conversion method. Conversion is performed (step S6). As the 2D / 3D conversion unit 107, for example, a known pseudo-stereoscopic image creation apparatus disclosed in Patent Document 1 by the present inventor can be used. The pseudo-stereoscopic image creation apparatus described in Patent Document 1 is based on the image content of a two-dimensional still image, and it is highly possible that the structure of a certain type of scene is relatively close based on experience. It is an apparatus that creates a pseudo three-dimensional image based on the so-called fail-safe idea, in which even if an erroneous determination is made, it is estimated to the extent that it is “selected”.

  Specifically, the pseudo-stereoscopic image creation apparatus described in Patent Document 1 includes a generation unit that generates a basic depth model indicating a depth value for each of a plurality of basic scene structures, and a non-stereoscopic image to be supplied. Calculating means for calculating a statistic of a pixel value in a predetermined area in the screen to estimate a scene structure; and combining means for combining a plurality of basic depth models with a combining ratio according to a value calculated by the calculating means And depth estimation data creating means for creating depth estimation data from the synthesis result synthesized by the synthesis means and the supplied non-stereo image, and a pseudo stereoscopic image (pseudo-image) from the depth estimation data and the non-stereo image. And a creation means for creating a stereo image).

  The plurality of basic depth models include a first depth model with a spherical concave surface, a second depth model in which the upper portion of the first depth model is replaced with an arched cylindrical surface instead of a spherical surface, There is a third depth model in which the upper part is a flat surface, the lower part is continuous from the flat surface, and the cylindrical surface is directed toward the front as it goes down.

  In the above calculation means, the evaluation value of the high frequency component of the luminance signal of the non-stereo image is calculated for each of the upper 20% area and the lower 20% area of the screen. Further, in the above synthesis means, a scene in which an empty or flat wall exists at the top of the screen when the high frequency component in the region of about 20% of the top of the screen is small, while being based on the first depth model. And the synthesis ratio of the second depth model in which the depth at the top of the screen is deepened is increased with respect to other depth models. On the other hand, if the high-frequency component in the area of about 20% of the lower part of the screen is small, the screen is recognized as a scene in which the flat ground or water surface continuously spreads toward the lower part of the screen, and the screen upper part is approximated as a distant view. For the lower part, the composition ratio of the third depth model, which decreases in depth as it goes down, is increased with respect to the other depth models.

  When the pseudo stereo image generated by the 2D / 3D conversion unit 107 having such a configuration is selected by the stereo / pseudo stereo selector unit 108, the pseudo stereo image is displayed on the stereo image display unit (view finder) 110. At the same time, it is recorded in the stereo image recording unit 109 (step S7).

  As described above, according to the video processing device 100 of the present embodiment, when the similarity between the left-eye image and the right-eye image is low and equal to or less than the threshold, it is determined that an appropriate parallax is not obtained in the input stereo image. Thus, a pseudo stereo image is displayed and recorded. Here, examples of the left-eye image and the right-eye image that are not similar to each other include, for example, a case where parallax in imaging is too strong by performing close-up or zooming near the tele end, or mistakes in shooting or editing. This also includes the case where one channel is missing. Such a stereo image in which the left-eye image and the right-eye image are not similar is inappropriate and has a problem in terms of biological influence and safety.

  Therefore, in the present embodiment, a more natural and safe pseudo stereo image is obtained by performing the video signal processing for generating the above pseudo stereo image only for the problematic frame period mentioned above in the content of the stereo image. It is possible to replace it.

  In FIG. 1, only one arrow is input from the stereo / pseudo-stereo selector 108 to the stereo image recording unit 109 and the stereo image display unit 110. Assume that a pair is entered.

  In addition, although an example in which signal processing is performed on a stereo image input from the imaging elements 103 and 104 has been described with reference to FIG. It is also possible to do.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. FIG. 3 shows a block diagram of a second embodiment of the video processing apparatus according to the present invention. In the figure, the same components as those in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted as appropriate.

  Compared with the video processing apparatus 100 of the first embodiment shown in FIG. 1, the video processing apparatus 200 of the second embodiment shown in FIG. 3 is a distance sensor 201 instead of deleting the similarity determination unit 105. And the stereo / pseudo stereo selector unit 202 is configured to select a stereo image or a pseudo stereo image in accordance with the distance detected by the distance sensor 201.

  The distance sensor 201 is a known infrared distance sensor for autofocus, but a stereo distance sensor using a stereo pair of imaging signals can also be used. The distance sensor 201 includes, for example, a shift in the position of the subject between the left-eye video signal (left-eye image) from the image sensor 103 and the right-eye video signal (right-eye image) from the image sensor 104, and the arrangement of the two left and right optical systems. This is a sensor that detects the distance to the subject triangulated using the above data, and has a known structure (see, for example, Japanese Patent Laid-Open Nos. 8-75456 and 3643157).

  Next, the operation of the video processing apparatus 200 of the second embodiment will be described with reference to the flowchart of FIG. In FIG. 3, the left-eye video signal output from the image sensor 103 out of the lens 101 and the image sensor 103 constituting the Lch optical system and the image sensor 104 out of the lens 102 and image sensor 104 constituting the Rch optical system. The output right-eye video signal is input as a stereo image to the channel selector unit 106 and the stereo / pseudo stereo selector unit 202, respectively (step S11).

  On the other hand, the distance sensor 201 detects the distance from the principal point of the lenses 101 and 102 on which the distance sensor 201 is installed to the subject (step S12). The distance detection method by the distance sensor 201 is known as described above. The distance sensor 201 supplies a numerical value indicating the detection result of the distance to the stereo / pseudo stereo selector unit 202. The position where the distance sensor 201 is installed is not limited to the principal points of the lenses 101 and 102, and may be an appropriate position before and after the lens 101 and 102.

  The stereo / pseudo-stereo selector 202 that constitutes the image selection unit compares a numerical value indicating the detection result of the distance from the distance sensor 201 with a predetermined threshold value (for example, 1 m) and the like. When the numerical value indicating the detection result is larger than the threshold value, it is determined that the distance to the subject is not too short, and the input stereo image is selected. The stereo / pseudo-stereo selector 202 determines that the distance to the subject is too short when the numerical value indicating the distance detection result is equal to or less than the threshold, and determines the appropriate parallax generated by the 2D / 3D converter 107. A pseudo stereo image is selected (step S13). This threshold value is determined experimentally.

  When a stereo image from the image sensors 103 and 104 is selected by the stereo / pseudo stereo selector unit 202, this is displayed as it is on the stereo image display unit (view finder) 110 and recorded in the stereo image recording unit 109. (Step S14).

  On the other hand, the channel selector unit 106 performs channel setting to determine which of the left and right images constituting the input stereo image is to be used after 2D / 3D conversion (step S15). Subsequently, the 2D / 3D conversion unit 107 receives an image of one channel determined to be used by the channel selector unit 106 (that is, a non-stereo image) as an input image, and the known ones such as Patent Document 1 described above are known. The pseudo stereo image is converted by the 2D / 3D conversion method (step S16).

  When the stereo / pseudo stereo selector unit 202 selects the pseudo stereo image generated by the 2D / 3D conversion unit 107, the pseudo stereo image is displayed on the stereo image display unit (view finder) 110 and the stereo image is displayed. It is recorded in the image recording unit 109 (step S17).

  As described above, according to the present embodiment, when the distance to the subject below the threshold is too short by performing close-up or zooming near the telephoto end, it is not a stereo image from the image sensors 103 and 104, but an appropriate amount. Since pseudo-stereo images with parallax are displayed, it is completely impossible to combine them as a stereo image for stereoscopic viewing, or even if possible, it can cause considerable fatigue and discomfort in stereoscopic viewing Therefore, it is possible to improve the impact on the living body and safety.

  In FIG. 3, although one arrow is input from the stereo / pseudo stereo selector unit 202 to the stereo image recording unit 109 and the stereo image display unit 110, a pair of stereo images or a pseudo stereo image is shown respectively. Assume that a pair is entered.

  In the flowchart of FIG. 4, the pseudo stereo image is generated only when the distance is a short distance that is equal to or less than the predetermined threshold (first threshold). However, the distance is longer than the second threshold. In some cases, a pseudo stereo image may be generated. In this case, it is possible to improve the biological effects and safety as compared to displaying a stereo image when the distance is too long.

(Third embodiment)
Next, a third embodiment of the present invention will be described. FIG. 5 shows a block diagram of a third embodiment of the video processing apparatus according to the present invention. In the figure, the same components as those in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted as appropriate.

  Compared with the video processing apparatus 100 of the first embodiment shown in FIG. 1, the video processing apparatus 300 of the third embodiment shown in FIG. 5 is a zoom mechanism 301 instead of deleting the similarity determination unit 105. The zoom state is determined by the zoom state determination unit 302, and a stereo image or a pseudo stereo image is selected according to the determination result. The zoom mechanism 301 has a known structure for zooming a lens system including the lenses 101 and 102. The zoom mechanism 301 and the zoom state determination unit 302 constitute zoom state determination means for determining the zoom state.

  Next, the operation of the video processing apparatus 300 according to the third embodiment will be described with reference to the flowchart of FIG. In FIG. 5, the left-eye video signal output from the image sensor 103 out of the lens 101 and the image sensor 103 constituting the Lch optical system and the image sensor 104 out of the lens 102 and image sensor 104 constituting the Rch optical system. The output right-eye video signal is input as a stereo image to the channel selector unit 106 and the stereo / pseudo stereo selector unit 303, respectively (step S21).

  On the other hand, the zoom state determination unit 302 determines the zoom state of the lens system including the lenses 101 and 102 by the zoom mechanism 301 (step S22). The determination of the zoom state by the zoom state determination unit 302 is performed by, for example, a position sensor that detects the position of the zoom lens when the lenses 101 and 102 include a zoom lens that moves on the optical axis and performs a zoom operation. A known method such as a method of determining the zoom state based on the detection result (for example, see Japanese Patent No. 2900962) can be used. As for the determination result of the zoom state, when zooming is performed in a state where the left and right lenses 101 and 102 are synchronized, it is sufficient to determine only about one of them, for example, a wide end; 0, a tele end; Shall be represented. The result of the zoom state determination is supplied to the stereo / pseudo-stereo selector 303 in FIG.

  The stereo / pseudo-stereo selector 303 constituting the image selection means compares the numerical value indicating the determination result of the zoom state from the zoom state determination unit 302 with a predetermined threshold value, and determines the zoom state. When the numerical value indicating the result is from the wide end of the threshold, the input stereo image is selected. When the determination result of the zoom state is the tele end from the threshold, the pseudo stereo image generated by the 2D / 3D conversion unit 107 is selected. Is selected (step S23). This threshold value is determined experimentally.

  When a stereo image from the image sensors 103 and 104 is selected by the stereo / pseudo-stereo selector 303, it is displayed on the stereo image display unit (viewfinder) 110 as it is and recorded in the stereo image recording unit 109. (Step S24).

  On the other hand, the channel selector unit 106 performs channel setting for determining which of the left and right images constituting the input stereo image is to be used after 2D / 3D conversion (step S25). Subsequently, the 2D / 3D conversion unit 107 receives an image of one channel determined to be used by the channel selector unit 106 (that is, a non-stereo image) as an input image, and the known ones such as Patent Document 1 described above are known. The pseudo stereo image is converted by the 2D / 3D conversion method (step S26).

  When the stereo / pseudo stereo selector unit 303 selects the pseudo stereo image generated by the 2D / 3D conversion unit 107, the pseudo stereo image is displayed on the stereo image display unit (viewfinder) 110 and the stereo image is displayed. It is recorded in the image recording unit 109 (step S27).

  As described above, according to the present embodiment, when zooming up to near the telephoto end, a NO determination result is obtained in step S23, and it is not a stereo image from the image sensors 103 and 104 but has an appropriate parallax. Since pseudo-stereo images are displayed, it is possible to reduce the great fatigue and uncomfortable feeling when stereoscopically merging stereo images, thereby improving the impact on the living body and safety. .

  In FIG. 5, although one arrow is input from the stereo / pseudo stereo selector 303 to the stereo image recording unit 109 and the stereo image display unit 110, a pair of stereo images or a pseudo stereo image is shown respectively. Assume that a pair is entered. In addition, this embodiment can apply both optical zoom and electronic zoom.

  Note that the stereo image or pseudo-stereo image obtained by the video processing apparatuses 100, 200, and 300 according to the above embodiments can be displayed not only by the stereo image display unit 110 but also by a stereo display apparatus. Stereo display device is a projection system using polarized glasses, a projection system or display system combining time-division display and liquid crystal shutter glasses, lenticular stereo display, parallax barrier stereo display, anaglyph stereo display Including head-mounted displays. In particular, a projection system including two projectors corresponding to each image of a stereo image is included.

  Further, in the stereoscopic display system including the above-described stereo display device, a configuration in which audio output is provided is also conceivable. In this case, for video content that does not have audio information such as a still image, a mode in which an environmental sound suitable for video is added can be considered.

100, 200, 300 Video processing device 101 Lch lens 102 Rch lens 103 Lch imaging device 104 Rch imaging device 105 Similarity determination unit 106 Channel selector unit 107 2D / 3D conversion unit 108, 202, 303 Stereo / pseudo stereo selector unit 109 Stereo image recording unit 110 Stereo image display unit 201 Distance sensor 301 Zoom mechanism 302 Zoom state determination unit

Claims (3)

Similarity determination means for determining the similarity between the left-eye image and the right-eye image constituting the stereo image;
2D / 3D conversion means for generating a pseudo stereo image based on one image selected from the left eye image and the right eye image;
The numerical value indicating the similarity is compared with a preset threshold value. When the numerical value indicating the similarity is equal to or less than the threshold value, the pseudo stereo image generated by the 2D / 3D conversion unit is selected, and the similarity is And an image selection means for selecting the input stereo image when a numerical value indicating the value is larger than the threshold value. A first optical system that captures a subject and outputs a left-eye image; a second optical system that captures the subject and outputs a right-eye image; and a distance sensor that detects a distance between the subject,
2D / 3D conversion means for generating a pseudo stereo image based on one image selected from the left eye image and the right eye image;
The numerical value indicating the detection result of the distance detected by the distance sensor is compared with a preset threshold value. When the numerical value indicating the detection result of the distance is equal to or less than the threshold value, the 2D / 3D conversion unit generates the Image selecting means for selecting a pseudo stereo image and selecting a stereo image composed of the input left eye image and right eye image when a numerical value indicating the detection result of the distance is greater than the threshold value. Video processing device. Zoom state determination for performing zoom processing on the first optical system that captures a subject and outputs a left-eye image and the second optical system that captures the subject and outputs a right-eye image, respectively, and determining the zoom state Means,
2D / 3D conversion means for generating a pseudo stereo image based on one image selected from the left eye image and the right eye image;
The numerical value indicating the determination result of the zoom state by the zoom state determination unit is compared with a preset threshold value. When the numerical value indicating the determination result of the zoom state is equal to or less than the threshold value, the 2D / 3D conversion unit generates the value. Image processing means for selecting a pseudo-stereo image and selecting a stereo image composed of the left-eye image and the right-eye image inputted when the numerical value indicating the zoom state is larger than the threshold value apparatus.

Images