JP2012065851A - Multi-view auto-stereoscopic endoscope system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a novel three-dimensional endoscope system which releases surgery staff from stresses specific to three-dimensional endoscope systems and supports achievement of a high-quality operation.SOLUTION: A stereoscopic camera for the three-dimensional endoscope acquires a stereoscopic image. Three-dimensional data associating a depth value of an imaging object with RGB values of the imaging object are formed on the basis of the stereoscopic image. A multi-view stereoscopic image is formed by integrating a plurality of stereoscopic images corresponding to a plurality of virtual viewpoints drawn on the basis of the three-dimensional data. By displaying the multi-view stereoscopic image on a multi-view auto-stereoscopic display device, multi-view auto-stereoscopic live image of an affected area is achieved.

Description

  The present invention relates to a stereoscopic endoscope system, and more particularly to a multi-view autostereoscopic endoscope system.

  In recent years, endoscopic surgery is rapidly spreading in the surgical field. Endoscopic surgery requires only a small incision to insert a surgical instrument such as forceps and an endoscopic scope, so that the surgical wound can be significantly reduced compared to conventional open surgery, The physical burden can be reduced. In endoscopic surgery that is commonly performed at present, an image of the affected area is displayed on a large monitor installed on the operating table, and is composed of a surgeon and assistant, anesthesiologists and nurses. The staff operates while looking at the monitor. However, it is difficult to accurately sense the “depth” of the affected area from the 2D image displayed on the monitor. From the viewpoint of spatial recognition, it is inferior to laparotomy where the affected area can be directly observed. It was.

  In this regard, various stereoscopic endoscope systems to which a 3D image system is applied have been recently studied. In the stereoscopic endoscope system, the operation staff can perform an operation while viewing a 3D image of the affected area. Japanese Patent Laying-Open No. 2007-041453 (Patent Document 1) discloses such a stereoscopic endoscope apparatus.

  Here, the principle of the eyeglass-type stereoscopic endoscope system that is most frequently used at present will be described. FIG. 20 shows a spectacle-type stereoscopic endoscope system 200. The glasses-type stereoscopic endoscope system 200 includes a stereo camera 201 at the distal end of an endoscope scope, and two arranged CCD cameras (a left-eye camera 202 and a right-eye camera 204) are used for an image for a left eye and a right eye of an object. Each image is captured and transmitted to the image output control unit 206. In the image output control unit 206, the image switching unit 208 alternately outputs the left-eye image and the right-eye image in accordance with the frame sequential method, and the display 210 displays the left-eye image and the right-eye image alternately. On the other hand, the surgical staff sees an image displayed on the display 210 with the dedicated 3D glasses 212 attached. Here, since the 3D glasses 212 include a liquid crystal shutter that alternately blocks the left and right fields of view in synchronization with the switching between the left-eye image and the right-eye image, the left-eye image appears only to the left eye, and the right-eye image Visible only to the right eye. As a result, the surgical staff combines the left and right images in the brain to obtain a three-dimensional image of the object.

  However, physical and mental stress due to dedicated 3D glasses has been reported for surgical staff using the above-described glasses-type stereoscopic endoscope system. Examples include eyestrain associated with the flicker phenomenon peculiar to active shutters and mental stress associated with wearing heavy special glasses for a long time. Furthermore, there is a problem due to the fact that a single viewpoint stereo image is projected in the glasses-type stereoscopic endoscope system. That is, when a plurality of surgical staff members perform an operation while viewing the display 210, there is no problem in the surgical staff members who are viewing the display 210 from the front because the actual viewpoint matches the viewpoint of the image. In the surgical staff seen from the above, the actual depth of the affected area and the sense of depth perceived from the images do not match, which causes virtual reality sickness (VR sickness). In surgery, since careful attention and high concentration are required for a long time, the cause of stress must be eliminated from the surgical environment as much as possible.

  With regard to the above points, in recent years, multi-view autostereoscopic display devices have been put into practical use. Japanese Patent Laying-Open No. 2005-86414 (Patent Document 2) discloses such a multi-viewpoint autostereoscopic display device. FIG. 21 is a conceptual diagram for explaining the mechanism of the multi-viewpoint autostereoscopic display device. The multi-view autostereoscopic display device 300 is an application of stereo photography technology such as a multi-view method and an integral photography method, which has been known for a long time. Is realized at the same time.

  In the multi-view autostereoscopic display device 300, a plurality of pixels (seven pixels in the example shown in FIG. 21) are stored as one unit in one lenticular lens, as shown by being surrounded by a broken line on the right side of FIG. ing. Each pixel is composed of three sub-pixels (RGB) (not shown) arranged in the vertical direction (perpendicular to the paper surface). The light of each pixel is refracted by a lenticular lens and emitted radially in different directions. By combining this multiplexed pixel and lenticular lens, a plurality of stereo images associated with a plurality of different viewpoints can be aggregated and simultaneously displayed on the display.

  Note that a multi-view stereo image to be displayed on the multi-view autostereoscopic display apparatus 300 can be created by three-dimensional computer graphics (3DCG). For example, when seven sets of stereo images corresponding to seven viewpoints (a group of left-eye images and right-eye images) are created by 3DCG, an object in which shape data is defined in advance is arranged in a virtual three-dimensional space. A set of virtual camera viewpoints is set, and a two-dimensional image of the object when perspectively projected on a virtual screen with each camera viewpoint as the origin is calculated and generated (rendering). Since a stereo image corresponding to one viewpoint includes an image corresponding to the left eye viewpoint and an image corresponding to the right eye viewpoint, it is necessary to calculate a 7 × 2 = 14 two-dimensional image for convenience.

  Each pixel data constituting the 14 two-dimensional image generated by the above-described procedure is formed so that the left-eye image and the right-eye image have appropriate parallaxes, and each image is formed at a predefined viewpoint. The appropriate pixel coordinates are calculated and assigned. As a result, a stereo image associated with each viewpoint is formed by being distributed to the left and right eyes of the user located at the viewpoint, and is also formed in accordance with the movement of the viewpoint of the user. The image transitions. Multi-viewpoint stereoscopic vision with the naked eye is realized by the mechanism described above.

JP 2007-041453 A JP 2005-84414 A

  The present invention has been made in view of the above-described problems in the prior art. The present invention is a novel technique that can release surgical staff from stress peculiar to a stereoscopic endoscope system and can support high-quality treatment. An object of the present invention is to provide a simple stereoscopic endoscope system.

  As a result of earnestly examining the stereoscopic endoscope system that can release the surgical staff from the stress peculiar to the stereoscopic endoscope system and support high-quality treatment, the inventor determined that the depth value of the imaging target and the RGB Based on the 3D data associated with the values, a plurality of stereo images corresponding to a plurality of virtual viewpoints are generated in real time and displayed on an existing multi-view autostereoscopic display device. The inventors have found that multi-view autostereoscopic viewing is possible, and have reached the present invention.

  That is, according to the present invention, a stereo camera, a multi-view stereo image generating unit that generates a multi-view stereo image based on a stereo image acquired by the stereo camera, and a multi-view stereo image for displaying the multi-view stereo image are displayed. A stereoscopic endoscope system including a viewpoint autostereoscopic display, wherein a multi-viewpoint stereo image generation unit generates 3D data in which a depth value and an RGB value for each pixel of a stereo image acquired by the stereo camera are associated with each other A stereoscopic endoscope system including a 3D data generating unit that renders and a rendering unit that renders a multi-view stereo image formed by aggregating a plurality of stereo images corresponding to a plurality of virtual viewpoints based on the 3D data. Is done.

  Furthermore, according to the present invention, there is provided a method of performing stereo matching based on dynamic programming in order to obtain a parallax for each pixel of a stereo image, wherein a plurality of pixels that include a pixel of interest and are continuous in a vertical direction are obtained. A method is provided which is defined as a luminance value acquisition region, and an average value or a power root of the luminance distance calculated for each of the plurality of pixels is defined as a cost in the stereo matching. In the present invention, the luminance value acquisition region is dynamically defined so as not to extend across an edge, and a weighting factor for oblique path selection in the stereo matching is increased according to the average value of the luminance distance. It is preferable to set so as to be. Further, in the present invention, the target scan line in the stereo image is defined at predetermined intervals, and the parallax of each pixel on the scan line is acquired by the stereo matching, and the parallax existing in the vertical direction has been acquired. If the parallax between the two pixels does not change with respect to the pixel located between the two pixels, the pixel located between the two pixels inherits the parallax between the two pixels, and the two pixels When the parallax changes, it is preferable to execute a series of processes such as newly performing the stereo matching only for the pixel located between the two pixels to obtain the parallax, It is preferable to repeat recursively until the parallax of all the pixels of the stereo image is acquired.

  As described above, according to the present invention, a novel stereoscopic endoscope system capable of releasing surgical staff from stress peculiar to a stereoscopic endoscope system and supporting high-quality treatment is provided. According to the present invention, in an open environment, it is possible to grasp a sense of depth from a continuous multi-viewpoint with respect to an object without a large number of people contacting a special observation tool all at once.

The functional block diagram of the multiview autostereoscopic endoscope system of this embodiment. The conceptual diagram for demonstrating the basic principle of a stereo vision measurement. The conceptual diagram for demonstrating the basic principle of a stereo vision measurement. The conceptual diagram for demonstrating the basic principle of a stereo vision measurement. The conceptual diagram of the conventional DP matching which uses the luminance distance of a single pixel as a cost. The conceptual diagram for demonstrating the luminance value acquisition area | region in this embodiment. The conceptual diagram for demonstrating the luminance value acquisition area | region in this embodiment. The conceptual diagram for demonstrating the luminance value acquisition area | region in this embodiment. The flowchart which shows the process which determines the luminance value acquisition area in this embodiment. The conceptual diagram for demonstrating the process which reduces the influence of a big color difference. The conceptual diagram for demonstrating the parallax inheritance process in this embodiment. The flowchart which shows the parallax inheritance process in this embodiment. The figure which shows four sets of stereo images used for the parallax measurement experiment. The figure which shows the result obtained about stereo image "Cones". The figure which shows the result obtained about stereo image "Teddy". The figure which shows the result obtained about stereo image "Tsukuba". The figure which shows the result obtained about stereo image "Venus". The figure which shows the relationship between parameter (alpha) and the average value of MDE / S. The figure which shows the relationship between parameter (alpha) and the average value of correct answer rate [%]. The figure which shows the conventional spectacles type | mold stereoscopic endoscope system. The conceptual diagram for demonstrating the mechanism of a multiview autostereoscopic display apparatus.

  Hereinafter, the present invention will be described with reference to embodiments shown in the drawings, but the present invention is not limited to the embodiments shown in the drawings. In the drawings referred to below, the same reference numerals are used for common elements, and the description thereof is omitted as appropriate.

  FIG. 1 shows a functional block diagram of a multi-view autostereoscopic system 100 according to an embodiment of the present invention. The multi-view autostereoscopic endoscope system 100 according to the present embodiment includes a stereo camera 201, a multi-view stereo image generating unit 102, and a multi-view autostereoscopic display 300.

  The stereo camera 201 can be referred to as an existing stereoscopic endoscope camera. The left-eye image and the right-eye image of the object captured by the two CCD cameras (the left-eye camera 202 and the right-eye camera 204) are transmitted to the 3D data generation unit 104 of the multi-viewpoint stereo image generation unit 102. The 3D data generation unit 104 generates 3D data based on the received left-eye image and right-eye image. Note that the 3D data in the present invention is information defined for each pixel and refers to information in which a depth value is associated with an RGB value.

  The rendering unit 106 renders a plurality of stereo images corresponding to a plurality of virtual viewpoints based on the 3D data generated by the 3D data generation unit 104. Specifically, the rendering unit 106 obtains the three-dimensional coordinate value of the object (affected part) in the coordinate system in the real space by calculation based on the depth value included in the 3D data. Furthermore, a plurality of stereo images corresponding to a plurality of virtual viewpoints are rendered based on the acquired three-dimensional coordinate values, and appropriate pixel coordinates are assigned to the rendered images and aggregated. As a result, one multi-view stereo image in which a plurality of stereo images corresponding to a plurality of virtual viewpoints are aggregated is rendered. The multi-view stereo image generated by the rendering unit 106 is transmitted to the multi-view autostereoscopic display 300 and displayed.

  According to the multi-viewpoint autostereoscopic system 100 of the present embodiment, the surgical staff does not need to wear dedicated glasses, and can perform treatment in an open environment. Also, in this system, the actual depth of the subject and the sense of depth perceived from the image coincide with each other when viewing the display 300 from any direction. Never become. In addition, since motion parallax can be obtained by changing the viewpoint little by little from left to right and up and down, it is possible to perceive a more natural sense of depth with respect to the object (affected part) than when observing a single viewpoint stereoscopic image. It becomes possible. This will contribute to the improvement of the accuracy of the treatment.

  As described above, in a conventional spectacle-type stereoscopic endoscope system, a single viewpoint stereo image (left-eye image and right-eye image) acquired for an object (affected part) is alternately displayed as it is. In the multi-view autostereoscopic system 100 of the present embodiment, the single-view stereo acquired for the object (affected part) is realized. Once 3D data is generated from an image by calculation, a plurality of stereo images corresponding to a plurality of virtual viewpoints are reconstructed based on the 3D data, and one piece of the plurality of stereo images is aggregated. By displaying as a multi-viewpoint stereo image, multi-viewpoint autostereoscopic viewing of a live-action image of the affected area is realized.

  Here, in order to put this system into practical use, first of all, the depth value in 3D data, which is the basis for reconstructing a plurality of stereo images corresponding to a plurality of virtual viewpoints, is calculated with high accuracy. There is a need to. This is because an erroneous depth value cannot reproduce a stereo image that reflects the correct sense of depth, resulting in a decrease in observation accuracy of the affected area. Secondly, since it is necessary to display a live-action moving image of the affected part in real time, it is necessary to ensure the real-time property of the depth value calculation processing speed. In the present invention, means for clearing the above two problems is implemented in the depth value calculation unit 108 of the 3D data generation unit 104. Hereinafter, the configuration of the depth value calculation unit 108, which is a characteristic configuration of the multi-viewpoint autostereoscopic system 100 according to the present embodiment, will be described in detail step by step.

  Prior to the description of the depth value calculation unit 108 in the present embodiment, the basic principle of stereo vision measurement used when acquiring a depth value from a stereo image will be described. In the following description, the aforementioned “left-eye camera” and “right-eye camera” are referred to as “main camera” and “sub-camera”. FIG. 2 is a conceptual diagram for explaining the basic principle of stereo vision measurement. In FIG. 2, the two-dimensional coordinates (i, h) on the main camera image and the sub-camera image are compared with the three-dimensional coordinates (x, y, z) in the coordinate system in the real space (world coordinate system). Two-dimensional coordinates (j, h) are defined. After the camera is modeled as a pinhole camera, the focal point of the main camera is P, the focal point of the secondary camera is P ', and the distance between the focal points of both cameras (baseline) The length (long) is shown as b, and the distance (focal length) from the focus to the screen (each camera image) is shown as f. Note that the focal length f is obtained by prior camera calibration so as to have an appropriate value when the camera is modeled as a pinhole camera.

  Here, as shown in FIG. 2A, the projection point on the main camera image of an arbitrary point A on the object is M, the projection point on the sub camera image of point A is S, and the focus of the sub camera. When P ′ and the sub camera image are translated in the x-axis direction and completely overlapped with the focal point P of the main camera and the main camera image, the positional relationship shown in FIG. 2B is obtained. Here, the parallax in stereo vision measurement can be obtained as a separation distance (i−j) between the projection point M (i, h) on the main camera image and the projection point S (j, h) on the sub camera image. Here, since the triangle PAA ′ and the triangle PSM shown in FIG. 2B are similar to each other, the following expression (1) is established.

  When the above formula (1) is modified, the following formula (2) is obtained.

  According to the above formula (2), it is understood that the distance Z (that is, the depth value) to an arbitrary point A on the object can be derived by obtaining the parallax (ij), and as a result, the depth value is calculated. It will be understood that the speed of the process depends on the speed of the parallax (ij) calculation process. Here, when the acquired image of the camera is an RGB digital image, the projection point M and the projection point S can be replaced with the pixel M and the pixel S of the RGB digital image. In particular, it can be said that this is equivalent to the process of obtaining the coordinate values of the pixel M and the pixel S.

  FIG. 3A shows a main camera image and a sub camera image when an object is imaged. In the stereo vision measuring device, the main camera and the sub camera are arranged in parallel equiposition, so that the pixel on the sub camera image (hereinafter referred to as the sub pixel) corresponding to the pixel on the main camera image (hereinafter referred to as the main pixel). ) Should be on a scan line at the same height. Therefore, obtaining the correct combination of the pixel M on the main camera image and the pixel S on the sub camera image corresponding to an arbitrary point on the object is the same height as shown in FIG. It will be understood that this results in a matching problem (so-called stereo matching) of two groups of pixels on the scan line (ie h coordinate values are equal).

  FIG. 4A shows the result of stereo matching between the main camera scanning line and the sub camera scanning line shown in FIG. In FIG. 4A, the matching result of the main pixel and the sub-pixel is indicated by an arrow line.

  FIG. 4B shows a matrix for solving a matching problem between a pixel group on the main camera scanning line and a pixel group on the sub camera scanning line. The vertical axis and horizontal axis of the matrix correspond to the j coordinate axis of the sub camera image and the i coordinate axis of the main camera image, respectively, and each intersection of the matrix corresponds to a set of main pixels and sub pixels.

  Each intersection of the matrix is given an absolute value of a difference in luminance value between the main pixel and the sub-pixel corresponding to the intersection (hereinafter referred to as luminance distance) as a cost for matching. Here, a correct pixel combination (that is, the pixel M and the pixel S) in which the same target point A is reflected can be considered as a pixel combination having the smallest difference in luminance values. Therefore, in the above-described matching problem, in the matrix shown in FIG. 4B, among the paths from the lower left start point to the upper right end point, the sum of the costs (luminance distance) given to each intersection point on the path is the highest. It can be replaced by a search problem of a smaller optimal path. This search problem can be handled by stereo matching based on dynamic programming (hereinafter referred to as DP matching) for the following reasons.

  That is, as shown in FIG. 4A, if the pixel M and the pixel S are correctly combined, the pixel existing on the left side of the pixel M in the other correct combination is on the left side of the pixel S. Should be combined with the pixels present in. Considering this in the light of the matrix shown in FIG. 4B, the correct combination (intersection) immediately before the combination of the pixel M and the pixel S is always left, below, It means that it exists in one of the three diagonally lower left. In other words, as a column of the optimum combination up to the correct combination of the pixel M and the pixel S, as viewed from the intersection of the pixel M and the pixel S, a column of three combinations reaching the three intersections of left, lower, and diagonally lower left Choose the best one from the list.

  In DP matching using the matrix shown in FIG. 4B, the total cost D (i, j) of the optimum path reaching the matrix intersection (i, j) can be expressed by the following equation (3). In the following formula (3), d (i, j) represents the luminance distance (cost) given to the intersection (i, j) of the matrix, and D (i-1, j), D (i− 1, j-1) and D (i, j-1) are the intersection (i-1, j), intersection (i-1, j-1), intersection (i, j-1) just before the intersection (i, j) j-1) represents the total cost of each optimum path, and λ represents a weighting factor for selecting an oblique route.

  In DP matching, as shown in FIG. 4 (c), D (i-1, j), D (i-1, j-1), D (i, j-1) obtained immediately before, The process of substituting d (i, j) and the weighting factor λ into the above equation (3) and selecting the one that gives the minimum value min as the optimum path reaches from the start point to the end point of the matrix shown in FIG. By repeating the above, it is possible to determine the optimum path that minimizes the sum of the costs (luminance distance) given to each intersection on the path among the paths from the start point to the end point.

  The principle of stereo vision measurement using DP matching has been described above. Next, a characteristic configuration included in the depth value calculation unit 108 in the multi-viewpoint autostereoscopic system 100 of the present embodiment will be described. The feature of the depth value calculation unit 108 in the present embodiment is that a new DP matching method obtained by improving the DP matching method in the conventional stereo vision measurement described above is employed. That is, in the new DP matching method in the present embodiment, first, a cost for DP matching is newly defined in order to improve the accuracy rate. Second, in order to improve the DP matching accuracy rate, a configuration is adopted in which the weighting factor λ for diagonal path selection is adaptively determined. Furthermore, the depth value calculation unit 108 in the present embodiment thirdly employs a configuration for omitting the matching process in order to speed up the process. Hereinafter, the first to third contents described above will be described in detail in order.

(About introduction of new cost definition in DP matching)
In DP matching in the conventional stereo vision measurement, as shown in FIG. 5A, the luminance value m (i, h) of the main pixel (i, h) of interest and the sub-pixel (j, The absolute value of the difference between the luminance values s (j, h) of h) is defined as the luminance distance d (i, j), and this luminance distance d (i, j) is defined as the cost in DP matching. However, in such a method that uses the luminance distance for a single pixel as a cost, it is inevitably affected by noise, and this reduces the matching accuracy rate. Furthermore, in the method that uses the luminance distance for a single pixel as a cost, ambiguity cannot be eliminated in a region where pixels of the same color (that is, the same luminance value) continue on the scanning line, and the matching accuracy rate is improved. There was a limit. This point will be described with reference to FIG.

  FIG. 5B shows matching based on the luminance distance of a single pixel. Here, when the luminance values of the pixel (b) and the pixel (c) on the sub camera image are equal, the luminance distance between the target pixel (b) of the main camera image and the pixel (b) on the sub camera image is the main camera. The ambiguity remains because it is equal to the luminance distance between the target pixel (A) of the image and the pixel (C) on the sub camera image. With respect to the problems of the conventional method, the present invention solves this by defining the cost using the luminance values of a plurality of pixels including the pixel of interest. Hereinafter, this point will be described with reference to FIG.

In the present invention, a plurality of pixels including the target pixel and continuing in the vertical direction (h-axis direction) are defined as the luminance value acquisition region. Specifically, as indicated by a thick line on the left side of FIG. 6A, the positive direction ε of the h coordinate axis perpendicular to the scanning line with the main pixel (i, h) of interest as the reference (0) A pixel group having a _U- th pixel as an upper end pixel and a negative direction ε _L- th pixel (ε _L is a negative value) in the h coordinate axis is specified as a lower end pixel, and these are defined as a luminance value acquisition region G _m To do. Similarly, with the sub-pixel (j, h) to be matched as a reference (0), the ε _U- th pixel in the positive direction of the h coordinate axis is the top pixel, and the ε _L- th pixel is in the negative direction of the h coordinate axis identify the group of pixels as a pixel, it defines them as luminance value acquisition region G _s. That is, the luminance value acquisition region G _s is defined to have the same size as the luminance value acquisition region G _m .

In the present invention, the average value of the luminance range calculated for each pixel included in the luminance value acquisition region G _m and area average luminance distance d _AVE., To define this area average luminance distance d _AVE. As the cost in DP matching . The area average luminance distance d _{AVE. Is} shown in the following formula (4). In the following formula (4), ε represents the added value of the h coordinate, N represents the number of pixels constituting the luminance value acquisition region G (ε _U −ε _L + 1), and m represents the luminance value of the main pixel. S indicates the luminance value of the sub-pixel.

The present invention adopts the area average luminance distance d _{AVE. As the} cost in DP matching has the following advantages. First, the present invention employs a configuration in which an average value of luminance distances calculated for a plurality of pixels including the target pixel is given as a value with respect to the target pixel. Even if noise due to differences or differences in sensitivity of CCD elements is included in the pixel of interest, the influence of noise is leveled by averaging the luminance distance, and as a result, the matching accuracy rate is reduced due to noise. Is suppressed. Second, since the values of ε _U and ε _L that define the luminance value acquisition region G when determining the region average luminance distance d _AVE are dynamically determined according to the shape of the object, such a region average The matching using the luminance distance d _AVE substantially uses spatial information in the vicinity of the target pixel, and as a result, the matching accuracy rate is significantly improved as compared with the case where a single pixel is used. Hereinafter, this point will be described with reference to FIG.

FIG. 6B shows matching using the area average luminance distance d _AVE as a cost. In the example shown in FIG. 6B, the luminance value acquisition region G _s including the sub-pixel (b) includes a pixel having a color different from that of the sub-pixel (b) on the sub-pixel (b). The area average luminance distance d _AVE between the _target pixel (A) and the sub-pixel (B) is larger than the area average luminance distance d _AVE between the _target pixel (A) and the sub-pixel (C). As a result, it becomes possible to select the pixel (c) of the sub camera image as the correct answer. Incidentally, FIG. 6 (b), for ease of understanding, the color brightness value acquisition region G _s showed significantly different example, if the color of the pixel of the luminance value acquisition region G was the same color However, since a certain degree of dispersion can be expected in the luminance values of the pixels in the region, the above-described effect can be obtained.

  Next, in the present invention, a description will be given of the fact that the matching accuracy rate can be further improved by adopting a configuration in which the optimum luminance value acquisition region G is dynamically set.

FIG. 7A shows a main camera image and a sub camera image when two objects are imaged. In the example shown in FIG. 7 (a), the magnitude of the luminance value acquisition region G _m is assumed to be fixed. Here, when the main pixels fancy-through is the object of the quadrangular (a) the target pixel, the pixel (i) of the luminance value acquisition region G _m, while the object behind the triangle bleeds through, the sub-pixels in the luminance value acquisition region G _s corresponding to the main pixel (i) (c) is not incorporated-through is the object behind the triangle, it happens such background visible on captured. In such a case, the area average luminance distance d _{AVE of} the main pixel (A) and the sub-pixel (A ′) becomes large, and as a result, there is a possibility that the combination of both is not determined as the correct answer. That is, depending on the settings how the magnitude of the luminance value acquisition region G _m, will be noise pickup returned, it could lead to thereby reduce the matching accuracy rate. In this regard, as conceptually shown in FIG. 7B, the present invention obtains a luminance value that does not extend across the edge of the object reflected in the main camera image as a reference. by dynamically setting the region G _m, it is possible to prevent noise pickup improve matching accuracy rate. Hereinafter, dynamic setting of the luminance value acquisition region G will be described in detail.

  In the present invention, setting the luminance value acquisition region G is synonymous with determining the upper end pixel and the lower end pixel of the luminance value acquisition region G. Hereinafter, a case where the upper end pixel of the luminance value acquisition region G is determined will be described as an example. In the present invention, the pixel located on the edge region in the upper direction of the h coordinate axis when viewed from the target pixel (i, h) is determined as the upper end pixel. When the pixel existing on the h coordinate axis and the edge region are as shown in FIG. 8A when viewed from the pixel of interest (i, h), the pixel on the edge region is the pixel of interest. The highest pixel (i, h + 3) whose color does not change greatly as viewed from (i, h) is determined as the upper end pixel. In addition, when the pixel existing on the h coordinate axis as viewed from the target pixel (i, h) and the edge region are in the form as shown in FIG. 8B, the edge has priority over the color change, The uppermost pixel (i, h + 4) in the edge region is determined as the upper end pixel. As shown in FIG. 8C, when there is no edge region in the vicinity of the target pixel (i, h), the upper end pixel has a predetermined maximum number of pixels E (maximum addition value E of h coordinates). To decide. In the example shown in FIG. 8C, since E = 5, the pixel (i, h + 5) is determined as the upper end pixel.

  The process of determining the upper end pixel of the luminance value acquisition region G will be described based on the flowchart shown in FIG. First, the coordinate value (i, h) of the main pixel of interest is set (step 101), and the addition value ε of the h coordinate is set to “1” (step 102). Next, it is determined whether or not the pixel (i, h + ε) is on the edge region (step 103). Here, the determination of whether or not the pixel (i, h + ε) is on the edge region can be executed by the following procedure.

  First, with the pixel (i, h + ε) as a reference, the luminance change amount η in the i-axis direction and the luminance change amount ζ in the h-axis direction shown in the following equation (5) are obtained using a Sobel filter. In the following formula (5), m represents the luminance value of the main pixel.

Next, a value obtained by substituting η and ζ obtained for the pixel (i, h + ε) into the following equation (6) is a luminance change amount k in the pixel (i, h + ε), and the luminance change amount k is When the predetermined threshold value _Ke is exceeded, it is determined that the pixel (i, h + ε) is on the edge region.

When it is determined that the pixel (i, h + ε) is on the edge region (step 103, Yes), after proceeding to step 104 and setting the edge flag _fe , the pixel (i, h) of the pixel of interest It is determined whether or not the absolute value (luminance distance) of the difference between the luminance value m (i, h) and the luminance value of the pixel (i, h + ε) exceeds a predetermined threshold value K _C (step 105). If the luminance distance does not exceed the threshold value K _C (No at Step 105), the process proceeds to Step 106, where the addition value ε of the h coordinate is incremented by “1”, and then the addition value ε is the maximum of the predetermined h coordinate. It is determined whether or not the added value E (maximum number of pixels E) has been exceeded (step 107). When the addition value ε exceeds the maximum addition value E (step 107, Yes), the pixel (i, h + E) is defined as the upper end pixel, and the process ends (step 108).

  When the addition value ε does not exceed the maximum addition value E (No at Step 107), the process returns to Step 103, and whether or not the pixel (i, h + ε) that is one pixel above the previously determined pixel is on the edge region. Determine whether. Thereafter, each time the process of Step 104 → Step 105 → Step 106 → Step 107 → Step 103 is repeated, the h coordinate of the pixel to be determined is incremented by one.

Meanwhile, if the luminance distance exceeds the threshold value K _C in step 105 (step 105, Yes), the pixel (i, h + ε-1) immediately before the current pixel (i, h + ε). Is defined as the top pixel (step 109), and the process is terminated. On the other hand, when the pixel (i, h + ε) is not on the edge region (No in Step 103), it is determined whether or not an edge flag is set (Step 110). If the edge flag is set (step 110, Yes), it is determined that the edge region has been exceeded, and the pixel (i, h + ε-1) immediately before the current pixel (i, h + ε) is set as the upper end pixel. And the process ends (step 109). When the edge flag is not raised (No at Step 110), the process proceeds to Step 106, where the added value ε of the h coordinate is incremented by “1” as described above, and the subsequent processing is repeated.

In the present invention, after determining the lower end pixel of the luminance value acquisition region G in the same procedure as described above, a plurality of pixel groups from the upper end pixel to the lower end pixel are dynamically set as the luminance value acquisition region G. . Having thus described the cost of the DP matching in the present invention, in the present invention, instead of the area average luminance distance d _AVE. In configuration to matching cost, matching power root of area average luminance distance d _AVE. Cost By adopting the configuration, the matching accuracy rate can be further improved. Hereinafter, this point will be described. In the following description, the part referring to a single pixel should be referred to as referring to the luminance value acquisition region G.

  FIG. 10A shows a main camera image and a sub camera image when three objects having different colors are imaged. In the example shown in FIG. 10A, the main camera image includes three objects, red, green, and yellow. In the sub camera image, the red object is a green object. It is hidden behind. In this case, in the matching using the main pixel (A) in which the red object is reflected as the target pixel, there is no correct answer on the scanning line of the sub camera image, and an incorrect combination must be determined. In other words, DP matching in stereo vision measurement cannot completely eliminate inappropriate matching in principle.

  In such a case, it is desirable that all the sub-pixels on the scanning line of the sub-camera image are equivalent from the viewpoint of matching cost, but this is not the case. This is because the luminance value of the pixel of the RGB digital image is three-dimensional vector information consisting of the R value, G value, and B value, and in particular, in the case of a combination of colors having a complementary color relationship, the sum of the differences between the values. This is because the luminance distance is larger.

  For example, the luminance value of the main pixel (A) on the red object is [R = 255, G = 0, B = 0], and the luminance value of the sub-pixel (A) on the green object is [R = 0, G = 255, B = 0], and the luminance value of the sub-pixel (c) on the yellow object is [R = 255, G = 255, B = 0]. As a result, the cost (luminance distance = 510) given to the set of the main pixel (a) and the sub-pixel (b) is given to the cost (luminance distance = 510) given to the set of the main pixel (a) and the sub-pixel (c). 255). A large difference in cost caused by such a large color difference may adversely affect the search for the optimum path.

  Here, for convenience of explanation, it is assumed that a pixel group consisting of 10 pixels is matched. In the first pass, out of 10 pixel groups, 7 pixel groups are correct (cost = 1), the remaining 3 pixel groups are incorrect combinations, and the cost of the incorrect combination is the same. Are each “10”. On the other hand, in the second pass, out of 10 pixel groups, 9 pixel groups are correct (cost = 1), and the remaining 1 pixel group is an incorrect combination. It is assumed that the cost is “50” because there is a large color difference in the combination. In this example, the total cost of the first path is “59”, and the total cost of the second path is “37”. As a result, not the second path with a high correct answer rate but the first path with a low correct answer rate may be determined as the optimum matching.

Regarding the above-described problems, in the present invention, the power root of the area average luminance distance d _AVE. Described above is defined as the cost in DP matching. The cost c (i, j) is shown in the following formula (7). In the following formula (7), the root r is a fixed value, and an appropriate value is set. For example, in the present invention, the root r can be 512 to 1024.

FIG. 10B is a diagram showing the relationship between the area average luminance distance d _AVE. And the cost c (i, j). As shown in FIG. 10B, when the area average luminance distance d _AVE. Is defined as the cost c (i, j) as it is, the cost c is directly proportional to the area average luminance distance d _AVE. (i, j) but increases, area average luminance distance d _AVE. cost c (i, j) the power roots when defined as, in the area average luminance distance d _AVE. value is less range, area average luminance distance d _AVE. cost c (i, j) in accordance with an increase in it is increased in the area average luminance distance d _AVE. value is large range, the rate of increase in the cost c (i, j) Decrease gradually. That is, by defining the power root of the region average luminance distance d _{AVE. As} the cost in DP matching, it is possible to appropriately estimate the total cost of the path including inappropriate matching, and as a result, the matching accuracy rate is calculated. Further improvement can be achieved. As described above, the details of the cost in DP matching employed by the depth value calculation unit 108 in the present embodiment have been described. Subsequently, the weighting factor λ for oblique path selection, which is the second feature of the depth value calculation unit 108, has been described. A configuration that is adaptively determined will be described.

(Regarding the configuration for adaptively determining the weighting factor λ for oblique route selection)
In the DP matching according to the present invention, the total cost C (i, j) of the optimum path to the intersection (i, j) of the matrix shown in FIG. 4B can be expressed by the following equation (8). In the following equation (8), c (i, j) is the cost given to the intersection (i, j) of the matrix (that is, the power of the area average luminance distance d _AVE. Shown in the above equation (7)) _. C (i-1, j), C (i-1, j-1), and C (i, j-1) are the intersection points (i-1, j) immediately before the intersection point (i, j). j), the total cost of each optimum path to the intersection (i-1, j-1) and the intersection (i, j-1), and λ represents a weighting factor for selecting an oblique route.

Here, if the value of the weighting factor λ for selecting the oblique route in the above equation (8) is set large, it becomes difficult to adopt the oblique route, and if the value of the weighting factor λ is set small, it becomes easy to adopt the oblique route. . In the conventional DP matching, the weighting factor λ is a fixed value. In the present invention, the weighting factor λ is set according to the size of the area average luminance distance d _{AVE. At} the intersection (i, j) of the matrix _. Change adaptively. That is, in the present invention, the weighting factor λ _{(i, j)} shown in the following equation (9 ₎ is employed instead of the weighting factor λ (fixed value) for the oblique path selection in the above equation (8). In the following formula (9), w is a fixed value, and an appropriate value is set. In the present invention, w = 0.15 to 0.35.

In DP matching, even when there is no appropriate matching target for the pixel of interest, it is possible to appropriately process two or more pixels by associating one pixel with a vertical or horizontal path. Therefore, there is no need to perform additional processing for that purpose. Here, whether the path takes an oblique path or a vertical or horizontal path largely depends on the value of the weighting factor λ _{(i, j)} . Therefore, in the present invention, the matching accuracy rate is further improved by adaptively changing the weighting factor λ _{(i, j)} . That is, in the present invention, the larger the cost c (i, j) given to the matrix intersection (i, j) (that is, the greater the possibility that there is no appropriate matching target for the pixel of interest). Since the value of the weighting factor λ _{(i, j)} increases, the path is configured to easily take a vertical or horizontal path. The configuration for adaptively determining the weighting coefficient λ for diagonal route selection has been described above. Next, the third feature of the depth value calculation unit 108, that is, the omission of matching processing will be described.

(About omission of matching process)
As described above with reference to FIG. 3, DP matching in stereo vision measurement is performed independently for each scanning line. In conventional stereo vision measurement, for example, when the vertical width of the main camera image is 300 pixels, 300 times of DP matching is necessary to obtain the parallax of all the pixels. In this regard, the present invention pays attention to the fact that there are many pixel areas in which the parallax does not change in the h-axis direction of the image, and the already acquired parallax is inherited for the pixel area that is assumed to have no change in the parallax. As a result, the processing amount is greatly reduced, thereby realizing real-time processing time. The specific contents will be described below based on the flowcharts shown in FIGS.

  The following description will be made taking the main camera image shown in FIG. 11A as an example. Note that the main camera image shown in FIG. 11A has 25 (0 to 24) scanning lines.

  First, in step 201, a normal matching process is executed for the bottom scanning line (h coordinate value = 0), and parallax for each pixel is acquired. A scanning line for which parallax acquisition has been completed is marked as “parallax acquired” (hereinafter the same). Thereafter, in step 202, the parameter ε is set to “0”.

  Next, in step 203, the h coordinate value (= ε + α) of the scanning line that is separated from the scanning line for which the parallax has been acquired (in this case, the scanning line with h coordinate value = 0) is specified, and the scanning is performed. A normal matching process is performed on the line to obtain a parallax for each pixel. In the present invention, “α” is a parameter for defining the inheritance range of parallax, and is a fixed value of an appropriate size defined as a multiple of 2. In the example shown in FIG. 11, α = 8. Is set. Therefore, h coordinate value = ε + α = 0 + 8 = 8 is designated, and a normal matching process is executed for the scanning line with h coordinate value = 8 to obtain parallax.

Next, in step 204, a scanning line marked as “parallax has been acquired” is searched, and an h coordinate value of a scanning line (hereinafter referred to as an intermediate scanning line) existing in the middle of adjacent scanning lines is determined. specify. The h coordinate value of the intermediate scanning line is calculated by adding h coordinate value = h _U of the upper scanning line and h coordinate value = h _L of the lower scanning line of adjacent scanning lines and dividing this by 2. You can ask for it. In this case, h coordinate value = 0 of the lower scanning line and h coordinate value = 8 of the upper scanning line are added and divided by 2 to obtain a value “4” as the h coordinate value of the intermediate scanning line. Become. When the intermediate scanning line (h coordinate value = 4) is designated, the process proceeds to step 205, and “parallax inheritance processing” is executed for each pixel (i, h) of the designated intermediate scanning line. Hereinafter, the “parallax inheritance process” in the present invention will be described with reference to FIG.

  In the “parallax inheritance process”, the parallax of the pixel (i, h + n) on the upper scanning line having the same i coordinate value as the pixel (i, h) of the designated intermediate scanning line and the lower scanning line The parallax of the pixel (i, h-n) is compared. As a result, when the two parallaxes are equal, the parallax is inherited by the pixel (i, h) of the intermediate scanning line, and when the two parallaxes are different, the pixel (i, h) of the intermediate scanning line is changed to “ Mark "Needs Action".

  In the example shown in FIG. 11B, when attention is paid to the pixel (0, 4) of the intermediate scanning line (h coordinate value = 4), the lower scanning line having the same i coordinate as the pixel (0, 4). The parallax (= 5) of the pixel (0,0) at (h coordinate value = 0) is equal to the parallax (= 5) of the pixel (0,8) of the upper scanning line (h coordinate value = 8). In this case, it is estimated that there is no parallax change in the pixel (0, 4) sandwiched between them, and the parallax = 5 is inherited by the pixel (0, 4). Focusing on the pixel (7, 4) of the intermediate scanning line, the parallax (= 4) of the pixel (7, 0) of the lower scanning line and the parallax (=) of the pixel (7, 8) of the upper scanning line Since 4) is equal, similarly, the parallax = 5 is inherited by the pixel (7, 4). As described above, according to the method of the present invention, the parallax can be acquired without requiring the matching process for a part of the region on the intermediate scanning line, so that the processing amount can be greatly reduced. The processing speed can be significantly increased.

  On the other hand, focusing on the pixel (3,4) of the intermediate scanning line, the parallax (= 5) of the pixel (3,0) on the lower scanning line of the pixel (3,4) and the pixel (3 on the upper scanning line) , 8) have different parallax (= 3). In this case, it is determined that the parallax needs to be acquired again for the pixel (3,4) sandwiched between them, and the pixel (3,4) is marked as “necessary processing”.

  As a result of the above-described processing being executed for all pixels of the intermediate scanning line (h coordinate value = 4), pixel (0,4), pixel (1,4), pixel of the intermediate scanning line (h coordinate value = 4) For (2,4), pixel (7,4), pixel (8,4), pixel (9,4), the already calculated parallax is inherited, pixel (3,4), pixel (4,4) Pixel (5, 4) and pixel (6, 4) are marked as “required processing”. Thereafter, the normal matching process is executed only for the pixel area marked as “necessary processing”. As a result, parallax is acquired for all the pixels of the intermediate scanning line (h coordinate value = 4), and the intermediate scanning line (h coordinate value = 4) is marked as “parallax already acquired”.

  Next, in step 206, it is determined whether or not the maximum value of h coordinate value differences between a plurality of scanning lines marked “parallax already acquired” = 1. When the maximum value of the difference of the h coordinate values between the scanning lines is not 1 (No at Step 206), an area where the parallax between the two parallax-acquired scanning lines is not acquired remains. The processing from step 204 to step 205 is executed again.

  In the case of the example shown in FIG. 11B, the scanning lines marked “Parallax acquired” are scanning lines (h coordinate value = 0), scanning lines (h coordinate value = 4), scanning lines (h coordinate value). = 8) and the maximum value of the difference of the h coordinate values between the parallax-acquired scanning lines is 4 (No at Step 206), the process returns to Step 204 to specify a new intermediate scanning line. . Specifically, a parallax-acquired scanning line (h-coordinate value = 0) and an intermediate scanning line (h-coordinate value = 2) with respect to the parallax-acquired scanning line (h-coordinate value = 4) are designated, and the parallax-acquired scanning line An intermediate scanning line (h coordinate value = 6) with respect to (h coordinate value = 4) and a parallax-acquired scanning line (h coordinate value = 8) is designated. Thereafter, the process proceeds to step 205, and the above-described parallax inheritance processing is executed for the newly designated intermediate scanning line (h coordinate value = 2) and intermediate scanning line (h coordinate value = 6).

  Thereafter, in step 206 again, it is determined whether or not the maximum value of the difference of h coordinate values between a plurality of scanning lines marked as “parallax already acquired” = 1, in this case, the parallax acquired scanning line Since the maximum value of the difference in the h coordinate value is 2 (step 206, No), the processing of steps 204 to 205 is repeated once again. As a result, the maximum value of the difference of h coordinate values between the parallax-acquired scanning lines is 1 (step 206, Yes). Since the maximum value of the coordinate value difference between the scanning lines = 1 means that there is no unprocessed area sandwiched between the two parallax-acquired scanning lines, the processing count C is incremented (step 207), the process proceeds to Step 208. In step 208, it is determined whether or not a predetermined target processing count has been achieved. Here, the target processing times is set to a value obtained by dividing the total number of scanning lines of the main camera image minus 1 by the parameter α (a multiple of 2). In the example shown in FIG. 11A, since the total number of scanning lines of the main camera image is “25” and the parameter α = 8, (25−1) ÷ 8 = 3 times is the target processing number.

  In this case, since the processing count C = 1 (step 208, No), the process proceeds to step 209. In step 209, a value obtained by adding the parameter α to the parameter ε is set as a new parameter ε (0 + 8 = 8), and the process proceeds to step 203. In step 203, a scanning line having an h coordinate value (= ε + α = 8 + 8 = 16) is designated, and a normal matching process is executed for the scanning line to obtain parallax. Thereafter, the same processing as described above is executed for the seven scanning lines existing between the parallax-acquired scanning line (h coordinate value = 8) and the parallax-acquired scanning line (h coordinate value = 16). In step 209, the parameter ε is further updated (8 + 8 = 16), and further, between the parallax acquired scanning line (h coordinate value = 16) and the parallax acquired scanning line (h coordinate value = 24). The same processing as described above is executed for the existing seven scanning lines. As a result, the processing count C = 3 (step 208, Yes), the acquisition of the parallax of all the pixels on all the scanning lines (25) is completed, and the processing ends.

  As described above, according to the configuration of the depth value calculation unit 108 employed by the present invention, the depth value in 3D data can be calculated with high accuracy, and the depth value calculation processing speed can be calculated in real time. Sex is guaranteed. As a result, a stereoscopic endoscope system capable of providing a stereoscopic image of a live-action image of an affected area in multiple viewpoints in real time to a naked eye surgical staff is realized.

  As described above, the present invention has been described with the embodiments. However, the present invention is not limited to the above-described embodiments, and other functions and effects of the present invention are within the scope of embodiments that can be considered by those skilled in the art. As long as it plays, it is included in the scope of the present invention. Needless to say, the parallax calculation method employed by the depth value calculation unit of the present invention described above is not limited to a stereoscopic endoscope, and can be applied to all techniques using stereo vision measurement.

  Hereinafter, the multi-view autostereoscopic system of the present invention will be described more specifically with reference to examples. However, the present invention is not limited to the examples described later.

  A computer (CPU: Intel Core2 Quad Q9300 / memory: 3GB / OS: Windows (registered trademark) XP (32bit) with a new DP matching method adopted by the depth value calculation unit of the multi-view autostereoscopic system of the present invention )) Parallax measurement experiment was conducted. In this experiment, four sets of stereo images ("Cones", "Teddy", "Tsukuba", "") obtained from http://vision.middlebury.edu/stereo/ shown in FIGS. Venus ") was used.

In this experiment, the computer of Example 1 was operated under the following conditions. That is, for dynamic setting of the luminance acquisition region G, (1) the maximum number of pixels E = 3 (upper side), −3 (lower side), (2) the threshold value K _{C of the} luminance distance for the pixel of interest (i, h) = 10, (3) The threshold value K _e of the luminance change amount k was set to 400.

The weighting factor λ _{(i, j)} at the tip of the oblique path is set to w = 0.25 in the following equation (9).

  Further, for the cost c (i, j), the root r = 512 in the following formula (7) is used.

  On the other hand, as a comparative example, a parallax measurement experiment using a computer (with the same specifications as in Example 1) on which a conventional stereo vision measurement method was implemented was performed. That is, the luminance distance of a single pixel is defined as a cost, and the weighting coefficient λ = 0.4 (fixed value) for oblique path selection.

<Verification of accuracy rate>
FIGS. 14-17 each show the results obtained for four stereo image pairs (“Cones”, “Teddy”, “Tsukuba”, “Venus”). 14A shows the correct parallax prepared in advance, FIG. 14B shows the parallax of the comparative example, and FIG. 14C shows the parallax of the first embodiment (FIG. 15). The same applies to FIG. 16).

  Table 1 below shows the correct answer rate [%] in four sets of stereo images (“Cones”, “Teddy”, “Tsukuba”, “Venus”) for Example 1 and the comparative example. In this experiment, it is assumed that “correct answer” = deviation with respect to the correct parallax is within one pixel, and the correct answer rate [%] = the number of correct pixels / the total number of pixels capable of calculating the parallax × 100.

  As shown in Table 1 above, the accuracy rate of Example 1 was 10 points or more higher than that of the comparative example.

<Verification of high-speed processing>
In addition to the configuration of the first embodiment described above, four sets of stereo images (“Cones”, “Teddy”) shown in FIGS. The parallax measurement experiment was conducted using "Tsukuba" and "Venus". In this experiment, the parallax was measured by parallel processing in which the measurement target image was divided into four in the X-axis direction. In this experiment, five conditions (parameter α = 2, 4, 8, 16, 32) are set for the parameter α for defining the inheritance range of the parallax. It was. Table 2 below shows MDE / S (Million Disparity Estimations per second) for four pairs of stereo images (“Cones”, “Teddy”, “Tsukuba”, “Venus”) for Examples 1 to 6. FIG. 18 shows the relationship between the parameter α and the average value of MDE / S. MDE / S is an index of calculation speed that does not depend on the size of an image defined as (maximum parallax) × (total number of pixels of image) × (number of images processed per second) × 10 ⁶ It is.

  As shown in Table 2 and FIG. 18, the value of MDE / S increased almost in proportion to the value of parameter α. Further, Table 3 below shows the accuracy rate [%] in the four sets of stereo images (“Cones”, “Teddy”, “Tsukuba”, “Venus”) of Examples 1 to 6. FIG. 19 shows the relationship between the parameter α and the average value of the correct answer rate [%].

  As shown in Table 3 and FIG. 19, the correct answer rate [%] tended to decrease slightly as the value of the parameter α increased, but still maintained a considerably high value as compared with the comparative example. .

  As described above, according to the present invention, a novel stereoscopic endoscope system capable of releasing surgical staff from stress peculiar to a stereoscopic endoscope system and supporting high-quality treatment is provided. The With the widespread use of the multi-view autostereoscopic system of the present invention, it is expected that the surgical operation will be further advanced, and as a result, even one person will be saved many lives.

DESCRIPTION OF SYMBOLS 100 ... Multi viewpoint autostereoscopic system 102 ... Multi viewpoint stereo image generation part 104 ... 3D data generation part 106 ... Rendering part 108 ... Depth value calculation part 200 ... Glasses type | mold stereoscopic endoscope system 201 ... Stereo camera 202 ... Left eye Camera 204 ... Right-eye camera 206 ... Image output control unit 208 ... Image switching unit 210 ... Display 212 ... 3D glasses 300 ... Multi-viewpoint autostereoscopic display device 302 ... Lenticular lens array

Claims (14)

A stereo including a stereo camera, a multi-view stereo image generating unit that generates a multi-view stereo image based on a stereo image acquired by the stereo camera, and a multi-view autostereoscopic display for displaying the multi-view stereo image An endoscope system,
The multi-viewpoint stereo image generation unit
A 3D data generation unit that generates 3D data in which a depth value and an RGB value for each pixel of a stereo image acquired by the stereo camera are associated;
A rendering unit that draws a multi-view stereo image formed by aggregating a plurality of stereo images corresponding to a plurality of virtual viewpoints based on the 3D data,
Stereoscopic endoscope system. The 3D data generation unit includes a depth value calculation unit,
The depth value calculation unit
Obtaining a parallax for each pixel by performing stereo matching based on dynamic programming for the stereo image, and calculating a depth value for each pixel based on the parallax,
The stereoscopic endoscope system according to claim 1. The depth value calculation unit
Defining a plurality of pixels including the target pixel in the vertical direction as a luminance value acquisition region, and defining an average value of luminance distances calculated for each pixel of the plurality of pixels as a cost in the stereo matching;
The stereoscopic endoscope system according to claim 1 or 2. The depth value calculation unit
Defining a plurality of pixels including the target pixel in the vertical direction as a luminance value acquisition region, and defining a power root of an average value of luminance distances calculated for each pixel of the plurality of pixels as a cost in the stereo matching;
The stereoscopic endoscope system according to claim 1 or 2. The depth value calculation unit
Dynamically defining the luminance value acquisition region that does not extend across an edge;
The stereoscopic endoscope system according to claim 3 or 4. The depth value calculation unit
A weighting factor for oblique path selection in the stereo matching is set so as to increase according to the average value of the luminance distances.
The three-dimensional endoscope system according to any one of claims 3 to 5. The depth value calculation unit
The target scanning line in the stereo image is defined at predetermined intervals, and the parallax of each pixel on the scanning line is acquired by the stereo matching, and then between the two pixels that have acquired the parallax in the vertical direction. When the parallax between the two pixels does not change with respect to the pixel located, when the parallax between the two pixels is inherited by the pixel located between the two pixels, and the parallax between the two pixels changes Performs a series of processes such as newly obtaining the parallax by performing the stereo matching only for the pixel located between the two pixels,
The stereoscopic endoscope system according to any one of claims 2 to 6. The depth value calculation unit
Recursively repeating the series of processes until the parallax of all the pixels of the stereo image is acquired,
The stereoscopic endoscope system according to claim 7. A method of performing stereo matching based on dynamic programming to obtain parallax for each pixel of a stereo image,
A plurality of pixels including the target pixel that are continuous in the vertical direction are defined as a luminance value acquisition region, and an average value of luminance distances calculated for each pixel of the plurality of pixels is defined as a cost in the stereo matching, how to. A method of performing stereo matching based on dynamic programming to obtain parallax for each pixel of a stereo image,
A plurality of pixels including the target pixel that are continuous in the vertical direction are defined as a luminance value acquisition region, and a power root of an average value of luminance distances calculated for each pixel of the plurality of pixels is defined as a cost in the stereo matching. A method characterized by.   The method according to claim 9 or 10, wherein the luminance value acquisition region is dynamically defined so as not to extend across an edge.   The method according to any one of claims 9 to 11, wherein a weighting coefficient for selecting an oblique path in the stereo matching is set so as to increase in accordance with an average value of the luminance distance.   The target scanning line in the stereo image is defined at predetermined intervals, and the parallax of each pixel on the scanning line is acquired by the stereo matching, and then between the two pixels that have acquired the parallax in the vertical direction. When the parallax between the two pixels does not change with respect to the pixel located, when the parallax between the two pixels is inherited by the pixel located between the two pixels, and the parallax between the two pixels changes The method according to any one of claims 9 to 12, wherein a series of processing is performed such that only a pixel located between the two pixels is newly subjected to the stereo matching to obtain parallax. The method described.   The method according to claim 13, wherein the series of processes is recursively repeated until parallax of all pixels of the stereo image is acquired.