KR102060880B1 - Endomicroscopy using single lens camera and operating method thereof - Google Patents

Abstract

An endoscope microscope using a monocular camera equipped with a camera system composed of a single optical system and an image sensor and a method of operating the same are disclosed. An endoscope microscope using a monocular camera according to an embodiment is located in the distal end of the endoscope microscope, a single optical system and an image sensor-the image sensor is composed of a plurality of pixels for processing a light beam having a plurality of wavelengths for each wavelength A camera system comprising at least one pixel set; And at least one processor configured to calculate a distance between the endoscope microscope and an object located around the endoscope microscope using at least two images acquired by the image sensor.

Description

Endoscopic Microscope Using Monocular Camera and its Operation Method {ENDOMICROSCOPY USING SINGLE LENS CAMERA AND OPERATING METHOD THEREOF}

The following description relates to an endoscope that can directly observe the inside of the organ or body cavity, and more specifically, an endoscope microscope using a monocular camera and a technique for operating the same.

An endoscopic microscope is a medical device that is inserted into the body where the lesion cannot be seen directly without surgery or incision. In accordance with the development of image processing technology, the endoscope microscope photographs each part of the body cavity with a camera system mounted on the endoscope microscope, and then the lesions of each part can be observed in detail through the captured image. A technique for such an endoscope is disclosed in Korean Patent Laid-Open Publication No. 10-2014-0116341.

However, because the conventional endoscope only provides a two-dimensional image, the lesion is difficult to accurately detect only the two-dimensional image in the color similar to the surrounding tissue, but the height is different.

Accordingly, there is a demand for a technology capable of capturing a 3D image having a depth applied to the image in an endoscope microscope.

One embodiment proposes an endoscope microscope capable of obtaining a three-dimensional image to which the depth is applied, and a method of operating the same.

In particular, one embodiment uses the two images obtained from the image sensor while minimizing the cross-sectional area of the camera system by minimizing the cross-sectional area of the camera system by having a structure including a single optical system and an image sensor included in the endoscope microscope A technique for calculating the distance (depth) between an endoscope microscope and an object located around the endoscope microscope is proposed.

According to one embodiment, the endoscope microscope using a monocular camera, a single optical system and an image sensor, the image sensor is composed of a plurality of pixels for processing a light beam having a plurality of wavelengths for each wavelength while being located at the tip of the endoscope microscope A camera system comprising at least one set of pixels; And at least one processor configured to calculate a distance between the endoscope microscope and an object located around the endoscope microscope using at least two images acquired by the image sensor.

According to an aspect of the present invention, an offset formed with an Offset Pixel Aperture (OPA) having a center position deviating from the center of the one pixel, on one of the plurality of pixels included in the pixel set of the image sensor. A mask is disposed, and the at least one processor excludes an image obtained through any one pixel on which the offset mask is disposed and any one pixel on which the offset mask is disposed among the plurality of pixels. The distance between the endoscope microscope and the object may be calculated by using a disparity between the images acquired through the other one pixel.

According to another aspect, the OPA formed in the offset mask may have a center position that is offset in any one direction horizontally, vertically or diagonally with respect to the center of one pixel on which the offset mask is disposed.

According to another aspect, the offset mask may be disposed on top of any one of the pixels having the maximum light transmission characteristics of the plurality of pixels.

According to another aspect of the present invention, any one pixel on which the offset mask is disposed may include a beam of light processed by any one pixel except for any one pixel among the plurality of pixels, on which the offset mask is disposed. It is possible to process light rays of different wavelengths that are distinct from the wavelength.

According to another aspect, when the image sensor includes a plurality of pixel sets, an OPA of an offset mask disposed on an upper part of one pixel in any one of the plurality of pixel sets may include the plurality of pixel sets. The pixel set may have a center position that is offset from the center of the OPA of the offset mask disposed above any one pixel in any one of the pixel sets except for the one pixel set.

According to another aspect, the at least one processor further comprises: an image obtained through any one pixel in which the offset mask is disposed in the one pixel set and the offset mask in the other one pixel set; The distance between the endoscope microscope and the object may be calculated by using a parallax between images acquired through one pixel disposed above.

According to another aspect, the OPA may be formed on the offset mask to have any one of a circle, an ellipse, a triangle, a square, or a polygon.

According to another aspect, an upper portion of any one of the plurality of pixels included in the pixel set of the image sensor blocks a light ray at a periphery of the bundle of light rays and injects only a central light ray into the one pixel. And a mask (Pixel Aperture) having a center position coincident with any one of the pixels is disposed in the mask, and the at least one processor is configured to select one pixel on which the mask is disposed. The endoscopic microscope and the object using a blur change between an image obtained through the image and an image obtained through any one of the pixels except for any one of the pixels on which the mask is disposed. The distance between them can be calculated.

According to another aspect, the single optical system is formed with a plurality of apertures, each of which introduces light rays of different wavelengths, the plurality of apertures having a central position coinciding with each other, wherein the at least one processor is An image obtained by processing a light ray having a wavelength introduced through any one of the plurality of apertures, and any one of the plurality of apertures except the one of the plurality of apertures. The distance between the endoscope microscope and the object may be calculated by using a blurring change between images obtained by processing a light beam having an incoming wavelength.

According to another aspect, the single optical system is formed with a plurality of apertures, each of which injects light rays of different wavelengths, the plurality of apertures having a center position deviated from each other, the at least one processor, An image obtained by processing a light ray having a wavelength introduced through one of the plurality of apertures and an inflow through any one of the plurality of apertures except the one of the plurality of apertures The distance between the endoscope microscope and the object may be calculated using the parallax between the images obtained by processing light rays having a predetermined wavelength.

According to another aspect, any one of a plurality of pixels included in the pixel set of the image sensor may include a center sub-pixel to which a light ray at a center of the bundle of rays is incident and a light ray at a periphery of the bundle of rays. And at least one peripheral sub-pixel incident, wherein the at least one processor is configured to blur between an image obtained through the center sub-pixel and an image obtained by merging a signal obtained through the at least one peripheral sub-pixel. The change may be used to calculate the distance between the endoscope microscope and the object.

According to another aspect, the at least one processor, when the at least one peripheral sub-pixel is provided with, the endoscope microscope and by using the parallax between the images respectively obtained through the plurality of peripheral sub-pixels The distance between the objects can be calculated.

According to another aspect, the at least one processor may perform an operation for avoiding collision between the endoscope microscope and the object based on a distance between the endoscope microscope and the object.

According to another aspect, the camera system may be located at the tip of the curved portion that can be bent or curved freely in the endoscope microscope.

According to an embodiment, the method of operating an endoscope microscope using a monocular camera may include an image sensor constituting a camera system located at the distal end of the endoscope microscope together with a single optical system, wherein the image sensor processes light rays having a plurality of wavelengths for each wavelength. At least one pixel set consisting of a plurality of pixels, the method comprising: obtaining two images of an object located around the endoscope microscope; And calculating, at at least one processor, the distance between the endoscope microscope and the object using the two images,

According to an aspect of the present invention, an offset formed with an Offset Pixel Aperture (OPA) having a center position deviating from the center of the one pixel, on one of the plurality of pixels included in the pixel set of the image sensor. A mask is disposed, and the obtaining of the two images comprises: obtaining a first image through any pixel on which the offset mask is disposed; And acquiring a second image through any one of the pixels except for any one of the plurality of pixels in which the offset mask is disposed, and calculating a distance between the endoscope microscope and the object. The calculating may include calculating a distance between the endoscope microscope and the object by using a disparity between the first image and the second image.

According to another aspect, when the image sensor includes a plurality of pixel sets, an OPA of an offset mask disposed above any one pixel in any one of the plurality of pixel sets includes: Obtaining the two images has a center position deviating from the center of the OPA of the offset mask disposed on top of any one pixel in any one of the pixel sets other than the one of the pixel sets. Acquiring a first image through any one pixel in which the offset mask is disposed above the one pixel set; And acquiring a second image through one pixel in which the offset mask is disposed in the other one pixel set, and calculating the distance between the endoscope microscope and the object comprises: Computing the distance between the endoscope microscope and the object using the parallax between the first image and the second image.

According to another aspect, an upper portion of any one of the plurality of pixels included in the pixel set of the image sensor blocks a light ray at a periphery of the bundle of light rays and injects only a central light ray into the one pixel. A mask (Pixel Aperture) having a center position coincident with any one of the pixels is disposed thereon, and obtaining the two images comprises: a mask having the mask disposed thereon. Acquiring a first image through a pixel of; And acquiring a second image through any one of the plurality of pixels except for any one of the pixels on which the mask is disposed, and calculating a distance between the endoscope microscope and the object. The method may include calculating a distance between the endoscope microscope and the object by using a blur change between the first image and the second image.

According to another aspect, the single optical system is formed with a plurality of apertures, each of which injects light rays of different wavelengths, the plurality of apertures having a central position coinciding with each other, and obtaining the two images. The method may include obtaining a first image based on a light ray having a wavelength passing through an aperture of any one of the plurality of apertures; And acquiring a second image based on a light ray having a wavelength passing through any one of the plurality of apertures except the one of the plurality of apertures, wherein the second image is between the endoscope microscope and the object. The calculating of the distance may include calculating a distance between the endoscope microscope and the object by using a blur change between the first image and the second image.

According to another aspect, the single optical system is formed with a plurality of apertures, each of which injects light rays of different wavelengths, the plurality of apertures having a displaced center position, and acquires the two images. The method may include obtaining a first image based on a light ray having a wavelength passing through an aperture of any one of the plurality of apertures; And acquiring a second image based on a light ray having a wavelength passing through any one of the plurality of apertures except the one of the plurality of apertures, wherein the second image is between the endoscope microscope and the object. The calculating of the distance may include calculating a distance between the endoscope microscope and the object by using a parallax between the first image and the second image.

According to another aspect, any one of a plurality of pixels included in the pixel set of the image sensor may include a center sub-pixel to which a light ray at a center of the bundle of rays is incident and a light ray at a periphery of the bundle of rays. Comprising at least one peripheral sub-pixel incident, wherein acquiring the two images comprises: acquiring a first image through the central sub-pixel; And generating a second image by merging the signals acquired through the at least one peripheral subpixel, wherein calculating the distance between the endoscope microscope and the object comprises: the first image and the second image; Computing the distance between the endoscope microscope and the object using the blurring change between the images.

According to another aspect, the generating of the second image may include generating a third image through one of the plurality of peripheral subpixels when the at least one peripheral subpixel is provided. Obtaining; And acquiring a fourth image through any one of the plurality of peripheral subpixels except for one of the peripheral subpixels, and calculating a distance between the endoscope microscope and the object. The step may include calculating a distance between the endoscope microscope and the object by using a parallax between the third image and the fourth image.

According to another aspect, the method of operating the endoscope microscope using the monocular camera may further include performing an operation for avoiding collision between the endoscope microscope and the object based on a distance between the endoscope microscope and the object. Can be.

One embodiment may propose an endoscope capable of obtaining a three-dimensional image to which the depth is applied, and an operation method thereof.

In particular, one embodiment uses the two images obtained from the image sensor while minimizing the cross-sectional area of the camera system by minimizing the cross-sectional area of the camera system by having a structure including a single optical system and an image sensor included in the endoscope microscope Therefore, a technique for calculating a distance (depth) between an endoscope microscope and an object located around the endoscope microscope can be proposed.

FIG. 1 is a diagram for explaining a principle of calculating a distance between an image sensor and a subject by using parallax between two images.
2 to 4 are diagrams for describing a principle of calculating a distance between an endoscope microscope and an object in a camera system of an endoscope microscope to which an offset pixel aperture (OPA) is applied, according to an exemplary embodiment.
5 is a diagram illustrating a schematic structure of an endoscope according to an embodiment.
6 is a diagram illustrating a schematic structure of a camera system mounted on an endoscope according to an embodiment.
7 is a diagram for describing a camera system of an endoscope microscope to which an OPA is applied, according to an exemplary embodiment.
8 is a diagram for describing a camera system of an endoscope microscope to which OPA is applied, according to another exemplary embodiment.
9 is a diagram for describing a camera system of an endoscope microscope to which an OPA is applied, according to another embodiment.
FIG. 10 is a diagram illustrating two images acquired by the camera system of the endoscope microscope shown in FIG. 9.
FIG. 11 is a diagram for describing a camera system of an endoscope microscope to which a pixel aperture (PA) is applied, according to an exemplary embodiment.
12 is a diagram for describing a camera system of an endoscope microscope to which a dual aperture (DA) is applied, according to an exemplary embodiment.
13 is a diagram for describing a camera system of an endoscope microscope to which DA is applied, according to another exemplary embodiment.
14 is a diagram for describing a camera system of an endoscope microscope to which a subpixel structure is applied, according to an exemplary embodiment.
15 is a flowchart illustrating a method of operating an endoscope according to an embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

In addition, terms used in the present specification are terms used to properly express preferred embodiments of the present invention, which may vary depending on the intention of the viewer, the operator, or customs in the field to which the present invention belongs. Therefore, the definitions of the terms should be made based on the contents throughout the specification. For example, in this specification, the singular also includes the plural unless specifically stated otherwise in the text. Also, as used herein, “comprises” and / or “comprising” refers to one or more other components, steps, operations and / or elements. It does not exclude the presence or addition of devices.

In addition, it should be understood that various embodiments of the present invention are different from one another, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it should be understood that the position, arrangement, or configuration of individual components in each of the exemplary embodiments presented may be changed without departing from the spirit and scope of the present invention.

In order to obtain a three-dimensional image to which the depth is applied, the depth of each pixel included in the two-dimensional image should be calculated. In this case, a conventional method of calculating the depth of each pixel included in the two-dimensional image includes a time of flight (TOF) for irradiating a laser to a subject (object) to be photographed and measuring the time that the light is returned. Method, a depth from stereo method that calculates depth by using parallax between images acquired in two or more camera systems, and light passing through each of a plurality of apertures formed in a single optical system in a single camera system. A method of calculating depth using parallax between images obtained by processing a signal (a parallax difference method using apertures), and processing an optical signal passing through each of a plurality of apertures formed in a single optical system in a single camera system There is a method of calculating the depth by using the blurring change between the acquired images.

On the other hand, the camera system included in the endoscope microscope according to an embodiment has a structure including a single optical system and an image sensor, minimizing the cross-sectional area of the camera system, and at the same time the two images obtained from the image sensor By using the method to calculate the distance (depth) between the image sensor of the endoscope microscope and the subject located around the endoscope microscope.

In particular, the camera system included in the endoscope microscope according to an embodiment calculates the distance between the endoscope microscope and the subject based on various depth calculation methods that can be utilized in the monocular camera. More specifically, the camera system included in the endoscope microscope includes an Offset Pixel Aperture based depth calculation method, a Pixel Aperture based depth calculation method, a DA (Dual Aperture) based depth calculation method, and a subpixel structure based depth calculation method. The distance between the endoscope microscope and the subject may be calculated using.

These depth calculation methods, based on the respective structures, will ultimately calculate the depth using the parallax between the images or the blur variation between the images. Hereinafter, the DA-based depth calculation method for calculating depth using the parallax between images will be described, and then the OPA-based depth calculation method based on the DA-based depth calculation method will be described.

FIG. 1 is a diagram for explaining a principle of calculating a distance between an image sensor and a subject by using parallax between two images.

Referring to FIG. 1, in the camera system 100, a plurality of apertures are formed in the single optical system 110 to introduce light rays having different wavelengths, respectively. In particular, the plurality of apertures have a central position deviated from each other on the single optical system 110. For example, the single optical system 110 may be formed with a first aperture and a second aperture having a center position offset from each other by being offset in one of horizontal, vertical, or diagonal directions (ie, a center position of the first aperture). And the central positions of the second aperture do not coincide).

At this time, since the first aperture and the second aperture are formed to have a center position deviated from each other, the center of the first image obtained by the image sensor 120 processing a light ray having a wavelength introduced by the first aperture. The position does not coincide with the center position of the second image obtained by the image sensor 120 processing a light ray having a wavelength introduced by the second aperture.

Using this principle, the camera system can calculate the parallax p between the first image and the second image as in Equation 1. Hereinafter, the parallax between the first image and the second image means the parallax between the center position of the first image and the center position of the second image.

<Equation 1>

는 이미지 센서(120)에 초점이 맞은 피사체 거리를 나타낸다.In Equation 1, x represents the distance between the center position of the first aperture and the center position of the second aperture, f represents the focal length, and a represents the subject distance (the first principal of the single optical system 110 from the subject). Distance to the plane)

Denotes the distance of the subject in focus to the image sensor 120.

At this time, when the value of parallax p between the center position of the first image and the center position of the second image is changed from positive to negative or from negative to positive, the direction of parallax between the two images is changed. Therefore, according to the sign of the value of p, whether the subject with the edge is in the foreground or background based on the position can be distinguished.

Also, from the equation 1, the subject distance a can be calculated as in the equation 2.

<Equation 2>

는 이미지 센서(120)에 초점이 맞은 피사체 거리를 나타내고, f는 초점 거리를 나타내며, p는 제1 이미지 및 제2 이미지 사이의 시차를 나타내고, x는 제1 애퍼처의 중심 위치와 제2 애퍼처의 중심 위치 사이의 거리를 나타낸다.In equation 2,

Denotes the distance of the subject in focus to the image sensor 120, f denotes the focal length, p denotes the parallax between the first image and the second image, and x denotes the center position of the first aperture and the second aperture Represents the distance between the center position of the aperture.

Therefore, the distance c between the image sensor 120 and the subject may be determined as in Equation 3.

<Equation 3>

In Equation 3, a represents the object distance (the distance from the subject to the first main plane of the single optical system 110), and b represents the distance between the first main plane of the single optical system 110 and the image sensor 120. .

2 to 4 are diagrams for describing a principle of calculating a distance between an endoscope microscope and an object in a camera system of an endoscope microscope to which an offset pixel aperture (OPA) is applied, according to an exemplary embodiment. More specifically, FIG. 2 illustrates only one pixel to which the OPA is applied among the plurality of pixels in the image sensor included in the camera system of an endoscope for explaining the distance calculation principle used by the camera system of the OPA structure according to an embodiment. 3 to 4 are diagrams for explaining the relationship between the aperture and the OPA formed in a single optical system.

Referring to FIG. 2, one pixel 200 of an image sensor included in an OPA applied camera system according to an embodiment may have a center position deviated from the center 201 of one pixel 200. OPA 212 with 211 includes an offset mask 210 formed. Accordingly, the light rays introduced into the single optical system 220 of the camera system pass through the OPA 212 formed in the offset mask 210 via the micro lens 230 and the color filter 240 of any one pixel 200. The solution is incident on the photodiode 250.

Accordingly, through the principle shown in the drawing, O _{2, which} is a distance at which the center position 211 of the OPA 212 is offset from the center 201 of the pixel 200, forms an aperture on the single optical system 220. In this case, the center position 221 of the aperture has a relationship proportional to O ₁ , which is a distance offset from the center 222 of the single optical system 220 (which is the same as the center 201 of the pixel 200).

Thus, the OPA 212 having a center position 211 deviated from the center 201 of the pixel 200 is from the center 222 of the single optical system 220 (which is the same as the center 201 of the pixel 200). The aperture may be equal to the aperture formed on the single optical system 220 to have a misaligned center position 221, and as a result, as shown in FIG. 3, any of the image sensors in the case 310 is formed in the single optical system. In the case where the OPA is applied (320), the light incident on any one pixel of the image sensor is applied to the offset mask, as if the light incident on one pixel is the light of the center of which the edge region is blocked by the aperture formed in the single optical system. The edge region may be a light ray of the central portion blocked by the edge region. Similarly, as shown in FIG. 4, in the case where an aperture is formed in a single optical system (410), when the OPA is applied as if a light ray incident on one pixel of the image sensor is a light ray of an edge region by an aperture formed in the single optical system. The ray of light incident on any one pixel of the image sensor at 420 may be a ray of the edge region by an offset mask.

Accordingly, in the above-described Equation 2, x, which is the distance between the center position of the first aperture and the center position of the second aperture, is OPA 212 on the image sensor in case of a camera system including an image sensor to which the OPA 212 is applied. Corresponds to the distance between the center position 211 of the OPA 212 of the pixel 200 to which the applied pixel 200 and the center position of the normal pixel to which the OPA 212 is not applied, and between Equation 2 between the first image and the second image. The parallax p is the image obtained through the pixel 200 with the OPA 212 applied and the image obtained through the normal pixel without the OPA 212 in the case of a camera system including an image sensor with the OPA 212 applied thereto. It can correspond to the time difference between.

, 초점 거리인 f, OPA(212)가 적용된 픽셀(200)을 통해 획득된 이미지 및 OPA(212)가 적용되지 않은 일반 픽셀을 통해 획득된 이미지 사이의 시차인 p, 그리고 OPA(212)가 적용된 픽셀(200)의 OPA(212)의 중심 위치(211)와 OPA(212)가 적용되지 않은 일반 픽셀의 중심 위치 사이의 거리인 x를 이용하여 식 2와 같이 피사체 거리 a를 계산할 수 있으며, 피사체 거리 a에 단일 광학계(220)와 이미지 센서 사이의 거리인 b를 더해 식 3과 같이 이미지 센서로부터 피사체 사이의 거리를 산출할 수 있다.Accordingly, the camera system mounted on the endoscope microscope according to an embodiment may be a subject distance focused on an image sensor.

, The focal length f, p, which is the parallax between the image obtained through the pixel 200 with the OPA 212 applied and the image obtained through the normal pixel without the OPA 212 applied, and the OPA 212 applied The object distance a can be calculated using Equation 2 using x, which is the distance between the center position 211 of the OPA 212 of the pixel 200 and the center position of the general pixel to which the OPA 212 is not applied, The distance a between the single optical system 220 and the image sensor b may be calculated by adding b, which is a distance between the single optical system 220 and the image sensor.

As described above, the depth calculation principle of the image sensor to which the OPA 212 is applied under the single optical system 220 has been described as being based on the parallax difference method in the DA structure, but the present invention is not limited thereto, and two parallaxes are generated. The images may be based on various ways of calculating depth in the image.

Hereinafter, a schematic structure of an image sensor to which the OPA 212 is applied under a single optical system 220 and a camera system including the image sensor, and a case in which the above-described OPA structure is applied to use the above-described depth calculation principle. The detailed structure of the camera system to which the system is applied, the structure to which the DA is applied, and the subpixel structure to which the camera system is mounted is described in detail.

5 is a diagram illustrating a schematic structure of an endoscope according to an embodiment.

Referring to FIG. 5, the endoscope 500 according to an embodiment is a device inserted into an examinee's body and photographing an organ or a body cavity as a photographing target, and an operation unit 510 which the inspector can grip and operate. The insertion unit 520 is connected to the operation unit 510 and inserted into the organ or body cavity.

At this time, the insertion portion 520 may be composed of a freely bendable or bendable curved portion 521 and a hard tip portion 522 installed at the tip of the curved portion 521, and the tip portion 522 has an OPA under a single optical system. A camera system can be located that includes an applied image sensor.

Therefore, when the insertion unit 520 is inserted into the examinee's body by the inspector's manipulation, the organ or body cavity of the examinee's body may be photographed by the operation of the camera system located at the tip 522. A detailed description of the camera system will be described with reference to FIG. 6.

In addition, although not shown in the drawing, the endoscope 500 is an endoscope based on an operation of calculating the distance between the endoscope microscope 500 and the object located in the vicinity of the endoscope microscope 500 and the distance between the endoscope microscope and the object. At least one processor may perform an operation for avoiding collision between the microscope and the object. Hereinafter, the at least one processor will be described as being provided in the camera system. However, the present invention is not limited thereto and may be provided on a separate computer device connected to the endoscope 500 wirelessly or by wire.

6 is a diagram illustrating a schematic structure of a camera system mounted on an endoscope according to an embodiment.

Referring to FIG. 6, the camera system 600 according to an exemplary embodiment has a monocular camera structure, and specifically includes a single optical system 610, an image sensor 620, and at least one processor 630. can do.

The single optical system 610 may be formed with a basic aperture for allowing the R (Red), G (Green), B (Blue) and W (White) light beams to be incident, but is not limited thereto. As will be described later, in one embodiment, DA (DA having mutually coincident center positions and introducing light rays of different wavelengths or DA having displaced center positions and light rays of different wavelengths) may be applied.

The image sensor 620 may acquire images by including at least one pixel set composed of a plurality of pixels for processing, for each wavelength, light rays (having a plurality of wavelengths) flowing through the single optical system 610. . According to the exemplary embodiment, the image sensor 620 includes any one of the pixels to which the OPA is applied as described above, thereby acquiring a first image through any one of the pixels to which the OPA is applied, and the other of the other ones to which the OPA is not applied. The second image may be acquired through the pixel. Detailed description thereof will be described with reference to FIG. 7.

In another example, the image sensor 620 includes any one of the pixels to which the PA has been applied, thereby acquiring the first image through any one of the pixels to which the PA has been applied, and through any other pixel to which the PA has not been applied. A second image may be obtained. A detailed description thereof will be described with reference to FIG. 11.

In another example, the image sensor 620 processes each of light rays of different wavelengths introduced through a single optical system 610 to which DAs having misaligned center positions are applied to each other. An image may be obtained, and the first and second images having a blur difference may be obtained by processing each of light rays having different wavelengths introduced through a single optical system 610 to which DAs having coincident center positions are applied. You may. Detailed description thereof will be described with reference to FIGS. 12 to 13.

In addition, any one pixel in the pixel set included in the image sensor 620 has a sub-pixel structure, so that the image sensor 620 acquires the first image through the center sub-pixel and acquires at least one peripheral sub-pixel. Through the second image can be obtained. Detailed description thereof will be described with reference to FIG. 14.

The at least one processor 630 calculates the distance between the endoscope microscope and the object located around the endoscope microscope using at least two images acquired by the image sensor 620. For example, a parallax exists between a first image acquired through one pixel to which the OPA is applied and a second image acquired through any other pixel to which the OPA is not applied. Accordingly, the at least one processor 630 may calculate the distance between the endoscope microscope and the object by using the first image and the second image having parallax based on the principle described above with reference to FIGS. 1 to 4. .

In another example, a blur difference exists between a first image acquired through one pixel to which the PA is applied and a second image acquired through any other pixel to which the PA is not applied. Accordingly, the at least one processor 630 may calculate a distance between the endoscope microscope and the object by using a blur change between the first image and the second image.

In another example, when a DA having a central position coinciding with each other and introducing light rays having different wavelengths is applied, a first obtained based on light rays having a wavelength passing through the aperture of any one of the plurality of apertures A blur difference exists between the image and the second image obtained based on the light of the wavelength passing through any one of the apertures except the one of the plurality of apertures. Accordingly, the at least one processor 630 may calculate the distance between the endoscope microscope and the object by using the blur change between the first image and the second image.

In another example, when a DA having misaligned center positions and introducing light rays having different wavelengths is applied, the first image acquired based on light rays having wavelengths passing through any one of the plurality of apertures are applied. And a parallax between the second image acquired based on a light ray having a wavelength passing through any one of the plurality of apertures except for the one of the plurality of apertures. Accordingly, the at least one processor 630 may calculate the distance between the endoscope microscope and the object by using the parallax between the first image and the second image.

In another example, when a subpixel structure is applied, a blur difference exists between a first image acquired through a center subpixel and a second image generated by merging a signal obtained through at least one peripheral subpixel. do. Accordingly, the at least one processor 630 may calculate the distance between the endoscope microscope and the object by using the blur change between the first image and the second image. Also, when a subpixel structure is applied, when a plurality of at least one peripheral subpixel is provided, a third image and a plurality of peripheral subpixels acquired through any one of the plurality of peripheral subpixels. The parallax exists between the fourth image acquired through any one of the peripheral subpixels except for the one of the peripheral subpixels. Accordingly, the at least one processor 630 may calculate the distance between the endoscope microscope and the object by using the parallax between the third image and the fourth image.

That is, the at least one processor 630 uses any one of the parallax or blur variation between the images, depending on which of the OPA structure, PA structure, DA structure or sub pixel structure is applied to the camera system 600. The distance between the endoscope microscope and the object can be calculated.

7 is a diagram for describing a camera system of an endoscope microscope to which an OPA is applied, according to an exemplary embodiment.

Referring to FIG. 7, an image sensor 700 included in an OPA applied camera system according to an embodiment may include a plurality of pixels 711, 712, and 713 for processing light rays having a plurality of wavelengths for each wavelength. , At least one pixel set 710 including 714. In the drawing, the image sensor 700 is illustrated as having nine pixel sets, and the pixel set 710 includes R pixels, G pixels, B pixels, and W pixels, but is limited or limited thereto. It doesn't work. That is, the pixel set 710 may be composed of W, R, B, and W pixels instead of R, G, B, and W pixels.

In particular, any one of the plurality of pixels 711, 712, 713, and 714 included in the pixel set 710 in the image sensor 700 may be disposed on the pixel (eg, the fourth pixel 714). An offset mask 714-4 having an OPA 714-3 having a center position 714-2 that is offset from the center 714-1 of the pixel (fourth pixel 714) may be disposed. Hereinafter, the center 714-1 of the pixel 714 means a center point on the top surface of the pixel 714, and the center 714-2 of the OPA 714-3 is the center point of the hole of the OPA 714-3. Means. That is, the OPA 714-3 formed in the offset mask 714-4 has the center 714-1 of any one pixel (fourth pixel 714) on which the offset mask 714-4 is disposed. It may have a center position 714-2 which is offset in the direction of any one of horizontal, vertical or diagonal with respect to.

Also, hereinafter, the OPA 714-3 may be formed on the offset mask 714-4 in the form of a circle, an ellipse, a triangle, a square, or a polygon. As described above, the OPA 714-3 does not have a limitation in shape, but only needs to have a center position 714-2 deviated from the center 714-1 of the pixel (fourth pixel 714).

Accordingly, the offset mask of the first image and the plurality of pixels 711, 712, 713, and 714 obtained through one pixel (the fourth pixel 714) on which the offset mask 714-4 is disposed. Any one pixel (for example, the first pixel 711, the second pixel 712, or the third pixel 713 except for any one pixel where the 714-4 is disposed (the fourth pixel 714) is disposed. Parallax exists between the second images acquired through any one of the pixels).

Accordingly, the at least one processor may include the first image and the plurality of pixels 711, 712, 713, and 714 obtained through any one pixel (the fourth pixel 714) on which the offset mask 714-4 is disposed. ) Any one of the pixels (for example, the first pixel 711, the second pixel 712, or the third) except for one pixel (the fourth pixel 714) in which the offset mask 714-4 is disposed. The distance between the endoscope microscope and the object (more precisely, the distance between the image sensor 700 and the object) can be calculated using the parallax between the second image acquired through any one of the pixels 713. Can be.

In this case, the offset mask 714-4 may be any one pixel having a maximum light transmission characteristic among the plurality of pixels 711, 712, 713, and 714 (eg, a fourth pixel that is a W pixel). 714)). This is because the offset mask 714-4 reduces the amount of light incident on the photodiode, and therefore, in order to acquire an image through the pixel on which the offset mask 714-4 is disposed, the light transmission characteristic of the pixel must be higher than the reference value. to be.

In addition, as shown in the drawing, any one pixel (the fourth pixel 714) on which the offset mask 714-4 is disposed is offset from among the plurality of pixels 711, 712, 713, and 714. Any one pixel (eg, the first pixel 711, the second pixel 712, or the third pixel 713 except for any one pixel (fourth pixel 714) on which the mask 714-4 is disposed. Can treat light rays of different wavelengths that are distinct from the wavelengths of light rays processed by either pixel. In other words, the offset mask 714-4 is disposed above any one pixel having a different wavelength of the light beam to be processed among the plurality of pixels 711, 712, 713, and 714. It is possible to obtain an image that is distinct from an image acquired through any other pixel not disposed above.

As such, the endoscope microscope according to an embodiment includes a camera system having a structure of an image sensor 700 to which an OPA 714-3 is applied under a single optical system, thereby minimizing the cross-sectional area of the camera system, A technique for calculating a distance (depth) between an endoscope microscope and an object using a parallax between two images (first image and second image) acquired by the image sensor 700 may be proposed.

In addition, according to an exemplary embodiment, the endoscope microscope to which the OPA is applied may include an offset mask (or an offset mask) in one pixel (fourth pixel 714) of the plurality of pixels 711, 712, 713, and 714 included in the pixel set 710. Instead of arranging 714-4, an offset mask 714-4 having a center position deviated from each other in each of the two pixels is used to take advantage of the parallax between images acquired through each of the two pixels. The distance between the endoscope microscope and the object may be calculated. Detailed description thereof will be described with reference to FIG. 8.

8 is a diagram for describing a camera system of an endoscope microscope to which OPA is applied, according to another exemplary embodiment.

Referring to FIG. 8, the image sensor 800 included in the camera system of an endoscope microscope to which an OPA is applied according to another embodiment may include a plurality of pixels 811, 812, which process light rays having a plurality of wavelengths for each wavelength. At least one pixel set 810 including 813 and 814 may be provided. For example, the pixel set 810 may be composed of a first W pixel 811, an R pixel 812, a B pixel 813, and a second W pixel 814.

At this time, the center 811-1 of any one pixel 811 is located above one of the pixels 811 of the plurality of pixels 811, 812, 813, and 814 included in the pixel set 810. An offset mask 811-4 with an OPA 811-3 having a shifted center position 811-2 may be disposed. In addition, in the image sensor 800, the other pixel 814 is also on the top of any other pixel 814 except for the pixel 811 of the plurality of pixels 811, 812, 813, and 814. An offset mask 814-4 with an OPA 814-3 having a center position 814-2 deviating from the center 814-1 may be disposed.

In particular, in the image sensor 800, the OPA 811-3 of the offset mask 811-4 disposed on the first W pixel 811 is offset on the second W pixel 814. The center position 811-2 deviated from the center 814-2 of the OPA 814-3 of the mask 814-4.

Accordingly, there is a parallax between the first image acquired through the first W pixel 811 and the second image obtained through the second W pixel 814. Thus, the at least one processor uses the parallax between the first image acquired through the first W pixel 811 and the second image acquired through the second W pixel 814 to determine the distance between the endoscope microscope and the object. (More precisely, the distance between the image sensor 800 and the object) can be calculated.

FIG. 9 is a diagram illustrating a camera system of an endoscope microscope to which an OPA is applied according to another embodiment, and FIG. 10 is a view illustrating two images acquired by the camera system of the endoscope microscope shown in FIG. 9.

9, an image sensor 900 included in an OPA applied camera system according to another embodiment may include a plurality of pixels 911 and 912 for processing light rays having a plurality of wavelengths for each wavelength. And at least two pixel sets 10 and 920 including 913, 914, 921, 922, 923, and 924. In the drawing, the image sensor 900 is illustrated as having nine pixel sets, and the pixel sets 910 and 920 include R pixels, G pixels, B pixels, and W pixels, but the present invention is not limited thereto. It is not limited or limited. That is, the pixel set 910, 920 may be composed of W, R, B, and W pixels instead of R, G, B, and W pixels.

In this case, any one of the pixels 911, 912, 913, and 914 included in any one pixel set (eg, the first pixel set 910) in the image sensor 900 (eg, the second). On the top of the four pixels 914, an offset mask in which an OPA 914-3 is formed having a center position 914-2 that is offset from the center 914-1 of one pixel (fourth pixel 914). 914-4 may be deployed. In addition, the plurality of pixels 921, 922, 923, and 924 included in any one pixel set (eg, the second pixel set 820) except for the first pixel set 910 in the image sensor 900. ) Also has a center position 924-2 that is offset from the center 924-1 of any one pixel (eg, the fourth pixel 924). The offset mask 924-4 on which the OPA 924-3 is formed may be disposed.

In particular, in the image sensor 900, any one of the plurality of pixels 911, 912, 913, and 914 included in any one pixel set (first pixel set 910) (fourth pixel 914). The OPA 914-3 of the offset mask 914-4 disposed on the upper side of the)) includes a plurality of pixels 921, 922, and the like included in the other one pixel set (the second pixel set 920). A center position 914-deviated from the center 924-2 of the OPA 924-3 of the offset mask 924-4 disposed above any one of the pixels 923 and 924 (the fourth pixel 924). May have 2). For example, if the OPA 914-3 formed in the first pixel set 910 has a center position 914-2 deviated to the right from the center 914-1 of the pixel 914, the second pixel set The OPA 924-3 formed at 920 may have a center position 924-2 deviated to the left from the center 924-1 of the pixel 924.

Accordingly, the offset mask 924-4 is obtained from the first image 1010 and the second pixel set 920 obtained through the pixel 914 having the offset mask 914-4 formed in the first pixel set 910. As shown in FIG. 10, parallax exists between the second image 1020 acquired through the formed pixel 924.

Accordingly, the at least one processor may include at least one pixel having an offset mask 914-4 disposed thereon in one of the plurality of pixel sets 910 and 920 (the first pixel set 910). Any other pixel set except for the first image 1010 obtained through 914 and any one pixel set (first pixel set 910) of the plurality of pixel sets 910 and 920 (second) Distance between the endoscope microscope and the object using the parallax between the second image 1020 obtained through any one pixel 924 in which the offset mask 924-4 is disposed on the pixel set 920. More precisely, the distance between the image sensor 900 and the object) can be calculated.

As such, the endoscope according to another embodiment is a camera system having the structure of the image sensor 900 to which the OPAs (914-3, 924-3) are applied differently for each pixel set (910, 920) under a single optical system. By minimizing the cross-sectional area of the camera system to achieve miniaturization, the distance (depth) between the endoscope microscope and the object is determined by using the parallax between the two images 1010 and 1020 acquired by the image sensor 900. Can offer the calculating technology

FIG. 11 is a diagram for describing a camera system of an endoscope microscope to which a PA (Pixel Aperture) is applied, according to an exemplary embodiment.

Referring to FIG. 11, an image sensor 1100 included in a camera system of an endoscope microscope to which a PA is applied according to an embodiment may include a plurality of pixels 1111, 1112, and 1113 for processing light rays having a plurality of wavelengths for each wavelength. , At least one pixel set 1110 configured as 1114. In the drawing, the image sensor 1100 is illustrated as having nine pixel sets, and the pixel set 1110 includes an R pixel, a G pixel, a B pixel, and a W pixel. It doesn't work. That is, the pixel set 1110 may be composed of W, R, B, and W pixels instead of R, G, B, and W pixels.

In particular, any one of the plurality of pixels 1111, 1112, 1113, and 1114 included in the pixel set 1110 in the image sensor 1100 (eg, the fourth pixel 1114) An offset mask 1114-3 with a PA 1114-2 having a center position 1114-1 that coincides with the center of the pixel of the fourth pixel 1114-1 may be disposed. Hereinafter, the center of the pixel 1114 refers to the center point at the top surface of the pixel 1114, and the center 1114-1 of the PA 1114-2 refers to the center point of the hole of the PA 1114-2.

In addition, hereinafter, the PA 1114-2 may be formed on the offset mask 1114-3 having any one of a circle, an ellipse, a triangle, a square, or a polygon. As such, the PA 1114-2 does not have a shape limitation.

Thus, the offset mask 1114-3 blocks the light rays at the periphery of the bundle of light rays, and injects only the light rays at the center portion to the fourth pixel 1114, and the first image acquired through the fourth pixel 1114 is offset. The mask 1114-3 becomes sharper than the second image obtained through the other pixels 1111, 1112, and 1113 not disposed thereon. That is, the first image obtained through one pixel 1114 having the offset mask 1114-3 disposed thereon and the other pixels 1111, 1112, which do not have the offset mask 1114-3 disposed therein. There is a blur difference between the second images acquired through 1113.

Accordingly, the at least one processor may include the first image and the plurality of pixels 1111, 1112, and 1113 obtained through any one pixel (the fourth pixel 1114) on which the offset mask 1114-3 is disposed. 1114, except for any one pixel (the fourth pixel 1114) in which the offset mask 1114-3 is disposed (eg, the first pixel 1111, the second pixel 1112, or The distance between the endoscope microscope and the object (more precisely, the distance between the image sensor 1100 and the object) using the blurring change between the second images obtained through any one of the third pixels 1113). Can be calculated.

As such, the endoscope microscope according to the embodiment includes a camera system having a structure of the image sensor 1100 to which the PA 1114-2 is applied under a single optical system, thereby minimizing the cross-sectional area of the camera system, A technique for calculating a distance (depth) between an endoscope microscope and an object using a blur change between two images (first image and second image) acquired by the image sensor 1100 may be proposed. Detailed descriptions for calculating depths using blurring variations between images are omitted because they depart from the technical spirit of the present invention.

12 is a diagram for describing a camera system of an endoscope microscope to which a dual aperture (DA) is applied, according to an exemplary embodiment.

Referring to FIG. 12, a single optical system 1200 included in a camera system of an endoscope microscope to which DA is applied, according to an embodiment, introduces a plurality of apertures each having a central position coinciding with each other by introducing light beams having different wavelengths. 1210 and 1220 are formed. For example, in the single optical system 1200, a first aperture 1210 for introducing a light of an RGB wavelength and a second aperture 1220 for introducing a light of an IR wavelength may be formed with a central position coincident with each other. Can be. For example, the first aperture 1210 may be implemented in the form of a filter that blocks the light of the IR wavelength, and the second aperture 1220 may be formed in the form of a hole in the center of the first aperture 1210. have. Hereinafter, although the first aperture 1210 introduces the light of the RGB wavelength and the second aperture 1220 introduces the light of the IR wavelength, the first aperture 1210 is not limited thereto or limited thereto. The second aperture 1220 may be implemented in various ways under the constraint of introducing light beams having different wavelengths.

Accordingly, the light rays of the wavelengths introduced through the first image and the second apertures 1220 obtained by processing the light rays of the wavelengths introduced through the first aperture 1210 are processed by an image sensor (not shown). A blur difference exists between the second images processed and acquired by the image sensor.

Accordingly, the at least one processor may include a plurality of first images and a plurality of first images obtained by processing light rays having a wavelength introduced through one of the plurality of apertures 1210 and 1220 (the first aperture 1210). A light ray having a wavelength introduced through any one of the apertures 1210 and 1220 of the second aperture 1220 except for the first aperture 1210 of the apertures 1210 and 1220. Using the blur change between the second images obtained by processing, the distance between the endoscope microscope and the object (more precisely, the distance between the image sensor 1100 and the object) may be calculated.

As such, the endoscope microscope according to an embodiment includes a camera system in which a plurality of apertures 1210 and 1220 are formed to introduce light rays of different wavelengths, respectively, having a central position coincident with each other on a single optical system. Minimize the cross-sectional area of the camera system and at the same time calculate the distance (depth) between the endoscope and the object by using the blurring change between the two images (first image and second image) acquired by the image sensor You can suggest a technique. Similarly, detailed descriptions for calculating depths using blurring variations between images are omitted because they depart from the technical spirit of the present invention.

13 is a diagram for describing a camera system of an endoscope microscope to which DA is applied, according to another exemplary embodiment.

Referring to FIG. 13, in a single optical system 1300 included in a camera system of an endoscope microscope to which DA is applied, a plurality of apertures having different center positions from each other by introducing light beams having different wavelengths ( 1310, 1320 are formed. For example, in the single optical system 1300, the first aperture 1310 for introducing the light of the RGB wavelength and the second aperture 1320 for introducing the light of the IR wavelength may be formed with the center positions deviated from each other. have. Hereinafter, while the first aperture 1310 introduces the light of the RGB wavelength and the second aperture 1320 introduces the light of the IR wavelength, the first aperture 1310 is formed side by side with the same size, but is limited or limited thereto. Instead, the first aperture 1310 and the second aperture 1320 may be implemented in various ways under the constraint of introducing light rays having different wavelengths and having misaligned center positions. For example, the first aperture 1310 and the second aperture 1320 may be formed so that at least some regions overlap or the second aperture 1320 may be formed on the first aperture 1310.

Accordingly, the light rays of the wavelengths introduced through the first image and the second aperture 1320 obtained by processing the light rays of the wavelengths introduced through the first aperture 1310 are processed by an image sensor (not shown). A parallax exists between the second images processed and acquired by the image sensor.

Accordingly, the at least one processor may include a plurality of first images and a plurality of first images obtained by processing light beams having a wavelength introduced through one of the plurality of apertures 1310 and 1320 (the first aperture 1310). Light rays of a wavelength introduced through any one of the apertures (the second aperture 1320) except for the one of the apertures 1310 and 1320 of the first (the first aperture 1310). The distance between the endoscope microscope and the object (more precisely, the distance between the image sensor 1300 and the object) may be calculated using the parallax between the second images obtained by processing. As described above with reference to FIG. 1, a method of calculating depth by using parallax between images in the plurality of apertures 1310 and 1320 is omitted.

As described above, the endoscope microscope according to another embodiment includes a camera system in which a plurality of apertures 1310 and 1320 are formed to introduce light rays having different wavelengths, respectively, having a center position deviated from each other on a single optical system. Minimizing the cross-sectional area of the camera system and minimizing the cross-sectional area of the camera system, while calculating the distance (depth) between the endoscope microscope and the object using the parallax between two images (first image and second image) acquired by the image sensor You can suggest a technique.

14 is a diagram for describing a camera system of an endoscope microscope to which a subpixel structure is applied, according to an exemplary embodiment.

Referring to FIG. 14, an image sensor 1400 included in a camera system of an endoscope microscope to which a sub-pixel structure is applied according to an embodiment may include a plurality of pixels 1411 and 1412 that process light rays having a plurality of wavelengths for each wavelength. , At least one pixel set 1410 including 1413 and 1414. In the drawing, the image sensor 1400 is illustrated as having nine pixel sets, and the pixel set 1410 includes R pixels, G pixels, B pixels, and W pixels. It doesn't work. That is, the pixel set 1410 may be composed of W, R, B, and W pixels instead of R, G, B, and W pixels.

In particular, any one of the plurality of pixels 1411, 1412, 1413, and 1414 (eg, the fourth pixel 1414) included in the pixel set 1410 in the image sensor 1400 may have a subpixel structure. Can be. More specifically, the fourth pixel 1414 includes a center subpixel 1414-1 through which a light ray at the center of the bundle of light beams is incident and at least one peripheral subpixel 1414-2 through which light rays from the periphery of the bundle of light rays are incident , 1414-3). Hereinafter, the fourth pixel 1414 having the subpixel structure will be described as including two peripheral subpixels 1414-2 and 1414-3, but the present invention is not limited thereto.

Therefore, the first image acquired through the center subpixel 1414-1 is sharper than the second image generated by merging the signals acquired through the peripheral subpixels 1414-2 and 1414-3. That is, a blur difference exists between a first image acquired through the center subpixel 1414-1 and a second image generated by merging a signal obtained through the neighboring subpixels 1414-2 and 1414-3. Done.

Accordingly, the at least one processor may be configured to merge the first image acquired through the center subpixel 1414-1 and the signal obtained through the neighboring subpixels 1414-2 and 1414-3. The blur change between the two may be used to calculate the distance between the endoscope microscope and the object (more precisely, the distance between the image sensor 1400 and the object).

Also, the center of the first peripheral subpixel 1414-2 and the center of the second peripheral subpixel 1414-3 of the two peripheral subpixels 1414-2 and 1414-3 are horizontal, vertical, or diagonal. In a direction displaced, a third image and a second peripheral generated using a signal acquired through the first peripheral subpixel 1414-2 of the two peripheral subpixels 1414-2 and 1414-3. Horizontal, vertical, or diagonal parallaxes exist between the fourth images generated by using signals acquired through the subpixels 1414-3.

Accordingly, the at least one processor may use the parallax between the third image acquired through the first peripheral subpixel 1414-2 and the fourth image acquired through the second peripheral subpixel 1414-3. The distance between the microscope and the object (more precisely, the distance between the image sensor 1400 and the object) can be calculated.

As such, the endoscope according to an embodiment utilizes a camera system including an image sensor 1400 to which a subpixel structure is applied under a single optical system, thereby minimizing the cross-sectional area of the camera system and simultaneously minimizing the cross-sectional area of the camera system. ), A technique for calculating a distance (depth) between an endoscope microscope and an object by using a blur variation or parallax between two images (first image and second image) acquired in FIG.

15 is a flowchart illustrating a method of operating an endoscope according to an embodiment. Hereinafter, the method of operating the endoscope may be performed by the endoscope including the camera system described above with reference to FIGS. 2 to 14 and at least one processor.

Referring to FIG. 15, in operation S1510, an image sensor constituting a camera system together with a single optical system may acquire two images photographing an object located around the endoscope microscope.

In operation S1520, at least one processor may calculate a distance between the endoscope microscope and the object using the two images.

In this case, the image sensor may include at least one pixel set consisting of a plurality of pixels for processing a light ray having a plurality of wavelengths for each wavelength, and according to an embodiment, OPA is applied, PA is applied, The sub pixel structure can be applied. Further, according to the embodiment, DA may be formed in a single optical system constituting the camera system.

Therefore, when OPA is applied to the image sensor, in operation S1510, the image sensor acquires a first image through any one of the plurality of pixels in which an offset mask is disposed, and the offset among the plurality of pixels. The second image may be acquired through any one of the pixels (pixels in which the offset mask is not disposed above) except for any one pixel in which the mask is disposed thereon, and in operation S1520, the at least one processor may include: The distance between the endoscope microscope and the object may be calculated using the parallax between the first image and the second image.

In addition, when the image sensor includes a plurality of pixel sets, the OPA of the offset mask disposed on top of any one pixel in any one of the plurality of pixel sets may be set to any one of the plurality of pixel sets. In any one pixel set except for the pixel set, the pixel may have a center position that is offset from the center of the OPA of the offset mask disposed above the one pixel. Accordingly, in operation S1510, the image sensor acquires a first image through one pixel in which an offset mask is disposed in any one of the plurality of pixel sets, and the remaining one of the plurality of pixel sets. In one pixel set, a second image may be acquired through one pixel having an offset mask disposed thereon, and in operation S1020, the at least one processor may use a parallax between the first image and the second image. The distance between the endoscope microscope and the object can be calculated.

In addition, when the PA is applied to the image sensor, in step S1510, the image sensor acquires a first image through any one of the plurality of pixels in which a mask is disposed, and any of the plurality of pixels. The second image may be acquired through any one pixel except for one pixel, and in operation S1520, the at least one processor may use the blurring between the first image and the second image between the endoscope and the object. The distance of can be calculated.

In addition, when a subpixel structure is applied to the image sensor, in operation S1510, the image sensor acquires a first image through a center subpixel through which a light ray at a center of the pixels of the subpixel structure is incident. A second image may be generated by merging a signal obtained through a peripheral subpixel into which a ray of a periphery of one of the pixels is incident, and in operation S1520, at least one processor may generate a second image between the first image and the second image. The blur variation can be used to calculate the distance between the endoscope microscope and the object. Furthermore, when a plurality of at least one peripheral subpixel is provided, in operation S1510, the image sensor acquires a third image through one of the plurality of peripheral subpixels and the plurality of peripheral subpixels. The fourth image may be acquired through any one of the peripheral subpixels other than the one of the peripheral subpixels among the pixels, and in operation S1520, the at least one processor may generate a fourth image between the third image and the fourth image. The parallax may be used to calculate the distance between the endoscope microscope and the object.

In addition, when the image sensor is formed in a general structure and DA is applied on a single optical system (OPA, PA, or a sub-pixel structure is not applied, a plurality of plural arcs having a central position coinciding with each other by introducing light beams of different wavelengths) are applied. If the apertures are formed on a single optical system), in step S1510 the image sensor acquires a first image based on the light of the wavelength passing through the aperture of any one of the plurality of apertures and the plurality of apertures. Among them, a second image may be acquired based on a light ray having a wavelength passing through any one of the apertures except for the aperture, and in operation S1520, the at least one processor may generate the first image and the second image. The blur variation between the images can be used to calculate the distance between the endoscope microscope and the object.

On the other hand, when the image sensor is formed in a general structure and DA is applied on a single optical system (OPA, PA, or sub-pixel structure is not applied, a plurality of plural arcs having different center positions in which light rays of different wavelengths are introduced but are shifted from each other) If the apertures are formed on a single optical system), in step S1510 the image sensor acquires a first image based on the light of the wavelength passing through the aperture of any one of the plurality of apertures and the plurality of apertures. Among them, a second image may be acquired based on a light ray having a wavelength passing through any one of the apertures except for the aperture, and in operation S1520, the at least one processor may generate the first image and the second image. The parallax between the images can be used to calculate the distance between the endoscope microscope and the object.

Although not shown in the drawings, the method of operating the endoscope further includes performing operations for avoiding collision between the endoscope microscope and the object based on the distance between the endoscope microscope and the object, in addition to steps S1510 to S1520. You may. As such, the operation for avoiding the collision may be to automatically manipulate the endoscope so that the endoscope does not collide with the object by using the distance between the endoscope and the object, but is not limited thereto. In one example, the operation for avoiding the collision may be performed according to an operation input from the user in response to the distance between the endoscope microscope and the object being provided to the user operating the endoscope microscope.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable PLU (programmable). It can be implemented using one or more general purpose or special purpose computers, such as logic units, microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and may configure the processing device to operate as desired, or process independently or collectively. You can command the device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium or device in order to be interpreted by or provided to the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. In this case, the medium may be to continuously store a computer executable program or to temporarily store the program for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a single or several hardware combined, not limited to a medium directly connected to any computer system, it may be distributed on the network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And ROM, RAM, flash memory, and the like, configured to store program instructions. In addition, examples of another medium may include a recording medium or a storage medium managed by an app store that distributes an application, a site that supplies or distributes various software, a server, or the like.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or, even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (24)

In an endoscope microscope using a monocular camera,
A single optical system and an image sensor, the image sensor including at least one pixel set consisting of a plurality of pixels for processing wavelengths of light having a plurality of wavelengths, positioned at the distal end of the endoscope microscope Camera system; And
At least one processor configured to calculate a distance between the endoscope microscope and an object located around the endoscope microscope using at least two images acquired by the image sensor
Including,
On top of any one of a plurality of pixels included in the pixel set of the image sensor,
An offset mask having an Offset Pixel Aperture (OPA) having a center position deviated from the center of the one pixel is disposed,
The at least one processor,
Between an image obtained through one pixel on which the offset mask is disposed and an image obtained through any other pixel except any one of the plurality of pixels on which the offset mask is disposed An endoscope microscope using a monocular camera for calculating the distance between the endoscope microscope and the object using a disparity. delete The method of claim 1,
OPA formed in the offset mask,
An endoscope using a monocular camera having a center position that is offset in the direction of any one of the horizontal, vertical or diagonal with respect to the center of any one pixel disposed above the offset mask. The method of claim 1,
The offset mask is,
An endoscope microscope using a monocular camera, disposed on top of any one of the plurality of pixels having the maximum light transmission characteristics. The method of claim 1,
One pixel in which the offset mask is disposed above,
An endoscope microscope using a monocular camera, wherein the offset mask of the plurality of pixels processes a light beam having a wavelength different from that of the light beam to be processed in any one pixel except for any one pixel disposed above. The method of claim 1,
When the plurality of pixel sets are included in the image sensor, the OPA of the offset mask disposed on any one pixel in any one of the plurality of pixel sets is
An endoscope microscope using a monocular camera having a center position deviating from the center of an OPA of an offset mask disposed above any one pixel in any other pixel set except the one of the plurality of pixel sets . The method of claim 6,
The at least one processor,
An image obtained through one pixel on which the offset mask is disposed in the set of one pixel and an image obtained through any pixel on which the offset mask is disposed in the other set of pixels An endoscope microscope using a monocular camera, to calculate the distance between the endoscope microscope and the object using the parallax between. The method of claim 1,
The OPA,
An endoscope microscope using a monocular camera, formed on the offset mask to have any one of a circle, an ellipse, a triangle, a square or a polygon. delete delete delete In an endoscope microscope using a monocular camera,
A single optical system and an image sensor, the image sensor including at least one pixel set consisting of a plurality of pixels for processing wavelengths of light having a plurality of wavelengths, positioned at the distal end of the endoscope microscope Camera system; And
At least one processor configured to calculate a distance between the endoscope microscope and an object located around the endoscope microscope using at least two images acquired by the image sensor
Including,
Any one of a plurality of pixels included in the pixel set of the image sensor,
A central subpixel into which a light ray in the center of the bundle of light beams is incident, and at least one peripheral subpixel into which a light ray in the periphery of the bundle of light rays is incident;
The at least one processor,
A monocular camera for calculating a distance between the endoscope microscope and the object using a blurring change between an image obtained through the center subpixel and an image obtained by merging a signal obtained through the at least one peripheral subpixel. Endoscopic Microscope. The method of claim 12,
The at least one processor,
When the at least one peripheral subpixel is provided in plural, the distance between the endoscope microscope and the object is calculated using the parallax between the images acquired through the plurality of peripheral subpixels, respectively. Endoscopic Microscope. The method according to any one of claims 1 to 12,
The at least one processor,
And an operation to avoid collision between the endoscope microscope and the object based on the distance between the endoscope microscope and the object. The method according to any one of claims 1 to 12,
The camera system,
An endoscope microscope using a monocular camera, which is located at the tip of the curved portion that can be bent or curved freely in the endoscope microscope. In the operation method of the endoscope microscope using a monocular camera,
An image sensor constituting a camera system located at the distal end of the endoscope microscope with a single optical system, the image sensor including at least one pixel set consisting of a plurality of pixels for processing a light beam having a plurality of wavelengths for each wavelength Obtaining at least two images of an object located around the endoscope microscope; And
Calculating, at at least one processor, the distance between the endoscope microscope and the object using the two images
Including,
On top of any one of a plurality of pixels included in the pixel set of the image sensor,
An offset mask having an Offset Pixel Aperture (OPA) having a center position deviated from the center of the one pixel is disposed,
Acquiring the two images,
Acquiring a first image through any one pixel on which the offset mask is disposed; And
Acquiring a second image through any one of the plurality of pixels except for any one pixel in which the offset mask is disposed.
Including;
Calculating the distance between the endoscope microscope and the object,
Calculating a distance between the endoscope microscope and the object by using a disparity between the first image and the second image
Operation method of the endoscope microscope using a monocular camera comprising a. delete The method of claim 16,
When the plurality of pixel sets are included in the image sensor, the OPA of the offset mask disposed on any one pixel in any one of the plurality of pixel sets is
Has a center position deviated from the center of the OPA of the offset mask disposed on top of any one pixel in any one of the plurality of pixel sets except for any one of the pixel sets;
Acquiring the two images,
Acquiring a first image through any one pixel in which said offset mask is disposed on said one pixel set; And
Acquiring a second image through any one pixel in which the offset mask is disposed on the remaining one pixel set;
Including,
Calculating the distance between the endoscope microscope and the object,
Calculating a distance between the endoscope microscope and the object by using a parallax between the first image and the second image
Operation method of the endoscope microscope using a monocular camera comprising a. delete delete delete In the operation method of the endoscope microscope using a monocular camera,
An image sensor constituting a camera system located at the distal end of the endoscope microscope with a single optical system, the image sensor including at least one pixel set consisting of a plurality of pixels for processing a light beam having a plurality of wavelengths for each wavelength Obtaining at least two images of an object located around the endoscope microscope; And
Calculating, at at least one processor, the distance between the endoscope microscope and the object using the two images
Including,
Any one of a plurality of pixels included in the pixel set of the image sensor,
A central subpixel into which a light ray in the center of the bundle of light beams is incident, and at least one peripheral subpixel into which a light ray in the periphery of the bundle of light rays is incident;
Acquiring the two images,
Acquiring a first image through the center subpixel; And
Generating a second image by merging a signal acquired through the at least one peripheral subpixel
Including;
Calculating the distance between the endoscope microscope and the object,
Calculating a distance between the endoscope microscope and the object using a blur change between the first image and the second image
Operation method of the endoscope microscope using a monocular camera comprising a. The method of claim 22,
Generating the second image,
Acquiring a third image through any one of the plurality of peripheral subpixels when the at least one peripheral subpixel is provided; And
Acquiring a fourth image through any one of the plurality of neighboring subpixels except for one neighboring subpixel;
Including,
Calculating the distance between the endoscope microscope and the object,
Calculating a distance between the endoscope microscope and the object by using a parallax between the third image and the fourth image
Operation method of the endoscope microscope using a monocular camera comprising a. The method according to any one of claims 16 to 22,
Performing an operation for avoiding collision between the endoscope microscope and the object based on a distance between the endoscope microscope and the object
Operation method of the endoscope microscope using a monocular camera further comprising.

Images