JP2013228798A - Image processing device, imaging device, endoscope, program and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, an imaging device and the like that can perform image synthetic processing to even an image signal of low contrast by suppressing occurence of an artifact.SOLUTION: An image processing device 100 is configured to comprise: an image acquisition unit 110 that performs acquisition processing of a plurality of image signals; an evaluation value calculation unit 120 that performs calculation processing of an evaluation value on the basis of an acquired image signal; and an image synthetic unit 130 that performs synthetic processing of the plurality of image signals on the basis of the evaluation value. The image acquisition unit 110 acquires a plurality of image signals different in an in-focus object position, and acquires a plurality of pattern image signals different in an in-focus object position photographed by projecting a geometric pattern onto a subject. The evaluation value calculation unit 120 calculates a first evaluation value on the basis of the plurality of pattern image signals, and calculates a second evaluation value on the basis of the plurality of image signals. The image synthetic unit 130 performs comparison processing of the first evaluation value with the second evaluation value, and performs synthetic processing of the plurality of image signals on the basis of a result of the comparison processing.

Description

  The present invention relates to an image processing device, an imaging device, an endoscope, a program, an image processing method, and the like.

  In the case where two subjects located at distant positions are imaged using the imaging device, only one subject may be in focus and the other subject may not be in focus. For example, in a scene where an endoscope is used to discover or observe an affected area, if a portion that is not in focus is reflected, the possibility that a doctor misses the affected area increases. For this reason, in such a case, it is desirable that all subjects appearing in the captured image are in focus.

  At this time, there are cases where both subjects can be brought into focus, for example, by adjusting the aperture of the imaging device to adjust the depth of field. However, depending on the distance between the two subjects, it may not be possible to bring both subjects into focus by simply adjusting the depth of field.

  Even in such a case, for example, a technique disclosed in Patent Document 1 is disclosed as a technique for acquiring an omnifocal image. In the method disclosed in Patent Document 1, the sharpness is calculated for each pixel of a plurality of image signals, and the first reference value obtained from the sharpness is spatially smoothed to calculate the second reference value. Then, an image synthesis process is performed based on the calculated second reference value to obtain an omnifocal image.

JP 2010-20758 A

  In the prior art disclosed in Patent Document 1 described above, the sharpness is calculated from a plurality of image signals, and a composite image is generated based on a reference value obtained from the sharpness. For this reason, in a region with a low texture in the image signal, that is, a subject with low contrast, the sharpness and the reference value for generating a composite image obtained from the sharpness cannot be calculated stably, and artifacts occur in the composite image. There is a case.

  According to some aspects of the present invention, an image processing device, an imaging device, an endoscope, a program, an image processing method, and the like that can perform image composition processing while suppressing the occurrence of artifacts even with low-contrast image signals Can be provided.

  One aspect of the present invention is an image acquisition unit that performs acquisition processing of a plurality of image signals, an evaluation value calculation unit that performs evaluation value calculation processing based on the acquired image signals, and the plurality of the plurality of image signals based on the evaluation values. An image synthesizing unit that performs image signal synthesizing processing, wherein the image acquisition unit acquires the plurality of image signals having different in-focus object positions from the imaging unit, and projects a geometric pattern on the subject to perform imaging The plurality of pattern image signals having different in-focus object positions are acquired from the imaging unit, and the evaluation value calculation unit calculates a first evaluation value based on the plurality of pattern image signals, and A second evaluation value is calculated based on an image signal, and the image composition unit performs a comparison process between the first evaluation value and the second evaluation value, and based on a result of the comparison process, the plurality of images Perform the above signal synthesis process It is related to the image processing apparatus according to claim.

  In one aspect of the present invention, the first evaluation value is calculated from the signal of the image captured by projecting the geometric pattern, and the second evaluation value is calculated from the signal of the image captured without projecting the geometric pattern. Then, image composition processing is performed based on the calculated first evaluation value and second evaluation value.

  As a result, even with an image signal having a low contrast, image synthesis processing can be performed while suppressing the occurrence of artifacts.

  In the aspect of the invention, the evaluation value calculation unit may calculate the first evaluation value for each synthesis target pixel that is a pixel that is a target of the synthesis process in the plurality of pattern image signals, and The second evaluation value is calculated for each of the synthesis target pixels of the image signal, and the image synthesis unit performs a comparison process between the first evaluation value and the second evaluation value for each of the synthesis target pixels, Based on the comparison processing result, the synthesis processing of the plurality of image signals may be performed for each synthesis target pixel.

  As a result, the first evaluation value and the second evaluation value can be calculated for the pixel to be image-combined, and comparison processing is performed on all the pixels to be combined, for example, focusing of the image It is possible to determine the state or the like and perform more accurate image composition processing.

  In the aspect of the invention, the evaluation value calculation unit may provide the calculated first evaluation value for each combination target pixel that is a target of the combination process in the plurality of pattern image signals. If it is determined whether or not the threshold value is equal to or less than a threshold value, and it is determined that the first evaluation value is equal to or less than the given threshold value, an interpolation process may be performed on the first evaluation value.

  As a result, for example, even when the dot pattern is sparse, it is possible to obtain a first evaluation value that is effective as an evaluation value for image synthesis processing for all the synthesis target pixels.

  In one aspect of the present invention, when the evaluation value calculation unit determines that the first evaluation value is equal to or less than the given threshold, the interpolation process includes a zero-order interpolation process and a linear interpolation process, Any one of the morphological expansion processes may be performed on the first evaluation value.

  Thereby, it is possible to interpolate the first evaluation value determined as not being effective as the evaluation value of the image composition processing based on the other first evaluation value calculated from the same pattern image signal.

  In the aspect of the invention, the image acquisition unit acquires the plurality of pattern image signals from the imaging unit, and projects a geometric pattern different from the previously projected geometric pattern onto the subject. The plurality of complementary pattern image signals are acquired from the imaging unit, and the evaluation value calculation unit determines that the first evaluation value calculated from the plurality of pattern image signals is equal to or less than the given threshold value. Processing for recalculating the first evaluation value based on the plurality of complementary pattern images may be performed as the interpolation processing on the synthesis target pixel.

  As a result, it is possible to replace the first evaluation value determined not to be effective as the evaluation value of the image composition processing with the first evaluation value corresponding to the same composition target pixel calculated from the complementary pattern image signal. Become.

  In the aspect of the invention, the image synthesis unit may calculate the second evaluation value as a given threshold value for each synthesis target pixel that is a target of the synthesis process in the plurality of pattern image signals. If the second evaluation value is determined to be equal to or less than the given threshold value, the combining process is performed based on the first evaluation value, and the second evaluation value is determined based on the first evaluation value. When it is determined that the evaluation value is larger than the given threshold value, the combining process may be performed based on the second evaluation value.

  Thereby, it is possible to select an effective evaluation value from the first evaluation value and the second evaluation value and perform image composition processing.

  In one aspect of the present invention, when the image synthesis unit determines that the second evaluation value is equal to or less than the given threshold value, the image synthesis unit calculates an image synthesis coefficient based on the first evaluation value. , Performing the synthesis process for each of the synthesis target pixels using the image synthesis coefficient, and determining that the second evaluation value is greater than the given threshold value, based on the second evaluation value, An image synthesis coefficient may be calculated, and the synthesis process may be performed for each synthesis target pixel using the image synthesis coefficient.

  Thereby, for example, it is possible to smoothly synthesize a switching portion between the in-focus area and the out-of-focus area.

  In the aspect of the invention, the image acquisition unit acquires the first image signal and the first image signal and the second image signal having a different focused object position from the imaging unit. Acquiring a first pattern image signal captured by projecting a geometric pattern onto a subject, a first pattern image signal and a second pattern image signal having a different focused object position from the imaging unit; The evaluation value calculation unit calculates the first evaluation value for the first pattern image signal and the second pattern image signal for each synthesis target pixel that is a pixel that is a target of the synthesis process, and The second evaluation value for the first image signal and the second image signal is calculated, and the image synthesis unit is configured to calculate the first image signal and the second image signal for each synthesis target pixel. The second evaluation value for If it is determined whether or not both of the second evaluation values are equal to or less than the given threshold, the first evaluation value is determined based on the first evaluation value. When the image signal and the second image signal are combined, and when it is determined that both the second evaluation values are larger than the given threshold, the first evaluation value is based on the second evaluation value. The combining process of the image signal and the second image signal may be performed.

  As a result, even when two image signals and two pattern image signals are acquired, it is possible to select an effective evaluation value from the first evaluation value and the second evaluation value and perform image composition processing, etc. become.

  In the aspect of the invention, the evaluation value calculation unit may calculate a luminance value of the plurality of pattern image signals as the first evaluation value.

  As a result, for example, even when effective contrast or the like cannot be calculated in the image composition process, it is possible to calculate other evaluation values used for the image composition process.

  In the aspect of the invention, the evaluation value calculation unit may calculate the second evaluation value representing a contrast or an edge component of the plurality of image signals.

  This makes it possible to perform image composition processing based on the contrast or edge component of the image.

  In the aspect of the invention, the image acquisition unit may acquire the plurality of pattern image signals captured by projecting the geometric pattern set based on an optical degradation function from the imaging unit. .

  This makes it possible to evaluate the in-focus state of each pixel position on which the geometric pattern is projected based on the luminance value of the pattern image.

  In one aspect of the present invention, the image acquisition unit is configured such that the inter-dot distance d is d ≧ 2R with respect to the blur radius R based on the optical degradation function (d and R are positive integers). The plurality of pattern image signals captured by projecting the set dot pattern may be acquired.

  This makes it possible to project a dot pattern onto the subject so that the dots do not overlap.

  In one aspect of the present invention, the image acquisition unit acquires, from the imaging unit, the plurality of pattern image signals captured by projecting the geometric pattern using a coherent light source and a spatial light modulator. Also good.

  This makes it possible to acquire a pattern image signal in which a clear arbitrary geometric pattern is reflected.

  In one aspect of the present invention, the image acquisition unit outputs the plurality of pattern image signals captured by projecting the geometric pattern using the coherent light source having an invisible wavelength and the spatial light modulator. You may acquire from an imaging part.

  This makes it possible to acquire the pattern image signal of the near point pattern image captured at the same timing as the near point image and the pattern image signal of the far point pattern image captured at the same timing as the far point image. Become.

  Another aspect of the invention relates to an imaging device including the image processing device, the imaging unit, and a pattern projection unit that projects the geometric pattern onto the subject.

  Another aspect of the present invention relates to an endoscope including the image processing device, the imaging unit, and a pattern projection unit that projects the geometric pattern onto the subject.

  Moreover, the other aspect of this invention is related with the program which functions a computer as said each part.

  According to another aspect of the present invention, a plurality of image signals with different in-focus object positions are acquired from an imaging unit, and a plurality of pattern images with different in-focus object positions are captured by projecting a geometric pattern onto a subject. A signal is acquired from the imaging unit, a first evaluation value is calculated based on the plurality of pattern image signals, a second evaluation value is calculated based on the plurality of image signals, and the first evaluation value and the The present invention relates to an image processing method characterized in that a comparison process with a second evaluation value is performed, and a synthesis process of the plurality of image signals is performed based on a result of the comparison process.

The system configuration example of this embodiment. 2A to 2C are explanatory diagrams of image signals. FIG. 3A to FIG. 3C are explanatory diagrams of pattern image signals. FIG. 4A to FIG. 4C are explanatory diagrams of geometric patterns. FIG. 5A to FIG. 5C are explanatory diagrams of image composition processing. An example of an image composition process result. The flowchart explaining the flow of a process of this embodiment. The flowchart explaining the flow of an interpolation process in detail. The flowchart explaining the flow of an image synthesis process in detail.

  Hereinafter, this embodiment will be described. First, an overview of the present embodiment will be described, and then a system configuration example will be described. Details of processing performed in this embodiment will be described in detail with reference to flowcharts and the like. Finally, the technique of this embodiment will be summarized. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline When an image pickup apparatus is used to image two subjects located at distant positions, only one subject may be in focus and the other subject may not be in focus. For example, in a scene where an endoscope is used to discover or observe an affected area, if a portion that is not in focus is reflected, the possibility that a doctor misses the affected area increases. For this reason, in such a case, it is desirable that all subjects appearing in the captured image are in focus.

  At this time, there are cases where both subjects can be brought into focus, for example, by adjusting the aperture of the imaging device to adjust the depth of field. However, depending on the distance between the two subjects, it may not be possible to bring both subjects into focus by simply adjusting the depth of field. Specifically, if the distance between two subjects is equal to or smaller than a predetermined threshold, it can be dealt with only by adjusting the depth of field, but if the distance is larger than the predetermined threshold, Just adjusting the depth of field is not enough.

  Even in such a case, as a technique for acquiring an omnifocal image, for example, the technique disclosed in Patent Document 1 is disclosed. In the method disclosed in Patent Document 1, the sharpness is calculated for each pixel of a plurality of image signals, and the first reference value obtained from the sharpness is spatially smoothed to calculate the second reference value. Then, an image synthesis process is performed based on the calculated second reference value to obtain an omnifocal image.

  However, in the conventional technique disclosed in Patent Document 1 described above, the sharpness, the first reference value, and the second reference value can be stably calculated in a region where the texture in the image signal is small, that is, an object with low contrast. In some cases, artifacts may occur in the composite image. Here, the artifact refers to data distortion, noise, and the like generated at the generation (synthesis) stage of the image signal.

  Therefore, the image processing apparatus or the like according to the present embodiment projects a geometric pattern onto a subject, and calculates an evaluation value for generating a composite image based on the luminance value of the pattern-projected image (pattern image). In other words, image composition processing is performed using a pattern image in which a geometric pattern is projected onto a subject and a luminance change is intentionally generated. By performing the processing as described above, it is possible to calculate an evaluation value for stably generating a composite image even in an area with less texture, that is, a low-contrast subject, and it is possible to generate a higher-quality composite image. .

  More specifically, for example, when the present embodiment is applied to a scene where the affected part is found or observed using the endoscope described above, an omnifocal image can be generated as a composite image. According to this, it is possible to generate an image that makes it easier for a doctor or the like to observe the inside of the body, and it is possible to prevent oversight of the affected area.

2. System Configuration Example Next, FIG. 1 shows a configuration example of an image processing apparatus 100 of the present embodiment and an imaging apparatus (endoscope) 400 including the same.

  The image processing apparatus 100 includes an image acquisition unit 110, an evaluation value calculation unit 120, and an image composition unit 130. Examples of the imaging apparatus 400 including the image processing apparatus 100 include an imaging unit 200, an endoscope 400 including a pattern projection unit 300, and the like. Note that the image processing apparatus 100 and the imaging apparatus (endoscope) 400 including the image processing apparatus 100 are not limited to the configuration in FIG. 1, and some of these components are omitted or other components are added. Various modifications such as these are possible.

  Next, the connection of each part is demonstrated. The image acquisition unit 110 is connected to the evaluation value calculation unit 120 and the image composition unit 130. The evaluation value calculation unit 120 is connected to the image composition unit 130. The imaging unit 200 is connected to the image acquisition unit 110.

  Next, the outline and configuration of processing performed in each unit will be described.

  First, the image acquisition unit 110 performs processing for acquiring a plurality of image signals from the imaging unit 200. The image acquisition unit 110 may acquire an analog signal as an image signal, or may acquire a digital signal. The image acquisition unit 110 may be an I / F unit (communication unit) with the imaging unit 200. The I / F unit (communication unit) may perform communication using a wired or internal bus, or may perform communication wirelessly.

  Next, the evaluation value calculation unit 120 performs evaluation value calculation processing based on the acquired image signal.

  Then, the image composition unit 130 performs composition processing of a plurality of image signals based on the evaluation value.

  The functions of the evaluation value calculation unit 120 and the image composition unit 130 can be realized by hardware such as various processors (CPU or the like), ASIC (gate array or the like), a program, or the like. In addition to the above-described components, the image processing apparatus 100 may include a buffer, a storage unit, an output unit (presentation unit), and the like (not shown).

  The imaging unit 200 includes an optical system (not shown), an aperture, an imaging device, a focused object position control unit, and the like. In the imaging unit 200, the focused object position control unit controls the optical system, the diaphragm, and the imaging device, and captures a plurality of images with different focused object positions. Furthermore, the imaging unit 200 may include a preprocessing unit (not shown) and the like. The preprocessing unit may perform processing for converting a captured image into a digital signal.

  Furthermore, the pattern projection unit 300 projects a geometric pattern (pattern light) onto the subject. The pattern projection unit 300 may include a coherent light source and a spatial light modulator (not shown). The coherent light source is, for example, a laser beam, and has a high coherence. When the coherent light source passes through the diffraction grating, a clear pattern based on the diffraction grating is formed. The spatial light modulator is, for example, an LCD (Liquid Crystal Display), and can form an arbitrary geometric pattern that acts as a diffraction grating instantly. By configuring the pattern projection unit 300 as described above, a clear arbitrary geometric pattern can be projected instantaneously. In addition, it is good also as a structure which uses a non-visible light wavelength for a coherent light source, and adds the light-receiving part for non-visible light to an image pick-up element.

3. Details of Processing Hereinafter, the processing flow of the present embodiment will be described with reference to the flowchart of FIG.

  First, the image acquisition unit 110 acquires a plurality of image signals from the imaging unit 200 (S101). In the present embodiment, two image signals are acquired by the image acquisition unit 110, but an arbitrary number of three or more image signals may be acquired. It is desirable to use a telecentric optical system for the optical system or adjust the angle of view by performing a geometric transformation process or the like so that the angle of view of these image signals does not change. The same applies to the pattern image signal in step S102 described later.

  In step S101, the image acquisition unit 110 acquires image signals having different in-focus object positions. Specifically, the image acquisition unit 110 acquires an image captured by the imaging unit 200 as illustrated in FIG.

  First, as a premise, the subject SB1 and the subject SB2 are at different positions as shown in FIG. At this time, the imaging unit 200 controls the optical system, the diaphragm, and the imaging device by the focused object position control unit, and aligns the focused object position with the subject SB1, and displays an image as shown in FIG. Further, an image as shown in FIG. 2C is acquired by aligning the focused object position with the subject SB2.

  Hereinafter, the image IM1 in FIG. 2B is referred to as a near point image (first image) IM1, and the image IM2 in FIG. 2C is referred to as a far point image (second image) IM2. The near-point image (first image) IM1 is when the focused object position is adjusted to a position closer to the imaging unit 200 (position of the subject SB1) than when the image IM2 in FIG. 2C is captured. This is a captured image. On the other hand, the far-point image (second image) IM2 has its focused object position aligned with a position (the position of the subject SB2) farther from the imaging unit 200 than when the image IM1 in FIG. This is an image taken at the time of shooting.

  Further, as shown in FIGS. 2B and 2C, in the near-point image IM1 and the far-point image IM2, the in-focus area and the out-of-focus area are limited by the depth of field by the optical system. Is present. In the in-focus area, the letter “A” written on the subject SB1 and the subject SB2 is clearly projected (the right-side area in FIG. 2B), and in the out-of-focus area, the letter “A” is projected in a blurred manner. (The left region in FIG. 2B). Originally, it is desirable to be able to acquire an omnifocal image in which both of the characters “A” written on the subject SB1 and the subject SB2 are clearly projected, but in this example, due to the limitation of the depth of field described above, An omnifocal image cannot be acquired simply by adjusting the depth of field and the position of the focused object. Therefore, an omnifocal image is generated by image synthesis processing by the following method.

  Next, the image acquisition unit 110 acquires a plurality of pattern image signals having different in-focus object positions (S102). Specifically, the image acquisition unit 110 acquires an image captured by the imaging unit 200 as illustrated in FIG.

  The positions of the subject SB1 and the subject SB2 in FIG. 3A are the same as those in FIG. 2A, but when the pattern projection unit 300 projects a geometric pattern onto the subject SB1 and the subject SB2, the imaging unit 200 2 differs from FIG. 2A in that it captures the subject. As a result, the captured pattern image is as shown in FIG. 3 (B) and FIG. 3 (C). Originally, in step S102, it is desirable to be able to acquire a pattern image showing the same subject, that is, a pattern image showing the same scene, at the same timing as when the image was captured in step S101. Therefore, it is desirable to perform the process of step S102 as soon as possible after the process of step S101.

  In the following, the image IM3 in FIG. 3B is referred to as a near point pattern image (first pattern image) IM3, and the image IM4 in FIG. 3C is referred to as a far point pattern image (second pattern image) IM4. . In the near point pattern image (first pattern image) IM3, the focused object position is adjusted to a position closer to the imaging unit 200 (position of the subject SB1) than when the image IM4 in FIG. 3C is captured. This is an image taken at the time of shooting. On the other hand, the far-point pattern image (second pattern image) IM4 has a focused object position at a position farther from the imaging unit 200 (position of the subject SB2) than when the image IM3 in FIG. It is the image imaged when it was put together.

  Further, as shown in FIG. 3B and FIG. 3C, the near-point pattern image IM3 and the far-point pattern image IM4 are not focused on the in-focus area as in the near-point image and far-point image. A focal area exists. In the in-focus area, the projected geometric pattern is clearly shown (the right area in FIG. 3B), and in the out-of-focus area, the projected geometric pattern is shown in a blurred manner (FIG. 3B ) In the left area). Therefore, whether or not each pixel position on which the pattern is projected is within the depth of field, that is, the focused state of each pixel position can be evaluated (determined) based on the luminance value of the pattern image. Actually, in the portion where the geometric pattern is projected onto the character “A” described in the subject SB1 and the subject SB2, a part of the character “A” is also displayed, but FIG. 3B and FIG. ) Is omitted for the sake of illustration.

  In addition, in a scene where the above pattern image is specifically captured by an endoscope, there is no influence of external light. Therefore, when only a geometric pattern is projected at the time of imaging, a clear pattern image signal is obtained. In many cases, an imaging scene with an endoscope includes a low-contrast subject and an area with little texture, and imaging is often performed in the dark. Therefore, the projected geometric pattern can be obtained clearly, and the method of this embodiment is a particularly effective scene.

  Here, a dot pattern PIM is shown in FIG. 4A as an example of a geometric pattern to be projected. In the above-described examples of FIGS. 2A to 2C and FIGS. 3A to 3C, the dot pattern is projected. The projected geometric pattern is not limited to the dot pattern, but can be any geometric pattern such as a lattice pattern, a line pattern PIM2 as shown in FIG. 4B, a random pattern PIM3 as shown in FIG. A pattern may be projected. The method for determining the geometric pattern will be described later.

  Next, the evaluation value calculation unit 120 calculates a first evaluation value for each synthesis target pixel based on the near point pattern image signal and the far point pattern image signal (S103). Specifically, the first evaluation value is a luminance value of the near point pattern image signal and the far point pattern image signal.

  Similarly, the evaluation value calculation unit 120 calculates a second evaluation value for each synthesis target pixel based on the near point image signal and the far point image signal (S104). Specifically, the second evaluation value is the contrast or edge component of the image signal.

  For example, when contrast is used as the second evaluation value, calculation is performed using Expression (1) within a predetermined region centered on the pixel position to be calculated.

C is the contrast, I _max is the maximum luminance value (pixel value) in the predetermined area, and I _min is the minimum luminance value (pixel value) in the predetermined area. When an edge component is used as the second evaluation value, an edge component by an existing edge extraction filter such as a Sobel filter or a Laplacian filter is calculated.

  Here, the first evaluation value, which is the luminance value of each pixel of the near point pattern image and the far point pattern image, is higher in the focused area of each image than the unfocused area according to the optical degradation function. The in-focus area has a lower value than the in-focus area. Further, since the projected geometric pattern is sparse, the near-point pattern image and the far-point pattern image are also sparse, and pixels that are not affected by the geometric pattern (for example, the black portion of the near-point pattern image IM3 in FIG. 3B). There is a pixel P1).

  In a pixel that is not affected by such a geometric pattern, the first evaluation value becomes (almost) the same in the near point pattern image and the far point pattern image, and even if the calculated first evaluation value is used, It cannot be determined whether the pixel is in focus or out of focus.

A specific example will be described. For example, the black portion pixel P1 and the white portion pixel P3 of the near point pattern image IM3 in FIG. 3B are both out of focus. On the other hand, in the black portion of the far point pattern image IM4 in FIG. 3C, the pixel P2 at the position corresponding to the pixel P1 and the pixel P4 at the position corresponding to the pixel P3 are both in focus. At this time, when the first evaluation value v _P3 obtained from the pixel P3 is compared with the first evaluation value v _P4 obtained from the pixel P4, v _P3 ≦ Th ≦ v _P4 (Th is a predetermined threshold), It can be determined that the pixel P4 is in focus and the pixel P3 is out of focus. However, the first evaluation value v _P1 obtained from the pixel _P1 and the first evaluation value v _P2 obtained from the pixel P2 become (almost) the same value, and even if these values are compared, the pixel P1 It is impossible to determine whether the pixel P2 is in focus or not in focus.

  Therefore, for pixels that are not affected by the geometric pattern, it is necessary to recalculate the first evaluation value (interpolate the first evaluation value) for each synthesis target pixel (S105). In the case of imaging in the dark like an endoscope, pixels that are not affected by the geometric pattern have very low luminance, and therefore can be identified by providing a given threshold value for the first evaluation value. .

  Here, the detailed processing flow of the interpolation processing of the first evaluation value is shown in the flowchart of FIG. Since the process of FIG. 8 shows the flow of the interpolation process performed on a certain synthesis target pixel, it is necessary to perform the process of FIG. 8 on all synthesis target pixels in step S105 of FIG. There is.

  First, it is determined whether or not the first evaluation value for the near point pattern image signal (first pattern image signal) is equal to or less than a given threshold value (S201).

  If it is determined that the first evaluation value for the near point pattern image signal is greater than the given threshold value, the process ends without performing the interpolation process.

  On the other hand, when it is determined that the first evaluation value for the near point pattern image signal is equal to or less than the given threshold value, the first evaluation value for the far point pattern image signal (second pattern image signal) is given. It is determined whether it is below the threshold value (S202). In the present embodiment, the given threshold value used in step S202 is the same value as the threshold value used in step S201, but a different value may be used.

  If it is determined in step S202 that the first evaluation value for the far-point pattern image signal is greater than the given threshold value, the process ends without performing the interpolation process.

  On the other hand, when it is determined that the first evaluation value for the near point pattern image signal (first pattern image signal) is equal to or less than a given threshold value, interpolation processing of the first evaluation value is executed (S203). Here, for example, a value obtained by performing interpolation processing from a pixel that is equal to or higher than a peripheral threshold by existing zero-order interpolation, linear interpolation, morphological expansion processing, or the like is set as a new first evaluation value. Alternatively, as will be described later, interpolation processing may be performed based on a complementary pattern image obtained by changing the position of the geometric pattern or the like. Thus, the first evaluation value interpolation process is completed.

  Then, the image composition unit 130 performs image composition processing for each composition target pixel (S106). When both the second evaluation values obtained from the near point image and the far point image are equal to or less than a predetermined threshold, the image is based on the first evaluation value because the contrast and edge components cannot be stably calculated. Perform synthesis processing. On the other hand, when the second evaluation values obtained from the near point image and the far point image are both larger than a predetermined threshold, the contrast and edge components are pixels that can be stably calculated. Based on this, image composition processing is performed.

  Here, the detailed processing flow of the image composition processing is shown in the flowchart of FIG. Since the process of FIG. 9 shows the flow of the image composition process performed for a certain compositing target pixel, in step S106 of FIG. 7, the process of FIG. 9 is performed for all compositing target pixels. There is a need.

  First, it is determined whether or not the second evaluation value for the near point image signal (first image signal) is equal to or less than a given threshold (S301).

  If it is determined that the second evaluation value for the near point image signal (first image signal) is equal to or less than a given threshold, the second evaluation value for the far point image signal (second image signal) is also It is determined whether it is below a given threshold (S302). In the present embodiment, the given threshold value used in step S302 is the same value as the threshold value used in step S301, but a different value may be used.

  And when it determines with the 2nd evaluation value about a far point image signal (2nd image signal) being below a given threshold value, an image synthesis coefficient is calculated using a 1st evaluation value (S303). .

Here, the image synthesis coefficient c (x, y) is obtained by the following equation (2), for example. In the following formula (2), v _near is a first evaluation value obtained from the _near point pattern image, and v _far is a first evaluation value obtained from the _far point pattern image.

  In addition, the image synthesis coefficient c (x, y) may be obtained by the following equation (3), for example. In the following formula (3), α is a normalization coefficient.

  According to this, instead of binary determination of 0 and 1 (either the near-point image or the far-point image), a coefficient for blending the near-point image and the far-point image is calculated as an image synthesis coefficient, and the in-focus area It is possible to smooth the part where the out-of-focus area is switched.

  On the other hand, when it is determined in step S301 that the second evaluation value for the near point image signal (first image signal) is larger than a given threshold value, or in step S302, the far point image signal (second image signal). If it is determined that the second evaluation value for the signal) is greater than a given threshold value, an image synthesis coefficient is calculated using the second evaluation value (S304).

Also in step S304, the image synthesis coefficient is obtained by the above equation (2), the above equation (3), and the like. However, in the case of obtaining the image synthesis coefficient using the second evaluation value, v _near is replaced with the second evaluation value obtained from the _near- point image in the above expressions (2) and (3), and v _far Is replaced with the second evaluation value obtained from the far point image.

Then, after step S303 or step S304, image composition processing is performed using the calculated image composition coefficient (S305). Specifically, the image composition processing is performed according to the following equation (4). In Expression (4), a (x, y) is a pixel value of coordinates (x, y) of the composite image, and a ₁ (x, y) is a pixel of coordinates (x, y) of the near-point image. The value a ₂ (x, y) represents the pixel value of the coordinates (x, y) of the far point image.

  Here, a specific example of the image composition processing will be described with reference to FIGS. First, an image SIM1 in FIG. 5A is an enlarged view of a part of the near point image in FIG. 2A, and an image SIM2 in FIG. 5B is an image of the far point image in FIG. It is the figure which expanded a part. Then, the image SCIM in FIG. 5C is a result of combining the image SIM1 in FIG. 5A and the image SIM2 in FIG. 5B in accordance with the image synthesis coefficient obtained by the equation (2) and the equation (4). It is an obtained image. 5A to 5C, A1 to A16 and B1 to B16 each represent a synthesis target pixel. Furthermore, the synthesis target pixels (specifically, A1 to A7, A9, A10, A13, B8, B11, B12, and B14 to B16) that are hatched are in an out-of-focus state, and other hatched lines It is assumed that the compositing target pixel not covered by is in a focused state. In actuality, there may be a compositing target pixel in which one part is in an in-focus state and the other part is in an out-of-focus state. Or it demonstrates as what is in a non-focus state.

First, focus on the pixel at the coordinates (1, 1) of the composite image SCIM. As described above, in order to obtain the pixel at the coordinates (1, 1) of the composite image SCIM, the pixel A1 at the same position in the near-point image SIM1 is compared with the pixel B1 at the same position in the far-point image SIM2. Decide which pixel to use. At this time, the first evaluation value (or second evaluation value) v _near obtained from the pixel A1 of the image SIM1 is obtained from the first evaluation value (or second evaluation value) v _far obtained from the pixel B1 of the image SIM2. Becomes smaller. Therefore, since the image synthesis coefficient c (1, 1) = 0 is obtained from the equation (2), a (1, 1) = a ₂ (1, 1) is obtained from the equation (4). Therefore, the pixel B1 of the image SIM2 is selected as the pixel at the coordinates (1, 1) of the composite image SCIM. When similar processing is performed for other pixels, a composite image SCIM as shown in FIG. 5C is obtained.

  As described above, the image compositing process is finished, these processes are executed for all compositing target pixels, and the process of the present embodiment is finished.

  Then, the near point image IM1 shown in FIG. 2B and the far point image IM2 shown in FIG. 2C are synthesized according to the equation (4) using the first evaluation value and the second evaluation value described above. The resulting image AFIM is shown in FIG. As described with reference to FIGS. 5A to 5C, when the in-focus areas of the near-point image IM1 and the far-point image IM2 are synthesized, an image AFIM that is focused on the entire image is obtained. It is done.

  Thereafter, the composite image AFIM may be transferred to an output unit (presentation unit) (not shown) and output to a display or the like.

4). Method of this Embodiment The image processing apparatus 100 according to this embodiment described above includes an image acquisition unit 110 that performs acquisition processing of a plurality of image signals, and an evaluation value calculation unit that performs evaluation value calculation processing based on the acquired image signals. 120 and an image composition unit 130 that performs composition processing of a plurality of image signals based on the evaluation value. First, the image acquisition unit 110 acquires a plurality of image signals having different in-focus object positions from the imaging unit 200, and projects a plurality of pattern image signals having different in-focus object positions that are captured by projecting a geometric pattern onto the subject. Obtained from the imaging unit 200. Next, the evaluation value calculation unit 120 calculates a first evaluation value based on the plurality of pattern image signals, and calculates a second evaluation value based on the plurality of image signals. Then, the image composition unit 130 performs a comparison process between the first evaluation value and the second evaluation value, and performs a combination process of a plurality of image signals based on the result of the comparison process.

  Here, the image signal refers to a signal representing an image obtained by imaging a subject. The image signal may be an analog signal or a digital signal.

  The evaluation value is a value for evaluating at what ratio the first image signal and the second image signal are used in the image composition processing. The evaluation value can also be said to be a value used for evaluating whether or not the image is in focus, that is, a value used for evaluating the degree of focusing of the image. Further, it can be said that the evaluation value is a value that changes based on an optical deterioration function described later.

  Here, the evaluation value calculation unit 120 may calculate a second evaluation value representing the contrast or edge component of a plurality of image signals.

  That is, the second evaluation value may be the contrast or edge component itself, or may be another value representing the contrast or edge component.

  Here, the contrast (contrast value) may be, for example, a value obtained by the above-described equation (1).

  Further, as described above, the edge component may be a value calculated by an existing edge extraction filter such as a Sobel filter or a Laplacian filter.

  This makes it possible to perform image composition processing based on the contrast or edge component of the image.

  Further, the evaluation value calculation unit 120 may calculate the luminance values of the plurality of pattern image signals as the first evaluation value.

  When the luminance value of the pattern image signal is used as the first evaluation value, the first evaluation value is larger in the in-focus state than in the out-of-focus state, and in the out-of-focus state. The first evaluation value is smaller than that in the in-focus state. That is, the first evaluation value increases as the degree of focusing of the image increases.

  Further, since the geometric pattern (pattern light) is projected, the luminance value of the pixel onto which the geometric pattern is projected does not become so small that it cannot be calculated.

  As a result, for example, even when effective contrast or the like cannot be calculated in the image composition process, it is possible to calculate other evaluation values used for the image composition process.

  Such a first evaluation value is obtained from a pattern image signal. The pattern image signal refers to a signal of an image captured by projecting a geometric pattern. In the present embodiment, pattern image signals having different in-focus object positions are acquired. Specific examples include the signal of the near point pattern image IM3 in FIG. 3B and the signal of the far point pattern image IM4 in FIG. 3C.

  Examples of the geometric pattern include patterns described with reference to FIGS. 4 (A) to 4 (C). However, the geometric pattern is not limited to the examples shown in FIGS. 4A to 4C, and other patterns may be used.

  Further, the in-focus object position refers to a position on the object side where an object reflected in an image captured through a lens is in an in-focus state. That is, the focused object position is a position on the subject side that is currently focused.

  As described above, in this embodiment, the first evaluation value is calculated from the signal of the image captured by projecting the geometric pattern, and the second evaluation value is calculated from the signal of the image captured without projecting the geometric pattern. To do. Then, image composition processing is performed based on the calculated first evaluation value and second evaluation value.

  Since the first evaluation value is not affected by the contrast or texture of the subject, an evaluation value effective for the image composition process can be calculated even for a subject with low contrast or a subject with little texture. That is, it is possible to calculate an evaluation value for stably generating a composite image even in an area with less texture and a subject with low contrast.

  As a result, even with an image signal having a low contrast, image synthesis processing can be performed while suppressing the occurrence of artifacts.

  Also, by performing such processing, it is possible to generate an omnifocal image in which the entire image is in focus from a plurality of image signals.

  In addition, the evaluation value calculation unit 120 calculates a first evaluation value for each synthesis target pixel that is a pixel to be synthesized in a plurality of pattern image signals, and a second value for each synthesis target pixel of the plurality of image signals. An evaluation value may be calculated. Then, the image synthesis unit 130 performs a comparison process between the first evaluation value and the second evaluation value for each synthesis target pixel, and performs a synthesis process of a plurality of image signals for each synthesis target pixel based on the comparison process result. You may go.

  Here, the synthesis target pixel means at least one pixel to be subjected to image synthesis processing. The synthesis target pixel may be one pixel or a group of pixels composed of a plurality of pixels. For example, four pixels of 2 pixels in the vertical direction and 2 pixels in the horizontal direction may be collectively treated as the synthesis target pixels.

  As a result, the first evaluation value and the second evaluation value can be calculated for the pixel to be image-combined, and comparison processing is performed on all the pixels to be combined, for example, focusing of the image It is possible to determine the state or the like and perform more accurate image composition processing.

  Next, a method for determining the inter-dot distance d of the dot pattern PIM as shown in FIG. First, as shown in FIG. 3B or FIG. 3C, the projected dots are captured in a clear form within the depth of field according to the optical degradation function of the optical system. The image is blurred outside. The optical deterioration function is, for example, a point spread function (PSF), which is a function representing a response of the optical system to a point light source.

  That is, if the degree of blur of the projected dots is determined, it can be determined whether or not each pixel position onto which the geometric pattern is projected is within the depth of field. Generally, a clearly imaged part has a higher brightness than a blurred part, and a blurred part has a lower brightness than a clearly imaged part. It is also possible to determine the degree of image blur.

  Therefore, the image acquisition unit 110 may acquire a plurality of pattern image signals captured by projecting a geometric pattern set based on the optical degradation function from the imaging unit 200.

  This makes it possible to evaluate (determine) whether or not each pixel position onto which the geometric pattern is projected is within the depth of field, that is, the focused state of each pixel position based on the luminance value of the pattern image. Become.

  However, when the projected dots overlap, the change in luminance value due to the optical degradation function cannot be calculated accurately. Therefore, when the blur radius is R at a predetermined subject distance due to the optical degradation function of the optical system, it is necessary to set the inter-dot distance d to be 2R or more so that the projected dots do not overlap.

  That is, the image acquisition unit 110 projects a dot pattern set such that the inter-dot distance d is d ≧ 2R (d and R are positive integers) with respect to the blur radius R based on the optical degradation function. A plurality of pattern image signals captured in this manner may be acquired.

  This makes it possible to project a dot pattern onto the subject so that the dots do not overlap.

  However, when the inter-dot distance d is determined in this way, the projected dot pattern becomes sparse. Here, the sparse dot pattern indicates that there is a pixel in which no dot is projected in the pattern image. As described above, when d ≧ 2R, it is always sparse.

  For example, in a scene where the present embodiment is applied to an endoscope, an image in a dark body is picked up, so the brightness of a portion where no dots are projected is very small. It may not be used effectively as an evaluation value for processing.

  Therefore, in the present embodiment, when the calculated first evaluation value is equal to or less than a given threshold value, the first evaluation value may be interpolated (recalculated).

  That is, the evaluation value calculation unit 120 determines whether or not the calculated first evaluation value is equal to or less than a given threshold value for each synthesis target pixel that is a pixel that is a target of the synthesis process in a plurality of pattern image signals. When it is determined that the first evaluation value is equal to or less than a given threshold value, an interpolation process may be performed on the first evaluation value.

  In the comparison process between the first evaluation value and a given threshold, it is possible to add a modification such as inversion of the sign, and such a modification is also within the scope of the present invention.

  As a result, even if the dot pattern is sparse, it is possible to obtain a first evaluation value that is effective as an evaluation value for image composition processing for all the compositing target pixels.

  In addition, when the evaluation value calculation unit 120 determines that the first evaluation value is equal to or less than a given threshold value, any one of zero-order interpolation processing, linear interpolation processing, and morphological expansion processing is performed as the interpolation processing. May be performed on the first evaluation value.

  Thereby, it is possible to interpolate the first evaluation value determined as not being effective as the evaluation value of the image composition processing based on the other first evaluation value calculated from the same pattern image signal.

  As another method of interpolation processing, the spatial pattern modulator is controlled, the projection pattern is moved in the horizontal and vertical directions, or the light and dark are reversed. By capturing a single pattern image) and a far point pattern image (second pattern image) a plurality of times, a complementary pattern image that interpolates a sparse area can be acquired.

  That is, the image acquisition unit 110 acquires a plurality of pattern image signals from the imaging unit 200, and projects a plurality of complementary pattern image signals captured by projecting a geometric pattern different from the previously projected geometric pattern onto the subject. You may acquire from 200. Then, the evaluation value calculation unit 120 applies the first evaluation value calculated from the plurality of pattern image signals to the synthesis target pixel determined to be equal to or less than a given threshold based on the plurality of complementary pattern images. The process of recalculating the evaluation value may be performed as an interpolation process.

  Here, a geometric pattern different from the previously projected geometric pattern (hereinafter referred to as a complementary geometric pattern) means that all the synthesis target pixels are obtained by combining the previously projected geometric pattern and the complementary geometric pattern projected this time. On the other hand, it refers to a geometric pattern on which pattern light is projected. For example, when the previously projected geometric pattern is a line pattern PIM2 as shown in FIG. 4B, it indicates a line pattern in which all white lines of PIM2 are moved downward by a predetermined distance. However, pattern light does not necessarily have to be projected onto all synthesis target pixels in two geometric patterns. When three or more geometric patterns, for example, geometric patterns A to C are combined, all synthesis is performed. The pattern light may be projected onto the target pixel. In this case, for the geometric pattern A, the geometric patterns B and C are complementary geometric patterns.

  The complementary pattern image signal refers to an image signal captured by projecting a complementary geometric pattern.

  As a result, it is possible to replace the first evaluation value determined not to be effective as the evaluation value of the image composition processing with the first evaluation value corresponding to the same composition target pixel calculated from the complementary pattern image signal. Become.

  Furthermore, even if all the used geometric patterns are combined, there may be a synthesis target pixel on which pattern light is not projected. In this case, as the interpolation process, any one of a zero-order interpolation process, a linear interpolation process, and a morphological expansion process may be further performed. That is, both the zero-order interpolation process and the interpolation process using the complementary pattern image may be performed.

  Now, the first evaluation value and the second evaluation value are obtained as described above. Basically, the image composition processing is performed based on the second evaluation value. However, when the second evaluation value is not valid as the evaluation value of the image composition process, the image composition process is performed based on the first evaluation value. The case where the second evaluation value is not valid is a case where the second evaluation value becomes a very small value because the contrast of the subject is low or the texture is small as described above.

  Therefore, in the present embodiment, it is determined whether or not the second evaluation value is effective as an evaluation value for the image composition processing by performing a determination process whether or not the second evaluation value is equal to or less than a given threshold value. If it is determined that the second evaluation value is valid as the evaluation value for the image composition process, the second evaluation value is used to perform the image composition process, and the second evaluation value is valid as the evaluation value for the image composition process. If it is determined that it is not, the image composition processing may be performed using the first evaluation value.

  That is, the image synthesis unit 130 performs a determination process as to whether or not the calculated second evaluation value is equal to or less than a given threshold value for each synthesis target pixel that is a pixel that is a target of the synthesis process in the plurality of pattern image signals. If it is determined that the second evaluation value is less than or equal to the given threshold value, a synthesis process is performed based on the first evaluation value, and if it is determined that the second evaluation value is greater than the given threshold value May perform the synthesis process based on the second evaluation value.

  In the comparison process between the second evaluation value and a given threshold value, it is possible to add a modification such as inversion of the sign, and such a modification is also within the scope of the present invention.

  As a result, when it is determined that the second evaluation value is valid as the evaluation value for the image composition processing, the image composition processing is performed using the second evaluation value, and the second evaluation value is used as the evaluation value for the image composition processing. If it is determined that the image is not valid, it is possible to perform image composition processing using the first evaluation value.

  In the image composition process, an image composition coefficient may be calculated from the first evaluation value or the second evaluation value.

  That is, when the image synthesis unit 130 determines that the second evaluation value is equal to or less than a given threshold value, the image synthesis unit 130 calculates an image synthesis coefficient based on the first evaluation value, and uses the image synthesis coefficient to combine the target pixel. When it is determined that the second evaluation value is greater than a given threshold value, an image synthesis coefficient is calculated based on the second evaluation value, and each synthesis target pixel is used using the image synthesis coefficient. A synthesis process may be performed.

  Here, the image synthesis coefficient specifically refers to c (x, y) in the above-described formula (4), and is obtained by the above-described formula (2), formula (3), or the like. However, the image synthesis coefficient is not limited to the value calculated by Expression (2) or Expression (3), and may be calculated by an arbitrary method.

  Thus, for example, by using the image synthesis coefficient calculated by Expression (3), it is possible to smoothly synthesize the switching portion between the in-focus area and the out-of-focus area.

  The method of the present embodiment described above is applied to the example using the above-described two image signals and two pattern image signals, in other words, as follows.

  The image acquisition unit 110 acquires the first image signal and the first image signal and the second image signal having a different in-focus object position from the imaging unit 200, and is imaged by projecting a geometric pattern onto the subject. Alternatively, the first pattern image signal and the second pattern image signal having a different focus object position from the first pattern image signal may be acquired from the imaging unit 200. Then, the evaluation value calculation unit 120 calculates a first evaluation value for the first pattern image signal and the second pattern image signal for each synthesis target pixel that is a pixel that is a target of the synthesis process, and first The second evaluation value for the image signal and the second image signal may be calculated. Further, the image synthesis unit 130 performs a determination process for determining whether or not both of the second evaluation values for the first image signal and the second image signal are equal to or less than a given threshold value for each synthesis target pixel. When it is determined that both the two evaluation values are equal to or less than the given threshold value, the first image signal and the second image signal are combined based on the first evaluation value, and both the second evaluation values are given. When it is determined that the value is larger than the given threshold value, the first image signal and the second image signal may be combined based on the second evaluation value.

  Here, the first image signal refers to a signal representing the above-described near-point image, for example, a signal representing the image IM1 in FIG. On the other hand, the second image signal refers to a signal representing the far point image described above, for example, a signal representing the image IM2 in FIG.

  Here, the first pattern image signal refers to a signal representing the above-described near point pattern image, for example, a signal representing the image IM3 in FIG. 3B. On the other hand, the second pattern image signal refers to a signal representing the far-point pattern image described above, for example, a signal representing the image IM4 in FIG.

  As a result, when it is determined that the second evaluation values for the first image signal and the second image signal are both equal to or less than a given threshold, a second evaluation value effective for the image composition process is calculated. It can be determined that the first image signal and the second image signal are combined based on the first evaluation value. Further, when it is determined that the second evaluation values for the first image signal and the second image signal are both greater than a given threshold value, a second evaluation value effective for the image composition process is calculated. And the first image signal and the second image signal can be combined based on the second evaluation value.

  That is, even when two image signals and two pattern image signals are acquired, it is possible to select an effective evaluation value from the first evaluation value and the second evaluation value and perform image composition processing, etc. Become. In this case as well, the first evaluation value interpolation processing or the like may be performed.

  The image acquisition unit 110 may acquire a plurality of pattern image signals captured by projecting a geometric pattern using a coherent light source and a spatial light modulator from the imaging unit 200.

  This makes it possible to acquire a pattern image signal in which a clear arbitrary geometric pattern is reflected.

  Further, the image acquisition unit 110 may acquire a plurality of pattern image signals captured by projecting a geometric pattern using a coherent light source having a non-visible wavelength and an optical spatial modulator from the imaging unit 200.

  This makes it possible to acquire the pattern image signal of the near point pattern image captured at the same timing as the near point image and the pattern image signal of the far point pattern image captured at the same timing as the far point image. Become.

  Further, the image processing apparatus 100 according to the present embodiment may be a component of the imaging apparatus 400 or the endoscope 400. The imaging device 400 and the endoscope 400 may include the image processing device 100, the imaging unit 200, and a pattern projection unit 300 that projects a geometric pattern onto a subject.

  Note that the image processing apparatus 100 or the like according to the present embodiment may realize part or most of the processing by a program. In this case, the image processing apparatus 100 according to the present embodiment is realized by a processor such as a CPU executing a program. Specifically, a program stored in the information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. The configurations and operations of the image processing apparatus 100, the imaging apparatus 400, the endoscope 400, and the like are not limited to those described in the present embodiment, and various modifications can be made.

100 image processing apparatus, 110 image acquisition unit, 120 evaluation value calculation unit,
130 image composition unit, 200 imaging unit, 300 pattern projection unit,
400 Imaging device (endoscope)

Claims (18)

An image acquisition unit for acquiring a plurality of image signals;
An evaluation value calculation unit that performs an evaluation value calculation process based on the acquired image signal;
An image composition unit that performs composition processing of the plurality of image signals based on the evaluation value;
Including
The image acquisition unit
The plurality of image signals having different in-focus object positions are acquired from the imaging unit, and the plurality of pattern image signals having different in-focus object positions captured by projecting a geometric pattern onto a subject are acquired from the imaging unit,
The evaluation value calculation unit
Calculating a first evaluation value based on the plurality of pattern image signals, and calculating a second evaluation value based on the plurality of image signals;
The image composition unit
An image processing apparatus that performs a comparison process between the first evaluation value and the second evaluation value, and performs the synthesis process of the plurality of image signals based on a result of the comparison process. In claim 1,
The evaluation value calculation unit
The first evaluation value is calculated for each synthesis target pixel that is a pixel that is the target of the synthesis process in the plurality of pattern image signals, and the second evaluation value is calculated for each synthesis target pixel of the plurality of image signals. Calculate
The image composition unit
A comparison process between the first evaluation value and the second evaluation value is performed for each synthesis target pixel, and the synthesis process of the plurality of image signals is performed for each synthesis target pixel based on the comparison process result. An image processing apparatus. In claim 1 or 2,
The evaluation value calculation unit
In each of the plurality of pattern image signals, a determination process is performed to determine whether or not the calculated first evaluation value is equal to or less than a given threshold value for each synthesis target pixel that is a pixel that is a target of the synthesis process. An image processing apparatus characterized by performing an interpolation process on the first evaluation value when it is determined that the evaluation value is equal to or less than the given threshold value. In claim 3,
The evaluation value calculation unit
When it is determined that the first evaluation value is less than or equal to the given threshold value, any one of a zero-order interpolation process, a linear interpolation process, and a morphological expansion process is performed as the interpolation process. An image processing apparatus characterized by being performed on In claim 3 or 4,
The image acquisition unit
The plurality of pattern image signals are acquired from the imaging unit, and a plurality of complementary pattern image signals captured by projecting a geometric pattern different from the previously projected geometric pattern onto the subject are acquired from the imaging unit,
The evaluation value calculation unit
The first evaluation value calculated from the plurality of pattern image signals is determined based on the plurality of complementary pattern images for the synthesis target pixel determined to be equal to or less than the given threshold value. An image processing apparatus that performs a recalculation process as the interpolation process. In any one of Claims 1 thru | or 5,
The image composition unit
For each synthesis target pixel that is a pixel that is a target of the synthesis process in the plurality of pattern image signals, a determination process is performed to determine whether the calculated second evaluation value is equal to or less than a given threshold value,
If it is determined that the second evaluation value is less than or equal to the given threshold value, the combining process is performed based on the first evaluation value,
An image processing apparatus, wherein when it is determined that the second evaluation value is greater than the given threshold value, the composition processing is performed based on the second evaluation value. In claim 6,
The image composition unit
When it is determined that the second evaluation value is equal to or less than the given threshold, an image synthesis coefficient is calculated based on the first evaluation value, and the synthesis target pixel is used for each synthesis target pixel using the image synthesis coefficient. Perform the synthesis process,
If it is determined that the second evaluation value is greater than the given threshold value, the image synthesis coefficient is calculated based on the second evaluation value, and for each synthesis target pixel using the image synthesis coefficient An image processing apparatus that performs the composition processing. In any one of Claims 1 thru | or 7,
The image acquisition unit
The first image signal, the first image signal, and the second image signal having a different in-focus object position are acquired from the imaging unit, and the first image signal is captured by projecting a geometric pattern onto the subject. A pattern image signal, the first pattern image signal, and a second pattern image signal having a different focused object position are acquired from the imaging unit,
The evaluation value calculation unit
The first evaluation value for the first pattern image signal and the second pattern image signal is calculated for each synthesis target pixel that is a pixel that is a target of the synthesis process, and the first image signal and Calculating the second evaluation value for the second image signal;
The image composition unit
For each of the synthesis target pixels, a determination process is performed to determine whether or not both of the second evaluation values for the first image signal and the second image signal are equal to or less than a given threshold value,
If it is determined that both of the second evaluation values are less than or equal to the given threshold value, the composition processing of the first image signal and the second image signal is performed based on the first evaluation value,
When it is determined that both of the second evaluation values are larger than the given threshold value, the synthesis processing of the first image signal and the second image signal is performed based on the second evaluation value. An image processing apparatus. In any one of Claims 1 thru | or 8.
The evaluation value calculation unit
An image processing apparatus, wherein brightness values of the plurality of pattern image signals are calculated as the first evaluation value. In any one of Claims 1 thru | or 9,
The evaluation value calculation unit
An image processing apparatus that calculates the second evaluation value representing a contrast or an edge component of the plurality of image signals. In any one of Claims 1 thru | or 10.
The image acquisition unit
An image processing apparatus, wherein the plurality of pattern image signals captured by projecting the geometric pattern set based on an optical degradation function are acquired from the imaging unit. In claim 11,
The image acquisition unit
Based on the optical degradation function, the plurality of images captured by projecting a dot pattern set such that the inter-dot distance d is d ≧ 2R with respect to the blur radius R (d and R are positive integers) An image processing apparatus characterized by acquiring a pattern image signal. In any one of Claims 1 to 12,
The image acquisition unit
An image processing apparatus, wherein the plurality of pattern image signals captured by projecting the geometric pattern using a coherent light source and a spatial light modulator are acquired from the imaging unit. In claim 13,
The image acquisition unit
An image processing apparatus, wherein the plurality of pattern image signals captured by projecting the geometric pattern using the coherent light source having an invisible wavelength and the spatial light modulator are acquired from the imaging unit. An image processing device according to any one of claims 1 to 14,
The imaging unit;
A pattern projection unit that projects the geometric pattern onto the subject;
An imaging apparatus comprising: An image processing device according to any one of claims 1 to 14,
The imaging unit;
A pattern projection unit that projects the geometric pattern onto the subject;
The endoscope characterized by including. An image acquisition unit for acquiring a plurality of image signals;
An evaluation value calculation unit that performs an evaluation value calculation process based on the acquired image signal;
As an image composition unit that performs composition processing of the plurality of image signals based on the evaluation value,
Make the computer work,
The image acquisition unit
The plurality of image signals having different in-focus object positions are acquired from the imaging unit, and the plurality of pattern image signals having different in-focus object positions captured by projecting a geometric pattern onto a subject are acquired from the imaging unit,
The evaluation value calculation unit
Calculating a first evaluation value based on the plurality of pattern image signals, and calculating a second evaluation value based on the plurality of image signals;
The image composition unit
A program for performing comparison processing between the first evaluation value and the second evaluation value, and performing the synthesis processing of the plurality of image signals based on the comparison processing result. A plurality of image signals with different in-focus object positions are acquired from the imaging unit, and a plurality of pattern image signals with different in-focus object positions captured by projecting a geometric pattern onto a subject are acquired from the imaging unit,
Calculating a first evaluation value based on the plurality of pattern image signals, and calculating a second evaluation value based on the plurality of image signals;
An image processing method comprising performing comparison processing between the first evaluation value and the second evaluation value, and combining the plurality of image signals based on a result of the comparison processing.