JP2012239644A - Image processing apparatus, endoscope apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an endoscope apparatus and an image processing method, etc., properly correcting blur even if an imaging visual field changes.SOLUTION: The image processing apparatus includes: an image obtaining part for obtaining a plurality of images in time series via an imaging part which can change the imaging visual field; a first motion information detection part for detecting motion information of the imaging visual field according to the operation of moving the imaging visual field of the imaging part as first motion information; a second motion information detection part for detecting second information which is motion information between images based on the plurality of images obtained in time series; and a blur-correction processing part for correcting blur of the images according to the first motion information and second motion information.

Description

  The present invention relates to an image processing apparatus, an endoscope apparatus, an image processing method, and the like.

  Conventionally, methods such as an electronic shake correction process and an optical shake correction process have been widely used as a technique for correcting a shake of a moving image such as a consumer video camera.

  In the endoscope, for example, as disclosed in Patent Document 1, a technique is disclosed in which the amount of movement of the scope tip is detected and blur correction processing is performed based on the detected amount of movement.

  On the other hand, in an endoscope, an angle knob provided in a scope is operated to change an image field (hereinafter referred to as an imaging field or an observation field). Specifically, the imaging field of view is changed by changing the degree of curvature of the scope tip (hereinafter, angle) in accordance with the amount of operation of the angle knob.

Japanese Patent Laid-Open No. 5-49599

  As described above, consider the case where the electronic blur correction process is performed when it is desired to change the imaging field (observation field) by operating the angle knob. In this case, although the doctor wanted to change the imaging field of view by operating the angle knob, it has been found that there is a problem that the imaging field of view is fixed by electronic blur correction processing.

  On the other hand, Patent Document 1 discloses a technique for blur correction processing using information on the amount of movement of the scope tip. However, in the endoscope apparatus, the imaging field of view fluctuates due to the influence of the pulsation of the living body that is the subject. Therefore, there is a problem that the blur correction process does not function sufficiently only with information on the amount of movement of the scope tip.

  According to some aspects of the present invention, it is possible to provide an image processing apparatus, an endoscope apparatus, an image processing method, and the like that can realize an appropriate blur correction process even when the imaging field of view changes.

  One aspect of the present invention is an image acquisition unit that acquires a plurality of images in time series via an imaging unit that can change an imaging field, and the imaging field that corresponds to an operation of moving the imaging field of the imaging unit. A first motion information detector that detects the motion information of the second as motion information, and a second motion information that detects motion information between images based on the plurality of images acquired in time series. The present invention relates to an image processing apparatus including a motion information detection unit and a blur correction processing unit that performs a blur correction process on the image according to the first motion information and the second motion information.

  According to one aspect of the present invention, motion information of the imaging field corresponding to an operation for moving the imaging field is detected as first motion information, and motion information between a plurality of images is detected as second motion information. And the blurring correction process according to these 1st motion information and 2nd motion information is performed.

  In this way, it is possible to realize blur correction processing that reflects not only the second motion information that is motion information between a plurality of images but also the first motion information that is motion information of the imaging field of view according to the operation. Therefore, it is possible to provide an image processing apparatus capable of realizing an appropriate blur correction process even when the imaging field of view changes.

  Another aspect of the present invention includes an image acquisition unit that acquires a plurality of frame images in time series via an imaging unit that can change the imaging field of view by an instruction to change the imaging field, and the imaging unit that receives the change instruction. A first motion information detecting unit that detects first motion information indicating the motion of the imaging field of view, a second motion information detecting unit that detects second motion information indicating the motion of the entire image of the frame image, and the first The present invention relates to an image processing apparatus including a motion compensation processing unit that performs motion compensation processing according to one motion information and the second motion information on the frame image.

  Another aspect of the present invention relates to an endoscope apparatus including the image processing apparatus described above and an operation unit for performing an operation of moving the imaging field of view of the imaging unit.

  According to another aspect of the present invention, movement information of the imaging field of view according to an operation of acquiring a plurality of images in time series and moving the imaging field of view of the imaging unit via an imaging unit capable of changing the imaging field of view. Is detected as first motion information, second motion information that is motion information between images is detected based on the plurality of images acquired in time series, and the first motion information and the second motion information are detected. This relates to an image processing method for performing blur correction processing corresponding to the image on the image.

  According to another aspect of the present invention, a plurality of frame images are acquired in time series via an imaging unit capable of changing the imaging field by an instruction to change the imaging field, and the imaging field of the imaging unit according to the change instruction First motion information indicating the motion of the frame image, second motion information indicating the motion of the entire image of the frame image is detected, and blur correction processing corresponding to the first motion information and the second motion information is performed on the frame. This relates to an image processing method performed on an image.

FIG. 1A to FIG. 1C are explanatory diagrams for a comparative example of the present embodiment. Explanatory drawing of the method of this embodiment. 3A and 3B are explanatory diagrams of the first technique of the present embodiment. FIGS. 4A and 4B are explanatory diagrams of the second method of the present embodiment. 1 is a first configuration example of an image processing apparatus and an endoscope apparatus according to the present embodiment. An example of a color filter of a Bayer arrangement. An example of the light transmission characteristic of a color filter. The example of the insertion part (endoscope scope) which has an angle knob. Explanatory drawing about the change of an imaging visual field. Explanatory drawing of an electronic blurring correction process. Explanatory drawing of an electronic blurring correction process. Explanatory drawing of an electronic blurring correction process. Explanatory drawing of an electronic blurring correction process. Explanatory drawing of an electronic blurring correction process. Explanatory drawing of an electronic blurring correction process. The structural example of a 2nd motion vector detection part. Explanatory drawing of the setting method of a local region. The structural example of a 1st motion vector detection part. The structural example of a 3rd motion vector calculation part. 2 shows a second configuration example of an image processing apparatus and an endoscope apparatus according to the present embodiment. Explanatory drawing of the change direction of an imaging visual field. The structural example of a 1st motion vector detection part. Explanatory drawing about the relationship between the variation | change_quantity of a rotation angle, and the motion vector on an image. Explanatory drawing about the relationship between the variation | change_quantity of a rotation angle, and the motion vector on an image. Explanatory drawing about the relationship between the variation | change_quantity of a rotation angle, and the motion vector on an image. Explanatory drawing about the relationship between the variation | change_quantity of a rotation angle, and the motion vector on an image. The structural example of a 3rd motion vector calculation part.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the outline of the image processing method of this embodiment will be described. 1A to 1C show the method of Patent Document 1 described above as a method of a comparative example.

  In the method of this comparative example, as shown in FIG. 1A, a cutout frame of an image signal to be read from the memory is set based on the moving distance (moving speed) of the distal end portion of the insertion portion of the endoscope apparatus. FIG. 1A shows an example of setting the cutout frame before and after movement when the distal end portion of the insertion portion is displaced from the first bending position A to the second bending position B. The memory control unit generates a read address corresponding to the cutout frame set as shown in FIG. 1A, so that the corresponding image signal is read from the memory.

  When a jiggling operation is performed in which the distal end portion of the insertion portion of the endoscope apparatus is vibrated in small increments, the imaging region for the subject 31 changes from A to B as shown in FIG. Moving. At this time, if the image signal stored in the memory is read as it is, the display position of the subject 31 is different in each of A and B, as shown on the left side of FIG. .

  Therefore, in the method of this comparative example, a cutout frame as shown in FIG. 1A is set, and image data is read from the memory, thereby realizing blur correction processing and a good endoscopic image without image blurring. Is generated. That is, in the method of the comparative example, image blurring caused by the jiggling operation is prevented by performing blurring correction processing based on the moving distance (moving speed) of the distal end portion of the insertion portion.

  However, as described above, in the endoscope apparatus, the imaging field of view fluctuates due to the influence of the pulsation of the living body that is the subject. It cannot be detected using the moving distance. Therefore, in the method of the comparative example shown in FIGS. 1A to 1C, it is impossible to perform the blur correction process for such image blur due to the pulsation of the living body.

  In the endoscope apparatus, an angle knob for instructing to change the imaging field of view (observation field of view) is provided in the operation unit of the insertion unit, and the doctor operates the angle knob by operating the angle knob. Can observe the subject in the desired field of view.

  However, when such an angle knob operation is performed, if the blur correction process of the endoscope apparatus is activated, the imaging field of view is fixed by this blur correction process, and the doctor can capture the desired imaging. The subject cannot be observed in the field of view.

  In the present embodiment, in order to solve such a problem, as shown in FIG. 2, the first motion information and the second motion information are detected, and blurs corresponding to the first motion information and the second motion information are detected. Image blurring is corrected by performing correction processing. For example, as illustrated in FIG. 2, third motion information is obtained based on the first motion information and the second motion information, and blur correction processing is performed based on the obtained third motion information.

  Here, the first motion information is motion information of the imaging field according to an operation of moving the imaging field (observation field) of the imaging unit. For example, the first motion information is motion information indicating the movement of the imaging field of the imaging unit in response to an instruction to change the imaging field. On the other hand, the second motion information is inter-image motion information (inter-frame motion information) detected based on a plurality of images acquired in time series via an imaging unit capable of changing the imaging field of view. For example, the second motion information is motion information indicating the motion of the entire frame image.

  As shown in FIG. 2, the third motion information is obtained as information obtained by excluding the component corresponding to the first motion information from the second motion information. For example, when the second motion information is detected as motion information between images and the first motion information is detected in response to an operation for moving the imaging field of view, the blur correction process due to the operation for moving the imaging field of view is not performed. The third motion information is obtained by performing a correction process that excludes (cancels or reduces) the component of the first motion information on the second motion information. Then, blur correction processing is performed based on the obtained third motion information. In this way, when the doctor (user) consciously performs an operation to move the imaging field of view, this operation is erroneously recognized as a blur, and the imaging field of view is fixed against the doctor's intention. The occurrence of the situation can be suppressed.

  Here, as a method for obtaining the third motion information, the first method described with reference to FIGS. 3A and 3B and the second method described with reference to FIGS. 4A and 4B. A method can be assumed. The first method is a method realized by a first configuration example of FIG. 5 described later, and the second method is a method realized by a second configuration example of FIG.

  For example, as shown in FIG. 3A, in the first method, the third method is based on the second motion information acquired in the non-operation state (non-operation period) in which the operation for moving the imaging field of view is not performed. Find movement information. Specifically, the second movement information in the operation state (operation period) in which the operation for moving the imaging field of view is performed is not used, and the third movement is performed based on the second movement information in the non-operation state. Information is obtained, and blur correction processing is performed based on the obtained third motion information. As a result, information obtained by excluding the component of the first motion information from the second motion information is obtained as the third motion information, and the blur correction process can be performed.

  More specifically, as shown in FIG. 3B, it can be detected based on the first motion information whether or not it is in the operating state. For example, it is determined based on the first motion information whether the operation state (operation period) or the non-operation state (non-operation period). Specifically, when the operation of moving the imaging field of the imaging unit is an operation performed using the operation unit, whether or not the operation state is based on operation information (such as the rotation angle of the angle knob) of the operation unit. To detect.

  Further, as shown in FIG. 3B, in the operation state (in the operation period), the second movement information is not integrated, and in the non-operation state (in the non-operation period), the second The third motion information is obtained by performing integration processing on the motion information. In this case, a process of attenuating the previous calculation result in the integration process by the attenuation coefficient is performed. Specifically, the integration result is obtained based on the integration result up to the previous time, the attenuation coefficient for attenuating the integration result up to the previous time, and the current second motion information. If it is determined that the magnitude of the motion represented by the third motion information (for example, the magnitude of the third motion vector) is larger than a given threshold value, the blur correction process is turned off. For example, the third motion information that does not perform the blur correction process is output.

  On the other hand, as shown in FIG. 4A, in the second method, difference information between the first motion information and the second motion information is obtained as third motion information. That is, a difference process (subtraction process) between the first motion information and the second motion information is performed to obtain the third motion information.

  Specifically, as shown in FIG. 4B, the first motion information is detected as a first motion vector, and the second motion information is detected as a second motion vector. Then, the third motion information is obtained as a third motion vector, and blur correction processing is performed based on the obtained third motion vector.

  For example, as shown in FIG. 4B, a difference motion vector representing the difference between the first motion vector and the second motion vector is obtained by performing a difference process between the first motion vector and the second motion vector. Then, a third motion vector is obtained based on the obtained difference motion vector. Specifically, for example, the difference motion vector is averaged to obtain an average motion vector, and the third motion vector is obtained based on the difference between the difference motion vector and the average motion vector. For example, the third motion vector is obtained by performing integration processing on the difference between the difference motion vector and the average motion vector. At this time, as shown in FIG. 4B, it is determined whether or not the magnitude of the difference motion vector is larger than a given threshold value. If it is determined that the difference is larger than the given threshold, the magnitude of the differential motion vector is set to 0 (given value).

  According to the first and second methods of the present embodiment as described above, when performing blur correction processing according to the second motion information that is motion information between images, the second motion information is converted into the first motion information. The blur correction process can be performed with the information excluding. Therefore, it is possible to provide an image processing apparatus, an endoscope apparatus, and the like that can realize an appropriate shake correction process even when the imaging field of view changes.

2. First Configuration Example FIG. 5 shows a first configuration example of an image processing apparatus according to this embodiment and an endoscope apparatus (endoscope system) including the image processing apparatus.

  The endoscope apparatus of the first configuration example of FIG. 5 is a configuration example that realizes the first technique of FIGS. 3A and 3B, and includes a light source unit 100, an insertion unit 200, and an image. The processing apparatus 300, the display part 400, and the external I / F part 500 are included. Further, the image processing apparatus 300 of the first configuration example includes an image acquisition unit 308, a second motion vector detection unit 330, an operation state acquisition unit 340, a first motion vector detection unit 350a, and a third motion vector calculation unit. 360 a, a shake correction processing unit (image extraction unit) 370, a display image generation unit 380, and a control unit 390. The image acquisition unit 308 includes, for example, an interpolation processing unit 310 and a luminance image generation unit 320. Note that the endoscope apparatus and image processing apparatus of the present embodiment are not limited to the configuration shown in FIG. 5, and various modifications such as omitting some of these components or adding other components are possible. It is. Further, the image processing apparatus 300 of the present embodiment can be applied to apparatuses other than the endoscope apparatus.

  The light source unit 100 includes a white light source 110 that generates white light and a lens 120 that collects the white light on the light guide fiber 210.

  The insertion portion 200 (endoscope scope in a narrow sense) is formed to be elongated and bendable so as to be inserted into a body cavity. Specifically, the insertion portion 200 has a shape as shown in FIG.

  The insertion unit 200 includes a light guide fiber 210 for guiding the light collected by the light source unit 100, an illumination lens 220 for diffusing the light guided by the light guide fiber 210 and irradiating the subject, and an imaging unit 228. And an operation unit 250 for changing the angle of the distal end of the insertion unit 200. The imaging unit 228 includes, for example, a condenser lens 230 that collects reflected light from the subject, and an imaging element 240 that detects the reflected light collected by the condenser lens 230. Note that the configuration of the imaging unit 228 is not limited to this.

  The image sensor 240 has, for example, a Bayer color filter as shown in FIG. As shown in FIG. 7, the three types of color filters r, g, and b shown in FIG. 6 are 580 to 700 [nm] for the r filter, 480 to 600 [nm] for the g filter, and 400 to 500 [nm] for the b filter. nm] light.

  Specifically, the operation unit 250 includes, for example, the angle knob 10 shown in FIG. By operating the angle knob 10, the angle of the distal end of the insertion portion 200 can be controlled. In the present embodiment, the angle knob 10 is composed of two types of angle knobs (for example, 12 and 14 in FIG. 8), and for convenience, will be referred to as a vertical angle knob and a horizontal angle knob. The angle knob 10 is simply referred to as an angle knob as appropriate. This angle knob is used when changing an imaging field (synonymous with an observation field; the same applies hereinafter) in endoscopic diagnosis. The operation unit 250 is connected to an operation state acquisition unit 340 described later.

  Here, the relationship between the operation of the angle knob and the change in the imaging field (observation field) will be described. Here, the xy coordinate system shown in FIG. 9 is used as the image coordinate system. In the present embodiment, when the vertical angle knob is operated, the imaging field of view changes in the “V” direction (y-axis direction) shown in FIG. When the horizontal direction angle knob is operated, the imaging field of view changes in the “H” direction (x-axis direction) shown in FIG.

  Next, general electronic shake correction processing will be described. First, it is assumed that the tip of the insertion unit 200 (imaging unit 228) is at a position indicated by P1 in FIG. At this time, the image sensor 240 acquires the image shown in FIG. At time t, it is assumed that the distal end of the insertion portion 200 has moved to the position indicated by P2 in FIG. In this case, the image acquired by the image sensor 240 is as shown in FIG. 12, and the displacement amount indicated by Q1 in FIG. 12 is recognized as blurring.

  The above-described blur problem does not use the image acquired by the image sensor 240 as a display image as it is, but displays only a partial region of the image acquired by the image sensor 240 as shown in A1 of FIG. This can be solved by using the image as an image and shifting the coordinates of this region by an amount corresponding to the displacement amount indicated by Q1.

  Specifically, at time t−1, the area indicated by A2 in FIG. 14 is used as a display image, and at time t, the area indicated by A3 in FIG. 14 is used as a display image. Here, the displacement amount indicated by Q1 in FIG. 14 is equivalent to the displacement amount indicated by Q1 in FIG. In addition, what is necessary is just to detect this displacement amount (equivalent to a motion vector) by a well-known block matching process using the image acquired at the time t-1 and the time t.

  If the above blur correction process is performed, the images used as display images at time t-1 and time t are both images shown in FIG. 15, and the image blur shown in FIG. 12 can be corrected.

  On the other hand, as described above, in endoscopic diagnosis, it is common to change the imaging field (observation field) by operating the angle knob. Therefore, when the electronic blur correction process described above is applied to an endoscope apparatus, the imaging field of view is fixed by the electronic blur correction process even though the angle knob is operated to change the imaging field of view. Cause the problem of end.

  Therefore, in the present embodiment, as described in detail later, the above-described problem is solved by performing a blur correction process in consideration of the operation of the angle knob.

  The external I / F unit 500 is an interface for performing input from the user to the endoscope apparatus, a power switch for turning on / off the power, a mode for switching shooting modes and other various modes. It includes a switching button and the like. In addition, the external I / F unit 500 outputs input information to the control unit 390.

  The interpolation processing unit 310 is connected to the blur correction processing unit 370 and the luminance image generation unit 320. The luminance image generation unit 320 is connected to the first motion vector detection unit 350 a and the second motion vector detection unit 330. The operation state acquisition unit 340 is connected to the first motion vector detection unit 350a. The first motion vector detection unit 350a and the second motion vector detection unit 330 are connected to the third motion vector calculation unit 360a. The third motion vector calculation unit 360a is connected to the shake correction processing unit 370. The blur correction processing unit 370 is connected to the display image generation unit 380. The display image generation unit 380 is connected to the display unit 400. The control unit 390 is connected to the second motion vector detection unit 330 and the shake correction processing unit 370 and controls them.

  The image acquisition unit 308 includes an interpolation processing unit 310 and a luminance image generation unit 320. The image acquisition unit 308 is not limited to this configuration, and any image acquisition unit 308 may be used as long as it can acquire a plurality of images in time series via the imaging unit 228 (insertion unit 200) that can change the imaging field of view.

  The interpolation processing unit 310 performs an interpolation process on the image acquired by the image sensor 240. As described above, since the image sensor 240 has the Bayer array color filter shown in FIG. 6, each pixel of the image acquired by the image sensor 240 has any one signal value among R, G, and B signals. The other two types of signals are missing. Therefore, the interpolation processing unit 310 performs interpolation processing on each pixel of the acquired image to interpolate the missing signal value, and each pixel has all the signal values of R, G, and B signals. Generate an image. Here, as the interpolation process, for example, a known bicubic interpolation process may be used. Here, the image generated by the interpolation processing unit 310 is referred to as an RGB image.

  The interpolation processing unit 310 outputs the generated RGB image to the shake correction processing unit 370 and the luminance image generation unit 320.

  The luminance image generation unit 320 generates a luminance image from the RGB image output from the interpolation processing unit 310. Specifically, the luminance image is generated by calculating the luminance signal Y using the following equation (1) for all the pixels of the RGB image.

  The luminance image generation unit 320 outputs the generated luminance image to the second motion vector detection unit 330 and simultaneously outputs a trigger signal to the first motion vector detection unit 350a. The first motion vector detection unit 350a can detect a motion vector at the same timing as the second motion vector detection unit 330 by operating based on the trigger signal. Details of the first motion vector detection unit 350a will be described later.

  The second motion vector detection unit 330 (second motion information detection unit, second motion information calculation unit in a broad sense), based on the luminance image output from the luminance image generation unit 320, the second motion vector (Vec_x2, Vec_y2). ) Is detected (calculated).

  Here, the configuration and operation of the second motion vector detection unit 330 will be described. As illustrated in FIG. 16, the second motion vector detection unit 330 includes a local region setting unit 331, a motion vector calculation unit 332, and a frame memory 333. The luminance image generation unit 320 and the control unit 390 are connected to the local region setting unit 331. The local region setting unit 331 is connected to the motion vector calculation unit 332. The frame memory 333 is bidirectionally connected to the motion vector calculation unit 332. The motion vector calculation unit 332 is connected to the third motion vector calculation unit 360a.

  The local area setting unit 331 sets a plurality of local areas as shown in FIG. 17 for the luminance image output from the luminance image generation unit 320. The size of the local area set here may be set to a fixed value in advance, or the external I / F unit 500 may be configured to allow the user to set an arbitrary value. In this case, the setting value set by the user is output from the external I / F unit 500 to the control unit 390. Further, the setting value is output from the control unit 390 to the local region setting unit 331.

  The local region setting unit 331 outputs the luminance image and the local region information to the motion vector calculation unit 332.

  The motion vector calculation unit 332 calculates a motion vector from the luminance image output from the local region setting unit 331 and the image held in the frame memory 333. As will be described later, the frame memory 333 holds a luminance image (hereinafter referred to as a past image) acquired one frame before the luminance image of the current frame. Accordingly, the motion vector calculated by the motion vector calculation unit 332 is an inter-frame motion vector (inter-image motion vector).

  The motion vector calculation unit 332 calculates motion vectors for all local regions set by the local region setting unit 331. Furthermore, the average value of the motion vectors calculated from all the local regions is output to the third motion vector calculation unit 360a as the second motion vector (Vec_x2, Vec_y2). Thereafter, the motion vector calculation unit 332 outputs the luminance image of the current frame to the frame memory 333. The frame memory 333 holds the luminance image output from the motion vector calculation unit 332 as a past image. Therefore, the frame memory 333 holds the luminance image of the previous frame (the previous frame in a broad sense).

  Here, for example, a known block matching process may be used to calculate the motion vector. The block matching process is a technique for searching the target image for the position of a block having a high correlation with respect to an arbitrary block of the reference image. The relative shift amount between blocks corresponds to the motion vector of the block. In the present embodiment, the luminance image corresponds to the reference image, and the past image corresponds to the target image.

  For example, an absolute error SAD may be used as a method for searching for a highly correlated block in the block matching process. This is a technique for obtaining the position of I 'having a high correlation with I, where I is the block area in the reference image and I' is the block area in the target image. SAD is defined by the following equation (2) where the pixel position in each block region is pεI and qεI ′, and the signal value of each pixel is Lp, Lq.

  Here, p and q each have a two-dimensional value, I and I 'have a two-dimensional area, and pεI indicates that the coordinate p is included in the area I. Further, “‖m 表 す” represents a process of acquiring the absolute value of the real value m.

  The operation state acquisition unit 340 acquires operation information of the angle knob in the operation unit 250. The operation information acquired here is the rotation angle [deg] of each of the horizontal angle knob (horizontal operation unit in a broad sense) and the vertical angle knob (horizontal operation unit in a broad sense). Here, the rotation angle of the horizontal angle knob is expressed as θh [deg], and the rotation angle of the vertical angle knob is expressed as θv [deg]. Further, when the angle knob is not operated (when the angle knob is in the initial position), “0 [deg]” is defined, and the clockwise direction of the angle knob is defined as a positive direction.

  The operation state acquisition unit 340 outputs the rotation angles θh and θv [deg] to the first motion vector detection unit 350a.

  The first motion vector detection unit 350a (first motion information detection unit, first motion information calculation unit in a broad sense) is output from the operation state acquisition unit 340 when a trigger signal is output from the luminance image generation unit 320. The first motion vectors (Vec_x1, Vec_y1) are detected by a method described later based on the rotation angles θh and θv [deg] to be output to the third motion vector calculation unit 360a. In the present embodiment, the first motion vector (Vec_x1, Vec_y1) represents the moving direction of the distal end of the insertion unit 200.

  As shown in FIG. 18, the first motion vector detection unit 350a includes a change amount calculation unit 351, a rotation angle storage unit 352, and a motion vector calculation unit 353a. The operation state acquisition unit 340 is connected to the change amount calculation unit 351. The change amount calculation unit 351 is bidirectionally connected to the rotation angle storage unit 352. Further, the change amount calculation unit 351 is connected to the motion vector calculation unit 353a. The motion vector calculation unit 353a is connected to the third motion vector calculation unit 360a.

  The change amount calculation unit 351 calculates the following expression (3) from the rotation angles θh and θv [deg] output from the operation state acquisition unit 340 and the rotation angles θh_mem and θv_mem [deg] held in the rotation angle storage unit 352. ) Is used to calculate the rotation angle variation θh_dif, θv_dif [deg].

  Here, as will be described later, the rotation angle storage unit 352 holds the rotation angle at the time point one frame before (when the previous trigger signal is output). Therefore, the above equation (3) corresponds to the amount of change in the rotation angle between frames.

  The change amount calculation unit 351 outputs the rotation angle change amounts θh_dif and θv_dif [deg] calculated by the above equation (3) to the motion vector calculation unit 353a. After that, the change amount calculation unit 351 outputs the rotation angles θh and θv [deg] output from the operation state acquisition unit 340 to the rotation angle storage unit 352. Therefore, the rotation angle storage unit 352 holds the rotation angle at the time point one frame before.

  The motion vector calculation unit 353a calculates the first motion vector (Vec_x1, Vec_y1) based on the rotation angle change amounts θh_dif, θv_dif [deg]. In the present embodiment, the first motion vector (Vec_x1, Vec_y1) represents the moving direction of the distal end of the insertion unit 200. Specifically, the value of the motion vector is set to “0” when the distal end of the insertion unit 200 moves in each direction of the xy axis, and “1” when the tip does not move.

For example, when the change amount of the rotation angle is “θh_dif = θv_dif = 0” (when the angle knob is not operated), it can be determined that the distal end of the insertion unit 200 has not moved, so the first motion vector (Vec_x1, Vec_y1) is (1, 1). On the other hand, when the change amount of the rotation angle is “θh_dif ≠ 0, θv_dif = 0” (when the horizontal angle knob is operated), the distal end of the insertion portion 200 moves in the horizontal direction (x-axis direction). Therefore, the first motion vector (Vec_x1, Vec_y1) is (0, 1). Similarly, when considering other cases, the first motion vectors (Vec_x1, Vec_y1) are as follows.
Change amount of rotation angle First motion vector (Vec_x1, Vec_y1)
θh_dif = 0, θv_dif = 0: (1, 1)
θh_dif ≠ 0, θv_dif = 0: (0, 1)
θh_dif = 0, θv_dif ≠ 0: (1, 0)
θh_dif ≠ 0, θv_dif ≠ 0: (0, 0)
Based on the first motion vector (Vec_x1, Vec_y1) and the second motion vector (Vec_x2, Vec_y2), the third motion vector calculation unit 360a (third motion information calculation unit in a broad sense) uses the method described later. Three motion vectors (Vec_x3, Vec_y3) are calculated.

  As illustrated in FIG. 19, the third motion vector calculation unit 360 a includes a motion vector integration unit 361 a and a motion vector storage unit 362. The first motion vector detection unit 350a, the second motion vector detection unit 330, and the control unit 390 are connected to the motion vector integration unit 361a. The motion vector integration unit 361a and the motion vector storage unit 362 are connected bidirectionally. The motion vector integration unit 361a is connected to the shake correction processing unit 370.

  The motion vector integrating unit 361a uses the first motion vector (Vec_x1, Vec_y1), the second motion vector (Vec_x2, Vec_y2), and the motion vector (Vec_x_mem, Vec_y_mem) held in the motion vector storage unit 362. Three motion vectors (Vec_x3, Vec_y3) are calculated. Here, as will be described later, the motion vector storage unit 362 holds the third motion vector at the time point one frame before (previous frame).

  The following equation (4) is used to calculate the third motion vector (Vec_x3, Vec_y3).

  Note that Cx and Cy shown in the above equation (4) are attenuation coefficients, and have positive real values smaller than 1. The effect of the attenuation coefficients Cx and Cy will be described later.

  The motion vector integration unit 361a outputs the third motion vector (Vec_x3, Vec_y3) to the blur correction processing unit 370. Further, the motion vector accumulation unit 361a outputs the third motion vector (Vec_x3, Vec_y3) to the motion vector storage unit 362. Accordingly, the motion vector storage unit 362 holds the third motion vector at the time point one frame before. Note that the initial value of the motion vector held in the motion vector storage unit 362 is (0, 0).

  Through the above processing, the integrated value of the motion vector from the initial frame to the current frame is calculated as the third motion vector (Vec_x3, Vec_y3). Here, the initial frame means an image acquired by the image sensor 240 immediately after the blur correction process is started.

  In the present embodiment, when the angle knob is operated as described above, the value of the first motion vector is “0”. Therefore, the second motion vector (inter-frame motion vector) detected during the period in which the angle knob is operated is not integrated as the third motion vector. Specifically, for example, when the horizontal angle knob is operated, the x-axis component (Vec_x1) of the first motion vector is “0”, so the inter-frame motion vector (Vec_x2) in the x-axis direction is Not accumulated. Therefore, the third motion vector does not include the movement due to the operation of the angle knob, and includes only the movement due to the influence of camera shake or the pulsation of the living body that is the subject.

  The start of the blur correction process may be configured by the user from the external I / F unit 500. For example, the blur correction process may be started when the power is turned “ON”. A switch for controlling the start / end of processing may be provided.

  The shake correction processing unit (image extraction unit) 370 extracts an image from the RGB image output from the interpolation processing unit 310 based on the third motion vector (Vec_x3, Vec_y3) output from the third motion vector calculation unit 360a. And output to the display image generation unit 380. Thereby, blur correction processing is realized. Here, the extracted image is referred to as an R′G′B ′ image. Specifically, image extraction is performed using information on the image size and start point coordinates defined by the following equations (5) and (6).

  First, the size (imx ', imy') of the R'G'B 'image is defined by the following equation (5), where the size of the RGB image is (imx, imy).

  In the above equation (5), space_x and space_y are natural numbers. For space_x and space_y, fixed values may be determined in advance, or the external I / F unit 500 may set arbitrary values.

  Further, the start point coordinates (st_x, st_y) when extracting the R′G′B ′ image are obtained by the following equation (6).

  As shown in the above equation (6), when the third motion vector (Vec_x3, Vec_y3) is smaller than space_x, space_y, based on the third motion vector (Vec_x3, Vec_y3), R′G′B 'Because the start point coordinates (st_x, st_y) when extracting the image are controlled, the blur correction process functions effectively. However, when the third motion vector (Vec_x3, Vec_y3) is larger than space_x, space_y, the start point coordinates (st_x, st_y) when extracting the R′G′B ′ image are fixed values, and blur The correction process will not work.

  On the other hand, the third motion vector (Vec_x3, Vec_y3) is an integrated value of motion vectors from the initial frame to the current frame, as shown in the above equation (4). Therefore, for example, when the blur direction has a characteristic that occurs only in a certain direction, the value of the third motion vector (Vec_x3, Vec_y3) increases with time and exceeds space_x, space_y. Is assumed. In this case, the blur correction process does not function as described above.

  Therefore, in the present embodiment, as shown in the above equation (4), the third motion vector (Vec_x_mem, Vec_y_mem) at the time point one frame before is multiplied by the attenuation coefficients Cx, Cy. Here, the subtraction coefficients Cx and Cy are real numbers smaller than 1. Accordingly, the third motion vector (Vec_x3, Vec_y3) gradually decreases with time, and it is possible to suppress the third motion vector (Vec_x3, Vec_y3) from becoming larger than space_x, space_y.

  The display image generation unit 380 performs existing white balance, color conversion, gradation conversion, and electronic zoom processing on the extracted image output from the shake correction processing unit 370 to generate a display image. Further, the display image generation unit 380 outputs the generated display image to the display unit 400.

  With the above processing, it is possible to solve the above-described problem that the imaging field of view is fixed when blur correction processing is performed on an image. Specifically, for example, when the vertical angle knob is operated, the problem that the imaging field of view is fixed can be solved by not performing the blur correction process in the y-axis direction of the image.

  In the present embodiment, as described above, the blur correction process considering the operation of the horizontal and vertical angle knobs is shown, but the present invention is not limited to this. As another method, for example, when any one of the horizontal and vertical angle knobs is operated, the blur correction process may be turned off.

  In this case, when any one of the angle knobs is operated, the blur correction processing unit 370 may output the RGB image output from the interpolation processing unit 310 to the display image generation unit 380 as it is. The operation state acquisition unit 340 may be configured to output a trigger signal indicating that the angle knob is operated to the shake correction processing unit 370 when any angle knob is operated. The problem that the imaging field of view is fixed can be solved even by using the above method.

  As described above, the image processing apparatus 300 according to the present embodiment includes the image acquisition unit 308, the first motion information detection unit 350a, the second motion information detection unit 330, and the shake correction processing unit 370 as illustrated in FIG. including. In addition, the endoscope apparatus of the present embodiment includes an image processing device 300 and an operation unit 250 for performing an operation of moving the imaging field of view of the imaging unit 228 (insertion unit 200).

  The image acquisition unit 308 acquires a plurality of images in time series via the imaging unit 228 (insertion unit 200) that can change the imaging field of view. For example, a plurality of frame images are acquired in time series.

  The first motion information detection unit 350a detects the motion information of the imaging field in accordance with the operation of moving the imaging field of the imaging unit 228 as the first motion information. In other words, the first motion information indicating the motion of the imaging field of the imaging unit 228 according to the change instruction is detected. Specifically, for example, the first motion information (first motion vector) is detected by performing the processing described in the above equation (3) or the like.

  Further, the second motion information detection unit 330 detects second motion information that is motion information between images (inter-frame motion information) based on a plurality of images (a plurality of frame images) acquired in time series. For example, the second motion information indicating the motion of the entire frame image is detected. Specifically, for example, the second motion information (second motion vector) is detected by performing the processing described in Equation (2) above.

  The blur correction processing unit 370 performs blur correction processing on the image (frame image) according to the first motion information and the second motion information. In other words, the blur correction process reflecting the first motion information and the second motion information is performed on the image obtained by the imaging unit 228. Specifically, for example, image blur correction processing is realized by performing image extraction processing as shown in the above formulas (5) and (6).

  In this way, when an operation for moving the imaging visual field consciously is performed, it is possible to suppress the occurrence of a situation in which this operation is erroneously detected as blurring and the imaging visual field is fixed. Further, by using the second motion information, for example, image blur due to the influence of the pulsation of a living body that is a subject can be suppressed.

  Here, the imaging field of view (observation field of view) is a field of view range (field of view region) where the imaging unit 228 (insertion unit) views the subject, and is a field of view range to be observed. Further, in FIG. 9, the imaging field of view can be changed is that the imaging field of view can be changed in the H direction (x-axis direction) which is the horizontal direction, the V direction (y-axis direction) which is the vertical direction, and the synthesis direction thereof. . The imaging field of view is changed according to a user's change instruction or the like. Specifically, for example, it is performed by an operation of moving the imaging visual field via the operation unit 250. Moreover, the image acquired in time series is an image (captured image) acquired over a plurality of frames, for example.

  The operation of moving the imaging field of the imaging unit 228 is an operation performed via the operation unit 250 realized by, for example, an angle knob shown in FIG. This operation may be an operation of a human user, or may be an automatic operation (operation executed by a predetermined algorithm) performed by a medical robot or the like. Further, the motion information of the imaging visual field according to the operation for moving the imaging visual field is motion information generated due to the operation of moving the imaging visual field (the instruction for changing the imaging visual field is performed), for example, FIG. This is information indicating the movement of the imaging field of view in the H direction, the V direction, and the synthesis direction thereof.

  Further, the motion information between images is, for example, information representing the motion of a subject between a plurality of images (frame images) acquired in time series, for example, motion information represented by Q1 in FIG.

  In addition, the blur correction processing according to the first motion information and the second motion information is the first motion information and the second motion information itself or information obtained from the first motion information and the second motion information (for example, third motion information). Is a blur correction process performed based on the above.

  The first motion information and the second motion information are, for example, the first motion vector and the second motion vector, respectively, but are not limited thereto. For example, information that is mathematically or arithmetically equivalent to the first motion vector and the second motion vector may be used. Further, the first motion vector and the second motion vector need only have information on the direction of the motion vector.

  In the present embodiment, the third motion information calculation unit 360a obtains the third motion information based on the first motion information and the second motion information. Then, the shake correction processing unit 370 performs shake correction processing based on the obtained third motion information. For example, the electronic blur correction process described with reference to FIGS.

  That is, such blur correction processing is normally performed based only on the second motion information that is inter-image motion information. In this regard, in the present embodiment, not only such second motion information but also blur correction processing that reflects first motion information that is motion information of the imaging field of view according to an operation of moving the imaging field of view is performed. Specifically, as described in FIG. 2 and the like, the third motion information calculation unit 360a obtains, as the third motion information, information obtained by excluding the component corresponding to the first motion information from the second motion information. Based on the third motion information obtained in this way, the blur correction process is performed. Specifically, for example, the third motion information (third motion vector) is obtained by performing the processing described in Equation (4) and the like.

  If it does in this way, a blurring correction process can be performed using the 3rd motion information which reflected the 1st motion information to the 2nd motion information. Therefore, since the third motion information is regarded as normal inter-image motion information (inter-frame motion vector), for example, the blur correction process can be realized by an algorithm such as a normal electronic blur correction process. Simplification of processing can be achieved.

  Here, in the first configuration example of FIG. 5, the processing for obtaining information excluding the component corresponding to the first motion information from the second motion information as the third motion information is shown in FIGS. 3A and 3B. This is realized by the first method described in (1).

  Specifically, the third motion information calculation unit 360a is in a non-operation state among an operation state (operation period) in which an operation for moving the imaging field of view is performed and a non-operation state in which no operation is performed (non-operation period). The third motion information is obtained based on the second motion information acquired at the time (second motion information acquired in the non-operation period). That is, the second motion information acquired in the operation state is regarded as corresponding to the motion information (first motion information) according to the operation to move the imaging field of view, and for example, the second motion information acquired in the non-operation state. The third motion information is calculated using only the motion information. As described above, for example, when the amount of change in the rotation angle by the angle knob is θh_dif = θv_dif = 0, the first motion vector, which is the first motion information, is regarded as being in a non-operation state. This can be realized by setting Vec_x1, Vec_y1) = (1, 1).

  In this way, it is possible to realize the shake correction process only by determining whether the operation state is in the non-operation state, and processing the second motion information in the non-operation state as the third motion information. become. Therefore, the process can be further simplified, and for example, a blur correction process with a small processing load can be realized.

  More specifically, the third motion information calculation unit 360a does not perform integration processing for the second motion information in the operation state, and performs integration processing for the second motion information in the non-operation state. Thus, the third motion information is obtained. Specifically, in the above-described equation (4), in the non-operation state, (Vec_x1, Vec_y1) = (1, 1) is satisfied, so that the integration process is performed on the second motion information. Three motion information (Vec_x3, Vec_y3) is obtained.

  If it does in this way, it will become possible to obtain the 3rd movement information based on the 2nd movement information acquired at the time of a non-operation state only by controlling accumulation processing, and it can attain simplification of a process. .

  As shown in FIG. 19, the third motion information calculation unit 360 a includes an integration unit 361 a (motion vector integration unit) that performs integration processing, and a storage unit 362 (motion vector storage unit) that stores the results of the integration processing. . Then, the integration unit 361a is based on the integration results up to the previous time (for example, the integration result one frame before) stored in the storage unit 362 and the current second motion information (second motion information in the current frame). Then, the integration calculation is performed to obtain the current integration result, and the obtained current integration result is written in the storage unit 362. That is, the integration process shown in the above equation (4) is realized by the third motion information calculation unit 360a configured as shown in FIG.

  In this way, calculation of the third motion information by the integration process can be performed by the third motion information calculation unit 360a having a simple configuration.

  Also, in this case, the integration unit 361a in FIG. 19 performs the previous integration result (for example, the first frame integration result), the attenuation coefficient for attenuating the previous integration result, and the second motion information of this time. Based on (the integration result of the current frame), the integration calculation is performed to obtain the current integration result. When it is determined that the magnitude of the motion represented by the third motion information is greater than a given threshold, the blur correction processing unit 370 turns off the blur correction process.

  Specifically, as shown in the above equation (4), attenuation coefficients Cx and Cy (Cx <1, Cy <1) are set, and the attenuation coefficients Cx and Cy are held in, for example, the storage unit 362. By multiplying the motion vector (Vec_x_mem, Vec_y_mem), the integration process for attenuating the previous integration result is realized.

  If such an attenuation coefficient is used, the third motion information gradually decreases with time, and the magnitude of the motion represented by the third motion information is larger than a given threshold value (space_x, space_y). It becomes possible to suppress the increase.

  Then, as described in the above equation (6), when the magnitude of the motion represented by the third motion information is larger than a given threshold value (space_x, space_y), the blur correction process is turned off. . That is, in Expression (6), the start point coordinates (st_x, st_y) when extracting the R′G′B ′ image are fixed values, so that the blur correction process is not executed.

  In this way, for example, when the blur direction has a characteristic that occurs only in a certain direction, the function of the blur correction process can be turned off. For example, as described later, when a doctor inserts or removes a scope, the direction of blurring occurs only in a certain direction. In such a case, blurring correction processing can be turned off. It becomes like this.

  In addition, the first motion information detection unit 350a detects whether or not the operation state is based on the first motion information. For example, in the above-described equation (3), when both θh_dif and θv_dif are 0, it is determined that the non-operation state (non-operation period), and when either one is not 0, the operation state ( Operation period). In this way, it is possible to determine whether the operation state or the non-operation state by only determining the first motion information, and the processing can be simplified.

  For example, the operation of moving the imaging field of the imaging unit 228 is an operation performed using the operation unit 250. In this case, the first motion information detection unit 350a detects whether or not the operation state is set based on the operation information of the operation unit 250. That is, the operation state acquisition unit 340 in FIG. 5 acquires operation information from the operation unit 250, and the first motion information detection unit 350a determines whether the operation state is based on the operation information acquired in this way. It is determined whether the operation state. In this way, it is possible to determine whether or not an operation for moving the imaging field of view has been performed via the operation unit 250 by a simple process using the operation information.

  For example, in the case of an endoscope apparatus, the operation unit 250 is the angle knob 10 in the endoscope scope shown in FIG. 8, for example. Then, the first motion information detection unit 350a detects that it is in the operation state when the rotation angle of the angle knob varies with time. Specifically, when one of θh_dif and θv_dif, which is the amount of change in the rotation angle, is not 0, it is determined that the operation state is set. On the other hand, when both θh_dif and θv_dif are 0, it is determined that the state is a non-operation state.

  In addition, the second motion information detection unit 330 detects the second motion information as a second motion vector. Then, the third motion information calculation unit 360a obtains a motion vector in a direction orthogonal to the moving direction of the imaging view among the detected second motion vectors as the third motion vector that is the third motion information.

  Specifically, as described above, when θh_dif = 0 and θv_dif = 0, the first motion vector (Vec_x1, Vec_y1) is set to (1, 1), and when θh_dif ≠ 0 and θv_dif = 0. , (Vec_x1, Vec_y1) is set to (0, 1). Then, the third motion vector (Vec_x3, Vec_y3) is obtained from the first motion vector (Vec_x1, Vec_y1) obtained in this way as in the above-described equation (4). By doing in this way, it becomes possible to obtain | require the motion vector of the direction orthogonal to the moving direction of an imaging visual field as a 3rd motion vector among 2nd motion vectors.

  In FIG. 5, the imaging unit 228 is provided at the distal end of the insertion unit 200 that is inserted into a subject such as a person. In this case, the operation of moving the imaging field of the imaging unit 228 is an operation of bending the bending member at the distal end of the insertion unit 200 and directing the distal end in a given direction. That is, this is an operation in which the bending member at the distal end of the endoscope scope which is the insertion unit 200 is bent and directed in a given direction (a direction desired by a doctor or the like).

  Specifically, the operation of moving the imaging field of the imaging unit 228 is an operation performed using the operation unit 250 that controls the bending state of the bending member. Taking the endoscope apparatus as an example, the operation unit 250 is an angle knob for controlling the bending state of the bending member as shown in FIG.

  That is, most of the insertion portion 200 is a flexible portion having flexibility, and the distal end portion is a hard portion because an illumination window, an observation window, and the like are provided. An angle portion is provided between the hard portion and the soft portion, and the angle portion is bent using an angle knob (angle operation device) so that the hard portion of the insertion portion 200 can be directed in a desired direction. Become. The operation for moving the imaging field of view is not limited to such an operation, and various modifications can be made depending on the apparatus to which the image processing apparatus of the present embodiment is applied.

  When the operation for moving the imaging field is an operation performed using the operation unit 250, the first motion information detection unit 350a performs an operation for moving the imaging field of the imaging unit based on the operation information of the operation unit 250. Detect and obtain first motion information.

  Taking an angle knob as an example, the rotation angles θh and θv are acquired by the operation state acquisition unit 340 as operation information. As described above, the rotation angle changes θh_dif and θv_dif are obtained from the rotation angles θh and θv, and the first motion vector (Vec_x1, Vec_y1) is obtained from the rotation angle changes θh_dif and θv_dif. Then, the third motion vector (Vec_x3, Vec_y3) is obtained by the above equation (4). In this way, the first motion information can be detected from the operation information with a simple detection process.

  The first motion information detection unit 350 a detects the moving direction of the imaging field of the imaging unit 228 based on the operation information of the operation unit 250. For example, the movement direction in the H direction or the V direction in FIG. 9 is detected. In this way, the direction of the first motion vector that is the first motion information can be detected.

  Specifically, the operation unit 250 includes a horizontal direction operation unit for moving the imaging field of the imaging unit in the horizontal direction H in FIG. Taking an angle knob as an example, the horizontal operation unit is a horizontal angle knob (RL angle knob) shown in FIG. In this case, the first motion information detection unit 350a detects the movement of the imaging field of the imaging unit 228 in the horizontal direction based on the operation information of the horizontal direction operation unit. For example, the movement of the imaging visual field in the horizontal direction H in FIG. 9 is detected.

  The operation unit 250 includes a vertical operation unit for moving the imaging field of the imaging unit in the vertical direction V in FIG. Taking an angle knob as an example, the vertical operation unit is a vertical angle knob (UD angle knob) shown in FIG. In this case, the first motion information detection unit 350a detects the movement of the imaging field of the imaging unit 228 in the vertical direction based on the operation information of the vertical direction operation unit. For example, the movement of the imaging visual field in the vertical direction V in FIG. 9 is detected.

  In this way, when a doctor or the like performs an operation of changing the imaging field of view in the horizontal direction or the vertical direction using an angle knob or the like, the change in the imaging field due to this operation is regarded as image blurring and the imaging field of view is changed. It becomes possible to deter the situation that is fixed.

3. Second Configuration Example FIG. 20 shows a second configuration example of the image processing apparatus of the present embodiment and an endoscope apparatus (endoscope system) including the image processing apparatus.

  The endoscope apparatus of the second configuration example of FIG. 20 is a configuration example that realizes the second technique of FIGS. 4A and 4B, and includes a light source unit 100, an insertion unit 200, and an image. The processing apparatus 300, the display part 400, and the external I / F part 500 are included. The image processing apparatus 300 of the second configuration example includes an image acquisition unit 308, a second motion vector detection unit 330, an operation state acquisition unit 340, a first motion vector detection unit 350b, and a third motion vector calculation unit. 360 b, a shake correction processing unit 370, a display image generation unit 380, and a control unit 390. The image acquisition unit 308 includes, for example, an interpolation processing unit 310 and a luminance image generation unit 320. Note that the endoscope apparatus and the image processing apparatus according to the present embodiment are not limited to the configuration shown in FIG. 20, and various modifications such as omitting some of these components or adding other components are possible. It is. The configuration of the light source unit 100, the display unit 400, the external I / F unit 500, and the like is the same as that of the first configuration example of FIG. In the following description, descriptions of processes and configurations similar to those of the first configuration example will be omitted as appropriate.

  The insertion unit 200 includes a light guide fiber 210, an illumination lens 220, an imaging unit 228, an operation unit 250, a lens control unit 260, and a memory 270. The imaging unit 228 includes, for example, a condenser lens 230 and an imaging element 240. Except for the condensing lens 230, the lens control unit 260, and the memory 270, the configuration is the same as the first configuration example of FIG. In the following description, the insertion unit 200 is also simply referred to as a scope (endoscope scope).

  The lens control unit 260 is connected to the condenser lens 230. Further, the memory 270 is connected to the control unit 390.

  The memory 270 holds an identification number unique to each scope. Therefore, the control unit 390 can identify the type of the connected scope by referring to the identification number in the memory 270.

  In the present embodiment, it is assumed that the condenser lens 230 can control the focus position d and the zoom position f. Here, the focus position d represents the distance between the condenser lens 230 and the subject when the subject is in focus. The zoom position f represents the distance between the optical center of the condenser lens 230 and the image sensor 240.

  The in-focus position d can be controlled, for example, in the range of dmin to dmax [mm] (dmax> dmin). Generally, when the in-focus position d is small, the subject can be closer to the subject, so that magnified observation is possible. The zoom position f can be controlled in the range of fmin to fmax [mm], for example. In general, when the zoom position f is large, the angle of view becomes narrow, so that magnified observation is possible. Therefore, in the present embodiment, the case where the in-focus position d is “dmax” and the zoom position f is “fmin” is defined as a normal observation state (non-magnification observation state). On the other hand, when the in-focus position d and the zoom position f are values other than those described above, it is defined as being in a magnified observation state. The focusing position d and the zoom position f of the condenser lens 230 are controlled (changed) by the lens control unit 260 (focusing position control unit, zoom position control unit).

  Moreover, what is necessary is just to set it as the structure which a user sets from the external I / F part 500, for example, the value of the focus position d and the zoom position f. In this case, first, the focus position d and zoom position f set by the user are output to the control unit 390. Then, the control unit 390 outputs the set focus position d and zoom position f to the lens control unit 260, and the lens control unit 260 changes the focus position d and zoom position f of the condenser lens 230. Further, the control unit 390 outputs the in-focus position d and the zoom position f to the first motion vector detection unit 350b described later.

  The control unit 390 is connected to the first motion vector detection unit 350b, the second motion vector detection unit 330, the blur correction processing unit 370, and the lens control unit 260. Since the processes other than the first motion vector detection unit 350b and the third motion vector calculation unit 360b are the same as those in the first configuration example, the description thereof is omitted.

  Here, the rotation angle of the angle knob (operation unit 250) and the direction of change in the imaging field of view will be described. In this embodiment, when the horizontal angle knob is rotated clockwise, the imaging field of view changes in the “Ha” direction of FIG. 21, and conversely when rotated counterclockwise, the imaging field of view is changed. It is defined as changing in the “Hb” direction of FIG. Similarly, in the case of the vertical angle knob, when the angle knob is rotated clockwise, the imaging field of view changes in the “Va” direction of FIG. 21, and when the angle knob is rotated counterclockwise. The imaging field is defined as changing in the “Vb” direction of FIG.

  In the present embodiment, for example, when the horizontal angle knob is rotated by θ [deg], the angle at the scope tip also changes by θ [deg].

  As illustrated in FIG. 22, the first motion vector detection unit 350b (first motion information detection unit) includes a change amount calculation unit 351, a rotation angle storage unit 352, and a motion vector calculation unit 353b. The control unit 390 is connected to the motion vector calculation unit 353b. The motion vector calculation unit 353b is connected to the third motion vector calculation unit 360b. The operation state acquisition unit 340 is connected to the change amount calculation unit 351. Since the change amount calculation unit 351 and the rotation angle storage unit 352 are the same as those in the first configuration example, description thereof is omitted.

  The motion vector calculation unit 353b includes the rotation angle changes θh_dif and θv_dif [deg] output from the change calculation unit 351, the in-focus position d [mm] and the zoom position f [mm] output from the control unit 390. Based on the above, the first motion vector (Vec_x1, Vec_y1) is obtained. In the present embodiment, the first motion vector (Vec_x1, Vec_y1) is a motion vector resulting from the operation of the angle knob, as will be described later.

  Here, the relationship between the change amounts θh_dif and θv_dif [deg] of the rotation angle and the motion vector on the image will be described. Here, for the sake of explanation, consider the case where only the horizontal angle knob is operated (θv_dif = 0).

  Now, assume that the tip of the insertion unit 200 and the subject are in the positional relationship shown in FIG. 23 at a certain time t-1. In this case, it is assumed that the image shown in FIG. In FIG. 23, “S” corresponds to a fulcrum when the angle of the scope is changed. “L” is a distance between the fulcrum and the tip of the insertion portion 200, and “d” is a focus position.

  Here, consider a case where the horizontal angle knob is operated counterclockwise by θh_dif at the next time t. At this time, the angle at the tip of the insertion portion 200 also changes by θh_dif as shown in FIG. In this case, the image sensor 240 acquires the image shown in FIG. “Dx_img” in FIG. 26 corresponds to a motion vector resulting from the operation of the angle knob.

  The motion vector “dx_img” resulting from the operation of the angle knob corresponds to “dx” shown in FIG. Therefore, the motion vector “dx_img” resulting from the operation of the angle knob can be calculated using the following equation (7).

  Note that “dx” illustrated in FIG. 24 is a distance (for example, [mm]) in real space, and “dx_img” illustrated in FIG. 26 is a distance [pixels] on the image. Therefore, in order to calculate the motion vector “dx_img” resulting from the operation of the angle knob, the conversion coefficient “Cod [pixels / mm]” shown in the above equation (7) is required.

  Here, the conversion coefficient “Cod” is determined by the focus position d [mm], the zoom position f [mm], and the pixel pitch P [mm / pixels] of the image sensor. Specifically, the conversion coefficient “Cod” can be calculated using the following equation (8).

  From the above, the first motion vector (Vec_x1, Vec_y1) can be calculated using the following equation (9). The motion vector calculation unit 353b calculates the first motion vector (Vec_x1, Vec_y1) using the following equation (9), and outputs the first motion vector to the third motion vector calculation unit 360b.

  In the above equations (8) and (9), the value of “L, P” varies depending on the connected scope, but the control unit 390 is connected by referring to the identification number held in the memory 270. It is possible to identify the type of scope that is present and obtain the values of “L, P”.

  Based on the first motion vector (Vec_x1, Vec_y1) and the second motion vector (Vec_x2, Vec_y2), the third motion vector calculation unit 360b calculates a third motion vector (Vec_x3, Vec_y3) by a method described later.

  As shown in FIG. 27, the third motion vector calculation unit 360b (third motion information calculation unit) includes a motion vector subtraction unit 363, a motion vector determination unit 364, an average motion vector calculation unit 365, and a motion vector integration unit. 361b and a motion vector storage unit 362. The first motion vector detection unit 350b and the second motion vector detection unit 330 are connected to the motion vector subtraction unit 363. The motion vector subtraction unit 363 is connected to the motion vector determination unit 364. The motion vector determination unit 364 is connected to the average motion vector calculation unit 365 and the motion vector integration unit 361b. The average motion vector calculation unit 365 is connected to the motion vector integration unit 361b. The motion vector integration unit 361b and the motion vector storage unit 362 are connected bidirectionally. The motion vector integration unit 361b is connected to the shake correction processing unit 370.

  The motion vector subtraction unit 363 performs a process of subtracting the first motion vector (Vec_x1, Vec_y1) from the second motion vector (Vec_x2, Vec_y2). Here, the motion vector after the subtraction process is expressed as a differential motion vector (Vec_sub_x, Vec_sub_y).

  The difference motion vector (Vec_sub_x, Vec_sub_y) is calculated using the following equation (10).

  The motion vector subtraction unit 363 outputs the difference motion vector (Vec_sub_x, Vec_sub_y) calculated by the above equation (10) to the motion vector determination unit 364.

  As described above, the first motion vector is a motion vector resulting from an angle operation (a motion vector resulting from an operation for moving the imaging field of view), and the second motion vector is an inter-frame motion vector (an inter-image motion vector). is there. Therefore, the differential motion vector calculated by the above equation (10) corresponds to a motion vector obtained by excluding the motion vector resulting from the angle operation from the inter-frame motion vector. However, this differential motion vector includes motion vectors resulting from the insertion / removal of the scope, in addition to camera shake and blur due to the pulsation of the subject.

  Therefore, when blur correction processing is performed based on the difference motion vector, there is a problem that the field of view is fixed by blur correction processing even though it is desired to change the imaging visual field by inserting or removing the scope. Therefore, in this embodiment, in order to improve the above-described problem, motion vectors resulting from the insertion / extraction of the scope are excluded by a method described later.

  The motion vector determination unit 364 performs determination processing using threshold values (th_x, th_y) on the differential motion vectors (Vec_sub_x, Vec_sub_y). As a specific determination process, for example, the following expression (11) is used.

  The motion vector determination unit 364 outputs the difference motion vector (Vec_sub_x, Vec_sub_y) after the determination processing according to the above equation (11) to the average motion vector calculation unit 365 and the motion vector integration unit 361b.

  The average motion vector calculation unit 365 calculates an average motion vector (Vec_ave_x, Vec_ave_y) from the difference motion vector (Vec_sub_x, Vec_sub_y) output from the motion vector determination unit 364.

  Here, the differential motion vector detected from the current frame (time t) is (Vec_sub_xt, Vec_sub_yt), and the differential motion vector detected n frames before the current frame is (Vec_sub_xt-n, Vec_sub_yt-n). write.

  At this time, the average motion vector (Vec_ave_x, Vec_ave_y) may be calculated using, for example, the following equation (12).

  However, in the above equation (12), M corresponds to the number of frames used for the averaging process. For example, the value of M may be M = 10. Also, Fnum represents the number of frames acquired since the start of blur correction (Fnum = 1 in the initial frame).

  The motion vector accumulation unit 361b includes a differential motion vector (Vec_sub_x, Vec_sub_y) output from the motion vector determination unit 364, an average motion vector (Vec_ave_x, Vec_ave_y) output from the average motion vector calculation unit 365, and a motion vector storage unit 362. The third motion vector (Vec_x3, Vec_y3) is calculated using the motion vector (Vec_x_mem, Vec_y_mem) held in the. Here, the motion vector storage unit 362 holds the third motion vector at the time point one frame before, as will be described later.

  The following equation (13) is used to calculate the third motion vector (Vec_x3, Vec_y3).

  The third motion vector calculation unit 360b outputs the third motion vector (Vec_x3, Vec_y3) to the shake correction processing unit 370. Further, the third motion vector calculation unit 360b outputs the third motion vector (Vec_x3, Vec_y3) to the motion vector storage unit 362. Accordingly, the motion vector storage unit 362 holds the third motion vector at the time point one frame before. Note that the initial value of the motion vector held in the motion vector storage unit 362 is (0, 0).

  Through the above processing, the integrated value of the motion vector from the initial frame to the current frame is calculated as the third motion vector (Vec_x3, Vec_y3).

  As described above, in this embodiment, the motion vector (first motion vector) resulting from the operation of the angle knob is subtracted from the inter-frame motion vector (second motion vector) using Equation (10). Thus, the third motion vector is calculated. Therefore, it is possible to solve the problem that the imaging visual field is fixed when the angle knob is operated. Furthermore, by using the methods shown in equations (11) to (13), it is possible to exclude motion vectors caused by scope insertion / removal.

  Here, the effects of the equations (11) to (13) will be described. In general, when a doctor inserts and removes a scope, the imaging field of view varies greatly. Therefore, when the scope is inserted / removed, the inter-frame motion vector becomes large, so that the difference motion vector calculated by Expression (10) also becomes large. Therefore, as shown in the equation (11), when the magnitude of the detected differential motion vector is larger than the threshold value (th_x, th_y), the differential motion vector is forcibly set to zero, and blur correction processing is performed. Do not apply.

  In addition, when the doctor inserts / removes the scope, the same operation is continued for a certain period. For this reason, the direction and magnitude of the differential motion vector detected during this period has a characteristic that it gradually changes over a plurality of frames.

  On the other hand, the motion vector resulting from the pulsation of the living body or camera shake has a feature that its direction and size change frequently. Therefore, in the present embodiment, as shown in the equation (12), by calculating the average of the difference motion vectors detected from a plurality of frames, the motion vector component due to the pulsation of the living body or camera shake is excluded. Is possible. Accordingly, the average motion vector calculated by Expression (12) is a motion vector resulting from the insertion / extraction of the scope.

  Therefore, as shown in Expression (13), by subtracting the average motion vector (Vec_ave_x, Vec_ave_y) from the differential motion vector (Vec_sub_x, Vec_sub_y), only the motion vector due to the pulsation or camera shake of the living body is obtained. It is possible to calculate.

  With the above processing, it is possible to solve the above-described problem that the imaging field of view is fixed when blur correction processing is performed on an image. Specifically, the image field of view is obtained by performing blur correction processing using the motion vector calculated by subtracting the motion vector resulting from the operation of the angle knob and the insertion / extraction of the scope from the inter-frame motion vector. Solve fixed issues.

  In the present embodiment, as described above, the blur correction process considering the operation of the horizontal and vertical angle knobs is shown, but the present invention is not limited to this. In general, blurring of an image caused by the effects of camera shake and pulsation of a subject has a feature that appears remarkably during magnified observation. Therefore, as another method, for example, the above-described blur correction process may function only during magnified observation.

  In this case, when the in-focus position d set by the user is “dmax” and the zoom position f is “fmin” (corresponding to normal observation), the blur correction processing unit 370 is output from the interpolation processing unit 310. The RGB image may be output to the display image generation unit 380 as it is. In this case, the control unit 390 does not perform the blur correction process on the blur correction processing unit 370 when the in-focus position d set by the user is “dmax” and the zoom position f is “fmin”. May be output.

  As described above, also in the second configuration example of the present embodiment illustrated in FIG. 20, the image acquisition unit 308 acquires a plurality of images in time series, and the first motion information detection unit 350 b moves the imaging field of view. The motion information of the imaging visual field corresponding to the operation is detected as the first motion information, and the second motion information detection unit 330 detects the second motion information that is motion information between images. Then, the blur correction processing unit 370 performs a blur correction process according to the first motion information and the second motion information.

  Further, the third motion information calculation unit 360b obtains the third motion information based on the first motion information and the second motion information. Specifically, information obtained by excluding the component of the first motion information from the second motion information is obtained as third motion information, and the blur correction processing unit 370 performs the blur based on the third motion information thus obtained. Perform correction processing.

  In the first configuration example of FIG. 20, unlike the first configuration example of FIG. 5, the third motion information calculation unit 360 b performs the first operation as described in FIGS. 4A and 4B. Difference information between the first motion information and the second motion information is obtained as third motion information. For example, the third motion information is obtained by performing an operation such as the above-described equation (10).

  In this way, it is possible to realize a blur correction process with higher performance than in the first configuration example. For example, when the angle operation is performed in the horizontal direction by the angle knob in the first configuration example, the blur correction process in the horizontal direction based on the second motion information that is the inter-image motion information is canceled. The same applies when an angle operation is performed in the vertical direction. That is, the first configuration example in FIG. 5 has advantages in that the processing can be simplified and the processing load can be reduced as compared with the second configuration example, but the second configuration information obtained in the operation state can be obtained. There is a disadvantage that blur correction processing based on motion information is not performed.

  On the other hand, according to the second configuration example of FIG. 20, even when an angle operation is performed in the horizontal direction or the vertical direction, the blur correction process based on the second motion information is performed in parallel therewith. It becomes possible to do. For example, when the angle operation is performed in the horizontal direction, information excluding the first motion information component corresponding to the movement in the horizontal direction from the second motion information is calculated as the third motion information. Based on the third motion information, the blur correction process between images is performed as usual. Therefore, it is possible to realize a blur correction process with higher performance than in the first configuration example.

  The first motion information detection unit 350b detects the first motion information as the first motion vector, and the second motion information detection unit 330 detects the second motion information as the second motion vector, and calculates the third motion information. The unit 360b obtains the third motion information as the third motion vector. Then, the shake correction processing unit 370 performs a shake correction process based on the third motion vector.

  In this way, it is possible to obtain the third motion vector by calculating the motion vectors, and to perform general blur correction processing using the motion vector based on the obtained third motion vector. Can be simplified.

  Specifically, the third motion information calculation unit 360b obtains a difference motion vector representing a difference between the first motion vector and the second motion vector, and obtains a third motion vector based on the difference motion vector. For example, as shown in the above equation (10), a difference motion vector (Vec_sub_x, Vec_sub_y) is obtained, and a third motion vector (Vec_x3, Vec_y3) is calculated using the difference motion vector.

  In this way, it is possible to obtain the third motion vector by a simple calculation process using vector calculation.

  If the third motion information calculation unit 360b determines that the magnitude of the difference motion vector is greater than a given threshold value, the third motion information calculation unit 360b sets the magnitude of the difference motion vector to zero. For example, as shown in the above equation (11), when the magnitude of the difference motion vector (Vec_sub_x, Vec_sub_y) is larger than the threshold value (th_x, th_y), it is forcibly set to zero and the blur correction process is performed. Don't break.

  In this way, the blur correction process can be forcibly turned off when the doctor is inserting or removing the scope.

  In addition, the third motion information calculation unit 360b may perform an average motion vector difference process to obtain an average motion vector, and obtain a third motion vector based on the difference between the difference motion vector and the average motion vector. For example, an average motion vector (Vec_ave_x, Vec_ave_y) is obtained, and a difference motion vector and an average motion vector as shown in the above equation (13) are processed to calculate a third motion vector. In this way, it is possible to obtain only the motion vector resulting from, for example, the pulsation of the living body or camera shake as the third motion vector.

  Further, as shown in FIG. 27, the third motion information calculation unit 360b stores an integration unit 361b (motion vector integration unit) that performs an integration process on the difference between the difference motion vector and the average motion vector, and stores the result of the integration process. A storage unit 362 (motion vector storage unit) is included. Then, the integration unit 361b converts the previous integration result stored in the storage unit 362 (for example, the integration result one frame before) and the difference between the current differential motion vector and the average motion vector (difference in the current frame). Based on this, an integration calculation is performed to obtain a current integration result. Then, the obtained integration result is written in the storage unit 362. That is, the integration process shown in the above equation (13) is realized by the third motion information calculation unit 360b configured as shown in FIG.

  In this way, calculation of the third motion information by the integration process can be executed by the third motion information calculation unit 360b having a simple configuration.

  In the second configuration example of FIG. 20, the focus position of the imaging unit 228 can be changed. For example, as described above, when the user sets the in-focus position using the external I / F unit 500, the lens control unit 260 changes the set in-focus position under the control of the control unit 390. Then, the first motion information detection unit 350b detects the first motion information based on the operation information of the operation for moving the imaging visual field of the imaging unit 228 and the focus position information that is information indicating the focus position. For example, as shown in the above equation (9), the first motion information (Vec_x1, Vec_y1) is detected based on (θh_dif, θv_dif) corresponding to the operation information and the focus position information d.

  In this way, by changing the in-focus position, even if the relationship between the distance in the real space and the distance on the image changes, it is possible to obtain appropriate first motion information that reflects the operation information. It becomes possible.

  In the second configuration example, the imaging unit 228 can change the zoom position in addition to the in-focus position. For example, when the zoom position is set by the external I / F unit 500, the lens control unit 260 changes to the set zoom position. Then, the first motion information detection unit 350b performs the first motion information based on (θh_dif, θv_dif) corresponding to the operation information, the focus position information d, and the zoom position information f that is information indicating the zoom position. (Vec_x1, Vec_y1) is detected.

  In this way, by changing the zoom position, even if the relationship between the distance in the real space and the distance on the image changes, it is possible to obtain appropriate first motion information that reflects the operation information. become.

  In addition, the blur correction processing unit 370 may perform the blur correction processing when the imaging unit 228 is in the enlarged observation state. For example, in the magnified observation state, the in-focus position d is reduced so that it can be closer to the subject. Further, the zoom position f is increased to narrow the angle of view so that the subject can be observed in a more magnified state.

  In such a magnified observation state, image blur due to the pulsation of the living body and camera shake is likely to occur compared to the normal observation state (non-magnification observation state).

  In this regard, if the shake correction processing unit 370 performs shake correction processing in the magnified observation state, even if there is a pulsation or a camera shake of the living body in the magnified observation state, image blur caused by the motion is minimized. It becomes possible to suppress. In addition, in the magnified observation state, when the doctor performs an angle operation using the angle knob to search for a non-specimen, it is possible to effectively prevent a problem that the imaging field of view is fixed due to the operation. become. In addition, by not performing the blur correction process in the normal observation state, it is possible to prevent a situation where an unnecessary blur correction process is performed when the blur correction is not necessary.

  In the present embodiment, the first motion information detector 350b detects the first motion information as the first motion vector. Specifically, the first motion vector detection unit 350b detects the moving direction and the moving amount of the imaging field of view as the first motion vector. The second motion information detection unit 330 also detects the second motion information as a second motion vector.

  If it does in this way, a 3rd motion vector will be obtained by vector operation of the 1st motion vector and the 2nd motion vector, and general blur amendment processing using a motion vector will be performed based on the obtained 3rd motion vector. And the processing can be simplified.

  Further, if the first motion information is detected as a first motion vector representing the moving direction and moving amount of the imaging field of view, the first motion vector reflecting not only the moving direction of the imaging field of view but also the moving amount can be obtained. it can. Then, it is possible to obtain the third motion vector from the difference between the first motion vector and the second motion vector to realize the blur correction process. Therefore, compared to the first configuration example using the first motion vector that reflects only the moving direction of the imaging field of view, it is possible to use the first motion vector that reflects the amount of movement of the imaging field of view. As a result, it is possible to realize blur correction processing that more accurately reflects the movement of the imaging field of view.

  The first and second configuration examples to which the present invention is applied and the modification examples thereof have been described above, but the present invention is not limited to these first and second configuration examples and the modification examples as they are. In the implementation stage, the constituent elements can be modified and embodied without departing from the spirit of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the first and second configuration examples and modifications described above. For example, some constituent elements may be deleted from all the constituent elements described in the first and second configuration examples and modifications. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

  In the specification or the drawings, terms (first motion vector, second motion vector) described at least once together with different terms (first motion information, second motion information, third motion information, etc.) having a broader meaning or the same meaning. , The third motion vector, etc.) can be replaced by the different terms anywhere in the specification or drawings. The configurations and operations of the image processing apparatus and the endoscope apparatus are not limited to those described in the present embodiment, and various modifications can be made.

10 (12, 14) Angle knob, 100 light source, 110 white light source,
120 lens, 200 insertion section, 210 light guide fiber,
220 illumination lens, 228 imaging unit, 230 condenser lens, 240 imaging device,
250 operation unit, 260 lens control unit, 270 memory, 300 image processing device,
308 image acquisition unit, 310 interpolation processing unit, 320 luminance image generation unit,
330 second motion vector detection unit (second motion information detection unit), 331 local region setting unit,
332 motion vector calculation unit, 333 frame memory, 340 operation state acquisition unit,
350a, 350b first motion vector detection unit (first motion information detection unit),
351 change amount calculation unit, 352 rotation angle storage unit,
353a, 353b motion vector calculation unit,
360a, 360b third motion vector calculation unit (third motion information calculation unit),
361a, 361b motion vector integration unit, 362 motion vector storage unit,
363 motion vector subtraction unit, 364 motion vector determination unit,
365 average motion vector calculation unit, 370 blur correction processing unit (image extraction unit),
380 Display image generation unit, 390 control unit, 400 display unit, 500 external I / F unit

Claims (36)

An image acquisition unit that acquires a plurality of images in time series via an imaging unit that can change the imaging field of view;
A first motion information detection unit that detects motion information of the imaging field in accordance with an operation of moving the imaging field of the imaging unit as first motion information;
A second motion information detector that detects second motion information that is motion information between images based on the plurality of images acquired in time series;
A blur correction processing unit that performs blur correction processing on the image according to the first motion information and the second motion information;
An image processing apparatus comprising: In claim 1,
A third motion information calculation unit for obtaining third motion information based on the first motion information and the second motion information;
The blur correction processing unit
An image processing apparatus that performs the blur correction process based on the third motion information. In claim 2,
The third motion information calculation unit
An image processing apparatus, wherein information obtained by excluding a component corresponding to the first motion information from the second motion information is obtained as the third motion information. In claim 2,
The third motion information calculation unit
Based on the second motion information acquired in the non-operation state among the operation state in which the operation for moving the imaging field of view is performed and the non-operation state in which the operation is not performed, the third movement information is obtained. An image processing apparatus characterized by obtaining motion information. In claim 4,
The third motion information calculation unit
The integration process for the second motion information is not performed in the operation state, and the integration process for the second motion information is performed in the non-operation state, thereby obtaining the third motion information. An image processing apparatus. In claim 5,
The third motion information calculation unit
An integration unit for performing the integration process;
A storage unit for storing the result of the integration process,
The integrating unit is
Based on the previous integration result stored in the storage unit and the current second motion information, the integration operation is performed to obtain the current integration result, and the obtained current integration result is stored in the memory. An image processing apparatus, wherein the image processing apparatus writes data to a copy unit. In claim 6,
The integrating unit is
Based on the integration result up to the previous time, the attenuation coefficient for attenuating the integration result up to the previous time, and the current second motion information, the integration calculation is performed to obtain the current integration result,
The blur correction processing unit
An image processing apparatus, wherein when it is determined that the magnitude of motion represented by the third motion information is greater than a given threshold, the blur correction processing is turned off. In claim 4,
The first motion information detection unit includes:
An image processing apparatus that detects whether or not the operation state is based on the first motion information. In claim 8,
The operation of moving the imaging field of view of the imaging unit is an operation performed using an operation unit,
The first motion information detection unit includes:
An image processing apparatus that detects whether or not the operation state is set based on operation information of the operation unit. In claim 9,
The operation unit is an angle knob in an endoscope scope,
The first motion information detection unit includes:
An image processing apparatus that detects the operation state when the rotation angle of the angle knob varies with time. In claim 2,
The second motion information detector is
Detecting the second motion information as a second motion vector;
The third motion information calculation unit
An image processing apparatus, wherein a motion vector in a direction orthogonal to a moving direction of the imaging field of view is obtained as a third motion vector that is the third motion information among the second motion vectors. In claim 2,
The third motion information calculation unit
A difference information between the first motion information and the second motion information is obtained as the third motion information. In claim 2,
The first motion information detection unit includes:
Detecting the first motion information as a first motion vector;
The second motion information detector is
Detecting the second motion information as a second motion vector;
The third motion information calculation unit
Obtaining the third motion information as a third motion vector;
The blur correction processing unit
An image processing apparatus that performs the blur correction process based on the third motion vector. In claim 13,
The third motion information calculation unit
An image processing apparatus, wherein a difference motion vector representing a difference between the first motion vector and the second motion vector is obtained, and the third motion vector is obtained based on the difference motion vector. In claim 14,
The third motion information calculation unit
An image processing apparatus, wherein when the magnitude of the difference motion vector is determined to be greater than a given threshold, the magnitude of the difference motion vector is set to zero. In claim 14,
The third motion information calculation unit
An image processing apparatus, wherein an average motion vector is obtained by performing an averaging process on the difference motion vector, and the third motion vector is obtained based on a difference between the difference motion vector and the average motion vector. In claim 16,
The third motion information calculation unit
An integration unit for performing integration processing on the difference between the difference motion vector and the average motion vector;
A storage unit for storing the result of the integration process,
The integrating unit is
Based on the integration result up to the previous time stored in the storage unit and the difference between the current difference motion vector and the average motion vector, the integration operation is performed to obtain the current integration result. An image processing apparatus, wherein the result of integration is written into the storage unit. In claim 1,
The imaging unit is provided at a distal end of an insertion unit to be inserted into a subject,
The image processing apparatus according to claim 1, wherein the operation of moving the imaging field of view of the imaging unit is an operation of bending a bending member at a distal end of the insertion unit and directing the distal end in a given direction. In claim 18,
The image processing apparatus, wherein the insertion unit is an endoscope scope. In claim 18,
The image processing apparatus according to claim 1, wherein the operation of moving the imaging field of view of the imaging unit is an operation performed using an operation unit that controls a bending state of the bending member. In claim 20,
The image processing apparatus, wherein the operation unit is an angle knob that controls a bending state of the bending member. In claim 1,
The operation of moving the imaging field of view of the imaging unit is an operation performed using an operation unit,
The first motion information detection unit includes:
An image processing apparatus, wherein the first motion information is obtained by detecting the operation of moving the imaging field of view of the imaging unit based on operation information of the operation unit. In claim 22,
The first motion information detection unit includes:
An image processing apparatus that detects a moving direction of the imaging field of view of the imaging unit based on the operation information of the operation unit. In claim 23,
The operation unit is
A horizontal operation unit for moving the imaging field of view of the imaging unit in a horizontal direction;
The first motion information detection unit includes:
An image processing apparatus that detects a movement of the imaging field of view of the imaging unit in a horizontal direction based on operation information of the horizontal direction operation unit. In claim 23,
The operation unit is
A vertical direction operation unit for moving the imaging field of view of the imaging unit in a vertical direction;
The first motion information detection unit includes:
An image processing apparatus that detects movement of the imaging field in the vertical direction of the imaging field based on operation information of the vertical direction operation unit. In claim 1,
The imaging unit can change the in-focus position,
The first motion information detection unit includes:
An image processing apparatus that detects the first motion information based on operation information of the operation for moving the imaging field of view of the imaging unit and focusing position information that is information indicating the focusing position. . In claim 26,
The imaging unit can change the in-focus position and the zoom position,
The first motion information detection unit includes:
An image processing apparatus that detects the first motion information based on the operation information, the focus position information, and zoom position information that is information representing the zoom position. In claim 1,
The blur correction processing unit
An image processing apparatus that performs the blur correction process when the imaging unit is in a magnified observation state. In claim 1,
The first motion information detection unit includes:
An image processing apparatus that detects the first motion information as a first motion vector. In claim 29,
The first motion vector detection unit includes:
An image processing apparatus that detects a moving direction and a moving amount of the imaging field of view as the first motion vector. In claim 1,
The second motion information detector is
An image processing apparatus for detecting the second motion information as a second motion vector. An image acquisition unit for acquiring a plurality of frame images in time series via an imaging unit capable of changing the imaging field of view by an instruction to change the imaging field;
A first motion information detection unit that detects first motion information indicating a motion of the imaging field of view of the imaging unit according to the change instruction;
A second motion information detector for detecting second motion information indicating the motion of the entire frame image;
A blur correction processing unit that performs blur correction processing on the frame image according to the first motion information and the second motion information;
An image processing apparatus comprising: An image processing apparatus according to claim 1;
An operation unit for performing an operation of moving the imaging field of view of the imaging unit;
An endoscopic device comprising: An image processing apparatus according to claim 32;
An operation unit for performing an operation of moving the imaging field of view of the imaging unit;
An endoscopic device comprising: A plurality of images are acquired in time series via an imaging unit capable of changing the imaging field of view,
Motion information of the imaging field corresponding to an operation of moving the imaging field of the imaging unit is detected as first motion information;
Based on the plurality of images acquired in time series, second motion information that is motion information between images is detected;
An image processing method, wherein blur correction processing corresponding to the first motion information and the second motion information is performed on the image. A plurality of frame images are acquired in time series via an imaging unit capable of changing the imaging field of view by an instruction to change the imaging field of view,
Detecting first movement information indicating movement of the imaging field of the imaging unit according to the change instruction;
Detecting second movement information indicating movement of the entire frame image;
An image processing method, wherein blur correction processing corresponding to the first motion information and the second motion information is performed on the frame image.