JP5791976B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program that process an image of a measurement object to measure the size of the measurement object.

  Conventionally, blades in a jet engine are measured using an observation jig such as an endoscope. Patent Document 1 describes a technique suitable for measuring a blade or the like. In the technique described in Patent Document 1, a subject image obtained by imaging a subject and a CG image generated by computer graphics (CG) are displayed on a monitor, and the state of the object in the CG image and the state of the subject in the subject image are displayed. It is possible to change the CG image based on the user's instruction so as to match, and to perform measurement on the CG image for the portion to be measured.

  However, in the technique described in Patent Document 1, when measurement is performed on each subject image while switching a plurality of subject images with different appearances of the subject, the state of the object in the CG image is changed every time the subject image is switched. An operation to match the state of the subject in the subject image is required. For this reason, when measuring a plurality of blades periodically arranged in the jet engine, there is a problem that the above operation frequently occurs and takes time and effort.

  In contrast, Patent Document 2 automatically matches (patterns) a two-dimensional image obtained by imaging a subject and a 3D object based on data obtained using CAD (Computer Aided Design). A technique that solves the above-mentioned trouble by matching) is described. In the technique described in Patent Literature 2, first, when a 3D object is observed from a plurality of virtual viewpoints, a projection view of the 3D object observed at each viewpoint is generated, and the plurality of projection views are collected together. Save as a 2D model. Then, matching is performed between the two-dimensional image and the generated 2D model, and the projection map having the highest similarity with the 2D model is searched, and based on the position of the viewpoint when the projection map is generated The relative position between the subject in the two-dimensional image and the camera that captured the two-dimensional image is calculated.

Japanese Patent Laid-Open No. 3-102202 US Patent Application Publication No. 2009/0096790

  In the technique described in Patent Document 2, similarity is obtained by performing matching by image processing based on edges detected from an image using a two-dimensional image and a projection view (2D model) of a 3D object. It was calculated. According to this method, since image processing based on edges is performed, the processing time required for matching is long. In addition, since image processing is performed on each of a large number of projection views, the processing time required for matching is enormous.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an image processing method, and a program capable of reducing processing time.

  The present invention has been made to solve the above-described problem, and includes an image of a measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance. A display unit for displaying, and a designation unit for designating a first point on the image of the measurement object and a second point on the image of the object based on an instruction input via the input device; Based on the result of the geometric calculation, a calculation unit that performs a geometric calculation of the first graphic with the first point as a reference and the second graphic with the second point as a reference, An adjustment unit that adjusts the posture or position of at least one of the image of the measurement object and the image of the object, and is specified based on an instruction that is input via the input device after the posture or the position is adjusted. Calculate the spatial coordinates on the object corresponding to the measured position. Based on the spatial coordinates, an image processing apparatus characterized by comprising: a measurement unit for calculating the size of the object.

  In addition, the present invention provides a display unit that displays an image of a measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance, and an input device. Based on an input instruction, a designation unit that designates a first point on the image of the measurement object, a first graphic that is based on the first point, and an image of the object are recorded in association with each other. A calculation unit that performs geometric calculation with the second graphic that is based on the second point that is used as a reference, and based on a result of the geometric calculation, at least an image of the measurement object and an image of the object An adjustment unit that adjusts one posture or position, and after the posture or the position is adjusted, spatial coordinates on the object corresponding to a measurement position designated based on an instruction input via an input device Calculate, based on the calculated spatial coordinates, the object A measuring unit for calculating a size, an image processing apparatus characterized by comprising a.

  In the image processing apparatus of the present invention, the first point and the second point include three or more points designated based on an instruction input via the input device. .

  In the image processing apparatus according to the aspect of the invention, the calculation unit may calculate the first point and the first point based on a specified order of the first point and a specified order of the second point. The geometric calculation is performed in association with two points.

  In the image processing apparatus according to the aspect of the invention, the calculation unit may use a first figure on a plane with the first point as a reference, and a point on the space of the object corresponding to the second point as a reference. The geometrical calculation with the second figure in the space is performed.

  In the image processing apparatus according to the aspect of the invention, the calculation unit may perform a geometric calculation between the first graphic on the plane and the third graphic obtained by projecting the second graphic on the space onto the plane. It is characterized by that.

  In the image processing device according to the aspect of the invention, when the first point is specified, the specifying unit further indicates the transmittance of the image of the object, and the image of the object when the second point is specified. It is characterized in that the transmittance is set to be larger than the transmittance.

  In the image processing apparatus of the present invention, the display unit displays the image of the measurement object and the image of the object, and then displays the image of the measurement object and the object until the geometric calculation is completed. The posture and position of the image are maintained, and after the geometric calculation is completed, the image of the measurement object and the image of the object whose posture or position is adjusted are displayed again.

  Further, the present invention provides a step in which a display unit displays an image of a measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance, and an input device. The designation unit designates a first point on the image of the measurement object and a second point on the image of the object based on an instruction input via The calculation unit performs a geometric calculation between the first graphic and the second graphic based on the second point, and the adjustment unit is configured to measure the measurement object based on the result of the geometric calculation. A step of adjusting the posture or position of at least one of the image of the object and the image of the object, and after the posture or the position is adjusted, the measurement unit specifies based on an instruction input via the input device The spatial coordinates on the object corresponding to the measured position Out, based on the calculated spatial coordinates, an image processing method characterized by comprising the steps of: calculating a size of the object.

  In the image processing method of the present invention, the first point and the second point include three or more points specified based on an instruction input via an input device, and the calculation unit includes: The geometric calculation is performed by associating the first point with the second point based on the designated order of the first point and the designated order of the second point. It is characterized by that.

  In addition, the present invention provides a display unit that displays an image of a measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance, and an input device. Based on an input instruction, a designation unit that designates a first point on the image of the measurement object and a second point on the image of the object, and a first that is based on the first point And a calculation unit that performs geometric calculation of the second figure based on the second point, and based on the result of the geometric calculation, an image of the measurement object and an image of the object A space on the object corresponding to a measurement position designated based on an instruction input via an input device after the posture or the position is adjusted Coordinates are calculated, and based on the calculated spatial coordinates, the size of the object is calculated. A measuring unit for calculating a is a program for causing a computer to function as a.

  According to the present invention, geometric calculation of the first graphic and the second graphic is performed, and based on the result of the geometric calculation, at least one posture or position of the measurement object image and the object image is determined. By adjusting, the processing time can be reduced.

It is a block diagram which shows the structure of the blade test | inspection system by one Embodiment of this invention. It is a block diagram which shows the structure of the endoscope apparatus with which the blade test | inspection system by one Embodiment of this invention is provided. It is a block diagram which shows the structure of the blade test | inspection system (modification) by one Embodiment of this invention. It is a block diagram which shows the structure of the blade test | inspection system (modification) by one Embodiment of this invention. It is a block diagram which shows the structure of PC with which the blade test | inspection system (modification) by one Embodiment of this invention is provided. It is a reference figure showing the screen of 3D measurement software in one embodiment of the present invention. It is a reference figure showing the relation between 3D object and camera pose in one embodiment of the present invention. It is a reference figure showing the relation between 3D object and camera pose in one embodiment of the present invention. It is a reference figure showing the relation between 3D object and camera pose in one embodiment of the present invention. It is a reference figure showing the relation between 3D object and camera pose in one embodiment of the present invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the initial value of the camera pose in one Embodiment of this invention. It is a reference figure for demonstrating the initial value of the camera pose in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the camera pose setting process in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the reference point (measurement) designation | designated process in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the reference point (3D) designation | designated process in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the matching process in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the measurement process in one Embodiment of this invention. It is a reference figure for demonstrating the content of the measurement process in one Embodiment of this invention. It is a reference figure for demonstrating the calculation method of the space coordinate in one Embodiment of this invention. It is a reference figure for demonstrating the calculation method of the space coordinate in one Embodiment of this invention. It is a reference figure for demonstrating the calculation method of the space coordinate in one Embodiment of this invention. It is a reference figure for demonstrating the calculation method of the space coordinate in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure showing a reference point and a reference figure in one embodiment of the present invention. It is a reference diagram for explaining the contents of the matching process in the pan / tilt direction in one embodiment of the present invention. It is a reference diagram showing a data list in an embodiment of the present invention. It is a reference diagram showing a data list in an embodiment of the present invention. It is a reference diagram showing a data list in an embodiment of the present invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the matching process of the roll direction in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference diagram for explaining the contents of the matching process of the zoom direction in one embodiment of the present invention. It is a graph which shows the relationship between the side length (3D) and the zoom direction position of a camera pose in one Embodiment of this invention. It is a flowchart which shows the procedure of the operation | movement by 3D measurement software in one Embodiment of this invention. It is a reference figure for demonstrating the content of the matching process of the shift direction in one Embodiment of this invention. It is a reference figure for explaining the modification of one embodiment of the present invention. It is a reference figure for explaining the modification of one embodiment of the present invention. It is a reference figure for explaining the modification of one embodiment of the present invention. It is a block diagram which shows the function structure of CPU of the control computer with which the blade test | inspection system by one Embodiment of this invention is provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration of a blade inspection system according to the present embodiment. In the jet engine 1, a plurality of turbine blades 10 (or compressor blades), which are inspection objects, are periodically arranged at predetermined intervals. Further, a turning tool 2 that rotates the turbine blade 10 in the rotation direction A at a predetermined speed is connected to the jet engine 1. In the present embodiment, the turbine blade 10 is always rotated while an image of the turbine blade 10 is captured.

  In the present embodiment, an endoscope apparatus 3 (corresponding to the image processing apparatus of the present invention) is used to acquire an image of the turbine blade 10. An endoscope insertion portion 20 of the endoscope device 3 is inserted inside the jet engine 1, and an image of the rotating turbine blade 10 is captured by the endoscope insertion portion 20. The endoscope apparatus 3 stores 3D measurement software for performing three-dimensional measurement of the turbine blade 10.

  FIG. 2 shows the configuration of the endoscope apparatus 3. The endoscope apparatus 3 includes an endoscope insertion unit 20, an endoscope apparatus main body 21, a monitor 22, and a remote controller (remote controller) 23. An imaging optical system 30a and an imaging element 30b are built in the distal end of the endoscope insertion unit 20. The endoscope apparatus body 21 includes an image signal processing unit (CCU) 31, a light source 32, a bending control unit 33, and a control computer 34.

  In the endoscope insertion unit 20, the imaging optical system 30a collects light from the subject (subject) and forms a subject image on the imaging surface of the imaging element 30b. The image sensor 30b photoelectrically converts the subject image to generate an image signal. The imaging signal output from the imaging element 30 b is input to the image signal processing device 31.

  In the endoscope apparatus body 21, the image signal processing device 31 converts the image signal from the image sensor 30 b into a video signal such as an NTSC signal and supplies it to the control computer 34, and further, as an analog video output if necessary. Output to the outside.

  The light source 32 is connected to the distal end of the endoscope insertion portion 20 through an optical fiber or the like, and can irradiate light to the outside. The bending control unit 33 is connected to the distal end of the endoscope insertion unit 20 and can bend the distal end up and down and left and right. The light source 32 and the bending control unit 33 are controlled by the control computer 34.

  The control computer 34 includes a RAM 34a, a ROM 34b, a CPU 34c, a network I / F 34d, an RS232C I / F 34e, and a card I / F 34f as external interfaces. The RAM 34a is used for temporarily storing data such as image information necessary for software operation. The ROM 34b stores a series of software (programs) for controlling the endoscope apparatus 3, and 3D measurement software described later is also stored in the ROM 34b. The CPU 34c executes various control operations and the like using the data stored in the RAM 34a according to the software instruction code stored in the ROM 34b.

  The network I / F 34d is an interface for connecting to an external PC through a LAN cable, and can develop video information output from the image signal processing device 31 to the external PC. The RS232C I / F 34e is an interface for connecting to the remote controller 23, and various operations of the endoscope apparatus 3 can be controlled by the user operating the remote controller 23. The card I / F 34f can freely attach and detach various memory cards 50 that are recording media. When the memory card 50 is attached, data such as image information stored in the memory card 50 can be taken in or recorded in the memory card 50 under the control of the CPU 34c.

  As a modification of the configuration of the blade inspection system according to the present embodiment, the configuration shown in FIG. 3 may be used. In this modification, the video terminal cable 4 and the video capture card 5 are connected to the endoscope apparatus 3, whereby the video captured by the endoscope apparatus 3 is displayed on the PC 6 (corresponding to the image processing apparatus of the present invention). Can also be incorporated into The PC 6 is illustrated as a notebook PC in FIG. 3, but may be a desktop PC or the like. The PC 6 stores 3D measurement software for performing three-dimensional measurement of the turbine blade 10.

  Further, in FIG. 3, the video terminal cable 4 and the video capture card 5 are used for capturing video to the PC 6, but a LAN cable 7 may be used as shown in FIG. The endoscope apparatus 3 includes a network I / F 34d that can develop the captured video on the LAN network. Then, the video can be captured by the PC 6 through the LAN cable 7.

  FIG. 5 shows the configuration of the PC 6. The PC 6 includes a PC main body 24 and a monitor 25. A control computer 35 is built in the PC main body 24. The control computer 35 includes a RAM 35a, an HDD (hard disk drive) 35b, a CPU 35c, and a network I / F 35d and a USB I / F 35e as external interfaces. The control computer 35 is connected to the monitor 25, and video information, a software screen, and the like are displayed on the monitor 25.

  The RAM 35a is used for temporarily storing data such as image information necessary for software operation. A series of software is stored in the HDD 35b to control the endoscope apparatus, and 3D measurement software is also stored in the HDD 35b. In the present embodiment, the storage folder for storing the image of the turbine blade 10 is set in the HDD 35b. The CPU 35c executes various control operations and the like using the data stored in the RAM 35a in accordance with the software instruction code stored in the HDD 35b.

  The network I / F 35d is an interface for connecting the endoscope apparatus 3 and the PC 6 by the LAN cable 7, and can input the video information output from the endoscope apparatus 3 to the PC 6 into the PC 6. The USB I / F 35 e is an interface for connecting the endoscope apparatus 3 and the PC 6 by the video capture card 5, and can input video information output from the endoscope apparatus 3 as analog video to the PC 6.

  The blade inspection system shown in FIGS. 3 and 4 can obtain the same effects as the blade inspection system shown in FIG. In particular, the blade inspection system shown in FIGS. 3 and 4 is effective when the performance of the endoscope apparatus is inferior to that of the PC and the operation speed of the endoscope apparatus is not sufficient.

  Next, the screen of the 3D measurement software will be described. FIG. 6 shows a main window of the 3D measurement software. A main window 600 shown in FIG. 6 is displayed on the monitor 22 when the user starts 3D measurement software. The CPU 34c performs processing based on various CUI operations in the main window 600 according to 3D measurement software.

  The display of the main window 600 is performed according to control by the CPU 34c. The CPU 34 c generates a graphic image signal (display signal) for displaying the main window 600 and outputs it to the monitor 22. When the video captured by the endoscope apparatus 3 (hereinafter referred to as a measurement image) is displayed in a superimposed manner on the main window 600, the CPU 34c converts the image data captured from the image signal processing apparatus 31 into a graphic image signal. And the processed signal (display signal) is output to the monitor 22.

  When updating the GUI display state on the main window 600, the CPU 34c generates a graphic image signal corresponding to the updated main window 600 and performs the same processing as described above. Processing related to display of windows other than the main window 600 is the same as described above. Hereinafter, the process in which the CPU 34c generates a graphic image signal for displaying (including updating) the main window 600 and the like is referred to as the process for displaying the main window 600 and the like.

  By using the GUI (graphical user interface) function, the user operates the main window 600 via the remote controller 23, moves the cursor C superimposed on the main window 600, and inputs an instruction such as a click. Various GUI operations on the main window 600 can be performed. Hereinafter, functions of various GUIs will be described.

  In the upper right part of the main window 600, a [File selection] box 610 is arranged. In addition, a [Measurement Image] box 611 is arranged in the upper left portion of the main window 600. [File selection] box 610 is a box for selecting a measurement image displayed in [Measurement image] box 611 and selecting CAD data corresponding to a 3D object displayed in [Measurement image] box 611. .

  The CAD data is data representing the three-dimensional shape of the turbine blade 10 calculated in advance using CAD. As the format of the CAD data, an STL (Standard Triangulated Language) format or the like is used. A 3D object is a CG object constructed based on the contents of CAD data. Details of the GUI and operation in the [Select File] box 610 are not described.

  The [Measurement Image] box 611 is a box for displaying a measurement image IMG obtained by imaging the turbine blade 10 that is a measurement object, and displaying a 3D object OB superimposed on the measurement image IMG. As will be described later, the user operates the [Measurement Image] box 611 to change the camera pose, specify the reference point, specify the measurement point, and the like.

  In the lower left part of the main window 600, a [Display setting] box 620 is arranged. In the [Display setting] box 620, a GUI related to the display setting of the 3D object OB displayed in the [Measurement image] box 611 is arranged. The function of each GUI in the [Display Settings] box 620 is as follows.

  [Transmittance] bar 621 is a bar for setting the display transmittance of the 3D object. The [Transmittance] bar 621 can be moved (slid) in the horizontal direction (lateral direction), and when the user moves the [Transmittance] bar 621, the display transmittance of the 3D object changes.

  For example, when the transmittance is set large, the 3D object OB is displayed with a transmittance close to transparency, and when the transmittance is set small, the 3D object OB is displayed without being transmitted. As will be described later, when the user designates a reference point in the measurement image IMG in the [Measurement Image] box 611, it is preferable to make the measurement image IMG easier to see by setting the transmittance of the 3D object OB large. Further, when the user designates a reference point for the 3D object OB, it is preferable to make the 3D object OB easier to see by setting the transmittance of the 3D object OB small.

  [Display method] radio button 622 is a radio button for setting the display method of 3D object OB. The “display method” radio button 622 has two setting items “shading” and “wire frame”. When “shading” is selected, the 3D object OB is displayed with the wire frame and the surface being filled. When “wire frame” is selected, the 3D object OB is only a wire frame as shown in FIG. Is displayed in the state.

  [Display method] radio button 623 is a radio button for setting the display color of 3D object OB. The “display method” radio button 623 has two setting items “light blue” and “yellow”. [Display method] The display color of the 3D object OB can be switched by setting the radio button 623.

  [Moving direction] radio button 624 is a radio button for setting the moving direction of the camera pose. The camera pose is a parameter indicating the orientation (pose) of the 3D object OB, that is, the direction and position from which the 3D object OB is viewed. [Movement direction] radio button 624 has two setting items of “pan / tilt” and “roll / zoom”. When “pan / tilt” is selected, the camera pose can be rotated in the pan / tilt direction by moving the cursor C up / down / left / right in the [Measurement Image] box 611 by the user. When “Roll / Zoom” is selected, the camera pose can be rotated in the roll / zoom direction by the same operation.

  Below the [Measurement Image] box 611, a [Current Position] box 630 is arranged. [Current position] box 630 is a box for displaying the surface coordinates of 3D object OB at the cursor position in real time. The surface coordinates of the 3D object OB are indicated in mm as coordinates in the spatial coordinate system. When the user moves the cursor C in the [Measurement Image] box 611, the value in the [Current Position] box 630 is also changed in real time. For example, when the cursor C is positioned on the 3D object OB, the surface coordinates of the 3D object OB are calculated and displayed in the [current position] box 630. When the cursor C is not positioned on the 3D object OB, “null” is displayed in the “current position” box 630. A method for calculating the surface coordinates of the 3D object OB will be described later with reference to FIGS.

  Below the [Current Position] box 630, a [Camera Pose] box 640 is arranged. [Camera Pose] box 640 is a box for displaying a camera pose in real time. When the user changes the camera pose, the value in the [Camera Pose] box 640 is changed to real time. The camera pose is indicated in mm as coordinates in the spatial coordinate system.

  On the right side of the [Camera Pose] box 640, a [Shift Position] box 650 is arranged. [Shift position] box 650 is a box for displaying the shift position of 3D object OB in [Measurement image] box 611. The shift position of the 3D object OB is indicated in pixels as coordinates in the plane coordinate system.

  The 3D object OB is displayed in the center of the [Measurement Image] box 611, and the display position does not change even when the camera pose is changed. However, the subject shown in the measurement image IMG is not necessarily located at the center of the image. Therefore, after the execution of the 3D matching process, which is a process for matching the measurement image IMG and the 3D object 3D, the 3D object OB is not positioned at the center of the [Measurement Image] box 611 but on the subject reflected in the measurement image. Should be.

  The shift position indicates the relative position of the 3D object OB from the center of the [Measurement Image] box 611. Hereinafter, the moving direction of the shift position in the plane coordinate system is referred to as a shift direction. The user cannot manually change the shift position arbitrarily. The shift position is calculated by the CPU 34c after executing the 3D matching process.

  A “matching / measurement” box 660 is arranged below the “file selection” box 610. In the [matching / measurement] box 660, GUIs related to 3D matching processing and measurement are arranged. The function of each GUI in the [matching / measurement] box 660 is as follows.

  [Camera pause] button 661a is a button for changing the camera pause. After the [Camera Pause] button 661a is pressed, the user can move the cursor C up, down, left, and right in the [Measurement Image] box 611a to change the camera pose. A [Reset] button 661b is arranged on the right side of the [Camera Pause] button 661a. When the [Reset] button 661b is pressed, the camera pose is set to an initial value.

  The “reference point (measurement)” button 662a is a button for designating a reference point (measurement) of the measurement image IMG. The reference point (measurement) is a point on the measurement image IMG serving as a reference when the CPU 34c executes the 3D matching process. After the “reference point (measurement)” button 662a is pressed, in the “measurement image” box 611, the user moves the cursor C and clicks or the like at the position to be designated, so that the object shown in the measurement image IMG is displayed. A reference point (measurement) can be designated for the specimen. The reference point (measurement) is indicated in pixels as coordinates in the plane coordinate system. In addition, when the [reference point (measurement)] button 662a is pressed, the display transmittance of the 3D object OB is automatically set to be large, and the measurement image IMG is easy to see. Further, a [Clear] button 662b is arranged on the right side of the [Reference point (measurement)] button 662a. When the [Clear] button 662b is pressed, all the reference points (measurement) that have already been specified are cleared. It will be in the state before specification.

  [Reference point (3D)] button 663a is a button for designating a reference point (3D) of the 3D object OB. The reference point (3D) is a point on the 3D object that serves as a reference when the CPU 34c executes the 3D matching process, similarly to the reference point (measurement). After the [Reference Point (3D)] button 663a is pressed, the user moves the cursor C in the [Measurement Image] box 611 and performs an operation such as clicking at a position where the reference point (3D) is desired to be specified. A reference point (3D) can be designated for the 3D object OB. The reference point (3D) is indicated in mm as coordinates in the spatial coordinate system. Further, after the “reference point (3D)” button 663a is pressed, the display transmittance of the 3D object OB is automatically set to be small, and the 3D object OB becomes easy to see. In addition, a [Clear] button 663b is arranged on the right side of the [Reference Point (3D)] button 663a. When the [Clear] button 663b is pressed, all the reference points (3D) that have already been specified are cleared. It will be in the state before specification.

  [3D matching] button 664 is a button for executing 3D matching processing. After the [3D matching] button 664 is pressed, the CPU 34c executes 3D matching processing based on two sets of reference points (reference point (measurement) and reference point (3D)) designated by the user. At this time, the CPU 34c performs the 3D matching process so that the positions of the two sets of reference points match each other. As a result of the 3D matching process, the subject in the measurement image IMG and the 3D object OB are displayed so as to substantially match, and the subject in the measurement image IMG and the 3D object OB are in a state suitable for measurement.

  [Measurement] button 665a is a button for designating a measurement point. A measurement point is a point used as a reference when performing measurement. After the [Measurement] button 665a is pressed, the user moves the cursor C in the [Measurement Image] box 611 and performs an operation such as clicking at a position where a measurement point is desired to be specified. Measurement points can be specified. The CPU 34c executes measurement processing based on the designated measurement point. Further, a [Clear] button 665b is arranged on the right side of the [Measurement] button 665a. When the [Clear] button 665b is pressed, all the measurement points that have already been designated are cleared, Become.

  Below the “matching / measurement” box 660, a “measurement result” box 670 is arranged. [Measurement result] box 670 is a box for displaying the measurement result. When the user designates a measurement point in the [Measurement Image] box 611, measurement processing is executed based on the designated measurement point, and the measurement result is displayed in the [Measurement Result] box 670.

  In the lower right part of the main window 600, an [End] button 680 is arranged. [Exit] button 680 is a button for ending the 3D measurement software. When the [Exit] button 680 is pressed, all the software operations are completed and the main window 600 is closed (hidden).

  Next, the relationship between the 3D object and the camera pose will be described with reference to FIG. As shown in FIG. 7, the 3D object OB1 and the viewpoint 700 are in a virtual space corresponding to the real space. Although the position of the 3D object OB1 is fixed, the position of the viewpoint 700 is freely changed by the user. The center of gravity of the 3D object OB1 has a line-of-sight center 701, and a straight line extending from the viewpoint 700 in the line-of-sight direction 702 always faces the line-of-sight center 701. The position of the line-of-sight center 701 is fixed.

  Between the 3D object OB1 and the viewpoint 700, there is a rectangular screen plane 703. The screen plane 703 corresponds to the “Measurement Image” box 611. The vertical and horizontal sizes of the screen plane 703 are fixed values. A projection image obtained by projecting the 3D object OB1 onto the screen plane 703 is the 3D object OB displayed in the [Measurement Image] box 611.

  The screen plane 703 is always perpendicular to the line-of-sight direction 702, and a straight line extending from the viewpoint 700 in the line-of-sight direction 702 always passes through the center 704 of the screen plane 703. The distance 706 from the viewpoint 700 to the center 704 of the screen plane 703 is a fixed value, but the distance from the viewpoint 700 to the line-of-sight center 701 can be freely changed by the user.

  The direction in which the screen plane 703 is directed is indicated by an upward vector 705. The upward vector 705 is a unit vector that is parallel to the screen plane 703 and indicates which direction the upward direction of the screen plane 703 is.

  Among the items shown in FIG. 7, there are three parameters constituting the camera pose: a viewpoint position, a line-of-sight center position, and an upward vector. The relationship between the 3D object and the camera pose when the camera pose is changed will be described below with reference to FIGS.

  FIG. 8A shows the relationship between the 3D object and the camera pose when the camera pose is changed in the pan / tilt direction. The pan direction is a direction (pan direction 803) when the viewpoint 800 is moved perpendicularly to the upward vector 802 while the distance from the viewpoint 800 to the line-of-sight center 801 is fixed. The tilt direction is a direction (tilt direction 804) when the viewpoint 800 is moved in parallel with the upward vector 802 while the distance from the viewpoint 800 to the line-of-sight center 801 is fixed. When the camera pose is changed in the pan / tilt direction as shown in FIG. 8A, the 3D object OB projected on the screen plane 805 is rotated in the vertical and horizontal directions as shown in FIG. 8B. I understand.

  FIG. 9A shows the relationship between the 3D object and the camera pose when the camera pose is changed in the roll direction. The roll direction is a direction (roll direction 904) when the screen plane 903 is rotated around the axis of the viewpoint direction 902 from the viewpoint 900 toward the viewpoint center 901 while the position of the viewpoint 900 is fixed. When the camera pose is changed in the roll direction as shown in FIG. 9A, the 3D object OB projected on the screen plane 903 rotates around the center of the screen plane 903 as shown in FIG. 9B. I understand.

  FIG. 10A shows the relationship between the 3D object and the camera pose when the camera pose is changed in the zoom direction. The zoom direction is a direction (zoom direction 1003) when the viewpoint 1001 is moved in parallel with the line-of-sight direction 1002 while the upward vector 1000 is fixed. As shown in FIG. 10A, when the camera pose is changed in the zoom direction, the 3D object OB projected on the screen plane 1004 is enlarged or reduced as shown in FIG. 10B.

  As described above, when the camera pose is changed, the position / direction of the screen plane changes. Accordingly, the display of the 3D object projected on the screen plane also changes, and as a result, the display of the 3D object displayed in the [Measurement Image] box 611 also changes. The CPU 34c detects a camera pose change instruction input by the user via the remote controller 23, and performs processing for displaying the 3D object in the [Measurement Image] box 611 in accordance with the change instruction.

  Next, the operation flow of the 3D measurement software will be described with reference to FIG. In the following description, not only operations related to all GUIs in the main window 600 but only operations related to some GUIs will be described. Specifically, [Measurement Image] box 611, [Camera Pose] button 661a, [Reference Point (Measurement)] button 662a, [Reference Point (3D)] button 663a, [3D Matching] button 664, [Measurement] button 665a, [Measurement Result] box 670, and [End] button 680 operations will be described, but other GUI operations will not be described.

  In step SA, the CPU 34c activates the 3D measurement software. Specifically, based on an activation instruction input by the user via the remote controller 23, the CPU 34c reads 3D measurement software stored in the ROM 34b into the RAM 34a and starts an operation according to the 3D measurement software. In step SB, the CPU 34c performs a process for displaying the main window 600.

  In step SC, the CPU 34c performs an initialization process. The initialization process is a process of setting initial states of various GUIs in the main window 600 and setting initial values of various data recorded in the RAM 34a. Details of the initialization process will be described later.

  In step SD, the CPU 34c performs camera pose setting processing. The camera pose setting process is a process for roughly matching the subject in the measurement image in the [Measurement Image] box 611 with the 3D object based on an instruction to change the camera pose input by the user. Details of the camera pose setting process will be described later.

  In step SE, the CPU 34c performs a reference point (measurement) designation process. The reference point (measurement) designation process is a process for designating (setting) a reference point based on an instruction for designating a position on the subject in the measurement image in the [Measurement Image] box 611 input by the user. is there. Details of the reference point (measurement) designation process will be described later.

  In step SF, the CPU 34c performs a reference point (3D) designation process. The reference point (3D) designation process is a process for designating (setting) a reference point based on an instruction for designating a position on the 3D object in the [Measurement Image] box 611 input by the user. Details of the reference point (3D) designation process will be described later.

  In step SG, the CPU 34c performs 3D matching processing. The 3D matching process is to match the measurement image displayed in the [Measurement Image] box 611 with the 3D object based on two sets of reference points (reference point (measurement) / reference point (3D)) designated by the user. It is a process to make. Details of the 3D matching process will be described later.

  In step SH, the CPU 34c performs a measurement process. In the measurement process, a measurement point is designated (set) based on an instruction for designating a position on a 3D object in the [Measurement Image] box 611 input by the user, and based on the designated measurement point, the object is measured. This is a process of calculating the size. Details of the measurement process will be described later.

  In step SI, the CPU 34c confirms whether or not the [Finish] button 680 has been pressed by the user. If the user presses the [END] button 680, the process proceeds to step SJ. If the user does not press the [Finish] button 680, the process proceeds to step SD. In step SJ, the CPU 34c hides the main window 600 and ends the operation of the 3D measurement software.

  Next, the operation flow of the initialization process in step SC will be described with reference to FIG. In step SC1, the CPU 34c reads a predetermined measurement image file and CAD data recorded on the memory card 50 into the RAM 34a. In step SC2, the CPU 34c calculates a camera pose (initial value) based on the read CAD data.

  Regarding the viewpoint position in the camera pose, as shown in FIG. 13, the CPU 34c sets the coordinates (x, y, z) = (0, 0, 0) of the origin of the spatial coordinate system as an initial value (viewpoint 1300). As for the line-of-sight center position in the camera pose, as shown in FIG. 13, the CPU 34c calculates the center-of-gravity position of all spatial coordinates in the CAD data, and sets the coordinates as initial values (viewpoint center 1301). This line-of-sight center position is a unique value for each CAD data, and this value will not change even if the camera pose is changed thereafter. Of the camera poses, for the upward vector, as shown in FIG. 13, the CPU 34c sets the unit vector having the largest Z-direction component among the unit vectors orthogonal to the straight line connecting the viewpoint 1300 and the line-of-sight center 1301 as the initial value (upward Vector 1302). Here, the unit vector having the largest Z-direction component is used as the initial value of the upward vector, but it is not limited to the unit vector having the largest Z-direction component.

  In step SC3, the CPU 34c records the camera pose (initial value) calculated in step SC2 in the RAM 34a as the current camera pose. The current camera pose is a currently set camera pose, and the 3D object is displayed based on the current camera pose.

  In step SC4, the CPU 34c displays a measurement image IMG in the [Measurement Image] box 611 as shown in FIG. 14, and further executes a process for superimposing and displaying the 3D object OB with a predetermined transmittance. At this time, the 3D object OB is displayed as a plan view projected on the screen plane based on the calculated camera pose (initial value). When the process of step SC4 ends, the initialization process ends.

  Next, the flow of the camera pose setting process in step SD will be described with reference to FIG. In step SD1, the CPU 34c confirms whether or not the [camera pause] button 661a has already been pressed (the process in step SD3 has already been performed). If the [camera pause] button 661a is pressed, the process proceeds to step SD4. If the [camera pause] button 661a is not pressed, the process proceeds to step SD2.

  In step SD2, the CPU 34c checks whether or not the [Camera Pause] button 661a has been pressed by the user. If the [Camera Pause] button 661a is pressed, the process proceeds to step SD3. If the [Camera Pause] button 661a is not pressed, the camera pose setting process ends.

  In step SD3, the CPU 34c performs processing for highlighting the [Camera Pause] button 661a as shown in FIG. The [Camera Pose] button 661a is highlighted in order to notify the user that the camera pose can be changed at present.

  In step SD4, as shown in FIG. 16B, the CPU 34c moves the cursor C up, down, left, and right while operating the remote controller 23 and clicking the cursor C in the [Measurement Image] box 611 ( Drag operation) is detected, and the camera pose is changed based on the result of detecting the operation of the cursor C. At this time, the user changes the camera pose so that the subject DUT shown in the measurement image and the 3D object OB roughly match. The camera pose can be changed in the aforementioned pan, tilt, roll, and zoom directions. At this time, the CPU 34c detects an operation instruction of the cursor C input via the remote controller 23, and calculates the changed camera pose based on the operation instruction.

  In step SD5, the CPU 34c overwrites and records the changed camera pose as the current camera pose in the RAM 34a. In step SD6, the CPU 34c performs a process for redisplaying the 3D object based on the current camera pose. As a result, the 3D object OB whose camera pose has been changed is displayed in the [Measurement Image] box 611 as shown in FIG. When the process of step SD6 ends, the camera pose setting process ends.

  Next, the flow of the reference point (measurement) designation process in step SE will be described with reference to FIG. In step SE1, the CPU 34c confirms whether or not the [reference point (measurement)] button 662a has already been pressed (steps SE3 and SE4 have already been performed). If the [reference point (measurement)] button 662a is pressed, the process proceeds to step SE5. If the [reference point (measurement)] button 662a is not pressed, the process proceeds to step SE2. .

  In step SE2, the CPU 34c confirms whether or not the [reference point (measurement)] button 662a has been pressed by the user. If the [reference point (measurement)] button 662a is pressed, the process proceeds to step SE3. If the button is not pressed, the reference point (measurement) designation process ends.

  In step SE3, the CPU 34c performs processing for highlighting the [reference point (measurement)] button 662a as shown in FIG. The reason why the [reference point (measurement)] button 662a is highlighted is to notify the user that a reference point can be designated for the measurement image.

  In step SE4, the CPU 34c performs processing for changing the transmittance of the 3D object OB and redisplaying the 3D object OB with the changed transmittance as shown in FIG. 18B. The transmittance set here is a large value, and the 3D object OB is almost transparent, and the measurement image is easy to see. Although not particularly illustrated, when a reference point (3D) that has already been specified exists, the 3D object OB is temporarily hidden. This is also to make the measurement image easy to see.

  In step SE5, the CPU 34c operates the remote controller 23 and clicks with the cursor C in order to designate a reference point (measurement) for the subject DUT shown in the measurement image in the [Measurement Image] box 611. The operation to be performed is detected, and the coordinates of the designated reference point are calculated based on the result of detecting the operation of the cursor C. The coordinates of the reference point (measurement) calculated at this time are plane coordinates (pixel units) in the measurement image.

  In step SE6, the CPU 34c records the coordinates of the designated reference point (measurement) in the RAM 34a. In step SE7, the CPU 34c performs processing for displaying the designated reference point (measurement) in a superimposed manner on the measurement image. As a result, as shown in FIG. 18C, the reference points (measurements) R1, R2, and R3 are superimposed and displayed on the measurement image. When the process of step SE7 ends, the reference point (measurement) designation process ends.

  Next, the flow of the reference point (3D) designation process in step SF will be described with reference to FIG. In step SF1, the CPU 34c checks whether or not the [reference point (3D)] button 663a is already pressed (steps SF3 and SF4 have already been performed). If the [reference point (3D)] button 663a is pressed, the process proceeds to step SF5. If the [reference point (3D)] button 663a is not pressed, the process proceeds to step SF2. To do.

  In step SF2, the CPU 34c checks whether or not the [reference point (3D)] button 663a is pressed by the user. If the [reference point (3D)] button 663a is pressed, the process proceeds to step SF3. If the [reference point (3D)] button 663a is not pressed, the reference point (3D) designation process ends.

  In step SF3, the CPU 34c performs processing for highlighting the [reference point (3D)] button 663a as shown in FIG. The reason why the [reference point (3D)] button 663a is highlighted is to notify the user that a reference point can be currently specified for the 3D object.

  In step SF4, the CPU 34c performs processing for changing the transmittance of the 3D object OB and redisplaying the 3D object OB with the changed transmittance as shown in FIG. 20B. The transmittance set here is a small value, and the 3D object OB is easy to see. At this time, although not particularly illustrated, if a reference point (measurement) that has already been specified exists, the measurement image is temporarily hidden. This is also for making the 3D object OB easy to see.

  In step SF5, the CPU 34c operates the remote controller 23 and clicks with the cursor C in order to designate a reference point (3D) for the subject DUT shown in the measurement image in the [Measurement Image] box 611. The operation to be performed is detected, and the coordinates of the designated reference point are calculated based on the result of detecting the operation of the cursor C. The coordinates of the reference point (3D) calculated at this time are spatial coordinates (in mm units) on the surface of the 3D object, and the CPU 34c first calculates the plane coordinates (in pixel units) of the designated reference point. Spatial coordinates (in mm units) are calculated from the calculated plane coordinates.

  The reference point (3D) designated by the user must be associated with the reference point (measurement) already designated. In the present embodiment, the CPU 34c associates the reference point (3D) with the reference point (measurement) based on the order in which the user designates the reference point (measurement) and the order in which the reference point (3D) is designated. And More specifically, the CPU 34c associates the point designated first among the reference points (measurement) with the point designated first among the reference points (3D), and the second among the reference points (measurement). The point designated in the reference point (3D) is associated with the second designated point,..., The nth designated point of the reference point (measurement) and the reference point (3D) The nth designated point is associated. The above method is an example, and the present invention is not limited to this.

  As shown in FIG. 18C, the upper left reference point (measurement) R1, the upper right reference point (measurement) R2, and the lower right reference point (measurement) R3 of the subject DUT are designated. After the designation of the reference point (measurement) is completed, the user designates the reference point (measurement) for the reference point (3D) on the 3D object OB corresponding to the reference point (measurement) on the subject DUT. Specify in the same order. As shown in FIG. 20 (c), reference points (measurements) R1 ′, R2 ′, R3 ′ are designated at positions on the 3D object corresponding to the reference points (measurements) R1, R2, R3 on the subject DUT. ing.

  In step SF6, the CPU 34c records the coordinates of the designated reference point (3D) in the RAM 34a. In step SF7, the CPU 34c performs processing for displaying the designated reference point (3D) in a superimposed manner on the 3D object. As a result, as shown in FIG. 20C, the reference points (3D) R1 ', R2', R3 'are superimposed and displayed on the 3D object OB. When the process of step SF7 ends, the reference point (3D) designation process ends.

  Note that the CPU 34c may record the coordinates of the designated reference point (3D) in the CAD data, or may record it in another file linked (associated) with the CAD data. As a result, when the same CAD data is read out again in step SC1, the processing in steps SF1 to SF5 can be omitted. Further, the reference point (3D) is not necessarily designated in step SF, and may be recorded in the CAD data in advance by the endoscope device 3 or the PC 6 or linked to the CAD data ( It may be recorded in another file (associated).

  Next, the flow of the 3D matching process in step SG will be described with reference to FIG. In step SG1, the CPU 34c confirms whether or not the [3D matching] button 664 has been pressed by the user. If the [3D matching] button 664 is pressed, the process proceeds to step SG2. If the [3D matching] button 664 is not pressed, the 3D matching process ends.

  In step SG2, the CPU 34c confirms whether all reference points have already been designated. Specifically, the CPU 34c checks whether or not three reference points (measurement) and three reference points (3D) have already been specified. If all reference points have been specified, the process proceeds to step SG3. If no reference point has been specified, the 3D matching process ends. In step SG3, the CPU 34c reads the coordinates of all reference points recorded in the RAM 34a.

  In step SG4, the CPU 34c performs a matching process in the pan / tilt direction based on the coordinates of the designated reference point. Details of the matching process in the pan / tilt direction will be described later. In step SG5, the CPU 34c performs a roll direction matching process based on the coordinates of the designated reference point. Details of the roll direction matching processing will be described later.

  In step SG6, the CPU 34c performs zoom direction matching processing based on the coordinates of the designated reference point. Details of the matching processing in the zoom direction will be described later. In step SG7, the CPU 34c performs a shift direction matching process based on the coordinates of the designated reference point. Details of the matching process in the shift direction will be described later.

  In step SG8, the CPU 34c performs processing for redisplaying the 3D object in the [Measurement Image] box 611. At this time, the 3D object is displayed with its posture and position adjusted based on the camera pose and shift position finally calculated in steps SG4 to SG7. FIG. 22 shows the subject and the 3D object shown in the measurement image after the matching process. As shown in FIG. 22, it can be seen that the subject shown in the measurement image and the 3D object substantially match, that is, the two match well. When the process of step SG8 ends, the 3D matching process ends.

  Next, the flow of the measurement process in step SH will be described with reference to FIG. In step SH1, the CPU 34c confirms whether or not the [Measurement] button 664 has already been pressed (step SH3 has already been performed). If the [Measure] button 664 is pressed, the process proceeds to step SH4. If the [Measure] button 664 is not pressed, the process proceeds to step SH2.

  In step SH2, the CPU 34c checks whether or not the [Measurement] button 664 has been pressed by the user. If the [Measure] button 664 is pressed, the process proceeds to step SH3. If the [Measure] button 664 is not pressed, the measurement process ends.

  In step SH3, the CPU 34c performs processing for highlighting the [Measurement] button 664 as shown in FIG. The [Measurement] button 664 is highlighted in order to notify the user that a measurement point can be currently specified for the 3D object.

  In step SH4, the CPU 34c detects an operation in which the user operates the remote controller 23 and clicks with the cursor C in order to specify a measurement point for the 3D object in the [Measurement Image] box 611. Based on the result of detecting the operation, the coordinates of the designated measurement point are calculated. The CPU 34c records the calculated coordinates of the measurement point in the RAM 34a. The coordinates of the measurement points recorded at this time are spatial coordinates (in mm units) on the 3D object surface.

  In step SH5, the CPU 34c calculates a measurement result based on the coordinates of the designated measurement point, and records the calculated measurement result in the RAM 34a. In the present embodiment, the spatial distance between two measurement points (distance between two points) is measured. The measurement result calculated in step SH5 is a spatial distance between two measurement points that have already been specified. When the designated measurement point is only one point, the measurement result is not calculated, and the process proceeds to step SH6.

  In step SH6, the CPU 34c performs a process for superimposing and displaying the designated measurement point on the 3D object as shown in FIGS. FIG. 24B shows the [Measurement Image] box 611 when the first measurement point P1 is designated, and FIG. 24C shows the [Measurement Image] when the second measurement point P2 is designated. ] Box 611. Further, the CPU 34c performs processing for displaying the calculated measurement result in the [Measurement Result] box 670 as shown in FIG. When the designated measurement point is only one point, the measurement result is not displayed, and the process of step SH6 ends.

  When the process of step SH6 ends, the measurement process ends. In the above description, the case of measuring the distance between two points is taken as an example, but the same applies to the case of measuring an area or the like where three or more measurement points are designated.

  Next, a method of calculating spatial coordinates (three-dimensional coordinates) on the surface of the 3D object at the designated measurement point will be described with reference to FIGS. FIG. 26 shows the relationship between a part of the 3D object and the viewpoint E in the three-dimensional space.

  The 3D object is composed of a plurality of triangular spatial planes. The direction from the viewpoint E toward the center of gravity G of the 3D object is defined as the line-of-sight direction. A screen plane SC that intersects the line-of-sight direction perpendicularly is set between the viewpoint E and the 3D object.

  When the user designates a measurement point on the 3D object in the [Measurement Image] box 611, the CPU 34c sets the measurement point S on the screen plane SC as shown in FIG. A spatial straight line passing through the measurement point S and the viewpoint E is defined as a straight line L. Then, the CPU 34c searches for all the triangles that intersect with the straight line L from among a plurality of triangles constituting the 3D object. As a method for determining whether or not a straight line and a space triangle intersect, for example, Tomas Moller's intersection determination method can be used. In this example, it is determined that the triangles T1 and T2 are triangles that intersect with the straight line L as shown in FIG.

  Then, as shown in FIG. 29, the CPU 34c calculates intersections between the straight line L and the triangles T1 and T2, and sets the calculated intersections as intersections F1 and F2. Here, since it is desired to calculate the spatial coordinates on the surface of the 3D object, the CPU 34c selects one of the intersections F1, F2 that is closer to the viewpoint E. In this case, the CPU 34c calculates the spatial coordinates of the intersection point F1 as the spatial coordinates on the 3D object surface. In the above description, only two triangles determined to intersect the straight line L are used. However, depending on the shape of the 3D object and the line-of-sight direction, it may be determined that more triangles intersect. Also in this case, the intersection of the straight line L and the triangle is obtained, and the intersection closest to the viewpoint E is selected from the obtained intersections.

  As described above, the spatial coordinates of the measurement point can be calculated. The spatial coordinates of the reference point (3D) in step SF5 can also be calculated in the same manner as described above.

  Next, the flow of matching processing in the pan / tilt direction in step SG4 will be described with reference to FIG. The purpose of the matching process in the pan / tilt direction is that a triangle formed by reference points (measurements) and a triangle formed by projected points obtained by dropping the reference points (3D) on the screen plane are closest to each other. It means finding a camera pose. When these triangles are similar, the pan / tilt direction of the line of sight when the subject imaged in the measurement image is imaged and the pan / tilt direction of the line of sight observing the 3D object are substantially the same. It can be said.

  Hereinafter, as shown in FIG. 31A, projection points Rp1 'to Rp3' obtained by dropping the reference points (3D) R1 'to R3' on the screen plane 3100 are referred to as projection points (3D). Furthermore, as shown in FIG. 31B, a triangle 3102 constituted by reference points (measurements) R1 to R3 is referred to as a reference graphic (measurement), and as shown in FIG. A triangle 3101 constituted by Rp1 ′ to Rp3 ′ is referred to as a reference graphic (3D).

  In step SG401, the CPU 34c calculates a vertex angle (measurement) and records the calculated vertex angle (measurement) in the RAM 34a. The vertex angle (measurement) is the angles A1 to A3 of the three vertices R1 to R3 of the reference graphic (measurement) 3200 as shown in FIG.

  In step SG402, the CPU 34c rotates the camera pose by −31 degrees in the pan / tilt directions, respectively. Steps SG403 to SG407 are repeatedly performed because the vertex angle (3D) is sequentially calculated while rotating the camera pose in the pan / tilt direction as shown in FIG. 32 (b). The vertex angle (3D) is angles A1 'to A3' of three projection points (3D) Rp1 'to Rp3' of the reference graphic (3D) 3201, as shown in Fig. 32 (b).

  As described above, the reference point (measurement) and the reference point (3D) are associated with each other in the order in which the reference points are designated, and the angles A1 to A3 and the angles A1 ′ to A3 are also associated with each other in this order. It has been. In FIG. 32, the angle A1 and the angle A1 'are associated, the angle A2 and the angle A2' are associated, and the angle A3 and the angle A3 'are associated.

  In step SG403, the CPU 34c rotates the camera pose by +1 deg in the pan direction. In steps SG403 to SG407, the CPU 34c repeats the process until the rotation angle in the pan direction of the camera pose reaches +30 deg. The CPU 34c rotates the camera pose in the pan direction by +1 deg from -30 deg to +30 deg. As a result, the series of processes of steps SG403 to SG407 are repeated 61 times.

  In step SG404, the CPU 34c rotates the camera pose by +1 deg in the tilt direction. In steps SG404 to SG407, the CPU 34c repeats the process until the rotation angle in the tilt direction of the camera pose reaches +30 deg. The CPU 34c rotates the camera pose in the tilt direction by +1 deg from -30 deg to +30 deg. As a result, the processes of steps SG404 to SG407 are repeated 61 times. In the repetition processing of steps SG403 to SG407, the camera pose is rotated from −30 deg to +30 deg. However, the range in which the camera pose is rotated is not necessarily within this range.

  In the camera pose setting process of step SD, when the user changes the camera pose, the camera pose is set in the repetition process of steps SG403 to SG407 depending on the degree of matching between the subject shown in the measurement image and the 3D object. The range required for rotation will change. When the range is wide, the user only needs to perform rough matching, but instead, the processing time of 3D matching becomes longer. If the range is narrow, the processing time for 3D matching may be short, but instead the user needs to perform matching in some detail.

  In step SG405, the CPU 34c records the current rotation angle in the pan / tilt direction in the RAM 34a. FIG. 33 shows the rotation angle recorded in the RAM 34a. In step SG405, the CPU 34c does not record the rotation angle in the RAM 34a. Instead, every time the camera pose is rotated in the pan / tilt direction, the CPU 34c presents the data list prepared in the RAM 34a as shown in FIG. The rotation angle in the pan / tilt direction is additionally recorded line by line. As will be described later, various data such as the vertex angle (3D) can be recorded in this data list in association with the rotation angle in the pan / tilt direction.

  In step SG406, the CPU 34c calculates a projection point (3D) and records the calculated projection point (3D) in the RAM 34a. In step SG407, the CPU 34c calculates the vertex angle (3D) and records the calculated vertex angle (3D) in the RAM 34a. At this time, as shown in FIG. 33 (b), the CPU 34c associates the vertex angle (3D) with the rotation angle in the pan / tilt direction, and additionally records the data list line by line.

  When the repetition process of steps SG403 to SG407 is completed, the process proceeds to step SG408. The data list at this point is as shown in FIG. 33C, and is composed of data for 61 × 61 rows. In step SG408, the CPU 34c rotates the camera pose by −30 degrees in the pan / tilt directions. Since the rotation angle in the pan / tilt direction is +30 degrees at the time when the repetition processing of steps SG403 to SG407 is completed, the camera pose is returned to the original state here by rotating by −30 degrees.

In step SG409, the CPU 34c calculates the difference between the vertex angle (measurement) and the vertex angle (3D). Specifically, the CPU 34c calculates the absolute values D1 to D3 of the differences between the vertex angles (measurement) A1 to A3 and the vertex angles (3D) A1 ′ to A3 ′ as in the expressions (1) to (3). calculate.
D1 = | A1-A1 ′ | (1)
D2 = | A2-A2 ′ | (2)
D3 = | A3-A3 ′ | (3)
Furthermore, as shown in FIG. 34A, the CPU 34c additionally records the difference in the vertex angle in the data list in association with the rotation angle in the pan / tilt direction.

  In step SG410, the CPU 34c calculates an average value of the differences D1 to D3. Further, the CPU 34c additionally records the average value in the data list in association with the rotation angle in the pan / tilt direction as shown in FIG. 34 (b).

  In step SG411, the CPU 34c searches the data list for the minimum average value. FIG. 35 shows a state where 0.5 is searched as the minimum value in the data list.

  In step SG412, the CPU 34c reads from the data list the rotation angle in the pan / tilt direction when the average value is minimum. Specifically, as shown in FIG. 35, the CPU 34c reads the rotation angle in the pan / tilt direction associated with the minimum average value from the data list.

  In step SG413, the CPU 34c rotates the camera pose in the pan / tilt direction by the rotation angle read in step SG412. When a 3D object is displayed in this camera pose, as shown in FIGS. 32C and 32D, the vertex angle (measurement) after rotation and the vertex angle (3D) are in good agreement, and the reference figure ( It can be seen that the measurement) and the reference figure (3D) are similar.

  In step SG414, the CPU 34c overwrites and records the camera pose at this time in the RAM 34a as the current camera pose. Here, the 3D object is not re-displayed based on the current camera pose. When the process of step SG414 ends, the matching process in the pan / tilt direction ends.

  Next, the flow of the matching process in the roll direction in step SG5 will be described using FIG. The purpose of the matching process in the roll direction is to find a camera pose that most closely matches the angle in the rotation direction of the reference figure (measurement) and the reference figure (3D). When the angle of rotation of each reference figure is close, the rotation angle of the line of sight observing the subject in the measurement image and the rotation angle of the line of sight observing the 3D object are almost the same. It can be said that

  In step SG501, the CPU 34c calculates a relative angle (measurement) and records the calculated relative angle (measurement) in the RAM 34a. The relative angle (measurement) is an angle Ar1 to Ar3 between a straight line 3700 extending in the vertical direction in the measurement image and the three sides of the reference graphic (measurement) as shown in FIG. The relative angle (measurement) at this time is an angle in the clockwise direction from the straight line 3700 to the side.

  In step SG502, the CPU 34c calculates a projection point (3D) and records the calculated projection point (3D) in the RAM 34a. In step SG503, the CPU 34c calculates a relative angle (3D) and records the calculated relative angle (3D) in the RAM 34a. The relative angle (3D) is an angle Ar1 ′ to Ar3 ′ between a straight line 3701 extending in the vertical direction on the screen plane and the three sides of the reference graphic (3D) as shown in FIG. Since the screen plane corresponds to the [Measurement Image] box 611 in which the measurement image is displayed, the directions of the straight line 3700 and the straight line 3701 coincide. The relative angle (3D) at this time is an angle in the clockwise direction from the straight line 3701 to the side.

In step SG504, the CPU 34c calculates the difference between the relative angle (measurement) and the relative angle (3D). Specifically, the CPU 34c calculates the respective differences Dr1 to Dr3 between the relative angles (measurements) Ar1 to Ar3 and the relative angles (3D) Ar1 ′ to Ar3 ′ as in the expressions (4) to (6). To do.
Dr1 = Ar1-Ar1 ′ (4)
Dr2 = Ar2-Ar2 '(5)
Dr3 = Ar3-Ar3 ′ (6)

  In step SG505, the CPU 34c calculates an average value of the differences Dr1 to Dr3, and records the calculated average value in the RAM 34a. In step SG506, the CPU 34c rotates the camera pose in the roll direction by the average value calculated in step SG505. When a 3D object is displayed in this camera pose, it can be seen that the relative angle (measurement) after rotation and the relative angle (3D) are in good agreement, as shown in FIGS.

  In step SG507, the CPU 34c overwrites and records the camera pose at this time in the RAM 34a as the current camera pose. Here, the 3D object is not re-displayed based on the current camera pose. When the process of step SG507 ends, the roll direction matching process ends.

  Next, the flow of the matching process in the zoom direction in step SG6 will be described with reference to FIG. The purpose of the matching process in the zoom direction is to find a camera pose that most closely matches the size of the reference graphic (measurement) and the reference graphic (3D). When the sizes of the respective reference figures are close, the position in the zoom direction of the line of sight observing the subject shown in the measurement image and the position in the zoom direction of the line of sight observing the 3D object are substantially the same. it can.

  In step SG601, the CPU 34c calculates a side length (measurement) and records the calculated side length (measurement) in the RAM 34a. The side length (measurement) is the lengths L1 to L3 of three sides of a triangle formed by reference points (measurements) R1 to R3 as shown in FIG.

  In step SG602, the CPU 34c calculates a projection point (3D) and records the calculated projection point (3D) in the RAM 34a. In step SG603, the CPU 34c calculates the side length 1 (3D), and records the calculated side length 1 (3D) in the RAM 34a. The side length 1 (3D) is the lengths L1 'to L3' of the three sides of the reference graphic (3D) as shown in FIG.

  In step SG604, the CPU 34c overwrites and records the camera pose at this time as the camera pose 1 in the RAM 34a. In step SG605, the CPU 34c moves the camera pose by a predetermined value in the zoom direction as shown in FIG.

  In step SG606, the CPU 34c calculates a projection point (3D), and records the calculated projection point (3D) in the RAM 34a. In step SG607, the CPU 34c calculates the side length 2 (3D) and records the calculated side length 2 (3D) in the RAM 34a. As shown in FIG. 39B, the side length 2 (3D) is the length Lz1 ′ to Lz3 ′ of the three sides of the reference graphic (3D) after the camera pose is moved by a predetermined value in the zoom direction. It is. In step SG608, the CPU 34c overwrites and records the camera pose at this time as the camera pose 2 in the RAM 34a.

  In step SG609, the CPU 34c calculates a zoom amount and records the calculated zoom amount in the RAM 34a. The zoom amount is the amount of movement in the zoom direction of the camera pose such that the side length (3D) and the side length (measurement) match. The relationship between the side length 1, 2 (3D) and the camera pose 1, 2 Is calculated from Since there are three sides, three zoom amounts are also calculated.

  FIG. 40 shows the relationship between the side length (3D) and the zoom direction position of the camera pose. As shown by a graph 4000 in FIG. 40, both are in a linear proportional relationship. By using the graph 4000, it is possible to calculate the movement amount when moving the camera pose in the zoom direction when the side length (3D) and the side length (measurement) are matched.

  In step SG610, the CPU 34c calculates an average value of the three zoom amounts, and records the calculated average value in the RAM 34a. In step SG611, the CPU 34c moves the camera pose in the zoom direction by the average value calculated in step SG611. When a 3D object is displayed in this camera pose, it can be seen that the side length (measurement) and the side length (3D) after movement are in good agreement, as shown in FIGS.

  In step SG612, the CPU 34c overwrites and records the camera pose at this time in the RAM 34a as the current camera pose. Here, the 3D object is not re-displayed based on the current camera pose. When the process of step SG612 ends, the zoom direction matching process ends.

  Next, the flow of the matching process in the shift direction in step SG7 will be described using FIG. The purpose of the matching process in the shift direction is to move the 3D object in the shift direction so that the subject shown in the measurement image matches the 3D object in the [Measurement Image] box 611. Since this process determines the shift position of the 3D object, the camera pose is not calculated.

  In step SG701, the CPU 34c calculates a centroid point (measurement), and records the calculated centroid point (measurement) in the RAM 34a. The center-of-gravity point (measurement) is a triangular center-of-gravity point G constituted by reference points (measurements) R1 to R3 as shown in FIG.

  In step SG702, the CPU 34c calculates a projection point (3D) and records the calculated projection point (3D) in the RAM 34a. In step SG703, the CPU 34c calculates a centroid point (3D) and records the calculated centroid point (3D) in the RAM 34a. The center-of-gravity point (3D) is a triangular center-of-gravity point G ′ constituted by projection points (3D) Rp1 ′ to Rp3 ′ as shown in FIG.

  In step SG704, the CPU 34c calculates the shift amount, and records the calculated shift amount in the RAM 34a. The shift amount is a relative position of the barycentric point (measurement) to the barycentric point (3D) (plane coordinate system, pixel unit). In step SG705, the CPU 34c moves the 3D object in the shift direction by the shift amount calculated in step SG704. When a 3D object is displayed in this camera pose, it can be seen that the center of gravity (measurement) and the center of gravity (3D) are in good agreement, as shown in FIG. When the process of step SG705 ends, the matching process in the shift direction ends.

  In the conventional example, matching processing is performed based on image data of a projection image (2D model) of a two-dimensional image and a 3D object. Therefore, it is often used for redisplay of a 3D object and statistical calculation of image data having a huge amount of data. It took a long time to process. However, in this embodiment, when executing the 3D matching process, it is only necessary to execute a simple geometric calculation based on a simple reference graphic specified by the user, and the processing time can be greatly reduced. Furthermore, the 3D object may be redisplayed only once after the 3D matching process is completed.

  Next, a modification of this embodiment will be described. In the above description, the CPU 34c executes the 3D matching process based on the three reference points. However, in the present modification, the CPU 34c executes the 3D matching process based on the four reference points. . In the matching process in the pan / tilt direction in step SG4, the CPU 34c calculates the rotation angle in the pan / tilt direction of the camera pose in which the reference figure (measurement) and the reference figure (3D) are closest to each other. Even if the rotation angle at which only the reference figure (triangle) is similar is calculated, the matching accuracy is not necessarily high, and an apparently incorrect rotation angle may be calculated. Further, in each matching process of steps SG5 to SG7, an incorrect rotation angle / movement amount may be calculated in the same manner.

  In view of the above, an object of the present modification is to improve the matching accuracy of the 3D matching process. In this modification, in the reference point (measurement) designation process in step SE, as shown in FIG. 43A, the user designates four reference points (measurements) R1 to R4, and the reference point (step SF) In the 3D designation process, as shown in FIG. 43B, the user designates four reference points (3D) R1 ′ to R4 ′. In this modification, in the 3D matching process in step SG, the CPU 34c calculates a reference graphic based on the four reference points.

  FIG. 44 shows the reference graphic (measurement) calculated in step SG. Four triangles 4400, 4401, 4402, and 4403 constituted by three of the four reference points (measurements) R1 to R4 are reference figures (measurements). FIG. 45 shows the reference graphic (3D) calculated in step SG. Four triangles 4500, 4501, 4502, and 4503 constituted by three of the four reference points (3D) Rp1 'to Rp4' serve as reference figures (3D).

  In the present modification, in the 3D matching process in step SG, the CPU 34c executes the 3D matching process based on these four triangles. For example, in the pan / tilt direction matching process in step SG4, the CPU 34c calculates the rotation angle of the camera pose in the pan / tilt direction. At this time, the average rotation angle calculated based on each of the four triangles is calculated. The value is used as the final rotation angle. Also in each matching process of steps SG5 to SG7, the CPU 34c uses the average value of the rotation angle / movement amount calculated based on each of the four triangles as the final rotation angle / movement amount.

  In this modification, when the user designates four reference points, the CPU 34c performs 3D matching processing based on the four reference figures configured by the reference points. Therefore, the CPU 34c can use many parameters when executing the 3D matching process, and as a result, the matching accuracy can be improved.

  In the present embodiment, measurement is performed by the CPU 34c performing the above-described processing according to 3D measurement software that is software (program) that defines a series of processing procedures and contents related to measurement. FIG. 46 shows a functional configuration necessary for the CPU 34c. In FIG. 46, only the functional configuration related to the measurement of the present embodiment is shown, and other functional configurations are omitted. The functional configuration of the CPU 34c includes an imaging control unit 340, a designation unit 341, a matching processing unit 342, a display control unit 343, and a measurement unit 344.

  The imaging control unit 340 controls the light source 32 and the bending control unit 33 and the imaging element 30b. The designation unit 341, based on an instruction input by the user via the remote controller 23, the reference point (measurement), the reference point (3D), the measurement corresponding to the designated position on the measurement image or the image of the 3D object. Specify (set) a point. The matching processing unit 342 calculates a reference graphic (measurement) and a reference graphic (3D) based on the reference point (measurement) and the reference point (3D) specified by the specifying unit 341, and refers to the reference graphic (measurement) and the reference graphic (measurement). The amount of camera pose change necessary for matching is calculated by geometric calculation with the figure (3D).

  The display control unit 343 controls the content and display state of the image displayed on the monitor 22. In particular, the display control unit 343 displays the measurement image and the 3D object in a matched state by adjusting the posture and position of the 3D object based on the change amount of the camera pose calculated by the matching processing unit 342. . In the above description, the posture and position of only the 3D object are adjusted. However, the present invention is not limited to this, and the posture and position of only the measurement image are adjusted, or the posture and position of the measurement image and 3D object are adjusted. Also good. The measurement unit 344 performs measurement processing based on the measurement points designated by the designation unit 341. It is also possible to replace part or all of the functional configuration shown in FIG. 46 with specific hardware configured by arranging analog circuits and digital circuits for realizing functions necessary for measurement.

  As described above, according to the present embodiment, geometric calculation of the reference graphic (measurement) and the reference graphic (3D) is performed, and at least one of the measurement image and the 3D object is performed based on the result of the geometric calculation. By adjusting the posture or position, the processing time required for matching can be reduced.

  Further, by associating the reference point (measurement) with the reference point (3D) based on the designated order of the respective points of the reference point (measurement) and the reference point (3D), the two can be associated with each other in a simple manner. And the processing time required for matching can be reduced.

  Also, by performing geometric calculation matching between a reference figure (measurement) on the measurement image and a reference figure (3D) obtained by projecting a triangle formed by reference points (measurement) on the 3D object onto the screen plane The processing time required for matching can be reduced while maintaining matching accuracy.

  Further, when the user designates the reference point (measurement), the transparency of the 3D object is set to be large, so that the user can easily see the measurement image, and the user designates the reference point (measurement). It can be made easier.

  In addition, during the 3D matching process, the 3D object having a high processing load is not redisplayed, and the 3D object is redisplayed at the end of the 3D matching process, thereby reducing the processing time required for the matching.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the above-described embodiments, and includes design changes and the like without departing from the gist of the present invention. .

  DESCRIPTION OF SYMBOLS 1 ... Jet engine, 2 ... Turning tool, 3 ... Endoscope apparatus, 4 ... Video terminal cable, 5 ... Video capture card, 6 ... PC, 7 ... LAN Cable 10, turbine blade 20, endoscope insertion section 21, endoscope apparatus main body 22, 25 monitor (display section) 23 remote control (input device) 24 ... PC body, 30a ... imaging optical system, 30b ... imaging device, 31 ... image signal processing device, 32 ... light source, 33 ... bending control unit, 34, 35. ..Control computer, 34a, 35a... RAM, 34b... ROM, 34c, 35c... CPU (designation unit, calculation unit, adjustment unit, measurement unit), 34d, 35d. F, 34e ... RS232C ... I / F, 34f ... Card I / F, 35b ... HDD, 35e ... USB ... I / F, 340 ... Imaging control unit, 341 ... Designation unit, 342 ... Matching processing unit, 343 ... display control unit, 344 ... measurement unit

Claims (11)

A display unit that displays an image of the measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance;
A designation unit for designating a first point on the image of the measurement object and a second point on the image of the object based on an instruction input via an input device;
A calculation unit that performs a geometric calculation between a first graphic that is based on the first point and a second graphic that is based on the second point;
An adjusting unit that adjusts the posture or position of at least one of the image of the measurement object and the image of the object based on the result of the geometric calculation;
After the posture or the position is adjusted, a spatial coordinate on the object corresponding to a measurement position designated based on an instruction input via an input device is calculated, and based on the calculated spatial coordinate, A measurement unit for calculating the size of the object;
An image processing apparatus comprising: A display unit that displays an image of the measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance;
A designation unit for designating a first point on an image of the measurement object based on an instruction input via an input device;
A calculation unit that performs a geometric calculation between a first graphic that is based on the first point and a second graphic that is recorded in association with the image of the object;
An adjusting unit that adjusts the posture or position of at least one of the image of the measurement object and the image of the object based on the result of the geometric calculation;
After the posture or the position is adjusted, a spatial coordinate on the object corresponding to a measurement position designated based on an instruction input via an input device is calculated, and based on the calculated spatial coordinate, A measurement unit for calculating the size of the object;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the first point and the second point include three or more points designated based on an instruction input via an input device.   The calculation unit associates the first point with the second point on the basis of the designated order of the first point and the designated order of the second point. The image processing apparatus according to claim 3, wherein scientific calculation is performed.   The calculation unit includes: a first graphic on a plane based on the first point; a second graphic on a space based on a point on the object space corresponding to the second point; The image processing apparatus according to claim 1, wherein geometric calculation is performed.   6. The calculation unit according to claim 5, wherein the calculation unit performs a geometric calculation between a first graphic on the plane and a third graphic obtained by projecting the second graphic on the space onto the plane. Image processing apparatus.   The specifying unit further sets the transmittance of the image of the object to be higher than the transmittance of the image of the object when specifying the second point when specifying the first point. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The display unit displays an image of the measurement object and an image of the object, and then maintains a posture and a position of the image of the measurement object and the image of the object until the geometric calculation ends. 8. The image of the measurement object and the image of the object whose postures or positions are adjusted are displayed again after the geometric calculation is finished. 8. An image processing apparatus according to 1. A display unit displaying an image of the measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance;
A step of designating a first point on the image of the measurement object and a second point on the image of the object based on an instruction input via the input device;
A calculation unit performing a geometric calculation of a first graphic with respect to the first point and a second graphic with respect to the second point;
A step of adjusting the posture or position of at least one of the image of the measurement object and the image of the object based on the result of the geometric calculation;
After the posture or the position is adjusted, the measurement unit calculates a spatial coordinate on the object corresponding to the measurement position specified based on an instruction input via the input device, and calculates the calculated spatial coordinate. Calculating a size of the object based on:
An image processing method comprising: The first point and the second point include three or more points designated based on an instruction input via an input device;
The calculation unit associates the first point with the second point on the basis of the designated order of the first point and the designated order of the second point. The image processing method according to claim 9, wherein a scientific calculation is performed. A display unit that displays an image of the measurement object and an image of the object corresponding to the measurement object and having a three-dimensional shape calculated in advance;
A designation unit for designating a first point on the image of the measurement object and a second point on the image of the object based on an instruction input via an input device;
A calculation unit that performs a geometric calculation between a first graphic that is based on the first point and a second graphic that is based on the second point;
An adjusting unit that adjusts the posture or position of at least one of the image of the measurement object and the image of the object based on the result of the geometric calculation;
After the posture or the position is adjusted, a spatial coordinate on the object corresponding to a measurement position designated based on an instruction input via an input device is calculated, and based on the calculated spatial coordinate, A measurement unit for calculating the size of the object;
As a program to make the computer function as.

Images