JP7183020B2 - Image processing device, image processing method, and program - Google Patents

Description

TECHNICAL FIELD The present invention relates to a technique for assisting camera positioning when photographing with a camera.

In recent years, due to the widespread use of digital cameras, there are increasing cases where digital cameras are used for purposes other than photography, which is their original purpose. For example, since a digital camera can acquire colors in two dimensions, it can be used as a two-dimensional colorimeter. There is an example of using it as a shape measuring instrument for calculating the shape. In the measurement of a subject having such a three-dimensional object shape, it is required to measure the variable angle reflection characteristics that change the apparent color from a plurality of different viewing angles. In this case, it is desirable to automatically control the spatial geometric conditions in photographing with a digital camera. As a technique for such automatic control, for example, Patent Document 1 discloses a technique for detecting the camera orientation using a gyro sensor or an acceleration sensor, and automatically taking a picture when it is determined that the camera orientation is predetermined. It is

Japanese Unexamined Patent Application Publication No. 2011-41050

With the technique described in Patent Document 1, it is possible to shoot when it is detected that the detected camera attitude has changed to a preset camera attitude. However, it is difficult for the user to grasp how to move the camera position when moving the camera position so as to meet preset geometric conditions.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to enable a user to easily match the position of a camera with a set geometric condition when moving the position of the camera so as to meet the set geometric condition.

The image processing apparatus of the present invention includes setting means for setting a geometric condition and an allowable range for the geometric condition when measuring an object on which markers are arranged, and imaging the object using an imaging means. Acquisition means for acquiring obtained image data; Extraction means for extracting the outline of the marker arranged on the object based on the image data; and Based on the image data and the set geometric conditions a calculation means for calculating a guide marker for guiding the moving direction of the imaging means; and a display control means for displaying the outline of the extracted marker, the calculated guide marker, and the allowable range on a display means. , is characterized by having

According to the present invention, when moving the position of the camera so as to meet the set geometric conditions, the user can easily match the position of the camera with the set geometric conditions.

It is a figure which shows the structural example of an image processing apparatus. It is a figure which shows the logical structure of an image processing apparatus. 4 is a processing flowchart in the image processing apparatus; It is a figure which shows an example of UI in 1st Embodiment. FIG. 4 is a diagram showing spherical coordinates; 4 is a state transition diagram showing state transition of UI in the first embodiment. FIG. FIG. 4 is a block diagram showing the logical configuration of a shape normal calculation unit; It is a schematic diagram explaining a marker. FIG. 4 is a diagram used for explaining markers and metadata; It is a flow chart of shape calculation processing. 6 is a flowchart of normal calculation processing; 6 is a flowchart of contour extraction processing; 4 is a flowchart of contour projection processing according to the first embodiment; FIG. 7 is a diagram showing an example of a UI for contour display in the first embodiment; It is a figure which shows an example of UI in 2nd Embodiment. FIG. 11 is a state transition diagram showing UI state transitions in the second embodiment. 9 is a flowchart of contour projection processing in the second embodiment; FIG. 11 is a diagram illustrating an example of a UI for contour display in the second embodiment;

Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the embodiments described below are examples of specific implementations of the present invention, and the present invention is not limited to the following embodiments.
In this embodiment, it is assumed that a marker, which will be described later, is arranged on the surface of the three-dimensional object that is the object to be measured. Also, in this embodiment, a marker having a plurality of point patterns is taken as an example of the marker, and a specific example thereof will be described later. In this embodiment, the relationship between the measurement geometric condition, the subject, and the camera position is fed back to the user based on the marker in the captured image of the subject of the measurement object and the preset measurement geometric condition. To make it easier for a user to position a camera with respect to conditions.

<First Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the invention. FIG. 1 shows the configuration of a computer 11 as an application example of the image processing apparatus of this embodiment.

The CPU 101 uses the RAM 103 as a work memory, executes an operating system (OS) and various programs stored in the ROM 102, hard disk drive (HDD) 17, various recording media, etc., and controls each configuration via the system bus 107. . Note that the programs executed by the CPU 101 include a program (image processing application) that executes a processing flow according to this embodiment, which will be described later. Note that the image processing application program according to the present embodiment is supplied via various recording media or a network, and can be executed by the CPU 101 by being developed in the RAM 103 or the like.

A general-purpose I/F 104 is a serial bus interface such as USB, and is connected to an input device 13 such as a mouse and a keyboard, a printer 14 , and the like via the serial bus 12 . The CPU 101 receives user input including user instructions from the input device 13 via the general-purpose I/F 104 .
The SATA I/F 105 is a serial ATA (SATA) interface, and is connected to the HDD 17 and a general-purpose drive 18 that reads and writes various recording media. The CPU 101 uses various recording media mounted on the HDD 17 or the general-purpose drive 18 as data storage locations for reading and writing via the SATA I/F 105 .
VC 106 is a video card for video interface, and is connected to display 15, which is a display device. CPU 101 displays a user interface (UI) provided by the program on display 15 via VC 106 .

Processing executed by the image processing application based on commands from the CPU 101 in the configuration shown in FIG. 1 will be described below.
FIG. 2 is a block diagram showing the logical configuration of the marker alignment guide display device 2 configured in the computer 11 by executing the image processing application of this embodiment. Note that FIG. 2 also shows an imaging device 1 for imaging the object to be measured.
The guide display device 2 shown in FIG. be.

The image data acquisition unit 21 acquires captured image data obtained by capturing an image of the subject of the measurement object by the imaging device 1 . The captured image data acquired by the image data acquisition section 21 is sent to the shape normal calculation section 22 and the contour extraction section 23 .

The shape normal calculation unit 22 extracts markers from the captured image acquired by the image data acquisition unit 21, and calculates the normal direction and the position and angle of the camera (imaging device 1) based on the extracted markers. . Camera angles are the azimuth and elevation angles at which the optical axis of the camera is pointing. Details of the normal direction calculation processing based on the marker and the camera position and angle calculation processing will be described later.

The contour extraction unit 23 extracts the contour of the marker extracted by the shape normal calculation unit 22 from the captured image acquired by the image data acquisition unit 21 . The details of the marker contour extraction processing will be described later. Information on the marker contours extracted from the captured image by the contour extraction unit 23 is sent to the contour display unit 26 .

The geometric condition setting unit 24 sets geometric conditions for measurement based on user instructions input via the input device 13 . Details of the geometric conditions set based on user instructions will be described later. Information on the geometric conditions set by the geometric condition setting unit 24 is sent to the marker contour projection unit 25 .

The marker contour projection unit 25 projects the subject on the preview screen based on the geometric condition set by the geometric condition setting unit 24, the normal direction calculated by the shape normal calculation unit 22, and the position and angle of the camera. Calculate a marker contour projected onto (measurement object). The details of the marker contour projection calculation processing in the marker contour projection unit 25 will be described later. Projection information of the marker contour calculated by the marker contour projection unit 25 is sent to the contour display unit 26 .

The contour display unit 26 displays the marker contour extracted from the captured image by the contour extraction unit 23 and the marker contour for projection calculated by the marker contour projection unit 25 on the preview screen displayed on the display 15. superimposed on the The details of the superimposed display of these marker contours on the preview screen will be described later.

FIG. 2 shows an example of a software configuration in which the computer 11 executes an image processing application, but each block in FIG. 2 may be realized by a hardware configuration. It may be realized by a hardware configuration.

FIG. 3 is a flow chart showing the flow of guide display processing in the guide display device 2 of this embodiment. In the following description, processing steps S301 to S305 in the flowchart of FIG. 3 are abbreviated as S301 to S305, respectively. The same applies to other flowcharts to be described later. The processing of the flowchart of FIG. 3 is started when the user operates the input device 13 and inputs a predetermined start instruction.

First, in S301, the image data acquisition unit 21 acquires captured image data, and the geometric condition setting unit 24 acquires geometric conditions set by the user.
Next, in S302, the shape normal line calculation unit 22 calculates the shape of the marker, the normal line of the measurement target point, and the position and angle of the camera from the captured image by processing to be described later.
Next, in S303, the contour extracting unit 23 extracts the contour of the marker from the captured image by processing described later.
Next, in S304, the marker contour projection unit 25 calculates the projection of the marker onto a plane whose normal is the geometric condition acquired in S301, by processing described later.
Then, in S305, the contour display unit 26 displays the two marker contours as a UI by the processing described later.

FIG. 4 is a diagram showing an example of the UI screen 4 displayed on the display 15. As shown in FIG.
In the UI example of FIG. 4, a preview window 401 is a display area for displaying a preview image captured by the camera (imaging device 1). A + marker 402 is a marker representing a measurement target point on the subject 41 . A text box 403 is an input area for the user to input an angle value or the like when editing the azimuth angle of the camera as a geometric condition. A text box 404 is an input area for the user to input an angle value or the like when editing the elevation angle of the camera as a geometric condition. The execution button 405 is an operation button that the user instructs when starting measurement based on the geometric conditions set by the input in the text boxes 403 and 404 .

FIG. 5 is a diagram showing an example of geometric conditions set in this embodiment, and schematically shows the positional relationship and geometric conditions (spherical coordinate values) between the subject and the camera using a spherical coordinate system.
Here, the radial coordinate is r, the angle formed by the z-axis and the radius is θ (zenith angle θ), and the angle formed by the projection of the radius and another axis (x-axis) on a plane perpendicular to the z-axis be ψ (azimuth angle ψ). It is also assumed that the coordinate origin is the measurement target point, and the normal line and the x-axis at the measurement target point are the same. Note that r of the radial coordinate will be described later.

FIG. 6 is a state transition diagram according to the display of the UI screen 4 in the guide display device 2 and user input.
In FIG. 6 , in state 601 , the UI screen is displayed on the display 15 after initial data is read, and then transitions to state 602 .
State 602 waits for user input, and transitions to state 603 when a point to be measured is set in the preview window 401 by an instruction from the user. If a user input is made to the text box 403 for editing the azimuth angle or if a user input is made to the text box 404 for editing the elevation angle, the state transitions to state 604 . Also, when the execution button 405 is pressed in state 602 , the state transitions to state 605 .

After the transition to state 603 and the setting of the measurement target point by the user, the transition to state 602 is made.
When the user sets the angles (azimuth, elevation) of the geometric conditions in the text boxes 403 and 404, the state 604 is entered.
When the execution button 405 is pressed in state 602 to transition to state 605, a measurement operation based on the set measurement target points and the set geometric conditions is started.

FIG. 7 is a block diagram showing the logical configuration of the shape normal calculator 22. As shown in FIG. Note that FIG. 7 also shows an imaging device 1 that acquires a live view image.
The shape normal calculation unit 22 is configured by having a shape calculation unit 701 and a normal calculation unit 702 .
Before describing details of the configuration and processing in the shape normal calculation unit 22 of FIG. 7, details of the marker and metadata corresponding to the marker 802 will be described with reference to FIGS. 8 and 9. FIG.

FIG. 8 is a diagram showing an example of a measurement object 8 (subject) imaged by the imaging device 1 and a marker 802 attached to the measurement object 8. As shown in FIG. The marker 802 is formed having a plurality of point patterns 801 for position acquisition of the imaging device 1 and an ID code area 803 in which the ID code assigned to the marker 802 is described.

FIG. 9 is a diagram for explaining the correspondence relationship between information held as metadata corresponding to the ID code in the ID code area 803 and markers. FIG. 9(a) is a diagram showing specific information recorded as metadata and an example of its arrangement. It is the figure which showed an example.

Here, in the metadata area 901 shown in FIG. 9A, the three-dimensional (X, Y, Z) coordinate values on the paper surface of the center point of each point pattern 801 of the marker 802 in FIG. This is a recording area, in which, for example, Nx×Ny rows are recorded. For example, the center point of the point pattern 801 at the upper left corner of the marker 802 in FIG. The three-dimensional coordinate values of points are recorded.

A metadata area 902 in FIG. 9A is an area for recording two-dimensional (u, v) coordinate values of the center point of each point pattern 801 of the marker 802. For example, Nx×Ny lines are recorded. It is For example, the center point of the point pattern 801 at the upper left corner of the marker 802 in FIG. 9B is set as the origin, and two-dimensional coordinate values in units of pixels are recorded with respect to the origin.

Further, a metadata area 903 in FIG. 9A is an area for recording polygon information. In the case of this embodiment, a polygon is determined by a shape formed with the center points of several point patterns 801 within the marker 802 as vertices. For example, as shown in FIG. 9(b), rectangles 804 with the central points of the four point patterns 801 as vertices are polygons. The polygon information consists of A/A (A is the same line number) representing a set of the line number of the area 901 and the line number of the area 902 as the information of the vertices of the rectangle, and the four vertices of the rectangle, for example, Recording is performed in order of rotation around the counterclockwise circumference of the rectangle. Polygons are not limited to rectangles, and may be triangular polygons obtained by halving rectangles along diagonal lines.

Details of the processing in the shape calculation unit 701 will be described below with reference to the flowchart shown in FIG.
First, in S1001, the shape calculation unit 701 acquires image data captured by the imaging device 1 via the image data acquisition unit 21 in FIG.
Next, in S<b>1002 , the shape calculation unit 701 extracts the ID code area 803 from the captured image data, and reads the ID code of the marker 802 from the image pattern of the extracted ID code area 803 . Specifically, the shape calculation unit 701 first binarizes the pixel values of the captured image. This binarization process is a binarization process in which a pixel having a pixel value equal to or greater than a predetermined threshold is rendered white, and a pixel having a pixel value less than the predetermined threshold is rendered black. Next, the shape calculation unit 701 extracts the edge position from the captured image after the binarization processing using the known Canny method, and groups the pixels having the edge position in the eight neighboring pixels as the same contour. contour extraction. Further, the shape calculation unit 701 selects a rectangular contour from among the plurality of extracted contour groups, and transforms the contour so that the ID code area extracted from the captured image has the same shape as the actual ID code area 803. . Then, the shape calculation unit 701 divides the pattern inside the ID code area obtained by the contour transformation process into blocks of 8×8 pixels, and reads the ID code based on the gradation of each block.
Furthermore, in S1003, the shape calculation unit 701 acquires metadata of the marker 802 corresponding to the ID code read in S1002.

Next, in S1004, the shape calculation unit 701 calculates center coordinates of each pattern region of each point pattern 801 of the marker 802 from the captured image data. Here, the shape calculation unit 701 first performs processing up to contour extraction in the same manner as in S1002, and acquires contour candidates that are circles or ellipses from the extracted contour group. Next, the shape calculation unit 701 calculates the area surrounded by the outlines of the circles or ellipses acquired as the outline candidates, and calculates the area according to the difference between the calculated area of each outline candidate and the area of the point pattern 801 set in advance. to rank the contour candidates. Further, the shape calculation unit 701 calculates the central coordinate values of the pattern areas of the contour candidates ranked 1 to 18, for example. Then, the shape calculation unit 701 calculates the center of each contour candidate so that the relative positional relationship of the center coordinate values of each contour candidate matches the relative positional relationship of the coordinate values described in the metadata area 902 . Sort coordinate values. As a result, the shape calculation unit 701 can easily determine the correspondence between the rectangle defined in the metadata area 903 and the center coordinates of the point pattern 801, which is the vertex of the rectangle, and the distortion of each rectangle in the captured image. can be identified and the shape can be obtained.

Next, normal direction calculation processing in the normal calculation unit 702 will be described.
A normal line calculation unit 702 calculates a normal line direction (for example, a three-dimensional vector) of each pixel from the shape data determined by the coordinate values calculated by the shape calculation unit 701 .
FIG. 11 is a flowchart showing the flow of normal direction calculation processing in the normal calculation unit 702 .

First, in S1101, the normal calculation unit 702 reads the three-dimensional coordinate values of the vertices of the rectangle 804 from the metadata. Specifically, the normal calculation unit 702 reads one line of polygon information shown in the area 903 from the metadata, and reads coordinate values corresponding to each vertex of the polygon from the area 901 .

Next, in S1102, the normal calculation unit 702 reads the coordinate values corresponding to the vertices whose coordinate values were read in S1101 from the center coordinates of the point pattern 801 calculated in S1004. Here, the order of the center coordinates of the point pattern 801 is sorted so as to match the order of the coordinate values already registered in the area 902 in S1004. Therefore, in S1102, the center coordinates corresponding to the vertex numbers described in the area 903 may be extracted, as in S1101.

Next, in S1103, the normal calculation unit 702 estimates a rotation matrix made up of coefficients r11 to r33 and a translation vector made up of coefficients t1 to t3 shown in Equation (1) below. As a result, the position and orientation of the imaging device 1 with respect to the plane of the rectangle 804 can be known.

fx and fy in equation (1) denote the focal length of the imaging device 1, and cx and cz denote the principal points of the imaging device 1, respectively, and are held as known values. (u, v) in equation (1) are the center coordinates of the point pattern 801 of the captured image data read in S1002, and (X, Y, Z) are the three-dimensional coordinate values corresponding to the metadata acquired in S1003. be. In S1103, the rotation matrix and translation vector are estimated from the correspondence between (u, v) and (X, Y, Z) at the four vertices of the polygon. This estimation method is known as, for example, the PnP (Perspective-n-Point) problem, and the rotation matrix and translation vector can be estimated from the correspondence between (u, v) and (X, Y, Z) of three or more points. .

Next, in S1104, the normal calculation unit 702 calculates the three-dimensional coordinates (x, y, z) of each vertex with the imaging device 1 as the origin, and the normal vector of the plane containing each vertex. The normal calculation unit 702 at this time first calculates the three-dimensional coordinates (x, y, z) of each vertex by the following equation (2).

In Equation (2), R is a rotation matrix and t is a translation vector, which are calculated in S1003. (X, Y, Z) are the three-dimensional coordinate values of each vertex in the metadata read in S1001. In this process, the (x, y, z) coordinate value of one vertex is used as the origin to calculate a vector to the (x, y, z) coordinate values of the other two vertices, and the direction indicated by the outer product vector is calculated as the normal vector. If the (X, Y, Z) coordinate values of the vertices have the same Z value, the normal vector can be a unit vector whose components are r13, r23, and r33 in the third column of the rotation matrix.

The normal calculation unit 702 performs the above-described processing of S1101 to S1104 for all rectangles, and acquires the three-dimensional (x, y, z) coordinate values of the vertices of each rectangle with the imaging apparatus 1 as the origin. These coordinates can be regarded as the three-dimensional coordinate values of the object 8 itself, even if the object 8 is, for example, a curved surface, if the rectangles are arranged at intervals that can be regarded as a plane.

Next, in S1105, the normal line calculation unit 702 determines three-dimensional (x, y, z) coordinate values of the center of the point pattern 801 for position acquisition. Here, the vertices of the rectangles calculated in the processing of S1101 to S1104 overlap between adjacent rectangles. Therefore, the normal calculation unit 702 recalculates the (x, y, z) coordinate values of the overlapping vertices. Specifically, the normal calculation unit 702 first calculates, in each rectangle, a plane that includes vertices that overlap adjacent rectangles and whose normal is the normal vector of the rectangle. Then, the normal calculation unit 702 calculates a straight line passing through the coordinate values of the non-overlapping vertices and the origin, and sets the intersection of the straight line and the plane as the new coordinate values of the non-overlapping vertices. By recalculating the coordinate values in this way, it is possible to maintain the normal vector of each rectangle. Note that the three-dimensional coordinate value of the center of the corresponding point pattern 801 may simply be the average value of the (x, y, z) coordinate values of the overlapping vertices.

After that, in S1106, the normal calculation unit 702 calculates the normal vector of each point pattern 801 for position acquisition from the normal vector calculated for each rectangle. This process is performed, for example, by calculating the average value of normal vectors of all rectangles having the center of each point pattern 801 as a vertex.

Processing for extracting the outline of the marker from the preview image in the outline extracting section 23 will be described below.
FIG. 12 is a flow chart showing the flow of marker contour extraction processing in the contour extraction unit 23 .
First, in S1201, the outline extraction unit 23 acquires the center coordinates and the identification number of each point pattern of the marker on the image plane calculated in S1004. As an example, the point pattern identification numbers are numbers assigned in counterclockwise order within the marker.

Next, in S1202, the outline extraction unit 23 generates a solid line connecting the center coordinates of the point patterns with the identification number i and the identification number i+1 as a line to be added to the preview image. The solid line information is sent to the outline display section 26 and displayed, whereby the solid line connecting the center coordinates of the point patterns is superimposed and displayed on the preview image.

Next, in S1203, the outline extraction unit 23 determines whether or not the processing of S1202 has been performed for all point patterns. return.
Proceeding to S1204, the contour extraction unit 23 generates a solid line connecting the central coordinates of the point patterns with the identification number N−1 and the identification number 0 as a line to be added to the preview image. Note that N represents the total number of marker point patterns. The solid line information is sent to the outline display section 26 and displayed, whereby the solid line connecting the center coordinates of the point patterns is superimposed and displayed on the preview image.

Processing for projecting a marker contour in the marker contour projection unit 25 will be described below.
FIG. 13 is a flow chart showing the flow of the process of calculating the projection of the contour of the marker 802 corresponding to the set geometric conditions from the preview image in the marker contour projection unit 25 and performing planar mapping.
First, in S1301, the marker contour projection unit 25 acquires the central coordinates and identification number of each point pattern in the three-dimensional space calculated in S1004, and the position of the imaging device 1 calculated in S1103.

Next, in S1302, the marker contour projection unit 25 calculates the two-dimensional coordinates (ui, vi) of the mapping destination of the point pattern with the identification number i using the equation (1) described in S1103.
Next, in S1303, the marker contour projection unit 25 determines whether or not the processing of S1302 has been performed for all point patterns. It is incremented and the process returns to S1302.

Proceeding to S1304, the marker contour projection unit 25 generates a dashed line between projection destination coordinates for the point patterns of identification number i and identification number i+1 as a dashed line to be added to the preview image. Information about the dashed lines is sent to the contour display unit 26 and displayed, whereby the dashed lines connecting the coordinates of the projection destinations of the point patterns are superimposed and displayed on the preview image.

Next, in S1305, the marker contour projection unit 25 determines whether or not the processing of S1304 has been performed for all point patterns. It is incremented and the process returns to S1304.
Proceeding to S1306, the marker contour projection unit 25 generates a dashed line connecting the center coordinates of the point patterns with the identification number N−1 and the identification number 0 as a dashed line to be added to the preview image. Information about this dashed line is sent to the contour display unit 26 and displayed, whereby a dashed line connecting the center coordinates of each point pattern of the marker is superimposed and displayed on the preview image.

The UI displayed on the display 15 by the contour display unit 26 will be described in detail below. 14(a) to 14(d) are diagrams schematically showing an example of the UI, and a window 401 is a display area for displaying the example shown in FIG. 4 and a preview image. 14(a) to 14(d), a solid line marker 1401 is a guide marker indicating the outline of the marker in the captured image, and a dashed line marker 1402 is a guide marker indicating the mapping of the outline of the marker under the set geometric conditions. is. Vertices 1403, 1404, 1405, 1406 are the four vertices of the marker contour. Vertices 1403', 1404', 1405', 1406' are the four vertices of the orthographically projected marker contour. A + marker 1407 is a guide marker indicating a measurement target point on the subject 41 .

FIG. 14(a) shows an example of marker display when the geometric conditions of the camera match the set geometric conditions, and FIGS. 14(b) and 14(c) show the It shows an example of marker display when the condition and the set geometric condition do not match. FIG. 14D shows a display example when the + marker 1407 of the measurement target point is different from the center of the angle of view.
According to the display examples of FIGS. 14A to 14D, the user can determine whether or not the geometric conditions of the camera match the set geometric conditions when the marker 1407 of the measurement target point is at the angle of view. It is easy to visually recognize whether the center is constant or not. By changing the position and angle of the imaging device 1 so that the solid-line marker 1401 and the dashed-line marker 1402 match, the user can adjust the position and angle of the imaging device 1 to match the set geometric conditions.

As described above, in the guide display device 2 of the first embodiment, the order of measurement is determined according to specified conditions for a plurality of measurement geometric conditions, and the set geometric conditions and the positional relationship between the subject and the camera are displayed to the user. giving feedback. As a result, according to the guide display device 2 of the first embodiment, the user can easily recognize how to move the camera, and easily adjust the camera position to the set geometric conditions. will be able to

Next, a second embodiment will be described.
In the first embodiment, an example is given in which the marker contour corresponding to the set geometric condition and the contour of the photographed marker are displayed to guide the moving direction of the camera. In the second embodiment, an example will be described in which the permissible range of movement is displayed in accordance with the display that guides the movement direction of the camera, thereby facilitating the alignment of the camera with the set geometric conditions at the pixel level. do. The configuration of the second embodiment is the same as that shown in FIGS. 1, 2, and 7, so illustration thereof is omitted.

FIG. 15 shows an example of a UI screen 1500 according to the second embodiment.
In the UI screen 1500 of FIG. 15, a window 1501 is a display area for displaying a preview image, and a +marker 1502 is a marker representing a measurement target point on the subject 41 . The text boxes 1503 and 1504 and the execution button 1506 are similar to the text boxes 403 and 404 and the execution button 405 shown in FIG. A text box 1505 is an input area for the user to input a value within the allowable range. In the case of this embodiment, it is assumed that the allowable angle range can be input according to the number of pixels.

FIG. 16 is a state transition diagram according to the display of the UI screen 1500 shown in FIG. 15 and user input in the guide display device 2 according to the second embodiment.
In FIG. 16 , in state 1601 , the UI is displayed on the display 15 after initial data is read, and then transitions to state 1602 .
The state 1602 waits for the user's input, and if the user's instruction is to set the measurement target point in the preview window 1501, the state 1603 is entered. When user input is made in the text box 1503 for editing the azimuth angle and the text box 1504 for editing the elevation angle in the geometric condition, the state transitions to state 1604 . Also, if the execute button 1506 is pressed in state 1602 , the state transitions to state 1606 . Also, in state 1602 , when the user inputs to the text box 1505 for editing the allowable range, the process transitions to state 1605 .

When transitioning to state 1603, the user can set the measurement target point, and transition to state 1602 when the measurement target point is set.
When the transition to state 1604 is made, the angle of the geometric condition can be set by the user.
After transitioning to state 1605 , the user can set the allowable range of angles, and after input is made in the text box 1505 , transition is made to state 1602 .
Then, when the execution button 1506 is pressed in the state 1602 to transition to the state 1606, the measurement operation based on the set measurement target point and the set geometric condition is started.

Processing for projecting a marker contour in the marker contour projection unit 25 of the second embodiment will be described below.
FIG. 17 is a flow chart showing the flow of processing for performing planar mapping corresponding to the set geometric conditions on the preview image in the marker contour projection unit 25 of the second embodiment. Since the processing from S1701 to S1706 is the same as the processing from S1301 to S1306 shown in FIG. 13, description thereof will be omitted.

After S1706, when proceeding to S1707, the marker contour projection unit 25 uses the number of pixels B set in the text box 1505 for each pixel P (x, y) forming the dashed line drawn in S1704 to S1706, Generate diagonal lines that are added to areas that meet the following conditions:

xB<shaded display area<x+B and yB<hatched display area<y+B

Then, the slanted line information is sent to the outline display section 26 and displayed, whereby the slanted lines connecting the point patterns are superimposed and displayed on the preview image.

FIG. 18 shows a UI display example displayed on the display 15 by the contour display unit 26 in the second embodiment. 18(a) to 18(c) are diagrams schematically showing an example of the UI, and a window 401 is a display area for displaying the example of FIG. 4 and a preview image. In FIGS. 18A to 18C, a solid line marker 1801 represents the contour of the marker in the captured image, and a diagonal line marker 1802 represents the map of the marker contour in the set geometric conditions. Vertices 1803, 1804, 1805, 1806 are the four vertices of the marker contour. Vertices 1803', 1804', 1805', 1806' are the four vertices of the orthographically projected marker contour. A + marker 1807 is a marker indicating a measurement target point on the subject 41 .

FIG. 18(a) shows an example of marker display when the geometric conditions of the camera and the set geometric conditions are within the allowable range. It shows an example of marker display when the condition and the set geometric condition are not within the allowable range.
According to these UI display examples of FIGS. 18(a) to 18(c), the user can easily visually check whether the geometric conditions of the camera and the set geometric conditions are within the allowable range. can recognize. By changing the position and angle of the imaging device 1 so that the solid line marker 1801 and the oblique line marker 1802 match, the user can adjust the position and angle of the imaging device 1 to match the set geometric conditions.

As described above, in the guide display device 2 of the second embodiment, by setting the allowable range for the set geometric condition, the user can easily match the camera position to the set geometric condition. become.

<Other embodiments>
In the above-described embodiment, the spherical coordinate system is used as the coordinate system for expressing the measurement geometric condition, but it may be expressed by an orthogonal coordinate system with the measurement target point as the origin. In that case, it is desirable to use a coordinate system in which the x-axis coincides with the normal line of the point of measurement symmetry. In the above-described embodiment, the projection of the marker contour is calculated based on the set geometric conditions and the normal line. A projection of the marker contour may be calculated based on the result. In the above-described embodiment, only the contour display of the marker is used as a guide display for the direction in which the camera is to be moved.

The imaging device 1 of the above-described embodiment is assumed to be a digital camera such as a digital single-lens reflex camera, a mirrorless single-lens camera, or a compact digital camera, but the imaging device 1 may be a camcorder, a medical camera, an industrial camera, or the like. may Alternatively, the imaging device may be a camera installed in a portable information terminal such as a tablet terminal, PHS, smart phone, feature phone, or portable game machine.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

1: imaging device, 2: guide display device, 21: image data acquisition unit, 22: shape normal calculation unit, 23: contour extraction unit, 24: geometric condition setting unit, 25: marker contour projection unit, 26: contour display Part 11: Computer 101: CPU

Claims (11)

setting means for setting a geometric condition for measuring an object on which the markers are arranged and an allowable range for the geometric condition ;
an acquisition means for acquiring image data obtained by imaging the object using an imaging means;
extracting means for extracting a contour of the marker placed on the object based on the image data;
calculation means for calculating a guide marker for guiding the movement direction of the imaging means based on the image data and the set geometric conditions;
display control means for displaying the extracted outline of the marker, the calculated guide marker, and the allowable range on a display means;
An image processing device comprising: The setting means
means for setting a measurement target point of the marker;
means for setting an azimuth angle and an elevation angle toward which the optical axis of the imaging means faces as the geometric conditions;
2. The image processing apparatus according to claim 1, comprising: The display control means is characterized in that the measurement target point of the marker, the extracted contour of the marker, and the calculated guide marker are superimposed on the image captured by the imaging means. The image processing apparatus according to claim 2. The display control means converts the measurement target point of the marker, the extracted contour of the marker, the calculated guide marker, and the contour representing the allowable range into an image captured by the imaging means. 3. The image processing apparatus according to claim 2 , wherein the image is displayed in a superimposed manner. The calculation means is
means for calculating a normal to the marker placed on the object based on the image data;
means for calculating the guide marker based on the normal and the set geometric condition;
5. The image processing apparatus according to any one of claims 1 to 4 , comprising: The marker placed on the object is formed with a plurality of point patterns and an ID code region of the marker,
6. The method according to claim 5 , wherein said calculating means calculates coordinate values based on positions of said plurality of point patterns of said image data, and calculates said normal line based on said calculated coordinate values. Image processing device. 7. The image processing apparatus according to claim 6 , wherein said calculating means acquires the shape of a polygon based on said calculated coordinate values. The calculation means is
sorting a plurality of coordinate values calculated based on the positions of the plurality of point patterns based on defined coordinate values;
8. The image processing apparatus according to claim 7 , wherein the shape of said polygon is calculated based on said plurality of coordinate values after said sorting. 3. The calculating means calculates a normal line of the polygon of the calculated shape, and acquires the position and orientation of the imaging means with respect to the plane of the polygon based on the normal line of the polygon. 9. The image processing device according to 8 . An image processing method executed by an image processing device,
a setting step of setting a geometric condition for measuring an object on which markers are arranged and an allowable range for the geometric condition ;
an acquisition step of acquiring image data obtained by imaging the object using imaging means;
an extracting step of extracting a contour of the marker arranged on the object based on the image data;
a calculation step of calculating a guide marker for guiding the moving direction of the imaging means based on the image data and the set geometric condition;
a display control step of displaying the extracted outline of the marker, the calculated guide marker, and the allowable range on a display;
An image processing method characterized by comprising: A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 9 .

Images