JP2006267026A - Image processing method, and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique capable of reducing manhours for index initial setting work required for estimating a camera position and posture by an index image, such as an installation position of an index, a posture thereof, a shape thereof, investigation of a dimension, preparation of the index and installation of the index, and capable of installing the index proper for estimating the camera position and posture, in a short time. <P>SOLUTION: When a virtual space simulated with a real space is set to be conformed with a coordinate system defined in the real space, a position in the real space corresponding to an indication position for arranging the index is found on an image of the virtual space viewed from a desired view point (S400), and an area in the real space is found to correspond to an area occupied on the image by the index (S404). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for obtaining the position and orientation of an image pickup apparatus that picks up an image of a real space in which a plurality of indices are arranged.

  Camera internal parameters such as camera focal length, principal point, lens distortion, etc. that exist inside the camera or camera external parameters that indicate how the camera exists relative to the outside world, such as camera position and orientation, are calibrated In order to do this, a plurality of indices with known coordinate values placed in the three-dimensional space coordinates are imaged by the camera to generate an index image, and the camera uses the spatial coordinate values of each index and the image coordinate values of the indices. There is an estimation method based on a calibration algorithm.

  The camera calibration algorithm includes a method based on an index in which all the indexes are not distributed on one plane (distributed three-dimensionally), and a method based on an index distributed in two dimensions (on a plane). It is widely used because of its advantages such as the ease of arranging indices in space and the development of a method that can also calibrate lens distortion, which is a nonlinear internal parameter.

In the past, many proposals have been made, including the proposal by Roger Tsusai et al. (See Non-Patent Document 1).
Roger Y. Tsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Metrology Using Off-the Shelf TVCameras and Lenses”, IIIE J. Roboticsand Automation, Vol. RA-3, No. 4, pp. 323-344, 1987

  In order to perform the process of estimating the camera position and orientation from the index image without depending on the position and orientation of the camera, the number of indices necessary for the position estimation algorithm that is always used must be present on the index image.

  In addition, in order to keep the accuracy of position and orientation estimation constant regardless of the camera position and orientation, the indices must always be arranged uniformly on the index image.

  In order to realize these, the following index initial setting work is currently performed in an environment for estimating the camera position and orientation.

(1) Examine the possible range of camera position and orientation. (Examination of camera position / posture movable range)
(2) A camera is installed in the position and orientation movable range examined in (1) and imaging is performed, and the installation position of the index is examined from the captured image. (Investigation of index position)
(3) Examine the shape of the index on the captured image when the index is installed at the index installation position studied in (2). (Finger index shape examination)
(4) The size of the index is examined so that the position and orientation estimation algorithm to be used can be recognized from the distance between the index setting position and the camera examined in (2). (Indicator dimension study)
(5) Prepare indicators based on the above considerations. (Indicator creation)
(6) Place the created index at the position examined in (2). (Indicator setting)
(7) A camera is installed in the position and orientation movable range examined in (1), and it is checked whether the position and orientation can be estimated from the captured image (index image). (Camera position and orientation estimation confirmation)
(8) A camera is installed in the position and orientation movable range examined in (1), and it is checked whether the accuracy of camera position and orientation estimation is uniform. (Camera position and orientation estimation accuracy check)
(9) If the position / orientation estimation fails in (7) or the position / orientation estimation accuracy changes greatly in (8), the process returns to (1) and is repeated.

  The index initial setting work requires the experience and familiarity of the worker, and is a sufficiently experienced worker and often requires repeated work, which takes time and labor.

  In view of the above problems, the present invention reduces the man-hours for initial setting work required for camera position and orientation estimation using index images, such as examination of index installation position, orientation, shape, and dimensions, index creation, and index installation. It is an object of the present invention to provide a technique for enabling an appropriate index for camera position and orientation estimation to be installed in a short time.

  It is another object of the present invention to maintain uniform accuracy of position and orientation estimation using an index image in a camera position and orientation movable range.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, when a virtual space that imitates the real space is set to match the coordinate system defined in the real space, a generation step of generating an image of the virtual space viewed from a desired viewpoint;
A display step of displaying the image generated in the generation step on a predetermined display screen;
On the image, a position indicating step for indicating a position where an index is arranged;
A first calculation step of obtaining a position in the real space corresponding to the position indicated on the image;
A setting step for setting an area occupied by the index on the image;
A second calculation step for obtaining a region in the real space corresponding to the region set in the setting step;
A storage step of storing data indicating the position calculated in the calculation step and data indicating the area calculated in the second calculation step in a memory.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, an acquisition step of acquiring a plurality of images by imaging at each position and orientation;
A calculation step of performing processing for each image for calculating the degree of variation in each image of each index shown in the image;
A first notification step of notifying a degree of variation obtained for each image.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, when a virtual space imitating a real space is set to match a coordinate system defined in the real space, a generation unit that generates an image of the virtual space viewed from a desired viewpoint;
Display means for displaying the image generated by the generating means;
On the image, position indicating means for indicating a position where an index is arranged;
First calculating means for obtaining a position in the real space corresponding to the position indicated on the image;
Setting means for setting an area occupied by the index on the image;
Second calculating means for obtaining an area in the real space corresponding to the area set by the setting means;
And storing means for storing data indicating the position calculated by the calculating means and data indicating the area calculated by the second calculating means.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, an acquisition unit that captures images at each position and orientation and acquires a plurality of images;
A calculation means for performing processing for each image to calculate the degree of variation of each index in the image in the image;
First informing means for informing the degree of variation obtained for each image.

  The configuration of the present invention reduces the man-hours for index initial setting work required for camera position and orientation estimation using index images, such as examination of index installation position, orientation, shape, dimensions, index creation, and index installation, in a short time. An index suitable for camera position and orientation estimation can be installed.

  In addition, the accuracy of position / orientation estimation using the index image can be kept uniform in the camera position / orientation movable range.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
In the present embodiment, a process for obtaining an arrangement position of an index to be arranged in the real space when the position and orientation of the imaging apparatus are fixed will be described. As described above, the purpose of use of the marker to be arranged is to estimate the position and orientation of the imaging device that captured the image using a real space image in which a plurality of markers are arranged (processing to obtain or correct). Used).

  FIG. 1 is a block diagram illustrating a hardware configuration of a computer that functions as an image processing apparatus according to the present embodiment. As this computer, a general PC (personal computer), WS (workstation), or the like is used.

  As shown in the figure, it comprises a CPU 110, a memory 120, an HDD (hard disk drive) 130, an I / F (interface) 150, a display device 170, a keyboard 180, and a mouse 190.

  The CPU 110 controls the entire computer and executes processes described later performed by the computer.

  The memory 120 can provide various areas such as an area for temporarily storing programs and data loaded from the HDD 130 and an area used when the CPU 110 executes each process.

  The HDD 130 stores an OS (operating system) and programs and data for causing the CPU 110 to execute processes described below performed by the computer. Some or all of these are loaded into the memory 120 according to control by the CPU 110. The CPU 110 is a processing target.

  The I / F 150 outputs a signal indicating an image or a character that is a processing result of the CPU 110 to the display device 170.

  The display device 170 is configured by a CRT, a liquid crystal screen, or the like, and displays the processing result by the CPU 110 using images, characters, or the like.

  The keyboard 180 and the mouse 190 can be operated by an operator of the computer to input various instructions to the CPU 110.

  Next, processing performed by a computer having such a configuration, that is, processing for obtaining the placement position of an index placed in the real space will be described below with reference to FIG. 2 showing a flowchart of the processing. Note that programs and data for causing the CPU 110 to execute the processing according to the flowchart of FIG. 10 are stored in the HDD 130 and are loaded into the memory 120 as needed under the control of the CPU 110 as necessary. By performing processing using the programmed program and data, the computer executes each processing described below.

  First, “virtual space data imitating real space” stored in the HDD 130 is loaded into the memory 120 (step S310). Here, “virtual space data imitating real space” means “model data for constructing the physical state of the real space in which the index is placed in the virtual space (hereinafter referred to as real space model data)” Is. In the present embodiment, this model data will be described as being composed of polygons.

  The virtual space is set so that its coordinate system matches the coordinate system defined in the real space (that is, the origin in the virtual space and the three axes orthogonal to each other at the origin ( (x-axis, y-axis, z-axis) are in agreement with those defined in the real space). Therefore, one point in the virtual space and one point in the real space corresponding to this one point are represented by the same coordinate value.

  Next, the CPU 110 displays a GUI for setting the position and orientation of the imaging device (viewpoint) arranged in the virtual space on the display screen of the display device 170, and receives an input of the position and orientation (step S314). FIG. 3 is a diagram showing a display example of a GUI for inputting the position and orientation of the viewpoint arranged in the virtual space, and the areas 351 to 356 are respectively the viewpoint position (x, y, z) and the attitude (azimuth, elevation, roll).

  An operator of this computer inputs a desired numerical value in each of the areas 351 to 356 using the keyboard 180 and the mouse 190. When the input is completed and the button image 357 is designated, the CPU 110 detects this and stores the position and orientation data input to the areas 351 to 356 in the memory 120. In the GUI shown in the figure, the position and orientation is defined by the Y-X-Z Euler angle. However, the position and orientation are not limited to this, and may be expressed by a rotation axis and a rotation angle. In that case, the configuration of the GUI displayed in step S314 needs to be changed as appropriate.

  Further, the input method of the viewpoint position and orientation is not limited to the form in which the numerical value is input as described above, and the viewpoint position and orientation is set by operating an icon on the display screen of the display device 170. Also good. In that case, the GUI shown in FIG. 4 is displayed on the display screen of the display device 170 in step S314.

  FIG. 4 is a diagram showing a display example of a GUI for setting the position and orientation of the viewpoint with icons. In the GUI shown in the figure, an image of the virtual space that is seen from the viewpoint of the position and orientation overlooking the real space model is displayed. 210 on this image is an icon for indicating the position and orientation of the viewpoint, the position of the end 215 of the icon 210 indicates the position of the viewpoint, and the direction of the icon indicates the attitude of the viewpoint (gaze direction). ing. Therefore, the operator of this computer can change and set the position and orientation of the viewpoint set in the virtual space by changing the position and orientation of this icon 210 using the keyboard 180 and mouse 190. .

  Further, in addition to the image displayed by the GUI shown in FIG. 4, a line segment 220 extending from the position of the icon 210 in the orientation direction of the icon 210 is displayed as shown in FIG. The position of the icon 210 may be changed by manipulating the position of the point 220 where the minute 220 intersects the real space model. In this case, the posture of the icon 210 (the posture of the viewpoint) is changed so that the line segment connecting the position of one end after the change and the position of the icon 210 is the line-of-sight direction. FIG. 5 is a diagram illustrating a display example of a GUI for setting the position and orientation of the viewpoint with icons.

  As described above, various methods for setting the position and orientation of the viewpoint are conceivable, and the method is not limited to one method.

  Returning to FIG. 2, the process proceeds to step S316, and an image of the virtual space seen from the viewpoint having the position and orientation set in step S314 is generated and displayed on the display screen of the display device 170 (step S316). In addition, since the process which produces | generates the image of the virtual space seen from the viewpoint which has a predetermined position and orientation is known, the description regarding this is abbreviate | omitted.

  FIG. 6 is a diagram illustrating an example of a virtual space image generated in step S316. This image is an image of a virtual space imitating a real space seen from the viewpoint having the position and orientation set in step S314.

  Next, the operator uses the keyboard 180 and the mouse 190 to set an index to be arranged in the real space on the virtual space image displayed in step S316.

  When the operator clicks a desired position on the virtual space image with the mouse 190, the CPU 110 displays an index icon at the click position on the image. Of course, when a plurality of locations are clicked, an index icon is displayed at each click position. FIG. 7 is a diagram illustrating a display example when four places are clicked on the virtual space image. Reference numerals 250a to 250d are index icons indicating positions clicked by the operator on the virtual space image. As the index icon, any icon may be used as long as the click position can be visually recognized.

  In addition, the display position of the index icon can be changed by moving and dragging the mouse cursor on the displayed index icon. Further, the index icon can be deleted by moving the index icon outside the image. In this way, one or more index icons can be arranged on the virtual space image, the position on the image can be moved, or deleted. In addition, about the operation method for that, it is not limited to the above.

  Accordingly, in step S318, when a click is detected on the virtual space image, the click position is recorded in the memory 120. When an operation for moving the position of the index icon is detected, the position of the index icon recorded in the memory 120 is recorded. When the operation is changed according to the operation and the operation for deleting the index icon is received, the position of the index icon recorded in the memory 120 is deleted. In this way, the CPU 110 manages the index icon position data set by the operator in the memory 120.

  In step S318, the index icon is displayed at the position on the virtual space image of each index icon related to the memory 120 (step S320).

  When an operation for ending the setting of the index on the virtual space image, for example, an operation of double-clicking the mouse 190 is detected, the process proceeds to step S324 via step S322. The processes of S318 and S320 are repeated.

  Then, using the position data of each index icon managed in the memory 120, it is determined which position in the real space the position indicated by the index icon on the virtual space image corresponds to ( Step S324).

  The process in step S324 will be described in detail. FIG. 8 is a diagram illustrating processing for obtaining a position in the real space corresponding to the position of the index icon on the virtual space image. In the figure, 280 is the viewpoint, 282 is the line of sight of the viewpoint 280, 270 is the virtual space image, 250a is the index icon, 260a is a straight line passing through the position of the viewpoint 280 and the position of the index icon 250a on the virtual space image 270, An intersection with the space model 800 is shown. Here, the line of sight 282 passes through the center position of the virtual space image 270.

  The virtual space image 270 is an image obtained by viewing the virtual space represented by the real space model 800 from the viewpoint 280. Therefore, the position of 260a is projected to the intersection position between the straight line connecting this position and the position of the viewpoint 280 and the image plane (virtual space image 270). This intersection position is the position of 250a.

  Therefore, if the intersection of the straight line passing through the position of the viewpoint 280 and the position of the index icon 250a on the virtual space image 270 and the real space model 800 is obtained, the position of this intersection is the position of the index icon 250a in the virtual space image 270. It can be obtained as a position in the real space corresponding to the position. Here, although the obtained coordinate position of the intersection is in the virtual space, as described above, the coordinate system of the virtual space and the real space is the same, so the coordinate value obtained in the virtual space is the real space as it is. It can be a coordinate value at.

  In this way, it is possible to determine to which position in the real space an arbitrary point on the virtual space image 270 corresponds. Note that a detailed method for obtaining the intersection is widely known, and a detailed description thereof is omitted here.

  When the instruction to save the data indicating the position in the real space corresponding to the position of each index icon obtained in step S324 in the virtual space image is input by the keyboard 180 or the mouse 190, the CPU 110 When this is detected, the process proceeds to step S332 via step S330, and is stored in the HDD 130 (step S332).

  According to the above description, according to the present embodiment, the indicator placement in the real space can be simulated with a simple operation on the computer before the indicator is actually placed in the real space. It is possible to reduce the work process until the placement.

  Note that the GUI operation method described in this embodiment is not limited to the above. In the present embodiment, the real space model is described as being composed of polygons. However, the present invention is not limited to this.

[Second Embodiment]
In the present embodiment, in addition to the calculation of the index placement position in the real space described in the first embodiment, the index shape is also obtained. Note that the same computer as that of the first embodiment is used for such processing.

  FIG. 9 is a flowchart of processing for obtaining the shape of an index in addition to calculating the position of the index in the real space. Note that programs and data for causing the CPU 110 to execute the processing according to the flowchart of FIG. 10 are stored in the HDD 130 and are loaded into the memory 120 as needed under the control of the CPU 110 as necessary. By performing processing using the programmed program and data, the computer executes each processing described below.

  First, the processing from step S310 to step S324 is performed, and the position data in the real space of each index icon set by the operator is stored in the HDD 130 (step S400).

  Here, the index image obtained when the real space in which the index is actually arranged at the index arrangement position determined in the above process is imaged by the imaging device arranged in the position and orientation of the viewpoint includes one or more indices. However, each index is reflected in various shapes and sizes. In the subsequent processing, the shape (shape, size) of each index is obtained so that each index appears in the same shape and size on the index image regardless of the distance to the imaging apparatus and the position and orientation relationship.

  For this purpose, first, the CPU 110 displays on the display screen of the display device 170 a GUI for inputting the size to which all the indices should be captured when the indices are captured on the index image, and waits for input (step S402). ). In other words, this input corresponds to setting an area indicated by each index on the index image. In the present embodiment, since all indexes appear as circles on the index image, the information to be input is the area of this circle.

  FIG. 10 is a diagram showing a GUI display example for inputting the size at which all indices should be captured when the indices are captured on the index image. The operator uses the keyboard 180 and the mouse 190 to enter the area 1001. A numerical value indicating the desired size (in this embodiment, the area of a circle) is input. When the button image 1002 is instructed after input, the CPU 110 detects this, advances the processing to step S404, and displays each index as a circle of the area input in step S402 on the index image. The shape of each index is obtained (step S404).

  Details of the processing in step S404 will be described. FIG. 11 is a diagram illustrating an example of the index image. As shown in the figure, four indicators 1101 to 1104 are shown on the image, but they are different in size and shape because they are different in distance and position / posture relationship from the viewpoint. In this case, even if the index is intended to be created with a sufficient size, it may not be used for estimating the position and orientation of the imaging apparatus because it falls below a threshold value to be regarded as an index on the image. In addition, the apparent index on the index image is distorted unless the line-of-sight direction of the imaging apparatus and the plane on the real space object on which the index is set face each other. In most cases in the imaging apparatus position / orientation estimation algorithm, the position of the index is often determined by the center of gravity of the area in the index image. However, when this center of gravity is obtained from the area, distortion may cause an error.

  In this embodiment, in view of this, the shape of each index is obtained in advance so that it is reflected on the index image with the same size and the same shape, and the index is created with the obtained shape and arranged in the real space. .

  FIG. 12 is a diagram showing indices that move onto the index image when the indices whose shapes are determined in step S404 are arranged in the real space. In the figure, reference numerals 1201 to 1204 denote indices shown on the index image. As shown in the figure, each index is shown in the same size and shape.

  Details of the process in step S404 will be described below.

  FIG. 13 is a diagram illustrating a relationship between an index displayed in a circle of a desired size on the index image and the imaging device. In the figure, Pe is the imaging device (viewpoint), d is the line of sight of the viewpoint Pe, m is the index image plane, C is the center position of the index image plane m, and is also the intersection position of the index image plane m and the line of sight d. W is a display area of an index displayed in a circle of a desired size, Mc is the center position of the display area W, Mr is a radius of the display area W, P1, P2,. A finite number of points arranged uniformly on (contour) are shown. The number of points is not particularly limited, but the more the number of points, the more finally obtained “index shape” can be obtained more accurately.

  First, the CPU 110 obtains line segments PeP1, PeP2,... PePn connecting the viewpoint Pe and each of P1, P2,.

  FIG. 14 is a diagram illustrating intersections that occur with the real space model when each of the line segments PeP1, PeP2,... PePn is extended. In the figure, p indicates a polygon where this intersection exists.

  If the “method for obtaining the intersection of the line segment and the real space model” used in the first embodiment is used, the intersection point of the line segment PeP1 and the real space model (polygon p in the figure) is P1 ′, The intersection point between the segment PeP2 and the real space model can be obtained as P2 ′, and the intersection point between the line segment PePn and the real space model can be obtained as Pn ′. Then, when P1 ', P2', and Pn 'are connected in order, a region shown by m' is obtained. This area corresponds to an area obtained by projecting the area W into the real space. Further, the center Mc 'of the region m' is at the position where Mc is projected in the real space. Thus, since the process which projects the point on a parameter | index image in real space has been demonstrated in 1st Embodiment, the description regarding this is abbreviate | omitted.

  Therefore, if a point corresponding to P1 ′, P2 ′,... Pn ′ is provided in the real space and an area m ′ obtained by sequentially connecting the points is used as an index, this index is an area on the index image. It appears as W.

  By performing such processing for other indexes in the same manner, the shape of each index can be obtained so that each index appears on the index image with the same shape as the region W.

  Accordingly, in step S404, a circular area having the area indicated in step S402 is set at the position of the index icon on the virtual space image displayed on the display screen of the display device 170, and on the circumference of this circular area. A plurality of points are set, and intersection points between the straight line from the viewpoint position and each point set on the circumference of the circular area and the real space model are obtained. The obtained position data of each intersection is the shape data of the index indicated by the index icon.

  As shown in FIG. 15, the projections hline and vline, hline and vline on the horizontal and vertical lines passing through Mc ′ and the projections of the index contours Pv1 ′, Pv2 ′, Ph1 ′ and Ph2 ′ are calculated. However, as shown in FIG. 16, if the index is printed on the printed image, the index can be set with an accurate posture.

  Returning to FIG. 9, when an instruction to save the data indicating the shape of each index obtained in step S404 is input by the keyboard 180 or the mouse 190, the CPU 110 detects this and performs the process in step S430. Then, the process proceeds to step S432 and is stored in the HDD 130 (step S432). Note that what is stored here is the position data itself. However, the present invention is not limited to this, and in the case of the above description, other information may be used as long as the data specifies the region m ′.

  Therefore, when creating an index using the obtained shape data, an image of the index with the center of gravity position, the horizontal line passing through the center of gravity position, and the vertical line is created using the position data of the point cloud indicated by the shape data. What is necessary is just to install what printed this on the paper in the calculated position and orientation.

  In addition, since all indexes are displayed on the index image with an equal size, even if an index-like object is detected from the index image, it can be determined that the index is not an index if the size is equal to or less than this equal size. Even if noise is added to the image, it can be handled without any problem.

  In this embodiment, the shape of the index is described as being circular, but is not limited to being circular. Further, in the present embodiment, the display size of the index shown on the index image is input by numerical input. However, the present invention is not limited to this, for example, an area including the position of the index icon on the virtual space image. May be designated directly with the mouse 190 or the like as the “region where the index is shown”.

[Third Embodiment]
In the first and second embodiments, the position and shape of the index are obtained assuming that the position and orientation of the imaging device are fixed. As a result, when the position and orientation of the imaging apparatus are fixed, the accuracy of the position and orientation estimation can be improved.

  In the present embodiment, even when the position and orientation are changed within a certain range, the position and orientation estimation accuracy of the imaging device is made uniform. Note that the computer used in the present embodiment is the same as that in the first embodiment.

  In this embodiment, the position / orientation movable range of the imaging apparatus is not a continuous value but a discrete finite value (as shown in FIG. 18, three sets of position / orientation (position: Ei). The case of (i = 1, 2, 3) and posture EiP (i = 1, 2, 3)) will be described.

  In the case described above, it is clearly a discrete value, but the position Ei (i = 1, 2, 3,... N) posture EiPj (i = 1, 2, 3,..., N, j = 1, 2, (3,...) M), if n and m are made sufficiently large, the discrete interval becomes small, and the position and orientation estimation accuracy can be made uniform in a substantially continuous movable range.

  FIG. 17 is a flowchart of processing performed to make the position and orientation estimation accuracy of the imaging apparatus uniform even when the position and orientation are changed within a certain range. Note that programs and data for causing the CPU 110 to execute the processing according to the flowchart of FIG. 10 are stored in the HDD 130 and are loaded into the memory 120 as needed under the control of the CPU 110 as necessary. By performing processing using the programmed program and data, the computer executes each processing described below.

  First, the real space model data is loaded into the memory 120 in the same manner as the processing in step S310 (step S410). Next, an input of the position of the imaging device is received using a GUI as shown in FIG. 4 (step S414). For example, when the operator moves the position of the icon 210 to a desired position and then presses a predetermined button on the keyboard 180 or the mouse 190, the CPU 110 detects this, and provides data indicating the position of the imaging device at that time. Register in the memory 120. If such processing is performed a plurality of times, a plurality of pieces of data indicating the imaging position are registered in the memory 120.

  Next, since the operator designates one point to be imaged in this GUI, the CPU 110 calculates an orientation for viewing this one point from each imaging position set in step S414, and calculates each calculated point. Is registered in the memory 120 (step S415). As described above, a plurality of sets of imaging positions and orientations are registered in the memory 120, but the registration form is not particularly limited.

  In this embodiment, the movable range Ei (i = 1, 2, 3) of the position of the imaging apparatus and the movable range EiP (i = 1, 2, 3) of the posture are registered in the memory 120.

  Next, the CPU 110 generates an image of the real space model (virtual space image) that can be seen from each position and orientation, and displays it on the display screen of the display device 170 (step S416). FIG. 19 is a diagram showing real space model images V 1, V 2, and V 3 that can be seen from the respective positions and orientations. Each of the images V 1 to V 3 is displayed as a list on the display screen of the display device 170.

  Next, the operator can perform operations such as placing, moving, and deleting the index icon on each image by the operation method described in the first embodiment. The CPU 110 accepts this operation, and manages the position data of each index icon in the memory 120 (step S418).

  In step S418, the index icon is displayed at the position on the virtual space image of each index icon related to the memory 120 (step S420).

  Here, when the operator places an index icon on the image V1 or moves the position of the placed index icon, the CPU 110 performs the same processing as the processing in step S324, and the index icon on the image V1. The position in the real space corresponding to the position at is obtained (step S418). Then, the obtained position is projected on the image planes of the images V2 and V3, and when projected into the images V2 and V3, an index icon is displayed at the projected position (step S420).

  That is, when an index icon is arranged in one image and the position in the real space corresponding to the arranged index icon can be reflected in other images, the index icon is arranged at the position where the index icon appears.

  The same applies to the case where the index icon is deleted. When the index icon is deleted in one image, the position of the deleted index icon in the real space is obtained, and the obtained position is projected on the image plane of the other image. In this case, when the image is projected in the image, the index icon displayed at the projected position is deleted.

  FIG. 20 is a diagram illustrating the relationship between the real space and the images V1 to V3. In this way, it is possible to confirm in which position an index placed on an image captured at a certain position / orientation E1, E1P appears in an image captured at another position / orientation E2, E2P, E3, E3P.

  Returning to FIG. 17, when an operation for ending the setting of the index on the virtual space image, for example, an operation of double-clicking the mouse 190 is detected, the process proceeds to step S424 via step S422. Unless detected, the processes of steps S418 and S420 are repeated.

Next, an index arrangement evaluation index is obtained for each of the images V1 to V3 (step S424). The index arrangement evaluation index indicates the degree of variation of the index in the image. If the index icon arrangement position in the image is (x _k , y _k ), k = 1, 2,. It is defined by the formula of

n
D = Σ {(x _k −x _l ) + (y _k −y _l )}
k, l = 1
This index arrangement evaluation index is the total value of the distances between the index icons in the image. The larger the value of D, the more the index icons arranged in the image vary. The smaller the indicator icons, the denser the indicator icons arranged in the image.

  Then, the calculated index arrangement evaluation index is displayed in the vicinity of the image (step S426). That is, the index arrangement evaluation index D1 obtained for the image V1 is displayed in the vicinity of the display position of the image V1, the index arrangement evaluation index D2 obtained for the image V2 is displayed in the vicinity of the display position of the image V2, and the index obtained for the image V3. The arrangement evaluation index D3 is displayed near the display position of the image V3.

  Here, when the values of the respective index arrangement evaluation indices D1 to D3 are close to each other (in other words, when the respective index arrangement evaluation indices D1 to D3 are within a predetermined range), which position This shows that the uniformity of the index in the image does not change much even when viewed from the posture.

  Therefore, the operator determines the uniformity by looking at the respective index arrangement evaluation indices D1 to D3 displayed on the display screen of the display device 170, and is desired when it is determined that further uniformity is necessary. In order to obtain the uniformity, the index icons are further arranged or the arranged ones are moved to obtain the index arrangement evaluation index again, and the operation of judging by looking at this is repeated.

  Therefore, the CPU 110 may calculate the difference between the index placement evaluation indices D1 to D3 and display the difference on the display screen of the display device 170 in order to notify the operator of the uniformity as a numerical value. And when each index arrangement | positioning evaluation index | exponent D1-D3 is settled in the predetermined range, you may make it alert | report by displaying that on the display screen of the display apparatus 170. FIG. In addition, about the alerting | reporting form, what used the audio | voice other than this can also be considered and it does not specifically limit.

  Therefore, the CPU 110 repeats the processing from step S418 to step S426 as long as the processing for arranging, moving, and deleting the index icon is performed. However, the operation to end such processing is performed by the keyboard 180 and the mouse 190. When an instruction to save the position data of all the indices managed in the memory 120 at this time is input, this is detected and the process is performed via steps S428 and S440. Then, the process proceeds to step S442, and the position data of all the indices managed in the memory 120 at this time are stored in the HDD 130 (step S442).

  The processing according to the present embodiment may be combined with the second embodiment. For example, after the index placement processing for one image is performed on images from viewpoints having various positions and orientations, The index may be edited, such as movement or deletion of the index placed on the image, or placement of a new index.

[Fourth Embodiment]
FIG. 21 is a diagram illustrating a hardware configuration of the computer according to the present embodiment. As illustrated in FIG. 21, the computer according to the present embodiment includes the configuration of the computer according to the first embodiment illustrated in FIG. 1. An image reading unit 140 is added, and the imaging device 160 is connected to the image reading unit 140.

  The imaging device 160 is a stereo camera device having a plurality of lenses, and is a device capable of acquiring a distance to a subject on an image taken together with photographing. Real space model data is automatically created from this distance and image. As a specific method for creating, a generally used technique may be used. In this embodiment, a stereo camera device is assumed. However, the present invention is not limited to this, and a device such as a laser scanner may be used.

  By using the real space model data generated in this way, it is not necessary to perform the actual space actual measurement and data input steps required when creating the real space model data, thereby reducing labor. it can.

  In addition, according to each of the above-described embodiments, it is possible to examine the index installation position, dimensions, and shape in advance, and the effect of making it unnecessary to occupy the camera installation site is obtained. This is particularly effective when a plurality of vendors work together and work at the same time, and are set up at a venue such as an exhibition or event where time constraints for installation and setting are severe.

[Other Embodiments]
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

1 is a block diagram illustrating a hardware configuration of a computer that functions as an image processing apparatus according to a first embodiment of the present invention. It is a flowchart of the process which calculates | requires the arrangement position of the parameter | index arrange | positioned in real space. It is a figure which shows the example of a display of GUI for inputting the position and orientation of the viewpoint arrange | positioned in virtual space. It is a figure which shows the example of a display of GUI for setting the position and orientation of a viewpoint with an icon. It is a figure which shows the example of a display of GUI for setting the position and orientation of a viewpoint with an icon. It is a figure which shows an example of the image of the virtual space produced | generated by step S316. It is a figure which shows the example of a display when four places are clicked on a virtual space image. It is a figure explaining the process which calculates | requires the position in the real space corresponding to the position of the index icon on a virtual space image. It is a flowchart of the process which calculates | requires the shape of a parameter | index in addition to calculation of the arrangement | positioning position of the parameter | index in real space. It is a figure which shows the example of a display of GUI for inputting the size which should show all the indices, when an index is reflected on an index image. It is a figure which shows an example of a parameter | index image. It is a figure which shows the parameter | index which moves on a parameter | index image when each parameter | index determined in step S404 is arrange | positioned in real space. It is a figure which shows the relationship between the parameter | index displayed with the circle | round | yen of desired size on an parameter | index image, and an imaging device. It is a figure which shows the intersection which arises with a real space model when each of line segment PeP1, PeP2, and PePn is extended. It is a figure which shows the intersection Pv1 ', Pv2', Ph1 ', Ph2' of the projection of hline and vline, hline and vline, and the projection of an index outline to p of the horizontal line and vertical line which pass through Mc '. It is a figure which shows an example of the printing image of a parameter | index. 10 is a flowchart of processing performed to make the position and orientation estimation accuracy of the imaging apparatus uniform even when the position and orientation are changed within a certain range. It is a figure explaining three sets of position and orientation. It is a figure which shows the image V1, V2, V3 of the real space model visible from each position and orientation. It is a figure which shows the relationship between real space and each image V1-V3. It is a figure which shows the hardware constitutions of the computer which concerns on the 4th Embodiment of this invention.

Claims (9)

A generation step of generating an image of the virtual space viewed from a desired viewpoint when a virtual space imitating the real space is set to match the coordinate system defined in the real space;
A display step of displaying the image generated in the generation step on a predetermined display screen;
On the image, a position indicating step for indicating a position where an index is arranged;
A first calculation step of obtaining a position in the real space corresponding to the position indicated on the image;
A setting step for setting an area occupied by the index on the image;
A second calculation step for obtaining a region in the real space corresponding to the region set in the setting step;
An image processing method comprising: a storage step of storing data indicating a position calculated in the calculation step and data indicating a region calculated in the second calculation step in a memory. In the first calculation step, an intersection position between the virtual space and a straight line connecting the position of the viewpoint and the position designated on the image is obtained,
2. The storage step stores the intersection point obtained in the first calculation step in the memory as a position in the real space corresponding to the position indicated on the image. An image processing method described in 1. In the second calculation step,
A point setting step of setting a finite number of points on the contour of the index on the image;
An intersection calculation step of performing a process for obtaining an intersection between the virtual space and the straight line connecting the point set in the point setting step and the viewpoint, for each point on the contour,
In the storing step, the intersection calculation step has been determined so that a region represented by the intersection group obtained in the intersection calculation step is a region in the real space corresponding to the region set in the setting step. 3. The image processing method according to claim 1, wherein data indicating the position of each intersection is stored in the memory. An acquisition step of capturing images at each position and orientation to acquire a plurality of images;
A calculation step of performing processing for each image for calculating the degree of variation in each image of each index shown in the image;
An image processing method comprising: a first notification step of notifying a degree of variation obtained for each image.   The image processing method according to claim 4, further comprising a second notification step of notifying that when the degree of variation obtained for each image is within a predetermined range. Generating means for generating an image of the virtual space viewed from a desired viewpoint when a virtual space imitating the real space is set to coincide with a coordinate system defined in the real space;
Display means for displaying the image generated by the generating means;
On the image, position indicating means for indicating a position where an index is arranged;
First calculating means for obtaining a position in the real space corresponding to the position indicated on the image;
Setting means for setting an area occupied by the index on the image;
Second calculating means for obtaining an area in the real space corresponding to the area set by the setting means;
An image processing apparatus comprising: storage means for storing data indicating a position calculated by the calculation means and data indicating an area calculated by the second calculation means. Acquisition means for capturing images at each position and orientation to acquire a plurality of images;
A calculation means for performing processing for each image to calculate the degree of variation of each index in the image in the image;
An image processing apparatus comprising: a first notification unit that notifies a degree of variation obtained for each image.   A program for causing a computer to execute the image processing method according to any one of claims 1 to 5.   A computer-readable storage medium storing the program according to claim 8.

Images