JPWO2013105205A1 - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

The image processing apparatus generates a registration image from an image obtained by capturing an object of interest from any direction with a single camera without using other devices such as a turntable and an acceleration sensor on which the object of interest is installed. The geometric transformation method determination means 16 is based on the first reference point determined by the first reference point projection means 13 and the second reference point set by the second reference point setting means 15 for each conversion target image. Then, parameters of a predetermined geometric transformation formula are calculated. The geometric conversion unit 17 performs geometric conversion on the conversion target image using the geometric conversion formula based on the parameters calculated by the geometric conversion method determination unit 16.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program, and in particular, generates an image in which a target object is aligned at an arbitrary position in the image from a plurality of conversion target images including the target object. The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  By continuously switching the image of the object of interest photographed from various distances and angles, and combining the position of the object of interest, the rotation direction, and the size, it is possible to present a video that is easy to see for the user. Hereinafter, an image obtained by combining the position, rotation direction, and size of the object of interest is referred to as an alignment image.

  One example of a method for capturing an alignment image in which the object of interest is located at the center of the image is described in Patent Documents 1 to 3 and Non-Patent Document 1. In the imaging method described in Patent Document 1, an object of interest is imaged from various angles using a plurality of cameras and a supporting unit that supports them.

  The imaging device described in Patent Document 2 captures an object of interest from various angles using a plurality of cameras and a turntable connected to a computer.

  In the image input method described in Patent Document 3, the object of interest is photographed from various angles with an unfixed camera, and the center position of the object of interest in the image is determined from the camera posture information detected by the posture detector. A two-dimensional geometric transformation matrix for alignment is calculated and image transformation is performed.

  In the image conversion method described in Non-Patent Document 1, an object of interest and a square marker are photographed from various distances and angles with an unfixed camera that moves in the horizontal direction, and based on camera posture information calculated from the square marker. A two-dimensional geometric transformation matrix for generating a registration image is calculated, and image transformation is performed.

JP 2004-264492 A JP 2007-72537 A Japanese Patent Laid-Open No. 2005-49999 JP 2000-97637 A

Manabu Nakano, Junji Taji, Yuzo Senda, “Generation of Low Computation Free Viewpoint Images Using Monocular Cameras and AR Markers”, Proc. 9 (3), 189-190, 2010-08-20

  However, in the methods and apparatuses described in Patent Documents 1 to 3, in order to generate the alignment image of the target object, a plurality of cameras and target objects whose installation locations are fixed are installed in addition to the single camera. A large number of devices such as a turntable and an acceleration sensor are required.

  For example, in the photographing method described in Patent Document 1, the support unit requires a number of devices such as a plurality of camera platforms for fixing the camera, a support frame that supports the camera platform, support arms, and columns. In addition, the photographer must arrange a plurality of cameras on the same straight line, in the same arc shape, on the same plane, or in the same shape, and preparation for taking a picture is very complicated.

  In the photographing apparatus described in Patent Document 2, the operation of the turntable is controlled by a computer. The cameras are installed at equal intervals on the arcuate imaging device installation base. Since the photographer can start photographing simply by placing the object of interest on the turntable, knowledge of the camera and work is not necessary. However, in the same way as the photographing method described in Patent Document 1, many devices other than the camera are required.

  In addition, the image input method described in Patent Document 3 does not require a plurality of cameras or an installation base for fixing the cameras. Therefore, the photographer only needs to photograph with the camera. However, similarly to the imaging devices described in Patent Literature 1 and Patent Literature 2, devices other than cameras such as an attitude detection device such as an acceleration sensor and a magnetic sensor are required.

  In addition, the image conversion method described in Non-Patent Document 1 does not require a plurality of cameras and an installation base for fixing the cameras, and can generate an image with only a single camera. However, the camera can move and shoot only in the horizontal direction, and the size of the object of interest cannot be adjusted when moving and shooting in the vertical direction.

  Therefore, the present invention can generate a registration image from an image obtained by capturing an object of interest from a free direction with a single camera without using other devices such as a turntable or an acceleration sensor for installing the object of interest. An object is to provide an image processing apparatus, an image processing method, and an image processing program.

  An image processing apparatus according to the present invention is an image processing apparatus that generates a registration image of a target object from a plurality of conversion target images including the target object, and for each conversion target image, a camera that estimates a camera position and orientation at the time of shooting Based on the position and orientation estimation means and the camera position and orientation estimated by the camera position and orientation estimation means, the first reference point space coordinates represented by the three-dimensional coordinates in each conversion target image are projected onto each conversion target image, First reference point projection means for generating a first reference point expressed by image coordinates in the conversion target image, second reference point conversion source coordinates expressed by image coordinates, and camera position and orientation are received, and the camera position and orientation are accepted. A second reference point setting means for setting a second reference point expressed by image coordinates that is a conversion destination of the first reference point, and a first reference point projecting means for each conversion target image. A geometric transformation method determining means for calculating a parameter of a predetermined geometric transformation formula based on the set first reference point and the second reference point set by the second reference point setting means; And a geometric transformation means for geometrically transforming the transformation target image using a geometric transformation formula based on the parameters calculated by the above.

  An image processing method according to the present invention is an image processing method for generating a registration image by performing geometric transformation on each of conversion target images, which are a plurality of images including an object of interest. The first reference point space coordinates represented by the three-dimensional coordinates in each conversion target image are projected on each conversion target image based on the estimated camera position and posture, and The first reference point expressed by the image coordinates is generated, the second reference point conversion source coordinates and the camera position and orientation expressed by the image coordinates are received, and the image that is the conversion destination of the first reference point according to the camera position and orientation A second reference point expressed in coordinates is set, and a parameter of a predetermined geometric transformation formula is calculated for each conversion target image based on the first reference point and the second reference point. Characterized by geometric transformation the converted image using the geometric transformation formula.

  An image processing program according to the present invention is an image processing program for generating a registration image by performing geometric transformation on each of conversion target images, which are a plurality of images including an object of interest. For the image, the process for estimating the camera position and orientation at the time of shooting and the first reference point space coordinates expressed by the three-dimensional coordinates in each conversion target image are projected onto each conversion target image based on the estimated camera position and orientation Then, the first reference point expressed by the image coordinates in each conversion target image, the second reference point conversion source coordinates expressed by the image coordinates, and the camera position / posture are received, and the first reference point is received according to the camera position / posture. Based on the first reference point and the second reference point for each conversion target image, a process for setting a second reference point expressed by image coordinates that is a conversion destination of one reference point, and a predetermined reference point A process for calculating what parameters of the conversion formulas, characterized in that to execute a process of geometric transformation the converted image using the geometric conversion equation according to the calculated parameters.

  According to the present invention, alignment is performed from an image obtained by capturing an object of interest from a free direction with a single camera without using a device such as a turntable on which the object of interest is installed or an acceleration sensor. An image can be generated.

It is a block diagram which shows the structural example of 1st Embodiment. It is a flowchart which shows an example of operation | movement of 1st Embodiment. It is explanatory drawing which shows the outline | summary of the conversion process of a 1st Example. It is explanatory drawing which shows the outline | summary of the conversion process of a 1st Example. It is explanatory drawing which shows the outline | summary of the conversion process of a 1st Example. It is a block diagram at the time of applying this invention to an information processing system. It is a block diagram which shows the outline | summary of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present invention.

  The image processing apparatus shown in FIG. 1 includes a camera position / posture estimation unit 11, a first reference point reception determination unit 12, a first reference point space coordinate projection unit 13, an image display unit 14, and a second reference point setting unit. 15, a geometric transformation method determining means 16, and a geometric transformation means 17.

  The camera position / orientation estimation unit 11 estimates the position and orientation of the camera that captured the conversion target image for each of a plurality of images including the target object (hereinafter referred to as a conversion target image), and determines the estimated position and the estimated orientation. Output to the first reference point space coordinate projection means 13. In order to estimate the position and orientation of the camera, various methods such as a method using a predetermined marker and a method using a correspondence between a known three-dimensional coordinate and an image coordinate where the coordinate is observed can be used. In addition, when a sensor such as an acceleration sensor or a gyro sensor is built in or attached to the camera and it is not necessary to prepare sensors separately, data obtained from them can be used.

  The first reference point reception determination unit 12 receives first reference point space coordinates represented by three-dimensional coordinates for the conversion target image as an input from the user. The first reference point reception determination means 12 receives a determination signal for the received first reference point space coordinates, and determines the first reference point space coordinates.

  The first reference point is a reference point on the input side among points that serve as a reference for input of geometric transformation and points that serve as a reference for output. That is, the first reference point is a reference point represented by position information (image coordinates) in the coordinate system of the conversion target image. The reference point is set for each conversion target image. The reference point may be, for example, two or more points that constitute the rotation axis of the camera in an alignment image (for example, a 360-degree rotated image of the object of interest) that is desired to be represented by a converted image.

  Note that the rotation axis of the camera is not the rotation axis of the camera at the time of actual shooting, but the axis that the user wants to use as the center of camera movement in the target alignment image. Further, the number of points necessary as reference points (number of corresponding points) can be set according to the geometric transformation method, but is 2 or more in any method. The second reference point is a reference point on the output side, and is a point to which the first reference point is converted. That is, the second reference point is a reference point represented by position information in the coordinate system of the converted image. Each point of the first reference point and each point of the second reference point have a correspondence relationship that they are image coordinates on which the same three-dimensional coordinates are projected. Further, in the generation of the alignment image, it is more preferable that the image coordinates indicating the same three-dimensional coordinates between the conversion target images, such as two or more points constituting the rotation axis of the camera, be used as the reference point.

  The first reference point space coordinate projecting means 13 is configured to generate first reference point space coordinates (or a line segment connecting them) based on the input first reference point space coordinate information and the camera position and orientation of the conversion target image. Is projected onto the image to be converted and output to the image display means 14 as the first reference point.

  For example, the first reference point space coordinate projection unit 13 obtains information (position information and the like) of the first reference point space coordinates in accordance with a user operation performed on each conversion target image displayed on the image display unit 14. The first reference point space coordinates may be received by inputting. The first reference point space coordinate projecting means 13 projects the received first reference point space coordinates to image coordinates based on the camera position and orientation estimated by the camera position and orientation estimating means 11, for example, and passes through the image display means 14. To display on the image to be converted.

  The first reference point space coordinate projection means 13 moves the first reference point on the conversion target image based on the information when the information of the first reference point space coordinate is changed. The information input of the first reference point space coordinates is performed, for example, through pointing with a mouse or the like or a keyboard. The first reference point space coordinate projecting means 13 finally receives a first reference point determination signal (information for instructing to determine the first reference point as a target of the first reference point determination signal). The first reference point space coordinates at that time may be determined as the first reference point. The first reference point space coordinate projection means 13 determines at least two points as the first reference point.

  The first reference point space coordinate projecting means 13 projects and displays two points set as initial values of the first reference point space coordinates in advance in the drawing area of the operation screen, and then displays the two points. The position may be moved to the user.

  When the first reference point space coordinate projection unit 13 determines the first reference point, the first reference point space coordinate projection unit 13 outputs the determined first reference point to the geometric transformation method determination unit 16.

  The image display means 14 displays an image. In the present embodiment, the image display means 14 displays a conversion target image, or displays an image in which first reference point space coordinates and second reference point conversion source coordinates described later are superimposed on the conversion target image. Further, the second reference point may be displayed within the frame of the converted image that is the image after conversion, or the converted image may be displayed. For example, the image display unit 14 may display image information of an operation screen including these displays.

  The second reference point setting unit 15 calculates a second reference point that is a conversion destination of the first reference point based on the camera position and orientation and the second reference point conversion source coordinates, and outputs the second reference point to the geometric conversion method determination unit 16. . The second reference point setting unit 15 receives the second reference point conversion source coordinate determination signal and determines the second reference point conversion source coordinates.

  For example, when the first reference point is a plurality of image coordinates indicating the positions of points (two or more) constituting the rotation axis of the camera in the conversion target image, the rotation of the camera in the converted image is performed. What is necessary is just to set it as the some image coordinate which shows the position which projects each point which comprises an axis | shaft.

  The second reference point conversion source coordinates may be, for example, the image coordinates of each point constituting the rotation axis when the camera is in an arbitrary position and orientation. However, the second reference point setting means 15 appropriately selects the second reference point conversion source coordinate based on the camera position and orientation when a camera position and orientation different from that when the second reference point conversion source coordinate is set is input. A second reference point is set.

  The second reference point determination signal is a signal indicating whether the user has determined whether the second reference point is calculated or not. For example, the second reference point is displayed together with the conversion target image via the image display means 14, and the user inputs a determination signal when satisfied. Information input of the second reference point determination signal is performed, for example, through pointing with a mouse or the like, a keyboard, or the like.

  Note that the second reference point conversion source coordinates may be set in advance. Each point of the second reference point is set in correspondence with each point of the first reference point. If preset, reading of the setting information corresponds to inputting the second reference point conversion source coordinates.

  Based on the first reference point output from the first reference point space coordinate projection means 13 and the second reference point output from the second reference point setting means 15, the geometric conversion method determination means 16 Determine the geometric transformation method to be applied to the image. Specifically, the geometric transformation method determination means 16 determines a parameter of a transformation formula (coordinate transformation formula) for geometric transformation to be applied to the transformation target image. The geometric transformation technique determination means 16 is used in the transformation technique based on a necessary number of first reference points and second reference points set for each conversion target image, for example, according to a preset transformation technique. A parameter of a predetermined conversion formula (coordinate conversion formula) may be determined. The geometric transformation technique determination means 16 outputs the decided parameter or a coordinate transformation formula including the decided parameter to the geometric transformation means 17 as information indicating the geometric transformation technique.

  The geometric conversion unit 17 performs geometric conversion on each conversion target image based on the geometric conversion method determined by the geometric conversion method determination unit 16 and outputs a converted image. The geometric conversion unit 17 obtains a converted image by converting pixel information of the conversion target image using, for example, a coordinate conversion formula input as information indicating a geometric conversion method. This conversion process includes a process of interpolating the converted image coordinates to compensate for pixel loss. The geometric conversion unit 17 may be configured to include the function of the geometric conversion method determination unit 16. That is, the geometric transformation method determining unit 16 may be mounted on the geometric transformation unit 17.

  The first reference point reception determination unit 12 and the second reference point setting unit 15 are realized by, for example, an information input device such as a mouse, a keyboard, and a touch panel, and a CPU (Central Processing Unit) operated by a program. Moreover, the image display means 14 is implement | achieved by CPU which operate | moves according to image display apparatuses, such as a monitor and a projector, and a program, for example. Further, the geometric transformation method determining means 16 and the geometric transformation means 17 are realized by, for example, hardware designed to perform specific arithmetic processing or the like, or a CPU that operates according to a program.

  Next, the operation of this embodiment will be described with reference to FIG. FIG. 2 is a flowchart showing an example of the operation of the image processing apparatus according to the present embodiment.

  In the example shown in FIG. 2, first, the camera position / orientation estimation means 11 estimates the camera position / orientation of the conversion target image (step S11). In the present embodiment, the camera position / orientation estimation unit 11 estimates the position / orientation of the camera for each input conversion target image. The camera position / orientation estimation means 11 estimates, for example, the three-dimensional coordinates (X, Y, Z) in the world coordinate system as the camera position from the captured image information, and uses the world coordinate system as the camera attitude. Estimate a 3 × 3 matrix representing the rotation with respect to.

  Next, the first reference point acceptance determination means 12 inputs a plurality of three-dimensional coordinates constituting the rotation axis candidates of the camera in the conversion target image as information on the first reference point space coordinates for at least one image ( Step S12). For example, the first reference point reception determination unit 12 displays an operation screen including an input field for inputting information on the first reference point space coordinates corresponding to the number of corresponding points necessary for the geometric conversion formula to be used, and displays the first screen to the user. Reference point space coordinate information may be input, or first reference point space coordinate information may be obtained by reading a preset initial value. Then, the inputted information on the first reference point space coordinate is output to the first reference point space coordinate projecting means 13.

  The first reference point space coordinate projecting means 13 projects the input first reference point space coordinates on the image based on the camera position and orientation when information on the first reference point space coordinates for a certain conversion target image is input. The first reference point is set (step S13).

  Next, the image display means 14 displays the first reference point and the image so as to overlap each other (step S14). For example, the image display unit 14 displays an operation screen for adjusting the position of the first reference point on the conversion target image. The image display means 14 may display an operation screen including a rotation axis (line segment) projected on the image by connecting a plurality of first reference points.

  The user confirms the position of the first reference point displayed in the process of step S14 in the conversion target image, and inputs the first reference point determination signal if satisfied. In addition, the user performs adjustment such as moving the position of the first reference point so that the length, inclination, and position of the line segment connecting the plurality of first reference points are appropriate as the rotation axis of the camera. At this time, the information on the first reference point is changed. Note that the first reference point space coordinates may be directly changed as the information of the first reference point and projected again. The first reference point determination signal is input to the first reference point reception determination unit 12 and the first reference point space coordinate projection unit 13.

  When receiving the first reference point determination signal, the first reference point reception determining unit 12 or the first reference point space coordinate projecting unit 13 determines the first reference point corresponding to the first reference point space coordinate (step S15). . The first reference point reception determination means 12 or the first reference point space coordinate projection means 13 is, for example, the first reference point displayed at that time as the first reference point that is the target of the first reference point determination signal. (More specifically, a line segment connecting the first reference points) may be selected. In the present embodiment, when the first reference point is determined from the first reference point space coordinates for one conversion target image, three-dimensional coordinates representing the first reference point space coordinates are determined. Based on the dimensional coordinates and the camera position and orientation of each image, the first reference point can be calculated for other conversion target images.

  Next, the second reference point setting means 15 inputs the second reference point conversion source coordinates (step S16). The second reference point setting means 15 may input the second reference point conversion source coordinates by reading information on the preset second reference point conversion source coordinates from the storage means or the like. Further, the second reference point setting means 15 is the same as the method used by the first reference point reception determination means 12, that is, by receiving information on the second reference point conversion source coordinates from the user, Point conversion source coordinates may be input.

  When receiving the second reference point conversion source coordinates, the second reference point setting means 15 sets the second reference point based on the camera position and orientation (step S17).

  Next, the image display unit 14 displays the first reference point, the second reference point, and the conversion target image in an overlapping manner (step S18). The image display unit 14 may display an operation screen for adjusting the position or the like of the second reference point on the conversion target image, as in the process of step S14. Further, the image display means 14 may display both or only one of the first reference point and the second reference point.

  When the user determines that the displayed second reference point is appropriate, the user inputs a second reference point conversion source coordinate determination signal (step S19). In the present embodiment, when the second reference point conversion source coordinates are determined for one conversion target image, the image coordinates from which the second reference points are set are determined, and the image coordinates and each image are determined. The second reference point can be calculated for other conversion target images based on the camera position and orientation. In addition, the first reference point determination signal in step S15 can be received in the process in step S19.

  When the first reference points and the second reference points corresponding to the number of corresponding points required for each conversion target image are determined, the geometric conversion method determining means 16 determines each conversion target from the first reference point and the second reference point. A geometric transformation method to be applied to the image is determined (step S20). The geometric transformation method determining means 16 determines the transformation formula by determining the parameters of the transformation formula of the geometric transformation to be applied to each transformation target image based on, for example, the number of reference points and the positions (image coordinates) in those images. Generate.

  Finally, the geometric conversion means 17 performs geometric conversion on each conversion target image using the conversion method (conversion formula) determined in step S20, and outputs the resulting image as a converted image (step). S21).

  In this embodiment, processing is performed for all the plurality of conversion target images in each step. However, the processing illustrated in FIG. 2 can be performed as a series of processing for one conversion target image. . In such a case, the above steps (steps S11 to S21) may be repeated for the number of conversion target images.

  In addition, with regard to step S16, step S18, and step S19, if processing has already been performed on any image in the conversion target image group, the processing for other conversion target images can be omitted. This is because the second reference point in the conversion target image can be obtained in step S17 without requiring the user to input the second reference point conversion source coordinates again. That is, once the second reference point conversion source coordinates are determined for a certain conversion target image, the second reference point for each image is uniquely determined based on the camera position and orientation for each image.

  In the present embodiment, each unit is realized by, for example, hardware designed to perform specific arithmetic processing or the like, or a CPU that operates according to a program.

  Hereinafter, specific examples will be described. FIG. 3 is an explanatory diagram showing an outline of the geometric transformation performed on the transformation target image in the first embodiment. In the example illustrated in FIG. 3, the camera performs shooting from the bottom to the top. {Image 1,..., Image m,..., Image n,.

  As an example, the first reference point space coordinate is a line segment perpendicular to the ground and passing through the center of gravity of the object of interest in the three-dimensional space (the top three-dimensional coordinate is q1 and the bottom three-dimensional coordinate is q2). The points are set as the image coordinates (coordinates on the conversion target image axis) of two points constituting the first reference point space coordinates in the conversion target image.

  In addition, the positions of the cameras that photographed {image 1, ..., image m, ..., image n, ...} are represented by {t1, ..., tm, ..., tn, ...}. , The camera posture is {R1,..., Rm,..., Rn,. The camera position t is, for example, a three-dimensional coordinate in the world coordinate system. The camera posture R is, for example, a 3 × 3 matrix representing rotation with respect to the world coordinate system.

  In FIG. 3, the ◇ mark and the X mark written on the upper conversion target image (images 1, m, n) are the first reference points set for the conversion target image (the ◇ mark is the upper end, × Represents the lower end). The symbol ◇ is the first point (first reference point) of the first reference point, and the symbol X is the second point (lower reference point) of the first reference point. Also, the ◇ and x marks on the lower converted images 1, m, n indicate the second reference point that is the conversion destination of the first reference point set for the conversion target images 1, m, n. Represent.

  In FIG. 3, the first reference point is represented as “P1”, the second reference point is represented as “P2”, and the subscripts [A] and [B] are used, where A = image index and B = reference point index. It is represented. In the example shown in FIG. 3, the indexes are indicated by values starting from 1. For example, P1 [1] [1] represents image coordinates that are information of the first point (upper reference point) among the first reference points in the first conversion target image (image 1). P1 [1] [2] represents image coordinates which are information of the second point (lower end reference point) of the first reference points in the image 1.

  Further, for example, P2 [m] [1] is an image coordinate that is information on the first point (upper reference point) of the second reference point in the converted image m generated from the mth conversion target image (image m). Represents. P2 [m] [2] represents image coordinates which are information of the second point (lower end reference point) of the second reference point in the converted image m generated from the image m.

  In the present embodiment, first, the camera position / orientation estimation unit 11 estimates the camera position / orientation of each of the conversion target images by capturing the image. The camera position / orientation estimation means 11 estimates the camera position t and orientation R based on the appearance of the marker on the image registered in advance, for example, by printing on paper or the like. Further, the camera position / orientation estimation means 11 estimates the camera position t and orientation R from a plurality of combinations of corresponding points of known three-dimensional coordinates and image coordinates at which they are observed, for example. The posture R of the camera is expressed by the following expression (1) according to the expression by roll, pitch, and yaw. Moreover, according to the expression by the quaternion, it is expressed as the following expression (2).

  Φ represents rotation about the Z axis (roll), θ represents new rotation about the Y axis (pitch), and ψ represents rotation about the new X axis (yaw).

  In addition, when a sensor such as an acceleration sensor or a gyro sensor is built in or attached to the camera and it is not necessary to separately prepare sensors, data obtained from them can be used for camera position estimation. For example, the camera position t can be estimated by integrating the angular acceleration obtained from the acceleration sensor twice in the time direction. Further, the camera posture R can be estimated by integrating the angular velocity obtained from the gyro sensor in the time direction. Further, Patent Document 4 discloses a method for estimating a camera position by using an acceleration sensor or a gyro sensor together with a conversion target image.

  Next, the first reference point reception determination unit 12 receives the first reference point space coordinates as an input from the user, and outputs the first reference point space coordinates to the first reference point space coordinate projection unit 13.

  The first reference point space coordinate projection means 13 calculates a first reference point based on the camera position and orientation and the first reference point space coordinates, and outputs the first reference point to the image display means 14.

  The first reference point can be obtained by the following equation (3).

The sign on the left side of K [R ^T −R ^T t] X _i represents equal to a constant multiple. Note that the subscript ^T represents transposition of the matrix. K is an internal parameter of the camera. K may be determined in advance using, for example, a method of estimating camera internal parameters using projective transformation (for example, the method of Zhang et al.), Or the camera internal parameters may be calculated from three or more images. An estimation method (for example, the method of Pollefeys et al.) May be used at the same time as shooting. When estimating the camera internal parameters, the camera position / posture estimation means 11 outputs the camera internal parameters to the first reference point space coordinate projection means 13 in addition to the camera position / posture.

  The user inputs first reference point space coordinates (three-dimensional coordinates of q1 and q2) using a mouse or a keyboard while viewing the conversion target image displayed on the image display means 14 and the first reference point. The user, for example, uses the three-dimensional coordinate (0, 0, 0) as the initial value of the first point (upper reference point) of the first reference point space coordinates as the initial value of the second point (lower reference point). Enter three-dimensional coordinates (0, 1, 0).

  The image display unit 14 displays an image including the target object input as the conversion target image via an image display device such as a monitor or a projector included in the image display unit 14. At this time, the image display means 14 displays an operation screen including the conversion target image and an input interface such as a toolbar or a cursor.

  The user compares the first reference point displayed on the image display unit 14 with the object of interest so that the length, inclination, and position of the line segment connecting the first reference point are appropriate as the rotation axis of the camera. The first reference point space coordinates are adjusted, and if satisfied, a first reference point determination signal is input.

  The first reference point reception determination means 12 receives, for example, information on the first reference point space coordinates (three-dimensional coordinates of q1 and q2) for the currently displayed image # 1 as an input from the user. At this time, the image display means 14 displays the input first reference point on the conversion target image so that the user can confirm the position of the first reference point space coordinates in the conversion target image. . The user confirms the position and the like of the first reference point with respect to the displayed conversion target image, and inputs a first reference point determination signal when satisfied with the currently displayed first reference point. If the positional relationship is not satisfied, the user operates the input interface such as a toolbar or a cursor by using a mouse or a keyboard to switch validity / invalidity of the first reference point space coordinate or adjust the position. do it.

  When the first reference point determination signal is input, the first reference point space coordinate projection means 13 sets the image coordinates on which the first reference point space coordinates are projected as the first reference point, and outputs the image coordinates to the geometric transformation technique determination means 16. To do.

  Next, the second reference point setting unit 15 sets a second reference point that is a conversion destination of the first reference point, based on the second reference point conversion source coordinates and the camera position and orientation. In this embodiment, the user inputs the second reference point conversion source coordinates using a mouse, a keyboard, or the like, and sets the second reference point based on the second reference point conversion source coordinates and the camera position and orientation. In the present embodiment, if the user is satisfied with the calculated second reference point, the user inputs a second reference point determination signal, and the subsequent input of the second reference point conversion source coordinates is omitted.

  Hereinafter, the width of the converted image is w pixels and the height is h pixels.

  For example, when the user has photographed the target object from the front (camera position t1 in FIG. 3), the target object after geometric transformation is the center in the horizontal direction, and the center of gravity g of the target object is the center of the image (0.5w, 0). .2h) and input the second reference point conversion source coordinates so that the length is 80% in the vertical direction. For example, the image coordinates of the upper reference point of the second reference point conversion source coordinates are (0.5 w, 0.9 h), and the image coordinates of the lower reference point are (0.5 w, 0.1 h). Alternatively, the above setting may be determined in advance, and information on the second reference point conversion source coordinates may be read from a storage unit such as an external recording medium.

  For example, when the conversion target image is obtained by photographing the target object from an oblique direction (camera position tm in FIG. 3), the conversion target image is reduced so that the target object of the converted image is reduced in accordance with the rate of height increase from t1 to tm. What is necessary is just to calculate the upper end reference point and the lower end reference point of two reference points.

  A calculation example will be described with reference to FIG. First, it is assumed that the center of gravity g of the object of interest is the midpoint between q1 and q2, and its three-dimensional coordinate is g = (q1 + q2) / 2. The camera position in the spherical coordinate system with the center of gravity g as the origin is expressed as tm ′ = tm−g.

Next, the length L ₀ = | q1-q2 | of the first reference point space coordinates is expressed by Expression (4) as an apparent length L _m at tm ′.

θ is a declination of tm ′ with the center of gravity g as the center of the coordinate system as shown in FIG. Since the length of L _{0 in} the image coordinate system is l ₀ = 0.9h−0.1h = 0.8h, the length of Li in the image coordinate system is l _m = 0.8 h * cos θ.

Therefore, the second reference point in the image m taken at the camera position tm is the upper reference point P2 [m] [1] = (0.5w, 0.5h + 0.5l _m ), and the lower reference point P2 [m] [2 ] = (0.5w, 0.5h-0.5l _m )

Although the method for dynamically calculating the second reference point according to the camera position and orientation is shown here, the above calculation is performed in advance and information on the second reference point is stored in an external recording medium or the like. It may be read from the means and output. For example, the camera position and orientation are quantized at an appropriate interval, and l ₀ is calculated in advance based on the camera position and orientation based on Equation (4) and stored as a lookup table in the storage means. Then, _lo is read from the table closest to the input camera position and orientation and output. When reading from the storage means, the calculation of equation (4) for each input can be omitted.

  Further, the method of dynamically calculating the second reference point and the method of reading from the storage means may be combined. For example, when the second reference point is first output by a method of reading from the storage unit and the user requests a second reference point with higher accuracy, the method may be switched to a method of dynamically calculating. Conversely, the second reference point may be first output by a method of dynamic calculation, and when the user requests lower accuracy, the method may be switched to a method of reading from the storage means.

  Moreover, when there is no request | requirement by a user, you may switch according to the performance of a computer, or the capacity | capacitance of a memory | storage means. For example, when the image processing apparatus according to the present exemplary embodiment is activated, the capacity of the storage unit is confirmed, and when it is difficult to secure a lookup table, a method of dynamically calculating may be used.

  Further, when it is necessary to increase the capacity of the storage means for some reason during the operation of the image processing apparatus according to the present embodiment, the method may be switched to a method in which the lookup table is released and dynamically calculated.

  Next, the geometric transformation method determination unit 16 determines a geometric transformation method based on the first reference point and the second reference point, and outputs the geometric transformation method to the geometric conversion unit 17. In the present embodiment, since two points are selected as the first reference point and the second reference point, the geometric transformation method determining means 16 uses similarity transformation. The similarity transformation is given by the following equation (5).

u _i '= au _i −bv _i + c
v _i ′ = bu _i + av _i + d (5)

(U _i , v _i ) represents the upper reference point P1 [m] [1] and the lower reference point P1 [m] [2] of the first reference point in the conversion target image to be processed, and (u _i , v _i ′) represents the upper reference point P2 [m] [1] and the lower reference point P2 [m] [2] of the second reference point in the converted image generated from the conversion target image to be processed. Further, (a, b, c, d) are similarity transformation parameters. The subscript i represents the first point (upper reference point) when i = 1, and the second point (lower reference point) when i = 2.

  The geometric transformation method determining means 16 solves the linear simultaneous equations for (a, b, c, d) in the equation (5) when the number of points of the first reference point and the second reference point is 2, respectively. And (a, b, c, d) can be obtained.

  Further, when the number of points of the first reference point and the second reference point is 3 or more, the geometric transformation method determination means 16 obtains a least square solution of (a, b, c, d).

  The geometric transformation method determining means 16 may output the similarity transformation parameters to the geometric transformation means 17 or may output a similarity transformation expression.

  The geometric conversion means 17 performs geometric conversion on the conversion target image to be processed using the similarity conversion parameters (a, b, c, d) thus obtained, and outputs a converted image. In the present embodiment, the obtained similarity transformation parameter is substituted into the equation (5), and an image subjected to the similarity transformation can be obtained by executing it on all the pixels of the image. Here, since the image coordinates after conversion generally do not become integer values, the geometric conversion means 17 performs interpolation to compensate for pixel loss. Interpolation methods include a nearest neighbor method that uses the pixel value of the observation point closest to the interpolation point as the pixel value of the interpolation point, and linear interpolation of the pixel values of the four observation points around the interpolation point. Various methods such as a bilinear method, which is a method of setting the pixel value of the interpolation point, can be used. The geometric transformation means 17 outputs the image subjected to the similarity transformation as a transformed image.

  As described above, in the present embodiment, a single user operation can be performed from a group of images photographed freely in the up / down / left / right directions even without equipment such as an accelerometer built in or attached to a turntable or camera. An alignment image can be generated with the camera.

  The reason is that, for each of a plurality of images including the target object, the user designates a plurality of three-dimensional coordinates constituting the rotation axis of the camera, and a camera that captures the length of the designated three-dimensional coordinates in the image coordinate system This is because the calculation is performed according to the position and the geometric transformation is performed so that the centroids of the three-dimensional coordinates coincide in a plurality of images. In addition, by calculating the geometric transformation parameters using only the image information, no equipment other than the camera is required at the time of shooting. Further, the user only needs to determine the first reference point and the second reference point once. The reason is that once the first reference point and the second reference point are determined, the first reference point and the second reference point are uniquely determined based on the camera position and orientation for each image.

  In the above description, an example in which similarity transformation is performed is shown as a specific example. However, any geometric transformation such as affine transformation or projective transformation can be selected in addition to the similarity transformation. In that case, the first reference point reception determination unit 12 and the second reference point setting unit 15 may each receive two or more image coordinates and output them to the geometric transformation method determination unit 16. Then, it is possible to cope with this by changing the calculation method of the geometric transformation method determining means 16 according to the selected number of geometric transformation parameters and the transformation formula.

  For example, when affine transformation is selected, the required number of corresponding points (number of reference points) is at least three. The first reference point reception determination unit 12 and the second reference point setting unit 15 may each receive three or more image coordinates and output them to the geometric transformation method determination unit 16. The geometric transformation method determining means 16 obtains the transformation parameters (a, b, c, d, e, f) by the following formula (6) based on the output information of the first reference point and the second reference point. Good.

u _i '= au _i + bv _i + c
v _i ′ = du _i + ev _i + f (6)

  Further, for example, when selecting projective transformation, the required number of corresponding points is at least four. The first reference point reception determination unit 12 and the second reference point setting unit 15 may each receive four or more image coordinates and output them to the geometric transformation method determination unit 16. The geometric transformation method determining means 16 may obtain the transformation parameters (a1 to a8) by the following equation (7) based on the output information on the first reference point and the second reference point.

u _i '= (a1 * u _i + a2 * v _i + a3) / (a7 * u _i + a8 * v _i +1)
v _i ′ = (a4 * u _i + a5 * v _i + a6) / (a7 * u _i + a8 * v _i +1)
... Formula (7)

  Note that the present embodiment is not limited to an image processing apparatus realized by hardware or the like corresponding to each means, but can also be realized by an information processing system as shown in FIG.

  FIG. 6 is a block diagram when the image processing apparatus according to the present invention is mounted in an information processing system. The information processing system illustrated in FIG. 6 is a general information processing system including a processor 400, a program memory 401, and a storage medium 402.

  The storage medium 402 may be a storage area composed of separate storage media, or may be a storage area composed of the same storage medium. As the storage medium, a RAM (Random Access Memory) or a magnetic storage medium such as a hard disk can be used.

  In the program memory 401, the camera position / orientation estimation means 11, the first reference point reception determination means 12, the first reference point space coordinate projection means 13, the image display means 14, and the second reference point setting means 15 described above. In addition, a program for causing the processor 400 to perform processing of each part of the geometric transformation method determining unit 16 and the geometric transformation unit 17 is stored, and the processor 400 operates according to this program.

  The processor 400 may be a processor that operates according to a program such as a CPU, for example.

  Thus, the present invention can also be realized by a computer program. It should be noted that a means capable of operating by a program (for example, camera position / posture estimation means 11, first reference point reception determination means 12, second reference point setting means 15, geometric conversion method determination means 16, geometric conversion means 17, etc.) It is not necessary to operate all with a program, and a part may be configured with hardware. Moreover, you may implement | achieve as a separate unit, respectively.

  Next, the outline of the present invention will be described. FIG. 7 is a block diagram showing an outline of the present invention. As illustrated in FIG. 7, the image processing apparatus, for each conversion target image, is based on the camera position and orientation estimation unit 111 that estimates the camera position and orientation at the time of shooting, and the camera position and orientation estimated by the camera position and orientation estimation unit 111. Then, the first reference point space coordinates expressed by the three-dimensional coordinates in each conversion target image are projected onto each conversion target image, and the first reference point expressed by the image coordinates in each conversion target image is generated. The point projection unit 113 (in the embodiment, realized by the first reference point reception determination unit 12 and the first reference point space coordinate projection unit 13), the second reference point conversion source coordinates represented by the image coordinates, and the camera A second reference point setting unit 115 that receives a position and orientation and sets a second reference point expressed by image coordinates that is a conversion destination of the first reference point according to the camera position and orientation; A geometric transformation method determination unit that calculates parameters of a predetermined geometric transformation formula based on the first reference point determined by the reference point projection unit 113 and the second reference point set by the second reference point setting unit 115 116 and a geometric conversion unit 117 that performs geometric conversion on the conversion target image using the geometric conversion formula based on the parameters calculated by the geometric conversion method determination unit 116.

  Although a part or all of the above embodiments can be described as the following supplementary notes, the configuration of the present invention is not limited to the following configurations.

(Supplementary note 1) For each conversion target image, based on the camera position and orientation estimation means for estimating the camera position and orientation at the time of shooting, and the camera position and orientation estimated by the camera position and orientation estimation means, three-dimensional in each conversion target image A first reference point projecting means for projecting the first reference point space coordinates expressed by coordinates onto each conversion target image and generating a first reference point expressed by the image coordinates in each conversion target image; A second reference point that receives the second reference point conversion source coordinates and the camera position and orientation, and sets a second reference point expressed by image coordinates that are the conversion destination of the first reference point according to the camera position and orientation Based on the point setting means, the first reference point determined by the first reference point projection means, and the second reference point set by the second reference point setting means for each conversion target image. A geometric transformation method determining means for calculating a parameter of a predetermined geometric transformation formula; and a geometric transformation means for geometrically transforming the conversion target image using the geometric transformation formula based on the parameter calculated by the geometric transformation method decision means, Furthermore, an image processing apparatus comprising image display means for displaying at least one conversion target image.

(Supplementary note 2) The image processing apparatus according to Supplementary note 1, wherein the image display means is set by the first reference point generated by the first reference point projection means and the second reference point setting means. An image processing apparatus that displays one or both of two reference points.

(Supplementary note 3) The image processing device according to Supplementary note 1 or Supplementary note 2, wherein the geometric transformation means performs interpolation to compensate for pixel loss.

  While the present invention has been described with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  This application claims the priority on the basis of the JP Patent application 2012-2559 for which it applied on January 10, 2012, and takes in those the indications of all here.

  The present invention can be suitably applied to an application for generating a registration image from an image obtained by photographing a target object from a free direction with a single camera.

DESCRIPTION OF SYMBOLS 11 Camera position and orientation estimation means 12 First reference point reception determination means 13 First reference point space coordinate projection means 14 Image display means 15 Second reference point setting means 16 Geometric transformation method determination means 17 Geometric transformation means 111 Camera position and orientation estimation section 113 First reference point projection unit 115 Second reference point setting unit 116 Geometric transformation method determination unit 117 Geometric transformation unit 400 Processor 401 Program memory 402 Recording medium

Claims (10)

An image processing device that generates a registration image of a target object from a plurality of conversion target images including the target object,
Camera position and orientation estimation means for estimating the camera position and orientation at the time of shooting for each conversion target image;
Based on the camera position and orientation estimated by the camera position and orientation estimation means, the first reference point space coordinates expressed by the three-dimensional coordinates in each conversion target image are projected onto each conversion target image, and the image in each conversion target image is displayed. First reference point projection means for generating a first reference point represented by coordinates;
Accepts the second reference point conversion source coordinates and the camera position and orientation expressed by image coordinates, and sets the second reference point expressed by the image coordinates that are the conversion destination of the first reference point according to the camera position and orientation Second reference point setting means for
Based on the first reference point determined by the first reference point projection means and the second reference point set by the second reference point setting means for each conversion target image, a predetermined geometric conversion formula Geometric transformation method determination means for calculating the parameters of
An image processing apparatus comprising: a geometric conversion unit configured to perform geometric conversion on a conversion target image using a geometric conversion formula based on a parameter calculated by the geometric conversion method determination unit. The second reference point setting means dynamically calculates and sets the second reference point expressed by image coordinates that are conversion destinations of the first reference point according to the camera position and orientation. The image processing apparatus described. The second reference point setting means sets the second reference point by reading from the storage device the second reference point expressed by image coordinates that are the conversion destination of the first reference point according to the camera position and orientation. The image processing apparatus according to claim 2. The camera position and orientation estimation unit estimates the camera position and orientation of the conversion target image based on one or both of each conversion target image and camera position and orientation parameter information read from an external device. The image processing apparatus according to claim 1. First reference point space coordinate reception determining means for receiving at least one candidate of first reference point space coordinates expressed by three-dimensional coordinates in each conversion target image and determining one first reference point space coordinate from the candidates. The image processing apparatus according to any one of claims 1 to 4. The first reference point reception determination means determines the image coordinates obtained by projecting the first reference point space coordinates displayed on at least one conversion target image from at least one candidate of the first reference point space coordinates. The image processing apparatus according to claim 5, wherein a signal is received, and when the determination signal is received, the first reference point that is a target of the determination signal is determined as the first reference point in each conversion target image. The image according to any one of claims 1 to 6, wherein the second reference point setting unit receives the determination signal for the set second reference point and determines the second reference point. Processing equipment. The image processing apparatus according to claim 1, wherein the second reference point setting unit accepts at least one image coordinate as the second reference point conversion source coordinate. An image processing method for generating a registration image by performing geometric transformation on each of conversion target images, which are a plurality of images including a target object,
Estimate the camera position and orientation at the time of shooting for each conversion target image,
Based on the estimated camera position and orientation, the first reference point space coordinates expressed by the three-dimensional coordinates in each conversion target image are projected onto each conversion target image, and the first reference is expressed by the image coordinates in each conversion target image. Generate a reference point,
Accepts the second reference point conversion source coordinates and the camera position and orientation expressed by image coordinates, and sets the second reference point expressed by the image coordinates that are the conversion destination of the first reference point according to the camera position and orientation And
For each image to be converted, a parameter of a predetermined geometric transformation equation is calculated based on the first reference point and the second reference point,
An image processing method characterized by geometrically transforming a conversion target image using a geometric transformation formula based on a calculated parameter. An image processing program for generating a registration image by performing geometric transformation on each of conversion target images, which are a plurality of images including a target object,
On the computer,
For each conversion target image, a process for estimating the camera position and orientation at the time of shooting,
Based on the estimated camera position and orientation, the first reference point space coordinates expressed by the three-dimensional coordinates in each conversion target image are projected onto each conversion target image, and the first reference expressed by the image coordinates in each conversion target image. Processing to generate a reference point;
Accepts the second reference point conversion source coordinates and the camera position and orientation expressed by image coordinates, and sets the second reference point expressed by the image coordinates that are the conversion destination of the first reference point according to the camera position and orientation Processing to
A process for calculating a parameter of a predetermined geometric transformation equation based on the first reference point and the second reference point for each conversion target image;
An image processing program for executing a process of geometrically transforming an image to be transformed using a geometric transformation expression based on a calculated parameter.

Images