JP2015184773A - Image processor and three-dimensional object tracking method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the tracking of an operation of a target object in a three-dimensional space with high accuracy from a two-dimensional video image picked up by a monocular camera.SOLUTION: An image processor includes a boundary extraction part 11 for extracting a boundary of a target object from a picked-up two-dimensional image, a radiation shape generation part 12 for generating a radiation shape by respectively connecting each point of the boundary and a camera center, a sample point extraction part 14 for extracting a sample point to be a boundary when the target object is reflected in the two-dimensional image from three-dimensional data of the target object projected on the two-dimensional image, and a positioning part 15 for positioning the three-dimensional data so as to position the sample point on the radiation shape, acquires an accurate three-dimensional position of the target object from the positioned three-dimensional data through positioning of the three-dimensional radiation shape generated from the two-dimensional image and the three-dimensional data of the target object, and performs three-dimensional tracking of the target object by performing this acquisition in each prescribed frame interval of a two-dimensional video image.

Description

  The present invention relates to an image processing apparatus and a three-dimensional object tracking method, and is particularly suitable for use in an image processing apparatus that tracks a movement of a target object in a three-dimensional space from a two-dimensional image.

  Conventionally, many methods for tracking a target object in a single video image have been proposed in the field of image processing. In particular, tracking of a target object using a particle filter is a very well-known method. However, many of these target object tracking methods are tracking of a target object in a two-dimensional image, and it is not possible to realize tracking of an operation in a three-dimensional space.

  By the way, in the research of markerless AR (Augmented Reality), there are some studies on a system for recognizing a three-dimensional space in an image. In particular, PTAM (Parallel Tracking and Mapping) is well known as a research for recognizing a three-dimensional space using a monocular camera. In PTAM, a three-dimensional coordinate system is defined by recognizing a portion close to a plane from image feature points in a video, and a three-dimensional position is recognized from a relative positional relationship of image feature points between frames of the video. However, PTAM is designed for the purpose of space tracking, and is not sufficient in terms of recognition accuracy and tracking accuracy in a three-dimensional space for use in object tracking.

  In addition, as a technique for processing a two-dimensional image and grasping a three-dimensional position / posture of a target object, a technique described in Patent Document 1 is also proposed. An image processing apparatus described in Patent Document 1 is based on an image acquisition unit that acquires an input image generated by imaging a real space using an imaging device, and the position of one or more feature points that appear in the input image. A recognition unit for recognizing a relative position and orientation between the real space and the imaging apparatus, and an application unit for providing an augmented reality application using the recognized relative position and orientation.

  According to the technique described in Patent Document 1, it is possible to detect the three-dimensional position / posture of a target object from a two-dimensional image captured by a monocular camera. However, since the technique described in Patent Document 1 is a mechanism that identifies a characteristic part from a two-dimensional image and detects the three-dimensional position / posture of the target object based on the position of the feature point. The accuracy of detection largely depends on the number of feature points extracted and the extraction accuracy. Therefore, it is suitable for processing in a space with many feature points or a large space, but the processing accuracy for a space with few feature points or a limited narrow space deteriorates the accuracy of 3D position / posture detection. was there.

  On the other hand, there is a technique called motion capture as a method for performing three-dimensional tracking of an object. However, in the case of motion capture, a special measuring device is required. For this reason, there is a problem that it is difficult to install in a site where it is difficult to use these hardware environments.

JP 2013-225245 A

  The present invention has been made to solve such a problem. Regardless of the nature of the space to be processed, the present invention is based on the 2D image captured by the monocular camera in the 3D space of the target object. The purpose is to be able to accurately track the movement.

  In order to solve the above-described problems, in the present invention, a two-dimensional image as a still image is extracted from a two-dimensional video generated by imaging a real space using an imaging device at predetermined frame intervals, and the extraction is performed. By sequentially obtaining the spatial information of the three-dimensional position and orientation of the target object from the two-dimensional image of each frame, the movement of the target object in the three-dimensional space is tracked. Specifically, the boundary of the target object is extracted from the two-dimensional image of the current frame, and the radiation shape is defined in the three-dimensional space by a plurality of straight lines formed by connecting each point of the extracted boundary and the center position of the imaging device. Generate on the coordinate system. On the other hand, it becomes a boundary when the target object appears in the two-dimensional image from the three-dimensional data of the target object projected onto the two-dimensional image using the three-dimensional position of the target object obtained from the two-dimensional image of the previous frame. A point is extracted as a sample point of 3D data. Then, the three-dimensional data is aligned so that the sample point is positioned on the radial shape, and the spatial information of the three-dimensional position and orientation of the target object is acquired from the aligned three-dimensional data.

  According to the present invention configured as described above, a two-dimensional image is obtained by aligning a three-dimensional radial shape generated from a two-dimensional image of a target object imaged by an imaging device with the three-dimensional data of the target object. The two-dimensional space by and the three-dimensional space by three-dimensional data can be linked. Since the three-dimensional data has the three-dimensional position / posture of the target object, the accurate three-dimensional position / posture of the target object can be acquired from the aligned three-dimensional data. By performing such processing for each of the two-dimensional images acquired at predetermined frame intervals from the two-dimensional video imaged by the imaging device, the target object can be tracked. Thereby, it is possible to accurately track the movement of the target object in the three-dimensional space from the two-dimensional image captured by the monocular imaging device, regardless of the property of the space to be processed.

It is a block diagram which shows the function structural example of the image processing apparatus by 1st Embodiment. It is a figure for demonstrating the principle of the pinhole camera model used by this embodiment. It is a figure which shows the example of the radial shape produced | generated by the radial shape production | generation part of this embodiment. It is a figure which shows the relationship between the coordinate system of a two-dimensional image plane, and the coordinate system of a three-dimensional space. It is a figure which shows the result of having performed the position correction of three-dimensional data by the position correction part of this embodiment. 3 is a flowchart illustrating an operation example of the image processing apparatus according to the first embodiment. It is a block diagram which shows the function structural example of the image processing apparatus by 2nd Embodiment. It is a figure for demonstrating the labeling of a foreground and a background using a mixed normal distribution. It is a figure which shows the comparison with Cauchy distribution and normal distribution. 10 is a flowchart illustrating an operation example of the image processing apparatus according to the second embodiment.

(First embodiment)
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, a first embodiment of the invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a functional configuration example of an image processing apparatus 100 according to the first embodiment. As shown in FIG. 1, the image processing apparatus 100 according to the first embodiment includes a 2D video acquisition unit 10, a 2D image acquisition unit 20, and a tracking unit 30 as its functional configuration. The tracking unit 30 includes a boundary extraction unit 11, a radial shape generation unit 12, a three-dimensional data projection unit 13, a sample point extraction unit 14, a registration unit 15, a spatial information acquisition unit 16, and a specific functional configuration. A camera parameter storage unit 17 is provided.

  Each of the functional blocks can be configured by any of hardware, DSP (Digital Signal Processor), and software. For example, when configured by software, each functional block is actually configured by including a CPU, RAM, ROM, etc. of a computer, and a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory operates. Is realized.

  The main problem to be solved in the present embodiment is “determining the three-dimensional position / posture of the target object in the two-dimensional image”. Assuming that a 2D image captured by the monocular camera 200 is a continuous display of 2D images as still images, if the 3D position and orientation of the target object in each 2D image are determined, 3D tracking can be performed by continuously acquiring the position and orientation.

  An image on a two-dimensional image of a target object with a fixed three-dimensional position / posture can be easily obtained by using the correspondence between the image plane and the three-dimensional space. However, the problem to be solved in the present embodiment is the inverse problem, and the solution is not simple. In order to deal with this problem, in this embodiment, a pinhole camera model as shown in FIG. 2 is used to connect a two-dimensional image and a three-dimensional space.

  As shown in FIG. 2, the pinhole camera model has a camera center 21 and an image plane 22. A certain point 25 on the target object 23 is projected on the intersection 24 between the straight line 26 connecting the point 25 and the camera center 21 and the image plane 22. In other words, when the point 25 on the target object 23 becomes the point 24 on the boundary line of the image projected on the image plane 22, the target object 23 takes the camera center 21 and the point 24 at the point 25. It can be said that it touches the connecting straight line 26. That is, the point 25 that becomes the boundary of the image of the target object 23 projected on the two-dimensional image exists on a straight line 26 that connects the camera center 21 and the point 24 on the boundary of the target object of the two-dimensional image.

  Therefore, in the present embodiment, the “boundary of the target object projected on the two-dimensional image” is matched with the “boundary of the target object projected on the two-dimensional image plane”, and the three-dimensional data in that state is used. A method of acquiring the three-dimensional position / posture of the target object was adopted.

  Note that the distance between the camera center 21 and the image plane 22 is referred to as a focal length. The number of pixels per unit distance is called resolution. In the present embodiment, it is assumed that the monocular camera 200 is fixed and the target object moves. Further, calibration is performed in advance, and the focal length and resolution of the monocular camera 200 are calculated. These focal lengths and resolutions are stored in advance in the camera parameter storage unit 17 as camera parameters. The auto zoom function of the monocular camera 200 is disabled.

  Each functional block shown in FIG. 1 has a configuration for performing the processing as described above. The 2D video acquisition unit 10 acquires a 2D video generated by imaging a real space using the monocular camera 200. The 2D image acquisition unit 20 acquires a 2D image as a still image from the 2D video acquired by the 2D video acquisition unit 10 every n frame intervals (n is an arbitrary number equal to or greater than 1).

  In the example of FIG. 1, a monocular camera 200 is connected to the image processing apparatus 100 such as a personal computer, and the 2D video acquisition unit 10 acquires the 2D video captured by the monocular camera 200 in real time. Although shown, the present invention is not limited to this. For example, a 2D image captured by the monocular camera 200 may be stored in a memory, and the 2D image stored in the memory may be captured later by the 2D image acquisition unit 10.

  The tracking unit 30 acquires the spatial information of the three-dimensional position and orientation of the target object from the two-dimensional image of each frame acquired by the two-dimensional image acquisition unit 20, thereby performing the operation of the target object in the three-dimensional space. To track. Specifically, the three-dimensional position and orientation of the target object are acquired by the function blocks 11 to 17 described below.

  The boundary extraction unit 11 extracts the boundary of the target object from the 2D image acquired in the current frame by the 2D image acquisition unit 20. The boundary of the target object in the two-dimensional image is a line that hits the boundary between the image of the target object and the background image, and the boundary line of the configuration surface of the target object is not always the boundary of the target object in the two-dimensional image. Absent. For example, after extracting the target object from the two-dimensional image by the foreground extraction process, the boundary extraction unit 11 extracts the boundary from the extracted target object. The extraction of the boundary can be performed by, for example, a so-called edge detection process (a process of specifying a location where the brightness or color of the image changes sharply (discontinuously)).

  The radial shape generation unit 12 generates a radial shape by a plurality of straight lines formed by connecting the center position of the monocular camera 200 and each point of the boundary extracted by the boundary extraction unit 11. FIG. 3 is a diagram illustrating an example of a radiation shape generated by the radiation shape generation unit 12. In FIG. 3, for convenience of illustration, only four straight lines 36 constituting the radial shape are shown. As shown in FIG. 3, the radial shape generation unit 12 has a camera center 31 obtained from the camera parameters stored in the camera parameter storage unit 17 and each point on the boundary of the target object in the two-dimensional image captured on the image plane 32. A radial shape is generated on a coordinate system in a three-dimensional space by a plurality of straight lines 36 each formed by connecting with the line 34.

  In this embodiment, the relationship between the coordinate system of the two-dimensional image plane and the coordinate system of the three-dimensional space is defined as shown in FIG. That is, in the coordinate system on the two-dimensional image plane 32, the width direction is the w axis and the height direction is the h axis. On the other hand, in the coordinate system of the three-dimensional space, the camera center 31 is the origin, the direction perpendicular to the image plane 32 from the camera center 31 is the z axis, and the direction orthogonal to the z axis and parallel to the w axis of the image plane 32 is x. A direction perpendicular to the axis and the z-axis and parallel to the h-axis of the image plane 32 is defined as a y-axis. The intersection of the z axis and the image plane 32 is the image center.

  The three-dimensional data projection unit 13 projects three-dimensional data representing the target object on the two-dimensional image based on the pinhole camera model. The three-dimensional data projected here is, for example, three-dimensional mesh data in which the same shape as the target object is expressed by a plurality of meshes made of triangles or quadrangles. When processing the first frame immediately after the start of tracking, three-dimensional data is projected to a predetermined initial position. Although the initial position is arbitrary, it is preferable to set a position where the target object reflected in the two-dimensional image is projected in the vicinity of a certain position as the initial position.

  That is, the position of the target object shown on the two-dimensional image is known from the boundary extracted by the boundary extraction unit 11. The position of the monocular camera 200 is also known from the camera parameters stored in the camera parameter storage unit 17. Therefore, from this information, the approximate position of the target object in the three-dimensional space can be estimated by the pinhole camera model.

  On the other hand, at the time of processing for the second and subsequent frames, the 3D data projecting unit 13 represents the target object using the 3D position of the target object obtained from the 2D image of the previous frame by the spatial information acquisition unit 16. Project dimensional data.

  The sample point extraction unit 14 determines a plurality of points that are boundaries when the target object appears in the two-dimensional image from the three-dimensional data of the target object projected onto the two-dimensional image by the three-dimensional data projection unit 13. Extract as data sample points. In FIG. 3, reference numeral 33 denotes a target object represented by three-dimensional data projected at a position before alignment, and reference numeral 35 denotes a plurality of sample points extracted from the target object 33 based on the three-dimensional data. In the example of FIG. 3, four sample points 35 corresponding to the four boundary points 34 on the two-dimensional image (the positions before the alignment of the three-dimensional data do not correspond exactly to the boundary points 34). Show.

  For example, when the three-dimensional data to be projected is mesh data, the target object is represented by a plurality of meshes having a triangular or quadrangular shape formed by connecting three or four nodes. When the normal is extended from the surface of the mesh, the angle of the normal viewed from the center of the camera is approximately 90 degrees at the location that becomes the boundary of the target object. Therefore, the sample point extraction unit 14 extends normals from the surfaces of a plurality of meshes having a certain node in common, and confirms the angle of the normal viewed from the camera center. When a mesh having a normal angle smaller than 90 degrees and a mesh larger than 90 degrees coexist, one certain node is extracted as a sample point. The sample point extraction unit 14 performs this process on a plurality of nodes, and thereby extracts a plurality of points that become boundaries when the target object appears in the two-dimensional image as sample points of the three-dimensional data.

  For convenience of explanation, FIG. 3 shows a state in which the three-dimensional data of the target object 33 is actually projected on the two-dimensional image. However, the three-dimensional data is actually projected on the two-dimensional image. There is no need to display. That is, it is possible to virtually project the three-dimensional data of the target object 33 and extract a plurality of sample points by calculation.

  The alignment unit 15 performs alignment of the three-dimensional data so that the sample points extracted by the sample point extraction unit 14 are positioned on the radial shape generated by the radial shape generation unit 12. For example, the alignment unit 15 uses a so-called ICP (Iterative Closest Point) algorithm to obtain a correspondence between the straight line 36 of the radial shape and the sample point 35 from the nearest point, and repeats a conversion process that minimizes the obtained correspondence. Thus, alignment of the three-dimensional data is performed.

  FIG. 5 is a diagram illustrating a result of the alignment of the three-dimensional data performed by the alignment unit 15. As shown in FIG. 5, when the alignment of the three-dimensional data is performed, the plurality of sample points 35 extracted from the three-dimensional data are the plurality of straight lines 36 constituting the radial shape generated by the radial shape generating unit 12. It will be located above.

  The spatial information acquisition unit 16 acquires the spatial information of the three-dimensional position and orientation of the target object from the three-dimensional data aligned by the alignment unit 15 in this way. Since the three-dimensional data is three-dimensional mesh data, it originally has spatial information about the three-dimensional position / posture of the target object. Therefore, the spatial information acquisition unit 16 may acquire the spatial information of the three-dimensional position / posture possessed by the aligned three-dimensional mesh data.

  FIG. 6 is a flowchart showing an operation example of the image processing apparatus 100 according to the first embodiment configured as described above. The flowchart shown in FIG. 6 starts when the user performs an operation to turn on the image processing apparatus 100 and instruct to perform tracking of the target object.

  First, the 2D video acquisition unit 10 acquires from the monocular camera 200 a 2D video generated by imaging a real space including a target object whose 3D position / posture is to be grasped (step S1). The 2D image acquisition unit 20 acquires a 2D image for one frame from the 2D video acquired by the 2D video acquisition unit 10 (step S2).

  Next, the boundary extraction unit 11 extracts a boundary with the background of the target object from the two-dimensional image acquired by the two-dimensional image acquisition unit 20 (step S3). Furthermore, the radial shape generation unit 12 generates a radial shape on a coordinate system in a three-dimensional space by using a plurality of straight lines formed by connecting the center position of the monocular camera 200 and each point of the boundary extracted by the boundary extraction unit 11. (Step S4).

  On the other hand, the three-dimensional data projection unit 13 projects three-dimensional mesh data representing the target object based on the pinhole camera model (step S5). Then, the sample point extraction unit 14 determines, from the three-dimensional data of the target object projected on the two-dimensional image by the three-dimensional data projection unit 13, a point that becomes a boundary when the target object appears in the two-dimensional image. Data is extracted as sample points (step S6).

  Note that the processes in steps S2 to S6 are not necessarily performed in the order described above. For example, the processes of steps S5 to S6 may be performed first, and then the processes of steps S2 to S4 may be performed. Or you may make it perform the process of step S2-S4, and the process of step S5-S6 simultaneously. However, if the process of step S3 is performed prior to the process of step S5, the position of the target object shown on the two-dimensional image can be known. It is possible to estimate the vicinity as an initial position for projecting the three-dimensional data.

  Next, the alignment unit 15 uses, for example, an ICP algorithm so that the sample points extracted by the sample point extraction unit 14 are positioned on the radial shape generated by the radial shape generation unit 12. Are aligned (step S7). Thereafter, the spatial information acquisition unit 16 acquires the spatial information of the three-dimensional position and orientation of the target object from the three-dimensional data aligned by the alignment unit 15 (step S8).

  Next, the image processing apparatus 100 determines whether or not tracking of the target object is to be ended (step S9). For example, when the user performs an operation of turning off the power of the image processing apparatus 100 or an operation for stopping the tracking process, the image processing apparatus 100 determines that the tracking is finished, and the process of the flowchart illustrated in FIG. finish. On the other hand, when the tracking is not finished, the process returns to step S1, and the process for the next frame is continued.

  As described above in detail, according to the first embodiment, the alignment between the three-dimensional radial shape generated from the two-dimensional image of the target object imaged by the monocular camera 200 and the three-dimensional data of the target object. Through this, it is possible to connect the two-dimensional space based on the two-dimensional image and the three-dimensional space based on the three-dimensional data. Since the three-dimensional data has the three-dimensional position / posture of the target object, the accurate three-dimensional position / posture of the target object can be acquired from the aligned three-dimensional data. By performing such processing for each of the two-dimensional images acquired at predetermined frame intervals from the two-dimensional video imaged by the monocular camera 200, the target object can be tracked. Accordingly, the operation of the target object in the three-dimensional space from the two-dimensional image captured by the monocular camera 200 is performed regardless of the nature of whether the space to be processed is a space having many feature points or a large space. Can be accurately tracked.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 7 is a block diagram illustrating a functional configuration example of the image processing apparatus 100 ′ according to the second embodiment. In FIG. 7, those given the same reference numerals as those shown in FIG. 1 have the same functions, and therefore redundant description is omitted here.

  As shown in FIG. 7, the tracking unit 30 ′ according to the second embodiment further includes a position correction unit 18 as a functional configuration. Further, instead of the boundary extraction unit 11 and the three-dimensional data projection unit 13, a boundary extraction unit 11 'and a three-dimensional data projection unit 13' are provided. The boundary extraction unit 11 ′ includes a foreground extraction processing unit 11 a and a boundary extraction processing unit 11 b as specific functional configurations.

  The foreground extraction processing unit 11a extracts a target object from the two-dimensional image acquired by the two-dimensional image acquisition unit 20 by performing foreground extraction processing using learning data generated from the teacher data. The boundary extraction processing unit 11b extracts a boundary from the target object extracted by the foreground extraction processing unit 11a. Similarly, in the first embodiment described above, after extracting the target object from the two-dimensional image by the foreground extraction process, the boundary is extracted from the extracted target object. In the second embodiment, foreground extraction is performed. This is performed by processing using machine learning with teacher data.

  In order to improve the tracking of the target object, it is desirable to perform the foreground extraction process and the boundary extraction process in real time (at high speed). As a method for performing foreground extraction, foreground extraction processing using a graph cut is well known, for example. However, this method has a high extraction accuracy, but requires a solution to a problem called a minimum cut problem, and therefore has a high processing cost. Therefore, it is not suitable for real-time processing required for tracking. Therefore, in this embodiment, although the extraction accuracy is somewhat inferior, foreground extraction processing is performed using learning data generated from teacher data as foreground extraction suitable for high-speed real-time processing. Then, in the alignment process, which is a post-process, a process that allows noise generated by foreground extraction is performed.

  First, generation of learning data will be described. In the present embodiment, the image of the target object and the background image are used as teacher data. The foreground extraction processing unit 11a creates learning data by modeling the data extracted from these teacher data with a mixed normal distribution. A mixed normal distribution is a probability distribution created by multiplying a plurality of normal distributions by a weight that gives a sum of 1 and taking the sum, and has a plurality of peaks. The mixed normal distribution is a probability distribution often used when modeling data having a plurality of frequency peaks in the data.

  Here, a method for creating a model will be described. The foreground extraction processing unit 11a expresses the color of each pixel of the teacher data with a three-dimensional coordinate value for model creation (here, this coordinate value is referred to as a color coordinate value). RGB values and HSV values are well known as color representation methods. In the present embodiment, it is assumed that an HSV value that is an expression close to grasping a human sensory color is used. The HSV value is expressed as one point in a conical coordinate system. In addition, although the example which uses an HSV value as a color model is demonstrated here, another model (for example, CIE-Lab) etc. may be sufficient.

  The foreground extraction processing unit 11a converts the RGB values of each pixel constituting the two-dimensional image into HSV values, and further converts them into corresponding coordinate values when a cone representing the HSV coordinate system is embedded in the three-dimensional space. Thus, the RGB value of each pixel is converted into a three-dimensional color coordinate value. Here, the foreground extraction processing unit 11a converts the RGB values of the foreground and background pixels into three-dimensional color coordinate values, and calculates the probability distribution of these color coordinate values by maximum likelihood estimation of a mixed normal distribution. Maximum likelihood estimation is a technique for calculating a probability distribution that gives the highest probability of occurrence of given data. The calculated mixed normal distribution is used as learning data.

  Each normal distribution of this mixed normal distribution can be labeled with either foreground or background. That is, when many foreground color coordinate values are gathered near the center of the normal distribution, the normal distribution is labeled as the foreground, and when many background color coordinate values are gathered, the normal distribution is labeled as the background. In the example of FIG. 8, foreground data is collected near the left normal distribution, and background data is collected near the right normal distribution. Therefore, the normal distribution on the left is labeled as a normal distribution representing the foreground, and the normal distribution on the right is labeled as a normal distribution representing the background.

  Next, a foreground extraction method using the learning data generated as described above will be described. The foreground extraction processing unit 11a determines the foreground or background for each pixel of the two-dimensional image acquired by the two-dimensional image acquisition unit 20, and extracts the foreground extracted by extracting the pixel determined to be the foreground.

  Specifically, the foreground extraction processing unit 11a calculates the Mahalanobis distance between the color coordinate value converted from the RGB value of each pixel of the two-dimensional image and the center of each normal distribution of the learning data. The Mahalanobis distance is a distance considering distribution distribution. The foreground extraction processing unit 11a determines whether each pixel is the foreground or the background by setting the label (foreground or background) of the normal distribution that minimizes the Mahalanobis distance as the label of the pixel.

  Since this process only calculates the Mahalanobis distance for each pixel, the calculation is very fast. Moreover, since the adjacent relationship between pixels is not considered, all pixels can be calculated independently. For this reason, this method is very compatible with high speed by parallel processing using a plurality of CPUs and GPGPU (General-purpose computing on graphics processing units).

  The foreground extraction processing unit 11a extracts the pixels determined to be the foreground after determining the foreground or the background for each pixel of the two-dimensional image acquired by the two-dimensional image acquisition unit 20 as described above. The target object is extracted with. According to this method, a place different from the target object is also extracted as the foreground, and noise other than the target object may remain. Therefore, as will be described below, alignment processing is performed in consideration of this noise.

  The position correction unit 18 corrects the position of the three-dimensional data projected on the two-dimensional image by the three-dimensional data projection unit 13 '. Specifically, the center positions of the target object extracted from the two-dimensional image of the previous frame by the boundary extraction unit 11 ′ and the target object extracted from the two-dimensional image of the current frame are respectively calculated, Based on the difference amount, the position where the three-dimensional data is projected is corrected. That is, at the time of processing for the second and subsequent frames, the 3D data projecting unit 13 ′ uses the 3D position of the target object obtained from the 2D image of the previous frame to project the 3D data on the 2D image. The position is determined. The position correction unit 18 converts the difference amount described above into a difference amount in the three-dimensional space and adds it to the position of the three-dimensional data of the previous frame, thereby correcting the position where the three-dimensional data is projected.

  Here, a method for calculating the center position of the target object on the two-dimensional image becomes an issue. As described above, the target object extraction processing result in the boundary extraction unit 11 ′ includes noise. Therefore, if the pixel values of the target object extracted by the boundary extraction unit 11 'are simply averaged, the center position is shifted due to the influence of noise. Therefore, it is necessary to estimate the center position that is robust against noise. Accordingly, the position correction unit 18 calculates the center position of the target object by applying maximum likelihood estimation of the Cauchy distribution to the pixel value of the target object extracted by the boundary extraction unit 11 ′.

  FIG. 9 is a diagram comparing the Cauchy distribution and the normal distribution. It is known that the Cauchy distribution has a higher probability of the tail portion than the normal distribution and has a robust property against noise. Therefore, by estimating the center of the target object pixel by maximum likelihood estimation using the Cauchy distribution, the influence of noise generated by the boundary extraction unit 11 ′ is minimized, and the center position of the target object is determined. It can be captured more accurately.

  Then, the three-dimensional data projection unit 13 ′ projects the three-dimensional data at the position on the two-dimensional image corrected by the position correction unit 18. The position corrected by the position correction unit 18 is a position where the target object actually exists in the current frame as compared to the position set using the three-dimensional position of the target object obtained from the two-dimensional image of the previous frame. It is close to. Thereby, the alignment of the three-dimensional data performed by the alignment unit 15 based on the ICP algorithm can be performed with higher accuracy.

  FIG. 10 is a flowchart showing an operation example of the image processing apparatus 100 according to the second embodiment configured as described above. The flowchart shown in FIG. 10 starts when the user performs an operation to turn on the image processing apparatus 100 ′ and instruct to perform tracking of the target object.

  First, the 2D video acquisition unit 10 acquires from the monocular camera 200 a 2D video generated by imaging a real space including a target object whose 3D position / orientation is to be grasped (step S11). The 2D image acquisition unit 20 acquires a 2D image for one frame from the 2D video acquired by the 2D video acquisition unit 10 (step S12).

  Next, the boundary extraction unit 11 ′ extracts the target object from the two-dimensional image acquired by the two-dimensional image acquisition unit 20 by the foreground extraction process, and then extracts the boundary between the extracted target object and the background (step S13). ). Further, the radial shape generation unit 12 places the radial shape on a coordinate system in a three-dimensional space by a plurality of straight lines formed by connecting the center position of the monocular camera 200 and each point of the boundary extracted by the boundary extraction unit 11 ′. Generate (step S14).

  On the other hand, the three-dimensional data projection unit 13 'projects the three-dimensional data on the two-dimensional image based on the pinhole camera model (step S15). Here, the position at which the three-dimensional data is projected is an arbitrary position when the first frame is processed, and is the three-dimensional position of the target object obtained from the two-dimensional image of the previous frame during the second and subsequent frames. is there. The position correction unit 18 corrects the position set by the three-dimensional data projection unit 13 'in this way (step S16).

  The three-dimensional data projection unit 13 'projects the three-dimensional mesh data representing the target object at the position corrected by the position correction unit 18 (step S17). Then, the sample point extraction unit 14 determines, from the three-dimensional data of the target object projected on the two-dimensional image by the three-dimensional data projection unit 13, a point that becomes a boundary when the target object appears in the two-dimensional image. Data is extracted as sample points (step S18).

  Next, the alignment unit 15 uses, for example, an ICP algorithm so that the sample points extracted by the sample point extraction unit 14 are positioned on the radial shape generated by the radial shape generation unit 12. Are aligned (step S19). Thereafter, the spatial information acquisition unit 16 acquires the spatial information of the three-dimensional position and orientation of the target object from the three-dimensional data aligned by the alignment unit 15 (step S20).

  Next, the image processing apparatus 100 ′ determines whether or not to finish tracking of the target object (Step S <b> 21). Here, when it is determined that the tracking is to be ended, the processing of the flowchart illustrated in FIG. 10 is ended. On the other hand, when the tracking is not finished, the process returns to step S11 and the process for the next frame is continued.

  As described above in detail, according to the second embodiment, the object is extracted by performing the process of extracting the target object from the two-dimensional image and the process of aligning the three-dimensional data projected on the two-dimensional image at high speed. It is possible to track the movement of the target object in the three-dimensional space with high accuracy following the object and minimizing the influence of remaining noise by speeding up the extraction process of the target object.

  In the first and second embodiments described above, an example in which mesh data is used as an example of three-dimensional data has been described. However, if the data has the three-dimensional position and orientation of the target object, data other than mesh data is used. It may be used. For example, CAD data may be used. The CAD data is composed of a plurality of curved surface expressions in a three-dimensional space and expressions representing boundary lines. With regard to each curved surface expression and boundary line, considering projection on the image plane toward the camera center, CAD data is expressed as a curved surface expression and boundary line expression in a two-dimensional space on the image plane. At this time, it is possible to acquire the boundary line when the target object appears in the two-dimensional image by acquiring all the curved surfaces constituting the target object and the outermost peripheral line of the shape projected from the boundary line. The sample point extraction unit 14 extracts a plurality of sample points from the boundary line. However, it is preferable that the CAD data is projected after being converted into mesh data in that the processing becomes faster. If other 3D data can be converted into mesh data, this processing can be applied.

  In the first and second embodiments described above, an example in which the target object is extracted by the foreground extraction process and then the boundary is extracted from the extracted target object as the process of extracting the target object boundary from the two-dimensional image. However, the present invention is not limited to this. That is, the boundary of the target object may be extracted by other known methods.

  In the first and second embodiments, the example in which the radial shape is generated by a plurality of straight lines formed by connecting the camera center and each point of the boundary of the target object extracted from the two-dimensional image has been described. The radial shape does not necessarily have to pass through all points on the boundary. That is, some representative points may be extracted from the boundary of the target object, and the radial shape may be generated by a plurality of straight lines formed by connecting the extracted representative points and the camera center.

  In the first and second embodiments, the example in which the position correction of the three-dimensional data is performed using the ICP algorithm has been described. However, the present invention is not limited to this. For example, other algorithms used for alignment of a three-dimensional point group such as Softassign algorithm and EM-ICP algorithm may be used.

  In addition, each of the first and second embodiments described above is merely an example of a specific example for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. It will not be. That is, the present invention can be implemented in various forms without departing from the gist or the main features thereof.

DESCRIPTION OF SYMBOLS 10 2D image acquisition part 20 2D image acquisition part 11, 11 'Boundary extraction part 12 Radial shape generation part 13, 13' Three-dimensional data projection part 14 Sample point extraction part 15 Position alignment part 16 Spatial information acquisition part 17 Camera parameter Storage unit 18 Position correction unit 100, 100 ′ Image processing device 200 Monocular camera

Claims (7)

A two-dimensional video acquisition unit that acquires a two-dimensional video generated by imaging a real space using an imaging device;
A two-dimensional image acquisition unit that acquires a two-dimensional image as a still image at predetermined frame intervals from the two-dimensional image acquired by the two-dimensional image acquisition unit;
A tracking unit that tracks the movement of the target object in the three-dimensional space by acquiring spatial information of the three-dimensional position and orientation of the target object from the two-dimensional image of each frame acquired by the two-dimensional image acquisition unit; With
The tracking part
A boundary extraction unit that extracts a boundary of the target object from the two-dimensional image acquired in the current frame by the two-dimensional image acquisition unit;
A radial shape generation unit that generates a radial shape on a coordinate system of a three-dimensional space by a plurality of straight lines formed by connecting the center position of the imaging device and each point of the boundary extracted by the boundary extraction unit;
Three-dimensional data for projecting the three-dimensional data representing the target object onto the two-dimensional image using the three-dimensional position of the target object obtained from the two-dimensional image of the previous frame acquired by the two-dimensional image acquisition unit. A projection unit;
A sample point extraction unit for extracting a point that becomes a boundary when the target object appears in the two-dimensional image from the three-dimensional data of the target object projected on the two-dimensional image; ,
An alignment unit that aligns the three-dimensional data so that the sample points extracted by the sample point extraction unit are positioned on the radial shape generated by the radial shape generation unit;
An image processing apparatus comprising: a spatial information acquisition unit that acquires spatial information of the three-dimensional position and orientation of the target object from the three-dimensional data aligned by the alignment unit. The center position of the target object extracted from the two-dimensional image of the previous frame and the center position of the target object extracted from the two-dimensional image of the current frame are respectively calculated by the boundary extraction unit, and based on the difference between the two center positions. A position correction unit for correcting the position for projecting the three-dimensional data;
The image processing apparatus according to claim 1, wherein the three-dimensional data projection unit projects three-dimensional data representing the target object at a position on the two-dimensional image corrected by the position correction unit. . The boundary extraction unit performs a foreground extraction process using learning data generated from teacher data, thereby extracting the target object from the two-dimensional image;
The image processing apparatus according to claim 2, further comprising a boundary extraction processing unit that extracts the boundary from the target object extracted by the foreground extraction processing unit.   The image processing apparatus according to claim 3, wherein the position correction unit calculates a center position of the target object by applying maximum likelihood estimation of a Cauchy distribution to pixel values of the two-dimensional image. .   The alignment unit obtains a correspondence between the radial shape generated by the radial shape generation unit and the sample point extracted by the sample point extraction unit from the nearest point, and performs a conversion process that minimizes the obtained correspondence. The image processing apparatus according to claim 1, wherein the three-dimensional data is aligned by repeating.   6. The three-dimensional data according to claim 1, wherein the three-dimensional data is three-dimensional mesh data in which the same shape as the target object is expressed by a plurality of meshes made of triangles or quadrilaterals. Image processing device. A first step in which a 2D video acquisition unit of the image processing apparatus acquires a 2D video generated by imaging a real space using an imaging device;
A second step in which a two-dimensional image acquisition unit of the image processing apparatus acquires a two-dimensional image as a still image from the two-dimensional video acquired by the two-dimensional video acquisition unit;
A third step in which a boundary extraction unit of the image processing apparatus extracts a boundary of the target object from the two-dimensional image acquired in the current frame by the two-dimensional image acquisition unit;
The radial shape generation unit of the image processing apparatus converts the radial shape on a coordinate system in a three-dimensional space by a plurality of straight lines formed by connecting the center position of the imaging device and each point of the boundary extracted by the boundary extraction unit. A fourth step of generating
The three-dimensional data projection unit of the image processing apparatus uses the three-dimensional position of the target object obtained from the two-dimensional image of the previous frame acquired by the two-dimensional image acquisition unit to represent the three-dimensional data representing the target object A fifth step of projecting onto the two-dimensional image;
The sample point extraction unit of the image processing apparatus determines, from the three-dimensional data of the target object projected onto the two-dimensional image, a point that becomes a boundary when the target object appears in the two-dimensional image. A sixth step of extracting as sample points of
The alignment unit of the image processing apparatus performs alignment of the three-dimensional data so that the sample point extracted by the sample point extraction unit is positioned on the radial shape generated by the radial shape generation unit. 7 steps,
An eighth step in which the spatial information acquisition unit of the image processing apparatus acquires the spatial information of the three-dimensional position and orientation of the target object from the three-dimensional data aligned by the alignment unit;
The processing from the first step to the eighth step is performed at predetermined frame intervals of the two-dimensional video, and the spatial information of the three-dimensional position and orientation of the target object is obtained from the two-dimensional image of each frame. A three-dimensional tracking method for tracking movement of the target object in a three-dimensional space.

Images