JP2010152557A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that it is difficult to track an object whose shape changes, in visual tracking. <P>SOLUTION: Captured video image data are read for each frame, and tracking start is determined based on the presence of an object to be tracked(S20, S22). When the tracking start is determined, an edge image of the image frame is generated (S24). On the other hand, when the control dot sequence of a B spline curve representing the shape of the object to be tracked is represented by the linear sum of the control dot sequence of the B spline representing a plurality of prepared reference shapes, particles are distributed in the space of the set of coefficients on each control dot sequence (S26). Furthermore, the particles are distributed even in a space of shape space vectors (S28), and likelihood observation and probability density distribution of the particles are obtained (S30). A curve obtained by weighting and averaging each parameter by the probability density distribution is generated as the result of tracking (S32). <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to information processing technology, and more particularly to an image processing apparatus that analyzes the position and shape of an object in an input image, and changes thereof, and an image processing method executed there.

  Visual tracking has a wide range of applications such as computer vision, especially visual surveillance in the security field, analysis / classification, editing of recorded video in the AV field, man-machine interface, and human-to-human interface, that is, video conferencing and videophone. Expected. Therefore, many studies have been made for the purpose of improving tracking accuracy and processing efficiency. In particular, there is a lot of research on applying particle filters, which are attracting attention as a time series analysis method for signals added with non-Gaussian noise that cannot be handled by Kalman filters, especially for the Condensation (Conditional Density Propagation) algorithm. Is famous (see, for example, Non-Patent Document 1 to Non-Patent Document 3).

In this Condensation algorithm, the object to be tracked is defined by a contour line having an arbitrary shape composed of a B-spline curve or the like. For example, in the case of a human head, tracking can be performed by defining an Ω-shaped curve with a B-spline. This is because the shape of the head basically does not change in response to human movements such as turning and bending, so the head shape can be expressed only by translating, expanding and contracting, and rotating the Ω-shaped curve. (For example, refer to Patent Document 1).
Contour tracking by stochastic propagation of conditional density, Michael Isard and Andrew Blake, Proc.European Conf. On Computer Vision, vol. 1, pp.343-356, Cambridge UK (1996) CONDENSATION-conditional density propagation for visual tracking, Michael Isard and Andrew Blake, Int. J. Computer Vision, 29, 1, 5-28 (1998) ICondensation: Unifying low-level and high-level tracking in a stochastic framework, Michael Isard and Andrew Blake, Proc 5th European Conf.Computer Vision, 1998 JP 2007-328747 A

  As described above, the Condensation algorithm is a very effective method in terms of calculation load, accuracy, and the like when tracking an object such as a human head, a ball, and an automobile that hardly changes in shape. On the other hand, when the shape of the object changes and the object cannot be expressed simply by translating, expanding, contracting, and rotating a specific shape, it is difficult to track with high accuracy. For this reason, a technique capable of recognizing changes in the shape and position of an object with a small amount of calculation has been desired.

  The present invention has been made in view of such problems, and an object thereof is to provide an image processing technique capable of recognizing a change in the shape and position of an object without increasing a calculation load. .

  One embodiment of the present invention relates to an image processing apparatus. The image processing apparatus defines a reference shape storage unit that stores a plurality of parameters that define contour lines of a plurality of reference shapes, and sets of coefficients for each parameter in a linear sum of the plurality of parameters stored in the reference shape storage unit. Thus, the apparatus includes a target shape determining unit that outputs the contour shape of the target object in the image as a linear sum.

  The image processing apparatus further includes an image acquisition unit that acquires moving image stream data including a first image frame and a second image frame obtained by imaging an object, and the object shape determination unit is defined by a set of coefficients. A shape prediction unit that generates and extinguishes particles used for the particle filter based on the estimated existence probability distribution of the target object in the first image frame in the coefficient set space, and makes a transition based on a predetermined transition model; An observation unit that observes the likelihood of each particle by matching the contour line of the object in the second image frame with the candidate contour determined by the particle, and the second image frame based on the likelihood observed by the observation unit The estimated existence probability distribution in the coefficient set space of the target object is calculated, and each part is calculated based on the estimated existence probability distribution. By performing weighted set of coefficients Ikuru, and contour acquisition unit for estimating the contour shape of the object in the second image frame may be provided.

  Here, the “first image frame” and the “second image frame” may be adjacent image frames in the image stream, or may be image frames positioned apart from each other. In general object tracking in which tracking is performed in the forward direction of the time axis, the “first image frame” is an image frame temporally prior to the “second image frame”. Is not limited to this. A “candidate contour” is a contour line of a part or the whole of an object. “Likelihood” is a degree representing how close the candidate contour is to the object. For example, when the tracking candidate is a two-dimensional figure, the degree of overlap with the object, the distance to the object, etc. Is a numerical value.

  “Particles” are introduced in the particle filter, which is one of the methods for estimating the current state from past information and current observation information. The sampling frequency of the parameter to be observed exists in the parameter space. It is expressed by the number of particles to be played.

  Another embodiment of the present invention relates to an image processing method. This image processing method uses a step of reading a plurality of parameters defining outlines of a plurality of reference shapes from a storage device, determining a set of coefficients for each parameter in a linear sum of the parameters, and using the set of determined coefficients And expressing the contour line of the object in the image as a linear sum and outputting it.

  Note that any combination of the above-described components, and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded, and the like are also effective as an aspect of the present invention. .

  According to the present invention, information related to changes in the position and shape of an object can be acquired with a small amount of calculation.

  First, in order to clarify the features and effects of the present embodiment, an outline of visual tracking using a particle filter will be described. FIG. 1 is a diagram for explaining a visual tracking method when a person is a tracking target. The person image 150 is one of image frames constituting an image stream of a moving image generated by a photographed video or computer graphics, and a person 152 to be tracked is shown.

  In order to track the movement of the person 152, an Ω-shaped curve 154 that approximates the shape of the head contour of the person 152 is described in a known expression. On the other hand, the person image 150 including the person 152 is subjected to edge extraction processing to obtain an edge image. Then, by changing a parameter that defines the curve 154, the curve 154 is translated, expanded, contracted, and rotated, and an edge in the vicinity thereof is searched for. Identify the value. The tracking of the person 152 progresses by repeating the above processing for each frame. Here, the edge generally refers to a portion having an abrupt change in image density or color.

  In order to perform matching between the curve 154 with various values of the specified parameter and the head contour of the person 152, a probability distribution prediction technique using a particle filter is introduced. In other words, the number of samplings of the curve 154 is increased or decreased according to the probability distribution of the object in the parameter space in the immediately preceding frame, and the tracking candidates are narrowed down. As a result, it is possible to focus on a portion having a high existence probability, and to perform highly accurate matching efficiently.

  For example, Non-Patent Document 3 (ICondensation: Unifying low-level and high-level tracking in a stochastic framework, Michael Isard and Andrew Blake, Proc 5th European Conf. Computer Vision, 1998). Here, the description will be focused on the points according to the present embodiment.

  First, an Ω-shaped curve 154 is described as a B-spline curve. A B-spline curve is defined by n control points (Q0,..., Qn) and n knots (s0,..., Sn). These parameters are set in advance so as to form a basic curve shape, in this case, an Ω-shaped curve. The curve obtained by the setting at this time is hereinafter referred to as template Qt. In the case of tracking the person 152 in the person image 150 shown in FIG. 1, the template Qt is an Ω shape, but the shape is changed depending on the tracking target. That is, if the tracking target is a ball, the shape is circular, and if it is a palm, the shape is a hand.

  Next, a shape space vector x is prepared as a conversion parameter for changing the state of the template. The shape space vector x is composed of the following six parameters.

Here, (shift _x , shift _y ) is a translation amount in the (x, y) direction, (extend _x , extend _y ) is a magnification, and θ is a rotation angle. Then, using the action matrix W for causing the shape space vector x to act on the template Qt, the curve after deformation, that is, the candidate curve Q can be described as follows.

  If Expression 2 is used, the template can be translated, stretched, and rotated by appropriately changing the six parameters constituting the shape space vector x, and the candidate curve Q can be changed variously depending on the combination.

  Then, for a plurality of candidate curves expressed by changing the parameters of the template Qt such as the control point and knot interval and the six parameters constituting the shape space vector x, the edges of the person 152 in the vicinity of each knot Explore. Thereafter, the likelihood density distribution in the six-dimensional space defined by the six parameters constituting the shape space vector x is estimated by obtaining the likelihood of each candidate curve from the distance to the edge or the like.

  FIG. 2 is a diagram for explaining a probability density distribution estimation method using a particle filter. In the figure, for the sake of easy understanding, among the six parameters constituting the shape space vector x, the change of a certain parameter x1 is shown on the horizontal axis, but actually the same processing is performed in the 6-dimensional space. Is called. Here, it is assumed that the image frame whose probability density distribution is to be estimated is the image frame at time t.

  First, using the probability density distribution on the parameter x1 axis estimated in the image frame at time t−1, which is the frame immediately before the image frame at time t (S10), particles at time t are generated (S12). ). Filtering is performed until then, and if the particle already exists, its splitting and disappearance are determined. The probability density distribution represented in S10 is obtained discretely corresponding to the coordinates in the parameter space, and the probability density is higher as the circle is larger.

  The particle is a materialization of the value of the parameter x1 to be sampled and the sampling density. For example, the parameter x1 region where the probability density was high at time t-1 is sampled by increasing the particle density. In the range where the probability density is low, sampling is not performed by reducing the number of particles. Thereby, for example, many candidate curves are generated near the edge of the person 152, and matching is performed efficiently.

  Next, particles are transitioned on the parameter space using a predetermined motion model (S14). The predetermined motion model is, for example, a Gaussian motion model, an autoregressive prediction motion model, or the like. The former is a model in which the probability density at time t is Gaussian distributed around each probability density at time t-1. The latter is a method that assumes a second-order or higher-order autoregressive prediction model acquired from sample data. For example, it is estimated from a change in past parameters that a person 152 is moving at a certain speed. In the example of FIG. 2, the motion of the parameter x1 in the positive direction is estimated by the autoregressive prediction type motion model, and each particle is changed in that way.

  Next, the likelihood of each candidate curve is obtained by searching for the edge of the person 152 in the vicinity of the candidate curve determined by each particle using the edge image at time t, and the probability density distribution at time t is obtained. Estimate (S16). As described above, the probability density distribution at this time is a discrete representation of the true probability density distribution 400 as shown in S16. Thereafter, by repeating this, the probability density distribution at each time is represented in the parameter space. For example, if the probability density distribution is unimodal, that is, if the tracked object is unique, the final parameter is the weighted sum of each parameter value using the obtained probability density. Thus, a curve estimated as the contour of the tracking target is obtained.

The probability density distribution p (x _t ⁱ ) at time t estimated in S16 is calculated as follows.

Here, i is a number uniquely given to a particle, p (x _t ⁱ | x _t ⁱ , u _t−1 ) is a predetermined motion model, and p (y _t | x _t ⁱ ) is a likelihood.

  The methods described so far are tracked on the premise that the initially set template shape is maintained to some extent, so if the shape change of itself is small, such as a human head, less calculation is required. This is very effective in that it can be accurately tracked by quantity. On the other hand, there is a problem that it cannot cope with the shape change of an object that cannot be expressed only by translation, expansion and contraction, and rotation. Therefore, in the present embodiment, a parameter set that defines the shape of the tracking target is expressed by a linear sum of a plurality of parameter sets prepared in advance, and a change in the shape of the tracking target is also estimated by adjusting the coefficient. This enables tracking corresponding to the shape change of the object.

Hereinafter, a case where a control point sequence defining a B-spline curve is adopted as a parameter set expressed by a linear sum will be described. First, N control point sequences Q ₀ , Q ₁ ,..., Q _N are prepared. Each control point sequence is composed of n control points as described above, and each defines a B-spline curve having a different shape. Then, the control point sequence Qsum that defines the B-spline curve representing the estimated shape of the object is a linear sum of the N control point sequences as follows.

Here, the coefficients α ₀ , α ₁ ,..., Α _N are weights for the prepared control point sequence, and a set of the coefficients α ₀ , α ₁ ,..., Α _N (hereinafter also referred to as coefficient set α). ) To express the shape of the object. Then, in addition to the shape space vector x, each particle is defined by a coefficient set α, and then the likelihood of each particle is observed, and the probability density distribution in the space of the coefficient set α is calculated in the same manner as Equation 3.

  By expressing the shape of a certain object by a linear sum of parameters that define a plurality of shapes prepared in advance, an intermediate shape of the prepared shape (hereinafter referred to as a reference shape) can be expressed. Therefore, the amount of calculation can be reduced compared to a method of preparing all image data of all shapes of the object and performing matching. Further, in the present embodiment, by using this simple expression method, the transition probability of the coefficient set α is set, thereby efficiently searching with a small amount of calculation and improving the accuracy. Basically, similarly to the shape space vector x, sampling is performed by generating and annihilating each particle in the space of the coefficient set α according to the probability density distribution in the space, and making transition according to a predetermined model. Then, by further generating, annihilating, and transitioning the particles according to the probability density distribution in the space of the shape space vector x, candidate contour lines are determined, and the respective likelihoods are observed.

3 and 4 are diagrams for explaining the value of the coefficient set α and the transition model. This figure shows an example in which a hand that makes a janken is tracked, and three types of reference B-spline curves of “Guu”, “Choki”, and “Par” are prepared. Assuming that the control point sequences defining these reference shapes are Q ₀ , Q ₁ , and Q _{2 in} Equation 4, respectively, when the shape of the tracking target is “gu”, the coefficient set α (α ₀ , α ₁ , α ₂ ) = (1.0, 0.0, 0.0). Similarly, α = (0.0, 1.0, 0.0) when “Choki”, and α = (0.0, 0.0, 1.0) when “Pa”. In this way, if the current time is in one of the reference shapes of “goo”, “chokki”, and “pah”, the other two reference shapes at the next time, namely “goo” or “chokki” or The probability P toward “pa” is 0.5.

  Here, even if the coefficient set α is slightly deviated from the above-mentioned sequence representing the reference shape, it can be considered that it is actually a reference shape. Therefore, a range of the coefficient set α that can be considered as the reference shape is set in advance. . For example, in the space defined by the coefficient set α, all the shapes defined by α within a predetermined Euclidean distance from (1.0, 0.0, 0.0) are set to be regarded as “goo”. In FIG. 3, it is assumed that the current time shape is a black circle 102 and the coefficient set α is (0.9, 0.1, 0.0). If this state is set to be “goo”, the probability P of transitioning from that state to “chokki” and “pa” is set to 0.5, respectively.

  Or, it is considered that there are a few more transitions to “Choki”, and the transition probability to “Choki” is based on the Euclidean distance of (1.0, 0.0, 0.0) and (0.9, 0.1, 0.0). Weighting is performed to increase the value. Then, after distributing the particles according to the transition probability, the Gaussian distribution 104 centering on the coefficient set α of the black circle 102 which is the current state, and a predetermined coefficient which is within the range of “Gu” and which goes to “Pa” Particles are distributed with a Gaussian distribution 106 centered on the set α.

In FIG. 4, it is assumed that the state at the current time is a black circle 108 and the coefficient set α is outside the range that can be regarded as “goo” and the range that can be regarded as “chokki” (0.4, 0.6, 0.0). In this case, it is determined that the transition is made to either “goo” or “choki”, and the particles are distributed in the Gaussian distribution 110 centering on the coefficient set α of the black circle 108 in the current state. Note that the Gaussian distributions 104, 106, and 110 in FIGS. 3 and 4 are actually distributions in a three-dimensional space defined by the coefficient set α (α ₀ , α ₁ , α ₂ ). At this time, for example, the standard deviation of the distribution in the direction of the line segment connecting the coefficient sets α representing the reference shapes (“Gu” and “Choki” in the example of FIG. 4) regarded as the arrival point of the transition may be increased. . In this way, many particles can be arranged in a shape having a high transition probability, and the sampling efficiency and tracking accuracy are improved.

  The particle distribution is not limited to that described above, and may be a Gaussian distribution having the same standard deviation in all directions, or a model other than the Gaussian distribution may be introduced. For example, the regression prediction model may be introduced by acquiring the motion of the coefficient set α in a plurality of frames up to the current time. In this case, for example, when it can be determined from the past frame that the transition from “Gu” to “Choki” is proceeding at a constant speed, many particles are further distributed in the direction of proceeding to the “Choki” shape.

  The probability P of transition from one reference shape to another reference shape is P = 0.5 if the reference shapes are three types of “gu”, “choki”, and “par” as described above. The value varies depending on the number of reference shapes. If the number of reference shapes that can be changed from the reference shape here is N, the transition probability P to each reference shape is 1 / N. Depending on the object, the transition probabilities may not be equal and may be biased, or may be determined dynamically according to the events so far.

  In Expression 4, a linear sum of control point sequences is used as a parameter of a B-spline curve representing a shape to be tracked, but a knot linear sum that is a parameter defining the same B-spline curve may be used. However, in terms of processing, since the development from the control point to the knot is only once, it is more efficient to use the control point.

  FIG. 5 shows a configuration example of the visual tracking system in the present embodiment. The visual tracking system 10 includes an imaging device 12 that images the tracking target 18, a tracking device 14 that performs tracking processing, and a display device 16 that outputs image data captured by the imaging device 12 and tracking result data. The tracked object 18 may vary depending on the intended use of the visual tracking system 10, such as a person, an object, or a portion thereof.

  The connection between the tracking device 14 and the imaging device 12 or the display device 16 may be wired or wireless, and may be via various networks. Alternatively, any two or all of the imaging device 12, the tracking device 14, and the display device 16 may be combined and integrally provided. Depending on the usage environment, the imaging device 12 and the display device 16 may not be connected to the tracking device 14 at the same time.

  The imaging device 12 acquires data of an image including the tracking target 18 or an image of a certain place regardless of the presence or absence of the tracking target 18 at a predetermined frame rate. The acquired image data is input to the tracking device 14, and the tracking processing of the tracking target 18 is performed. The processing result is output as output data to the display device 16 under the control of the tracking device 14. The tracking device 14 may also serve as a computer that performs another function, and may implement various functions by using data obtained as a result of the tracking process, that is, position information and shape information of the tracking target 18. .

  FIG. 6 shows the configuration of the tracking device 14 in the present embodiment in detail. The tracking device 14 includes an image acquisition unit 20 that acquires input image data input from the imaging device 12, an image storage unit 24 that stores data necessary for tracking processing such as the input image data, and an edge image and the like from the input image data. An image processing unit 22 to generate, a tracking target region detection unit 26 that detects a tracking target region, a tracking start / end determination unit 28 that determines the start and end of tracking, a tracking processing unit 30 that performs tracking processing using a particle filter, A result storage unit 36 that stores final tracking result data and an output control unit 40 that controls output of the tracking result to the display device 16 are included.

  In FIG. 6, each element described as a functional block for performing various processes can be configured by a CPU, a memory, and other LSIs in terms of hardware, and a program for performing image processing in terms of software. It is realized by. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The image processing unit 22 extracts the contour to be tracked. Specifically, edge extraction processing is performed for each image frame of the input image data stored in the image storage unit 24 to generate an edge image. Here, a general edge extraction algorithm such as a canny edge filter or a Sobel filter can be used. Further, the image processing unit 22 may be mounted with a foreground extractor (not shown) that uses a background difference, and the tracking is performed by extracting the foreground including the tracking target from the input image as preprocessing of the edge extraction processing. The target edge may be extracted efficiently.

  The tracking target area detection unit 26 performs image analysis for each image frame of the input image data stored in the image storage unit 22 to detect a tracking target area. For example, a foreground extractor (not shown) using a background difference is mounted, and the presence or absence of a tracking target is determined from the foreground shape extracted from the image frame, and the region is detected. At this time, if the tracking target is a human head, face detection technology may be further applied. Alternatively, an area having a color different from the background color or a specific color may be detected as a tracking target by the color detector. Alternatively, the region to be tracked may be detected by pattern matching with a preset object shape.

  In addition to the imaging device 12, the visual tracking system 10 is provided with a temperature sensor that measures the heat distribution of the space to be imaged and a piezoelectric sensor that acquires the contact area of the tracking object two-dimensionally. An area to be tracked may be detected. Existing technology can be applied to the detection of an object by a temperature sensor or a piezoelectric sensor.

  The tracking start / end determination unit 28 determines whether to start or end tracking based on the detection result of the tracking target region by the tracking target region detection unit 26. Here, “end” may include a temporary stop of tracking by occlusion or the like. Tracking starts when the tracking target appears within the viewing angle of the imaging device or when it appears from behind the object, etc.When the tracking target leaves the viewing angle of the imaging device or enters the shadow, etc. To finish. By notifying the tracking processing unit 30 of the determination result, the tracking processing of the tracking processing unit 30 is started and ended.

  The tracking processing unit 30 includes a sampling unit 42, an observation unit 48, and a result acquisition unit 50. The sampling unit 42 includes a shape prediction unit 44 that performs sampling in the space of the coefficient set α and a shape space vector prediction unit 46 that performs sampling in the space of the shape space vector x. The shape prediction unit 44 performs particle generation and extinction processing based on the probability density distribution in the space of the coefficient set α estimated for the previous image frame at time t−1. Then, as in the example of the above-mentioned Janken, the particles are distributed according to a predetermined rule corresponding to the shape represented by each particle.

  The shape space vector prediction unit 46 performs particle generation and extinction processing based on the probability density distribution in the space of the shape space vector x estimated for the previous image frame at time t-1. Then, a predetermined motion model is applied to all particles to cause the particles to transition in the space. By the processing of the shape prediction unit 44 and the shape space vector prediction unit 46, a plurality of candidate curves in the image frame at time t can be determined in consideration of shape change and translation, expansion / contraction, and rotation. The sampling unit 42 starts the process when receiving a signal indicating the start of tracking from the tracking start / end determining unit 28 and ends the process when receiving a signal indicating the end of tracking.

  The observation unit 48 observes the likelihood of the candidate curve defined by each particle generated, disappeared, and transitioned by the sampling unit. As described above, the likelihood is determined by searching for an edge in the vicinity of each candidate curve on the edge image generated by the image processing unit 22 and estimating the distance to the edge for each candidate curve. The result acquisition unit 50 calculates the probability density distribution as shown in Expression 3 based on the likelihood observed by the observation unit 48 in each of the space of the coefficient set α and the space of the shape space vector x, and the weighted average parameter thereby A tracking result such as curve data obtained by the above is derived and stored in the result storage unit 36. The data is returned to the sampling unit 42 for use in the tracking process at the next time t + 1. The data stored in the result storage unit 36 may be the value of each parameter that has been weighted and averaged, and may be any of an image composed only of a curve determined by that, or image data formed by combining a curve and an input image. But you can.

  When there are a plurality of tracking targets, the result acquisition unit 50 may further track each tracking target using a template prepared for each, and combine the tracking results into one tracking result. Further, a case where a plurality of tracking targets overlap is detected based on the tracking result, and the tracking target hidden behind is taken out of the tracking processing target at a predetermined timing. As a result, even if the observation likelihood is temporarily lowered due to the tracking target moving behind another tracking target, it is possible to avoid outputting an inappropriate tracking result.

  By performing the above-described processing in the image processing unit 22 and the tracking processing unit 30 for each frame, the result storage unit 36 stores, for example, moving image data including the tracking result. In this case, by outputting the moving image data to the display device 16 under the control of the output control unit 40, it can be displayed that the contour line moves in the same manner as the movement of the tracking target. As described above, in addition to displaying the tracking result as a moving image, processing such as output to another arithmetic module may be appropriately performed according to the purpose of tracking.

  Next, the operation of the tracking device 14 configured as described above will be described. First, the imaging device 12 captures an imaging target at a predetermined frame rate in response to a user instruction input or the like. The captured image is input as input image data to the image acquisition unit 20 of the tracking device 14 and stored in the image storage unit 24. In addition, the image storage unit 24 stores three types of control point sequences that define a plurality of reference shapes, in the above example, “gu”, “choki”, and “par”. In such a state, the following tracking process is performed.

  FIG. 7 is a flowchart showing the procedure of the tracking process in the present embodiment. First, the tracking target area detection unit 26 reads the input image data stored in the image storage unit 24 for each frame, and detects an area where an object that can be a tracking target exists. The tracking start / end determination unit 28 determines whether to start tracking based on the result (S20, S22). For example, when an object having a predetermined size and shape that can be estimated as a palm appears as a foreground extracted from an image frame, it is determined to start tracking. The size and shape of the foreground that is the criterion for determination is determined in advance logically or experimentally.

  N of S20 and S22 is repeated until it is determined that the tracking is started, and when it is determined that the tracking is started (Y of S22), the tracking processing unit 30 starts the tracking process. Here, it is assumed that the time corresponding to the image frame determined to start tracking is t = 0, and the subsequent image frames correspond to time t = 1, 2, 3,. First, when the sampling unit 42 requests the image processing unit 22 to perform edge image generation processing, the image processing unit 22 generates an edge image of a t = 0 image frame (S24). At this time, the sampling unit 42 may also request an edge image generation process for the subsequent frame, and the image processing unit 22 may sequentially perform the process.

  The shape prediction unit 44 of the sampling unit 42 first performs sampling by arranging particles evenly in a predetermined region of the space of the coefficient set α (S26). When the tracking target area detection unit 26 detects that the tracking target is in one of the reference shapes by template matching or the like, the particles are locally distributed within a predetermined range of a coefficient set that defines the reference shape. You may make it make it. Next, the shape space vector prediction unit 46 performs sampling by arranging particles evenly in a predetermined region of the parameter space (S28). Then, the observation unit 48 observes the likelihood by matching the candidate curve determined by each particle and the edge image, and the result acquisition unit 50 applies the expression 3 to both the space of the coefficient set α and the shape space vector x. An initial value p (t = 0) of the density distribution is calculated (S30).

  The result acquisition unit 34 further finally determines a curve obtained by weighted average of each parameter according to the probability density distribution p (t = 0) as the shape and position of the tracking target at time t = 0, and the original input Desired tracking result data, such as combining with an image frame, is generated and stored in the result storage unit (S32).

  On the other hand, the image processing unit 22 reads the image frame at time t = 1 from the image storage unit 24 and generates an edge image (N in S34, S24). The sampling unit 42 generates a number of particles corresponding to the generated initial value p (t = 0) of the probability density distribution in the space of the coefficient set α, and distributes the particles according to the value of the coefficient set α (S26). Further, it is generated in the space of the shape space vector x, and the particles are respectively transitioned based on a predetermined motion model (S28). The number of particles to be generated is controlled in consideration of the processing load based on the amount of computation resources of the tracking device 14 and the required result output speed. The rules and motion models to be distributed are determined in advance according to the type of tracking target so that tracking accuracy can be obtained from a Gaussian motion model, an autoregressive prediction motion model, or the like.

  Then, the observation unit 30 observes the likelihood of each candidate curve defined by the particles after transition, and obtains the probability density distribution p (t = 1) at time t = 1 based on the result. (S30). The likelihood is observed by searching for a contour line in the vicinity of each candidate curve, using the edge image at time t = 1 generated by the image processing unit 22 in S24. When there are a plurality of tracking targets, the above process is performed for all the tracking targets. Then, the result acquisition unit 34 finally determines a curve obtained by weighted average of each parameter according to the probability density distribution p (t = 1) as the shape and position of the tracking target at time t = 1, and the original input Desired tracking result data, such as combining with an image frame, is generated and stored in the result storage unit (S32).

  The tracking start / end determining unit 28 determines whether to continue or end the tracking process (S34). For example, the tracking end is determined when a target having a predetermined size and shape that can be estimated as a palm does not appear as a foreground for a predetermined time. Alternatively, the end of tracking is determined when the occlusion state continues for a predetermined time, such as when a tracking target turns behind another tracking target in real space. Further, a situation in which the tracking target deviates from the angle of view of the imaging device 12 continues for a predetermined time is also detected by a method similar to occlusion and the tracking end is determined.

  If it is determined in S34 that the tracking process is not terminated (N in S34), an edge image is generated from the image frame at time t = 2, and the probability density distribution p (at time t = 1 obtained in S32) is obtained. Particles are manipulated using t = 1), and likelihood observation, probability density distribution calculation, and tracking result data generation for the frame at time t = 2 are performed (S24 to S32). Thereafter, the processing from S24 to S32 is repeated for each frame until the tracking start / end determination unit 28 determines tracking end (Y in S34) in S34. As a result, for example, moving image data that has the same shape and movement as the palm of the hand of a janken and whose contour line as a tracking result changes with time is stored in the result storage unit 36. When the output control unit 40 outputs the data to the display device 16 or a module that provides another function, the user can use the tracking result in a desired form.

  In the description so far, the method for expressing the reference shape of the palm with a B-spline curve has been mainly described. However, the tracking target is not limited to the palm, but the same thing should be done for the entire human body, animals, objects, etc. Can do. Further, the curve representing the shape of the tracking target, the method of expressing the straight line, and the parameters defining the shape are not limited to the B-spline curve or the control point.

  As described above, in this embodiment, visual tracking that can cope with a change in the shape of the tracking target is possible. Being able to cope with the shape change means that the shape of the object can be recognized. In the course of the calculation, the distribution of the coefficient set α that defines the shape of the next image frame is predicted by the transition model from the coefficient set α that defines the shape of the previous image frame. That is, not only the shape recognition of the object in the image frame at the current time but also the shape of the object in the subsequent image frame is predicted.

  By using this feature, it becomes possible to detect the movement of the user in front of the camera in real time with a minimum delay time due to various processes, and a user interface with excellent responsiveness can be provided. For example, when moving a virtual person drawn on the screen according to the movement of your body or operating a remote-controlled robot hand, you can reduce the time from information input to result output .

  In the above description, the output control unit 40 generates a moving image in which the contour moves in the same manner as the movement of the tracking target by combining the contour of the tracking target obtained as a result of the tracking process with the input image. An example was given. In the present embodiment, as described above, the contour line to be tracked can be accurately traced regardless of the presence or absence of the shape change. Using this feature, various visual effects can be given not only to the display of the contour line but also to the region of the object in the image or the region other than the object. An example will be described below.

  For example, when the outline of the hand is acquired by the tracking process, the position of the finger from the thumb to the little finger and the position of the nail of each finger can be specified. Here, the “position” may be the position of a point such as a feature point or the position of a surface having a finite area. In a configuration in which an image of a user's hand is captured and displayed on the display device, a nail image in which nail art is applied to the position of the nail is synthesized or a ring image is synthesized on the root of a desired finger. You can virtually try out art and try on a ring.

  Since the tracking device 14 can derive a contour line corresponding to the movement or shape change of the hand, the hand need not be in a predetermined position and in a predetermined state. Even if the orientation, size, etc. of the nails change depending on the orientation, size, depth direction, etc. of the hand, the nail art and rings fit the actual hand by deforming the prepared image accordingly. Can be synthesized, increasing the reality. Furthermore, since the inclination of the hand can be estimated by the movement of the contour line, if the image to be synthesized is changed depending on the inclination of the camera, such as the front and the side, it is possible to confirm the shadow and the light reflection.

  FIG. 8 shows the configuration of an image processing apparatus that performs image processing using the contour line acquired by the tracking process. The image processing device 70 includes a tracking device 14 that acquires the contour line of an object, an input unit 72 that receives an instruction input from a user, a part specifying unit 74 that specifies a position of a predetermined part of the object, and a position of the predetermined part A processing unit 76 that performs predetermined image processing based on the information, an output unit 78 that outputs the result of the image processing, and a processing data storage unit 80 that stores data used for image processing are included.

  The tracking device 14 can have the same configuration as the tracking device 14 shown in FIG. Note that, depending on the region of interest such as the head, the shape may not change, and in this case, the processing of the shape prediction unit 44 may be omitted as appropriate. On the other hand, when various shape changes can be predicted, such as a hand, tracking processing corresponding to those shapes is performed. Even in this case, by defining the shape of the object with the linear sum of the parameters that define the reference shape as described above, any shape can be expressed with only a small number of reference shape preparations. In the case of a hand, for example, as a reference shape, by preparing five shapes in which any one of five fingers is raised and the remaining four are gripped, the number of standing fingers can be from 1 to 5 Can express the hand.

  Further, the image to be processed is the one stored in the image storage unit 24 of the tracking device 14, but the image data input to the image processing device 70 from a separate imaging device is tracked in real time. In this case, image processing may be performed. The input unit 72 is an interface for the user to instruct the image processing apparatus 70 to start and end the process and to select the content of the processing process. The input unit 72 may be a general input device such as a keyboard, a mouse, a trackball, a button, or a touch panel, or may be a combination with a display device that displays options for input.

  The part specifying unit 74 acquires data of a curve representing the outline of the target object as a tracking result from the tracking device 14 and specifies the position of the target part such as a nail or a finger. The target part may be determined by the user selecting and inputting to the input unit 72, or may be set in advance. In any case, information related to the positional relationship between the contour line obtained from the tracking device 14 and the target part is stored in the processed data storage unit 80. In the example of the nail art described above, the position of the nail is specified by setting in advance a rule for deriving the nail region from the point indicating the fingertip and the thickness of the fingertip in the outline of the hand. Further, the part specifying unit 74 specifies the inclination of the object or the inclination of the target part from the contour line.

  FIG. 9 is a diagram for explaining an example of a method in which the part specifying unit 74 specifies the inclination of the object. In the figure, the state 82 is when the object 86 is viewed from the front, and the state 84 is when the object is rotated from the state 82 about the rotation axis 88 by the angle θ. Assuming that the width of the object perpendicular to the rotation axis 88 is W, the apparent width is also W in the state 82 as shown in the figure. On the other hand, in state 84, the width of the object appears to be Wcos θ. Therefore, for example, if the front image of the object is first taken as a calibration image, the rotation angle can be obtained from the apparent width using the relationship shown in FIG. The same applies to the inclination of the target part. Which direction is tilted appropriately uses information that can be acquired from the contour line, such as the position of the thumb. In this embodiment, since the movement of the contour line is sequentially traced, the rotation axis can be easily obtained by acquiring the movement of the object for a predetermined frame. Further, the time change of the rotation angle may be obtained from the movement of the object, and the inclination of the immediately following frame may be estimated.

  Returning to FIG. 8, the processing processing unit 76 performs a predetermined processing on the target site specified by the site specifying unit 74. The content of the processing may be determined by the user selecting and inputting to the input unit 72, or may be set in advance. Or the combination may be sufficient. For example, choices such as nail art colors and patterns are displayed on the display device, and a user's selection input is accepted. Then, the selected nail art image is read out from the processed data storage unit 80, and displayed on the nail portion of the input image obtained by capturing the user's hand. For this reason, the processing data storage unit 80 stores image data necessary for processing, such as texture data of an image to be synthesized such as a nail or 3D graphics data such as shape data.

  Moreover, since the site | part specific part 74 also specifies the inclination of a target site | part, the process part 76 also changes the image to synthesize | combine according to the said inclination. At this time, not only the inclination of the image to be synthesized is changed, but also a change in shadow and reflection of light according to the motion is expressed. Also, if the images to be combined overlap, such as when the target part overlaps, the rear part is specified based on the temporal change of the part and the contour line, and the hidden part of the composite image corresponding to the rear part is deleted. . For these processes, generally used techniques such as shading and hidden surface removal in the field of 3D graphics can be appropriately used. Furthermore, since the contour line obtained in the present embodiment can correspond to an arbitrary shape of the target object, no image processing is particularly performed when the target part is not visible on the screen. For example, if the hand is in the shape of “choki” and the instep is in front, the nail image is superimposed only on the index finger and the middle fingernail.

  The output unit 78 displays an image obtained as a result of the processing performed by the processing unit 76 or stores it as moving image data. Therefore, the output unit 78 is composed of a storage device such as a display device or a hard disk drive. In the case of a display device, the display device of the input unit 72 may be the same.

  Next, the operation of the image processing apparatus 70 configured as described above will be described. FIG. 10 is a flowchart showing a processing procedure of image processing performed by the image processing apparatus 70. First, the user inputs to the input unit 72 an instruction to start processing and selection of processing contents (S40). A multi-stage input mode may be employed, such as selecting a favorite item from the nail displayed on the display device after inputting a processing start instruction. In addition, a change in processing content such as reselecting another nail may be accepted at any time during the subsequent processing.

  Then, the tracking device 14 acquires an image of the target object at time t (S42), and acquires a contour line of the target object by performing tracking processing (S44). As described above, the image of the object may be acquired in real time by the user placing the object such as his / her hand on a predetermined place and capturing it in real time, or the moving image previously captured The image frame may be read from the image storage unit 24.

  Next, the site | part specific | specification part 74 specifies the position and inclination of the site | part according to the content of the process from the outline data acquired from the tracking apparatus 14 as above-mentioned (S46). Then, the specified information is transmitted to the processing unit 76 together with the image of the object. The processing unit 76 generates a processed image by performing the processing of the content selected by the user in S40 based on the information on the target part (S48). The output unit 78 performs output processing such as displaying the generated processed image (S50). While the user does not input a process end instruction to the input unit 72 (N in S52), the time t is incremented (S54), and the processes from S42 to S50 are performed on each image frame. When the user inputs an instruction to end the process, the process ends (Y in S52).

  By such an operation, it is possible to process an image that follows the movement of the object in consideration of shadows, changes in reflected light, occlusion, and the like. In the description so far, the main example has been an aspect in which the object is a hand and trial painting of nail art is performed in a virtual space, but the present embodiment can realize many other application examples. Hereinafter, application examples that can be realized by the image processing apparatus 70 will be described.

  FIG. 11 shows an example of a screen displayed on the display device of the output unit 78 when the image processing device 70 realizes an aspect of trying on clothes in a virtual space. Virtual try-on screen 90 includes a try-on image display area 92 and a clothes image display area 94. In this aspect, the user first stands in front of the imaging apparatus so that the whole body falls within the viewing angle. The image including the whole body of the user acquired by the imaging device is displayed in a try-on image display area 92 of the virtual try-on screen 90 displayed on the display device. If the imaging device is arranged in the same direction as the display device, the user can see an image obtained by capturing his / her whole body from the front.

  The clothes image display area 94 displays a list of clothes images that can be selected as try-on objects. For example, a clothing store or an auction exhibitor who receives an order for clothes via a network prepares their products as images. The image processing apparatus 70 acquires the image via the network in accordance with an instruction input from the user, and displays it in the clothes image display area 94. If the input unit 72 is a controller that can operate the pointer 96 displayed in the virtual try-on screen 90 at the user's hand, the user operates the controller to select a clothes to be tried on from the clothes image display area 94 with the pointer 96. can do.

  Then, according to the processing procedure shown in FIG. 10, an image in which the clothes selected from the clothes image display area 94 are combined with the user's body displayed in the try-on image display area 92 can be generated. When the image is displayed in the try-on image display area 92, the user can see his / her appearance after trying on the selected clothes. In this embodiment, the tracking device 14 tracks the contour of the user's head using an Ω-type template. In the case of the head, as described above, it can be traced by translation, expansion / contraction, and rotation of the Ω-type template, so the processing of the shape prediction unit 44 may be omitted.

  Then, the site | part specific | specification part 74 specifies the position and magnitude | size of a shoulder line among the omega-type head outlines which the tracking apparatus 14 output. The processing unit 76 then superimposes the clothes image on the user's image so that the shoulder line of the selected clothes image overlaps the identified shoulder line of the user. By repeating this process for each image frame at each time, it is possible to move the synthesized clothes image following the user's movements, and the user himself can try on clothes and move as if moving. it can.

  Even if the user does not have to face the imaging device and faces sideways or rotates, the part specifying unit 74 detects the orientation of the user's body according to the principle shown in FIG. The clothes image is rotated accordingly. For this purpose, images obtained by photographing clothes from a plurality of predetermined angles are stored in the processed data storage unit 80. The other angles are interpolated by a known method of 3D graphics. Whether the user's body is facing right or left may be estimated by the movement from the previous image frame as described above, or existing face detection technology is introduced to determine from the face orientation. May be.

  Note that the example in FIG. 11 illustrates a state in which the user is facing substantially rearward with respect to the imaging apparatus. When the imaging device and the display device are installed in the same direction, the user cannot see the virtual try-on screen 90 of the display device at this moment. Therefore, the processing unit 76 may detect a state in which the user is facing backward, and the processed image generated at that time may be controlled to delay display for a predetermined time in units of several seconds, for example. The state in which the user is facing backward is detected based on a temporal change in the width of the shoulder line of the user's contour line or the fact that no face is detected in the face detection process. By doing so, the user can confirm the back of the user who tried on the clothes.

  Further, when the processing unit 76 detects that the user is rotating from a change over time in the width of the shoulder line, the processing unit 76 may express that the clothes being tried on change in shape according to the rotation speed. . For example, widen the skirt of a skirt while trying it on, or inflate a blouse. If a table that associates the rotation speed with the degree of shape change is prepared according to the hardness of the clothing, the shape of the clothes, etc., the shape change according to the rotation speed can be applied with general 3D graphics technology. Can do. By doing in this way, the state of clothes can be confirmed at intervals closer to reality.

  Another application example that can be realized by the image processing apparatus 70 is mosaic processing. For example, a mosaic process can be applied only to the head of a person on a video image of the person photographed in advance. Also in this case, the tracking device 14 performs tracking processing of a person's head using an Ω-type template and acquires a contour line. The part specifying unit 74 specifies, for example, an area surrounded by an Ω-shaped outline and a line segment connecting the end points as a head area. The processing unit 76 performs mosaic processing on the identified area. By repeating this for image frames at each time, it is possible to generate a moving image that has been subjected to mosaic processing following the movement of a person.

  The tracking device 14 always acquires the outline of the head regardless of the orientation of the person's face. Therefore, it is difficult to specify by face detection or the like, and the head region can be specified even when a person turns sideways or scolds or faces backward. Then, in a situation where a person is identified even in the back of the head, etc., the face is not detected and the mosaic is removed, or the mosaic is regularly placed in an extra area including the area around the person so that the mosaic is not removed. It is possible to avoid the situation of giving. Thereby, it is possible to safely conceal information relating to the figure of a person while holding necessary information included in the image such as a situation around the person.

  As another application example that can be realized by the image processing device 70, there is information display of an object on an image. As an example, FIG. 12 shows a screen that displays information about players in a soccer game. The player information display screen 120 shown in the figure is, for example, a game relay video, and three players 122, 126, and 130 are within the viewing angle of the imaging apparatus. And processing which adds the image of the information tags 124 and 128 which consist of the area | region which displayed the information which shows each player's information, such as the arrow which points to a player, and a name, a back number, today's number of shots, on the head of the players 122 and 126 is carried out. It has been subjected. As shown in the figure, the sizes of the information tags 124 and 128 are changed according to the distance of the player from the imaging device.

  In this case, the tracking device 14 performs tracking processing of the player's head in the video during the match using the Ω-type template, and acquires the contour line. The part specifying unit 74 specifies the apex of the Ω-type outline as the top of the head, and acquires the size of the outline. The processing unit 76 determines the size of the information tag based on the correspondence relationship between the size of the contour line and the size of the information tag that is set in advance. Then, information on each player prepared in advance is read from the processed data storage unit 80 to generate an information tag image, which is displayed superimposed on the video during the game with the arrow pointing to the top of each player.

  Here, it is desirable to display the information tag so as not to cover other players. Therefore, the part specifying unit 74 specifies an area where no other player exists based on the information on the contour of the head of the player within the viewing angle, and the processing unit 76 displays an information tag in the area. Also good. By repeating this process for the image frames at each time, it is possible to generate a video of a game in which an information tag that follows the movement of the player is displayed.

  By changing the size of the information tag according to the distance from the imaging device to the player, it is possible to produce a sense of distance also in the information tag, and the information tag is complicated even if many people exist within the viewing angle This makes it easier to know which player's information tag it is. When the tracking device 14 detects an overlap of a plurality of players, the processing unit 76 displays the information tags so that the information tags of the players behind are also partially hidden by the information tag of the player in front. Also good.

  In addition, for the size of the information tag, either or both of an upper limit and a lower limit to be displayed may be set. In the example of FIG. 12, the information tag is not displayed for the player 130 farthest away because the size of the information tag is less than the lower limit. By setting the lower and upper limits for the size of the information tag, there is no need to display small information tags that cannot distinguish characters, or large information tags that cover large areas in the image, making it always easy to see It becomes.

  By displaying information tags in this way, it is easy to identify players in sports such as soccer and marathon that are performed by a large number of people in a wide area, and further, information on each player can be displayed with the status of the game and the players. You can easily grasp while watching the movement of the. The information tag may be switched between display and non-display by an instruction input to the input unit 72 by the user. The information tag can be used not only for sports videos but also for displaying information about drama characters and actors, displaying information about products in moving images, and the like. Furthermore, not only live-action images but also information on people or objects in a virtual space drawn by computer graphics may be displayed.

  According to the present embodiment described above, the shape to be tracked is expressed by a linear sum of control point sequences that define B-spline curves representing a plurality of reference shapes prepared in advance. Then, a coefficient set composed of coefficients related to each control point sequence is included in the parameters defining the particles. As a result, the Condensation algorithm that can only handle translation, expansion, contraction, and rotation of one template shape can be applied in an environment where the shape of the tracking target itself changes.

  In addition, since all intermediate shapes of the reference shape can be expressed by adjusting the coefficient set, the memory area to be used can be greatly reduced compared to preparing all the shapes that the object can take, and it is also useful for calculation. The number of parameters used can be reduced. In addition, since the coefficient set can be handled in the same way as the shape space vector, the conventional algorithm can be used as it is, and the advantage of the tracking process using the particle filter can be maintained without increasing the calculation amount. it can.

  Further, by introducing a transition model in the space of the coefficient set, the shape immediately after is predicted, and the particles are distributed in the vicinity of the coefficient set defining the shape. Thereby, the tracking process can be performed efficiently and accurately without increasing the number of particles. In general, shape recognition and tracking processing are separate processes, but they can be connected by the concept of particles, and simultaneous processing is possible with a simple algorithm.

  Setting a shape transition model and distributing particles based on the model is equivalent to predicting the shape of the object. This makes it possible to pre-read the user's hand in the janken, and to realize an interface with good responsiveness to the user's movement. It can also be applied to operating robots and medical instruments.

  Furthermore, it is possible to provide various functions by accurately obtaining the contour line of an object that performs at least one of shape change, translation, expansion / contraction, and rotation, and performing image processing using the information. it can. Specifically, trial application of nail art, try-on of rings and clothes, mosaic processing, addition of information tags, and the like can be performed. Conventionally, when cutting out the outline of an object in an image, it is necessary for the person to check and cut out the image frames one by one, especially in the case of moving images, the work cost was significant. . In the present embodiment, the contour line can be acquired accurately and easily even for a moving image. In addition, special conditions are not required for the input image, such as chroma key composition using blue screen or green screen, face detection technology, and the like.

  Thereby, it is possible to easily perform processing according to the movement of the object in addition to obtaining the contour line with a small calculation amount as compared with the conventional method. Since the tilt and overlap of objects can also be detected, the processing area and the shape of the image to be synthesized can be changed, and graphics processing such as shading and hidden surface removal can be further performed to express the virtual space more realistically. it can. In addition, since the area where the object exists and the area where the object does not exist can be specified according to the movement of the object, only the object is processed, or the area without the object is selected and processed. Therefore, from the viewpoint of design and information disclosure, a processed image corresponding to the user's needs can be generated flexibly.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in this embodiment, in order to mainly estimate the contour line of the tracking target, the contour line of the target object is expressed by a linear sum of parameters that define a reference shape prepared in advance. On the other hand, this expression method is not limited to the estimation of the contour line of the tracking target, but can be widely applied as an expression method for drawing an object. For example, it may be used for generating polygon data used on three-dimensional computer graphics. Even in such a case, the amount of memory to be used can be remarkably reduced as compared with a case where parameter sets of all shapes that can be expressed are prepared.

It is a figure for demonstrating the visual tracking method when a person is made into the tracking object. It is a figure explaining the technique of probability density estimation using a particle filter. It is a figure for demonstrating the value of a coefficient set and a transition model in this Embodiment. It is a figure for demonstrating the value of a coefficient set and a transition model in this Embodiment. It is a figure which shows the structural example of the visual tracking system in this Embodiment. It is a figure which shows the structure of the tracking apparatus in this Embodiment in detail. It is a flowchart which shows the procedure of the tracking process in this Embodiment. It is a figure which shows the structure of the image processing apparatus which performs an image processing process using the outline acquired by the tracking process in this Embodiment. It is a figure for demonstrating the example of the method at the time of the site | part specific part of this Embodiment specifying the inclination of a target object. It is a flowchart which shows the process sequence of the image process which the image processing apparatus of this Embodiment performs. It is a figure which shows the example of a screen displayed on a display apparatus, when the aspect which tries on clothes in virtual space is implement | achieved by the image processing apparatus of this Embodiment. It is a figure which shows the example of a screen which displays the information of the player in a soccer game with the image processing apparatus of this Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Visual tracking system, 12 Imaging apparatus, 14 Tracking apparatus, 16 Display apparatus, 20 Image acquisition part, 22 Image processing part, 24 Image memory | storage part, 26 Tracking object area | region detection part, 28 Tracking start end determination part, 30 Tracking processing part , 30 observation unit, 36 result storage unit, 40 output control unit, 42 sampling unit, 44 shape prediction unit, 46 shape space vector prediction unit, 48 observation unit, 50 result acquisition unit, 70 image processing device, 72 input unit, 74 Site specifying unit, 76 processing unit, 78 output unit, 80 processing data storage unit, 90 virtual try-on screen, 92 try-on image display region, 94 clothes image display region, 120 player information display screen.

Claims (14)

A reference shape storage unit that stores a plurality of parameters that define contour lines of a plurality of reference shapes;
An object shape determination unit that expresses and outputs the contour shape of an object in an image by the linear sum by determining a set of coefficients of each parameter in a linear sum of a plurality of parameters stored in the reference shape storage unit When,
An image processing apparatus comprising: An image acquisition unit that acquires moving image stream data including a first image frame and a second image frame obtained by imaging the object;
The object shape determination unit
In the coefficient set space defined by the set of coefficients, particles used for the particle filter are generated and disappeared based on the estimated existence probability distribution of the object in the first image frame in the space, and based on a predetermined transition model A shape prediction unit for transition;
An observation unit for observing the likelihood of each particle by matching the contour line of the object in the second image frame with the candidate contour defined by the particle;
An estimated existence probability distribution in the coefficient set space of the object in the second image frame is calculated based on the likelihood observed by the observation unit, and the coefficient set of each particle is weighted based on the estimated existence probability distribution A contour line acquisition unit that estimates a contour shape of the object in the second image frame by performing
The image processing apparatus according to claim 1, further comprising:   The image processing apparatus according to claim 1, wherein the parameter defining the contour line is a control point sequence when the contour line is represented by a B-spline curve.   The image processing apparatus according to claim 1, wherein the parameter defining the contour line is a knot sequence when the contour line is represented by a B-spline curve. The particles that the shape prediction unit has made transition to the shape space vector space defined by the shape space vector that defines the translation amount, magnification, and rotation angle of the contour line determined by each particle, A shape space vector prediction unit that generates and disappears based on the estimated existence probability distribution in the space and makes a transition based on a predetermined transition model,
The observation unit observes the likelihood of the particles that the shape space vector prediction unit has transitioned,
The contour acquisition unit further calculates an estimated existence probability distribution in the space of the shape space vector of the object in the second image frame based on the likelihood observed by the observation unit, and the estimated existence probability distribution 3. The translation amount, magnification, and rotation angle of the contour line of the object in the second image frame are further estimated by weighting the shape space vector of each particle based on the second image frame. Image processing device.   The shape prediction unit transitions particles generated and extinguished based on the estimated existence probability distribution of the object in the first image frame so as to form a Gaussian distribution centered on coordinates before transition in the coefficient set space. The image processing apparatus according to claim 2, wherein:   When the shape prediction unit detects that the shape defined by the particle is a shape between the first reference shape and the second reference shape based on the coordinates before the transition of the particle in the coefficient set space. In the coefficient set space, the standard deviation in the direction of the line connecting the coordinates representing the first reference shape and the coordinates representing the second reference shape has a Gaussian distribution larger than the standard deviation in the other direction. The image processing apparatus according to claim 6, wherein the particles are transitioned.   When the shape prediction unit detects that the shape defined by the particle is in the state considered as the reference shape based on the coordinates before the transition of the particle in the coefficient set space, the shape prediction unit may make a transition from the reference shape. The image processing apparatus according to claim 2, wherein the particles are distributed on the assumption that the probability of transition to each of the possible reference shapes is equal. Reading a plurality of parameters defining a plurality of reference shape contours from a storage device, and determining a set of coefficients for each parameter in a linear sum of the parameters;
Using the defined set of coefficients to represent and output the contour of the object in the image as the linear sum;
An image processing method comprising: Further including obtaining moving image stream data including a first image frame and a second image frame obtained by imaging an object and storing them in a memory;
The outputting step predicts a contour line of the object in the second image frame based on an estimated existence probability distribution of the object in the first image frame in a coefficient set space defined by the set of coefficients. Obtaining an estimated presence probability distribution of the object in the second image frame by comparing with a contour line of the object in the second image frame read from the memory;
Estimating a contour line of an object in the second image frame based on the estimated existence probability distribution and storing it in a memory;
The image processing method according to claim 9, further comprising: A function of reading a plurality of parameters defining contour lines of a plurality of reference shapes from a storage device, and determining a set of coefficients of each parameter in a linear sum of the parameters;
A function of expressing and outputting the outline of the object in the image by the linear sum using the set of the determined coefficients;
A computer program for causing a computer to realize the above. Causing a computer to further realize a function of acquiring moving image stream data including a first image frame and a second image frame obtained by capturing an object and storing them in a memory;
Based on the estimated existence probability distribution of the object in the first image frame in the coefficient set space defined by the set of coefficients, the contour line of the object in the second image frame is predicted and read from the memory A function of obtaining an estimated presence probability distribution of the object in the second image frame by comparing with an outline of the object in the second image frame;
A function for estimating an outline of an object in the second image frame based on the estimated existence probability distribution and storing the outline in a memory;
The computer program according to claim 11, wherein a computer is implemented. A function of reading a plurality of parameters defining contour lines of a plurality of reference shapes from a storage device, and determining a set of coefficients of each parameter in a linear sum of the parameters;
A function of expressing and outputting the outline of the object in the image by the linear sum using the set of the determined coefficients;
The recording medium which recorded the computer program characterized by making a computer implement | achieve. Causing a computer to further realize a function of acquiring moving image stream data including a first image frame and a second image frame obtained by capturing an object and storing them in a memory;
Based on the estimated existence probability distribution of the object in the first image frame in the coefficient set space defined by the set of coefficients, the contour line of the object in the second image frame is predicted and read from the memory A function of obtaining an estimated presence probability distribution of the object in the second image frame by comparing with an outline of the object in the second image frame;
A function for estimating an outline of an object in the second image frame based on the estimated existence probability distribution and storing the outline in a memory;
14. A recording medium on which a computer program according to claim 13 is recorded.

Images