CN104103062A - Image processing device and image processing method - Google Patents

Abstract

The invention provides an image processing device and an image processing method. The image processing device comprises an object extraction unit, a matching unit, a feature extraction unit and a depth determination unit, wherein the object extraction unit is used for extracting an image of a target object from a monocular video sequence frame to serve as an object image; the matching unit is used for matching an observation object image with a reference object image to obtain a matching parameter, and the reference object image and the observation object image are object images extracted from the reference frame and the observation frame except from the reference frame; the feature extraction unit is used for the features reflecting depth changes of the observation object image and the reference object image in relative to an imaging plane by using the matching parameter; and the depth determination unit is used for determining the depth changes on the basis of the extracted features.

Description

Image processing device and image processing method

technical field

The present disclosure generally relates to video image processing, and in particular, relates to an image processing device and an image processing method capable of determining a depth change of a target object relative to an imaging plane according to a monocular video sequence.

Background technique

Detecting a target object in an image and estimating the distance between the target object and the camera is widely used in various fields, such as visual surveillance, obstacle detection, and human-computer interaction. Most of the traditional target object depth detection techniques are based on the principle of stereo vision to estimate the target depth. In this technique, two calibrated cameras are required, and the depth of the target object is detected by analyzing the correspondence between image pairs from the two cameras.

Traditional techniques rarely address object depth estimation in monocular vision. In one method of target depth estimation using monocular images, such a scheme is disclosed: based on some assumptions made on the motion properties of moving target objects, such as maximum velocity, minimum velocity change, consistency, and continuous motion, etc., to detect the moving target; through this detection, the position of each moving target object in the monocular sequence image in the two consecutive images is output; then, according to the output position, the moving target is estimated by using the over-constrained method distance.

Contents of the invention

In the prior art such as the above solution, the motion feature of the object is used for depth estimation, and the algorithm is relatively complicated and the accuracy rate is not high.

An object of the present invention is to provide an image processing apparatus and an image processing method that perform depth detection by taking into account the change characteristics of the object image in each frame associated with the depth change of the target object without performing motion prediction, Thereby instead of analyzing the motion characteristics of the image.

According to one aspect of the present disclosure, an image processing device is provided, including: an object extraction unit, configured to extract an image of a target object from a frame of a monocular video sequence as an object image; a registration unit, configured to convert the observed object image Registration is performed on the reference object image to obtain registration parameters, the reference object image and the observation object image are object images extracted from the reference frame and the observation frame other than the reference frame respectively; the feature extraction unit is used to use the registration parameters to extract features capable of reflecting depth changes in the image of the observed object compared with the image of the reference object relative to the imaging plane; and a depth determination unit configured to determine the depth change based on the extracted features.

In one embodiment according to the present disclosure, the feature extraction unit may extract the scaling parameter reflecting the size change of the object image among the registration parameters as a feature.

In another embodiment according to the present disclosure, if S represents the scaling parameter of the observation frame with respect to the reference frame, then: when S>1+ε _a , the depth determination unit determines that the target object becomes shallow relative to the imaging plane; when S <1− _εb , the depth determination unit determines that the target object becomes deeper relative to the imaging plane; and otherwise, the depth determination unit determines that the depth change of the target object relative to the imaging plane is uncertain. Here, ε _a and ε _b are small positive numbers as a certain margin.

In another embodiment according to the present disclosure, the image processing device may further include a motion direction identification unit configured to identify the motion direction of the target object relative to the imaging plane. Wherein, when the image processing device is made to sequentially perform depth determination on the observation frames within a predetermined period, the motion direction identification unit may identify the motion direction according to the result sequence determined by the depth determination unit.

In another embodiment according to the present disclosure, when there are more than the first predetermined number of "darkening" in the result sequence, the motion direction identification unit may identify that the target object moves in a direction away from the imaging plane; and when the result When there are more than a second predetermined number of consecutive "shallows" in the sequence, the motion direction identification unit may identify that the target object is moving in a direction close to the imaging plane.

In another embodiment according to the present disclosure, when there are more than the third predetermined number of "deepening" in the result sequence in succession, and the continuous product of the corresponding scaling parameters S is less than the first threshold, the motion direction identifying unit may identify When the target object moves in a direction away from the imaging plane; and when there are more than a fourth predetermined number of "shallows" in the result sequence, and the product of the corresponding scaling parameters S is greater than the second threshold, the motion direction identification unit may The movement of the target object in a direction close to the imaging plane is identified.

In another embodiment according to the present disclosure, the feature extraction unit may include: an alignment unit for aligning the observation object image with the reference object image according to the translation parameters in the registration parameters; and a histogram generation unit for aligning The edge portion of the object image generates histograms of oriented gradients as features that can reflect depth changes.

In another embodiment according to the present disclosure, the object extraction unit may include a detection unit. The detection unit is used to detect the object image by using multi-size sliding windows, so as to determine the area where the object image is located.

In another embodiment according to the present disclosure, the extracting unit may include a segmentation unit. The segmentation unit is used for performing segmentation processing on the region where the object image is located, so as to extract the object image.

In another embodiment according to the present disclosure, the detection unit may detect the object image using a trained target object detector.

According to one aspect of the present disclosure, an image processing method is provided, including: extracting an image of a target object from a frame of a monocular video sequence as an object image; registering the image of the observed object against the image of the reference object to obtain a registration Parameters, the reference object image and the observation object image are the object images extracted from the reference frame and the observation frame outside the reference frame respectively; by using the registration parameters to extract, it can reflect that the observation object image is compared with the reference object image relative to the imaging plane features of the depth variation; and determining the depth variation based on the extracted features.

By implementing the image processing device and the image processing method according to the present invention, the algorithm for detecting the depth of the target object can be simplified, the relative depth change of the target object can be detected in real time, and the detection accuracy can be improved.

Description of drawings

The above and other objects, features and advantages of the present invention will be more easily understood with reference to the following description of the embodiments of the present invention in conjunction with the accompanying drawings. In the drawings, the same or corresponding technical features or components will be indicated by the same or corresponding reference numerals. The dimensions and relative positions of elements are not necessarily drawn to scale in the drawings.

Figure 1 is a schematic diagram illustrating the principle on which the invention is carried out.

FIG. 2 is a block diagram showing the structure of an image processing device 200 for depth detection according to an embodiment of the present disclosure.

FIG. 3 exemplarily shows a window in which a hand is detected and an image of a hand after the window is segmented.

FIG. 4 is a block diagram showing the structure of a feature extraction unit 400 according to an embodiment of the present disclosure.

FIG. 5 is a schematic diagram illustrating the use of the feature extraction unit 400 to align images and generate an edge feature histogram.

FIG. 6 is a block diagram showing the structure of an image processing device 600 for depth detection according to an embodiment of the present disclosure.

FIG. 7 is a flowchart illustrating an image processing method for depth detection according to an embodiment of the present disclosure.

FIG. 8 is a flowchart illustrating an image processing method for depth detection according to an embodiment of the present disclosure.

FIG. 9 is a block diagram showing an exemplary structure of a computer implementing the present disclosure.

Detailed ways

Embodiments of the present invention will be described below with reference to the drawings. It should be noted that representation and description of components and processes that are not related to the present invention and known to those skilled in the art are omitted from the drawings and the specification for the purpose of clarity.

For the convenience of description, the hand will be described below as an example of the target object. In a human-computer interaction system, an event can be triggered by pushing the hand forward or backward relative to the camera. The forward pushing or pulling of the hand can be determined by detecting the change of the distance (depth) of the hand relative to the imaging plane of the camera. It can be understood that the solution according to the present invention can be applied to any other target objects, such as vehicles, people (whole body or part), any pointing devices such as pointing sticks, etc.; and any other application scenarios, such as visual monitoring, obstacle detection, etc.

Figure 1 is a schematic diagram schematically illustrating the principle on which the invention is carried out. Given a sequence of images captured by a monocular camera, it is assumed that the object to perform depth detection along the optical axis is a hand appearing in the sequence of images. Figure 1 shows the depth change of two adjacent frames (frame t-1 and frame t) and the change of imaging size on the image plane when a person is doing the "forward" action. In Figure 1, d represents the distance from the hand to the camera; f represents the focal length of the camera; h typically represents the size of the hand; s represents the size of the image formed by a hand with a size h at a distance d on the imaging plane ; Δd represents the distance that the hand moves between two frames, more precisely, the distance that the hand moves between two frames in the depth direction (optical axis direction) relative to the imaging plane; Δs is the response to the hand in the depth direction The change in the image plane changes in the size of the image.

Through the geometric proportion relationship shown in the illustration, it can be seen that the relationship between the depth change Δd of the hand and the imaging size change Δs on the image plane is shown in formula (1):

$\frac{\mathrm{<span\; class="notranslate">\; \&Delta;s</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; \&Delta;d</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Delta;d</span>}}<span\; class="notranslate">\; \&ap;</span>\frac{\mathrm{<span\; class="notranslate">\; \&Delta;d</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Wherein, when comparing two consecutive frames or frames that are close to each other, Δd<<d, so "≈" is reasonable. It can be seen from formula (1) that the rate of change of the depth of the hand can be estimated by detecting the rate of change of the imaging size.

In addition, from the change in the size of the object image caused by the change of the object depth, it is associated that if the object images corresponding to different depths are aligned according to the corresponding positions of each part, then when the observation object image as the object image extracted from the observation frame and the object image as the When the object image extracted from the reference frame is shallower relative to the imaging plane compared to the reference object image, the observed object image will completely cover the reference object image in the aligned image; For example, when darkened relative to the imaging plane, the image of the observed object will be smaller than the image of the reference object in the aligned image, so that the image of the reference object cannot be completely covered. Visible: In both cases, the edge parts of the aligned object images will exhibit different characteristics. If these features are extracted, it is possible to obtain histograms that are significantly different. That is to say, the depth change of the observation object image compared to the reference object image can be determined according to the histograms of the edge features of the alignment object image in these two cases.

The present invention has been made in consideration of the above points.

FIG. 2 is a structural block diagram illustrating an image processing device 200 for determining a depth change of a target object relative to an imaging plane according to an embodiment of the present invention.

As shown in FIG. 2 , the image processing device 200 includes: an object extraction unit 201 , a registration unit 202 , a feature extraction unit 203 and a depth determination unit 204 .

The object extraction unit 201 extracts an image of a target object from a frame of a monocular video sequence as an object image for subsequent processing. Various image extraction methods known in the art can be used in the extraction unit 201, as long as the image of the target object can be identified and separated from the video frame to meet the needs of subsequent processing.

In one embodiment, for example, the extraction unit 201 may include a detection unit (not shown) for detecting the object image from the video frame. For example, in one example, the detection unit may use multi-sized sliding windows to detect the object image, so as to determine the area where the object image is located. Specifically, the detection unit can use a sliding window of a specific size to scan the video frame, and perform feature extraction on the image content in the sliding window and send the extracted features to the classifier to determine whether there is an object in the sliding window image. After scanning the entire frame with the sliding window, the size of the sliding window is adjusted, and then the above steps of scanning, extracting and determining are repeated until the region where the object image is located is determined.

The classifier used by the detection unit can be constructed and trained using any conventional features. In some embodiments, in order to detect the target object more accurately, a trained target object detector may be used to detect the object image. By using standard learning machine techniques, such as support vector machines, trained object detectors are able to detect object images more accurately. This is especially useful for objects with less texture or sharp edges, such as hands, and scenes with complex backgrounds or varying lighting. Furthermore, the use of trained object detectors also opens up the possibility to detect non-rigid objects. An example of a non-rigid object is a hand that changes pose during motion, or that rotates out of plane.

In some embodiments, in order to improve the accuracy of subsequent depth change detection processing, the extraction unit 201 may further include a segmentation unit (not shown). The segmentation unit is used to perform segmentation processing on the region where the detected object image is located (for example, the window where the object image is detected), so as to separate the object image as a foreground from a background region in the region. Segmentation can be performed using various methods commonly used in the art. For example, in an embodiment where a hand is used as a target object, a skin color model may be constructed to perform foreground and background separation on detected windows containing hands. FIG. 3 exemplarily shows the sliding window where the hand is detected, and the image of the hand as the object image obtained after the window area is segmented. Segmenting the hand image can effectively reduce the noise introduced into subsequent depth detection processing, thereby improving the accuracy of the depth detection result.

Returning to Figure 2, if one frame is taken as the reference frame in the monocular video sequence, and the frames other than the reference frame are taken as the observation frame, and the object image extracted from the reference frame is called the reference object image, the observation frame The object image extracted from is called the observation object image, and the registration unit 202 registers the observation object image with the reference object image, so as to obtain corresponding registration parameters. Registration parameters generally include: a translation parameter representing the translation occurring between object images, a rotation parameter representing the rotation of the object image in a plane, and a scaling parameter representing the size change of the object image.

The registration unit 202 may perform registration processing according to various methods known in the art. For example, you can refer to "An Iterative Image Registration Technique Using a Scale-Space Model" published by J. Lee, S.S. Young, R. Gutierrez-Osuna in Technical Report, CSE Department, Texas A&M University, 2011. In the embodiment of hand detection, considering the low resolution of the hand image, the region-based image registration method mentioned in the above literature can be used to directly match the pixel densities of the two images as a whole. This is achieved by embedding the dimension-space model in a nonlinear least-squares architecture.

After the registration unit 202 obtains the registration parameters of the observation object image and the reference object image, the feature extraction unit 203 uses the registration parameters from the registration unit 202 to extract Ratio features that vary in depth relative to the imaging plane.

In one embodiment, the feature extraction unit 203 may directly extract the scaling parameter S reflecting the size change of the object image among the registration parameters, as a feature reflecting the depth change of the observed object image relative to the imaging plane compared with the reference object image.

In the case of using the scaling parameter S as a feature reflecting the depth change, the depth determination unit 204 can determine the depth change of the observation object image compared with the reference object image in such a way: when S>1+ε _a , determine the target The object becomes shallow relative to the imaging plane; when S<1- _εb , it is determined that the target object becomes dark relative to the imaging plane; and, otherwise, it is determined that the depth change of the target object relative to the imaging plane is indeterminate. Here, ε _a and ε _b are small positive numbers serving as certain margins, and may take the same or different values. ε _a and ε _b can be taken as 0.05 or 0.1, for example, depending on specific design requirements.

For example: in the case of taking ε _a and ε _b as 0.1, when the scaling parameter S=1.15, the depth determination unit 204 determines the depth change as "shallow"(S>1+ε _a =1.1); when scaling When the parameter S=0.8, the depth determination unit 204 determines the depth change as "deepening" (at S<1- _εb =0.9); when the scaling parameter S=1.05, the depth determination unit 204 determines the depth change as "not OK"(0.9<S<1.1).

In another embodiment, instead of directly extracting the scaling parameter S as a feature to determine the depth change, the feature extraction unit 203 can align the image of the object to be determined according to the translation parameter in the registration parameters from the registration unit 202, and then The edge parts of the aligned images are analyzed to extract corresponding features.

FIG. 4 is a block diagram showing the configuration of a feature extraction unit 400 as one example of the feature extraction unit 203 . The feature extraction unit 400 may include an alignment unit 401 and a histogram generation unit 402 . The alignment unit 401 aligns the image of the observation object with the image of the reference object according to the translation parameter in the registration parameters. The histogram generation unit 402 generates a histogram of directional gradients for edge portions of the aligned object images as features reflecting depth changes. The histogram generation unit 402 may use any method known in the art to extract features of the edge portion of the alignment object image to generate a corresponding histogram.

See Figure 5. The left side of Fig. 5 exemplarily shows the image obtained after the alignment unit 401 aligns the observation object image and the reference object image according to the translation parameters. Since the reference frame and the observation frame are adjacent frames (or closely spaced frames) in this example, the respective edges of the observation and reference object images cannot be observed separately in the aligned image. However, by using the histogram generation unit 402 to perform feature extraction on the edge part of the aligned image and generate a corresponding directional gradient histogram, the size relationship between the observed and reference object images can be clearly distinguished.

For example, the histogram (a) in Figure 5 schematically shows a histogram in which the image of the observed object is closer ("lightened") to the imaging plane than the image of the reference object; histogram (b) in Figure 5 schematically shows A histogram showing that the observed object image is farther ("darkened") from the imaging plane than the reference object image. From the comparison of (a) and (b), it can be seen that the generated feature histograms will have obvious differences when the image of the observed object is shallower or darker relative to the imaging plane compared with the image of the reference object. Please note: the histogram shown in Figure 5 is schematic. Depending on the method used to extract the features of the edge, different forms of histograms can be obtained, as long as these histograms can reflect different depth changes. Can.

The depth determining unit 204 may determine the depth variation of the object image between frames according to the histogram provided by the histogram generating unit 402 .

In application scenarios such as human-computer interaction, it is not enough to only determine the depth change in two frames of a video image sequence, but it is necessary to determine the motion of the target object in a specific time period, that is, a specific sequence of frames relative to the imaging plane. The direction is determined. Then, a corresponding operation is triggered according to the moving direction of the target object. For example, taking the hand as an example of the target object, when it is determined that the movement direction of the hand is moving in a direction close to the imaging plane of the camera (that is, the hand "pushes forward"), a specific function can be turned on; and when it is determined that the direction of the hand is moving away from the camera Certain functions are turned off when the direction of the imaging plane is moved (i.e. the hand is "pulled back"). In order to realize this interactive function, in some embodiments, the image processing device further includes a motion direction identification unit for identifying the motion direction of the target object relative to the imaging plane.

FIG. 6 is a block diagram showing the structure of an image processing apparatus 600 according to an embodiment of the present invention. The image processing device 600 includes: an object extraction unit 601 , a registration unit 602 , a feature extraction unit 603 , a depth determination unit 604 and a motion direction identification unit 605 . Because the structures and functions of the object extraction unit 601, the registration unit 602, the feature extraction unit 603 and the depth determination unit 604 are respectively the same as the object extraction unit 201, the registration unit 202, the feature extraction unit 203 and the depth determination unit explained in FIG. 2 The structures and functions of 204 are the same, so repeated descriptions thereof are omitted here, and only the motion direction identification unit 605 is described.

The motion direction identifying unit 605 can identify the motion direction of the target object relative to the imaging plane. For example, the image processing device 600 is made to sequentially perform depth determination on observation frames within a predetermined period, so as to obtain a depth determination result for each observation frame. Then, the motion direction identification unit 605 may identify the motion direction according to the depth determination result sequence of the observation frame input from the depth determination unit 604 .

It should be noted that: for the predetermined time period, one frame may be taken as the reference frame among the video frames within the time period. For example, the first frame or the last frame within the period may be taken as the reference frame but not limited to. Then, frames other than the reference frame within the predetermined period are used as observation frames. In addition, it is also possible to arbitrarily take a frame outside the period as the reference frame, for example, a frame immediately before the period. Then, all the frames within the predetermined period are taken as observation frames. This can be determined based on design needs. Although it is also possible to take a frame in the middle of the video sequence corresponding to the predetermined period as the reference frame, generally, for the sake of convenience, the video frame at one end of the observed frame sequence is usually taken as the reference frame. The following example illustrates an embodiment of the recognition performed by the motion direction recognition unit 605 .

In one embodiment, the earliest frame within a predetermined period is taken as a reference frame, and other subsequent frames are used as observation frames, and the image processing device 600 is made to sequentially perform depth determination processing on each observation frame. Since the above has been fully described, the description of the processing of object extraction, registration, feature extraction, depth determination, etc. performed on the observation frame is omitted here. The depth determination unit 604 provides the obtained depth determination results of each observed frame to the motion direction identification unit 605 . The motion direction identifying unit 605 identifies the motion directions according to the obtained sequence of depth determination results for each observation frame.

For example, when more than n consecutive occurrences of "deepening" appear in the sequence of depth determination results, the motion direction recognition unit recognizes that the target object moves in a direction away from the imaging plane. For example, a hand pulling back movement is recognized. However, when more than m "shallows" appear consecutively in the sequence of depth determination results, the motion direction recognition unit recognizes that the target object moves in a direction close to the imaging plane. For example, a forward thrust movement of a hand is recognized. Wherein, n and m are preset positive integers, which can take the same or different values. The sizes of n and m can be determined according to design requirements.

In some embodiments where the scaling parameter S is used for depth determination, the moving direction of the target object can also be identified according to both the depth determination result and the scaling parameter S. For example, when there are more than K "darkening" in the result sequence from the depth determination unit 604, and the scaling parameters S _i (i=1,...,K ) is less than the predetermined threshold TH1, the motion direction identification unit 605 may identify that the target object is moving in a direction away from the imaging plane. In addition, when there are more than L "shallow" consecutively in the result sequence from the depth determination unit 604, and the scaling parameters S _j corresponding to these L "shallow" observation object images (j=1,...,L ) is greater than the predetermined threshold TH2, the motion direction identification unit 605 may identify that the target object is moving in a direction close to the imaging plane. Wherein, K and L are preset positive integers, which can take the same or different values. The sizes of K and L and TH1 and TH2 can be determined according to design requirements.

The movement direction of the target object relative to the imaging plane is identified by combining the number of consecutive occurrences of specific determination results and the scaling parameters corresponding to these consecutive occurrences, which improves the accuracy of the identification result. In addition, relatively slow and insignificant motion of the moving object relative to the imaging plane can be excluded by controlling the magnitudes of the predetermined thresholds TH1 and TH2.

The image processing method used by the image processing device according to the embodiment of the present disclosure will be described below with reference to FIG. 7 and FIG. 8 .

Fig. 7 is a flowchart illustrating an image processing method for detecting the depth of a target based on a monocular video sequence according to an embodiment of the present disclosure.

In step S701, an image of a target object is extracted from a frame of a monocular video sequence as an object image. Various image extraction methods known in the art can be used, as long as the image of the target object can be identified and separated from the video frame to meet the needs of subsequent processing.

In one embodiment, multi-sized sliding windows may be used to detect the object image, so as to determine the region where the object image is located. Specifically, a sliding window of a specific size can be used to scan video frames, and by performing feature extraction on the image content in the sliding window and feeding the extracted features into a classifier to determine whether there is an object image in the sliding window. After scanning the entire frame with the sliding window, the size of the sliding window is adjusted, and then the above steps of scanning, extracting and determining are repeated until the region where the object image is located is determined.

The classifier used can be constructed and trained using any conventional features. In some embodiments, in order to detect the target object more accurately, a trained target object detector may be used to detect the object image. By using standard learning machine techniques, such as support vector machines, trained object detectors are able to detect object images more accurately.

In some embodiments, in order to improve the accuracy of the subsequent depth change detection processing, segmentation processing can also be performed on the region where the detected object image is located (for example, the window where the object image is detected), so as to use the object image as the foreground and the object image. The background region in the region is separated. Segmentation can be performed using various methods commonly used in the art. For example, in an embodiment where a hand is used as a target object, a skin color model may be constructed to perform foreground and background separation on detected windows containing hands. Segmenting the hand image can effectively reduce the noise introduced into subsequent depth detection processing, thereby improving the accuracy of the depth detection result.

Take a frame in the sequence of video frames as a reference frame, and use frames other than the reference frame as observation frames. The object images extracted from the reference frame and the observation frame are referred to as a reference object image and an observation object image, respectively. Then in step S702, the image of the observation object is registered to the image of the reference object, so as to obtain corresponding registration parameters. The registration process can be performed according to various methods known in the art. I won't go into details here.

After the registration parameters of the observation object image and the reference object image are obtained, in step S703, features that can reflect the depth change of the observation object image and the reference object image relative to the imaging plane are extracted by using the obtained registration parameters.

In one embodiment, the scaling parameter S reflecting the size change of the object image among the registration parameters may be directly extracted as a feature reflecting the depth change of the observed object image relative to the imaging plane compared with the reference object image.

In the case of taking the scaling parameter S as a feature reflecting the depth change, the depth change of the observed object image compared with the reference object image can be determined in such a way: when S>1+ε _a , it can be determined that the target object is relative to The imaging plane becomes shallow; when S<1- _εb , it can be determined that the target object becomes darker relative to the imaging plane; and, otherwise, it can be determined that the depth change of the target object relative to the imaging plane is indeterminate. Here, ε _a and ε _b are small positive numbers serving as certain margins, and may take the same or different values. ε _a and ε _b can be taken as 0.05 or 0.1, for example, depending on specific design requirements.

In another embodiment, instead of directly extracting the scaling parameter S as a feature to determine the depth change, the image of the object to be determined can be aligned according to the translation parameter in the registration parameter, and then the edge part of the aligned image is analyzed, to extract the corresponding features.

For example, the image of the observed object can be aligned with the image of the reference object according to the translation parameter in the registration parameters. Then, a histogram of oriented gradients is generated for the edge part of the aligned object images as a feature reflecting the depth variation. Any method known in the art may be used to extract features of the edge portion of the alignment object image to generate a corresponding histogram. Usually, by performing feature extraction on the edge parts of the aligned images and generating corresponding histograms of oriented gradients, the size relationship between the observed and reference object images can be clearly distinguished.

In step S704, the depth variation of the object image between frames may be determined according to the histogram obtained in step S703. Then, the processing ends.

According to the depth detection method of the embodiment shown in FIG. 7 , it is possible to detect the depth of an object in a video in real time while maintaining a low computing load.

In application scenarios such as human-computer interaction, it is sometimes necessary to determine the motion direction of the target object in a specific sequence of frames relative to the imaging plane, so as to trigger corresponding operations according to the motion direction of the target object. For example, taking the hand as an example of the target object, when it is determined that the movement direction of the hand is moving in a direction close to the imaging plane of the camera (that is, the hand "pushes forward"), a specific function can be turned on; and when it is determined that the direction of the hand is moving away from the camera Certain functions are turned off when the direction of the imaging plane is moved (i.e. the hand is "pulled back").

FIG. 8 is a flowchart illustrating a method for identifying a moving direction of a target object relative to an imaging plane according to an embodiment of the present disclosure. Since the processing of steps S801 to S804 in FIG. 8 is the same as the processing of steps S701 to S704 described in conjunction with FIG. 7 , repeated description will not be repeated here, and only step S805 will be described.

In step S805, the moving direction of the target object relative to the imaging plane is identified. For example, the processing in steps S801 to S804 may be sequentially performed on the observation frames within a predetermined period, so as to obtain a depth determination result for each observation frame. Then, in step S805, the motion direction may be identified according to the obtained depth determination result sequence of the observation frame.

It should be noted that: for the predetermined time period, one frame may be taken as the reference frame among the video frames within the time period. For example, the first frame or the last frame within the period may be taken as the reference frame but not limited to. Then, frames other than the reference frame within the predetermined period are used as observation frames. In addition, it is also possible to arbitrarily take a frame outside the period as the reference frame, for example, a frame immediately before the period. Then, all the frames within the predetermined period are taken as observation frames. This can be determined based on design needs. Although it is also possible to take a frame in the middle of the video sequence corresponding to the predetermined period as the reference frame, generally, for the sake of convenience, the video frame at one end of the observed frame sequence is usually taken as the reference frame. The following example illustrates the embodiment of identifying the motion direction.

For example, when more than n consecutive occurrences of "deepening" appear in the sequence of depth determination results, it can be identified that the target object is moving in a direction away from the imaging plane. For example, a hand pulling back movement is recognized. However, when more than m "shallows" appear consecutively in the sequence of depth determination results, it can be identified that the target object is moving in a direction close to the imaging plane. For example, a forward thrust movement of a hand is recognized. Wherein, n and m are preset positive integers, which can take the same or different values. The sizes of n and m can be determined according to design requirements.

In some embodiments where the scaling parameter S is used for depth determination, the moving direction of the target object can also be identified according to both the depth determination result and the scaling parameter S. For example, when there are more than K "darkening" consecutively in the determination result sequence obtained in step S804, and the scaling parameters S _i (i=1,...,K ) is less than the predetermined threshold TH1, it can be identified that the target object is moving in a direction away from the imaging plane. In addition, more than L "shallows" appear consecutively in the depth determination result sequence, and the continuous product of the scaling parameters S _j (j=1,...,L) corresponding to these L "shallowed" observation object images is greater than When the threshold TH2 is predetermined, the motion direction identification unit 605 may identify that the target object is moving in a direction close to the imaging plane. Wherein, K and L are preset positive integers, which can take the same or different values. The sizes of K and L and TH1 and TH2 can be determined according to design requirements.

Please note: Although one frame in the video sequence is taken as the reference frame in the above embodiment, and other frames are used as the observation frame to register with the reference frame, it is also possible to use such a method: that is, the previous frame in the video sequence One frame is used as a reference frame, the next frame is used as an observation frame, and so on. According to design needs to decide.

Hereinafter, an exemplary structure of a computer implementing the data processing device of the present invention is described with reference to FIG. 9 . FIG. 9 is a block diagram showing an exemplary structure of a computer implementing the present invention.

In FIG. 9 , a central processing unit (CPU) 901 executes various processes according to programs stored in a read only memory (ROM) 902 or programs loaded from a storage section 908 to a random access memory (RAM) 903 . In the RAM 903, data required when the CPU 901 executes various processes is also stored as necessary.

The CPU 901 , ROM 902 , and RAM 903 are connected to each other via a bus 904 . An input/output interface 905 is also connected to the bus 904 .

The following components are connected to the input/output interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 908 , including a hard disk, etc.; and a communication section 909, including a network interface card such as a LAN card, a modem, and the like. The communication section 909 performs communication processing via a network such as the Internet.

A drive 910 is also connected to the input/output interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read therefrom is installed into the storage section 908 as necessary.

In the case of implementing the above-described steps and processing by software, the programs constituting the software are installed from a network such as the Internet or a storage medium such as the removable medium 911 .

Those skilled in the art should understand that such a storage medium is not limited to the removable medium 911 shown in FIG. 9 in which the program is stored and distributed separately from the method to provide the program to users. Examples of the removable medium 911 include magnetic disks, optical disks (including compact disk read-only memory (CD-ROM) and digital versatile disk (DVD)), magneto-optical disks (including mini disks (MD) and semiconductor memories. Alternatively, the storage medium may be ROM 902 , a hard disk or the like contained in the storage section 908, in which programs are stored, and are distributed to users together with a method containing them.

In the foregoing specification, the invention has been described with reference to specific embodiments. However, those of ordinary skill in the art understand that various modifications and changes can be made without departing from the scope of the present invention as defined in the claims.

The present invention can also be realized in the following embodiments:

1. An image processing device, comprising:

The object extraction unit is used to extract the image of the target object from the frame of the monocular video sequence as the object image;

A registration unit, configured to register the observation object image with the reference object image to obtain registration parameters, the reference object image and the observation object image are object images extracted from the reference frame and the observation frame other than the reference frame, respectively;

a feature extraction unit for extracting a feature capable of reflecting a depth change of the observed object image with respect to the imaging plane compared with the reference object image by using the registration parameter; and

A depth determination unit is configured to determine a depth change based on the extracted features.

2. The image processing apparatus according to item 1, wherein the feature extraction unit extracts, as a feature, a scaling parameter reflecting a change in size of the subject image among the registration parameters.

3. The image processing device according to item 2, wherein, denoting the scaling parameter of the observation frame with respect to the reference frame by S, then the depth determination unit:

When S>1+ _εa , it is determined that the target object becomes shallower relative to the imaging plane;

When S<1- _εb , it is determined that the target object becomes darker relative to the imaging plane; and

Otherwise, it is determined that the depth variation of the target object relative to the imaging plane is uncertain;

Here, ε _a and ε _b are small positive numbers as a certain margin.

4. The image processing device according to any one of items 1 to 3, further comprising a motion direction identification unit for identifying the motion direction of the target object relative to the imaging plane;

Wherein, when the image processing device is made to sequentially perform depth determination on the observation frames within a predetermined period, the motion direction identification unit can identify the motion direction according to the result sequence determined by the depth determination unit.

5. The image processing device according to item 4, wherein the motion direction recognition unit:

identifying motion of the target object in a direction away from the imaging plane when there are more than a first predetermined number of consecutive "darkenings" in the resulting sequence; and

When there are more than a second predetermined number of consecutive "shallows" in the resulting sequence, it is identified that the target object is moving in a direction closer to the imaging plane.

6. The image processing device according to item 4, wherein the motion direction recognition unit:

When there is a continuous third predetermined number or more of "deepening" in the result sequence, and the continuous product of the corresponding scaling parameters S is less than the first threshold, it is identified that the target object is moving in a direction away from the imaging plane; and

When there are more than a fourth predetermined number of consecutive "shallows" in the result sequence, and the product of the corresponding scaling parameters S is greater than the second threshold, it is identified that the target object is moving in a direction close to the imaging plane.

7. The image processing device according to item 1, wherein the feature extraction unit comprises:

an alignment unit, configured to align the image of the observation object with the image of the reference object according to the translation parameters in the registration parameters; and

The histogram generating unit is configured to generate a histogram of directional gradients for the edge portion of the aligned object image as a feature capable of reflecting depth changes.

8. The image processing apparatus according to any one of items 1 to 7, wherein the object extraction unit includes a detection unit configured to detect the object image using a multi-sized sliding window to determine the region where the object image is located.

9. The image processing apparatus according to item 8, wherein the extracting unit includes a segmentation unit for performing segmentation processing on the region where the object image is located to extract the object image.

10. The image processing device according to item 8 or 9, wherein the detection unit detects the object image using a trained target object detector.

11. An image processing method, comprising:

Extract the image of the target object from the frame of the monocular video sequence as the object image;

Registering the observation object image with the reference object image to obtain registration parameters, the reference object image and the observation object image are object images extracted from the reference frame and the observation frame other than the reference frame, respectively;

extracting features that reflect changes in depth of the image of the observed object relative to the imaging plane compared to the image of the reference object by utilizing the registration parameters; and

Depth variation is determined based on the extracted features.

12. The image processing method according to item 11, wherein, among the registration parameters, a scaling parameter reflecting a size change of the subject image is extracted as a feature.

13. The image processing method according to item 12, wherein, using S to represent the scaling parameter of the observation frame for the reference frame, then:

When S>1+ _εa , it is determined that the target object becomes shallower relative to the imaging plane;

When S<1- _εb , it is determined that the target object becomes darker relative to the imaging plane; and

Otherwise, it is determined that the depth variation of the target object relative to the imaging plane is uncertain;

Here, ε _a and ε _b are small positive numbers as a certain margin.

14. The image processing method according to any one of items 11 to 13, further comprising: identifying a motion direction of the target object relative to the imaging plane based on a sequence of depth determination results sequentially obtained for observation frames within a predetermined period.

15. The image processing method according to item 14, wherein,

identifying motion of the target object in a direction away from the imaging plane when there are more than a first predetermined number of consecutive "darkenings" in the resulting sequence; and

When there are more than a second predetermined number of consecutive "shallows" in the resulting sequence, it is identified that the target object is moving in a direction closer to the imaging plane.

16. The image processing method according to item 14, wherein,

When there is a continuous third predetermined number or more of "deepening" in the result sequence, and the continuous product of the corresponding scaling parameters S is less than the first threshold, it is identified that the target object is moving in a direction away from the imaging plane; and

When there are more than a fourth predetermined number of consecutive "shallows" in the result sequence, and the product of the corresponding scaling parameters S is greater than the second threshold, it is identified that the target object is moving in a direction close to the imaging plane.

17. The image processing method according to item 11, wherein extracting features reflecting depth changes comprises:

aligning the image of the observed object with the image of the reference object according to the translation parameter in the registration parameters; and

A histogram of oriented gradients is generated for the edge part of the aligned object image as a feature that can reflect the depth variation.

18. The image processing method according to any one of items 11 to 17, wherein extracting the object image comprises: detecting the object image using multi-sized sliding windows to determine the region where the object image is located.

19. The image processing method according to item 18, wherein extracting the target image comprises: performing segmentation processing on the region where the target image is located to extract the target image.

20. The image processing method according to item 18 or 19, wherein extracting the object image comprises detecting the object image using a trained target object detector.

Claims (10)

1. An image processing device, comprising: The object extraction unit is used to extract the image of the target object from the frame of the monocular video sequence as the object image; A registration unit, configured to register the observation object image with the reference object image to obtain registration parameters, the reference object image and the observation object image are respectively extracted from the reference frame and the observation frame other than the reference frame object image of a feature extraction unit for extracting a feature capable of reflecting a depth change of the observed object image relative to the imaging plane compared with the reference object image by using the registration parameters; and a depth determination unit, configured to determine the depth variation based on the extracted features. 2. The image processing apparatus according to claim 1, wherein the feature extraction unit extracts, as the feature, a scaling parameter reflecting a size change of an object image among the registration parameters. 3. The image processing device according to claim 2, wherein, using S to represent the scaling parameter of the observation frame for the reference frame, then the depth determination unit: When S>1+ _εa , it is determined that the target object becomes shallower relative to the imaging plane; When S<1- _εb , it is determined that the target object becomes darker with respect to the imaging plane; and Otherwise, it is determined that the depth change of the target object relative to the imaging plane is uncertain; Here, ε _a and ε _b are small positive numbers as margins for the determination. 4. The image processing device according to any one of claims 1 to 3, further comprising a motion direction identification unit configured to identify the motion direction of the target object relative to the imaging plane; Wherein, when the image processing device is made to sequentially perform depth determination on the observation frames within a predetermined period of time, the motion direction identification unit can identify the motion direction according to the sequence of results determined by the depth determination unit. 5. The image processing device according to claim 4, wherein the motion direction identification unit: identifying that the target object is moving in a direction away from the imaging plane when there are more than a first predetermined number of consecutive "darkenings" in the sequence of results; and The target object is identified as moving in a direction closer to the imaging plane when there are more than a second predetermined number of "shallows" in succession in the sequence of results. 6. An image processing method, comprising: Extract the image of the target object from the frame of the monocular video sequence as the object image; Registering the observation object image with respect to the reference object image to obtain registration parameters, the reference object image and the observation object image are object images extracted from the reference frame and the observation frame other than the reference frame, respectively; extracting features capable of reflecting depth changes in the image of the observed object relative to the imaging plane compared with the image of the reference object by utilizing the registration parameters; and The depth variation is determined based on the extracted features. 7. The image processing method according to claim 6, wherein, among the registration parameters, a scaling parameter reflecting a size change of an object image is extracted as the feature. 8. The image processing method according to claim 7, wherein, using S to represent the scaling parameter of the observed frame for the reference frame, then: When S>1+ _εa , it is determined that the target object becomes shallower relative to the imaging plane; When S<1- _εb , it is determined that the target object becomes darker with respect to the imaging plane; and Otherwise, it is determined that the depth change of the target object relative to the imaging plane is uncertain; Here, ε _a and ε _b are small positive numbers as margins for the determination. 9. The image processing method according to any one of claims 6 to 8, further comprising: identifying the target object relative to the The direction of motion of the imaging plane. 10. The image processing method according to claim 9, wherein, identifying that the target object is moving in a direction away from the imaging plane when there are more than a first predetermined number of consecutive "darkenings" in the sequence of results; and The target object is identified as moving in a direction closer to the imaging plane when there are more than a second predetermined number of "shallows" in succession in the sequence of results.