JP2021174537A - Image processing device, image processing method, and machine-readable storage medium - Google Patents

Abstract

To provide an image processing device, an image processing method, and a machine-readable storage medium.SOLUTION: An image processing device includes a detection unit configured to detect 2D key points relating to a plurality of objects in an area including the plurality of objects in an image, in which the objects partially overlap with each other; a candidate key point set selection unit configured to select a candidate key point set for one object of the plurality of objects from the 2D key points; an estimation unit configured to estimate 3D models of the one object based on the candidate key point set; a 3D model selection unit configured to select an optimized 3D model from among the estimated 3D models; and an acquisition unit configured to project the optimized 3D model onto a plane of the image to obtain the one object separated.SELECTED DRAWING: Figure 1

Description

The present invention relates to the technical field of image processing, and more particularly to an image processing apparatus for object separation, an image processing method, and a machine-readable storage medium.

Analysis of objects in video or images (eg, human hand, human body, etc.) is one of the important tasks in the field of computer vision. The task of object separation is very difficult and can affect the results of the analysis, as multiple objects in a video or image often overlap each other.

For example, in hand motion analysis, as one of the commonly used methods, first, the position and posture information of each hand in a video frame, for example, a segmentation area, a hand key point, etc. are acquired. After that, features and design rules may be acquired to classify the operation. However, in many practical applications, the operation waiting for analysis needs to be completed in cooperation with two hands. Therefore, in the captured video frame, the two hands often overlap each other, which makes it difficult to separate the hands, and even if the corresponding areas of both hands can be accurately and completely segmented, the hands Is still difficult to separate.

In the conventional method, an attempt was made to solve such a problem by using the hand key point information. Based on the idea of "bottom (bottom) to top", these methods first detect keypoints in the hand area and then 1 corresponding keypoints for each hand according to some policies. Divide into groups. Generally speaking, the keypoint detection results are relatively accurate, but the appearance of the two hands is so similar that no matter what policy is adopted, the keypoint division results will be. It's unsatisfactory, in other words, it doesn't separate the two hands in the image well.

Therefore, how to accurately separate an object in an image containing a plurality of objects, for example, how to accurately separate one hand in an image containing a plurality of human hands is an important research subject in this field. one of.

An object of the present invention is to provide an image processing apparatus, an image processing method, and a machine-readable storage medium capable of separating one object in an image containing a plurality of objects.

According to one aspect of the invention, an image processing apparatus is provided, which
A detection unit that detects two-dimensional 2D key points related to the plurality of objects in a region including a plurality of objects in an image, of which the plurality of objects partially overlap;
Candidate key point set selection unit that selects a candidate key point set for one of the plurality of objects from the 2D key points;
An estimation unit that estimates a 3D model of the one object based on the candidate key point set;
A 3D model selection unit that selects an optimized 3D model from among the estimated 3D models; and an acquisition unit that obtains the one object separated by projecting the optimized 3D model onto the plane of the image. ..

According to another aspect of the invention, an image processing method is provided, which is:
A step of detecting a two-dimensional 2D key point relating to the plurality of objects in a region including a plurality of objects in an image, wherein the plurality of objects partially overlap.
A step of selecting a candidate key point set for one of the plurality of objects from the 2D key points;
A step of estimating a 3D 3D model of the one object based on the candidate key point set;
It includes a step of selecting an optimized 3D model from the estimated 3D models; and a step of projecting the optimized 3D model onto the plane of the image to obtain the one object separated.

According to another aspect of the present invention, a machine-readable storage medium is provided, in which a program product in which a machine-readable command code is stored is carried, in which the command code is stored by a computer. When read and executed, the computer can realize the image processing method according to the present invention.

By constructing a 3D model for one of a plurality of objects included in an image by using the image processing apparatus, the image processing method, and the machine-readable storage medium according to the present invention, the image including the plurality of objects can be selected. One object can be separated.

It is a block diagram of the structure of the image processing apparatus which concerns on embodiment of this invention. It is a block diagram of the structure of the image processing apparatus which concerns on other Examples of this invention. It is a block diagram of the structure of the 3D model selection unit in the image processing apparatus which concerns on embodiment of this invention. It is a block diagram of the structure of the image processing apparatus which concerns on other Examples of this invention. It is a flowchart of the image processing method which concerns on embodiment of this invention. It is a flowchart of the image processing method which concerns on other Examples of this invention. It is a block diagram of an exemplary configuration of a general-purpose personal computer that can realize the image processing apparatus and method according to the embodiment of the present invention.

Hereinafter, preferred embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. It should be noted that such an embodiment is merely an example and does not limit the present invention.

In addition, the subject referred to in the text usually has a plurality of joint parts, and the plurality of joint parts can form different postures by performing flexible relative movements. Specifically, the target is a human hand, a human body, or the like. For example, with respect to one human hand, the plurality of joint sites may include a metacarpophalangeal joint, an interphalangeal joint, a radial carpal joint, and the like. Further, for example, with respect to the human body, the plurality of joint parts may include a shoulder joint, an elbow joint, a wrist joint, a hip joint, a knee joint, an ankle joint, and the like of the human body. Also, the key points mentioned in the text usually correspond to the plurality of joint parts described above, but the key points may correspond to all or a part of all the plurality of joint parts of interest. Also, keypoints may be referred to as 2D keypoints in video frames or images. The 3D model of the target can be constructed by the 2D key points of the target, and the 3D model can be limited by the relative positions of the plurality of joint parts of the target in space. Further, the key point may be referred to as a 3D key point in the 3D model.

Hereinafter, with reference to FIG. 1, how the image processing apparatus according to the embodiment of the present invention separates one object in an image including a plurality of objects will be described.

FIG. 1 is a block diagram of the configuration of the image processing apparatus 100 according to the embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 100 according to the embodiment of the present invention may include a detection unit 110, a candidate key point set selection unit 120, an estimation unit 130, a 3D model selection unit 140, and an acquisition unit 150.

The detection unit 110 can detect 2D key points relating to a plurality of objects in a region including the plurality of objects in the image. In the image, multiple objects partially overlap each other. Here, when a plurality of objects partially overlap each other, each object of the plurality of objects that need to be separated in the image and at least one of the other objects partially overlap each other. Point to that.

Further, the candidate key point set selection unit 120 can select a candidate key point set for one of a plurality of targets from the 2D key points. Here, the candidate key point set indicates a set of 2D key points that can estimate a 3D model of a specific target.

Further, the estimation unit 130 can estimate the 3D model of the one object based on the candidate key point set.

Further, the 3D model selection unit 140 can select an optimized 3D model from the estimated 3D models.

In addition, the acquisition unit 150 can project the optimized 3D model onto the plane of the image to obtain the one object separated.

As a result, the image processing apparatus 100 according to the embodiment of the present invention constructs a 3D model for one object among the plurality of objects included in the image, thereby forming the one object in the image including the plurality of objects. The object can be separated.

Preferably, the detection unit 110 can detect a plurality of target 2D key points by a method of deep learning, and the candidate key point set selection unit 120 is for a candidate key point set based on the detection result of deep learning. 2D key points can be selected. For example, the position of each target key point can be determined based on the heat map output by the neural network. Also, since a plurality of objects in the image partially overlap each other, the key points of the plurality of objects may also overlap each other, in other words, the key point at a certain position may belong to the plurality of objects. be. Therefore, it is necessary to generate the target candidate key point set by appropriately selecting the target key point. For example, when separating two objects, if there are two detection results in the entire area containing multiple objects for a 2D key point at a certain position, any one of them should be included in the generated set. If there is only one detection result, it can be included in the generated set, but it is not necessary to include it; if there is no detection result, the generated set has the key point of the corresponding position. Not included.

In addition, the candidate key point set selection unit 120 can further reduce the number of candidate key point sets by using geometric constraints on the one object. This makes it possible to remove irrational candidate keypoint sets and effectively reduce the number of candidate keypoint sets for objects that need to be separated.

Preferably, the estimation unit 130 can estimate the 3D model by performing deep learning with a neural network or by optimizing using a general-purpose model predefined for the one object. The process can be realized by a conventional method. Specifically, the object can be represented by a shape parameter and a posture parameter. The 3D model can be modeled with shape and orientation parameters corresponding to the pre-defined general purpose model and the concrete model, and then projected onto a video frame or image based on the camera parameters. All the above parameters can be obtained by calculating based on the method of deep learning, and can also be obtained by solving the optimization problem. After estimating the values of all parameters based on the candidate key point set of interest, the corresponding 3D model can be constructed.

Preferably, as shown in FIG. 2, the image processing apparatus 200 according to another embodiment of the present invention may further include a segmentation unit 260. The detection unit 210, the candidate key point set selection unit 220, the estimation unit 230, the 3D model selection unit 240, and the acquisition unit 250 shown in FIG. 2 are the detection unit 110, the candidate key point set selection unit 120, and the estimation unit 130 shown in FIG. For 3D model selection unit 140 and acquisition unit 150. Therefore, the detailed description thereof will be omitted here.

The segmentation unit 260 can segment each frame in the video, that is, an area containing a plurality of objects in the image. Segmentation of an area containing a plurality of objects can be realized by different methods. For example, the segmentation unit 260 can segment a region containing a plurality of objects in an image by a method of deep learning. Alternatively, the segmentation unit 260 can segment the region containing the plurality of objects by other methods, for example, based on the color characteristics of the portion containing the plurality of objects in the image.

For example, when segmenting the hand region within each frame in the video, if the main scene in the input video frame is the human hand, then the video frame will have skin color. The task of hand region segmentation can be completed by directly finding the region, and for more complex scenes, hand region segmentation can be achieved by adopting a method of deep learning.

Subsequently, the segmentation unit 260 can provide the detection unit 210 with an area including a plurality of segmented objects so that the two-dimensional key point can be detected.

Thereby, the image processing apparatus 200 according to the embodiment of the present invention can more accurately detect the 2D key points of the plurality of objects by extracting the area including only the plurality of objects that need to be separated. .. This can help separate one object from an image that contains multiple objects.

As a preferred example, FIG. 3 shows the configuration of a 3D model selection unit in the image processing apparatus according to the embodiment of the present invention. The 3D model selection unit 300 shown in FIG. 3 corresponds to the 3D model selection unit 140 shown in FIG. 1 and the 3D model selection unit 240 shown in FIG. As shown in FIG. 3, the 3D model selection unit 300 may include a first selection unit 310 and a second selection unit 320.

The first choice unit 310 can reduce the estimated number of 3D models due to geometric constraints on the one object that needs to be separated. Specifically, the first choice unit 310 can use some geometric constraints on each 3D model to determine if the model is reasonable. After that, the first selection unit 310 can delete an irrational 3D model among the 3D models.

In addition, the second-choice unit 310 can calculate the estimated effectiveness score of the 3D model based on at least one of the following, i.e. 1) in the candidate keypoint set. The distance between the 2D keypoint and the corresponding 3D keypoint in the 3D model at the projection position on the plane of the image; and 2) the projection of the estimated 3D model image on the plane and multiple objects. The area of overlap with the area.

Thereby, the 3D model selection unit 300 according to the embodiment of the present invention can select the optimized 3D model from the estimated 3D models.

In order to better understand the technical proposal of the present invention, the image processing apparatus of the present invention will be described in more detail below.

Here, the separation of the two human hands in the image is a preferred embodiment, the detection units 110 and 210, the candidate key point set selection units 120 and 220, the estimation units 130 and 230, and the 3D model selection units 140, 240. And 300, and the processing that can be performed by the acquisition units 150 and 250 will be described in detail. For convenience, in the following description, only the detection unit 110, the candidate key point set selection unit 120, the estimation unit 130, the 3D model selection unit 140, and the acquisition unit 150 may be used. Also, it should be understood that these processes can be similarly applied to the detection unit 210, the candidate key point set selection unit 220, the estimation unit 230, the 3D model selection units 240 and 300, and the acquisition unit 250.

First, the detection unit 110 detects 2D key points for two hands in the hand segmentation region in the image, and transmits the detected 2D key points to the candidate key point set selection unit 120. Can be done. For the detection of 2D hand key points, a deep learning method is usually adopted, and the position of each key point of the hand can be determined based on the heat map output by the neural network. When there are two hands in the hand segmentation area, a maximum of two detection results can be obtained for each key point.

After that, the candidate key point set selection unit 120 generates a candidate key point set corresponding to the same hand. If there are two detection results in the entire hand segmentation area for a key point at a certain position on the hand, any one of them can be included in the generated set; only one detection result In some cases, it can be included in the generated set, but it is not necessary; if there is no detection result, the generated set does not include the key points of the corresponding positions.

Therefore, if there are a total of N keypoints defined in advance for each hand, the generated candidate keypoint set corresponding to the same hand will contain a maximum of N keypoint detection results. included. Further, by limiting the minimum value of the number of key point detection results included in each set, the 3D hand model can be accurately estimated in the next step. Furthermore, an irrational set of key points can be removed based on the geometric constraints of the hand, for example, each finger described (represented) based on a 2D key point bends in only one direction. Constraints can be used.

Thereby, the candidate key point set selection unit 120 can acquire the candidate key point set corresponding to the same move and provide it to the estimation unit 130.

Subsequently, the estimation unit 130 can estimate the corresponding hand 3D model based on one given hand 2D candidate key point set. One concrete hand 3D model can be described and modeled based on the shape and orientation parameters corresponding to the pre-defined general hand model and the concrete hand model, and the camera. It can be projected onto a video frame based on the parameters. All the above parameters can be calculated by the method of deep learning, and can also be obtained by solving the optimization problem. After estimating the values of all parameters based on the hand 2D keypoint set, the corresponding hand 3D model can be constructed and the positions of N hand 3D keypoints can be calculated. can.

Subsequently, the 3D model selection unit 140 can select an optimized 3D model from the estimated hand 3D models and provide the optimized 3D model to the acquisition unit 150.

For example, the first choice unit 310 can determine if the model is reasonable based on some hand geometric constraints. The constraints that can be adopted may include the following: a) there should be no large range of overlap between different parts of the model; b) 3D key points on the same finger It should be co-planar; c) Each finger described based on a 3D keypoint bends in only one direction, and so on.

Subsequently, for example, the second selection unit 310 may define one effectiveness score for each rational hand 3D model, which score can be calculated based on information from the two aspects. On one side, the second selection unit 310 calculates the distance between each detected 2D keypoint and its corresponding 3D keypoint and the projected point in the video frame and corresponds to all detected keypoints. The smaller the average value of the above-mentioned distances, the higher the effectiveness score of the 3D model. In another aspect, the second selection unit 310 compares the entire hand segmentation area with the projected area in the video frame of the 3D hand model, and ideally the projected area is a subset of the entire hand segmentation area. Moreover, a part of the boundary between the projection area and the entire hand segmentation area overlaps. The larger the overlapping area of the two regions, the longer the boundary and the higher the effectiveness score of the 3D model. Finally, the second choice unit 310 determines the best 3D model based on the calculated effectiveness score.

The acquisition unit 150 then projects the optimized 3D model onto the plane of the video frame or image to acquire one separated hand. The optimized 3D hand model corresponds to one hand in the projected area on the video frame, and the other part of the entire hand segmentation area corresponds to the other hand. In such a way, the two hands in the video frame or image can be successfully separated.

As a result, according to the image processing apparatus according to the embodiment of the present invention, a 3D model has been introduced for hand separation. Similarly, based on the idea of "bottom (bottom) to top", first, all key points in the corresponding area of the two hands are detected. Subsequently, a three-dimensional hand model is constructed for the candidate key point set corresponding to each same hand, and the optimum one is selected from many models. The most dominant 3D hand model corresponds to one hand in the projected area on the video frame, and the other part of the entire hand area corresponds to the other hand. By using the valid information provided by the 3D model, more accurate hand separation results can be obtained.

Although the embodiment relates to the separation of two human hands, it can also be applied to the separation of two human bodies in an image. The difference is that it is necessary to convert the key point positions of both, the general-purpose model, etc. based on a specific case.

Hereinafter, the configuration of the image processing apparatus according to another embodiment of the present invention will be described with reference to FIG. The image processing device 400 shown in FIG. 4 includes a detection unit 410, a candidate key point set selection unit 420, an estimation unit 430, a 3D model selection unit 440, an acquisition unit 450, and a determination unit 460. The detection unit 410, the candidate key point set selection unit 420, the estimation unit 430, the 3D model selection unit 440 and the acquisition unit 450 are the detection units 110 and 210, the candidate key point set selection units 120 and 220, the estimation units 130 and 230, and 3D. Performs processing similar to model selection units 140, 240 and 300, and acquisition units 150 and 250. Therefore, the detailed description thereof will be omitted here. Further, the image processing device 400 in FIG. 4 may include a segmentation unit similar to the segmentation unit 260 in FIG.

As described with respect to the image processing apparatus 100 in FIG. 1, the acquisition unit 450 can separate the first object in an image including a plurality of objects (assuming there are n objects).

The number n of the plurality of objects can be determined as follows, that is, when the detection unit 410 detects 2D key points for the plurality of objects in the image, the detected key points for different key point positions. The number with the largest number of is determined as the target number. For example, if there are five human bodies in the image and the torso of the five human bodies block each other, but the heads do not block each other, then five key points with respect to the head position can be detected. In this case, if the number of detected key points relating to the positions of other parts of the body is less than 5, it can be determined that there are a total of 5 objects in the image. Alternatively, if the number of objects in the video frame or image for which motion analysis needs to be performed is determined in advance, the number of objects can be artificially specified.

Since this embodiment relates to the case of three or more objects as compared to the previous embodiment, the image processing apparatus 400 may further include a determination unit 460. It can determine if all the objects in the image have been separated. When the determination unit 460 determines that all the objects are not separated, the candidate key point set selection unit 420 can select the candidate key point set for the second object in the image. Further, the estimation unit 430 can estimate the 3D model of the second target based on the candidate key point set of the second target, and the 3D model selection unit 440 can estimate the 3D model of the second target from among the estimated 3D models. An optimized 3D model of the second object can be selected.

After that, the acquisition unit 450 is projected on the plane of the image of the optimized 3D model of the second object, and the part after removing (subtracting) the separated first object from the region containing the plurality of objects. By obtaining the intersection of, the second separated object can be obtained.

This allows the second object to be further separated in the image.

By such a method, the third object to the n-1th object can be continuously separated from the image, and finally, what is left in the area containing the plurality of objects is the nth object. .. In addition, the acquisition unit 450 is after the projection of the image of the optimized 3D model of the i-th object on the plane and the i-1 object separated before the projection from the region including the plurality of objects. By obtaining the intersection with the part, the separated i-th object can be obtained.

Thereby, the image processing apparatus 400 according to the embodiment of the present invention can separate a plurality of objects included in the image one by one.

Hereinafter, the image processing method according to the embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 5, the image processing method according to the embodiment of the present invention starts in step S110. In step S110, two-dimensional 2D key points relating to a plurality of objects are detected in a region including the plurality of objects in the image, and the plurality of objects partially overlap each other.

Subsequently, in step S120, a candidate key point set for one of the plurality of targets is selected from the 2D key points.

Subsequently, in step S130, a three-dimensional 3D model of the one object is estimated based on the candidate key point set.

Subsequently, in step S140, the optimized 3D model is selected from the estimated 3D models.

Subsequently, in step S150, the optimized 3D model is projected onto the plane of the image to acquire one separated object. Then complete the process.

According to the examples of the present invention, the method may further include segmenting a region containing a plurality of objects in an image by a method of deep learning.

According to the embodiment of the present invention, the 2D key points of a plurality of objects can be detected by the method of deep learning.

According to an embodiment of the present invention, the step of selecting a candidate key point set includes selecting a 2D key point for the candidate key point set based on the detection result of deep learning.

According to an embodiment of the present invention, the step of selecting a candidate key point set involves reducing the number of candidate key point sets using geometric constraints on one object.

According to the embodiment of the present invention, the estimation step of the 3D model is to estimate the 3D model by performing deep learning or optimization by a neural network using a predefined general-purpose model for one object. including.

According to an embodiment of the present invention, the step of selecting an optimized 3D model involves reducing the estimated number of 3D models using geometric constraints on one object.

According to an embodiment of the present invention, the step of selecting an optimized 3D model comprises calculating an estimated effectiveness score of the 3D model based on at least one of the following: 1) The distance between the 2D keypoint in the candidate keypoint set and the projected position of the image plane between the corresponding 3D keypoint in the 3D model; and 2) the estimated plane of the 3D model image. The area of overlap between the above projection and the area containing multiple objects.

As a result, according to the image processing method according to the embodiment of the present invention, by constructing a 3D model for one of a plurality of objects included in the image, the image including the plurality of objects is described as 1 Two objects can be separated.

Hereinafter, the image processing method according to another embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 6, the image processing method according to the embodiment of the present invention starts in step S210. In step S210, 2D key points relating to a plurality of objects are detected in a region including the plurality of objects in the image, and the plurality of objects partially overlap each other.

Subsequently, in step S220, a candidate key point set for one of a plurality of targets is selected from the 2D key points.

Subsequently, in step S230, the one target 3D model is estimated based on the candidate key point set.

Subsequently, in step S240, the optimized 3D model is selected from the estimated 3D models.

Subsequently, in step S250, one separated object is acquired by projecting the optimized 3D model onto the plane of the image. Specifically, the intersection of the projection of the image of the optimized 3D model of the object on the plane and the part after excluding all the objects separated before the region containing the plurality of objects is obtained. This makes it possible to obtain the separated object.

Subsequently, in step S260, it is determined whether all the objects have been separated. If it is determined that all the targets are not separated, the process returns to step S220, and the candidate key point set is continuously selected for the next one target. After that, the processing in steps S230, S240 and S250 is executed for the next one target.

In step S260, the process is terminated after it is determined that all the objects have been separated.

Thereby, the image processing method according to the embodiment of the present invention can separate a plurality of objects included in the image one by one.

Since various specific methods of carrying out the above-mentioned steps in the image processing method according to the embodiment of the present invention have already been described in detail above, the detailed description thereof will be omitted here.

Of course, each operation process of the image processing method of the present invention can also be realized by a computer-executable program stored in various machine-readable storage media.

Further, an object of the present invention may be realized by the following method, that is, a storage medium storing the above-mentioned executable program code is directly or indirectly provided to the system or device, and the system or device is provided. The computer or central processing unit (CPU) in the device reads and executes the above-mentioned program code. At this time, the embodiment of the present invention is not limited to a program as long as the system or device has a function capable of executing a program, and the program may be in any form, for example, an object-oriented program or an interpreter. It may be an executable program or a script program provided to the OS.

Further, the above-mentioned series of processes may be realized by software and / or firmware. When realized by software and / or firmware, a program constituting the software is installed on a computer having a dedicated hardware structure, for example, a general-purpose machine 1300 (for example, a computer) shown in FIG. 7 from a storage medium or a network. , The computer can perform various functions and the like when various programs are installed.

FIG. 7 is a block diagram of an exemplary configuration of a general-purpose personal computer capable of realizing the image processing apparatus and method according to the embodiment of the present invention.

In FIG. 7, the central processing unit (CPU) 1301 performs various processes based on the program stored in the ROM 1302 or the program rodged from the storage unit 1308 to the RAM 1303. The RAM 1303 can also store data and the like required when the CPU 1301 performs various processes according to the needs. The CPU 1301, ROM 1302 and RAM 1303 are connected to each other via bus 1304. The input / output interface 1305 is also connected to bus 1304.

Further, the following components are further connected to the input / output interface 1305, that is, an input unit 1306 including a keyboard and the like, an output unit including a display such as a liquid crystal display (LCD), and a speaker. 1307, a storage unit 1308 including a hard disk and the like, and a communication unit 1309 including a network interface card such as a LAN card and a modem. The communication unit 1309 performs communication processing via a network such as the Internet or LAN.

Drive 1310 may be connected to input / output interface 1305, if desired. The removable medium 711, for example, a semiconductor memory, is set in the drive 1310 as needed, and the computer program read from the medium can be installed in the storage unit 1308.

The present invention further relates to a computer storage medium in which the program is stored, the program being able to realize the method of the above-described embodiment when executed.

The present invention also provides a program product that includes a machine-readable command code. When such a command code is read and executed by the machine, the method according to the embodiment of the present invention described above can be executed. Correspondingly, carry such program products, such as magnetic disks (including floppy disks (registered trademarks)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (MD (registered trademarks)). ), And various storage media such as semiconductor storage devices are also included in the present invention.

The above-mentioned storage medium may include, but is not limited to, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor storage device, and the like.

Further, each operation (process) in the above method can be realized by a method of a computer-executable program stored in various machine-readable storage media.

In addition, the above examples will be further disclosed as additional notes as follows.

(Appendix 1)
It is an image processing device
A detection unit that detects two-dimensional 2D key points relating to the plurality of objects in a region including a plurality of objects in an image, wherein the plurality of objects partially overlap each other;
Candidate key point set selection unit that selects a candidate key point set for one of the plurality of objects from the 2D key points;
An estimation unit that estimates a 3D model of the one object based on the candidate key point set;
A 3D model selection unit that selects an optimized 3D model from among the estimated 3D models; and an acquisition unit that projects the optimized 3D model onto the plane of the image to obtain the one object separated. Device.

(Appendix 2)
The image processing apparatus according to Appendix 1.
An apparatus further comprising a segmentation unit for segmenting the region containing the plurality of objects in the image by a method of deep learning.

(Appendix 3)
The image processing apparatus according to Appendix 1.
The detection unit is a device that detects a plurality of target 2D key points by a method of deep learning.

(Appendix 4)
The image processing apparatus according to Appendix 3,
The candidate key point set selection unit is a device that selects a 2D key point for the candidate key point set based on the detection result of deep learning.

(Appendix 5)
The image processing apparatus according to Appendix 1.
The candidate key point set selection unit is a device that reduces the number of the candidate key point sets by using geometric constraints on the one object.

(Appendix 6)
The image processing apparatus according to Appendix 1.
The estimation unit is an apparatus that estimates the 3D model by performing deep learning or optimization by a neural network using a general-purpose model defined in advance for the one object.

(Appendix 7)
The image processing apparatus according to Appendix 1.
The 3D model selection unit is a device that reduces the estimated number of 3D models using geometric constraints on the one object.

(Appendix 8)
The image processing apparatus according to Appendix 1.
The 3D model selection unit calculates the estimated effectiveness score of the 3D model based on at least one of the following, i.e.
1) The distance between the 2D keypoint in the candidate keypoint set and the corresponding 3D keypoint in the 3D model at the projected position on the plane of the image; and 2) of the estimated 3D model. An apparatus that is an overlapping area between a projection of the image on a plane and the region including the plurality of objects.

(Appendix 9)
It is an image processing method
Two-dimensional 2D keypoints relating to the plurality of objects are detected in a region containing the plurality of objects in the image, and the plurality of objects partially overlap each other;
Select a candidate keypoint set for one of the plurality of objects from the 2D keypoints;
Estimate a 3D 3D model of the one object based on the candidate key point set;
A method comprising selecting an optimized 3D model from among the estimated 3D models; and projecting the optimized 3D model onto the plane of the image to obtain the one object separated.

(Appendix 10)
The method described in Appendix 9
A method further comprising segmenting the region containing the plurality of objects in the image by a method of deep learning.

(Appendix 11)
The method described in Appendix 9
A method of detecting 2D key points of the plurality of objects by a method of deep learning.

(Appendix 12)
The method according to Appendix 11,
The method of selecting a candidate key point set is a method of selecting a 2D key point for the candidate key point set based on the detection result of deep learning.

(Appendix 13)
The method described in Appendix 9
A method, wherein the selection step of the candidate key point set comprises reducing the number of the candidate key point sets using geometric constraints on the one object.

(Appendix 14)
The method described in Appendix 9
The 3D model estimation step is a method of estimating the 3D model by performing deep learning or optimization by a neural network using a general-purpose model defined in advance for the one object.

(Appendix 15)
The method described in Appendix 9
The optimized 3D model selection step is a method of reducing the estimated number of 3D models using geometric constraints on the one object.

(Appendix 16)
The method described in Appendix 9
The optimized 3D model selection step calculates the estimated effectiveness score of the 3D model based on at least one of the following:
1) The distance between the 2D keypoint in the candidate keypoint set and the corresponding 3D keypoint in the 3D model at the projected position on the plane of the image; and 2) of the estimated 3D model. A method, which is an area of overlap between the projection of the image on a plane and the region including the plurality of objects.

(Appendix 17)
The method according to any one of Appendix 9-16.
A method in which the one object is represented by a shape parameter and a posture parameter.

(Appendix 18)
The method described in Appendix 9
A method in which the one object is a human hand or body.

(Appendix 19)
The method described in Appendix 9
The method, wherein the number of the plurality of objects is two.

(Appendix 20)
A machine-readable storage medium
A program including a machine-readable command code is stored, and when the command code is read and executed by a computer, the computer can be made to execute the image processing method described in Appendix 9-19. ..

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention belongs to the technical scope of the present invention unless the gist of the present invention is deviated.

Claims (10)

It is an image processing device
A detection unit that detects 2D (two-dimensional) key points relating to the plurality of objects in a region including a plurality of objects in an image, wherein the plurality of objects partially overlap.
Candidate key point set selection unit that selects a candidate key point set for one of the plurality of objects from the 2D key points;
An estimation unit that estimates a 3D (three-dimensional) model of the one object based on the candidate key point set;
A 3D model selection unit that selects an optimized 3D model from among the estimated 3D models; and an acquisition unit that projects the optimized 3D model onto the plane of the image to obtain the one object separated. Device. The image processing apparatus according to claim 1.
An apparatus further comprising a segmentation unit for segmenting the region containing the plurality of objects in the image by a method of deep learning. The image processing apparatus according to claim 1.
The detection unit is a device that detects a plurality of target 2D key points by a method of deep learning. The image processing apparatus according to claim 3.
The candidate key point set selection unit is a device that selects a 2D key point for the candidate key point set based on the detection result of deep learning. The image processing apparatus according to claim 1.
The candidate key point set selection unit is a device that reduces the number of the candidate key point sets by using geometric constraints on the one object. The image processing apparatus according to claim 1.
The estimation unit is an apparatus that estimates the 3D model by performing deep learning or optimization by a neural network using a general-purpose model defined in advance for the one object. The image processing apparatus according to claim 1.
The 3D model selection unit is a device that reduces the estimated number of 3D models using geometric constraints on the one object. The image processing apparatus according to claim 1.
The 3D model selection unit is
1) The distance between the 2D keypoint in the candidate keypoint set and the corresponding 3D keypoint in the 3D model at the projected position on the plane of the image; and 2) the estimated 3D model. An apparatus for calculating an estimated effectiveness score of the 3D model based on at least one of a planar projection of the image and an overlapping area of the region containing the plurality of objects. It is an image processing method
A 2D (two-dimensional) key point for the plurality of objects is detected in an area containing the plurality of objects in the image, and the plurality of objects partially overlap;
Select a candidate keypoint set for one of the plurality of objects from the 2D keypoints;
Estimate a 3D (three-dimensional) model of the one object based on the candidate key point set;
A method comprising selecting an optimized 3D model from among the estimated 3D models; and projecting the optimized 3D model onto the plane of the image to obtain the one object separated. A program for causing a computer to execute the image processing method according to claim 9.

Images