JP2014002515A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus and an image processing method by which in an image technology for acquiring a two-dimensional image of a photographing object in real time, a two-dimensional image can be acquired in real time by changing (deforming or adding new object) an image of a photographing object without actually changing the image of the photographing object without impairing resolution of the two-dimensional image.SOLUTION: Since three-dimensional information of an object OB can be reproduced in a short time by linearly combining pieces of three-dimensional information of a plurality of models M, a two-dimensional image is created by using a linear combination coefficient obtained on the basis of a result of the linear combination and two or more pieces of two-dimensional information associated with the pieces of three-dimensional information of the selected two or more models so as to display a two-dimensional image with high definition in real time.

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method capable of synthesizing a two-dimensional image of an object.

  In recent years, with the improvement in performance of personal computers and mobile terminals, still images are widely synthesized, and in CM and the like, CG and the like are also synthesized with actual video. However, it is difficult to cut out the object currently being photographed from the background and synthesize it in real time even with the current technology. If real-time display is required, for example, a part of the object is cut out by chroma key processing. And combining with other images.

  On the other hand, in Patent Document 1, new 3D information is created (or transformed) from an input 2D image and acquired 3D information, and a 2D image created from the 3D image A system that synthesizes an original input two-dimensional image in a natural state is disclosed.

JP 2004-152164 A

  In Patent Document 1, two-dimensional information is generated by mapping three-dimensional information by mapping texture data to a polygon model or having color information corresponding to a voxel model, and cutting out one of them. Is generated. However, this method creates 3D information with a huge amount of data compared to 2D information, making it difficult to display high-definition images in real time. On the other hand, because of real-time display, if data is thinned out, only an image with low resolution can be obtained.

  The present invention has been made in view of the above-described problems, and in an image technique for acquiring a two-dimensional image of a shooting target in real time, the shooting target is actually changed (deformed or a new target is added). An object of the present invention is to provide an image processing apparatus and an image processing method capable of synthesizing a changed two-dimensional image and acquiring the two-dimensional image in real time without losing resolution.

The image processing apparatus according to claim 1,
A database storing a plurality of sample information groups in which the sample 3D information of the sample object and the sample 2D information associated with the sample 3D information are combined,
3D information acquisition means for acquiring 3D information of the object;
2D information acquisition means for acquiring 2D image data of an object;
A calculation means for synthesizing the three-dimensional information and the two-dimensional information,
Perform 3D information acquisition of the object by the 3D information acquisition means,
The calculation means selects two or more types of the sample three-dimensional information whose shape is close to the three-dimensional information of the object from the database, and linearly combines the selected two or more types of sample three-dimensional information. Reproduce the 3D information of objects,
Further, the calculation means, the linear combination coefficient obtained based on the implemented linear combination result, two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information, It is characterized by combining two-dimensional images using.

  According to the present invention, a plurality of sample information groups each including a set of sample three-dimensional information of a sample object and sample two-dimensional information associated with the sample three-dimensional information are stored in a database in advance. When performing 3D information acquisition of the object by the 3D information acquisition means, the calculation means selects two or more types of the sample 3D information that is similar in shape to the 3D information of the object from the database, Since the three-dimensional information of the object is reproduced by linearly combining the selected two or more types of sample three-dimensional information, the three-dimensional information of the object can be quickly combined by linearly combining the three-dimensional information of the known sample. Dimensional information can be reproduced. Therefore, the calculation means, the linear combination coefficient obtained based on the implemented linear combination result, two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information, By synthesizing a two-dimensional image using, a high-definition two-dimensional image can be displayed even in real time.

The image processing apparatus according to claim 2 is the invention according to claim 1,
A second set of the plurality of sample information groups and second sample two-dimensional information associated with second sample three-dimensional information of a second sample object different from the sample object. A second database storing a plurality of third sample information groups obtained by combining the plurality of sample information groups, and a three-dimensional information acquisition means for acquiring the three-dimensional information of the object;
2D information acquisition means for acquiring 2D image data of an object;
Including calculation means to synthesize 3D information and 2D information,
The calculation means performs 3D information acquisition of the object by the 3D information acquisition means,
Further, the calculation means selects two or more types of the sample three-dimensional information whose shape is close to the three-dimensional information of the object from the second database, and linearly combines the selected two or more types of sample three-dimensional information. To reproduce the three-dimensional information of the object,
A two-dimensional image is obtained using a linear combination coefficient obtained based on the result of the implemented linear combination and two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information. It is characterized by combining.

  For example, in the invention of claim 1, for example, a model as a specimen object can be replaced with a consumer as an object. When the consumer arbitrarily changes the pose in front of the monitor, the image of the model with the same pose changed is displayed in real time. On the other hand, in the invention of claim 2, similarly, a plurality of types of clothes are previously photographed as the second specimen object and data is stored, so that the clothes worn by the model are desired by the consumer. Can be replaced with clothes. By doing so, it is possible to acquire a two-dimensional image obtained by combining the consumer and the clothes to be tried on and display them in real time.

The image processing apparatus according to claim 3 is the invention according to claim 1 or 2,
Before performing information acquisition of the target object, the basic shape information of the target object is acquired by the three-dimensional information acquisition unit and the two-dimensional information acquisition unit, and the basic shape of the sample information group is changed to the basic shape of the target object. A third database that stores information about new replacement specimen information,
3D information acquisition means for acquiring 3D information of the object;
2D information acquisition means for acquiring 2D image data of an object;
Including calculation means to synthesize 3D information and 2D information,
Perform 3D information acquisition of the object by the 3D information acquisition means,
The calculation means selects two or more types of the sample three-dimensional information whose shape is similar to the three-dimensional information of the object from the third database, and linearly combines the selected two or more types of sample three-dimensional information. To reproduce the 3D information of the object,
Further, the calculation means, the linear combination coefficient obtained based on the implemented linear combination result, two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information, It is characterized by combining two-dimensional images using.

  In the invention of claim 3, for example, by pre-photographing the consumer, information on the body shape and face is acquired, the database is corrected, and a new database is created. The two-dimensional image synthesized in this way has characteristics closer to consumers.

The image processing method according to claim 4 comprises:
Storing a plurality of sample information groups as a set of the sample 3D information of the sample object and the sample 2D information associated with the sample 3D information in a database;
Acquiring 3D information of the object;
Two or more types of the sample three-dimensional information having a shape close to that of the three-dimensional information of the object are selected from the database, and the three-dimensional information of the object is linearly combined with the selected two or more types of sample 3D information Steps to reproduce
A two-dimensional image is obtained using a linear combination coefficient obtained based on the result of the implemented linear combination and two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information. And a synthesizing step.

  According to the present invention, a plurality of sample information groups each including a set of sample three-dimensional information of a sample object and sample two-dimensional information associated with the sample three-dimensional information are stored in a database in advance. When the three-dimensional information acquisition of the object is performed, two or more types of the sample three-dimensional information whose shape is close to the three-dimensional information of the target object are selected from the database, and the selected two or more types of the sample three-dimensional Since the three-dimensional information of the object is reproduced by linearly combining information, the three-dimensional information of the object can be reproduced in a short time by linearly combining the known sample three-dimensional information. Therefore, using the linear combination coefficient obtained based on the implemented linear combination result and two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information, two-dimensional By synthesizing the images, a high-definition two-dimensional image can be displayed even in real time.

  According to the present invention, in an image technique for acquiring a two-dimensional image of a photographing object in real time, the changed two-dimensional image can be obtained without actually changing the photographing object (deformation or adding a new object). It is possible to provide an image processing apparatus and an image processing method that can synthesize and acquire the two-dimensional image in real time without losing resolution.

1 is a block diagram illustrating a configuration of an image processing apparatus 10. 2 is a flowchart illustrating an image processing method performed by the image processing apparatus 10. FIG. It is a figure for demonstrating the 1st image processing method. It is a figure for demonstrating the 2nd image processing method. It is a figure for demonstrating the 2nd image processing method. It is a block diagram which shows the structure of the image processing apparatus 10 'concerning a modification.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described. With reference to FIG. 1, the structure of the image processing apparatus 10 which concerns on embodiment of this invention is demonstrated. For example, the image processing apparatus 10 including a PC includes a database 11 built in a server (not shown), a three-dimensional information acquisition unit 13 that acquires three-dimensional information, and a two-dimensional information acquisition unit that acquires two-dimensional image data. 14, synchronization means 15, three-dimensional skeleton data selection means 16, three-dimensional skeleton data generation means 17, two-dimensional image generation means 18, two-dimensional image display means 19, normalization means 20, and rescaling means 21. The three-dimensional information acquisition means 13, the two-dimensional information acquisition means 14, and the synchronization means 15 constitute an information input means. The three-dimensional skeleton data generation means 17 and the two-dimensional image generation means 18 constitute a calculation means. FIG. 6 is a diagram showing a configuration of an image processing apparatus 10 ′ according to a modification, which is different from the image processing apparatus 10 of FIG. 1 in that it further includes a memory 12.

  FIG. 2 is a flowchart showing an image processing method by the image processing apparatus 10 according to the embodiment of the present invention. FIG. 3 is a diagram for explaining the image processing method. Here, a stereo camera is used as the three-dimensional information acquisition means 13 to acquire the model and subject data, but a three-dimensional scanner or the like may be used.

  In step S101 of FIG. 2, as a preprocessing, as shown in FIG. 3A, a plurality of models (persons) M as specimen objects are subjected to various standard actions (pauses), which are processed by a stereo camera. And 3D skeleton data (sample 3D information), which is 3D coordinate data, is acquired in the image processing apparatus, and the normalizing means 20 normalizes the obtained 3D skeleton data and corresponds to it. Two-dimensional image data (specimen two-dimensional information) is acquired by the two-dimensional information acquisition unit 14 and stored in the database 11 in a state of being associated with the synchronization unit 15. The two-dimensional image data can be obtained by converting the three-dimensional coordinate data into two dimensions. Normalization refers to processing for adjusting the distance between each point of the acquired 3D skeleton data to the distance between corresponding points of the reference 3D skeleton data. The method for determining the reference three-dimensional skeleton data is not particularly limited, but it is preferable to use the model having an average shape as a reference.

  Next, in step S102, as shown in FIG. 3B, the object person OB as the object is pre-photographed in a plurality of poses, and the two-dimensional information acquisition unit 14 acquires two-dimensional image data, The size information of preset parameters such as width, chest width, and arm width is acquired. Next, the scale ratio between the size information and the size of the parameter stored in the database 11 is stored in the rescaling means 21. Alternatively, in the modification shown in FIG. 6, the rescaler 21 converts the two-dimensional image obtained by rescaling the scale ratio over the two-dimensional image data stored in the database 11 and the corresponding three-dimensional skeleton data, It can also be stored in the memory 12. Thus, the preprocessing is completed.

  Furthermore, in step S103, the target person OB is actually photographed as shown in FIG. Furthermore, in step S104, the three-dimensional skeleton data selection means 16 accesses the database 11, and is the second closest to the three-dimensional skeleton data (A) closest to the three-dimensional skeleton data of the subject OB obtained by imaging. Three-dimensional skeleton data (B) is determined.

  In step S105, the three-dimensional skeleton data generation unit 17 creates an action frame based on the vector sum of the determined three-dimensional skeleton data. Specifically, if the three-dimensional skeleton data in the database close to the frame shot X of the actual skeleton (three-dimensional) data of the subject OB is A and B, it can be expressed as X = mA + nB (linear combination is performed) ). In step S106, the two-dimensional image generation means 18 assumes that the two-dimensional images x, a, and b correspond to the skeleton (three-dimensional) data X, A, and B, respectively, and the rescaling means 21 (or the memory 12 in FIG. 6). By enlarging or reducing (referred to as rescaling) a two-dimensional image based on the scale ratio from, and expressing it as x = ma + nb using the above coefficients (linear combination coefficients) m, n Perform image composition. When shooting continues, two-dimensional images are sequentially created in this way.

  In step S107, the two-dimensional image display means 19 as a monitor displays the synthesized two-dimensional image. If the process is not finished in step S108, the process can return to step S103 again to capture the target person OB, and similarly synthesize a two-dimensional image.

  According to the present invention, since the three-dimensional information of the object OB can be reproduced in a short time by linearly combining the three-dimensional information of the known model M, the linear combination coefficient obtained based on the linear combination result is obtained. 2D images can be quickly synthesized using two-dimensional information associated with 3D information of two or more selected models and two or more types of 2D information. It can be done.

  Next, a second embodiment will be described. In the first embodiment described above, the model M is wearing the clothes synthesized with the subject OB. In the second embodiment, the clothes selected by the subject OB can be combined with a two-dimensional image.

  4 and 5 are diagrams for explaining an image processing method according to the second embodiment. Here, the image processing apparatus 10 of FIG. 1 is used. First, as preprocessing, as shown in FIG. 4, a plurality of clothes C as second specimen objects are worn on a mannequin and photographed from various angles, and three-dimensional skeleton data (second dimension data) 3D information) is acquired, and the corresponding 2D image data (second 2D information) is acquired by the 2D information acquisition means 14 and correlated by the synchronization means 15. This is the second sample information group. Then, the 3D skeleton data and 2D image data of the model stored in the database 11 and the 3D skeleton data and 2D image data of the clothes C are set as a set (third sample information group), and the second Stored in the database 11 ′.

  Here, it is assumed that the target person OB has selected the desired clothes. By photographing the subject OB, the three-dimensional skeleton data selecting means 16 accesses the second database 11 ′, and selects the three-dimensional skeleton data closest to the selected clothes and the three-dimensional skeleton data associated therewith. The second closest 3D skeleton data is determined.

  The three-dimensional skeleton data generation means 17 creates an action frame by the vector sum of the determined three-dimensional skeleton data in the same manner as described above. The two-dimensional image generation means 18 synthesizes a two-dimensional image based on the action frame and action frame of the created subject OB. As a result, as shown in FIG. 5, for example, the subject OB virtually selects the clothing selected by the subject OB from the clothing options (C) displayed beside the two-dimensional image display means 19 as a monitor. A worn two-dimensional image is obtained. Therefore, a two-dimensional image of the subject OB who takes various poses with virtually tried-on clothes can be displayed in real time. Each time the subject OB re-selects clothes, the displayed clothes worn by the subject OB are converted to new ones.

Next, a third embodiment will be described. Here, a description will be given using the image processing apparatus 10 ′ of FIG.
The subject OB as an object is pre-photographed in a plurality of poses, the two-dimensional information acquisition means 14 acquires two-dimensional image data, and size information of preset parameters such as a waist width, a chest width, and an arm width To get. Next, a two-dimensional image obtained by rescaling the size ratio between the size information and the size of the parameter stored in the database 11 ′ over the two-dimensional image data stored in the database 11 by the rescaling means 21. And the three-dimensional skeleton data corresponding thereto can be stored in the memory 12, and the third sample information group stored in the database 11 'can be rescaled. Except for using the sample information group which is rescaled and stored in the memory 12, it is the same as the above-described embodiment.

  Next, a fourth embodiment will be described. In the first to third embodiments described above, the face of the subject OB is replaced with the feature of the model M. On the other hand, in the fourth embodiment, the characteristics of the subject OB can be combined with a two-dimensional image.

  Using the image processing apparatus 10 or 10 ', the subject OB is photographed as a target, and three-dimensional skeleton data (third three-dimensional information), which is three-dimensional coordinate data, is acquired, and the corresponding two-dimensional Image data (third two-dimensional information) is acquired by the two-dimensional information acquisition unit 14 and correlated by the synchronization unit 15. Then, only the face portion above the neck of the image data image-processed using the database 11 or the database 11 'is replaced with the face portion above the neck in the third two-dimensional information and displayed. The rest is the same as the above-described embodiment.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Database 11 'Second database 12 Memory 13 Three-dimensional information acquisition means 14 Two-dimensional information acquisition means 15 Synchronization means 16 Three-dimensional skeleton data selection means 17 Three-dimensional skeleton data generation means 18 Two-dimensional image generation means 19 Two-dimensional image display means

Claims (4)

A database storing a plurality of sample information groups in which the sample 3D information of the sample object and the sample 2D information associated with the sample 3D information are combined,
3D information acquisition means for acquiring 3D information of the object;
2D information acquisition means for acquiring 2D image data of an object;
A calculation means for synthesizing the three-dimensional information and the two-dimensional information,
Perform 3D information acquisition of the object by the 3D information acquisition means,
The calculation means selects two or more types of the sample three-dimensional information whose shape is close to the three-dimensional information of the object from the database, and linearly combines the selected two or more types of sample three-dimensional information. Reproduce the 3D information of objects,
Further, the calculation means, the linear combination coefficient obtained based on the implemented linear combination result, two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information, An image processing apparatus characterized in that a two-dimensional image is synthesized by using a computer. A second set of the plurality of sample information groups and second sample two-dimensional information associated with second sample three-dimensional information of a second sample object different from the sample object. A second database storing a plurality of third sample information groups obtained by combining the plurality of sample information groups, and a three-dimensional information acquisition means for acquiring the three-dimensional information of the object;
2D information acquisition means for acquiring 2D image data of an object;
Including calculation means to synthesize 3D information and 2D information,
The calculation means performs 3D information acquisition of the object by the 3D information acquisition means,
Further, the calculation means selects two or more types of the sample three-dimensional information whose shape is close to the three-dimensional information of the object from the second database, and linearly combines the selected two or more types of sample three-dimensional information. To reproduce the three-dimensional information of the object,
A two-dimensional image is obtained using a linear combination coefficient obtained based on the result of the implemented linear combination and two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information. The image processing apparatus according to claim 1, wherein the images are combined. Before performing information acquisition of the target object, the basic shape information of the target object is acquired by the three-dimensional information acquisition unit and the two-dimensional information acquisition unit, and the basic shape of the sample information group is changed to the basic shape of the target object. A third database that stores information about new replacement specimen information,
3D information acquisition means for acquiring 3D information of the object;
2D information acquisition means for acquiring 2D image data of an object;
Including calculation means to synthesize 3D information and 2D information,
Perform 3D information acquisition of the object by the 3D information acquisition means,
The calculation means selects two or more types of the sample three-dimensional information whose shape is similar to the three-dimensional information of the object from the third database, and linearly combines the selected two or more types of sample three-dimensional information. To reproduce the 3D information of the object,
Further, the calculation means, the linear combination coefficient obtained based on the implemented linear combination result, two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information, The image processing apparatus according to claim 1, wherein a two-dimensional image is synthesized using Storing a plurality of sample information groups as a set of the sample 3D information of the sample object and the sample 2D information associated with the sample 3D information in a database;
Acquiring 3D information of the object;
Two or more types of the sample three-dimensional information having a shape close to that of the three-dimensional information of the object are selected from the database, and the three-dimensional information of the object is linearly combined with the selected two or more types of sample three-dimensional information. Steps to reproduce
A two-dimensional image is obtained using a linear combination coefficient obtained based on the result of the implemented linear combination and two or more types of sample two-dimensional information associated with the selected two or more types of sample three-dimensional information. And an image processing method.