JPH11161821A - Three-dimensional form data processor and modeling system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a model whose feature on the sight of an object is reflected on a form and whose appearance is natural by specifying a part which requires correction based on two-dimensional photographing information of the object and generating the form model which is locally emphasized. SOLUTION: A data processor eytracts the area of a specified face element from a two-dimensional picture which color picture data shows. Eyebrows and pupils in the three-dimensional form model specified by form data are set to correction object parts Up and they are deformed, for example. In deformation, a mesh M vertical to a camera eye is projected on the correction object part and a correction vector (v) is calculated with a position on which a lattice point is projected as a sampling point (p). Then, the intersections (q) of the mesh M and the outlines of the correction object parts Up are obtained and a curve surface passing through the tip positions p2 of the correction vector (v) and the outline points (q) is obtained. The curve surface and the original three-dimensional form model are synthesized and the correction object parts Up are substituted for the obtained curve surface Up2. Thus, the positions of the pupils and the eyebrows are expressed by the forms.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to three-dimensional shape data processing for creating a model of an existing object.

[0002]

2. Description of the Related Art A portable non-contact type three-dimensional measuring device (three-dimensional camera) has been commercialized and used for data input to a CG system or a CAD system, body measurement, visual recognition of a robot, and the like. As a non-contact measurement method, a slit light projection method (light cutting method) is generally used, but a pattern light projection method, a stereo vision method, an interference fringe method, and the like are also known.

[0003] Also, three-dimensional CG software that can be used in personal computers and small three-dimensional cutting machines for hobbies are commercially available. If these are used, models and creations can be easily manufactured even in ordinary households.

[0004] On the other hand, a kind of vending machine for creating a face photograph sticker of a customer on the spot is gaining in popularity. The customer inserts coins for the fee and takes a desired pose in front of the camera while watching the monitor screen. Then, when a predetermined operation is performed, a sheet on which a fixed number of seals are arranged is created and discharged to an outlet. Most models have multiple options for the shape of the facial photograph, the imprint pattern, and the like.

[0005]

According to the above-described three-dimensional measuring apparatus, the shapes of various objects including a human body can be converted into data with the same ease as taking a photograph. Since it is a non-contact type, even when measuring a human body, the person to be measured does not feel troublesome. Therefore, it is conceivable to use this three-dimensional measuring device for creating a facial model that is not a facial photograph. That is, when combined with a three-dimensional processing machine, it is possible to measure a person's face and create a model with an appropriate magnification on the spot.

[0006] However, in the non-contact three-dimensional measurement, no facial element indistinguishable in shape does not appear on the model. For example, the so-called iris portion of the eyeball is not distinguished from the iris portion even though it is an important element characterizing the face. Eyebrows drawn with makeup tools also assimilate with the forehead.

[0007] In addition, the person concerned may sometimes feel uncomfortable with a model that faithfully reproduces the undulation of the face. This is because the color affects the perception of undulation, such as the warm-colored lips appearing more raised than they actually are. Even though the shape is faithful, the uncolored model looks flatter than a familiar face.

An object of the present invention is to make it possible to create a natural-looking model in which the visual characteristics of an object are reflected in the shape.

[0009]

According to the present invention, a portion requiring correction is specified based on two-dimensional photographing information of an object, and a locally enhanced shape model is generated.

An apparatus according to a first aspect of the present invention is a three-dimensional shape data processing apparatus for creating a model, which extracts a characteristic region satisfying a set condition from a two-dimensional image obtained by photographing an object. The correction is made to emphasize the portion corresponding to the extracted characteristic region to the shape model of the object obtained by the three-dimensional measurement of the contact.

According to a second aspect of the present invention, there is provided a three-dimensional shape data processing apparatus which divides a two-dimensional image obtained by photographing an object in color according to color conditions, and obtains a shape of the object obtained by non-contact three-dimensional measurement of the object. In the model, data correction is performed for each part corresponding to each area of the partitioned two-dimensional image to be deformed in accordance with the corresponding color information.

According to a third aspect of the present invention, there is provided a modeling system, comprising: a photoelectric conversion device for photographing an object; a three-dimensional measuring device for measuring the shape of the object in a non-contact manner; A three-dimensional shape data processing device for correcting data on the shape model of the object obtained by the three-dimensional measurement device based on the two-dimensional image.

A modeling system according to a fourth aspect of the present invention is configured to be capable of switching between an operation mode in which the data correction is performed and an operation mode in which the data correction is not performed.

[0014]

FIG. 1 is an external view of a three-dimensional model forming apparatus 1 according to the present invention. The three-dimensional model creation device 1 has a function of measuring the shape of an object and processing the material on the spot based on the measurement data, and is used as a vending machine for small articles in the shape of a customer's face. . The created article is a three-dimensional object in which a model of the face protrudes from a plate of a predetermined shape (for example, a square). It is also possible to add a specific undulating pattern to the plate surface (background portion). If an appropriate metal fitting is attached to such an article, it becomes an accessory such as a pendant, a broach, or a key holder. A metal fitting may be attached to the material in advance.

On the front surface of the upper half of the substantially life-size housing 10,
A light emitting window 12 and a light receiving window 14 for optical three-dimensional measurement are provided together with a display 16 for a user to check a pose. The light receiving window 14 is also used for two-dimensional color photography. The lower half of the housing 10 projects forward from the upper half, and the upper surface thereof is an operation panel 18. The product outlet 20 is provided on the front surface of the lower half.

The customer stands toward the three-dimensional model forming device 1 and inserts coins for a fee. Thereafter, when the user performs a start operation, the three-dimensional model creation device 1 measures the shape of the object existing within a certain range in front and displays a three-dimensional shape model (for example, a surface model) indicating the measurement result. Then, when the user performs a confirmation operation for instructing determination of the composition, the three-dimensional model creation device 1 starts three-dimensional processing according to the measurement result. The product is completed in a matter of minutes. The user takes out the product from the outlet 20.

FIG. 2 is a plan view of the operation panel 18. The operation panel 18 includes a start button 181, a confirmation button 182, a cancel button 183, a joystick 1
84 and a coin slot 185 are provided. The start button 181 is a start operation unit, and the confirmation button 182 is a confirmation operation unit. Joystick 18
Reference numeral 4 is used for changing the composition of the model. Rotation processing of the three-dimensional shape model is performed in response to a pan operation for tilting left and right, a tilt operation for tilting up and down, and a roll operation for rotating a knob, and the processing results are sequentially displayed. The cancel button 183 is an operation unit for instructing re-measurement when the user does not like the displayed three-dimensional shape model. However, the number of valid times is set in the cancel button 183, and it is not possible to instruct re-measurement without any restriction.

FIG. 3 is a functional block diagram of the three-dimensional model forming apparatus 1. The three-dimensional model creation device 1 includes a modeling system 1A that generates a three-dimensional shape model having a model size, and a processing system 1B that makes the three-dimensional shape model visible.

The modeling system 1A includes a photographing system 30 for converting (converting) the appearance information of a user who is an original object into digital data. The imaging system 30 includes a three-dimensional measurement device 34 for converting shape information into data by a slit light projection method, a two-dimensional imaging device 36 for converting color information into data, and a controller 38. Note that other methods may be used for the three-dimensional measurement instead of the slit light projection method. The shape data DS, which is information measured by the three-dimensional measuring device 34, and the color image data DC, which is image information obtained by the two-dimensional image capturing device 36,
Is input to Since the relative relationship between the camera coordinates of the three-dimensional measurement and the two-dimensional imaging is known, it is easy to align the three-dimensional shape model based on the shape data DS with the two-dimensional image. The data processing device 40 includes an image processing circuit (not shown), and performs various data processing including data correction specific to the present invention. The controller 42 of the data processing device 40 also performs overall control of the three-dimensional model creation device 1 and gives appropriate instructions to the controller 38 of the imaging system 30 and the controller 76 of the processing system 1B. The display 16 and the operation input system 80 are connected to the controller 42. The operation input system 80 includes the above-described operation panel 18 and a fee receiving mechanism.

On the other hand, the processing system 1B includes a processing device 72 for cutting a material such as a resin block, a material supply device 74 for supplying a material to a processing position and transporting the processed product to the outlet 20, a controller 76, An outlet sensor 78 is provided. The detection signal of the outlet sensor 78 is input to the controller 42.

It is also possible to adopt a circuit configuration in which the controller 42 is in charge of the control of the photographing system 30 and the processing system 1B, and the controller 38 and the controller 76 are omitted.

FIG. 4 is a perspective view showing an example of the mechanism configuration of the processing system 1B. The material supply device 74 has a stock section 210 for storing a total of eight types of materials. The storage spaces are provided on both sides of the linear transfer path 212, and four elevators 220 are arranged along the transfer path 212 in the storage spaces on each side. A plurality of materials of the same type are stacked on each elevator 220, and the vertical movement of the elevator 220 is controlled so that the uppermost material is located at a predetermined height. When one type of material suitable for the model to be created is designated, the designated material is transferred from the storage space to the transfer path 212 by the pushing rod 218 as the work 216.
Sent out to Then, the work 21 on the transfer path 212
6 is sent to the table 200 of the processing device 72 by the transfer rod 214 with the chuck.

In the table 200, the work 216 is fixed by two stoppers 202 and a clamp jig 204. Then, it is cut by a blade 208 attached to a rotating shaft 206 that can move up and down, left and right, and back and forth.

When the three-dimensional processing is completed, the work 216 is held by the chuck at the tip of the transfer rod 214, carried to the discharge-side end of the transfer path 212, and sent to the discharge port 222. Instead of using the transfer rod 214, the work 216 may be moved from the table 200 to the discharge port 222 in the form of a slide.

The mechanism configuration of the processing system 1B is not limited to the example. For example, the number of elevators can be reduced by arranging the same type of material horizontally on each of the multi-stage shelves, arranging an elevator at one end in the arrangement direction, and extruding the materials from the shelves to the elevators. The work may be carried from the storage position to the processing position to the discharge position by the arm robot. Instead of cutting, it is also possible to create a model by a method such as an additive manufacturing method (including a stereolithography method), a laser processing (thermal processing), and a molding processing (pressing and the like). In addition, regarding the material shape, the user may be able to select a desired external shape, or a material having the shortest processing time is automatically selected from a plurality of types of materials in which a standard face model has been created in advance. You may do so.

In the three-dimensional model creating apparatus 1 having the above configuration, in order to create a natural-looking face model, the three-dimensional shape model obtained by the three-dimensional measurement is automatically used by using the color photographing information of the face. The data correction to be deformed is performed by the data processing device 40.

FIG. 5 is a schematic diagram of partial data correction. The data processing device 40 first extracts a specific face element region from the two-dimensional image represented by the color image data DC. In the present embodiment, the eyebrows and irises (iris and pupil) are to be extracted. The extraction is performed in the following manner. By performing clustering in a specific color space (for example, L ^* a ^* b ^* color space), the two-dimensional image is divided into regions of the same hue. Labeling is performed on the result to extract a continuous area having the same hue. Matching is performed using a template created in advance based on statistics on the positions and colors of the face elements, and areas corresponding to eyebrows and irises are selected.

Next, the eyebrow and iris portions of the three-dimensional shape model specified by the shape data DS are corrected by the correction target portion U.
Set to p and perform moderate deformation. The deformation is performed in the following manner. A mesh M perpendicular to the camera line of sight is projected on the correction target portion Up, and the position where the grid point of the mesh M is projected on the correction target portion Up is set as a sampling point p. A correction vector v having a set length that is directed to the camera side from the sampling point p and is perpendicular to the model plane is calculated. The length of the correction vector v is set for each of the eyebrows and irises. For example, the eyebrows are longer.
An intersection (hereinafter referred to as a contour point) q between the mesh M and the contour of the correction target portion Up is obtained. A curved surface that passes through the position (tip point) p2 of the tip of the correction vector v and the contour point q is obtained. The obtained curved surface and the original three-dimensional shape model are synthesized. That is, the correction target portion Up of the three-dimensional shape model is replaced with the obtained curved surface Up2. As a result, the correction target portion Up rises, and the position of the iris is represented by the shape. The same applies to eyebrows drawn with eyebrows. Since the position of the contour point q does not change, the continuity of the surface is maintained between the eyebrows and irises and their surroundings.

The operation of the three-dimensional model forming apparatus 1 will be described below with reference to a flowchart. FIG. 6 is a main flowchart showing a schematic operation. After the power is turned on, the two-dimensional imaging and the display of the imaging result are repeated in a standby period waiting for the operation by the user (# 10, # 12, # 1).
4). In addition, a guidance message is displayed periodically. When a fee is input and the start button 181 is pressed, two-dimensional imaging is performed again and three-dimensional measurement is performed (# 1).
6, # 18). Predetermined data processing is performed (# 20), and the obtained three-dimensional shape model is displayed (# 22). At this time, the appearance is enhanced by applying a known graphic method such as adding a shadow. Then, it waits for an instruction operation. However,
The waiting time is finite, and after the time limit, it is considered that the confirmation operation has been performed.

When the joystick 184 is operated,
As described above, the three-dimensional shape model is rotated and displayed according to the operation (# 24, # 38). Cancel button 18
When 3 is pressed, the operation returns to the operation in the standby period (# 40, # 1).
0). However, in this case, there is no need for the customer to re-enter the fee, and if the start button 181 is pressed, re-measurement is performed.

When the confirm button 182 is pressed (# 2
6) Data for processing control is generated with reference to the processing condition database based on the three-dimensional shape model (# 28).
The material is processed (# 30). When the processing is completed, the product is discharged (# 32), and it is confirmed that the product has been taken out by the takeout port sensor 78, and the process returns to the standby operation (# 34, # 1).
0).

FIG. 7 is a flowchart showing the contents of the data processing of FIG. Here, as described above, the following processing including data correction for partially raising the projection and compression in the depth direction for shortening the processing time and intentionally flattening the design is performed.

A smoothing process is performed to eliminate abnormal data due to noise and to avoid excessive reproduction of fine irregularities (# 200). Perform re-sampling processing (# 21
0). This is a process of converting input data into data aligned with grid points at equal intervals projected in parallel from a certain direction in order to directly face the input data in the processing direction when the face is oblique. For example, if a person's face cannot be measured due to shadows below the ears, the data can be converted to represent a normal face-up face after three-dimensional measurement with the face facing up. If there is no measurement point at the position where the grid point is projected, linear interpolation is performed using the measurement values around the measurement point.
At this time, the projected direction is vertically above when processing, and each grid point has height data. Also,
Even when the input data is perspective projection, the input data can be converted to parallel projection data by this processing.

The missing part without data is complemented (# 22)
0). Various methods such as linear interpolation and weighted completion can be applied as the complementing method. For example, all missing parts of the data are replaced with fixed values (simple interpolation). As the fixed value, a set value, a minimum height, and an average value of the outer peripheral position of the face can be considered. If the missing part is completely surrounded by the valid data part, linear interpolation is performed from the surrounding data. In addition, for parts where accurate data cannot be obtained by optical three-dimensional measurement due to the nature of the target, such as black eyebrows and hair on a human face, the existing three-dimensional shape data is replaced. Is also good. In this case, a standard model of the face (front half of the head) is prepared, and the data of the standard model whose position and size have been adjusted are used for the data missing part. The adjustment of the standard model is performed in the following procedure. 2 in the same manner as in the case of data correction described above.
The eyes and mouth are extracted from the two-dimensional image, and the positions of three reference points are calculated. Then, the standard model is linearly converted so that each reference point of the standard model matches the actually measured shape model. In addition, such composition is not limited to the missing part of the face,
Applicable to any part.

After a three-dimensional shape model faithful to the real shape is obtained in each of the above-described processes, data correction specific to the present invention for realizing the eyebrows and irises is performed as described above (# 230). At this time, an emphasis may be made to slightly excite a specific portion such as a lip or a cheek. Also, whether or not to correct data,
It is also possible to provide a mode switching function so that the user or the device manager can select which part (face element) to correct when performing the operation.

A height compression process is performed to reduce the size of the three-dimensional model in the depth direction (# 240). That is,
Reduce the height difference in the depth direction to shorten the processing time.
A flat model is suitable for pendants and medals. Either the uniform compression method or the non-uniform compression method can be applied to the compression, and the compression can be used for each part.

The background portion of the three-dimensional shape model is detected (# 250). This is a pre-process for correcting the background portion. Leave the back of the customer as a blue background,
If the result of the color determination of the two-dimensional image is used, the background portion can be easily and reliably detected.

A background conversion for replacing the background portion with other data is performed (# 260). For example, since the background portion is extremely deep, the data is converted into data having a small depth in order to reduce the processing time. The data to be replaced may be plane data or three-dimensional plane data representing a pattern such as a flower or a tree or a geometric pattern.

The size adjustment is performed so that the full-size three-dimensional shape model is adapted to the product size (# 270). Further, resolution conversion is performed to adapt the data amount to the accuracy of the processing device 72 (# 280). In this processing, a mesh having a predetermined grid width is projected and resampled at grid points, but the projection direction is fixed to the vertical direction during processing. As a point of the resolution conversion (data number conversion), first, the constituent points of the shape model for machining are defined by the point pitch and the vector change amount, and the point pitch range corresponding to the vector change amount is stored in advance. Read from the specified characteristic data table and set. That is, the pitch is increased by thinning out the data, or the pitch is reduced by interpolating the data. When the measurement resolution is sufficiently large, only the thinning may be performed. If the resolution conversion function is provided, the resolution of the three-dimensional measuring device 34 is not limited, so that a usage form in which the measuring means is replaced depending on the application is allowed.

Lastly, positioning is performed by moving the origin of the coordinates in parallel so that the reference position of the three-dimensional shape model matches the reference position of machining (# 290). In the case of using a material in which predetermined irregularities have been formed in advance as described above in the processing, in the processing data generation processing (# 28 in FIG. 6) corresponding to the confirmation operation, 3
The cutting amount is calculated by comparing the three-dimensional shape model with the built-in irregularities.

FIG. 8 is a flowchart showing the contents of the partial data correction process of FIG. As described with reference to FIG. 5, first, eyebrows and irises are extracted from the two-dimensional image (# 231).
0). The correction target portion Up is set for the three-dimensional shape model (# 2320), and the correction vector v is calculated by projecting the mesh M (# 2340). Also, the correction target portion U
An intersection (contour point) q between the contour of p and the mesh M is obtained.
Either the correction target portion Up or the contour point q may be obtained first, and the operations for obtaining them may be performed in parallel.

Then, a surface including the tip point p2 of the correction vector v and the contour point q is obtained as a correction partial model (# 23).
60), the obtained surface and the original three-dimensional shape model are combined (# 2370).

In the above-described embodiment, an example in which a region of a specific color is extracted from a two-dimensional image and a model surface corresponding to the region is raised is described. The correction according to the color of the object may be made in the manner described above. The luminance signal of the two-dimensional image is re-mapped to the Munsell color space. Hue (hue), Value (brightness), Chroma
Among (saturation), which axis direction is used as a parameter is determined. Set the origin, the positive (outside) direction, and the magnification as the reference for correction. As image processing for the two-dimensional image, clustering is performed using the parameters determined in step (1), labeling is performed, and the two-dimensional image is partitioned into clusters (region division). Each partitioned area is associated with the surface of the three-dimensional shape model. As in the above embodiment, the mesh M is projected, the sampling points are determined, the correction vector v is calculated, and the model surface is deformed. At this time, the direction and magnitude of the correction vector v are calculated based on the HVC value of the corresponding area.

In the above-described embodiment, the three-dimensional model creation device 1 is assumed to be used as a vending machine. However, the data processing according to the present invention does not matter whether the model creation is paid or free. The size of the model is not limited to the reduced size,
It may be full size or enlarged size. The original object may be a creature other than a person or an inanimate object.

[0045]

According to the first to fourth aspects of the present invention,
It is possible to create a natural-looking model in which the visual characteristics of the object are reflected in the shape.

[Brief description of the drawings]

FIG. 1 is an external view of a three-dimensional model forming apparatus according to the present invention.

FIG. 2 is a plan view of an operation panel.

FIG. 3 is a functional block diagram of the three-dimensional model creation device.

FIG. 4 is a perspective view showing an example of a mechanism configuration of the processing system.

FIG. 5 is a schematic diagram of a partial data correction.

FIG. 6 is a main flowchart showing a schematic operation.

FIG. 7 is a flowchart showing the contents of the data processing of FIG. 6;

FIG. 8 is a flowchart showing the contents of a partial data correction process of FIG. 7;

[Explanation of symbols]

 40 data processing device (three-dimensional shape data processing device) 1A modeling system 36 two-dimensional imaging device (photoelectric conversion device) 34 three-dimensional measurement device DC color image data (two-dimensional image) DS shape data (shape model) Up Correction target part (Parts corresponding to characteristic regions)

Claims (4)

[Claims] 1. A feature region that satisfies a set condition is extracted from a two-dimensional image obtained by photographing an object, and a feature region extracted from a shape model of the object obtained by non-contact three-dimensional measurement of the object is extracted. A three-dimensional shape data processing apparatus for creating a model, characterized by adding data correction for deforming a portion corresponding to (1). 2. A two-dimensional image obtained by photographing an object in color is divided according to color conditions, and a shape model of the object obtained by non-contact three-dimensional measurement of the object is divided into two-dimensional images of the two-dimensional image. A three-dimensional shape data processing apparatus for creating a model, wherein data correction for deforming according to color information corresponding to each part corresponding to each area is performed. 3. A photoelectric conversion device for photographing an object, a three-dimensional measuring device for measuring a shape of the object in a non-contact manner, and a three-dimensional measuring device based on a two-dimensional image of the object obtained by the photoelectric conversion device. 3. A three-dimensional shape data processing apparatus according to claim 1 or 2, wherein data correction is performed on a shape model of the object obtained by a three-dimensional measurement apparatus. system. 4. The modeling system according to claim 3, wherein an operation mode in which the data correction is performed and an operation mode in which the data correction is not performed can be switched.