JP2015224909A - Three-dimensional measurement device and three-dimensional measurement method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional measurement device and a three-dimensional measurement method that can repair an omission of data while suppressing a decrease in quality of three-dimensional shape data in a limited time.SOLUTION: A three-dimensional measurement device includes: a small noise repair part (23a) which generates first loss repair information by repairing a loss included in three-dimensional information acquired by a stereo camera (11); a mask generation part (23b) which generates mask data such that a region which is not repaired in the first loss repair information is extracted; a large noise repair part (23c) which repairs a loss of the region which is not repaired in the first loss repair information using the first loss repair information and mask data; and a data composition part (23d) which generates composite loss repair information by putting the first loss repair information and second loss repair information together.

Description

  The present invention relates to a three-dimensional measurement apparatus and a three-dimensional measurement method for capturing an object with a stereo camera or a 3D sensor and acquiring three-dimensional shape data of the object.

  In recent years, a three-dimensional measuring apparatus equipped with a stereo camera, a 3D sensor, or the like and capable of easily acquiring three-dimensional shape data of an object has been developed, and is becoming popular in various fields.

  A conventional ordinary camera can only store luminance information and color information for each pixel in a two-dimensional image data format. On the other hand, the three-dimensional measuring apparatus can measure the distance to a person or an object to be photographed and save it in the form of three-dimensional shape data such as distance image data or point cloud data.

  In the three-dimensional measurement apparatus, a part of measurement data may be lost for various reasons. In order to compensate for this lack and improve the quality of the three-dimensional shape data, several methods have been proposed.

  For example, Patent Literature 1 discloses a data missing supplement method in a three-dimensional shape measuring apparatus that measures a human head. Specifically, in the three-dimensional shape measuring apparatus disclosed in Patent Document 1, as shown in FIG. 13, the three-dimensional shape of the head, which is the measurement object 102, is measured with the backboard 101 installed in the background. . Next, a pixel scan is performed to search for a data missing region existing between a pixel having a distance on the backboard 101 and a pixel having a head distance, and this pixel value is interpolated. As a result, it is possible to compensate for the lack of data that occurs near the head and the boundary between the head and the background.

  Non-Patent Document 1 discloses a processing method called an inpainting process that repairs a missing portion in an image.

JP 11-108633 A (published on April 23, 1999)

A.Telea, Journal of Graphic Tools 2004, Vol.9, No.1: pp.25-36 R. Szeliski, “Computer Vision: Algorithms and Applications”, Springer, 2011, Chapter 11

  By the way, when acquiring the three-dimensional shape data of an object using a three-dimensional measuring apparatus, a large omission due to illumination shine, occlusion, or the like may occur simultaneously with a small omission due to noise or the like. Note that the illumination is the reflection of the illumination light in the captured image by specular reflection that can be observed when the illumination angle and the shooting angle are the same. Occlusion is an area that can be seen from one of the left and right cameras but not from the other.

  However, in the conventional three-dimensional measuring apparatus, if this large omission is processed by simple linear interpolation, expansion / contraction, or the like as in the case of the small omission, unnatural shape data is obtained. On the other hand, if advanced processing such as in-painting processing that can repair large omissions is performed, the quality of the image after processing increases, but the amount of computation required for the processing is large, so the processing time becomes enormous. have.

  The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to provide a three-dimensional measuring apparatus capable of repairing data loss while suppressing deterioration in quality of three-dimensional shape data in a limited processing time. And providing a three-dimensional measurement method.

  In order to solve the above-described problem, the three-dimensional measurement apparatus according to one aspect of the present invention is configured to restore a defect included in the three-dimensional measurement information and a three-dimensional measurement unit that acquires the three-dimensional measurement information. First repair means for generating defect repair information; mask data generation means for generating mask data in which a portion of the first defect repair information that has not been repaired by the first repair means is extracted; Using the first defect repair information and the mask data, the defect of the portion of the first defect repair information that has not been repaired by the first repair means is repaired, and unit measurement is performed more than the first defect repair information. A second defect repairing unit that generates second defect repair information with a small amount of data per region, and a synthetic defect that generates the combined defect repair information by combining the first defect repair information and the second defect repair information. Repair information generation It is characterized in that it comprises a stage.

  In order to solve the above-described problem, the three-dimensional measurement method according to one aspect of the present invention includes a three-dimensional measurement step of acquiring three-dimensional measurement information, and repairing a defect included in the three-dimensional measurement information. A first repair step for generating defect repair information; a mask data generation step for generating mask data in which a portion of the first defect repair information that has not been repaired in the first repair step is extracted; Using the defect repair information of 1 and the mask data, the defect of the part not repaired in the first repair process in the first defect repair information is repaired, and unit measurement is performed more than the first defect repair information. A second defect repairing step for generating second defect repair information with a small amount of data per region, and a synthetic defect for generating synthetic defect repair information by synthesizing the first defect repair information and the second defect repair information. Repair information generation It is characterized in that it comprises a degree.

  According to one aspect of the present invention, it is possible to provide a three-dimensional measurement apparatus and a three-dimensional measurement method capable of repairing data loss while suppressing deterioration in quality of three-dimensional shape data in a limited processing time.

It is a block diagram which shows the structure of the face shape measuring apparatus as a three-dimensional measuring apparatus in Embodiment 1 of this invention. It is a flowchart which shows the production | generation operation | movement of the distance image in the distance image generation part of the said face shape measuring apparatus. (A) is a figure which shows an example of the left camera image of the to-be-photographed person's face image | photographed with the stereo camera of the said face shape measuring apparatus, (b) is image | photographed with the stereo camera of the said face shape measuring apparatus. It is a figure which shows an example of the right camera image of the image | photographing subject's face, (c) is a figure which shows the distance image produced | generated in the distance image generation part of the said face shape measuring apparatus, (d) is the said It is a figure which shows an example of the left camera image of a to-be-photographed person's face image | photographed with the stereo camera, giving a random two-dimensional dot pattern as a projection texture with the illuminating device of a face shape measuring apparatus, (e) It is a figure which shows an example of the right camera image of a to-be-photographed person's face image | photographed with the stereo camera, providing a random 2-dimensional dot pattern as a projection texture with the illuminating device of a face shape measuring apparatus. It is a flowchart which shows the restoration operation | movement in the restoration part of the said face shape measuring apparatus. (A) is a figure which shows an example of the distance image produced | generated in the distance image generation part of the said face shape measuring apparatus, (b) is restored by the small noise restoration part in the restoration part of the said face shape measuring apparatus. It is a figure which shows an example of the small noise repair distance image. (A) is a figure which shows an example of the mask image produced | generated by the mask production | generation part in the restoration part of the said face shape measurement apparatus, (b) is a large noise restoration part in the restoration part of the said face shape measurement apparatus It is a figure which shows an example of the repaired large noise repair distance image. (A) is a figure which shows an example of the mask image produced | generated from the distance image of the said face shape measuring apparatus, (b) is a figure which shows an example of the mask image produced | generated from the small noise repair distance image of the said face shape measuring apparatus. It is. (A) is a nose enlarged view showing an example of a mask image generated by omitting the expansion process in the mask generation unit of the face shape measuring apparatus, and (b) a mask image generated by performing the expansion process. It is a nose part enlarged view which shows an example. (A) is a figure which shows an example of the mask image A of the said face shape measuring apparatus, (b) is a figure which shows an example of the mask image B of the said face shape measuring apparatus, (c) is a figure in (a). It is a figure which shows the three-dimensional data at the time of performing a synthetic | combination process using the shown mask image A, (d) shows the three-dimensional data at the time of performing a synthetic | combination process using the mask image B shown in (b). FIG. (A) is a figure which shows an example of the distance image produced | generated in the distance image generation part of the said face shape measuring apparatus, (b) is restored by the small noise restoration part in the restoration part of the said face shape measuring apparatus. Is a diagram showing an example of a small noise repair distance image, (c) is a diagram showing an example of a repair distance image synthesized by the data synthesis unit in the repair unit of the face shape measuring device, (d) (D) is a figure which shows the three-dimensional data at the time of performing a synthetic | combination process using the distance image shown to (a), (e) is a figure which performs a synthetic | combination process using the small noise repair distance image shown to (b). It is a figure which shows the three-dimensional data at the time of performing, (f) is a figure which shows the three-dimensional data at the time of performing a synthetic | combination process using the repair distance image shown to (c). It is a block diagram which shows the structure of the body shape measuring apparatus as a three-dimensional measuring apparatus in Embodiment 2 of this invention. It is a block diagram which shows the structure of the monitoring camera apparatus as a three-dimensional measuring apparatus in Embodiment 3 of this invention. It is a figure which shows the conventional three-dimensional measuring apparatus.

[Embodiment 1]
One embodiment of the present invention is described below with reference to FIGS.

  The face shape measuring device as the three-dimensional measuring device of the present embodiment is a device for photographing a large number of human faces and acquiring the three-dimensional shape data.

  A configuration of a face shape measuring apparatus 1 as a three-dimensional measuring apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a face shape measuring apparatus 1 according to the present embodiment.

  As shown in FIG. 1, the face shape measuring apparatus 1 according to the present embodiment includes a stereo camera 11 as a three-dimensional measuring unit, an illumination device 12, a data processing circuit 20, a data display unit 13, and a data storage unit 14. Has been.

  The stereo camera 11 is a camera in which two cameras, a left camera and a right camera (not shown) are arranged apart from each other by a predetermined distance (base line length), and acquires face image data of the face F of the subject to be photographed. Therefore, it is arranged in front of the face shape measuring apparatus 1.

  The illuminating device 12 is a small projector, that is, a light projecting device, and can irradiate illumination light including a predetermined projection pattern onto the face F of the subject to be photographed at the time of photographing.

  The data processing circuit 20 is a circuit board for processing the 3D image data acquired by the stereo camera 11 to acquire 3D image information, and includes a control unit 21, a distance image generation unit 22, a restoration unit 23, and data Each block of the conversion unit 24 is configured. The repair unit 23 includes a small noise repair unit 23a as a first repair unit, a mask generation unit 23b as a mask data generation unit, a large noise repair unit 23c as a second repair unit, and a combined defect repair information generation unit. It consists of sub-blocks of the data synthesizer 23d. These blocks and sub-blocks can be arranged as independent dedicated processing circuits on the circuit board. Alternatively, each can be implemented as a subroutine on a program using a common arithmetic unit.

  The face shape measuring apparatus 1 according to the present embodiment is characterized in that large and small defects included in the acquired distance image data are repaired using the repair unit 23. The operation of the restoration unit 23 will be described later.

  The data display unit 13 is a display device such as a liquid crystal display, and can display 3D shape data of the face F of the person to be photographed obtained by measurement on the screen. Further, the data storage unit 14 can store the three-dimensional shape data of the face, and can extract them as needed or transmit them to the outside via a communication line (not shown).

  The control unit 21 of the data processing circuit 20 is connected to the stereo camera 11, the illumination device 12, the distance image generation unit 22, the restoration unit 23, the data display unit 13, and the data storage unit 14, and controls between these units. The entire control is performed by transmitting and receiving signals.

  Details of the face shape measurement process of the face shape measuring apparatus 1 having the above-described configuration will be described based on FIGS. 2 and 3A to 3E. FIG. 2 is a flowchart showing a distance image generating operation in the distance image generating unit 22 of the face shape measuring apparatus 1. FIG. 3A is a diagram illustrating an example of the left camera image 31L of the face F of the person to be photographed photographed by the stereo camera 11 of the face shape measuring apparatus 1. FIG. 3B is a diagram illustrating an example of the right camera image 31R of the face F of the person to be photographed, which is photographed by the stereo camera 11 of the face shape measuring apparatus 1. FIG. 3C is a diagram illustrating the distance image 34 generated by the distance image generation unit 22 of the face shape measuring apparatus 1. FIG. 3D is a diagram illustrating an example of the left camera image 31L of the face F of the person to be photographed, which is photographed by the lighting device 12 with a random two-dimensional dot pattern as a projection texture. (E) of FIG. 3 is a figure which shows an example of the right camera image 31R of the to-be-photographed person's face F image | photographed by providing the random two-dimensional dot pattern as a projection texture with the illuminating device 12. FIG.

  First, a face image is taken by the stereo camera 11 with the face F of the subject to be photographed facing the face shape measuring apparatus 1. A left camera and a right camera (not shown) constituting the stereo camera 11 are camera modules each combining a small CMOS image sensor, a condensing lens, and a DSP (Digital Signal Processor) for image signal processing. This camera module can perform pixel thinning or pixel interpolation by changing the setting from the outside, and can output with a lower number of pixels than during shooting. In this embodiment, the total number of pixels of the camera module is, for example, horizontal 2592 pixels × vertical 1944 pixels = approximately 5 million pixels, and is output with an image size of horizontal 1280 pixels × vertical 960 pixels by performing pixel thinning. doing. However, it is not necessarily limited to this size.

  Further, the camera module of the present embodiment has two modes, “still image mode” and “moving image mode”, which can be switched from the outside. In the “still image mode”, only one image is taken at a predetermined timing, and image data is output. On the other hand, in the “moving image mode”, it is possible to continuously capture images at a predetermined cycle and output image data. For the stereo camera 11 of the present embodiment, camera calibration data D0, which will be described later, including parameters such as camera focal length, lens distortion data, and the like is acquired in advance by a procedure called calibration. . The camera calibration data D0 is stored in the data storage unit 14 inside the face shape measuring apparatus 1.

  At the time of photographing a face image, a synchronization signal is transmitted from the control unit 21 to the left camera and right camera (not shown) of the stereo camera 11 and the illumination device 12. Along with this synchronization signal, the illumination device 12 irradiates the face with projection light, and the two cameras of the stereo camera 11 shoot at the same timing. Thereby, as shown in FIG. 2, the left camera image 31L and the right camera image 31R are obtained. An example of the left camera image 31L and the right camera image 31R is shown in FIGS. In the image of the present embodiment, the mannequin head is a subject to be photographed as the face F of the person who is the subject of photographing, and a part of an unnecessary area around the image is trimmed.

  These image data captured by the stereo camera 11 are sent to the distance image generation unit 22.

  The distance image generation unit 22 first performs a process called stereo parallelization (S1). Stereo parallelization removes image distortion caused by lens aberration and the like, and is further convenient in the case of searching for corresponding points later, and further, in the vertical (vertical) direction of the left camera image 31L and the right camera image 31R. This is a process of aligning the positions of. This process is performed by calculating using a coefficient obtained from camera calibration data D0 stored in advance in the face shape measuring apparatus 1. By this stereo parallelization processing, a parallelized left camera image 32L and a parallelized right camera image 32R are obtained.

  Next, a pre-filter such as an edge enhancement filter is applied to the two images, and a level difference between the left and right images due to a difference in illumination is corrected (S2). Thereafter, corresponding point search processing is performed between the left and right images (S3). In the corresponding point search process, with reference to the parallelized left camera image 32L, the corresponding pixel is searched for from the pixels of the parallelized right camera image 32R for each pixel of the parallelized left camera image 32L. Thereby, a pixel shift amount between corresponding points, that is, a parallax value can be obtained. By performing this operation for all the pixels of the left camera image 32L after parallelization, an image in which the parallax value is expressed in shades of pixels (grayscale) is obtained. This image is subjected to post-filtering processing such as filling a pixel for which no corresponding point is obtained with a pixel value representing invalid data (S4), and as a result, a parallax image 33 based on the left camera is obtained.

  In the stereo camera 11, the greater the distance to the point to be measured, the greater the parallax between the left and right images. For this reason, if the camera calibration data D0 is known in advance, the parallax value can be uniquely converted into a distance. By performing such parallax-distance conversion (S5) (S5), the distance image 34 based on the left camera can be generated.

  Such distance image generation processing by the distance image generation unit 22 can also be performed by hardware such as FPGA or ASIC on the circuit board. Alternatively, it can be performed as software as an image processing program. The details of the distance image generation method described above are described in Non-Patent Document 2, for example.

  In this way, the distance image 34 is generated by the distance image generator 22. The actually generated distance image is shown in FIG. In the distance image 34 of the present embodiment, the shorter the distance, the whiter the distance, the farther the distance. Expressed with black pixel values. In the present embodiment, since the parallax of the right camera image 32R after parallelization with respect to the left camera image 32L after parallelization is obtained with reference to the left camera, the visual field range of the obtained distance image 34 is after parallelization. It becomes the same as the visual field range of the left camera image 32L. That is, the pixel in the parallelized left camera image 32L corresponding to a certain pixel in the distance image 34 exists at the same position on the image.

  Note that when the change in shading is scarce on the face surface, the corresponding points of the left camera image 31L and the right camera image 31R may not be obtained by the above simple corresponding point search. In that case, instead of projecting simple illumination light from the illumination device 12, illumination light including an appropriate texture is projected toward the face F of the subject to be photographed to forcibly change the light and shade as a clue to search for corresponding points. Can be granted. Left and right images taken with a random two-dimensional dot pattern applied as the projection texture are shown in FIGS. Depending on the condition of the subject, it is possible to switch with / without texture projection, or to generate a distance image using image data taken with / without projection. It is also possible to simultaneously acquire images with and without projection by using infrared light as projection light and wavelength separation between visible light and infrared light.

  The left camera image 31L and the right camera image 31R in this embodiment have a bit depth of 8 bits per pixel, that is, 256 levels (gradation). The parallax image 33 obtained as a result of the corresponding point search is usually represented by a shift amount in units of one pixel. However, in the corresponding point search process of the present embodiment, the parallax is calculated in units of sub-pixels smaller than one pixel by performing an interpolation process after searching for the corresponding points. As a result, the parallax image 33 and the distance image 34 obtained by converting the parallax image 33 have a bit depth of 16 bits per pixel. The distance value of the distance image 34 according to the present embodiment is set to 0 (= 0 mm) to 9998 (= 0.1 mm units) with the lens principal point of the left camera as the origin and the depth direction along the camera optical axis as the positive direction. Values up to 999.8 mm).

  In the process of searching for corresponding points, the corresponding points of the left and right images may not be found for various reasons. For example, small non-measurable pixels of one pixel to several pixels occur when sporadic noise in an image or when a corresponding point happens to exist between pixels. This is hereinafter referred to as “small noise (small defect)”. Apart from this, unclearness due to lack of illumination light, overexposure due to excess illumination light, shine of illumination light, and occlusion of the stereo camera 11, etc., together with non-measurable pixels in units of several to several tens of pixels May occur. Hereinafter, this set of pixels is referred to as “large noise”. Note that the illuminance of illumination light refers to reflection of illumination light in a captured image by specular reflection that can be observed when the illumination angle and the shooting angle are the same. In addition, the occlusion of the stereo camera 11 refers to an area that can be seen from one of the left and right cameras but not from the other.

  In the present embodiment, among these small noise and large noise pixels, a value of 9999 indicating an abnormal value is assigned when measurement is clearly impossible, that is, when no corresponding point is found. However, even in a noise region, a corresponding point may be found by chance in an irrelevant pixel in the corresponding point search. In this case, an effective distance value (but not a surrounding noise pixel) is not an abnormal value. A value deviating from the distance value).

  As described above, the distance image 34 obtained by the distance image generation unit 22 includes large and small noise components. If the three-dimensional shape data is obtained from the distance image 34 as it is, data with very low accuracy may be obtained due to noise. In order to prevent this, the face shape measuring apparatus 1 according to the present embodiment is characterized in that the restoration unit 23 performs a restoration process for noise included in the distance image 34.

  Details of the repair process in the repair unit 23 will be described with reference to FIGS. 1, 4 and 5A and 5B. FIG. 4 is a flowchart showing the repair operation of the repair unit 23 in the face shape measuring apparatus 1 of the present embodiment. FIG. 5A is a diagram illustrating an example of the distance image 34 generated by the distance image generation unit 22 of the face shape measuring apparatus 1. FIG. 5B is a diagram illustrating an example of the small noise repair distance image 41 repaired by the small noise repair unit 23 a in the repair unit 23 of the face shape measuring apparatus 1.

  In the small noise restoration unit 23a in the restoration unit 23 shown in FIG. 1, the expansion process (S11) and the contraction process (S12) are successively performed on the distance image 34, as shown in FIG. These correspond to the first repairing means of the present invention. The expansion process and the contraction process are a kind of image morphology process. In the expansion process, the pixel value of the target pixel is replaced with the maximum pixel value among the surrounding pixels surrounding it. On the other hand, in the contraction process, on the contrary, the pixel value of the target pixel is replaced with the minimum pixel value among the surrounding pixels surrounding it. By continuously applying the expansion process and the contraction process, an isolated pixel having a pixel value larger or smaller than that of the surrounding area is filled with the surrounding pixel value. As a result, a relatively small noise (hereinafter referred to as “small noise”) pixel value including an unmeasurable value in the distance image 34 is removed, and a small noise repair distance image 41 is obtained. FIG. 5B shows a small noise repair distance image 41 obtained by performing small noise repair on the distance image 34 shown in FIG.

  In the small noise repair distance image 41 of FIG. 5B, it can be seen that fine noise components included in the distance image 34 of FIG. 5A are repaired. However, relatively large noise portions present on both sides of the nose are not repaired by the above processing and remain.

  The small noise repair distance image 41 has a bit depth of 16 bits. In FIG. 5B, the tone is expressed in gray scale after converting 16 bits to 8 bits. The same applies to an image having a 16-bit bit depth described below.

  In this embodiment, expansion processing and contraction processing are used as image processing for removing small noise performed by the first restoration means. However, there are a linear interpolation filter, a median filter, and the like as image processing that can be applied relatively easily as in the expansion processing and the contraction processing. For this reason, you may use these processes instead of the expansion / contraction process as needed. Also, depending on the magnitude of the small noise, the expansion process and the contraction process are not only applied once, but are applied twice in succession, for example, expansion-expansion-contraction-contraction, and noise twice as large is applied. It is also possible to remove it.

  Next, the mask generation unit 23b in the repair unit 23 generates a mask image based on the small noise repair distance image 41. In this case, first, a smoothing filter is applied to the small noise repair distance image 41 (S21), and a difference image is obtained by taking the difference between this and the image before application (S22). In this difference image, the portion where the degree of local change in the pixel value of the original image is larger is higher, and the smaller the value is, the closer to 0 is. For this reason, after applying a smoothing filter to this (S24), it binarizes with a suitable threshold value (S25). As a result, a binarized image is obtained in which when the degree of change in the pixel value of the original image is large, white (pixel value is the highest), and when the degree of change is black (pixel value is the lowest). This is called “mask image A42a”. Further, a smoothing filter is applied again to this image (S26), and a mask image B42b is obtained.

  Among these, the mask image A42a is used as a mask at the time of large noise restoration by the inpainting process in the large noise restoration unit 23c. On the other hand, the mask image B42b is used to determine a combination ratio between the large noise repaired image and the small noise repaired image in the data combining unit 23d.

  Subsequently, the large noise restoration unit 23c in the restoration unit 23 performs expansion processing and contraction processing from the small noise restoration distance image 41 obtained by the small noise restoration unit 23a and the mask image A42a obtained by the mask generation unit 23b. Repair large noise that cannot be removed.

  First, the large noise restoration unit 23c converts the bit depth of the small noise restoration distance image 41 from 16 bits to 8 bits (S31). Then, the image converted to 8-bit accuracy is repaired with a large noise by a method called inpainting processing (S32), and a large noise repair distance image 43 is output.

  Here, an example of the mask image A42a is shown in FIG. 6A, and an example of the large noise repair distance image 43 is shown in FIG. 6B. In the large noise repair distance image 43 of FIG. 6B, large noises such as both sides of the nose that could not be removed in the small noise repair distance image 41 shown in FIG. 5B are almost repaired. I understand that.

  The inpainting process is a process that uses the pixel information around the defect area to restore the pixel values inside the defect area when the image contains a defect and the position of the defect area is known. It is. The specific processing procedure is described in Non-Patent Document 1, etc., but adjacent pixel values are smoothly connected in order from the pixel closest to the boundary of the missing region using the continuity condition of the pixel value change. Thus, the pixel values are restored in order. Therefore, it is suitable for repairing the surface shape of an object whose surface is known to be smooth, such as the shape of a human face, body, or natural object.

  If a linear interpolation process is used instead of the in-painting process described here as a method for repairing large noise, discontinuous pixel value changes remain at the boundary of the repaired part, and the repaired image is not correct. It becomes a natural image. When the inpainting process is used, the image can be restored smoothly as described above, so that a natural restoration distance image can be obtained. If the same effect as described above can be obtained, a repair processing method other than the in-painting process can be applied.

  As described above, the inpainting process. Since the pixel values are restored in a procedure from the periphery to the center of the defect area, the calculation time becomes longer compared to a small noise removal method such as expansion and contraction. For this reason, in this embodiment, the time required for processing is reduced by reducing the bit depth to 8 bits before processing.

  In particular, when the area of the defect region is large, it takes time to repair by the inpainting process. For this reason, the mask image used for large noise restoration needs to be an image that reliably specifies only the large noise portion as a mask. For this reason, the small noise repair distance image 41 is used instead of the distance image 34 as the original image from which the mask generation unit 23b in the previous stage generates. Since the distance image 34 before the small noise restoration contains many fine noises in the image, if the difference from the smoothing filter is taken and binarized, as shown in FIG. All missing parts are extracted as a mask. On the other hand, when a mask is generated from the small noise repair distance image 41, only a large defect portion can be extracted as a mask as shown in FIG. As a result, since only the large noise can be targeted for restoration by the large noise restoration unit 23c at the subsequent stage, the efficiency of the restoration process can be increased.

  In the process of the mask generation unit 23b, an expansion process (S23) is performed between the difference process (S22) and the binarization process (S25), and the mask area is temporarily expanded. Without this expansion process, a gap is formed in the mask due to fine random components in large noise. Differences in the mask image with and without the expansion process are shown in FIGS. FIG. 8A is an enlarged view of the nose portion of the mask image when the expansion process is omitted. FIG. 8B is an enlarged view of the nose portion of the mask image A42a when the expansion process is performed.

  From FIG. 8A, it can be seen that many fine holes exist in the large noise portion of the mask image.

  If such a hole exists in the mask, when applying the inpainting process in the subsequent large noise repair unit 23c, the hole is determined to be a non-noise (normal) pixel, and an incorrect repair process is performed. End up. By sandwiching the expansion process, the gap between the masks can be filled, and the mask of the large noise part can be made into a correct shape. As a result, it is possible to correct large noise correctly in the inpainting process.

  The large noise repair distance image 43 obtained as described above is sent to the data synthesis unit 23d as shown in FIG. The data synthesis unit 23d synthesizes the large noise repair distance image 43 and the small noise repair distance image 41. First, the large noise repair distance image 43 is returned to the 16-bit bit depth (S41), and the synthesis process with the small noise repair distance image 41 is performed (S42). At the time of synthesis, the mask image B42b generated by the mask generation unit 23b is used to determine a synthesis ratio for each pixel of both images. That is, as the pixel value of the mask image B42b is closer to white, the ratio of the large noise repair distance image 43 is increased, and as it is closer to black, the ratio of the small noise repair distance image 41 is increased to synthesize both.

  Since the large noise repair distance image 43 obtained by the inpainting process has a bit depth of 8 bits, even if converted to 16 bits, the substantial bit depth remains 8 bits. Accordingly, for example, as a procedure different from the present embodiment, if the procedure is such that the small noise repair and the large noise repair are simply applied to the original image in order, the final repaired image The effective bit depth is 8 bits over the entire image. This means that the distance accuracy significantly decreases (to 1/256) with respect to the bit depth of 16 bits of the original distance image before repair.

  On the other hand, in the present embodiment, the small noise repair distance image 41 and the large noise repair distance image 43 are separately created and then combined to restore only the large noise portion at the 8-bit depth, and the remaining A final repair image can be obtained while maintaining a 16-bit depth for the portion. That is, this processing procedure can minimize a decrease in distance accuracy of the distance image 34 after repair.

  In this embodiment, in order to shorten the processing time of the inpainting process, the process is applied after the bit depth of the distance image 34 is reduced. However, instead of reducing the bit depth, the image size is reduced. However, the same effect can be obtained. In that case, the mask image A42a and the mask image B42b are also reduced in accordance with the processing, and then the process is applied. Then, the image size is restored to the original size by enlarging again at the time of composition after the process.

  The mask image B42b used for composition is created by smoothing the mask image A42a in the mask generation unit 23b, so that the pixel value gradually changes at the boundary between the large noise and the other portions. . Therefore, by synthesizing the large noise repair distance image 43 and the small noise repair distance image 41 based on this pixel value, there is an effect of making the joint at the boundary between the two inconspicuous.

  FIG. 9C shows a three-dimensional display of the three-dimensional shape data obtained when combining is performed using the mask image A42a shown in FIG. 9A. Similarly, FIG. 9D shows a three-dimensional display of the three-dimensional shape data obtained when combining is performed using the mask image B42b shown in FIG. 9B.

  In FIG. 9C, the seam is conspicuous at the boundary of the large noise part beside the nose, whereas in FIG. 9D, the boundary is blurred and becomes inconspicuous. Thus, it can be seen that more natural and smooth three-dimensional shape data can be obtained by using the mask image B42b to which the smoothing process is added.

  The data synthesizing unit 23d further smoothes the synthesized image (S43), erases a slightly remaining noise component, and finally outputs a repair distance image 44.

  The repair distance image 44 obtained by repairing as described above is sent to the data converter 24 as shown in FIG. In the data converter 24, a process called 3D reprojection (3D reprojection) is performed (S51). The distance image 34 holds, for each pixel, a numerical value of the distance in the depth direction (direction toward the subject along the optical axis) of the object existing in the pixel as viewed from the stereo camera 11. . If the distance value in the depth direction and the camera calibration data D0 of the stereo camera 11 are known, the distance values in the vertical and horizontal directions other than the depth direction can be obtained. As a result, the XYZ coordinate value of the object existing at that position is obtained for each pixel.

  In the case of the present embodiment, the coordinate value data is 1960 pixels wide × 960 pixels long × 3 dimensions × 16 bits, and is very large. Therefore, the data is compressed in a format suitable for storage. (S52). Specifically, processing such as reducing the number of vertical and horizontal pixels and deleting coordinate values of unnecessary (no face) regions is performed. As a result, three-dimensional point group data 45, which is finally the three-dimensional coordinate information of the face shape, is generated.

  Three-dimensional point cloud data 45 from the original distance image 34 shown in FIG. 10A, the small noise repair distance image 41 shown in FIG. 10B, and the repair distance image 44 shown in FIG. FIG. 10 (d), (e), and (f) show the three-dimensional display of these.

  In FIG. 10 (f), it can be seen that large noise existing in the data of FIGS. 10 (d) and 10 (e) is repaired, and smooth and high-quality three-dimensional data is obtained.

  As described above, the three-dimensional point cloud data 45 is stored in the data storage unit 14 and can be displayed on the screen of the data display unit 13 for confirmation. Further, they can be taken out as necessary, or transmitted to the outside via a communication line (not shown) to be used for various purposes.

  In this embodiment, the stereo camera 11 is used to acquire the distance image 34. However, as a camera capable of acquiring the distance image 34, a TOF (Time 0f Flight) method or a structured light irradiation method is used. It is also possible to use a three-dimensional camera.

  The three-dimensional face data obtained by the face shape measuring apparatus 1 according to the present embodiment can be used for sports, art, clothing, medical care, nursing care, games, amusement, advertising, and various other uses not limited to these. it can.

  As described above, the face shape measuring apparatus 1 according to the present embodiment can repair missing data while minimizing the deterioration of the quality of the three-dimensional face data in a limited processing time. .

  As described above, the face shape measurement apparatus 1 as the three-dimensional measurement apparatus according to the present embodiment is included in the stereo image 11 as the three-dimensional measurement unit that acquires the distance image 34 as the three-dimensional measurement information, and the distance image 34. A small noise repairing part 23a as a first repairing means for repairing the missing defect and generating a small noise repairing distance image 41 as first defect repairing information, and the small noise repairing part 23a of the small noise repairing distance image 41 Then, a mask generation unit 23b as mask data generation means for generating a mask image A42a or a mask image B42b as mask data obtained by extracting a portion that has not been repaired, a small noise repair distance image 41, a mask image A42a, or a mask image B42b Is used to repair a defect in a portion of the small noise repair distance image 41 that has not been repaired by the small noise repair portion 23a. A large noise restoration unit 23c as a second restoration means for generating a large noise restoration distance image 43 as second defect restoration information having a smaller data amount per unit measurement area than the small noise restoration distance image 41; A data synthesizing unit 23d serving as a synthetic defect repair information generating unit that synthesizes the repair distance image 41 and the large noise repair distance image 43 to generate a repair distance image 44 as composite defect repair information.

  Further, in the face shape measurement method as the three-dimensional measurement method of the present embodiment, a three-dimensional measurement process (S1 to S5) for acquiring the distance image 34, and a defect included in the distance image 34 are repaired to repair small noise. Mask image A42a or mask extracted from the first repairing step (S11 / S12) for generating the distance image 41 and the portion of the small noise repairing distance image 41 that has not been repaired in the first repairing step (S11 / S12) A mask data generation process (S21 to S26) for generating an image B42b, and a first repair process (S11 · S26) of the small noise repair distance image 41 using the small noise repair distance image 41 and the mask image A42a or the mask image B42b. In S12), the defect of the part that has not been repaired is repaired, and the large noise repair is performed with a smaller data amount per unit measurement area than the small noise repair distance image 41. A second restoration step (S31 / S32) for generating the isolated image 43, and a combined defect repair information generation step (S41) for generating the repair distance image 44 by synthesizing the small noise repair distance image 41 and the large noise repair distance image 43. To S43).

  According to the above configuration, in the present embodiment, first, a small defect that can be repaired by the small noise restoration unit 23a such as expansion / contraction processing is repaired, and the mask image A42a that cannot be repaired by the expansion / contraction processing or the like. Alternatively, the repair distance image 44 as three-dimensional measurement data obtained by applying the processing of the large noise repairing unit 23c such as inpainting processing only to the mask data portion of the mask image B42b and finally combining them to repair the defect is obtained. obtain.

  As a result, the processing by the large noise restoration unit 23c such as inpainting processing is limited to the minimum necessary mask data part, and the processing by the small noise restoration unit 23a such as expansion / contraction processing is performed in the other parts. As a result, it is possible to suppress a decrease in the three-dimensional measurement accuracy due to a reduction in resolution as a whole at the time of restoration, and it is possible to repair a deficiency in the three-dimensional measurement information at a high speed by preventing an enormous amount of calculation.

  Therefore, it is possible to provide the face shape measuring apparatus 1 and the face shape measuring method capable of repairing the data loss while suppressing the deterioration of the quality of the three-dimensional shape data with a limited processing time.

  Further, in the face shape measuring apparatus 1 according to the present embodiment, the amount of calculation per unit data amount is larger in the defect repair calculation by the large noise repair unit 23c than in the defect repair calculation by the small noise repair unit 23a. The large noise restoration unit 23c generates data amount reduction data in which the data amount per unit measurement area is reduced as compared with the small noise restoration distance image 41, and then performs a defect repair operation on the data amount reduction data.

  As a result, when processing by the large noise restoration unit 23c such as inpainting processing is performed, in order to reduce the amount of calculation, after generating data amount reduction data with reduced resolution before processing, the inpainting processing Perform defect repair operations.

  As a result, since the restoration processing time by the large noise restoration unit 23c can be shortened, the lack of data can be repaired with the limited processing time while minimizing the deterioration of the quality of the three-dimensional shape data.

  Further, in the face shape measuring apparatus 1 according to the present embodiment, the data synthesis unit 23d as the synthetic defect repair information generation unit applies to a portion of the small noise repair distance image 41 that has not been repaired by the small noise repair unit 23a. By combining the large noise repair distance image 43, a repair distance image 44 as composite defect repair information is generated.

  As a result, expansion / reduction that can limit the portion to be processed by the large noise restoration unit 23c such as in-painting processing that lowers the resolution to the minimum necessary portion and maintain the resolution of the other portions high. Processing by the small noise restoration unit 23a such as contraction processing is performed. Therefore, it is possible to suppress a decrease in the three-dimensional measurement accuracy due to a reduction in the resolution of the entire screen at the time of restoration.

  Further, in the face shape measuring apparatus 1 according to the present embodiment, the mask generation unit 23b as the mask data generation unit generates smoothing information obtained by smoothing the small noise repair distance image 41, and further the small noise repair distance image 41. And the smoothing information are generated, and the mask image A42a or the mask image B42b is generated using the difference information.

  That is, a portion where data is lost, that is, a portion where data jumps is a portion where data is discontinuously and extremely changed relative to the surroundings, that is, a portion such as an extremely steep mountain or valley. Therefore, when a smoothing process is performed on such a characteristic portion, the amount of change with respect to the original data increases. For this reason, by taking the difference between the original image and the smoothed image, it is possible to emphasize the portion that could not be restored by the small noise restoration unit 23a. By using this enhanced data, the mask image A42a or the mask image B42b can be created. Therefore, it is possible to clarify the processing range by the large noise restoration unit 23c.

  In the face shape measuring apparatus 1 according to the present embodiment, the small noise repair unit 23a repairs a defect included in the distance image 34 by using expansion processing and contraction processing to generate a small noise repair distance image 41. The large noise restoration unit 23c uses the inpainting process to repair a portion of the small noise restoration distance image 41 that has not been restored by the small noise restoration unit 23a.

  Thereby, it is possible to easily repair a small omission using the expansion process and the contraction process by the small noise repair unit 23a. Further, when processing by the large noise restoration unit 23c such as inpainting processing is performed, in order to reduce the amount of calculation, it is possible to limit the range and to process a large defect highly.

  Further, in the face shape measuring apparatus 1 according to the present embodiment, the large noise repair distance image 43 has a smaller bit depth or image size than the small noise repair distance image 41. Thereby, time can be shortened in the inpainting process by the large noise restoration part 23c.

  Further, in the face shape measuring apparatus 1 according to the present embodiment, the data synthesis unit 23d smoothes the boundary after the synthesis of the small noise restoration distance image 41 and the large noise restoration distance image 43. As a result, when synthesizing images with different bit depths with respect to the synthesized image, it is possible to obtain high-quality repair data without a sense of incongruity by blurring the boundary.

  In the face shape measuring apparatus 1 according to the present embodiment, the mask generation unit 23b performs an expansion process (S23) between the difference process (S22) and the binarization process (S25) to enlarge the mask area. Is going. That is, when there is no expansion process, a gap is formed in the mask due to fine random components in the large noise. On the other hand, by performing the expansion process, it is possible to fill the gap of the mask and make the mask of the large noise portion into a correct shape. As a result, it is possible to correct large noise correctly in the inpainting process.

  In the face shape measuring apparatus 1 according to the present embodiment, the measurement target is a human face, and the three-dimensional shape data of the surface of the human face is measured. Thereby, the three-dimensional shape data of the face surface of a person with high measurement quality can be obtained.

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIG. Configurations other than those described in the present embodiment are the same as those in the first embodiment. For convenience of explanation, members having the same functions as those shown in the drawings of the first embodiment are given the same reference numerals, and explanation thereof is omitted.

  The three-dimensional measuring apparatus according to the first embodiment relates to the face shape measuring apparatus 1 whose photographing object is the face F. However, the three-dimensional measuring apparatus according to the present embodiment is different in that the imaging target is a body shape measuring apparatus.

  The configuration and operation of the body shape measuring apparatus 2 as the three-dimensional measuring apparatus according to the present embodiment will be described with reference to FIG. FIG. 11 is a block diagram showing a configuration of the body shape measuring apparatus 2 as the three-dimensional measuring apparatus according to the present embodiment.

  As shown in FIG. 11, the body shape measuring apparatus 2 as the three-dimensional measuring apparatus of this embodiment acquires 3D shape data of the body B using a stereo camera 11 and is used for measuring the size of clothes. The

  The body shape measurement apparatus 2 includes a stereo camera 11 as an imaging unit, and captures a whole body image, a half body image, or a specific body part as a body B of a person to be imaged, and the left camera image 31L and the right camera. An image 31R is acquired. The distance image 34 is generated from these images by the same method as described in the first embodiment.

  At that time, an unmeasurable region may occur in the distance image 34 due to imaging noise, shine or shadow caused by sunlight or illumination light, or occlusion. For this non-measurable region, small noise repair, mask generation, large noise repair, and synthesis are performed by the same method as described in the first embodiment. Thereby, the non-measurable region of the distance image 34 is repaired, and the three-dimensional shape data of the body B to be imaged can be generated.

  The generated three-dimensional shape data of the body B can be used for applications such as size adjustment and customization of clothes, clothing, medical equipment, sports equipment, and the like by a program installed in the body shape measuring apparatus 2.

  As described above, the body shape measuring apparatus 2 according to the present embodiment can repair missing data while minimizing the deterioration in the quality of the three-dimensional shape data of the body B in a limited processing time.

  Other parts and operations are the same as those in the first embodiment, and a description thereof will be omitted.

[Embodiment 3]
The following will describe still another embodiment of the present invention with reference to FIG. Configurations other than those described in the present embodiment are the same as those in the first and second embodiments. For convenience of explanation, members having the same functions as those shown in the drawings of Embodiment 1 and Embodiment 2 described above are given the same reference numerals, and descriptions thereof are omitted.

  The three-dimensional measuring apparatus according to the second embodiment relates to the body shape measuring apparatus 2 whose imaging target is the body B. However, the three-dimensional measuring apparatus according to the present embodiment is different in that it is the monitoring camera apparatus 3 whose photographing object is an unspecified large number of persons M.

  The configuration and operation of the monitoring camera device 3 as the three-dimensional measurement device of the present embodiment will be described with reference to FIG. FIG. 12 is a block diagram illustrating a configuration of the monitoring camera device 3 as the three-dimensional measurement device of the present embodiment.

  As shown in FIG. 12, the monitoring camera device 3 as the three-dimensional measuring device according to the present embodiment acquires three-dimensional shape data of an unspecified number of persons M using a stereo camera 11 and monitors a suspicious person. It is used for that.

  The surveillance camera device 3 is an imaging device including a stereo camera 11 as an imaging unit. The stereo camera 11 is attached to a predetermined place outdoors or indoors. The stereo camera 11 takes an unspecified number of persons M that fall within the field of view of the stereo camera 11 and continuously captures the left camera image 31L and the right camera image 31R. get. The distance image 34 is generated from these images by the same method as described in the first embodiment.

  At that time, an unmeasurable region may occur in the distance image 34 due to imaging noise, shine or shadow caused by sunlight or illumination light, or occlusion. For this non-measurable region, small noise repair, mask generation, large noise repair, and synthesis are performed by the same method as described in the first embodiment. Thereby, the non-measurable region of the distance image 34 is repaired, and the three-dimensional shape data of the unspecified large number of persons M to be photographed can be generated.

  The generated three-dimensional shape data is collated with a database in the monitoring camera device 3. If the data conforms to an abnormal pattern registered in advance or does not conform to a normal pattern, it is determined that some of the unspecified large number of persons M to be imaged are suspicious persons, and an abnormal signal is transmitted. Output to the outside.

  In the surveillance camera device 3 according to the present embodiment, the distance image 34 is continuously acquired, so that the data amount becomes enormous as it is. For this reason, the amount of data is reduced as much as possible, for example, by reducing the image size to the minimum required for person verification or by compressing the generated three-dimensional data. .

  As described above, in the monitoring camera device 3 according to the present embodiment, data loss is achieved while minimizing the deterioration in the quality of the three-dimensional shape data of some persons in the unspecified large number of persons M in a limited processing time. Can be repaired.

  Other parts and operations are the same as those in the first embodiment, and a description thereof will be omitted.

[Embodiment 4]
Control blocks of the face shape measuring device 1 in the first embodiment, the body shape measuring device 2 in the second embodiment, and the monitoring camera device 3 in the third embodiment as the three-dimensional measuring device of the present invention, particularly the distance image generating unit 22. The restoration unit 23 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).

  In the latter case, the face shape measuring device 1, the body shape measuring device 2, and the surveillance camera device 3 as a three-dimensional measuring device include a CPU that executes instructions of a program that is software that realizes each function, the program, and various data. A ROM (Read Only Memory) or a storage device (these are referred to as “recording media”) recorded so as to be readable by a computer (or CPU), a RAM (Random Access Memory) for developing the program, and the like are provided. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[Summary]
The three-dimensional measuring device (face shape measuring device 1, body shape measuring device 2, monitoring camera device 3) according to aspect 1 of the present invention is a three-dimensional measuring unit (stereo camera 11) that acquires three-dimensional measurement information (distance image 34). ) And first repair means (small noise repair unit 23a) for repairing the defect included in the three-dimensional measurement information (distance image 34) and generating first defect repair information (small noise repair distance image 41). And mask data (mask image A42a or mask image) obtained by extracting a portion of the first defect repair information (small noise repair distance image 41) that has not been repaired by the first repair means (small noise repair unit 23a). B42b) for generating mask data (mask generator 23b), the first defect repair information (small noise repair distance image 41), and the mask data (mask image A42a). Using the mask image B42b) to repair the defect of the first defect repair information (small noise repair distance image 41) that has not been repaired by the first repair means (small noise repair unit 23a), Second repair means (large large repair distance image 43) for generating second defect repair information (large noise repair distance image 43) having a smaller data amount per unit measurement area than the first defect repair information (small noise repair distance image 41). The noise repair unit 23c), the first defect repair information (small noise repair distance image 41), and the second defect repair information (large noise repair distance image 43) are combined to produce composite defect repair information (repair distance image 44). ) And a synthetic deficiency repair information generating means (data synthesizing unit 23d).

  According to the above invention, the three-dimensional measurement means acquires three-dimensional measurement information such as a human face. At this time, a large gap may occur in the three-dimensional measurement information simultaneously with a small gap due to noise or the like. In this case, if a large omission is processed by simple expansion / contraction as in the case of a small omission, the shape data becomes unnatural. On the other hand, if advanced processing such as in-painting processing that can repair large omissions is performed, the quality of the image after processing increases, but the amount of computation required for the processing is large, so the processing time becomes enormous. have.

  Therefore, in the present invention, for example, a small missing portion is repaired by using the first repairing means that processes by expansion / contraction, for example. Next, in the present invention, a large omission is repaired by using second repairing means such as an inpainting process. Thus, in the present invention, since the first repairing means for repairing a small defect and the second repairing means for repairing a large defect are repaired in combination, both the small defect and the large defect can be repaired. it can.

  Here, for example, when a large omission is repaired using the second repairing means such as an inpainting process, the processing time is enormous as described above, because the amount of computation required for the process is large. Therefore, for example, it is conceivable to reduce the resolution of the entire screen of the first defect repair information and perform the process by the second repair means. However, the quality of the finally obtained image is deteriorated.

  Therefore, in the present invention, mask data is generated by extracting a portion of the first defect repair information that has not been repaired by the first repair unit, and the second repair unit includes the first defect repair information and the mask data. Are used to repair a defect in a portion of the first defect repair information that has not been repaired by the first repair means, thereby generating second defect repair information. Therefore, since the repair range of the second repair means is limited to the mask data range, the repair time can be shortened.

  Thereafter, in the present invention, the synthetic defect repair information generation means generates the composite defect repair information by synthesizing the first defect repair information and the second defect repair information, so that both the small defect and the large defect are repaired. Can be obtained.

  As described above, in the present invention, first, a small defect that can be repaired by the first repairing means such as expansion / contraction processing is repaired, and only the mask data portion that cannot be repaired by the expansion / contraction processing is in-painted. The processing by the second repairing means such as the processing is applied, and finally, they are synthesized to obtain the three-dimensional measurement data in which the defect is repaired.

  As a result, the portion to be processed by the second restoration means such as in-painting processing is limited to the minimum necessary mask data portion, and the processing by the first restoration means such as expansion / contraction processing is performed at other portions. As a result, it is possible to suppress a decrease in the three-dimensional measurement accuracy due to a reduction in resolution as a whole at the time of restoration, and it is possible to repair a deficiency in the three-dimensional measurement information at a high speed by preventing an enormous amount of calculation.

  Therefore, it is possible to provide a three-dimensional measuring apparatus capable of repairing data loss while suppressing deterioration in quality of three-dimensional shape data with a limited processing time.

  The three-dimensional measuring device (face shape measuring device 1, body shape measuring device 2, monitoring camera device 3) according to aspect 2 of the present invention is the same as the three-dimensional measuring device according to aspect 1 (face shape measuring device 1, body shape measuring device 2, In the monitoring camera device 3), the defect repair calculation by the second repair means (large noise repair section 23c) is more unit data than the defect repair calculation by the first repair means (small noise repair section 23a). When the amount of calculation per unit is large, the second repairing unit (large noise repairing unit 23c) calculates the data amount per unit measurement region more than the first defect repairing information (small noise repairing distance image 41). After generating the reduced data amount reduction data, it is preferable to perform a defect repair operation on the data amount reduction data.

  That is, for example, the calculation of the defect repair by the second repairing unit such as the inpainting process has a calculation amount per unit data amount rather than the calculation of the defect repairing by the first repairing unit processed by, for example, expansion / contraction Is big. For this reason, in the present invention, the second repair means generates the data amount reduction data in which the data amount per unit measurement area is reduced as compared with the first loss repair information, and then the data is reduced with respect to the data amount reduction data. Perform repair operations.

  As a result, when processing is performed by the second restoration means such as inpainting processing, in order to reduce the amount of computation, after generating data amount reduction data with reduced resolution before processing, the inpainting processing Perform defect repair operations.

  As a result, the repair processing time by the second defect repairing means can be shortened, so that the data loss can be repaired with the limited processing time while minimizing the deterioration of the quality of the three-dimensional shape data.

  The three-dimensional measuring device (face shape measuring device 1, body shape measuring device 2, monitoring camera device 3) in aspect 3 of the present invention is the same as the three-dimensional measuring device (face shape measuring device 1, body shape measuring device in aspect 1 or 2). 2. In the monitoring camera device 3), the combined defect repair information generating means (data synthesizing unit 23d) includes the first repair means (small noise) in the first defect repair information (small noise repair distance image 41). It is preferable that the combined defect repair information (repair distance image 44) is generated by synthesizing the second defect repair information (large noise repair distance image 43) with the portion that has not been repaired by the repair unit 23a).

  In other words, in the present invention, the first defect repair information is used for the portion where the defect can be repaired by the first repair means, and the second repair means for the portion where the defect cannot be repaired by the first repair means. The synthesis defect repair information is generated by synthesizing so as to use the second defect repair information.

  As a result, the location where the processing by the second restoration means such as the inpainting processing that lowers the resolution is limited to the minimum necessary portion, and the other portions can be maintained at a high resolution. Processing by the first restoration means such as contraction processing is performed. Therefore, it is possible to suppress a decrease in the three-dimensional measurement accuracy due to a reduction in the resolution of the entire screen at the time of restoration.

  The three-dimensional measuring device (face shape measuring device 1, body shape measuring device 2, monitoring camera device 3) in aspect 4 of the present invention is the same as the three-dimensional measuring device (face shape measuring device 1, body shape in aspect 1, 2, or 3). In the measurement device 2 and the monitoring camera device 3), the mask data generation unit (mask generation unit 23b) generates smoothing information obtained by smoothing the first defect repair information (small noise repair distance image 41), Further, difference information between the first defect repair information (small noise repair distance image 41) and smoothing information is generated, and the mask data (mask image A42a or mask image B42b) is generated using the difference information. Is preferred.

  That is, a portion where data is lost, that is, a portion where data jumps is a portion where data is discontinuously and extremely changed relative to the surroundings, that is, a portion such as an extremely steep mountain or valley. Therefore, when a smoothing process is performed on such a characteristic portion, the amount of change with respect to the original data increases. For this reason, by taking the difference between the original image and the smoothed image, it is possible to emphasize the portion that could not be restored by the first restoration means. Mask data can be created by using this emphasized data. Therefore, it is possible to clarify the processing range by the second restoration means.

  The three-dimensional measuring device (face shape measuring device 1, body shape measuring device 2, monitoring camera device 3) according to aspect 5 of the present invention is the three-dimensional measuring device according to any one of aspects 1 to 4 (face shape measuring device 1, In the body shape measuring device 2 and the monitoring camera device 3), the first restoration means (small noise restoration unit 23a) uses expansion processing and contraction processing for defects included in the three-dimensional measurement information (distance image 34). The first defect repair information (small noise repair distance image 41) is generated by repairing, and the second repair unit (large noise repair unit 23c) is configured to restore the first defect repair information (small noise repair). In the distance image 41), it is preferable to repair a defect of a portion that has not been repaired by the first repairing means (small noise repairing portion 23a) using an inpainting process.

  As a result, the first repair means repairs the defect included in the three-dimensional measurement information using the expansion process and the contraction process to generate the first defect repair information, so that the small defect can be easily repaired. it can. In addition, the second repairing unit repairs a defect in a portion of the first defect repairing information that has not been repaired by the first repairing unit using an inpainting process. Thereby, when processing by the second repairing means such as inpainting processing is performed, the range is limited in order to reduce the amount of calculation, and a large defect can be processed at a high level.

  The three-dimensional measurement method according to the sixth aspect of the present invention includes a three-dimensional measurement step (S1 to S5) for acquiring three-dimensional measurement information (distance image 34), and a defect included in the three-dimensional measurement information (distance image 34). Of the first defect repair information (small noise repair distance image 41), the first repair process (S11 / S12) for repairing and generating first defect repair information (small noise repair distance image 41), and the first defect repair information (small noise repair distance image 41) A mask data generation step (S21 to S26) for generating mask data (mask image A42a or mask image B42b) obtained by extracting a portion that has not been repaired in the first repair step (S11 / S12); and the first defect repair described above. Using the information (small noise repair distance image 41) and the mask data (mask image A42a or mask image B42b), the first defect repair information (small noise repair distance image 41) That is, the defect of the part that was not repaired in the first repair process (S11 / S12) is repaired, and the amount of data per unit measurement area is smaller than that of the first defect repair information (small noise repair distance image 41). A second repair process (S31 / S32) for generating second defect repair information (large noise repair distance image 43), the first defect repair information (small noise repair distance image 41), and a second defect repair. And a combined defect repair information generation step (S41 to S43) for combining information (large noise repair distance image 43) to generate composite defect repair information (repair distance image 44).

  According to the above invention, it is possible to provide a three-dimensional measurement method capable of repairing data loss while suppressing deterioration in quality of three-dimensional shape data in a limited processing time.

  It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims, and technical means disclosed in different embodiments are appropriately combined. Embodiments obtained in this manner are also included in the technical scope of the present invention.

  The present invention relates to a three-dimensional measurement apparatus and a three-dimensional measurement method for measuring a three-dimensional shape of an object, and is used as an independent apparatus for industrial, consumer, and other uses including cleaning, monitoring, and automatic conveyance. In addition, it may be used by being incorporated in a part of another device, or part or all of the device may be used as an integrated circuit (IC).

1 Face shape measuring device (3D measuring device)
2 Body shape measuring device (3D measuring device)
3. Surveillance camera device (3D measuring device)
11 Stereo camera (three-dimensional measuring means)
DESCRIPTION OF SYMBOLS 12 Illuminating device 13 Data display part 14 Data storage part 20 Data processing circuit 21 Control part 22 Distance image generation part 23 Restoration part 23a Small noise restoration part (1st restoration means)
23b Mask generator (mask data generator)
23c Large noise restoration section (second restoration means)
23d Data synthesizing unit (synthesis defect repair information generating means)
24 Data converter 31L Left camera image 31R Right camera image 32L Parallelized left camera image 32R Parallelized right camera image 33 Parallax image 34 Distance image (three-dimensional measurement information)
41 Small noise repair distance image (first defect repair information)
42a Mask image A (mask data)
42b Mask image B (mask data)
43 Large noise repair distance image (second defect repair information)
44 Repair distance image (composite defect repair information)
45 3D point cloud data B Body D0 Camera calibration data F Face M Unspecified number of people

Claims (6)

3D measurement means for acquiring 3D measurement information;
First repair means for repairing a defect included in the three-dimensional measurement information and generating first defect repair information;
Mask data generating means for generating mask data obtained by extracting a portion of the first defect repair information that has not been repaired by the first repair means;
Using the first defect repair information and the mask data, the defect of the portion that has not been repaired by the first repair means in the first defect repair information is repaired, and more than the first defect repair information. A second repair means for generating second defect repair information with a small amount of data per unit measurement area;
3. A three-dimensional measuring apparatus comprising: synthetic defect repair information generating means for generating synthesized defect repair information by synthesizing the first defect repair information and the second defect repair information. When the amount of calculation per unit data amount is larger than the operation of defect repair by the first repair unit, the operation of defect repair by the second repair unit,
The second repair means generates data amount reduction data in which the data amount per unit measurement area is reduced as compared with the first defect repair information, and then performs a defect repair operation on the data amount reduction data. The three-dimensional measuring apparatus according to claim 1. The synthetic defect repair information generating means includes
The synthetic defect repair information is generated by synthesizing the second defect repair information with respect to a portion of the first defect repair information that has not been repaired by the first repair means. The three-dimensional measuring apparatus according to 1 or 2. The mask data generating means
Smoothing information obtained by smoothing the first defect repair information is generated, further difference information between the first defect repair information and the smoothing information is generated, and the mask data is generated using the difference information. The three-dimensional measuring apparatus according to claim 1, 2, or 3. The first repair means repairs a defect included in the three-dimensional measurement information using an expansion process and a contraction process to generate the first defect repair information,
2. The second repairing unit repairs a defect in a portion of the first defect repair information that has not been repaired by the first repairing unit using an inpainting process. The three-dimensional measuring apparatus of any one of -4. A 3D measurement process for obtaining 3D measurement information;
A first repairing step of repairing a defect included in the three-dimensional measurement information to generate first defect repair information;
A mask data generation step for generating mask data obtained by extracting a portion of the first defect repair information that has not been repaired in the first repair step;
Using the first defect repair information and the mask data, the defect of the portion of the first defect repair information that has not been repaired in the first repair process is repaired. A second repair step for generating second defect repair information with a small amount of data per unit measurement area;
A three-dimensional measurement method comprising: a synthetic defect repair information generation step of generating synthetic defect repair information by synthesizing the first defect repair information and the second defect repair information.