JP2006267031A - Three-dimensional input apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To shorten the time required for imaging in a technique for imaging an object and measuring the three-dimensional shape and surface color of the object. <P>SOLUTION: For measuring the three-dimensional shape of the object, the object is imaged by a CCD, and a pixel-thinned image is formed from results of this imaging by thinning out some of a plurality of pixels constituting the overall image displaying the object. On the basis of the extracted pixel-thinned image, the three-dimensional shape of the object is measured. For measuring the surface color of the same object, the object is imaged by the CCD continuously to the imaging for three-dimensional shape measurement, and a pixel-unthinned image is formed from results of this imaging without thinning out any of a plurality of pixels constituting the overall image displaying the object. On the basis of the extracted pixel-unthinned image, the surface color of the object is measured. It is thereby possible to shorten the time required for imaging and improve accuracy in texture mapping while securing high texture resolution. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for imaging an object and measuring the three-dimensional shape and surface color of the object, and more particularly to a technique for shortening the imaging time.

  A technique for inputting an object three-dimensionally already exists (for example, see Patent Document 1). According to an example of this technique, an object is imaged, and the three-dimensional shape and surface color of the object are measured by calculation based on the imaging result.

  A three-dimensional input device for implementing this kind of technology generally includes a projection unit that projects a plurality of types of pattern light onto an object, an imaging unit that images the object, and a processing unit that processes the imaging result. Configured as follows.

  Specifically, the imaging unit (a) measures an object for each projection of each pattern light during a period in which a plurality of types of pattern light is sequentially projected onto the object by the projection unit in order to measure the three-dimensional shape of the object. It is designed to perform a three-dimensional measurement imaging process for imaging, and (b) a surface color measurement imaging process for imaging an object in order to measure the surface color of the object.

  In this imaging unit, in general, in order to form one frame of an image of an object, exposure of the imaging unit with light incident on the imaging unit from the object, and a signal for extracting an electrical signal reflecting the incident light from the imaging unit Removal is performed in that order. The exposure and signal extraction performed for each frame of the image constitute one individual imaging process.

  Therefore, when it is necessary to form a plurality of image frames (for example, the same number of image frames as the above-described plurality of types of pattern lights) for the same object by the imaging unit, the exposure and the signal extraction are alternately performed. As described above, it is necessary to sequentially perform a plurality of individual imaging processes.

Specifically, the processing unit includes a three-dimensional measurement calculation process for measuring the three-dimensional shape of the object based on an imaging result of the three-dimensional measurement imaging process by the imaging unit, and an imaging result of the surface color measurement imaging process by the imaging unit. It is designed so as to perform surface color measurement calculation processing for measuring the surface color of the object based on it.
JP 2000-105108 A

  In this type of three-dimensional input device, usually, an individual imaging process is performed a plurality of times in the imaging unit for three-dimensional measurement of an object, and an individual imaging process is performed in the imaging unit for measuring the surface color of the same object. At least once. A three-dimensional shape of an object is measured using a result of a plurality of individual imaging processes for three-dimensional measurement, and the measurement result is defined in real space. The surface color of the object is measured using the result of at least one individual imaging process for surface color measurement, and the measurement result is also defined in the same real space.

  The actual three-dimensional shape of the object and the actual surface color are spatially related to each other. However, the measurement of the three-dimensional shape and the measurement of the surface color are each performed using the results of a plurality of individual imaging processes performed at different times.

  On the other hand, the longer the time required for the plurality of individual imaging processes, the higher the possibility that the object and the imaging unit are relatively displaced during the execution period of the plurality of individual imaging processes. As the degree of relative displacement increases, the object represented by the captured image used for three-dimensional measurement and the object represented by the captured image used for surface color measurement will be spatially displaced. Probability is high. The higher the possibility, the higher the possibility that the three-dimensional measurement result and the surface color measurement result will not be spatially mapped correctly.

  In order to improve the accuracy of mapping the surface color measurement result to the three-dimensional measurement result (hereinafter referred to as “texture mapping accuracy”), the object represented by the captured image used for the three-dimensional measurement, and the surface color It is desirable to suppress a spatial shift from an object represented by a captured image used for measurement. An example of a measure for suppressing such a spatial shift is to shorten the entire imaging time for three-dimensional measurement and surface color measurement.

  Based on the knowledge described above, the present invention has been made to reduce the imaging time in a technique for imaging an object and measuring the three-dimensional shape and surface color of the object.

  The following aspects are obtained by the present invention. Each aspect is divided into sections, each section is given a number, and is described in a form that cites other section numbers as necessary. This is to facilitate understanding of some of the technical features that the present invention can employ and combinations thereof, and the technical features that can be employed by the present invention and combinations thereof are limited to the following embodiments. Should not be interpreted. That is, it should be construed that it is not impeded to appropriately extract and employ the technical features described in the present specification as technical features of the present invention although they are not described in the following embodiments.

  Further, describing each section in the form of quoting the numbers of the other sections does not necessarily prevent the technical features described in each section from being separated from the technical features described in the other sections. It should not be construed as meaning, but it should be construed that the technical features described in each section can be appropriately made independent depending on the nature.

(1) A three-dimensional input device for three-dimensionally inputting an object,
A projection unit that projects multiple types of pattern light onto an object;
An imaging unit that images the object, and (a) each pattern light during a period in which the plurality of types of pattern light is sequentially projected onto the object by the projection unit in order to measure a three-dimensional shape of the object. A three-dimensional measurement imaging process for imaging the object for each projection of the image, and (b) a surface color measurement imaging process for imaging the object in order to measure the surface color of the object, and the imaging unit Three-dimensional measurement calculation processing for measuring the three-dimensional shape of the object based on the imaging result of the three-dimensional measurement imaging processing, and measuring the surface color of the object based on the imaging result of the surface color measurement imaging processing by the imaging unit And a processing unit for performing surface color measurement calculation processing.
The imaging unit is formed by thinning out a pixel thinned image formed by thinning out any of a plurality of pixels constituting an entire image representing the object from the imaging result of the object, and without thinning out any of the pixels. It is configured to be able to selectively extract a pixel non-decimated image,
The processing unit performs the three-dimensional measurement calculation process based on the pixel thinned image extracted from the imaging result of the three-dimensional measurement imaging process by the imaging unit, while the surface color measurement imaging process by the imaging unit. A three-dimensional input device that performs the surface color measurement calculation processing based on the pixel non-thinned image extracted from the imaging result.

  In this three-dimensional input apparatus, since three-dimensional measurement calculation processing is performed on an object, one of a plurality of pixels constituting the entire image representing the object is thinned out from the imaging result of the object in the imaging unit. A three-dimensional measurement imaging process is performed to extract the pixel thinned image. Based on the extracted pixel thinning image, a three-dimensional measurement calculation process for measuring the three-dimensional shape of the object is performed.

  In this three-dimensional input device, surface color measurement calculation processing is further performed on the same object, and therefore, in the imaging unit, any of a plurality of pixels constituting the entire image representing the object is detected from the imaging result of the object. Surface color measurement and imaging processing is performed to extract a pixel non-thinned image formed without subtraction. Based on the extracted pixel non-thinned image, surface color measurement calculation processing for measuring the surface color of the object is performed.

  Therefore, according to this three-dimensional input device, both the imaging processing for the three-dimensional measurement calculation processing and the imaging processing for the surface color measurement calculation processing are performed by the pixel non-decimated image extracted from the imaging result of the object in the imaging unit. Compared to the case where the image is used as a reference image, it is easy to shorten the extraction time required for extracting the reference image from the imaging result of the object in the imaging unit. On the other hand, the shorter the take-out time, the shorter the imaging time during which the imaging unit continues to capture an object.

  Therefore, according to this three-dimensional input device, since the imaging time is short as described above, the imaging unit regards the object as a moving image, assuming that the object is fixed in space at least instantaneously. Regardless of whether it is a still image or a still image, the three-dimensional shape of the object is measured even if the relative position between the imaging unit and the object changes unexpectedly during the period of shooting as one frame of the image. In addition to the reduction in accuracy, the accuracy in which the surface color measured for the same object is spatially mapped to the three-dimensional shape of the object does not need to be reduced.

  In addition, according to the three-dimensional input device, the imaging processing for the three-dimensional measurement calculation processing and the imaging processing for the surface color measurement calculation processing are performed by using the pixel thinned image extracted from the imaging result of the object in the imaging unit. Compared to the case where the image is used as a reference image, the accuracy with which the surface color of the object is measured in the imaging unit is not reduced. As a result, a high texture resolution is realized.

(2) After the imaging of the object, the imaging unit extracts the pixel thinned image from the imaging result in a time shorter than the time required to extract the pixel non-thinned image. Input device.

  According to this three-dimensional input device, after imaging of an object by the imaging unit (in the later-described embodiment, after exposure of the CCD), the time is shorter than the time required to extract a pixel non-decimated image from the imaging result. A pixel-thinned image is taken out and output from the imaging unit. That is, for each imaging, the time required for an electrical signal (image signal) representing the captured image to be output from the imaging unit is short for a pixel-thinned image, but for a non-pixel-thinned image It's long.

  On the other hand, in imaging processing for three-dimensional measurement calculation processing, imaging by the imaging unit is usually performed a plurality of times, whereas in imaging processing for surface color measurement calculation processing, by the imaging unit Imaging is normally performed a number of times less than the number of times of imaging in the imaging process for the three-dimensional measurement calculation process. Therefore, the imaging process and the surface color measurement for the three-dimensional measurement calculation process are shorter as the time required for outputting the image signal from the imaging unit for each imaging in the imaging process for the three-dimensional measurement calculation process is shorter. The time required for the entire imaging process including the imaging process for the arithmetic process can be shortened.

  Further, when the time required for outputting an image signal from the imaging unit for each imaging in the imaging process for the three-dimensional measurement calculation process is short, the imaging process for the three-dimensional measurement calculation process is performed. When the imaging process for the surface color measurement calculation is performed later, there is no need for a long time interval between the preceding imaging process and the subsequent imaging process. As a result, these two types of imaging processing can be performed continuously without a large time interval, which contributes to the reduction of the total imaging time described above.

  Therefore, according to the three-dimensional input device according to this section, the time required for the image signal to be output from the imaging unit for each imaging in the imaging process for the three-dimensional measurement calculation process is the surface color measurement calculation. In order to perform three-dimensional measurement and surface color measurement of an object on the assumption that the time required for the image signal to be output from the imaging unit in imaging in the imaging process for processing is set, the imaging unit The imaging time that must be continued is reduced. As a result, the texture mapping accuracy is improved.

(3) One time three-dimensional measurement imaging process by the imaging unit and one surface color measurement imaging process by the imaging unit constitute one whole imaging process for the object,
The imaging unit sequentially performs individual imaging processing a plurality of times according to the number of the plurality of types of pattern light in one three-dimensional measurement imaging processing, while at least once in one surface color measurement imaging processing. The three-dimensional input device according to (1) or (2), which performs individual imaging processing.

(4) The three-dimensional input apparatus according to (3), wherein the imaging unit performs the surface color measurement imaging process after the three-dimensional measurement imaging process.

  According to this three-dimensional input device, the time required for the image signal to be output from the imaging unit for each imaging in the imaging process for the three-dimensional measurement calculation process is the imaging for the surface color measurement calculation process. Preceding imaging processing for three-dimensional measurement calculation processing and subsequent imaging processing for surface color measurement calculation on the premise that the time required for outputting an image signal from the imaging unit in imaging in processing is shorter There is no long time interval between them. As a result, these two types of imaging processing can be performed continuously without a large time interval, which contributes to the reduction of the total imaging time described above.

(5) The three-dimensional input device according to (4), wherein the imaging unit performs the surface color measurement imaging process continuously without performing other processes following the three-dimensional measurement imaging process.

  According to this three-dimensional input device, the preceding imaging process for the three-dimensional measurement calculation process and the subsequent imaging process for the surface color measurement calculation are more continuous without passing through other processes. Done. As a result, it becomes easier to shorten the entire imaging time.

(6) The imaging unit performs a plurality of individual imaging processes in the three-dimensional measurement imaging process and at least one individual imaging process in the surface color measurement imaging process in that order in the same cycle (5 3D input device according to item.

(7) The imaging unit performs a plurality of individual imaging processes in the three-dimensional measurement imaging process and at least one individual imaging process in the surface color measurement imaging process in that order at a video rate (6 3D input device according to item.

(8) The imaging unit projects the object on which each pattern light is projected and takes out the pixel thinned image to perform the three-dimensional measurement imaging process, and the pattern light is projected. The three-dimensional image according to any one of (1) to (6), wherein the low-speed imaging mode in which the three-dimensional measurement imaging process is performed is performed by imaging the object and taking out the pixel non-decimated image. Input device.

  According to this three-dimensional input device, if the high-speed imaging mode is selected and executed, the relative displacement between the imaging unit and the object because the object is a moving object or the positional variation of the imaging unit is large. Although the environment is large, it is easy to input an object three-dimensionally with high texture mapping accuracy. On the other hand, if the low-speed imaging mode is selected and executed, the object is three-dimensionally input with high shape measurement accuracy and high texture mapping accuracy on the condition that the relative displacement between the imaging unit and the object is small. It becomes easy.

  Therefore, according to this three-dimensional input device, it is possible to change the mode for extracting an image from the imaging unit in accordance with the presence or absence of the dynamic behavior of the object to be imaged and the user's request and preference. Usability of the three-dimensional input device is improved.

  The switching between the “high-speed imaging mode” and the “low-speed imaging mode” in this section is performed, for example, by the three-dimensional input device in response to a command from the user, or the three-dimensional input device It is possible to perform automatically based on the imaging result (movement of the object) and the usage state (movement of the device) of the three-dimensional input device.

(9) The imaging unit adds the illuminance detection values of a plurality of target pixels belonging to each of the plurality of pixel groups into which the plurality of pixels are grouped, for each pixel group, and the added illuminance Any one of the items (1) to (8), in which the pixel-thinned image is extracted from the imaging result of the object by an addition method in which illuminance detection values of a plurality of target pixels belonging to each pixel group are evenly distributed using A three-dimensional input device according to claim 1.

(10) The imaging unit selects, for each pixel group, a representative pixel representing the target pixel from a plurality of target pixels belonging to each of the plurality of pixel groups in which the plurality of pixels are grouped. The pixel-thinned image is extracted from the imaging result of the object by a selection method in which the illuminance detection values of a plurality of target pixels belonging to each pixel group are set using the illuminance detection values of the selected representative pixels ( The three-dimensional input device according to any one of items 1) to (8).

(11) The three-dimensional input device according to any one of (1) to (10), wherein the three-dimensional input device is a handheld type.

(12) A three-dimensional input method for inputting an object three-dimensionally,
A projection step of projecting multiple types of pattern light onto an object;
An imaging step of imaging the object, wherein: (a) each pattern light during a period in which the plurality of types of pattern light is sequentially projected onto the object by the projection step in order to measure a three-dimensional shape of the object; Performing a three-dimensional measurement imaging process for imaging the object for each projection of the image, and (b) performing a surface color measurement imaging process for imaging the object in order to measure the surface color of the object. 3D measurement calculation processing for measuring the 3D shape of the object based on the imaging result of the 3D measurement imaging processing, and measuring the surface color of the object based on the imaging result of the surface color measurement imaging processing by the imaging process And a processing step for performing surface color measurement calculation processing.
In the imaging step, a pixel-thinned image formed by thinning out any of a plurality of pixels constituting the entire image representing the object, and any pixels are not thinned out from the imaging result of the object. Select pixel non-decimated image and
The processing step performs the three-dimensional measurement calculation processing based on the pixel thinned image extracted from the imaging result of the three-dimensional measurement imaging processing in the imaging step, while the surface color measurement imaging processing in the imaging step. A three-dimensional input method for performing the surface color measurement calculation processing based on the pixel non-thinned image extracted from the imaging result.

  If the three-dimensional input method according to this section is implemented, it is possible to realize basically the same function and effect according to basically the same principle as the three-dimensional input apparatus according to the above (1).

(13) A program executed by a computer to implement the three-dimensional input method described in (12).

  If the program according to this section is executed by a computer, it is possible to realize basically the same operation effect according to basically the same principle as that of the three-dimensional input device according to the above section (1).

  The program according to this section can be interpreted so as to include not only a combination of instructions executed by a computer to fulfill its function, but also files and data processed in accordance with each instruction.

  In addition, this program may achieve its intended purpose by being executed by a computer alone, or may be intended to achieve its intended purpose by being executed by a computer together with other programs. it can. In the latter case, the program according to this section can be mainly composed of data.

  Hereinafter, one of more specific embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a perspective view showing the appearance of a three-dimensional input device 10 according to an embodiment of the present invention. The three-dimensional input device 10 obtains the three-dimensional information and surface color information of the subject S by a computer based on the projection of the striped pattern light onto the subject S (object), the imaging of the subject S, and the imaging result. It is designed to perform signal processing.

  As shown in FIG. 1, the three-dimensional input device 10 is a handheld type, and the user can move it by hand. In addition, the three-dimensional input device 10 controls the subject S so that the three-dimensional shape information of the subject S and the surface color information are accurately matched to each other with respect to the pixel position, regardless of the camera shake or the movement of the subject S. Designed to image. Therefore, in FIG. 1, an animal turtle is shown as an example of the moving subject S.

  In order to perform projection onto the subject S, imaging of the subject S, and acquisition of three-dimensional information and surface color information of the subject S, the three-dimensional input device 10 includes a projection unit 12 as shown in FIG. And an imaging unit 14 and a processing unit 16.

  The three-dimensional input device 10 operates according to a mode selected by the user from among a plurality of types of modes. These modes include a SLOW mode, a FAST mode, and an off mode. The SLOW mode is a low-speed imaging mode in which the subject S is imaged with high accuracy at low speed, and is an imaging accuracy priority mode. The FAST mode is a high-speed imaging mode that images the subject S at high speed with low accuracy, and is an imaging time priority mode. The off mode is a mode in which the operation of the three-dimensional input device 10 is stopped.

  The imaging unit 14 images the subject S, and from the imaging result, a pixel thinned image formed by thinning out any of a plurality of pixels constituting the entire image representing the subject S, and all the pixels are thinned out. It is configured such that it is possible to selectively extract the non-pixel thinned image formed without any delay. Furthermore, the imaging unit 14 is configured to take out the pixel-thinned image from the imaging result after taking the subject S in a time shorter than the time required to take out the pixel non-thinned image.

  A CCD is known as an element for imaging the subject S. As an example of this CCD, there is a CCD of a type that performs progressive scanning by an interline transfer method.

  As shown in FIG. 17A, in this type of CCD, a plurality of photodiode rows arranged in the vertical direction are arranged in the horizontal direction, whereby a plurality of photodiodes as photoelectric conversion elements are arranged. Dimensionally arranged. In this type of CCD, a plurality of vertical transfer CCD columns arranged in the vertical direction are arranged in the horizontal direction, so that a plurality of CCD elements as charge storage / transfer elements are arranged two-dimensionally. ing. In this type of CCD, a horizontal transfer CCD row is arranged in common to the vertical transfer CCD columns and at the end of each vertical transfer CCD column. An output circuit as a charge outlet common to all CCD elements is connected to the end of the horizontal transfer CCD row.

  In this type of CCD, when each photodiode is exposed, a signal charge corresponding to the exposure amount is generated in the photodiode. In response to the read command pulse, the signal charge generated in each photodiode is transferred to and stored in a corresponding one of the plurality of CCD elements in the vertical transfer CCD array. Thereafter, as each horizontal transfer command pulse is generated, signal charges are transferred in units of one CCD element in each vertical transfer CCD column. At this time, each vertical transfer CCD column is transferred from its end. The signal charge is transferred to the horizontal transfer CCD row. Further, as each horizontal transfer command pulse is generated, signal charges are transferred in units of one CCD element in the horizontal transfer CCD row, and at this time, signal charges are output from the end of the horizontal transfer CCD row. Is output. In parallel with the vertical transfer operation and the horizontal transfer operation, photoelectric conversion of an image in the photodiode is performed.

  17A to 17D show the transfer operation in this type of CCD in time series. FIG. 17A shows the movement of signal charges in the vertical blanking period, FIG. 17B shows the movement of signal charges in the first row horizontal blanking period, and FIG. The signal charge movement in the row horizontal transfer period is shown, and FIG. 17D shows the signal charge movement in the second row horizontal blanking period.

  In order to explain the transfer principle in this kind of CCD, FIG. 18 shows the timing of the voltage change of the output signal of the output circuit, the changes of the φ1 transfer pulse signal and the φ2 transfer pulse signal, and the change of the reset pulse signal. It is represented by a chart. Further, FIG. 19A shows a cross-sectional view of the structure of the CCD. Further, FIG. 19B shows how the potential distribution in each part of the CCD changes in association with each of the times T1 to T4 shown in FIG.

  As shown in FIG. 19A, the CCD includes a transfer channel, a φ1 transfer gate, a φ2 transfer gate, an output channel, a reset drain, a reset gate, and an output amplifier. Among them, the output channel, the reset drain, the reset gate, and the output amplifier constitute the aforementioned output circuit. The output amplifier is configured as a source follower circuit using FETs. The φ1 transfer pulse signal shown in FIG. 18 is output to the φ1 transfer gate, the φ2 transfer pulse signal is output to the φ2 transfer gate, and the reset pulse signal is output to the reset gate.

  As shown in FIG. 18, when the φ1 transfer pulse signal transitions from OFF to ON, as shown in FIG. 19, the potential of the current transfer channel corresponding to the φ1 transfer gate is raised (in FIG. 19B, it is lowered). As a result, charge is transferred from the transfer channel preceding the current transfer channel to the current transfer channel. As shown in FIG. 18, signal changes are given to the φ1 transfer gate and φ2 transfer gate in opposite phases, and as a result, as shown in FIG. Charge is transferred from the side to the downstream side.

  In this way, when the signal charge is transferred progressively in the horizontal transfer CCD row and reaches the end thereof, the signal charge is output to the output amplifier via the output channel. In the output amplifier, the signal is converted into a signal voltage having a potential corresponding to the signal charge.

  The signal output speed of the CCD is determined by the signal conversion speed in the output amplifier. Generally, the upper limit of the time from when the potential of the φ1 transfer gate is raised until the signal voltage is secured as shown in FIG. 18 is about 0.1 μs. Therefore, it takes about 0.5 s to output signal charges for all pixels from a CCD having 5 million pixels. Further, if the output pixel is VGA resolution (640 * 480 pixels), a time of 0.0307 s is required. This satisfies the moving picture output rate of 30 frames / second.

  As shown in FIG. 18, when the reset pulse signal transitions from OFF to ON, as shown in FIG. 19B, the signal charge moves from the output channel to the reset drain and is erased. Such a charge erasing circuit is provided in the aforementioned output circuit as a circuit common to a plurality of CCD elements. On the other hand, a similar charge erasing circuit is provided for each of the plurality of photodiodes. As a result, among the photocharges generated in the photodiodes, the charges generated for the pixels that are not selected and are not selected are vertical. It is erased directly at the most upstream position without being transferred to the transfer CCD column. This action enables selective pixel thinning, which will be described later.

  In order to explain the principle of selective transfer of this kind of CCD, FIG. 20A shows the signal change of the vertical transfer clock line V1, the signal change of the horizontal transfer clock line H1, and the horizontal transfer clock line H2. Each signal change is represented by a timing chart. Further, FIG. 20B shows a cross-sectional view of the structure of the CCD. Further, FIG. 20C shows how the potential distribution in each part of the CCD changes in association with each of times T5 and T6 shown in FIG.

  As shown in FIG. 20A, in the example in which the horizontal transfer clock line H1 is enabled, when the vertical transfer clock line V1 transitions from ON to OFF between times T5 and T6, FIG. And as shown on the left side in (c), the signal charge moves from the vertical transfer channel to the horizontal transfer channel. On the other hand, as shown in FIG. 20A, in the example in which the horizontal transfer clock line H2 is disabled, when the vertical transfer clock line V1 transitions from on to off between times T5 and T6, As shown on the right side of 20 (b) and 20 (c), the signal charge stays in the vertical transfer channel without moving from the vertical transfer channel to the horizontal transfer channel.

  Therefore, according to this CCD, it is possible to selectively transfer signal charges, and by using this, it is possible to thin out pixels by adding signal charges.

  As described above, in the field of imaging using this type of CCD, an addition method and a selection method are already known in order to extract a pixel-thinned image from the imaging result of the subject S.

  According to the addition method, the illuminance detection values of a plurality of target pixels belonging to each of a plurality of pixel groups in which a plurality of pixels constituting the entire image representing the subject S are grouped are added for each pixel group. The detected illuminance values of a plurality of target pixels belonging to each pixel group are evenly distributed using the added illuminance.

  On the other hand, according to the selection method, a representative pixel representing the target pixel is selected for each pixel group from a plurality of target pixels belonging to each of the plurality of pixel groups, and the selected representative pixel The detected illuminance values of a plurality of target pixels belonging to each pixel group are evenly distributed using the detected illuminance values.

  In the present embodiment, the imaging unit 14 is designed to extract a pixel-thinned image from the imaging result of the subject S in accordance with a preselected one of the addition method and the selection method.

  The thinned-out image processing mode for extracting a pixel-thinned image from the imaging result of the subject S is suitable for a low-speed imaging mode in which the subject S is imaged with high accuracy at low speed. On the other hand, the non-decimated image processing mode for extracting the pixel non-decimated image from the imaging result of the subject S is suitable for the high-speed imaging mode for imaging the subject S at high speed with low accuracy.

  Therefore, in this embodiment, when the FAST mode is selected by the user, the thinned image processing mode is set, while when the SLOW mode is selected by the user, the non-thinned image processing mode is set.

  As shown in FIG. 1, the three-dimensional input apparatus 10 includes a casing 20 having a substantially box shape.

  In the casing 20, as shown in FIG. 2, the projection part 12 and the imaging part 14 are each arrange | positioned in the state accommodated in the subcasing shown with a broken line. As shown in FIG. 1, the casing 20 includes a lens barrel 24 and a flash 26 in a posture in which each is partially exposed in front of the casing 20. The casing 20 further includes an imaging optical system 30 that is a part of the imaging unit 14 in a posture in which a part of the lens is exposed in front of the casing 20. The imaging optical system 30 receives image light representing the subject S at the exposed portion thereof.

  The lens barrel 24 protrudes from the front surface of the casing 20 as shown in FIG. 1, and accommodates a projection optical system 32 that is a part of the projection unit 12 as shown in FIG. The lens barrel 24 holds the projection optical system 32 in a state where the projection optical system 32 can be moved as a whole for focus adjustment. Further, the lens barrel 24 protects the projection optical system 32 from damage. A part of the lens of the projection optical system 32 that is a part of the projection unit 12 is exposed from the exposed end face of the lens barrel 24. The projection optical system 32 projects pattern light toward the subject S at the exposed portion.

  The flash 26 is a light source that emits light to supplement the insufficient light amount, and is configured using, for example, a discharge tube filled with xenon gas. Therefore, the flash 26 can be repeatedly used by discharging a capacitor (not shown) built in the casing 20.

  The casing 20 further includes a release button 40, a mode changeover switch 42, and a monitor LCD 44 on the upper surface thereof.

  The release button 40 is operated by the user to operate the three-dimensional input device 10. The release button 40 is composed of a two-stage push button type switch that can issue different commands depending on whether the user's operation state (pressed state) is “half-pressed state” or “full-pressed state”. ing. The operation state of the release button 40 is monitored by the processing unit 16. When the “half-pressed state” is detected by the processing unit 16, well-known auto focus (AF) and automatic exposure (AF) functions are activated, and the focus, aperture, and shutter speed are automatically adjusted. On the other hand, if the “fully pressed state” is detected by the processing unit 16, imaging or the like is performed.

  The mode changeover switch 42 is operated by the user to set the operation mode of the three-dimensional input apparatus 10 as one of a plurality of modes including the SLOW mode, the FAST mode, and the off mode. The operation state of the mode changeover switch 42 is monitored by the processing unit 16. When the operation state of the mode changeover switch 42 is detected by the processing unit 16, the process in the mode corresponding to the detected operation state is 3. This is performed in the dimension input device 10.

  The monitor LCD 44 is configured using a liquid crystal display, and receives an image signal from the processing unit 16 and displays an image to the user. The monitor LCD 44 displays, for example, an image (stereoscopic image) representing the detection result of the three-dimensional shape of the subject S.

  As shown in FIG. 1, the casing 20 further includes an antenna 50 as an RF (wireless) interface.

  The antenna 50 is connected to the RF driver 52 as shown in FIG. The antenna 50 wirelessly transmits the above-described data representing a stereoscopic image to an external interface (not shown) via the RF driver 52.

  FIG. 2 conceptually shows the internal configuration of the casing 20. As described above, the casing 20 mainly includes the projection unit 12, the imaging unit 14, and the processing unit 16.

  The projection unit 12 is a unit for projecting pattern light onto the subject S. As shown in FIG. 2, the projection unit 12 includes a substrate 60, a plurality of LEDs 62 (hereinafter referred to as “LED array 62A”), a light source lens 64, a projection LCD 66, and a projection optical system. 32 in series along the projection direction. Details of the projection unit 12 will be described later with reference to FIG.

  The imaging unit 14 is a unit for imaging the subject S. As shown in FIG. 2, the imaging unit 14 includes an imaging optical system 30 and a CCD (Charge Coupled Device) 70 in series along the incident direction of image light.

  As shown in FIG. 2, the imaging optical system 30 is configured using a plurality of lenses. The imaging optical system 30 automatically adjusts a focal length and a diaphragm by a well-known autofocus function to form an image of external light on the CCD 70.

  The CCD 70 is configured by arranging photoelectric conversion elements such as photodiode elements in a matrix. The CCD 70 generates a signal corresponding to the color and illuminance of the image formed on the surface of the CCD 70 via the imaging optical system 30 for each pixel. The generated signal is converted into digital data and output to the processing unit 16.

  As shown in the block diagram of FIG. 5, the processing unit 16 is electrically connected to the flash 26, the release button 40, and the mode changeover switch 42. Further, the processing unit 16 is electrically connected to the monitor LCD 44 via the monitor LCD driver 72, to the antenna 50 via the RF driver 52, and to the battery 74 via the power interface 76.

  The processing unit 16 is further electrically connected to the external memory 78 and the cache memory 80, respectively. The processing unit 16 is further electrically connected to the LED array 62A via the light source driver 84, to the projection LCD 66 via the projection LCD driver 86, and to the CCD 70 via the CCD interface 88. The flash 26 and the like are controlled by the processing unit 16.

  The external memory 78 is a detachable flash ROM, and can store captured images and three-dimensional information captured in the stereoscopic image mode. In order to configure the external memory 78, for example, an SD card, a compact flash (registered trademark) card, or the like can be used.

  The cache memory 80 is a storage device that can read and write data at high speed. The cache memory 80 is used, for example, to enable a captured image captured in the digital camera mode to be transferred to the cache memory 80 at a high speed and stored in the external memory 78 after being processed by the processing unit 16. The In order to configure the cache memory 80, for example, SDRAM, DDRRAM, or the like can be used.

  The power supply interface 76, the light source driver 84, the projection LCD driver 86, and the CCD interface 88 are constituted by various ICs (Integrated Circuits) that control the battery 74, the LED array 62A, the projection LCD 66, and the CCD 70, respectively.

  Here, details of the projection unit 12 will be described with reference to FIG. FIG. 3A is an enlarged view of the projection unit 12, and FIG. 3B is a front view of the light source lens 64.

  As described above, the projection unit 12 moves the substrate 60, the LED array 62A, the light source lens 64, the projection LCD 66, and the projection optical system 32 in the pattern light projection direction, as shown in FIG. It is equipped in series along.

  When the LED array 62A is attached to the substrate 60, electrical wiring is performed between the substrate 60 and the attached LED array 62A. As the substrate 60, for example, an insulating synthetic resin is applied to an aluminum substrate and then a pattern is formed by electroless plating, or a single layer or multilayer structure substrate having a glass epoxy base as a core is used. Can be produced.

  The LED array 62 </ b> A is a light source that emits radial light toward the projection LCD 66. In this LED array 62A, a plurality of LEDs (light emitting diodes) 62 are bonded on a substrate 60 via a silver paste in a staggered arrangement as shown in FIG. The substrate 60 and the plurality of LEDs 62 are electrically connected via bonding wires. The effect obtained by arranging the plurality of LEDs 62 in a staggered manner will be described in detail later with reference to FIG.

  As described above, in the present embodiment, since a plurality of LEDs 62 are used as the light source of the projection unit 12, electricity is converted into light as compared with the case where an incandescent bulb, a halogen lamp, or the like is used as the light source. Therefore, it is possible to easily improve the electro-optical conversion efficiency and suppress the generation of infrared rays and ultraviolet rays. Therefore, it is possible to easily achieve power saving, long life, and suppression of heat generation of the three-dimensional input device 10.

  As described above, the LED 62 has an extremely low heat ray generation rate as compared with a halogen lamp or the like, and therefore, a synthetic resin lens can be used for the light source lens 64 and the projection optical system 32. Therefore, the light source lens 64 and the projection optical system 32 can be made cheaper and lighter than when a glass lens is employed.

  Furthermore, in this embodiment, each LED 62 constituting the LED array 62A emits light of the same color, and specifically, an amber color using four elements of Al, In, Ga, and P as materials. It is configured to emit light. Therefore, it is not necessary to consider correction of chromatic aberration, which is a problem that must be taken into account when light of a plurality of colors is emitted. Therefore, it is also necessary to employ an achromatic lens in the projection optical system 32 in order to correct chromatic aberration. Absent. As a result, it is easy to provide a simple surface configuration and an inexpensive material projection means.

  Furthermore, in this embodiment, since the LED 62 is an amber LED of a four-element material that has an electro-optical conversion efficiency of about 80 [lumen / W] compared to other emission colors, a three-dimensional input is used. The device 10 can easily achieve high brightness, power saving, and long life.

  Specifically, in this embodiment, the LED array 62A is composed of 59 LEDs 62, and each LED 62 is driven at 50 [mW] (20 [mA], 2.5 [V]). The entire LED 62 is driven with power consumption of about 3 [W].

  Further, in the present embodiment, the brightness as the light flux value when the light emitted from each LED 62 passes through the light source lens 64 and the projection LCD 66 and is emitted from the projection optical system 32 is the case where the entire surface is irradiated. Is also set to about 25 ANSI lumens.

  In this embodiment, since the brightness of the emitted light from the projection unit 12 of the three-dimensional input device 10 is selected to that extent, for example, when the subject S is a human or animal face, the subject S Even if pattern light is projected onto the subject S in order to detect a three-dimensional shape, the subject S does not have to be dazzled. Therefore, according to the present embodiment, when the subject S is a person or an animal, it is easy to detect the three-dimensional shape of the subject S in a state where the subject S does not close his eyes.

  As shown in FIG. 3, the light source lens 64 is a lens that collects light emitted radially from the LED array 62A, and the material thereof is an optical resin typified by acrylic.

  As shown in FIG. 3A, the light source lens 64 includes a plurality of convex lens portions 90, a base portion 92 that supports the lens portions 90, an epoxy sealing material 94, and a plurality of positioning pins. 96.

  As shown in FIG. 3A, each lens unit 90 is projected from the base unit 92 toward the projection LCD 66 at a position facing the LEDs 62 of the LED array 62 </ b> A in the base unit 92. The epoxy sealing material 94 is filled in the concave portion 98 in which the LED array 62 </ b> A of the base portion 92 is to be hermetically accommodated, whereby the LED array 62 </ b> A is sealed in the concave portion 98. The epoxy sealing material 94 has a function of sealing the LED array 62A, and further has a function of bonding the substrate 60 and the light source lens 64 to each other.

  As shown in FIG. 3A, the plurality of positioning pins 96 protrude from the light source lens 64 toward the substrate 60 in order to relatively position the light source lens 64 and the substrate 60. It is installed. As shown in FIG. 3 (b), some of the plurality of positioning pins 96 are inserted into the long holes 100 formed in the substrate 60, while the rest are true holes formed in the substrate 60. The light source lens 64 is inserted into the circular hole 102, thereby fixing the light source lens 64 to the substrate 60 in a proper position.

  Thus, in this embodiment, since the light source lens 64, the LED array 62A, and the substrate 60 are spatially packed in the projection direction and stacked on each other, the assembly of the light source lens 64 and the like is compact. And space saving become easy.

  Further, in the present embodiment, the substrate 60 fulfills an additional function of holding the light source lens 64 in addition to the basic function of holding the LED array 62A. Therefore, according to the present embodiment, the addition of components that exclusively hold the light source lens 64 can be omitted, and as a result, the number of components of the three-dimensional input device 10 can be easily reduced.

  Furthermore, in this embodiment, as shown in FIG. 3A, the lens units 90 are arranged so as to face each LED 62 of the LED array 62A in a one-to-one relationship. Therefore, the radial light emitted from each LED 62 is efficiently condensed by each lens unit 18 facing each LED 62, and irradiated to the projection LCD 66 as radiation having high directivity as shown in FIG. Is done.

  The reason why the directivity is improved in this way is that, if light is incident on the projection LCD 66 substantially perpendicularly, the transmittance unevenness in the plane of the projection LCD 66 is suppressed and the image quality can be improved.

  The projection optical system 32 is a plurality of lenses for projecting light that has passed through the projection LCD 66 toward the subject S. These lenses are constituted by a telecentric lens composed of a combination of a glass lens and a synthetic resin lens. Telecentric means a configuration in which the principal ray passing through the projection optical system 32 is parallel to the optical axis in the incident side space and the position of the entrance pupil is infinite.

  Since the projection optical system 32 has a telecentric characteristic as described above and its incident NA is about 0.1, only light within a vertical ± 5 ° can pass through the diaphragm inside the projection optical system 32. The optical path of the projection optical system 32 is restricted.

  Therefore, in the present embodiment, due to the telecentricity of the projection optical system 32, it is easy to improve the image quality in combination with the configuration in which only light passing through the projection LCD 66 at the vertical ± 5 ° can be projected onto the projection optical system 32 It can be planned.

  Therefore, in this embodiment, in order to improve the image quality, the light emission angles from the respective LEDs 62 are aligned so that the light emitted from the respective LEDs 62 enters the projection LCD 660 substantially perpendicularly, and the light emitted from the respective LEDs 62 is provided. It is important that most of the light is incident on the projection optical system 32 within an incident angle range of vertical ± 5 °.

  Hereinafter, the arrangement of the LED array 62A will be described in more detail with reference to FIG. FIG. 4A is a side view showing a three-dimensional shape of light emitted from the light source lens 64. FIG. 4B is a graph showing the illuminance distribution of light incident on the incident surface 106 of the projection LCD 66 from one LED 62. FIG. 4C is a front view showing the LED array 62A partially enlarged. FIG. 4D is a graph showing a combined illuminance distribution of a plurality of lights incident on the incident surface 106 of the projection LCD 66 from the plurality of LEDs 62.

  As shown in FIG. 4A, the projection LCD 66 outputs the light emitted from the light source lens 64 as light having an illuminance distribution as shown in FIG. 4B in a state where the half-value spread half-angle θ is about 5 °. The light source lens 64 is designed to reach the light incident surface 106.

  Further, as shown in FIG. 4C, the plurality of LEDs 62 are arranged in a staggered pattern on the substrate 16. Specifically, a plurality of LED rows each having a plurality of LEDs 62 arranged in series in the horizontal direction at a pitch d are arranged in parallel in the vertical direction at a pitch equal to √3 / 2 times the pitch d. Furthermore, two LED rows adjacent to each other in the vertical direction are shifted from each other in the horizontal direction with a length equal to the pitch d.

  That is, in this embodiment, the arrangement of the LEDs 62 is a triangular lattice arrangement, and any LED 62 is separated from any other LED 62 adjacent thereto by a distance equal to the pitch d.

  In this embodiment, the length of the pitch d is previously set to be equal to or less than the full width at half maximum (FWHM (Full Width Half Maximun)) of the illuminance distribution given to the projection LCD 66 by the light emitted from one LED 62. Is set.

  Therefore, in this embodiment, the combined illuminance distribution of the light passing through the light source lens 64 and reaching the incident surface 106 of the projection LCD 66 is a substantially linear graph having small ripples as shown in FIG. As a result, the entire incident surface 106 of the projection LCD 66 is irradiated with light substantially uniformly. Therefore, according to the present embodiment, uneven illuminance in the projection LCD 66 is suppressed, and as a result, the pattern light is projected onto the subject S with high quality.

  FIG. 5 is a block diagram showing the electrical configuration of the three-dimensional input device 10. The processing unit 16 is configured mainly by a computer 110, and the computer 110 is configured to include a CPU 112, a ROM 114, and a RAM 116.

  The CPU 112 executes the program stored in the ROM 114 while using the RAM 116, thereby detecting the operation state of the release button 40, taking in the image data from the CCD 70, transferring and storing the taken image data, and mode switching. Various processes such as detection of the operation state of the switch 42 are performed.

  The ROM 114 includes a camera control program 114a, an imaging processing program 114b, a luminance image generation program 114c, a code image generation program 114d, a code boundary extraction program 114e, a lens aberration correction program 114f, and a triangulation calculation program 114g. Is stored.

  The camera control program 114a is executed to execute control of the entire three-dimensional input apparatus 10, and the control includes main processing conceptually shown in the flowchart of FIG.

  In order to detect the three-dimensional shape of the subject S, the imaging processing program 114b captures the subject S on which the pattern light is projected to obtain an image with pattern light, and further captures the subject S on which the pattern light is not projected. Then, it is executed to obtain a pattern light no image.

  The luminance image generation program 114c generates a plurality of luminance images respectively corresponding to the plurality of patterned light-present images based on the RGB values of the respective pixels acquired for the same subject S by the execution of the imaging processing program 114b.

  In the present embodiment, a plurality of types of pattern light are sequentially projected in time series on the same subject S, and the subject S is imaged each time each pattern light is projected. The RGB values of each pixel are acquired for each of the plurality of pattern light existence images thus captured, and as a result, the same number of luminance images as the types of pattern light are generated.

  The code image generation program 114d generates a code image in which a spatial code is assigned to each pixel from a binarized image generated by threshold processing for each of a plurality of luminance images generated by the execution of the luminance image generation program 114c. Executed to generate.

  Briefly, when the code image generation program 114d is executed, the interval between pattern lines in the luminance image of the subject S on which the narrowest interval between pattern lines is projected among the plurality of types of pattern light. Is acquired as a period, and the distribution of the entire luminance image of the period is acquired as a period distribution.

  When the code image generation program 114d is executed, a variable window whose size changes according to the acquired periodic distribution is set locally in the luminance image for each pattern light, thereby using the variable window. A threshold value is locally calculated and set for the entire luminance image by the filter processing. A binarized image is generated for each pattern light from the relationship between the threshold image representing the threshold distribution set in this way and the luminance image for each pattern light.

  A technique for locally calculating a threshold value for the entire luminance image by filtering using a variable window is disclosed in detail in Japanese Patent Application No. 2004-285736 of the present applicant. The contents of the specification are incorporated herein by reference.

  The code boundary extraction program 114e uses the code image generated by the execution of the code image generation program 114d and the luminance image generated by the execution of the luminance image generation program 114c, thereby converting the boundary coordinates of the code into subpixel accuracy. Run to ask for.

  The lens aberration correction program 114f is executed to correct the aberration of the imaging optical system 20 with respect to the code boundary coordinates obtained with subpixel accuracy by the execution of the code boundary extraction program 114e.

  The triangulation calculation program 114g is executed to calculate the three-dimensional coordinates of the real space related to the boundary coordinates from the boundary coordinates of the code subjected to the aberration correction by the execution of the lens aberration correction program 114f.

  As illustrated in FIG. 5, the RAM 116 includes a pattern light existence image storage unit 116 a, a pattern light no image storage unit 116 b, a luminance image storage unit 116 c, a code image storage unit 116 d, and a code boundary coordinate storage unit 116 e. , Aberration correction coordinate storage unit 116g, three-dimensional coordinate storage unit 116h, document orientation calculation result storage unit 116i, plane conversion result storage unit 116j, projection image storage unit 116k, period distribution storage unit 116p, threshold image The storage unit 116q, the binarized image storage unit 116r, and the working area 120 are allocated as storage areas.

  The pattern light present image storage unit 116a stores pattern light present image data representing the pattern light present image captured by the execution of the image capturing processing program 114b. The pattern light no-image storage unit 116b stores pattern light no-image data representing the pattern light no-image captured by the execution of the imaging processing program 114b.

  The luminance image storage unit 116c stores data representing a luminance image generated by executing the luminance image generation program 114c. The code image storage unit 116d stores data representing a code image generated by executing the code image generation program 114d. The code boundary coordinate storage unit 116e stores data representing the boundary coordinates of each code extracted with subpixel accuracy by the execution of the code boundary extraction program 114e.

  The aberration correction coordinate storage unit 116g stores data representing the boundary coordinates of the code that has been subjected to aberration correction by executing the lens aberration correction program 114f. The three-dimensional shape coordinate storage unit 116h stores data representing the three-dimensional coordinates of the real space calculated by executing the triangulation calculation program 114g.

  The projection image storage unit 116k stores a projection image that the projection unit 12 projects onto the subject S, that is, information regarding pattern light.

  The period distribution storage unit 116p, the threshold image storage unit 116q, and the binarized image storage unit 116r store data representing the period distribution, the threshold image, and the binarized image acquired by executing the code image generation program 114d, respectively.

  The working area 120 stores data temporarily used by the CPU 112 for its operation.

  Here, the camera control program 114a will be described with reference to FIG. When the camera control program 114a is executed by the computer 110, the main process described above is executed.

  In this main process, first, in step S101 (hereinafter simply referred to as “S101”; the same applies to other steps), the power source including the battery 74 is turned on. Next, in S102, the processing unit 16, peripheral interfaces, and the like are initialized.

  Subsequently, in S103, a key scan is performed to determine the operation state of the mode changeover switch 42. Thereafter, in S104, it is determined whether or not the SLOW mode is selected by the operation of the mode changeover switch 42. If it is assumed that the SLOW mode is selected this time, the determination is YES, and the above-described non-decimated image processing mode is set in S105. After execution of S105, S108, which will be described in detail later, is executed, and then the process returns to S103.

  On the other hand, this time, if it is assumed that the SLOW mode is not selected by the operation of the mode change switch 42, the determination in S104 is NO, and whether or not the FAST mode is selected by the operation of the mode change switch 42 in S106. Is determined. If it is assumed that the FAST mode is selected this time, the determination is YES, and the above-described thinned image processing mode is set in S107. After execution of S107, S108, which will be described in detail later, is executed, and then the process returns to S103.

  On the other hand, this time, if it is assumed that the FAST mode is not selected by the operation of the mode switch 42, the determination in S106 is NO, and whether or not the off mode is selected by the operation of the mode switch 42 in S112. Is determined. If it is assumed that the off mode is selected by operating the mode switch 42 this time, the determination is YES, and the main process of this time is immediately terminated. However, this time, the off mode is selected by operating the mode switch 42. If it is assumed that there is not, the determination is YES and the process returns to S103.

  FIG. 7 conceptually shows a flowchart of S108 in FIG. 6 as a stereoscopic image processing routine. By executing this stereoscopic image processing routine, stereoscopic image processing for detecting and displaying the three-dimensional shape of the subject S as a stereoscopic image is executed. In this stereoscopic image processing, the surface color of the same subject S is also detected. The three-dimensional color shape detection result is a combination of the detection result of the stereoscopic image and the detection result of the surface color in association with the position.

  In this stereoscopic image processing, first, in S1001, the finder image, that is, the same image as the image in the range visible through the imaging optical system 30 is displayed on the monitor LCD 44. Therefore, the user can confirm the captured image (imaging range) before actual imaging by viewing the image displayed on the monitor LCD 44.

  Next, in S1002, the operation state of the release button 40 is scanned, and then, in S1003, it is determined based on the scan result whether the release button 40 is in a half-pressed state. If it is in the half-pressed state, the determination is yes, and in S1004, the autofocus (AF) and automatic exposure (AE) functions are activated, thereby adjusting the focus, aperture, and shutter speed. If the release button 40 is not half-pressed, the determination in S1003 is NO, and the flow proceeds to S1010.

  After the execution of S1004, the operation state of the release button 40 is scanned again in S1005, and thereafter, in S1006, it is determined whether or not the release button 40 is fully pressed based on the scan result. If the release button 40 is not fully pressed, the determination in S1006 is NO and the process returns to S1002.

  If the release button 40 shifts from the half-pressed state to the fully-pressed state, the determination in S1006 becomes YES, and in S1007, a three-dimensional color shape detection process described later is executed, whereby the three-dimensional shape and surface color of the subject S Is detected.

  Briefly described, a three-dimensional color shape detection result for the subject S is generated by the three-dimensional color shape detection process. Here, the three-dimensional color shape detection result is a set of vertex coordinates obtained as a result of converting a plurality of spatial code boundary images detected in a spatial code image, which will be described later, into three-dimensional coordinates. Means that color shape information and polygon information are associated with each other. The color shape information is information representing a combination of real space coordinates and RGB values. The polygon information is information representing a combination of a plurality of vertices to be connected to each other in order to form a solid that three-dimensionally faces the subject S among the plurality of vertices.

  Thereafter, in S1008, the three-dimensional color shape detection result is stored in the external memory 78. Subsequently, in S1009, the three-dimensional color shape detection result is displayed on the monitor LCD 44 as a three-dimensional computer graphic image.

  Thereafter, in S1010, key scanning is performed in the same manner as in S103 in FIG. Subsequently, in S1011, it is determined whether or not the operation state of the mode switch 42 has changed. If there is no change, the determination in S1011 is YES and the process returns to S1001, but if there is a change, the determination in S1011 is NO and the current stereoscopic image processing ends.

  In the three-dimensional color shape detection process executed in S1007 in FIG. 7, the three-dimensional shape of the subject S is detected using a spatial coding method. Hereinafter, the spatial coding method will be described with reference to FIG. FIG. 8A shows a real space in which the three-dimensional coordinate system XYZ is set as viewed in the Y coordinate axis direction, a view as viewed in the X coordinate axis direction, and three types of masks A and B using pure binary codes. And C pattern. On the other hand, FIG. 8B shows three types of masks A, B, and C patterns using a gray code and a plurality of spatial codes.

  As shown in FIG. 8A, the spatial encoding method is an observation image that is an image of a subject S that is an observation target, and a projection light source (for example, a projector) that projects light (diffuse light) onto the subject S. This is a kind of technique for detecting the three-dimensional shape of the subject S by applying the principle of triangulation to an observer (for example, a camera) that observes the subject S. In this spatial coding method, as shown in FIG. 8A, the projection light source L (PROJECTOR) and the observation device O (CAMERA) are set apart by a distance d. Therefore, an arbitrary point P in the observation space can be specified if the direction ψ of the projection light and the direction θ seen from the observer O can be measured. In this spatial coding method, in order to specify an arbitrary position on the surface of the subject S, the observation space is coded by being divided into a plurality of elongated fan-shaped regions.

  In order to obtain a code at an arbitrary position on the surface of the subject S from the observation image, a plurality of types of stripe pattern light are projected onto the subject S in time series. Pattern light can be switched by preparing the same number of masks as the type of pattern light and mechanically exchanging the masks, or by forming a striped optical shutter array using a material having an electro-optic effect. It is possible to implement as an electronic system that electronically controls the light transmittance of each stripe in the optical shutter row. However, in the present embodiment, the latter electronic method is employed. Specifically, a plurality of types of mask patterns are reproduced or displayed in time series by the projection LCD 66.

  In the example shown in FIG. 8A, a mask is installed in a replaceable manner between the projection light source L and the subject S (square pole and cylinder). In this example, three types of masks A, B, and C having different patterns are prepared, and therefore, three types of pattern light are projected onto the subject S in time series.

  When the pattern light generated by the masks A, B, and C is projected onto the subject S, each of the eight fan-shaped areas is encoded into either the bright area “1” or the dark area “0”. When the light that has passed through the three masks A, B, and C is projected onto the subject S in that order, a code consisting of 3 bits is assigned to each fan-shaped area. These 3 bits are arranged in order from the most significant bit MSB corresponding to the first mask A to the least significant bit LSM corresponding to the last mask C. For example, in the example shown in FIG. 8A, the fan-shaped region to which the point P belongs is blocked by the masks A and B, whereas the light passes only by the mask C and becomes a bright region. 001 (A = 0, B = 0, C = 1) ”.

  Thus, a code corresponding to the direction ψ from the projection light source L is assigned to each fan-shaped region. On the other hand, if each bit plane of the memory is constructed by binarizing the light / dark pattern of the subject S on which each pattern light is projected for each mask, the horizontal position (address) of each bit plane image is determined by the observer O. This corresponds to the direction θ from If attention is paid to the memory contents of the three bit planes corresponding to the three masks for each bit (each pixel), a 3-bit code is acquired for each pixel. From this code, the direction ψ from the projection light source L of each fan-shaped region is specified. In a situation where the distance d is known, if the directions ψ and θ are specified, the three-dimensional coordinates of the point of interest on the surface of the subject S are specified by the principle of triangulation.

  FIG. 8A shows an example in which a space is coded using a pure binary code by using a plurality of masks such as masks A, B and C. FIG. 8B shows an example. Shows an example in which a space is coded by using a gray code whose Hamming distance between adjacent codes is always 1 as a space code by using a plurality of masks such as masks A, B and C.

  In the present embodiment, in the above-described three-dimensional color shape detection process, a spatial encoding method using a pure binary code or a spatial encoding method using a Gray code may be used.

  Details of this spatial coding method are described in, for example, Kosuke Sato and one other, “Distance Image Input by Spatial Coding”, IEICE Transactions, 85/3 Vol. J 68-D No3 p369-375.

  FIG. 9 conceptually shows a flowchart of S1007 in FIG. 7 as a three-dimensional color shape detection processing routine.

  In this three-dimensional color shape detection processing routine, first, in S1210, an imaging process is executed. If this imaging process is executed, for example, the striped pattern light is projected onto the subject S in time series from the projection unit 12 using a plurality of gray code mask patterns shown in FIG. . Further, a plurality of pattern light existence images obtained by imaging the subject S on which a plurality of types of pattern light are projected and a single pattern light no image obtained by imaging the same subject S on which the pattern light is not projected are acquired. Is done.

  When the imaging process ends, a three-dimensional measurement process is executed in S1220. When this three-dimensional measurement process is executed, the three-dimensional shape of the subject S is actually measured using a plurality of pattern-lighted images and one pattern-lightless image acquired by the above-described imaging process. Is done.

  When this three-dimensional measurement process is completed, a three-dimensional color shape detection result is generated by combining the three-dimensional shape measured for the subject S and the surface color in S1230. When this three-dimensional color shape detection result is generated, the current three-dimensional color shape detection process ends.

  In S1210 in FIG. 9, the imaging processing program 114b is executed for the above-described imaging processing. FIG. 10 conceptually shows the imaging processing program 114b in a flowchart.

  In this imaging processing program 114b, first, in S2001, a pattern number PN representing a mask pattern number used for forming pattern light is initialized to zero. Subsequently, in S2002, it is determined whether or not the current value of the pattern number PN is smaller than the maximum value PNmax. The maximum value PNmax is determined according to the total number of mask patterns used. For example, when 8 types of mask patterns are used, the maximum value PNmax is set to 8.

  This time, if it is assumed that the current value of the pattern number PN is smaller than the maximum value PNmax, the determination in S2002 is YES, and then in S2003, the current value of the pattern number PN among the plurality of types of mask patterns used. Projection of the PN-th mask pattern with the same number is started.

  Subsequently, in S2004, a projection process for projecting the PN-th mask pattern onto the subject S is performed. FIG. 11 conceptually shows the details of S2004 as a projection processing subroutine in a flowchart. By executing this projection processing subroutine, the projection processing for projecting the image stored in the projection image storage unit 37k from the projection unit 12 onto the subject S is executed.

  In this projection processing, first, in S3001, it is determined whether or not an image is stored in the projection image storage unit 116k. If it is not stored, the determination is no and the current projection process is immediately terminated. On the other hand, if it is stored, the determination is YES, and the image stored in the projection image storage unit 116k is transferred to the projection LCD driver 86 in S3002. Subsequently, in S3003, an image signal corresponding to the stored image is sent from the projection LCD driver 86 to the projection LCD 66, whereby an image is displayed on the projection LCD 66.

  Thereafter, in S3004, the light source driver 84 is driven. Subsequently, in S3005, the LED array 62A emits light by an electrical signal from the light source driver 84. This is the end of the current projection process.

  The light emitted from the LED array 62A reaches the projection LCD 66 via the light source lens 64. In the projection LCD 66, spatial modulation is performed in accordance with the image signal transmitted from the projection LCD driver 86. As a result, incident light on the projection LCD 66 is converted into image signal light and output. The image signal light output from the projection LCD 66 is projected as a projected image onto the subject S via the projection optical system 32.

  As described above, when the PN-th pattern light formed by the PN-th mask pattern is projected onto the subject S, subsequently, the subject onto which the PN-th pattern light is projected in S2005 in FIG. S is imaged by the imaging unit 14.

  By the imaging, a PN-th pattern light existence image obtained by imaging the subject S on which the PN-th pattern light is projected is acquired. The acquired pattern light present image is stored in the pattern light present image storage unit 116a in association with the corresponding pattern number PN.

  When the imaging is finished, the projection of the PN-th pattern light is finished in S2006. Subsequently, in S2007, the pattern number PN is incremented by 1 to project the next pattern light, and then the process returns to S2002.

  As a result of the execution of S2002 to S2007 being repeated the same number of times as the number of types of pattern light, if the current value of the pattern number PN becomes a value not smaller than the maximum value PNmax, the determination in S2002 becomes NO, and the current imaging process is performed. finish. Therefore, the same number of images with pattern light as the maximum value PNmax are acquired by one imaging process.

  Subsequently, in S2008, it is determined whether or not the flash mode is selected. If the flash mode is selected, the determination is YES, and the flash 26 is caused to emit light in S2009. If the flash mode is not selected, the determination in S2008 is NO and S2009 is skipped. In any case, the subject S is then imaged in S2010.

  This imaging is performed without projecting pattern light from the projection unit 12 onto the subject S for the purpose of measuring the surface color of the subject S. As a result, one pattern light no image is acquired for the subject 12. The acquired pattern light no image is stored in the pattern light no image storage unit 116b.

  This completes one execution of the imaging processing program 114b.

  FIG. 12 is a timing chart showing an example of the operation of the three-dimensional input apparatus 10 accompanying one execution of the imaging processing program 114b. This operation example is executed by the three-dimensional input device 10 when the user operates the release button 40 in a fully-pressed state while the FAST mode is selected by the user.

  FIG. 12A shows a state in which the CCD 70 is continuously exposed a plurality of times by the incident light from the subject S. In FIG. 12B, for each of these multiple exposures, light is converted into an electrical signal by the CCD 70 for each pixel in the entire image represented by the incident light from the subject S and output from the CCD 70. The signal output timing is represented by a timing chart. In FIG. 12C, timing when the image processing mode of the imaging unit 14 is temporally switched between the thinned-out image processing mode and the non-thinned-out image processing mode is represented by a timing chart.

  Further, in FIG. 12D, timings at which the state of the imaging unit 14 is temporally switched between a standby state and an operation state for imaging and signal extraction are shown in a timing chart. FIG. 12E shows the timing at which the projection image data representing each pattern light is transferred to the projection LCD 66 in a timing chart. FIG. 12F shows the timing at which the release button 40 is temporally switched between the non-operating state and the operating state in a timing chart.

  In the present embodiment, after the CCD 70 is exposed by incident light from the subject S, a signal reflecting the exposure is taken out from the CCD 70. One signal extraction corresponds to one exposure, and these exposure and signal extraction together constitute one individual imaging process.

  In the present embodiment, the acquisition of the three-dimensional shape information and the acquisition of the surface color information for the same subject S are continuously performed in that order.

  As described above, in order to acquire the three-dimensional shape information of the subject S, eight types of pattern light (pattern numbers PN = 0 to 7) are sequentially projected on the subject S, and the CCD 70 is projected for each projection of the pattern light. Exposure and signal extraction from the CCD 70 are performed. That is, in order to acquire the three-dimensional shape information of the subject S, the individual imaging processing for the subject S is sequentially performed a total of eight times. In FIG. 12, the number PN of the pattern light corresponding to each individual imaging process for acquiring the three-dimensional shape information is indicated by numbers “0” to “7”.

  In order to acquire the surface color information of the subject S, the CCD 70 is exposed once by the incident light from the subject S, and then the signal is extracted. That is, in order to acquire the surface color information of the subject S, the individual imaging process for the subject S is performed once. In FIG. 12, a single individual imaging process for acquiring surface color information is indicated by a symbol “c”.

  In imaging for obtaining three-dimensional shape information, it is indispensable to project pattern light as illumination light on the subject S, whereas in imaging for obtaining surface color information, illumination light is projected onto the subject S. It is selective to do. Specifically, in imaging for acquiring surface color information, when the amount of light received from the subject S is insufficient, the flash 26 is automatically caused to emit light, thereby projecting illumination light onto the subject S. Is done.

  Therefore, in the present embodiment, eight individual imaging processes for acquiring three-dimensional shape information and one individual imaging process for acquiring surface color information are continuously performed, for a total of nine individual imaging processes. Imaging processing is continuously performed. In the present embodiment, these nine individual imaging processes cooperate with each other to constitute one whole imaging process.

  In these nine individual imaging processes, nine exposures are sequentially performed on the same subject S, and these nine exposures are performed at the same cycle as the video rate, for example. The period in which these nine exposures are continuously performed is a period in which the influence appears in the imaging result of the CCD 70 when the relative position between the subject S and the three-dimensional input device 10 changes. This period is the imaging time of the three-dimensional input device 10. It means that the moving image imaging capability of the three-dimensional input device 10 is higher as the imaging time is shorter.

  In one operation example shown in FIG. 12, signal extraction in eight individual imaging processes for acquiring three-dimensional shape information is performed as a thinned image process. Accordingly, in each individual imaging process for obtaining three-dimensional shape information, an electrical signal is output from the CCD 70 after the CCD 70 has been exposed and after a necessary signal extraction time t1 has elapsed. The signal extraction time t1 is also referred to as one frame extraction time. For each pattern light projection, the three-dimensional necessary for the three-dimensional shape information to be output from the CCD 70 after the exposure of the CCD 70 is completed. It means shape information output time.

  On the other hand, in the example shown in FIG. 12, signal extraction in one individual imaging process for acquiring surface color information is executed as non-decimated image processing. Therefore, in one individual imaging process for acquiring surface color information, an electrical signal is output from the CCD 70 after the exposure of the CCD 70 and after a necessary signal extraction time t2. The signal extraction time t2 is also called one frame extraction time, and means the surface color information output time required until the surface color information is output from the CCD 70 for one frame after the exposure of the CCD 70 is completed.

  The signal extraction time t1 required for the thinned-out image processing is shorter than the signal extraction time t2 required for the non-thinned-out image processing. For example, the signal extraction time t1 is about 33 ms, while the signal extraction time t2 is about 0.5 s.

  As shown in FIGS. 12A and 12B, in one individual imaging process, signal extraction is started after the exposure is completed, but the exposure in the next individual imaging process is performed in the previous individual imaging process. This is started before the signal extraction at is finished. That is, the signal extraction in one individual imaging process and the exposure in the next individual imaging process are performed so as to partially overlap in time. However, the signal extraction in a certain individual imaging process ends before the exposure in the next individual imaging process ends.

  Accordingly, in the present embodiment, as shown in FIG. 12B, eight signal extractions for acquiring three-dimensional shape information are continuously performed with no time gap.

  Each signal extraction is completed in about 33 ms if it is executed as a thinned image process. Therefore, eight signal extractions are completed in about 0.26 s. Therefore, the length of the part (hereinafter referred to as “partial imaging time”) necessary for imaging for obtaining the three-dimensional shape information in the imaging time (total imaging time) shown in FIG. 12 is the length of the signal extraction time. Therefore, a length of about 0.26 s is required.

  On the other hand, if each signal is extracted as non-decimated image processing, about 0.5 s is required. Therefore, about 5 s is required for eight signal extractions, and accordingly, the corresponding partial imaging time is also required to be as long as that.

  As described above, when the signal extraction from the CCD 70 is executed as the thinned image processing, the imaging time is shortened. As a result, the 3 of the subject S can be obtained regardless of the movement of the subject S and the camera shake of the three-dimensional input device 10. Dimensional shape can be measured with high accuracy.

  Further, as shown in FIG. 12, in the present embodiment, each of the second to eighth exposures for acquiring the three-dimensional shape information and the exposure for acquiring the surface color information are immediately preceding each other. The process is started without waiting for the end of signal extraction corresponding to the exposure. The signal extraction corresponding to the preceding exposure and the subsequent exposure are performed in parallel with each other, whereby nine signal extractions are continuously performed with no time gap. Therefore, the imaging time shown in FIG. 12, that is, the time required to continuously perform both the imaging for acquiring the three-dimensional shape information and the imaging for acquiring the surface color information is shortened.

  Specifically, since each signal extraction is completed in about 33 ms when it is executed as a thinned-out image process, nine signal extractions are completed in about 0.3 s, and thus correspond to it. The entire imaging time can be as long as that.

  Temporarily, one individual imaging process for surface color information acquisition, that is, surface color measurement imaging process (signal extraction is executed as non-decimated image processing) is performed eight times for acquiring three-dimensional shape information. When individual imaging processing, that is, three-dimensional measurement imaging processing (signal extraction is executed as a thinned-out image processing) is performed later, three-dimensional shape information is obtained until signal extraction in the preceding surface color measurement imaging processing is almost completed. You have to wait for the first exposure for acquisition. The waiting time is approximately equal to the length of the signal extraction time t2 and is about 0.5 s.

  In this case, there is a slightly longer time interval between the exposure for acquiring the surface color information and the first exposure for acquiring the three-dimensional shape information, and the total imaging time shown in FIG. 12 becomes longer. On the other hand, when the subject S is a stationary object and the degree of camera shake of the three-dimensional input device 10 is not large, it is not a problem that the entire imaging time is slightly long. On the other hand, as shown in FIG. 1, when the subject S is a moving object or when the degree of camera shake of the three-dimensional input device 10 is large, if the entire imaging time is long, the surface color information and the three-dimensional shape Information does not match each other sufficiently accurately with respect to the pixel position. That is, the texture mapping accuracy is lowered.

  In contrast, in the present embodiment, as shown in FIG. 12, eight individual imaging processes for acquiring three-dimensional shape information are performed first, and one individual imaging process for acquiring surface color information is performed later. Done. As a result, the eighth exposure for acquiring the preceding three-dimensional shape information and the subsequent exposure for acquiring the surface color information are performed eight times for acquiring the preceding three-dimensional shape information. It becomes possible to carry out continuously with the same period. Therefore, according to the present embodiment, it is possible to reduce the total imaging time to about 0.3 s.

  Therefore, according to the present embodiment, it is possible to continuously perform exposure for acquiring three-dimensional shape information and exposure for acquiring surface color information at sufficiently short time intervals. Regardless of the degree of movement or the degree of camera shake of the three-dimensional input device 10, high texture mapping accuracy is realized.

  Therefore, according to the present embodiment, when the signal extraction from the CCD 70 for obtaining the three-dimensional shape information is executed in the thinned image processing mode (when the user selects the FAST mode), the high texture resolution, that is, the surface color A three-dimensional input device 10 suitable for capturing a moving image with high texture mapping accuracy while ensuring measurement accuracy is provided.

  Furthermore, in this embodiment, the user can appropriately change the image processing mode for acquiring the three-dimensional shape information into a thinned image processing mode, that is, a FAST mode, and a non-thinned image processing mode, that is, a SLOW mode. It is. If the user selects the FAST mode in an environment where a decrease in texture mapping accuracy is a concern, the texture mapping accuracy does not decrease in spite of such an environment. On the other hand, if the user selects the SLOW mode in an environment where there is no concern about a decrease in texture mapping accuracy, not only high texture mapping accuracy but also high three-dimensional shape measurement accuracy is realized.

  As described above, according to the present embodiment, the user sets the three-dimensional input device 10 according to the use environment of the three-dimensional input device 10 and the user's request for each of the three-dimensional shape measurement accuracy and the texture mapping accuracy. It can be arbitrarily changed, so that the usability of the three-dimensional input device 10 is improved.

  FIG. 13 conceptually shows a flowchart of S1220 in FIG. 9 as a three-dimensional measurement processing subroutine.

  In this three-dimensional measurement processing subroutine, first, in S4001, a luminance image is generated by executing the luminance image generation program 114c.

  In S4001, the luminance value is defined as the Y value in the YCbCr space, and from the RGB value of each pixel,

  Y = 0.2989 · R + 0.5866 · G + 0.1145 · B

It is calculated using the following formula. By obtaining the Y value for each pixel, a plurality of luminance images respectively corresponding to the plurality of pattern light existence images are generated. The generated luminance images are stored in the luminance image storage unit 116c in association with the pattern number PN. However, the formula used for calculating the luminance value is not limited to the above formula, and can be appropriately changed to another formula.

  Next, in S4002, the code image generation program 114d is executed. When the code image generation program 114d is executed, a plurality of generated luminance images are combined using the above-described spatial encoding method, thereby generating a code image in which a spatial code is assigned to each pixel. Is done. The code image is generated by binarization processing by comparing a luminance image related to a plurality of types of pattern light existence images stored in the luminance image storage unit 116c with a threshold image to which a luminance threshold is assigned for each pixel. . The generated code image is stored in the code image storage unit 116d.

  FIG. 14 conceptually shows the details of the code image generation program 114d in a flowchart. Since the technology employed in the code image generation program 114d is described in detail in Japanese Patent Application No. 2004-285736 of the present applicant, the contents of the patent application are described in this specification by referring to the patent application. Quote in the book.

  Hereinafter, the code image generation program 114d will be described in time series, but prior to that, it will be described in principle.

  In the present embodiment, a plurality of luminance images are generated for the same subject S (three-dimensional object) under a plurality of types of pattern light. Each of the pattern lights is formed such that a bright part, that is, a bright pattern line having a width, and a dark part, that is, a dark pattern line having a width, are alternately repeated at a constant period. The pattern lights are different from each other with respect to the period, and are each referred to as a pattern light having a pattern number PN. Among the pattern lights, the pattern light having the shortest cycle is the pattern light having the pattern number PN of 0, and the pattern light having the longest cycle is the pattern light having the pattern number PN of (PNmax-1). .

  Since any luminance image is acquired under the corresponding pattern light, it is formed as a pattern image in which bright pattern lines as bright portions and dark pattern lines as dark portions are alternately arranged. Since the interval, that is, the period between the pattern lines depends on the relative geometric relationship (relationship between position and orientation) between the three-dimensional input device 10 and the subject S, it is constant at all positions in each luminance image. Not necessarily. A plurality of luminance images respectively acquired under a plurality of types of pattern light are specified using the pattern number PN of the corresponding pattern light.

  In the present embodiment, any one of the plurality of luminance images is selected as the representative pattern image. A typical example of the representative pattern image is a luminance image corresponding to one of a plurality of types of pattern light having a minimum pattern line cycle. This is a luminance image having a pattern number PN of 0.

  In the luminance image obtained by imaging the subject S on which the pattern light is projected, the luminance value changes spatially and periodically in the direction of the pixel column. In the graph representing the periodic change, there is an envelope that touches at a plurality of lower peak points (lowest luminance points). This envelope represents a spatial change in the luminance value in the luminance image obtained by imaging the same subject S in the non-irradiated state, that is, the luminance value of the background light of the subject S. For a luminance image in which such an envelope exists, in order to binarize the luminance value of each pixel accurately by threshold processing, it is desirable to change the threshold according to the pixel position. That is, it is desirable to adaptively change the threshold value by tracking the actual luminance value change of the luminance image.

  Based on such knowledge, in this embodiment, a filter window for calculating a threshold value by performing filter processing on the luminance image is set locally, and the threshold value suitable for the position by the filter processing is Set locally for the luminance image. If a window is set at a certain local position in the luminance image, the luminance value of the pixel existing in the window is extracted and referred to from among the plurality of pattern lines constituting the luminance image, and the window is set. A threshold value corresponding to the local position is set.

  The window used in this embodiment is a rectangular window. When this rectangular window is adopted, the luminance values of the pixels constituting the plurality of pattern lines existing in the rectangular window are extracted, and the threshold value is calculated by using the same weighting coefficient for the luminance values. Is done. The window function of the rectangular window is defined by the weight coefficient.

  Further, when a rectangular window is adopted, the number of pixels existing in the line direction in the rectangular window can be made variable according to the line direction size in the line direction in which the pattern line extends. . On the other hand, the number of pattern lines and the number of pixels existing in the column direction in the rectangular window are variable according to the column direction size in the column direction in which a plurality of pattern lines are arranged in a row. can do.

  Therefore, when a rectangular window is adopted, the threshold value calculated from the luminance image changes by setting the window in the luminance image depending on the column direction size of the rectangular window. Therefore, when it is necessary to adaptively change the threshold value, the column direction size of the rectangular window may be adaptively changed.

  In the present embodiment, the size of the window configured as a rectangular window is such that the number of pattern lines existing in the window is an integral multiple of the interval between the pattern lines, that is, the period (for example, the period in which bright pattern lines are repeated) It is desirable to set so that there is. That is, it is desirable to set the size of the window so that the same number of bright pattern lines and dark pattern lines exist in the window. With this setting, it is possible to obtain a desired threshold with high accuracy by calculating the average value of the luminance values of a plurality of pattern lines existing in the window.

  However, even on the same luminance image, the pattern line period may vary depending on the location. Therefore, when the size of the window is fixed, the number of pattern lines existing in the window varies depending on the location, and the threshold setting accuracy is lowered.

  In the present embodiment, among a plurality of luminance images, an image captured under the pattern light having the minimum pattern line period, that is, a luminance image having a pattern number PN of 0 is selected as the representative pattern image. Is done. Furthermore, in this embodiment, the window VW set locally with respect to the representative pattern image is configured as a variable window whose size is variable. Thereby, the size of the variable window VW is changed in conformity with the actual pattern line period of the representative pattern image.

  Therefore, according to the present embodiment, even if the pattern line period in the representative pattern image varies according to the position in the column direction, the size of the variable window VW is changed so as to follow it, and as a result, the pattern line period varies. Nevertheless, the number of bright and dark pattern lines existing in the variable window VW is kept constant. In the present embodiment, the threshold value TH is acquired for each local position where the variable window VW is set for the representative pattern image. The threshold value TH for each local position is accurately obtained based on the variable window VW having the optimum size for each local position.

  In addition, the size of the variable window VW that keeps the number of bright and dark pattern lines constant is minimum in a luminance image having a pattern number PN of zero. Therefore, by selecting a luminance image whose pattern number PN is 0 as a representative pattern image, the size of the minimum variable window VW can be made, and the calculation load of filter processing after using the variable window VW can be suppressed. It becomes possible.

  In the present embodiment, the variable window VW is configured as a rectangular window having a variable size. The size of the variable window VW is set so as to be variable in the column direction of the representative pattern image but fixed in the line direction.

  In the present embodiment, the size of the variable window VW, that is, the size in the column direction of the representative pattern image is set so as to adaptively reflect the actual pattern line period of the representative pattern image. Therefore, in order to set the size of the variable window VW, it is necessary that the actual pattern line period distribution of the representative pattern image is known in advance.

  Therefore, in this embodiment, prior to setting the size of the variable window VW, a fixed window with a fixed size is set for the representative pattern image. A plurality of continuous pixels captured by the set fixed window are selected as a plurality of target pixels, and an actual pattern line period distribution of the representative pattern image is acquired based on the luminance values of the selected target pixels. The

  In the present embodiment, further, FFT (Fast Fourier Transform) processing is performed on the luminance values of a plurality of pixels of interest in the representative pattern image, whereby the luminance value in the column direction of the representative pattern image. Intensity (eg, power spectrum) is acquired for each frequency component of the change. Here, the “frequency component” means the number of repetitions in which the change in luminance value is repeated when a plurality of target pixels captured by one fixed window are traced in the column direction.

  In the present embodiment, each of a plurality of continuous pixels continuously arranged in the column direction in the representative pattern image is sequentially selected as a target pixel, and the pattern line period for each of the selected target pixels is the representative pattern image. Obtained based on the luminance value distribution.

  The code image generation program 114d has been described in principle above. Hereinafter, the code image generation program 114d will be described in time series with reference to FIG.

  In the code image generation program 114d, first, in S5001, a luminance image obtained by imaging the subject S on which pattern light having a pattern number PN of 0 is captured is read as a representative pattern image from the luminance image storage unit 116c.

  Next, in S5002, the pattern line cycle is calculated for each pixel that is continuously arranged in the column direction in the representative pattern image based on the read luminance image for the representative pattern image by the above-described approach by FFT conversion. Is done. The calculated plurality of pattern line periods are stored in the period distribution storage unit 116p in association with each pixel (each column direction pixel position).

  Subsequently, in S5003, the characteristics of the variable window VW are set locally based on the plurality of calculated pattern line periods. In the present embodiment, the line direction size of the variable window VW is set so as not to change regardless of the position on the representative pattern image where the variable window VW is set, whereas the column direction of the variable window VW The size is set so as to correspond to an integral multiple of the pattern line period calculated in association with each column direction pixel position.

  After that, in S5004, a variable window VW is set for the representative pattern image in a plane along the line direction and the column direction and in association with each pixel. Thereby, for each pixel, an average value of luminance values of a plurality of pixels existing in the variable window VW is calculated as a local threshold value. In S5004, a threshold image in which the calculated threshold value is assigned to each pixel is further generated. The generated threshold image is stored in the threshold image storage unit 116q.

  Subsequently, in S5005, the pattern number PN is initialized to 0. Thereafter, in S5006, it is determined whether or not the current value of the pattern number PN is smaller than the maximum value PNmax. Since the current value of the pattern number PN is 0 this time, the determination is no and the process moves to S5007.

  In S5007, the luminance value of the luminance image assigned the pattern number PN equal to the current value of the pattern number PN and the threshold value of the generated threshold image are compared with each other for each pixel. The comparison result is reflected in the binarized image for each pixel. Specifically, when the luminance value of the luminance image is larger than the threshold value, data representing “1” is stored in the binarized image storage unit 116r in association with the corresponding pixel position in the binarized image, On the other hand, when the luminance value of the luminance image is not greater than the threshold value, data representing “0” is stored in the binarized image storage unit 116r in association with the corresponding pixel position in the binarized image.

  Thereafter, in S5008, the pattern number PN is incremented by one. Subsequently, returning to S5006, it is determined whether or not the current value of the pattern number PN is smaller than the maximum value PNmax. Again, if it is smaller than the maximum value PNmax, the determination is no and the process moves to S107.

  As a result of the execution of S5006 to S5008 being repeated the same number of times as the number of types of pattern light, if the current value of the pattern number PN becomes a value not smaller than the maximum value PNmax, the determination in S5006 becomes YES and the process proceeds to S5009.

  In S5009, for each pixel, the binarization corresponding to the luminance image in which the pixel value (“1” or “0”) from the same number of binarized images as the maximum value PNmax is the pattern number PN is 0. The spatial code is extracted from the image in accordance with the order from the binary image corresponding to the luminance image having the pattern number PN of (PNmax-1), and the spatial code arranged in the order from the least significant bit LSM to the most significant bit MSB is generated. The The number of bits of the spatial code for each pixel is the same as the maximum value PNmax. By generating a spatial code for each pixel, a spatial code image corresponding to the current subject S is generated. The generated spatial code is stored in the spatial code storage unit 116d in association with each pixel position. For example, when the maximum value PNmax is 8, the generated spatial code has a value in the range of 0 to 255.

  Thus, one execution of the code image generation program 114d is completed.

  Thereafter, in S4003 in FIG. 13, a code boundary coordinate detection process is performed by executing the code boundary extraction program 114e. Since the coding by the above-described spatial coding method is performed in units of pixels, the bright / dark boundary line in the actual pattern light and the spatial code boundary line in the generated code image (a certain spatial code is assigned) An error in sub-pixel accuracy occurs between the region and the boundary between the region assigned with another spatial code. Therefore, this code boundary coordinate detection process is performed for the purpose of detecting the boundary coordinate value of the spatial code with subpixel accuracy.

  For example, when 255 discrete reference line positions intersecting the line direction of each pattern light are set in the CCD coordinate system, the maximum value PNmax is 8 (the boundary is 255 because it has 256 spatial codes). By S4003 (execution of the code boundary extraction program 114e) in FIG. 13, boundary coordinate values of a maximum of about 65,000 spatial codes are detected.

  The detected code boundary coordinate value is stored in the code boundary coordinate storage unit 116e. The code boundary coordinate values are defined in a CCD coordinate system ccdx-ccdy, which is a two-dimensional coordinate system set on the imaging plane of the CCD 70.

  Subsequently, in S4004, a lens aberration correction process is performed by executing the lens aberration correction program 114f. In this lens aberration correction processing, the actual imaging position of the light beam incident on the imaging optical system 30 that is affected by the aberration of the imaging optical system 30 is the ideal lens. If so, it is a process of correcting the image so as to approach the ideal image formation position where the image should be formed.

  By this lens aberration correction processing, the code boundary coordinate value detected in S4003 is corrected so that an error caused by distortion of the imaging optical system 30 or the like is removed. The code boundary coordinates corrected in this way are stored in the aberration correction coordinate storage unit 116g.

  Neither the code boundary coordinate detection process nor the lens aberration correction process is indispensable for understanding the present invention, and is disclosed in detail in the specification of the applicant's Japanese Patent Application No. 2004-105426. The detailed description is omitted in this specification by citing it by referring to it.

  Thereafter, in S4005, real space conversion processing based on the principle of triangulation is performed by executing the triangulation calculation program 114g. When this real space conversion processing is performed, the above-described code boundary coordinate values on the CCD coordinate system ccdx-ccdy and subjected to aberration correction are set in the real space according to the principle of triangulation. It is converted into a three-dimensional coordinate value on the real space coordinate system XYZ which is a three-dimensional coordinate system, and as a result, a three-dimensional coordinate value as a three-dimensional color shape detection result is acquired. The acquired three-dimensional coordinate value is stored in the three-dimensional coordinate storage unit 116h.

  Here, referring to FIG. 15, a method of converting a two-dimensional coordinate value on the CCD coordinate system ccdx-ccdy into a three-dimensional coordinate value on the real space coordinate system XYZ according to the principle of triangulation. This will be described in detail.

  In the present embodiment, the real space coordinate system XYZ is fixed to the three-dimensional input device 10, and the subject S that is the imaging target is imaged on the real space coordinate system XYZ. As shown in FIG. 15, the real space coordinate system XYZ extends in the horizontal direction, the Y axis extends in the vertical direction, and the Z axis corresponds to the imaging optical system 30 with respect to the three-dimensional input device 10. It is positioned so as to extend in the optical axis direction. FIG. 15A shows the real space coordinate system XYZ observed in the X-axis direction, and FIG. 15B shows the real space coordinate system XYZ Y-axis. It is shown as observed in the direction. The real space coordinate system XYZ is set with respect to the three-dimensional input device 10 so that the origin is located at a position VPZ away from the input pupil position of the imaging optical system 30 along the Z axis. .

  In the real space coordinate system XYZ, the projection angle from the projection unit 12 to the subject S is expressed as “θp”, and the distance between the optical axis of the imaging optical system 30 and the optical axis of the projection unit 12 is “ D ”. The projection angle θp is uniquely specified by a spatial code assigned to each pixel.

  In the real space coordinate system XYZ, further, the intersection Y between the straight line in which the optical path where the reflected light from the target point on the subject S enters the CCD 70 is extended in the opposite direction and the XY plane is used. The coordinate value is expressed as “Ytarget”, and the X coordinate value is expressed as “Xtarget”. In the real space coordinate system XYZ, the field of view of the imaging optical system 30 in the Y direction is further defined as a region from a point represented by “Yftop” to a point represented by “Yfbottom”. , The field of view in the X direction is defined as a region from a point represented by “Xfstart” to a point represented by “Xfend”. Further, the length (height) of the CCD 70 in the Y-axis direction is represented by “Hc”, and the length (width) in the X-axis direction is represented by “Wc”.

  On the real space coordinate system XYZ defined in this way, the three-dimensional coordinate values (X, X, Y) on the real space coordinate system corresponding to the arbitrary coordinate values (ccdx, ccdy) on the CCD coordinate system of the CCD 70. Y, Z) is

  (A) a target point target (X, Y, Z) on the subject S (indicated by “(a)” with a leader line in FIG. 15);

  (B) The input pupil position of the imaging optical system 30 (shown as “(b)” with a leader line in FIG. 15);

  (C) The output pupil position of the projection optical system 32 (shown as “(c)” with a leader line in FIG. 15);

  (D) Intersection (Xtarget, Ytarget) between the input pupil position of the imaging optical system 30 and a straight line passing through the target point on the subject S and the XY plane (shown as “(d)” with a leader line in FIG. 15) .)When,

  (E) The intersection of the output pupil position of the projection optical system 32 and the straight line passing through the target point on the subject S and the XY plane (shown as “(e)” with a leader line in FIG. 15).

It is obtained by solving the following five expressions showing the relationship regarding

(1) Y = (PPZ−Z) tan θp−D + cmp (Xtarget)

(2) Y = − (Ytarget / VPZ) Z + Ytarget

(3) X = − (Xtarget / VPZ) Z + Xtarget

(4) Ytarget = Yftop− (ccdcy / Hc) × (Yftop−Yfbottom)

(5) Xtarget = Xfstart + (ccdcx / Wc) × (Xfend−Xfstart)

  However, “cmp (Xtarget)” in the equation (1) is a function for correcting the deviation between the imaging optical system 20 and the projection unit 12, and can be regarded as 0 in an ideal case where there is no deviation.

  Further, in this real space conversion process, the coordinate value (ccdx, ccdy) of an arbitrary point in the real image is converted into the coordinate value (ccdcx, ccdcy) in the image captured by the ideal camera. This conversion is performed using the following three expressions, that is, an approximate expression for camera calibration.

(6) ccdcx = (ccdx−Centx) / (1 + dist / 100) + Centx

(7) ccdcy = (ccdy−Centy) / (1 + dist / 100) + Centy

(8) hfa = arctan [(((ccdx−Centx) ² + (ccdy−Centy) ² ) ^0.5 ) × pixellength / focallength]

  However, the aberration amount dist (%) is described as dist = f (hfa) using the function f of the half field angle hfa (deg). Further, the focal length of the imaging optical system 30 is expressed by “focallength (mm)”, the ccd pixel length is expressed by “pixelwell (mm)”, and the coordinate value of the center of the lens in the CCD 70 is (Centx, Centy). ).

  In this real space coordinate system XYZ, as shown in FIG. 15, an XY plane and an XY plane in which light paths from which light beams are incident on the target point target on the subject S are extended in the same direction from the projection unit 12. The Y-coordinate value of the intersection with is expressed as “Yptarget” and the X-coordinate value as “Xptarget”. Furthermore, the output pupil position of the projection unit 12 is defined as (0, 0, PPZ). Further, the field of view of the projection unit 12 in the Y direction is defined as a region from the point represented by “Ypftop” to the point represented by “Ypfbottom”, and the field of view in the X direction is represented by “Xpfstart”. To the point represented by “Xpfend”. Further, the length (height) in the Y-axis direction of the projection LCD 66 is represented by “Hp”, and the length (width) in the X-axis direction is represented by “Wp”.

  In step S4005, in order to spatially discretely measure the three-dimensional shape of the subject S as a collection of a plurality of three-dimensional vertices, the two-dimensional code image intersects with the line direction of each pattern light. Referenced spatially with respect to the baseline of the book. As a result, not only a plurality of three-dimensional vertices respectively corresponding to a plurality of discrete points on the outer boundary of the code image are acquired, but also a plurality of discrete points (detected in S4003) inside the code image. A plurality of three-dimensional vertices respectively corresponding to the boundary coordinate points of the codes are acquired.

  FIG. 16 conceptually shows a flowchart of S1230 in FIG. 9 as a three-dimensional color shape detection result generation subroutine.

  First, in S6001, the RGB values (R luminance value, G luminance value, and B luminance value) corresponding to each real coordinate space coordinate value of the three-dimensional vertex group coordinate-converted to the above-described real space three-dimensional coordinate system are described above. Extracted from the surface color image.

  The relationship between the real space coordinate system and the plane coordinate system that defines the surface color image is geometrically associated with each other by the triangulation calculation described above. That is, there is a function used for mapping a code image, that is, a plane coordinate system defining a shape image, which is a two-dimensional image for measuring the three-dimensional shape of the subject S, to a real space three-dimensional coordinate system by calculation. In this case, by using an inverse function of the function, a real space three-dimensional coordinate system can be mapped by calculation to a plane coordinate system that defines a surface color image. Therefore, in S6001, it is possible to extract the surface color value, that is, the RGB value, corresponding to each three-dimensional vertex from the two-dimensional surface color image.

  In step S6002, color shape information is generated by combining corresponding real space coordinate values and RGB values for each vertex. Further, the generated color shape information is stored locally in the working area 120 in association with the corresponding vertex directly or indirectly.

  Subsequently, in S6003, in order to approximately represent the surface shape of the subject S by dividing it into triangles, which are examples of a plurality of polygons, among the plurality of vertices acquired for the subject S, the distance A plurality of vertices adjacent to each other are grouped in groups of three. For each group, three vertices are connected to each other to form one polygon.

  Thereafter, in S6004, for each polygon, a combination of three vertices to be connected to form the polygon is locally stored in the working area 120 as polygon information, directly or indirectly associated with each polygon. .

  Thus, one execution of the three-dimensional color shape detection result generation subroutine is completed, and accordingly, one execution of the three-dimensional color shape detection processing routine shown in FIG. 9 is ended.

  As is apparent from the above description, in the present embodiment, the projection unit 12 (the barrel 24, the projection optical system 32, the LED 62, the light source lens 64, the projection LCD 66, and the projection LCD driver 86) and the computer 110 shown in FIG. The parts executing S2001 to S2004 and S2006 to S2009 in the above together constitute an example of the “projection unit” in the above item (1).

  Further, in the present embodiment, the imaging unit 14 (imaging optical system 30, CCD 70, and CCD interface 88) and the part of the computer 110 that executes S2005 and S2010 in FIG. This constitutes an example of the “imaging unit”.

  Furthermore, in the present embodiment, the processing unit 16 (particularly related to the imaging unit 14 among the plurality of steps shown in FIGS. 6, 7, 9, 10, 13, 14, and 16 of the computer 110). The part that executes the steps excluding the step to be performed) constitutes an example of the “processing unit” in the item (1).

  Furthermore, in the present embodiment, the imaging process performed by the computer 110 executing S2005 in FIG. 10 via the imaging unit 14 constitutes an example of the “three-dimensional measurement imaging process” in the section (1), The imaging process performed when the computer 110 executes S2010 in FIG. 10 via the imaging unit 14 constitutes an example of the “surface color measurement imaging process” in the same section.

  Further, in the present embodiment, the arithmetic processing performed when the computer 110 executes S106 to S108 in FIG. 6 (except for the part corresponding to S1230 in FIG. 9) is the “three-dimensional measurement” in the item (1). An example of “calculation process” is configured, and the calculation process performed by the computer 110 executing S1230 in FIG. 9 configures an example of “surface color measurement calculation process” in the same section.

  As is clear from the above description, in the present embodiment, the method implemented in the three-dimensional input device 10 to acquire the three-dimensional shape information and surface color information of the subject S relates to the item (12). It constitutes an example of a “three-dimensional input method”.

  Further, in the present embodiment, S2001 through S2004 and S2006 through S2009 in FIG. 10 together constitute an example of the “projection step” in the above item (12).

  Further, in the present embodiment, the portions that execute S2005 and S2010 in FIG. 10 together constitute one example of the “imaging process” in the above item (12).

  Furthermore, in the present embodiment, the step (12) except for the step related to the imaging unit 14 among the plurality of steps shown in FIGS. 6, 7, 9, 10, 13, 14, and 16 is performed. It constitutes an example of the “processing step” in the section.

  Furthermore, in the present embodiment, S2005 in FIG. 10 constitutes an example of “three-dimensional measurement imaging process” in the item (12), and S2010 in FIG. 10 represents an example of “surface color measurement imaging process” in the same term. It is composed.

  Furthermore, in the present embodiment, S106 to S108 (excluding the portion corresponding to S1230 in FIG. 9) are an example of “three-dimensional measurement calculation processing” in the above item (12). S1230 in FIG. It constitutes an example of “surface color measurement calculation processing”.

  In addition, in this embodiment, a CCD is used as an image sensor and pixel thinning is performed by selective charge transfer. However, pixel thinning is similarly performed using another element, for example, a CMOS. It is possible to carry out the present invention in an embodiment.

  Although one embodiment of the present invention has been described in detail with reference to the drawings, this is an exemplification and is based on the knowledge of those skilled in the art including the aspects described in the section of [Disclosure of the Invention]. The present invention can be implemented in other forms with various modifications and improvements.

It is a perspective view which shows the external appearance of the three-dimensional input device 10 according to one Embodiment of this invention. It is a top view which shows the structure inside the casing 20 in FIG. FIG. 3 is an enlarged plan view showing a projection unit 12 in FIG. 2 and an enlarged front view showing a light source lens 64 in FIG. 2. FIG. 3 is a side view for explaining the arrangement of the plurality of LEDs 62, a graph showing the illuminance distribution individually realized by the one LED 62, a front view showing the plurality of LEDs 62, and the plurality of LEDs 62. It is a graph which shows the synthetic | combination illumination intensity distribution implement | achieved synthetically by. FIG. 2 is a block diagram conceptually showing an electrical configuration of the three-dimensional input device 10 shown in FIG. 1. 6 is a flowchart conceptually showing main processing executed in the camera control program in FIG. 5. 7 is a flowchart conceptually showing stereoscopic image processing executed in S108 in FIG. FIG. 8 is a plan view and a side view for explaining the principle of the spatial coding method employed in the stereoscopic image processing of FIG. 7 and a plan view showing two sets of mask patterns. FIG. 8 is a flowchart conceptually showing the three-dimensional color shape detection processing executed in S1007 in FIG. 7 as a three-dimensional color shape detection processing routine. 8 is a flowchart conceptually showing S1210 in FIG. 7 as an imaging processing program 114b. 11 is a flowchart conceptually showing a projection process executed in S2004 in FIG. 10 as a projection process subroutine. It is a timing chart for demonstrating an example of the action | operation of the three-dimensional input device 10 shown in FIG. FIG. 10 is a flowchart conceptually showing S1220 in FIG. 9 as a three-dimensional measurement processing subroutine. 14 is a flowchart conceptually showing a code image generation program 114d executed in S4002 in FIG. In step S4005 in FIG. 13, coordinate conversion performed between the two-dimensional CCD coordinate system and the three-dimensional real space coordinate system, and between the two-dimensional LCD coordinate system and the three-dimensional real space coordinate system. It is the side view and top view for demonstrating coordinate transformation. FIG. 8 is a flowchart conceptually showing S1230 in FIG. 7 as a three-dimensional color shape detection result generation routine. It is a top view for demonstrating progressive scanning of an interline transfer type CCD sensor. It is a timing chart for demonstrating the charge transfer principle of the CCD sensor shown in FIG. FIG. 18 is a cross-sectional view of the CCD sensor shown in FIG. 17 and a graph for explaining the temporal transition of the potential distribution. FIG. 18 is a timing chart for explaining the principle of selective charge transfer of the CCD sensor shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 3D input device 12 Projection part 14 Imaging part 16 Processing part 70 CCD
110 computers

Claims (13)

A three-dimensional input device for three-dimensionally inputting an object,
A projection unit that projects multiple types of pattern light onto an object;
An imaging unit that images the object, and (a) each pattern light during a period in which the plurality of types of pattern light is sequentially projected onto the object by the projection unit in order to measure a three-dimensional shape of the object. A three-dimensional measurement imaging process for imaging the object for each projection of the image, and (b) a surface color measurement imaging process for imaging the object in order to measure the surface color of the object, and the imaging unit Three-dimensional measurement calculation processing for measuring the three-dimensional shape of the object based on the imaging result of the three-dimensional measurement imaging processing, and measuring the surface color of the object based on the imaging result of the surface color measurement imaging processing by the imaging unit And a processing unit for performing surface color measurement calculation processing.
The imaging unit is formed by thinning out a pixel thinned image formed by thinning out any of a plurality of pixels constituting an entire image representing the object from the imaging result of the object, and without thinning out any of the pixels. It is configured to be able to selectively extract a pixel non-decimated image,
The processing unit performs the three-dimensional measurement calculation process based on the pixel thinned image extracted from the imaging result of the three-dimensional measurement imaging process by the imaging unit, while the surface color measurement imaging process by the imaging unit. A three-dimensional input device that performs the surface color measurement calculation processing based on the pixel non-thinned image extracted from the imaging result.   The three-dimensional input device according to claim 1, wherein after the imaging of the object, the imaging unit extracts the pixel-thinned image from the imaging result in a time shorter than a time required to extract the pixel non-thinned image. One time three-dimensional measurement imaging process by the imaging unit and one surface color measurement imaging process by the imaging unit constitute one whole imaging process for the object,
The imaging unit sequentially performs individual imaging processing a plurality of times according to the number of the plurality of types of pattern light in one three-dimensional measurement imaging processing, while at least once in one surface color measurement imaging processing. The three-dimensional input device according to claim 1, wherein the individual imaging process is performed.   The three-dimensional input device according to claim 3, wherein the imaging unit performs the surface color measurement imaging process after the three-dimensional measurement imaging process.   The three-dimensional input device according to claim 4, wherein the imaging unit performs the surface color measurement imaging process continuously without performing other processes following the three-dimensional measurement imaging process.   6. The imaging unit according to claim 5, wherein the imaging unit performs a plurality of individual imaging processes in the three-dimensional measurement imaging process and at least one individual imaging process in the surface color measurement imaging process in that order in the same cycle. 3D input device.   The imaging unit performs a plurality of individual imaging processes in the three-dimensional measurement imaging process and at least one individual imaging process in the surface color measurement imaging process in that order at a video rate. 3D input device.   The imaging unit captures the object on which the pattern light is projected and takes out the pixel-thinned image to perform the three-dimensional measurement imaging process, and the object on which the pattern light is projected The three-dimensional input device according to claim 1, which selectively executes a low-speed imaging mode in which the three-dimensional measurement imaging processing is performed by taking out the image and taking out the pixel non-decimated image.   The imaging unit adds illuminance detection values of a plurality of target pixels belonging to each of a plurality of pixel groups into which the plurality of pixels are grouped, and uses the added illuminance. The three-dimensional image according to any one of claims 1 to 8, wherein the pixel-thinned image is extracted from the imaging result of the object by an addition method in which illuminance detection values of a plurality of target pixels belonging to each pixel group are evenly distributed. Input device.   The imaging unit selects, for each pixel group, a representative pixel representing the target pixel from a plurality of target pixels belonging to each of a plurality of pixel groups in which the plurality of pixels are grouped. The pixel-thinned image is extracted from the imaging result of the object by a selection method in which illuminance detection values of a plurality of target pixels belonging to each pixel group are set using the detected illuminance detection values of the representative pixels. The three-dimensional input device according to any one of 8.   The three-dimensional input device according to any one of claims 1 to 10, wherein the three-dimensional input device is a handheld type. A three-dimensional input method for inputting an object three-dimensionally,
A projection step of projecting multiple types of pattern light onto an object;
An imaging step of imaging the object, wherein: (a) each pattern light during a period in which the plurality of types of pattern light is sequentially projected onto the object by the projection step in order to measure a three-dimensional shape of the object; Performing a three-dimensional measurement imaging process for imaging the object for each projection of the image, and (b) performing a surface color measurement imaging process for imaging the object in order to measure the surface color of the object. 3D measurement calculation processing for measuring the 3D shape of the object based on the imaging result of the 3D measurement imaging processing, and measuring the surface color of the object based on the imaging result of the surface color measurement imaging processing by the imaging process And a processing step for performing surface color measurement calculation processing.
In the imaging step, a pixel-thinned image formed by thinning out any of a plurality of pixels constituting the entire image representing the object, and any pixels are not thinned out from the imaging result of the object. Select pixel non-decimated image and
The processing step performs the three-dimensional measurement calculation processing based on the pixel thinned image extracted from the imaging result of the three-dimensional measurement imaging processing in the imaging step, while the surface color measurement imaging processing in the imaging step. A three-dimensional input method for performing the surface color measurement calculation processing based on the pixel non-thinned image extracted from the imaging result.   A program executed by a computer to implement the three-dimensional input method according to claim 12.

Images