JP2011039076A - Improved optical ranging camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an improved ranging camera which simultaneously acquires depth information from all the pixels within a scene. <P>SOLUTION: The apparatus for creating an image, indicating distances to objects in a scene, includes a modulated radiation source having a first modulation function which directs radiation toward the scene; a detector which detects the radiation reflected from the scene and modulated by a second modulation function, and generates, according to the detected modulated radiation, signals responsive to the distance to regions of the scene; a processor which receives the signals from the detector and forms an image, based on the signals, having an intensity value distribution indicative of the distance of the objects from the apparatus; and a controller which varies, at least one of the first and second modulation functions in response to the intensity value distribution of the image formed by the processor. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates generally to a system for generating a three-dimensional image, and more particularly to a camera system that captures an image of a scene and determines the distance to a subject and a portion of the subject in the scene.

  Various techniques for generating a three-dimensional image of a scene, that is, an image including depth and distance information, are well known. Typical methods include the so-called time-of-flight method, phase detection method, and triangulation method. In these techniques, it is generally required to scan an image with, for example, a laser beam to obtain point-by-point depth data. For many applications, however, it is desirable to obtain depth information or distance information simultaneously throughout the image.

  US Pat. No. 5,200,793, cited as a reference for the present invention, discloses a ranging array camera including a light source and a CCD or other sensing element array. The photoelectric modulator modulates both the illumination of the target scene by the light source and the reflected light returning from the scene to the detection element array, and the illumination light and the reflected light have the same modulation pattern. As the modulation frequency changes, the computer analyzes the intensity change at each pixel of the array to determine the distance from the camera to the portion of the scene represented by the pixel.

  S. Christie et al. (S. Christie, et al), page 1301 of “Measurement Science and Technology No. 6 (September 1995)”, cited as a reference of the present invention. On page 1308, a two-dimensional ranging sensor system is disclosed. The system includes a laser that illuminates the scene of interest, a gate that captures an image of the scene, and an enhanced CCD camera. The laser is modulated to obtain a desired length of illumination pulse and the camera is gated to receive light from the scene for an equal length of time. The camera gate pulse is synchronized with or delayed from the illumination pulse. The total intensity of light received by each pixel of the CCD array during the gate pulse is a function of the distance from the camera to the portion of the scene imaged by the pixel. The computer analyzes the light intensity received in this way across all the arrays and then outputs ranging information.

  Similarly, a range gate type camera is also known. Such cameras typically include a gate sensing array that operates in conjunction with a pulsed or strobe light source that illuminates the scene. The array is a gate, such as an "Intensified Lens System" manufactured by Imco Electro-optics Ltd., Essex, UK. It is gated by coupling a gated intensifier to it or other means known in the art. The gate timing of the sensing array is delayed with respect to the light source so that only objects within a desired distance range from the camera are captured. The camera cannot determine the distance to various objects or different points within the distance range.

  One object of the present invention is to provide an improved ranging camera that obtains depth information from all pixels in a scene substantially simultaneously.

  According to some aspects of the present invention, the ranging camera generates both an intensity image and a range image of the scene, and these images are registered with each other.

  According to one aspect of the invention, the ranging camera generates a three-dimensional image of the scene.

  According to another aspect of the invention, the ranging camera outputs information about the distance of one or more subjects in the scene from the camera. This distance information can be used to filter and / or further process the image obtained by the camera, or used as feedback to the camera itself, to detect that distance for greater selectivity and / or accuracy. The function can be adapted.

  In a preferred embodiment of the present invention, the ranging camera includes a radiation source for illuminating the scene of interest and a sensing array, preferably a CCD array, the intensity of the reflected light from the subject in the scene. And is described in the first PCT patent application having the name “Optical Ranging Camera” filed on the same day by the same applicant as the present application. This invention is assigned to the assignee of the present invention, the disclosure of which is hereby incorporated by reference. The radiation source is modulated, for example, by a mechanical shutter or a liquid crystal shutter, which alternately shuts off and opens the radiation source for a predetermined known period and time. The reflected wave is similarly modulated so that the reflected wave reaching the sensing array is preferably blocked and opened for a predetermined known period, which is the same as the period during which the radiated wave source is blocked and opened. Or, more preferably, may be different. The intensity of the radiated waves reaching the detection array and the intensity detected thereby is then processed to determine the distance of the subject in the scene from the camera.

  In some preferred embodiments of the present invention, the radiation source and the reflection source are modulated by one or more mechanical shutters or liquid crystal shutters as described above, as well as other known types of radiation modulators. Can be used. Thus, in some preferred embodiments of the present invention, one or more photoelectric modulators, such as KDP crystal modulators, modulate the radiation source and / or the detected radiation. In addition or in the alternative, in some aspects of the invention, the radiating wave source can be electrically modulated by applying an appropriate waveform to the circuit driving the radiating wave source. Additionally or alternatively, the reflected wave can be modulated by a well-known reset / readout technique in which the sensing array is first reset, i.e., the elements of the array are substantially contained therein. The collected charge is cleared and then after a known predetermined delay period, the collected charge on the element is read and measured.

  The radiation source preferably includes a laser light source, but may alternatively include other types of suitable light sources.

  In some preferred embodiments of the present invention, the ranging camera is described in a second PCT application named “Telecentric Optical Camera” filed on the same day as the present invention. Such a telecentric optical system is included, and the application is similarly assigned to the assignee of the present application, the disclosure of which is hereby incorporated by reference. Preferably, the reflected wave is condensed by a suitable condensing optical system and modulated by a photoelectric modulator such as a KDP crystal modulator. More preferably, the outgoing wave and the reflected wave are parallel light along a common optical axis. In one such preferred embodiment, the outgoing and reflected waves are modulated by the same photoelectric modulator, while in other preferred embodiments they are separately modulated by another modulator.

  In a preferred embodiment of the invention, the radiation source is modulated by a first waveform and the reflected wave is modulated by a second waveform. The shape of the first waveform and the second waveform and the delay time between these waveforms are a function of the intensity detected at each pixel of the detection array at a distance from the camera to the subject imaged by the pixel. Adjusted to give dependencies.

  Preferably, the first and second waveforms and the delay time therebetween are adjusted to define a distance “window”, ie, a distance range of the subject from the camera where the distance of the subject is determined. While the radiated wave source and the reflected wave are modulated by the first and second waveforms, respectively, the camera obtains an image of only the subject in the window. By appropriately adjusting the window, the processing of the detected radiated wave that determines the distance from the camera to the subject in the window is simplified and / or its accuracy is improved. Preferably, the window is also adjusted to maximize the dynamic range of the camera for subjects in the distance range from the camera corresponding to the portion of the scene of interest.

  In some preferred embodiments of the invention, the first and second waveforms are such that the radiated and / or reflected waves from the radiated wave source are blocked and opened multiple times between a single field or frame of the sensing array. The Preferably, the first and second waveforms include a plurality of pulses, and each pulse of the second waveform corresponds to the first waveform according to a predetermined, typically constant, delay time. Follow the pulse you want. Considering that the pulse width of the first and second waveforms is typically less than one microsecond while the duration of a standard video field is about 16 ms, a single field or frame It can be seen that hundreds or thousands of such pulses are included during the time. Such a multi-pulse modulation function is useful for increasing the S / N ratio of an image formed by a camera.

  In some preferred embodiments of the present invention, the window is adjusted to obtain an image of one or more objects of interest. Preferably, the first window includes a large distance range from the camera and is initially defined by modulating the radiated wave source and reflected wave using a relatively long pulse width waveform. . An image of the target subject is obtained in this window and the distance to at least one subject is determined. A second window is then defined that includes the at least one subject and preferably includes a range of distances that is narrower than the first window. In this way, a second image is obtained in which congestion of an undesired subject is reduced and a distance to the at least one target subject can be determined more accurately.

  Preferably, the window is subsequently adjusted to track the at least one subject of interest. As the distance of the subject from the camera changes, the modulation waveform changes so that the subject remains within the distance range encompassed by the window.

  Alternatively or in addition, the distance to the object of interest can initially be determined using a reconnaissance beam, such as a laser ranging device or other ranging devices known in the art. The window is then adjusted so that the subject is within the distance range encompassed by it.

In a preferred embodiment of the present invention, an image of a portion of the scene located at a distance from the camera outside the window is not obtained, or is obtained separately from another subject image in the scene. The present invention can then be used, for example, to distinguish one or more objects of interest from a distant background. In some preferred embodiments, the image of the object of interest is then combined with another image, such as a computer-processed or computer-generated image, and a desired mixed image using conventionally known techniques. Are combined to form
In some preferred embodiments of the present invention, the sensing array is a two-dimensional matrix array, such as a CCD array, where one or more light modulators modulate reflected light reaching the array, and different modulated light is the array. Associated with different rows or groups of adjacent rows. In this way, different distance windows, i.e. different distance ranges selected, are associated with different rows or groups of adjacent rows. Each such row or group of adjacent rows generates a scene line image containing depth information of the subject cut by the line. In order to generate an entire image of the scene including extensive depth information, the field of view of the array is typically scanned over the scene to obtain and combine multiple line images as described below.

  In some of these preferred embodiments, a liquid crystal shutter array is arranged with respect to the CCD array to separately modulate the reflected light reaching each row of the CCD array or a contiguous group of adjacent rows. Preferably, the liquid crystal array is driven so that the reflected light is incident during successive periods, where successive rows or successive groups of adjacent rows of the array are equal. Thus, each successive row or group receives light from the subject in successive distance windows in the scene and is used to determine the distance of such subject from the camera.

  More preferably, in the liquid crystal array, the first row of CCD arrays adjacent to the first edge receives light from a first distance window relatively close to the camera, and the first edge close to the second edge includes: Is driven so that the last row of the CCD array on the opposite side receives light from a final distance window relatively far from the camera. Between the first row and the last row, each successive row of the array receives light from successive distance windows, all of which distance windows are both substantially continuous, preferably substantially. Covers the distance between the first window and the last window.

Furthermore, in another preferred embodiment of the present invention, a liquid crystal shutter array is suitably arranged with respect to the CCD array, and separately modulates the reflected light reaching each pixel or adjacent group of pixels in the CCD array. So that each pixel or group of pixels has its own distance window associated with it. The shutter array is preferably driven appropriately and each distance window tracks the distance to the point of interest in the scene where the image is captured by each pixel or group of pixels. Thus, the distance window associated with each pixel or group of pixels can be adjusted to reduce image convergence and to include and generate an accurate reading of the distance to the point.
In some of these preferred embodiments, where each pixel has its own distance window, adjacent pixel groups are formed to cooperate as “super pixels”. Each such superpixel may include, for example, a 3 × 3 group of nine pixels on which light reflected from the scene is imaged along a substantially common axis. In the shutter array, each of the nine pixels receives light from a different distance window along the common axis. In this way, a complete image of the scene is captured, including depth information with the desired distance resolution, at the expense of lateral resolution. Superpixels of any desired size, such as 1 × 2 pixels, 2 × 2 pixels, or 4 × 4 pixels, can be used to achieve an optimal return of distance versus lateral resolution.

  Although the above embodiments have been described with respect to a liquid crystal shutter array, other preferred embodiments of the present invention include gated addressable microchannel arrays, arrays of photoelectric or acousto-optic modulators, etc. Other types of light modulator arrays can be used for similar purposes.

  In some preferred embodiments of the present invention, where different distance windows work sequentially with different rows of the sensing array, each row forming a line image of a portion of the scene, as described above, the field of view of the array is Scanned across a scene in "push bloom" image mode. In other words, the camera preferably scans across the scene in a direction transverse to the scene, and the line image formed by each row is in a direction substantially perpendicular to the length dimension of the row. Move in aspect and direction. The camera captures a series of line images from each row of the detector array along successive parallel lines.

  Furthermore, the field of view of the camera may be scanned and the effect is the same if the subject in the scene moves laterally with respect to the camera.

  Preferably, the capture of the image by the camera differs in the portion of the scene imaged by any row of the array (except the first row) at any point in the series of images captured by the camera. Synchronized with the movement or scanning of the sensing array with respect to the scene so that it is substantially the same as the portion of the scene imaged by the preceding row of the array at the sequentially preceding points, except having a distance window. ing. In this way, a complete three-dimensional image of the scene is obtained because, for each scene point, the intensity of the reflected light is measured as a function of distance from the camera over a wide distance range.

  In another preferred embodiment of the present invention, the camera includes scanning optics that focus and modulate the line image of the scene so that the line image is continuously projected onto each row of the sensing array. To be. Thus, there is a series of distance windows associated with each row of the array, the window being defined by the time that the line image pauses on the row. In each continuous distance window, the distance to the subject in the scene cut by the line image is determined by the method described above. The scanning optical system may include a high speed optomechanical scanner, an acousto-optic or photoelectric modulator, or other types of scanners known in the art.

  In some preferred embodiments of the present invention, the camera includes associated color filters, preferably three sensing arrays known in the art, red, green and blue. Alternatively, the camera may have only one or two sensing arrays with one or more strips or mosaic color filters known in the art, thereby obtaining a color image. This provides image intensity, color and depth information simultaneously for each pixel in the array.

  In some preferred embodiments of the present invention, two-dimensional intensity and / or color images and three-dimensional distance images (ie, pixel-by-pixel distance information) are used to generate and provide depth images to the camera user. Are registered and superimposed. Such depth images can be obtained using conventionally known methods, for example by drawing contours at equal distances on the two-dimensional image, and by using pseudo colors for the distances displayed in the black and white image. Further, it can be generated by isometric projection generated by a computer.

  In some preferred embodiments of the present invention, the camera is mounted on a mobile platform, eg, an airplane to obtain a topographical map of the land flying over it, or installed either underwater or on the water. Installed on platforms and platforms installed in space. Preferably, the camera operates in a push-bloom mode, as described above, and image capture by the camera is synchronized with the ground speed of the mobile platform. The image captured by the camera is used to draw the terrain and object figure below the platform and can be used to calculate the three-dimensional size of the terrain and object. In another preferred embodiment of the present invention, the camera is mounted on or cooperates with the robot, which uses 3D image information supplied from the camera to guide the desired work-in-motion.

  Alternatively or in addition, in some preferred embodiments of the present invention, the camera determines the various dimensions of the manufactured article and confirms deviations from standard values, for example, quality assurance or quality of the manufacturing process. Used for management. Similarly, the size of the load or item is determined by measuring the surface contour of what is stored in the load or container and whose size can be similarly known or measured using a camera. be able to.

  In another preferred embodiment of the invention, the camera is used with a model scanner to obtain a three-dimensional image of the object of interest. In one such preferred embodiment, these three-dimensional images are high-speed three-dimensional images, such as those known in the art, for reproducing one or more three-dimensional copies or models of the subject of interest, including surface color reproduction. Used together with the system for.

  In yet another embodiment of the invention, the camera is mounted on a vehicle for use in a system for recognizing possible obstacles and avoiding collisions by determining the distance from the vehicle to the obstacle, for example. The

  In another preferred embodiment of the present invention, the ranging camera described above is used with an endoscope to obtain a three-dimensional image of a region of interest within the patient's body.

  In another embodiment of the invention, the camera is coupled to a sensory matrix for use by a blind person. For example, the sensory matrix includes a number of pins that are driven to press the skin of a blind person, such as the forehead, and the pressure applied to the skin by each pin is blind to the eyes. It is proportional to the distance from a free person to the object at the corresponding point in the scene viewed by the camera. Preferably, the blind person controls the distance window of the camera so that the pressure range applied by the pin corresponds to a desired distance range from the blind person.

  Accordingly, in accordance with a preferred embodiment of the present invention, an apparatus is provided for generating an image that displays a distance to an object in a scene, the apparatus having a first modulation function and toward the scene. A modulated radiation wave source for projecting a radiation wave, and a reflected wave reflected from the scene and modulated by the second modulation function, and a distance to the scene area corresponding to the detected modulated wave A detector for generating a corresponding signal; a processor for receiving a signal from the detector and forming an image based on a signal having an intensity distribution indicating a distance of an object from the device; and the processor. And a controller that changes at least one of the first and second modulation functions corresponding to the intensity distribution of the image.

According to a preferred embodiment of the present invention, there is further provided an apparatus for generating an image showing a distance to an object in a scene, the apparatus having a first modulation function and light toward the scene. A reflected radiation source that is reflected from a plurality of parallel spaced segments in the scene, including a modulated radiation source that directs the light, a sensing modulator having a second modulation function, and a sensing array that includes a plurality of sensing elements In addition, a detector that generates a signal corresponding to the detected reflected wave, and at least some segments having a plurality of intervals, and forming an image having an intensity distribution indicating the distance from the device to the object And each of the plurality of spaced segments has a distance limit with respect to the device, the distance limit being determined by the sense modulator. And at least one of the segments having a plurality of intervals has a distance limit different from a limit of the segments having at least one other interval.
According to a preferred embodiment of the present invention, there is also provided a system for generating an enlarged image showing the distance to an object in a scene, the system comprising an apparatus for generating an image as described above The apparatus forms a plurality of images of a scene, and at least one of the plurality of images includes an apparatus including one or more regions of the scene not included in at least one other image of the image; An image processor for combining the plurality of images to indicate the distance to the object in the scene.

  According to a preferred embodiment of the present invention, there is further provided an apparatus for generating an image showing a distance to an object in a scene, the apparatus directing a radiated wave toward the scene. A radiation source, a telecentric optical system that receives and collimates the radiation reflected from the scene, a modulator that modulates the collimated radiation, and the modulated and collimated radiation A detector for detecting the formed image, and a processor for forming an image having an intensity value distribution indicating the distance of the object from the device according to the intensity distribution of the detected image, the radiation wave source and The detector is boresighted and the modulator that modulates the parallel reflected wave reflected from the scene is the scene by the radiation source. Does not modulate the radiated wave directed toward

  According to a preferred embodiment of the present invention, there is further provided an apparatus for generating an image showing a distance to an object in a scene, the apparatus having a first modulation function, A modulated radiation source that directs the radiation toward the surface, and a reflected wave from the scene modulated by the second modulation function, and in response to the detected modulated wave, to the area of the scene A first detector that generates a signal corresponding to a distance and a sighted wave that is reflected by the first detector and reflected from the scene and not modulated by the second modulation function are detected and detected. A second detector for generating a signal corresponding to the intensity of the radiated wave reflected from the area of the scene in response to the radiated wave, and a signal from the first and second detectors to receive the signal from Having an intensity value distribution indicative of the distance of the body, including a processor for forming an image based on the signal.

  According to a preferred embodiment of the present invention, there is further provided a robot guidance system, which includes an apparatus as described above for generating an image showing the distance to an object in a scene, and for guidance. A robot acting on the object in the scene using the distance to the object indicated by the device.

  According to a preferred embodiment of the present invention, there is further provided a system for determining the dimensions of an object, the system comprising an apparatus for generating an image of the object indicating the distance to the object region. And a computer for determining the size of the object using the distance indicated with the image. Preferably, the computer compares the dimensions with a standard one. Alternatively or additionally, the computer uses the determined dimensions to determine the volume of the object.

  According to yet another preferred embodiment of the present invention, there is also provided a system for generating a three-dimensional model of an object, the system for determining the dimensions of the object as described above, A rapid prototyping apparatus that receives the dimensions determined by the system for determining the dimensions and generates a three-dimensional model according to the dimensions. Preferably, the rapid prototyping apparatus receives image information from an apparatus for generating an image of the object, more preferably including color information, and adds the image information to the surface of the three-dimensional model.

  According to a preferred embodiment of the present invention, a vehicle guidance system is further provided, which indicates the distance to an object in the scene in front of the vehicle in the direction of vehicle movement as described above. An apparatus for generating an image and a processor that uses the image and the indicated distance to identify obstacles in the scene.

  In accordance with a preferred embodiment of the present invention, an endoscopic imaging system is provided that has a distal end for insertion into a subject's body and a proximal side for coupling an optical device thereto. And an image forming apparatus as described above, and the image forming apparatus projects a radiation wave into the body through the endoscope and is reflected therefrom. Coupled to the proximal end of the endoscope so as to detect radiated waves.

  According to a preferred embodiment of the present invention, there is further provided a system for generating a substantive image of a scene, the system comprising distance information about a plurality of points in the scene as described above. A range-finding camera to generate an image of the subject and an addressable matrix of tactile probes in contact with the surface of the subject's body, each probe depending on the distance to multiple points in the scene Apply pressure to the body surface.

  According to a preferred embodiment of the present invention, there is further provided a microchannel array device for modulating electron flow, the device being proximate to the microchannel plate through which electrons are accelerated. A plurality of switchable electrodes, each electrode modulating the acceleration of electrons in a corresponding region of the plate.

  Preferably, the microchannel array includes a photocathode, the photocathode receives electrons accelerated through the microchannel plate and generates a corresponding photon, and the electrode is above the photocathode. A conductive transparent material coated on the substrate.

  According to a preferred embodiment of the present invention, there is also provided a camera system for forming an image of a scene, the camera system generating a current of electrons in response to radiation waves incident on it from the scene. And a microchannel array device that generates electrons in response to the electron flow, and a detector that receives the photons and forms a scene image accordingly.

  According to a preferred embodiment of the present invention, a method for generating an image showing a distance to an object in a scene, the method comprising modulating a radiation wave from a radiation source with a first modulation function to produce the radiation. Projecting waves toward the scene, modulating the reflected wave from the scene by the second modulation function, detecting the reflected wave, and depending on the detected reflected wave, the distance between the scene areas Generating a corresponding signal, forming an image having an intensity value distribution of the distance to the object in the scene, and at least one first and second modulation function according to the intensity value distribution of the image Changing.

  According to a preferred embodiment of the present invention, there is further provided a method for generating an image showing a distance to an object in a scene, the method modulating a radiation wave from a radiation source by a first modulation function. Projecting radiation waves onto the scene, defining a segment having a plurality of parallel intervals in the scene, each having a distance limit, and reflection reflected from each segment having a parallel interval A wave is detected, and a signal corresponding to a distance between scene areas is generated according to the detected reflected wave, and an image having an intensity value distribution indicating a distance to an object in the scene is formed. And defining the segment having the plurality of intervals includes defining a distance limit for each of the plurality of segments, and having at least one plurality of intervals Distance limit is different from the distance limit of segments having at least one other spacing.

  According to a preferred embodiment of the present invention, there is further provided a method for generating an enlarged image showing a distance to an object in a scene, the method generating the first image by the method described above. And with respect to the scene, horizontally scanning segments with intervals, generating a second image of the scene in the same manner as the first image was generated, and generating the enlarged image To register the first image with the second image.

  According to a preferred embodiment of the present invention, there is also provided a method for generating a mixed image comprising a target object superimposed on a separately obtained background image, the method comprising: Generating a first image including a target object and indicating a distance to the object in the scene; determining a distance to the object as indicated by the first image; determining a distance to at least one object; Defining a range of distances to include, removing objects in the first image outside the distance region to form a filtered image, and superimposing the filtered image on the background image .

  According to a preferred embodiment of the present invention, a method for generating a mixed image of a scene, including a desired inserted image inserted into a selected zone in the mixed image, the method comprising: Defining a zone in the scene; indicating a distance to an object in the scene; generating a first image of the scene including the defined zone; Determining the position and distance to the zone as indicated by the first image, measuring the inserted image to fit within the boundaries of the zone according to the distance to the zone, and mixing Superimposing the measured insertion image on the zone of the first image to form a processed image.

  Preferably, the method further comprises identifying an object to be inserted in the first image, wherein the distance to the inserted object is less than the distance to the zone to form a filtered image of the object of interest. The first image is filtered so as to superimpose the filtered image of the object to be inserted on the inserted image in the zone.

  The invention will be better understood by reference to the following detailed description of the preferred embodiments of the invention and the accompanying drawings.

1 is a schematic diagram of a configuration of an optical distance measuring camera according to a preferred embodiment of the present invention. FIG. 2A is a schematic diagram of the configuration of a telecentric optical camera according to a preferred embodiment of the present invention. FIG. 2B is a schematic diagram of the configuration of another telecentric optical ranging camera according to another preferred embodiment of the present invention. FIG. 2C is a schematic diagram of the configuration of yet another optical ranging camera that includes two sensing arrays according to a preferred embodiment of the present invention. FIG. 3 is a schematic waveform diagram illustrating an overview of modulation for use with the camera of FIG. 1 or 2B according to a preferred embodiment of the present invention. FIG. 4 is a schematic diagram showing a scene formed by a distance measuring camera according to a preferred embodiment of the present invention. FIG. 5 is a schematic diagram illustrating a distance-dependent image filter function obtained by applying the waveform shown in FIG. 3 to the camera of FIG. 1 or 2B, according to a preferred embodiment of the present invention. FIG. 6A is a schematic diagram illustrating another modulation waveform diagram for use with the camera of FIG. 1 or FIG. 2B, in accordance with another preferred embodiment of the present invention. 6B is a schematic diagram of a distance dependent image filter function obtained by applying the waveform shown in FIG. 6A to the camera of FIG. 1 or 2B. FIG. 7 is a schematic diagram of an adapter-type optical ranging camera system according to a preferred embodiment of the present invention. FIG. 8A is a schematic diagram of an optical ranging camera system for generating a mixed video screen according to a preferred embodiment of the present invention. FIG. 8B is a schematic diagram of a video image generated by the system of FIG. 8A. FIG. 9 is a schematic diagram of an optical ranging camera system for generating a mixed video image according to another preferred embodiment of the present invention. FIG. 10 is a schematic diagram of an optical ranging line image camera according to a preferred embodiment of the present invention. FIG. 11 is a schematic exploded cross-sectional view of a shutter array, preferably for use with the camera of FIG. 10, in accordance with a preferred embodiment of the present invention. 12A is a diagram illustrating a schematic waveform diagram of a modulation for use with the camera of FIG. 10, in accordance with a preferred embodiment of the present invention. 12B is a schematic diagram of a series of distance-dependent image filter functions obtained by applying the waveform shown in FIG. 11A to the camera of FIG. 10 according to a preferred embodiment of the present invention. 13 is a schematic diagram of a terrain imaging system based on the line image camera of FIG. 10, according to a preferred embodiment of the present invention. 14 is a schematic diagram of an image scanning system based on the line imaging camera of FIG. 10, according to another preferred embodiment of the present invention. FIG. 15 is a schematic diagram of an optical ranging image camera according to another preferred embodiment of the present invention. 16 is a schematic diagram of a color stripe filter for use with the sensing array of the line image camera of FIG. 10 according to a preferred embodiment of the present invention. FIG. 17A is a schematic diagram of an optical ranging camera according to another more preferred embodiment of the present invention. FIG. 17B is a partial schematic diagram of a sensing array used in the camera of FIG. 17A to generate a multi-range window, according to a preferred embodiment of the present invention.

Referring to FIG. 1, there is shown an optical ranging camera of a preferred embodiment of the present invention as described in the second mentioned PCT application filed on the same day and cited as a reference of the present invention. Yes. Camera 20 is a sensing array 22, preferably a KAF 0400 CCD array from Eastman Kodak, Rochester, New York, manufactured by EG & G Reticon, Sunnyvale, CA. Includes a CCD array, such as the HS 0512J CCD array. The signal generated by the CCD 22 is processed by a video processor 24, which preferably is a three-dimensional digital image of the scene 26 imaged by the camera, or a two-dimensional image whose distance to the object of interest is shown below. Generate a video image.
The camera 20 further includes an optical system 28 that modulates the light received from the scene 26. The optical system 28 includes an entrance stop 30, a condenser lens 32, a light modulator 34, a focus lens 36, and an exit stop 38. The aperture 30 and the condenser lens 32 form a telecentric optical system, and the lens 32 enters the light received from the scene 26 into the modulator 34 in parallel. The optical modulator 34 preferably includes a photoelectric modulator made of a conventionally known crystal KDP or other suitable material, but instead, such as an acousto-optic modulator, a liquid crystal shutter or a high speed mechanical shutter, Other types of light modulators known in the art may be included. For ease of explanation, the optical system 28 is shown schematically here, but in practice it will be understood that the camera 20 generally includes other lenses and / or other known optical elements. Let's go.

  The camera 20 is also preferably a high power laser diode, such as an OPC-1003-813 laser diode from Opto Power Corporation of City of Industry, California. In order to obtain a substantially uniform illumination of the scene 26 by the laser diode, a suitable optical system known in the art is included. Alternatively, the light source 40 may include other suitable lasers and may include an incoherent light source such as a strobe lamp. The drive circuit 42 supplies input power to the light source 40. A light modulator 44, which may be of the same type as the modulator 34, is associated with the light source 40 to modulate the illumination of the scene 26. Further, the drive circuit 42 may provide appropriately modulated input power to the light source so as to modulate the illumination of the scene without the need for a further modulator 44.

  The controller 46 controls the modulator 34 to modulate the light reflected from the scene and the light incident on the array 22 and also controls the modulation of the illumination of the scene 26 by the modulator 44 and / or circuit 42. . Preferably, the controller 46 communicates with the video processor 24 so that the illumination and reflected light modulation may vary accordingly as described below.

  FIG. 2A shows another preferred embodiment of the present invention, including a telecentric optical ranging camera 50, filed on the same day and described in the second PCT patent application mentioned above, which is incorporated herein by reference. . Camera 50 includes sensing array 22, optical system 28, and light source 40. Camera 50 is generally similar to camera 20, except that in camera 50, a common modulator 52 modulates both the illumination from light source 40 and the reflected light returning from scene 26 back to array 22. The first difference is that. Modulator 52 preferably includes a KDP crystal, or alternatively, any other suitable type of modulator as described above. Illumination light from the light source 40 is substantially collimated by the projection lens 54 and is collimated with the optical axis 56 using a beam combiner 58.

  The modulator 52 may be used to apply a common modulation waveform to both illumination light and reflected light. In a preferred embodiment of the present invention, however, the modulator 52 is a controller for applying multiple light, preferably a first waveform applied to the illumination light and a second waveform applied to the reflected light as described below. 46 is driven.

  It will be appreciated that while the first and second waveforms are respectively applied, the modulator 52 generally transmits both illumination light and reflected light indiscriminately. Preferably, therefore, video processor 24 controls array 22, which generates a signal only in response to light transmitted by modulator 52 according to the second waveform. Thus, for example, in a preferred embodiment of the present invention where array 22 includes a CCD array as described above, video processor 24 synchronizes the electronic shutter of the array with the second waveform, as is well known. To drive. Similarly, the drive circuit 42 preferably drives the light source 40 in synchronization with the first waveform.

  FIG. 2B shows yet another preferred embodiment of the present invention that includes a telecentric optical ranging camera 59. The camera 59 includes a modulator 44 for modulating the illumination light of the scene 26 independently of the modulator 52 that modulates the reflected light, and the camera 50 shown in FIG. 2A. Is the same. Therefore, the same result is obtained when the modulation waveform applied to the camera 20 is also applied to the telecentric camera 59 as described below.

  FIG. 2C shows yet another preferred embodiment of the present invention that includes a telecentric optical ranging camera 64 having two sensing arrays 22 and 65. The sensing array 22 receives light that is modulated by the modulator 52, substantially as described with respect to the sensing array of FIG. 2B and as described in the other preferred embodiments described above. Array 65, however, receives light from scene 26 that is not modulated after reflection from the scene. The array 65 is aimed at the array 22 by a beam splitter 66. Image lens 67 and exit stop 68 thus form an unmodulated image of scene 26 on array 65, which is preferably registered with a substantially (modulated) image formed on array 22. The

  As noted above with respect to camera 59, the modulation waveform applied to camera 20 is also applied to camera 64, as described below, with similar results for images received by array 22. The usefulness of unmodulated images received by the array 65 will also be described below.

  Although the preferred embodiment illustrated in FIGS. 1, 2A, 2B and 2C includes a telecentric image element, the principles of the present invention are as shown and described in the first PCT patent application mentioned above. It will be appreciated that it can be applied to non-telecentric optical ranging cameras as well.

  FIG. 3 schematically shows the waveforms used to drive the modulators 44 and 34 shown in FIG. 1 to modulate the illumination light of the scene 26 and the reflected light therefrom, respectively. Both waveforms preferably have a rectangular pulse waveform, as shown. When each waveform is at its high value, each modulator passes light through it. When the waveform is at its low value, the modulator blocks light. Preferably, both waveforms 60 and 62 maintain their high values during period T to open each of the modulators and the timing at which modulator 34 opens is delayed by time τ with respect to the timing of modulator 44. ing.

  Although the waveform of FIG. 3 is rectangular, other waveforms can also be used, some examples of which are described below. It will also be appreciated that although the pulse waveform of FIG. 3 is idealized for simplicity of the following description, such idealized pulse waveform is not required for operation of the present invention.

  FIG. 4 is a schematic diagram of a scene 70 photographed by the camera 20 as shown in FIG. 1, and the camera is controlled by the waveform shown in FIG. The scene 70 includes a plurality of objects including a first subject 72, a second subject 74, and a third subject 76 disposed at distances D1, D2, and D3 from the camera 20, respectively. The subjects 72, 74 and 76 are illuminated by the light source 40 provided in the camera 20, they are reflected almost toward the camera, and the reflected light is collected by the optical system 28 and converged on the detection array 22. The Considering the period in which the modulators 34 and 44 operate in the CW mode, that is, with a constant opening, the reflected light from each subject is reflected in one or more elements of the detection array on which the image of the subject is formed. Produces a substantially constant illumination. The level of illumination has approximately a functional relationship, inter alia, with the distance of each subject from the camera and the special emissivity of the subject. In response to these substantially constant illuminations, the elements of the sensing array 22 generate their respective baseline signal levels S1, S2 and S3.

  When the modulators 34 and 44 are driven by the waveforms 62 and 60, respectively, however, the signal generated by the array 22 in response to the light reflected from the subjects 72, 74 and 76 reaches each subject and reaches the camera. Due to the propagation time required for light emitted by the light source 40 to return to 20, it differs from the baseline signal. This propagation time is generally represented by ti = 2Di / c, where c is the speed of light, and the subscript i represents the i-th subject in the scene 70.

  As shown in FIG. 3, the selection of values for T and τ defines a distance window 78 partitioned by a minimum distance and a maximum distance, Dmin and Dmax, respectively, where Dmin = (τ−T) c. / 2 and Dmax = (τ + T) c / 2, and when τ <T, Dmin = 0. When the modulators 34 and 44 are operating as shown, light from subjects outside the window 78, such as the subject 76 where D3> Dmax, is rejected by the modulator 34 and thus the array 22 Does not generate a substantial signal corresponding to such a subject.

Further, as shown in FIG. 5, the intensity of the signal Ii generated by the array 22 in response to an object in the window 78, such as the subject 72 or 74, is substantially linear with the distance of the subject from the camera. Dependent. The signals I1 and I2 corresponding to the subjects 72 and 74, respectively, are normalized by the following expression.
Ii = Ii / Siτf (1)
Here, Ii is a denormalized signal from the i-th subject, and f is an appropriate field rate or frame rate at which the array 22 is sampled.

  The baseline signal level Si is obtained by the sensing array 22 while the modulators 34 and 44 are temporarily open, as described above. Further, as shown in FIG. 2C, the sensing array 65 in the camera 64 may be used to obtain these baseline signals. Light incident on the array 22 is modulated to obtain distance information, as described above, while light incident on the array 65 is substantially unmodulated (except for the magnitude at which the light source 40 is modulated). . Therefore, the signal Ii and the baseline signal Si corresponding to the distance are obtained simultaneously.

  In some preferred embodiments of the present invention, where T and τ are chosen to be substantially less than 1 / f, the field or frame period, however, the waveform shown in FIG. It may be repeated many times in succession. In this case, the denormalized signal Ii increases in proportion to the number of times, and the waveform is repeated during each field or frame, which increase typically improves the signal-to-noise ratio of the image captured by the camera. To do. As is well known, if N is the number of times, the waveform is repeated during a single field, and the S / N ratio is improved depending on whether the noise is dominated by the light source or the circuit. The value will typically be in a range between the square root of N and N. Since T and τ are both typically less than 1 microsecond, the waveform may be repeated 1000 times (or more), preferably about 16 milliseconds (or more), so that An improvement in S / N of a factor of 30 or more is made.

The linear function dependence of the normalized signal with respect to the distance from the camera 20 is required for the light emitted by the light source 40 so that the waveform 60 reaches each subject and returns to the camera. This is a result of a changing overlap with the modulation waveform 62 of the reflected light of the illumination light 60 after being effectively delayed by time 2Di / c. This function has a maximum value when the subject is at the distance Di = τc / 2, and has a width (Dmax−Dmin) proportional to T. The distance from the camera 20 to the subject in the window 78 is thus determined from the normalized signal generated according to the subject. For example, the normalized signal I1 due to the subject 72 located near the center of the window is substantially larger than I2 due to the subject 74 located at a greater distance from the camera.

  Although waveforms 60 and 62 are described above as having equal width T, alternatively, these waveforms may have widths of T60 and T62, respectively. For example, T60 or T62 may be extended to provide stronger signals I1 and I2. However, as a result, the function dependence of the normalized signal with respect to the subject distance no longer has the triangular shape shown in FIG. 5, which is rather trapezoidal, and the difference in distance at the center of the window is not identified. Thus, in the preferred embodiment of the present invention, the widths of the waveforms 60 and 62 are equal. This method of determining subject distance is further described in the above-mentioned PCT patent application cited as a reference for the present invention. As described in these PCT patent applications, other modulation waveforms may be used to drive modulators 34 and 44 as well, but the general dependence of the normalized signal on subject distance and Application of the camera 20 as described herein is most easily described by reference to the rectangular pulse waveform shown in FIG.

  Although the waveforms 60 and 62 are shown as having an ideal rectangular shape, in practice, the waveforms provided to the projected and reflected light by the modulators 34 and 44, respectively, are illustrated, for example, below FIG. 6A. As shown, it has a finite rise time and fall time. As a result, the corners of the response function are rounded rather than sharp as shown in FIG. The principles of the present invention, however, are equally applicable when other non-ideal waveforms are used.

  It will also be appreciated from the above discussion that the center position of the window 78 may be controlled by changing τ and the width of the window may be controlled by changing T. Preferably, when the target object is identified in the scene, τ is appropriately controlled by the controller 46 so that the target object is maintained near the center of the window automatically or by the command of the camera operator. At that time, the normalized signal by the subject is maximum.

  Further, T is preferably controlled by the controller 46 to limit the width of the window 78 to include only the range of subject distances from the camera where the subject is located. This limitation on T, as shown in FIG. 5, increases the difference in normalized signal between subjects at different distances by increasing the slope of the dependence of the normalized signal on distance. Useful for increasing the accuracy of distance determination. Limiting T is further useful to reduce image “congestion” by erasing uninteresting objects from the image obtained by the array 22, such as subject 76 of FIG. .

  In some preferred embodiments of the invention, τ and T are controlled by controller 46 to optimize subject dynamic range and distance detection by camera 20. For example, when the scene contains several objects of interest at different distances from the camera, τ and T are controlled to shift the window center to the distance of the object farthest from the camera. You may do it. As is well known, the irradiance generated in the array 22 by the light reflected from the subject is substantially reduced to the third or fourth power of the subject from the camera. Thus, the signal generated (denormalized) in response to the farthest subject is generally the weakest signal, and shifting the window center to the farthest subject distance minimizes the dynamic range of this signal. To help. Similarly, if one of the subjects now has a substantially lower reflectivity than the other subject, the window will be centered on the low reflectivity subject so that the window will expand the dynamic range of the signal thereby. You may do it.

  FIG. 6A is a timing diagram of another preferred embodiment of the present invention, wherein the waveform 60 applied to modulate the illumination light and the waveform 62 applied to modulate the reflected light have a substantially trapezoidal shape. And includes a modulation waveform that is closer to reality than the idealized rectangular pulse of FIG. Further, the duration T60 of the waveform 60 is substantially longer than the duration T62 of the waveform 62.

  FIG. 6B schematically shows the normalized signal strength Ii received from the subjects 72, 74 and 76 as shown in FIG. 4 when the camera 20 is operating according to the waveform of FIG. 6A. . The normalized signal for a subject at a distance smaller than Dmin is substantially equal to zero, as described above. For a subject at a distance greater than Dmid and up to Dmax, such as subject 76 at distance D3, the normalized signal is substantially equal to one. Dmid is highly dependent on the rise time of T62 and waveforms 60 and 62. Dmax depends on the sum of time τ and T60. Preferably, Dmax is selected to be larger than the distance from the camera 20 of the farthest important subject in the scene. Further, Dmax may be chosen such that Dmax> c / f, where c is the speed of light and f is the camera field rate or frame rate, as previously described, and thus Dmic. All objects at greater distances will give a normalized signal substantially equal to one.

  Thus, within the desired distance range between Dmin and Dmid, the normalized signal is a monotonically increasing function, based on which the distances D1 and D2 of each of the objects 72 and 74 are It is determined based on the principle. Subjects at distances outside this range are preferably ignored. The above range may be increased or decreased by changing T62. For a subject within the target range, each value of the normalized signal corresponds to a unique distance, unlike the function shown in FIG. 5, in which it is between 0 and 1 The given value of the normalized signal corresponds to the distance between two different subjects. In the diagram of FIG. 5, these two distances must then generally be distinguished from one another by comparing the results obtained using other means, for example using different distance windows.

  FIG. 7 illustrates one preferred embodiment of the present invention, in which the camera 20 obtains 3D image information from a distance window 78 in the scene 70, the center distance and width of the window being It is controlled appropriately. Preferably, camera 20 initially receives two-dimensional image data while modulators 34 and 44 operate in CW mode, as described above. The image data is received by the image analyzer 80, which identifies the subject in the scene 70 and provides a video image to the operator via the console 82 and the associated video display 84. Image analyzer 80 and console 82 preferably include one or more digital computers adapted for image processing, as is well known. An operator identifies at least one object of interest, such as object 72, in scene 70, preferably using a trackball, mouse or other well-known pointing input device in conjunction with console 82, and an image analyzer. The subject is designated or instructed to 80 and / or another system controller 86.

  Once the target subject is identified, the system controller 86 requests an initial distance reading for the subject 72. As shown in FIG. 7, the initial distance reading can be obtained by aiming the subject 72 with a known distance measuring device 88 such as a laser distance measuring device and receiving the distance reading therefrom. The initial distance reading is used by the system controller 86 to determine appropriate values for τ and T to generate a desired distance window, preferably centered on the distance of the subject 72.

  In a further preferred embodiment of the present invention, the system controller 86 obtains an initial distance reading without using the ranging device 88. Preferably, modulators 34 and 44 are controlled by controller 46 to provide a wide window centered at the estimated distance of subject 72 using a relatively large value of T. When the distance of the subject 72 is not estimated well, τ may be scanned over a predetermined numerical range so as to generate a plurality of windows at different distances until the initial distance is found. Thereafter, T is preferably narrowed so that the window is narrowed closer to the range of distances including subject 72 and the other subject of interest, and as described above, the distance to these subjects is determined with optimum accuracy. Image congestion may be reduced.

  In the preferred embodiment described above, the object of interest is initially identified interactively by the operator, but in other embodiments of the invention, the image analyzer 80 is self-supporting based on pre-programmed image attributes. Alternatively, the target subject may be identified. For example, the image analyzer may be programmed to identify a vehicle, person or structure using conventionally known image analysis and / or automatic target recognition techniques. When these pieces of identification information are conveyed to the system controller 86, the system controller 86 may control the distance measuring device 88 and / or the camera 20 to obtain and determine the distance to the identified subject.

  However, once at least one subject of interest has been identified and its initial distance has been found, then image analyzer 80 preferably continues to track the subject and its distance from the camera. System controller 86 varies τ and T as appropriate to maintain window 78 at a desired width substantially centered on the subject.

  FIGS. 8A and 8B schematically illustrate one preferred embodiment of the present invention, in which the camera 20 is used with a well-known image processor 90 to generate the desired image.

  A mixed image is typically a person in front of a separately photographed background, and the image is common in entertainment and news reporting to artificially insert the desired subject photographed by a video camera. Used. Images of the intended person or other subject must generally be obtained against a well-known, predetermined color, typically a blue background. After an image of the target person or other subject is obtained, it is chroma-keyed, i.e., all pixels in the image of a predetermined color are identified and a background image obtained separately. Is a pixel in which the above color is identified, and is replaced for each pixel. In this color fixing method, a background of a predetermined color is fixed behind a person or a target subject, and the target person or other subject substantially includes no predetermined color. Request.

  As shown in FIG. 8A, however, camera 20 can selectively obtain images of persons 92 and 94 within window 96, but more distant buildings 98 are excluded from the obtained image. Preferably, τ and T are substantially equal, even though the normalized signal generated by the array 22 in response to images of the persons 92 and 94 is located farther from the camera 20 than the person 92. Adjusted to be equal. Such normalization of the normalized signal is useful for making a more accurate determination up to person 94, with a clearer reproduction of both persons 92 and 94, and also to produce a more pleasing image. Useful for.

  As shown in FIG. 8B, the image processor 90 receives the selectively obtained image from the camera 20 and generates a desired combined video image on the monitor 102 or records and records such combined video image. Or mix it with the background 100 for transmission. Neither fixed fixed curtains (backdrops) nor fixed colors are required. The background 100 may be an actual image, an image obtained separately, or a computer composite image or a combination of an actual image and a composite image.

  FIG. 9 schematically illustrates another preferred embodiment of the present invention, in which the camera 20 and image processor 90 are used with an advertising bulletin board 104 to display a desired picture, such as an advertising message. A mixed image artificially arranged inside the advertising bulletin board is generated. Such preferred embodiments can preferably be used in international sports broadcasts, for example to advertise specific to audiences in different countries.

  Accordingly, as shown, the camera 20 captures an image including the subject 106 and the advertising bulletin board 104, and determines the distance to the subject and the advertising bulletin board as described above. The image processor 90 additionally identifies the boundary 108 of the billboard 104 and uses known image analysis methods to determine the position and orientation of the billboard relative to the camera 20. The boundary 108 preferably includes easily identifiable features such as retroreflective strips, strips of infrared reflective material, or infrared LED light or other light source strings. Further, the billboard 104 may be color fixed to help the image processor 90 accurately determine the position and orientation. The image processor then inserts the desired image or message into the advertising bulletin board 104 in the image received from the camera 20 after appropriately adjusting the size and virtual viewing angle of the image or message. The image of the subject 106 is superimposed on the image or message in the advertisement bulletin board.

  FIG. 10 schematically illustrates a ranging line image camera 110 according to another preferred embodiment of the present invention. The camera 110 can be used in the same manner as known image capture devices such as linear array CCD cameras and scanners. Such devices generally generate an image (no depth information) of a narrow and linear portion of a subject or scene. The field of view of the device is then optically or mechanically scanned over the object or scene, and the multiple images so generated are combined to form a single, two-dimensional image. The As will be described below, the visual field line imaging camera 110 preferably scans the subject or scene as well to form an image that includes three-dimensional information.

  As shown in FIG. 10, the camera 110 includes a sensing matrix array 112, similar to the array 22 of the camera 20, preferably a CCD array. The array 112 includes a plurality of parallel rows 114 of sensing elements. Video processor 116 provides drive signals to array 112 and receives image response video signals therefrom. Similar to the camera 20, a line image camera 110 is shown forming an image of a scene 26, which is illuminated by a light source 40 driven by a drive circuit 42 via a modulator 44.

  The optical system 118 of the camera 110 includes a condenser lens 120, a linear shutter array 122, and an image lens 124. The shutter array 122 includes a plurality of narrow shutter elements 130, 132, 134,..., 136 and preferably includes a sensing array 112 having elements 130-136 oriented in a direction substantially parallel to the row 114. Is disposed substantially parallel to the surface of the. Controller 126 controls linear shutter array 122 by integral switching circuitry 128 and is preferably controlled by controller 126 to open each of shutter elements 130-136 according to a desired sequence. The controller 126 further controls the drive circuit 42 and the modulator 44 as well as receives and controls data from the video processor 116 as described above with reference to the controller 46 of the camera 20.

  Preferably, the shutter array 122 is a liquid crystal shutter array, or a gated microchannel element, or an array of photoelectric or acousto-optic modulation elements.

  FIG. 11 illustrates, for example, one preferred embodiment of the present invention, in which the shutter array 122 includes a gated array, shown in parallel exploded cross-sectional view. The array 122 includes a second glass plate 145 having a photoanode 137 coated on the first glass plate 138, a microchannel plate 139, and an integral switching network 128 superimposed thereon. Photoanode 137 and photocathode 143 include any known suitable material, such as a neighbor. The switching network 128 preferably includes a plurality of transparent electrodes 141, for example, strips of indium tin oxide (ITO) coated on the surface of the photocathode 143. For clarity of illustration, the glass plates 138 and 145 and the microchannel plate 139 are shown separately in the drawing, but in actual use, as is well known, these elements are generally contained within a vacuum case. Held close together.

  The microchannel plate 139 is preferably similar to the known normal microchannel plate, but the different groups of microchannels in the plate that make up the shutter elements 130, 132, 134,. They are individually switched or gated by corresponding electrodes 141.

  The electrodes are individually switchably connected through a network 128, preferably in the range of -200 VDC, to a known negative high voltage source through the network 128.

  As shown in FIG. 11, the operation of the shutter array 122 will be understood from the following example. Photons impinging on the photoanode 137 generate photoelectrons in the vicinity of the microchannel plate 139. When one of the electrodes 147 corresponding to the element 130 is connected to the negative high voltage, photoelectrons generated in a region close to the element 130 are accelerated through the microchannel of the plate 139 constituting the element 130. As is further known, secondary electrons are further generated in the microchannel. These electrons then pass through the electrode 147, enter the photocathode 143, and generate photons in the cathode region close to the element 130. Assuming that the remaining electrode 141 is not connected to a negative high voltage, however, this process does not occur in any of the other elements 132, 134,. In this way, the shutter element 130 is effectively open and the remaining shutter elements are effectively closed. The shutter elements 132, 134,..., 136 are preferably opened sequentially following the element 130 in the same manner.

  The condenser lens 120 forms an image of the light reflected from the scene 26 on the shutter array 122, and the shutter array 122 modulates the light as described below. The modulated light from the shutter array is then imaged onto the detection array 112 by the imaging lens 124. Preferably, the optical system 118 has each element 130-136 of the shutter array 122 above one row 114 of the sensing array 112, depending on the number and size of the elements 130-136 compared to the number and size of the rows 114. , Image onto a group of adjacent rows 114. In FIG. 10, the imaging lens 124 is shown as imaging the shutter array 122 onto the sensing array 112 at a magnification of approximately 1 ×, but the relative relationship between the shutter array 122 and the sensing array 112 is shown. It will be understood that depending on the dimensions and the relative dimensions of their elements 130-136 and rows 114, an appropriate magnification may be selected by proper selection and arrangement of imaging lenses 124. Further, when the size of the shutter array 122 and shutter elements 130-136 can be made small enough to fit the dimensions of the sensing array 112 and row 114, the shutter array 122 can be directly adjacent to the sensing array 112, It may be arranged and coupled to it by direct contact to better avoid the need to interpose the imaging lens 124 and simplify the optical system 118.

  FIG. 12A is a timing diagram that schematically illustrates switching and modulation pulses 140 applied to the modulator 44 and shutter array 122 of the camera 110 in accordance with one preferred embodiment of the present invention. The modulator 44 and the shutter element 130 are initially open and a rectangular wave with a pulse width T allows light to pass through it. Since the modulator 44 and the element 130 are close to each other, substantially the same modulation waveform is applied to the element 132.

  As element 132 closes, element 134 opens, and so on, down array 122 sequentially until the last element 136 is opened.

  As described above, as each of the shutter elements 130-136 is opened, the light reflected from the scene 26 is incident on the corresponding row 114 or row group of the sensing array 112.

  As shown in FIG. 12B, the modulation pattern 140 shown in FIG. 12A results in a distance window 142 having a series of successive overlaps defined by the shutter elements 130-136. Each row 114 or row group of the sensing array 112 receives light only from the subject of the scene 26 within the window defined by its corresponding shutter element 130-136. The widths of windows 142 and their respective center distances, along with normalized signal strength, are determined as described above with reference to FIGS. 3, 4, and 5, and also with reference to FIGS. 6A and 6B. The The mutual overlap of the distance windows 142 is used to eliminate the distance ambiguity problem encountered when the object of interest is contained only within a single distance window, eg, according to the diagram of FIG. Also good.

  It will be appreciated that the modulation pattern 140 and the resulting window 142 are shown here by way of example, and that other suitable modulation patterns can be used as well. For example, the modulation waveform applied to the modulator 44 may be similar to the waveform 60 shown in FIG. 6A and described above with reference to that drawing. The waveforms applied to the shutter elements 130 to 136 are the same as the waveform 52 of FIG. 6A, and may be sequentially applied as shown in FIG. 12A. In this case, a distance window having a series of overlaps is defined similar to that shown in FIG. 6B.

  Furthermore, the modulation pulses applied to the series of shutter elements 130-136 may be timed so that they overlap, and the corresponding distance windows have a greater overlap with their respective neighborhoods. Furthermore, it may be introduced between the modulation pulses so that the delay spreads away from the distance window. Further, the waveforms applied to the series of shutter elements 130-136 need not all have equal pulse widths, but have an increasing period from element to element as they go down the array. The distance window may be wider at a greater distance from the camera 110.

  It will be appreciated that each image frame captured by the line imaging camera 110 includes an array of image slices. Each image segment is substantially continuous in the vicinity and lateral direction, but captures subjects within different distance windows. Thus, an object that is on the boundary between two image sections has different levels of intensity normalized by the two sections, which are partly in one of the two sections. Appears but may not appear elsewhere. Thus, as previously mentioned by analogy with known and conventional linear array imaging devices, in preferred embodiments of the present invention, image slices are scanned across the scene optically and mechanically. Each target object within is captured in at least one image slice having a distance window containing the target object.

  FIG. 13 schematically illustrates the use of the line imaging camera 110 of the airborne system 146 for 3D image inspection. The system 146 includes a camera 110 that is mounted to look down from the airplane 148, with the length of the shutter elements 130-136 being substantially perpendicular to the flight direction of the airplane. Camera 110 typically includes buildings 152 and 154, vegetation 156 and terrain 158. As shown in FIG. 13, each shutter element 130, 132, 134,..., 136 of array 122 defines a respective image segment 160, 162, 164,. The sections preferably have substantially equal angles or cut thicknesses defined by the height of the shutter elements of the array 122, and have substantially equal depths, for example as shown in FIG. 12B. Has a distance window. In practice, any section number can be used until the number of rows 114 of array 112 is reached, but FIG. 13 includes only a few sections.

 As the plane 148 flies over the scene 150, the sections 160-166 scan over the scene in a "push-bloom" format so that each section is The entire two-dimensional image of the subject in the scene in the two-dimensional image of the subject in the scene within each distance window is captured line by line. The distance from the camera 110 to each subject in any one window of the intercept, eg, the building 152 in the intercept 166, is accurately determined from its normalized signal strength, as described above. If desired, the distance window may be adjusted accordingly to accommodate changes in altitude in the scene, such as the slope of the terrain 158. Preferably, the altitude and ground speed of the airplane 148 are known and controlled, but are most preferably constant, and the image processor 170 that receives the line image from the camera 110 combines both the line image and distance information of the scene 150. It can be incorporated into a full 3D image.

  More preferably, the array detector 112 is scanned at a rate that is synchronous with ground speed, and the area of the scene captured in the slice 160 in one field or frame obtained by the camera 110 is, for example, within the following field or frame. Is captured by a section 162 of Thus, video processor 194 or an associated image capture system can conveniently and accurately register multiple slices and corresponding depth windows.

  In some preferred embodiments of the present invention, the line image and distance information obtained by the camera 110 determines the lateral dimensions and elevation of the imaged terrain, such as the buildings 152 and 154 and the terrain 158. To be used in king dimension photogrammetry. These dimensions and altitudes may then be incorporated into, for example, accurate topographic maps and land survey maps. They can also be used to calculate the volume of things that are usually very difficult to determine with high accuracy from ordinary photographs or video images, such as the building 152.

  Although FIG. 13 shows a camera 110 mounted on an airplane 148, other similar preferred embodiments of the present invention where the camera is mounted on an aerial platform or an alternative space or maritime transport platform are also possible. is there. For example, the camera may be mounted on a marine vessel or a submersible vessel in order to create a topography under water or under the sea floor.

  FIG. 14 schematically illustrates another preferred embodiment of the present invention, in which the camera 110 is fixed and includes a number of line image slices 160 that include distance information about the subjects 174, 176, and 178. ~ 166 are generated. A mirror 180 attached to the rotating shaft 182 scans the line image across the scene 172, and the image processor 170 obtains an overall three-dimensional image of the scene in the same manner as described above with respect to FIG. it can. The shaft 182 is preferably rotated by a galvanometer, polygon scanner or other known means.

  FIG. 15 illustrates yet another preferred embodiment of the present invention, in which the line imaging camera 190 obtains a three-dimensional image of the scene 26 quickly. Camera 190 includes a sensing array 192 that preferably includes a plurality of rows 193, such as array 112 of cameras 110, and a video processor 194 coupled to array 192, similar to video processor 116. Yes.

  As in the preferred embodiment described above, the light source 40 is driven by the drive circuit 42 to illuminate the scene 26, while the modulator 44 alternately opens and closes the path of the illumination light 196 according to the desired modulation waveform. The cylindrical lens 198 converges the beam 196 such that a narrow beam 200 of illumination light is generated on the scene 26, the major axis of the light beam 200 of illumination light being perpendicular to the plane of FIG. Are substantially parallel. The scanning mirror 202 is rotated by a scanning motor 204 such as a well-known galvanometer device in order to scan the light beam 200 by vertically cutting the light beam 200 perpendicularly to the range of the scene 26. Such scanning can be similarly performed using, for example, a polygon mirror that is appropriately rotated by a servo motor.

  The light reflected from the scene 26 is collected on the array 192 by the optical system 210, scanned and converged. The optical system 210 includes an incident side stop 212, a condenser lens 214, an optical modulator 216, a converging lens 218, and an exit stop 220. The modulator 216 preferably includes an acousto-optic crystal that is driven by the controller 222 to deflect the image beam 224 received from the scene 26 to traverse the array 192 and scan the beam. . It will be appreciated that the image beam 224 includes a narrow line of reflected light, similar to the illumination beam 200.

  Controller 222 further controls and receives image data from video processor 194 and controls driver 42, modulator 44 and galvanometer 204 as described with reference to controller 126 shown in FIG.

  Preferably, the illumination beam 196 scans across the scene 26, the modulator 44 repeatedly opens and closes, and the illumination beam 200 illuminates a series of horizontal slices. For each such segment, the modulator 216 preferably, in turn, spans substantially all rows 193 of the array 192, starting from the top row and ending at the bottom row. 224 is scanned. Thus, the image beam 224 continues to remain in each of the rows 193 for a substantially equal pause period that is functionally equivalent to the pulse width T of the modulation waveform shown in FIGS. 3 and 12A. Similarly, as in FIG. 12A, the beam 224 reaches each row 193 with a delay time for the illumination beam 196 that is substantially equal to the accumulated pause period of the preceding row of the array.

  Thus, it will be appreciated that for each of the 26 slices defined by the position of the illumination beam 200 where the modulator 44 is open, the camera 190 forms a line image of the scene that includes distance information. I will. The distance information in the intercept is captured in subsequent distance information, with each window corresponding to a different row 193 of the array 192. As motor 204 and mirror 202 sweep beam 196 across scene 26, camera 190 forms a complete, three-dimensional image of the entire scene.

  In another preferred embodiment of the present invention, the camera 190 is secured to an airplane 148, for example, instead of the camera 110, or some other suitable type of mobile platform, as shown in FIG. Scan over the scene in push bloom mode. In this case, since the illumination beam 200 scans the scene due to the movement of the airplane, the motor 204 is not necessary. In addition, all distance windows obtained in a single field or frame of the turtle 190 are within the same image segment (unlike camera 110) and are combined cubics of individual line images generated by the camera. The display in the original image is simplified. The mirror 202 is preferably replaced by a beam combiner, such as the beam combiner 58 shown in FIG. 2, so that the illumination light beam 196 and the reflected light beam 224 are aimed.

  The preferred embodiments of the present invention described above can generally be applied to generate either black and white images or color images. Black and white images are generated by using visible light, infrared light or ultraviolet light and appropriate optics to illuminate the scene of interest, and using modulators and detectors that control and receive the light. be able to.

  Furthermore, the principles of the present invention can be similarly applied to generate three-dimensional images using other types of radiation waves. For example, an acoustic image, such as a sonar or ultrasound image, can be generated using a well-known acoustic sensor such as a microphone or piezoelectric crystal and an array of radiators, and depth information about the subject in the image. Can be determined by the present invention. For submarine acoustic images, as is well known, acoustic lenses can be conveniently used in place of the optical lenses shown in FIGS. 1 and 1A-C.

  FIG. 16 illustrates the application of sensing array 112 for use with camera 110 to generate a color image, according to a preferred embodiment of the present invention. Array 112 is covered by a color stripe filter 230 having a number of stripes 232, each of which substantially covers a single respective column of array 112 that is substantially perpendicular to row 114. The stripes 232 preferably alternately pass light of different colors, as is well known, in the order of red, green, blue, green, red, green, etc. Accordingly, red, green, or blue light is alternately incident on adjacent pixels in the row 114 of the array 112. (Preferably the light supplied by light source 40 is white light.) Signals from adjacent pixels are preferably combined to derive local luminance and chrominance information.

  Thus, each of a number of line images generated by the camera 110 typically includes color information in addition to intensity and depth information. These color line information is then preferably scanned as described with reference to FIG. 13 or FIG. 14, for example, to generate a three-dimensional color image.

  Similarly, the sensing array 22 of the camera 20 shown in FIG. 1, or the camera 50 of FIG. 2A or the camera 59 of FIG. 2B may be a color stripe filter, or a well-known color mosaic filter, as shown in FIG. May be included.

  In addition, a CCD sensing array with a suitable dichroic filter, such as a well-known dichroic prism assembly, generates a RGBD (red-green-blue-depth) image, i.e. a color image containing depth information. 20, camera 50, camera 59, or array 22 of cameras 240 described below.

  Further, in a preferred embodiment of the present invention, the 3D image information generated by any of the optical ranging cameras described above is fused and used to enhance the 2D intensity image generated by the camera. Similarly, the two-dimensional intensity image can be used to sharpen or enhance a three-dimensional image.

  It will be appreciated that the two-dimensional and three-dimensional images are essentially registered with each other because they are from the same sensing array. Thus, for example, image information of 2D and 3D images can be used to sharpen and / or smooth the contours of the image edges, and / or using well-known image filtering techniques for images close to such edges. It may be used to correct the color.

  FIG. 17A illustrates a pixel-wise ranging camera 240 according to yet another preferred embodiment of the present invention. Camera 240 includes a sensing matrix array 241 that includes a plurality of sensing elements 242, similar to array 112 shown in camera 110, as shown in FIG. The camera 240 also includes an optical system 243 that includes a liquid crystal shutter array 244 in addition to the lenses 120 and 124 as the camera 110 has.

  Other parts of the camera 240 shown in FIG. 17A are substantially the same in configuration and function as the parts of the camera 110.

  The shutter array 244 includes a plurality of shutter arrays 246. The optical system 243 is appropriately aligned and arranged with respect to the sensing array 241 and each shutter element 246 individually modulates the reflected light from the scene 26 that reaches each sensing element 242 of the array 241 or a group of adjacent sensing elements. To do. The shutter array 244 may be controlled by the controller 126 and modulated such that different shutter elements 246 open and close simultaneously or at different times. Thus, each sensing element 242 or group of elements may have its own distance window associated with it, depending on the modulation of the corresponding shutter element.

  The shutter array 244 is preferably operated appropriately so that each distance window tracks the distance to a point in the scene 26 where the image is captured by each sensing element 242 or group of elements. Good. Each such distance window may be adjusted to reduce and prevent image congestion and to include and generate an accurate reading of the distance to the point.

  In some preferred embodiments of the present invention using a camera 240, as schematically shown in FIG. 17B, the group of sensing elements 242 in the array 241 that are adjacent to each other are “super-pixels”. (pixels) "250, and is configured to operate in cooperation. Each such super pixel 250 includes, for example, a 3 × 3 group of nine elements 242 on which light reflected from the scene 26 is substantially along a common axis by the optical system 243. An image is formed. The shutter array 244 operates so that each of the nine elements 242 of the superpixel 250 receives light from different distance windows along the common axis. These windows are labeled W1-W9 in FIG. 17B and correspond, for example, to the shutter elements 246 associated with the nine sensing elements that open and close sequentially according to the modulation waveform illustrated in FIG. 12A. You may do it.

  In this way, a complete image of the scene, including depth information having the desired distance resolution, is captured in a single field or frame of the camera 240, at the expense of lateral resolution. For example, superpixels of a desired size of 1 × 2 pixels, 2 × 2 pixels, or 4 × 4 pixels may be used to achieve optimal convergence of subject distance in the scene.

  In some preferred embodiments of the present invention, a ranging camera, such as the camera 20 described above, is mounted on or associated with the robot, and the robot guides its operation in the desired job. 3D image information supplied from the camera is used.

  In another preferred embodiment of the invention, the camera 20, 50 or 110 is quality assurance or quality control in the manufacturing process, for example detecting the dimensions of the various articles being manufactured and checking how far they deviate from the standard values. Used to. Similarly, by measuring the contour of the surface of the bulk material, the volume of the material is determined. The volume of an object stored in a container, the size of which is also known or measured using the camera, can be determined by similar techniques.

  In another preferred embodiment of the invention, the camera 20, 50 or 110 is used in a model scanner to obtain a three-dimensional image of the object of interest. In one such preferred embodiment, these three-dimensional images are converted to a rapid three-dimensional model, as is well known, to reproduce one or more three-dimensional copies or models of the object of interest, including surface color reproduction. Used with the prototype.

  In yet another preferred embodiment of the invention, the camera 20 is mounted on a vehicle, for example for use in a collision avoidance system. Preferably, the camera 20 is part of an image analysis system that recognizes possible obstacles and uses the camera output to determine the distance from the vehicle to the obstacle.

  In another preferred embodiment of the present invention, a ranging camera such as that described above, such as camera 20 or 50, is used with an endoscope to obtain a three-dimensional image of a target site within a patient's body. The

  In another preferred embodiment of the present invention, camera 20 or camera 50 is connected to a sensor matrix for use by a blind person. The sensor matrix has a large number of pins, which press the skin of a blind person, for example the forehead, at which time the pressure applied to the skin by each pin is from that person by the camera. It is proportional to the distance to the object at the corresponding point of the viewed scene.

  Preferably, the blind person controls the distance window of the camera, and the range of pressure applied by the pin corresponds to the desired distance range from the person.

  It will be understood that the preferred embodiments described above have been described by way of example, and the scope of the present invention is defined only by the claims.

Claims (54)

An apparatus for generating an image showing a distance to a subject in a scene,
A modulated radiation source having a first modulation function and directing a modulated radiation wave to a scene, wherein a portion of the radiation wave from the radiation source is reflected from a plurality of points in the scene. A radiation source adapted to reach the device;
A detector for detecting a radiation wave reflected from the scene and modulated by a second modulation function, wherein the second modulation function defines a distance window for receiving a reflection from a position in the scene, and In response to the detected modulated radiation, the device generates a signal corresponding to a portion of the portion of the reflected radiation, the portion of the portion being a distance of a point in the scene and A detector corresponding to a position of the point within the distance window, wherein a portion of the portion is associated with a particular signal from the device;
A processor for receiving an image from the detector and forming an image based on the signal having an intensity value distribution indicating a distance of a subject within the distance window from the device;
The width of the distance window is adjusted by changing at least one of the first modulation function and the second modulation function by a variation corresponding to the distance indicated by the intensity value distribution of the image formed by the processor. A controller,
Including the device.   The controller changes at least one of the first modulation function and the second modulation function by an amount of change corresponding to the distance indicated by the intensity value distribution of the image formed by the processor, The apparatus of claim 1, wherein the apparatus adjusts timing and thereby adjusts a distance of the distance window.   The apparatus according to claim 1 or 2, wherein the first and second modulation functions include pulses having respective characteristic pulse widths, and wherein the controller changes the pulse width of at least one of the pulses.   The first and second modulation functions include pulses having inherent pulse widths that are substantially equal, and the controller varies the pulse widths of both the first modulation function and the second modulation function, and The apparatus of claim 3, wherein the apparatus adjusts the width of the distance window.   5. An image analyzer for identifying a target subject in a scene, wherein the controller changes the at least one modulation function according to a distance of the target subject from the device. The device according to item.   6. A range finder for determining a distance of a target object from the device, wherein the controller changes at least one modulation function according to the distance determined by the range finder. The device according to any one of the above.   7. The controller according to any one of claims 1 to 6, wherein the controller changes the at least one modulation function so that the detector detects only radiated waves reflected from an area of the scene at a target distance from the device. The device according to item.   8. An image processor comprising: an image processor that receives a first image formed by a processor in an area of a scene at a target distance and blends the first image into a second image to form a mixed image. Equipment.   The detector detects radiation waves reflected from the two target objects in the scene, and the controller changes the at least one modulation function so that the portions of the image corresponding to the two objects have substantially the same intensity. 9. A device according to any one of the preceding claims, having a value.   The apparatus according to claim 1, wherein the detector includes a color filter, and the image formed by the processor is a color image. A detector transformer for receiving the radiation wave reflected from the scene and modulating the detector according to the second modulation function;
Sensing including a plurality of sensing elements for sensing a radiation wave modulated by the sensing modulator and generating a signal corresponding to the distance to the scene segment in response to the sensed and modulated radiation wave An array,
The apparatus according to claim 1, comprising:   The apparatus of claim 11, wherein the sensing modulator comprises a shutter element array comprising a plurality of shutter elements. The radiation wave incident on the first sensing element is modulated by the first shutter element, the radiation wave incident on the second sensing element is modulated by the second shutter element, and the second modulation function is the first and First and second modulation subfunctions applied to the second shutter element, respectively,
Each of the segments having a plurality of intervals has a distance limit with respect to the device, the distance limit being determined by the sense modulator;
13. The apparatus of claim 12, wherein at least one of the plurality of spaced segments has a distance limit different from that of at least one other spaced segment.   14. An apparatus according to any one of the preceding claims, wherein a radiation source generates an irradiation line on the scene and the detector receives a corresponding irradiation line of reflected radiation.   The apparatus according to claim 14, comprising a scanning device that scans a scene with the irradiation beam.   16. An apparatus according to claim 14 or 15, wherein the sensed modulator deflects the reflected radiation wave line so as to rest sequentially over a period of time in each of a plurality of element rows of the sensing array.   The one distance limit of a segment having a plurality of intervals, wherein each of the plurality of rows of the array is associated with one of the plurality of spaced segments, and the reflected radiation wave line remains in the associated row. The apparatus of claim 16, wherein the apparatus is determined. A system for generating an enlarged image showing a distance to a subject in a scene,
18. An apparatus for generating an image according to any one of claims 1 to 17, wherein a plurality of consecutive images of a scene are formed, at least one of the plurality of images being at least one of the images. A device including one or more areas of the scene not included in the object,
An image processor that combines a plurality of images to form an enlarged image that indicates the distance to the subject in the scene;
Including system.   The system of claim 18 including means for moving the device laterally with respect to the scene.   The system of claim 19, wherein the device generates an image of the scene with a known relationship to the movement rate of the device with respect to the scene.   21. The system according to claim 19 or 20, wherein the means for moving the device includes a mobile platform on which the device is mounted.   The system of claim 21, wherein the mobile platform includes an airplane. An apparatus for generating an image showing a distance to a subject in a scene according to any one of claims 1 to 17,
Detecting a radiation wave reflected from the scene and not modulated by the second modulation function, and generating a signal according to the intensity of the light reflected from the scene area according to the detected radiation wave; A second detector bored with one detector;
A processor for receiving an image from the first detector and the second detector and forming an image based on the signal having an intensity value distribution indicating a distance of a subject from the device;
Including the device.   24. The apparatus of claim 23, wherein the radiation source is boresighted with the first and second detectors.   25. The apparatus of claim 23 or 24, wherein the signal from the first detector is normalized using the signal from the second detector. A system for guiding a robot, the device for generating an image showing a distance to a subject in a scene according to any one of claims 1 to 17 and 23 to 25;
A robot acting on an object in the scene using the distance to the object indicated by the guidance device;
Including system. A system for determining the size of an object,
An apparatus for generating an image of an object that indicates a distance to a region of the object according to any one of claims 1 to 17, 23 to 25;
A computer using the image and displayed distance to determine the size of the object;
Including system.   28. The system of claim 27, wherein the computer compares the size to a standard one.   29. A system according to claim 27 or 28, wherein the computer uses the determined dimensions to determine the volume of the object. A system for generating a three-dimensional model of an object,
A system for determining the dimensions of an object according to claim 27;
A rapid prototyping apparatus that receives dimensions determined by a system for determining dimensions and generates a three-dimensional model from the dimensions;
Including system.   31. The system of claim 30, wherein the rapid prototyping apparatus receives image information from an apparatus for generating an image of the object and applies the image information to a surface of a three-dimensional model. A vehicle guidance system,
An apparatus for generating an image showing the distance to an object in a scene ahead of the vehicle in the direction of motion according to any one of claims 1 to 17, 23 to 25;
A processor for identifying obstacles in the scene using the image and displayed distance;
Including system. A system for image formation of an endoscope,
An endoscope having a distal end for insertion into a patient's body and a base for coupling an optical device thereto;
26. Any one of claims 1 to 17, 23 to 25, coupled to the base of the endoscope so as to direct a radiated wave through the endoscope into the body and detect a reflected radiant wave therefrom. An apparatus for generating an image according to claim
Including system. A method for generating an image showing a distance to a subject in a scene,
Modulating a radiation wave from a radiation source by a first modulation function to direct the radiation wave to a scene;
Modulating a radiation wave reflected from the scene by a second modulation function, wherein the second modulation function defines a distance window that receives reflections from a position in the scene;
Detecting the reflected radiation, and generating a signal corresponding to a portion of the reflected radiation portion in response to the detected modulated radiation, wherein the portion of the portion is Corresponding to the distance of the point in the scene and the position of the point in the distance window and relating to the specific signal;
Forming an image having an intensity value distribution indicating a distance to a subject in the scene;
Adjusting the width of the distance window by changing at least one of the first modulation function and the second modulation function by a change amount corresponding to the intensity value distribution of the image;
Including methods.   At least one of the first modulation function and the second modulation function is changed by an amount of change corresponding to the distance indicated by the intensity value distribution of the image formed by the processor, and the timing and distance of the distance window 35. The method of claim 34, comprising adjusting.   36. A method according to claim 34 or 35, wherein the first and second modulation functions comprise pulses and changing at least one modulation function comprises changing a pulse width of at least one pulse.   37. The method of claim 36, wherein the pulse width of the first modulation function pulse is substantially greater than the pulse width of the second modulation function pulse. Identifying at least one object of interest in the scene,
Using the intensity value distribution of the image to determine the distance to the object;
38. The method of any one of claims 34 to 37, comprising:   40. The method of claim 38, wherein changing at least one modulation function includes changing the modulation function in response to a distance to at least one subject. Changing the modulation function according to the distance to the at least one subject defines a region of distance including the distance to the at least one subject;
Adjusting the modulation function such that detecting reflected radiation includes detecting radiation from one or more regions of the scene within the distance region;
40. The method of claim 39, comprising:   Identifying at least one subject of interest includes positioning a plurality of points on at least one subject and defining a region of distance includes defining a plurality of local regions 41. The method of claim 40, wherein each local region includes one of each of a plurality of points.   42. The adjustment according to claim 40 or 41, wherein adjusting the modulation function comprises adjusting the function such that radiated waves from a subject and a scene area outside a defined distance area are not detected. Method.   Adjusting the modulation function includes adjusting the function such that radiated waves from a subject and scene region outside a defined distance region have a substantially constant intensity value distribution. 42. A method according to claim 40 or 41.   Forming a first two-dimensional image, wherein the image includes radiation waves detected from one or more areas of the scene within the distance range, and to form a mixed image 44. A method according to any one of claims 34 to 43, comprising mixing the first two-dimensional image with a second image obtained separately or generated by synthesis.   Determining the position of a desired region in the first two-dimensional image, and mixing the first two-dimensional image with a second image obtained separately to place the second image in the desired region 44. The method of claim 43, comprising inserting. Detecting the radiated wave at each substantially different distance includes detecting the radiated wave from the target subject in the scene; and
46. A method according to any one of claims 34 to 45, wherein changing at least one modulation function comprises changing the modulation function such that the two subjects have approximately equal intensity values. Defining a segment having a plurality of parallel intervals in the scene, each having a distance limit;
Detecting radiation reflected from each of the segments having the parallel spacing;
In response to the detected radiation, generating a signal according to the distance to the scene area,
and,
Forming an image having an intensity value distribution indicating a distance to a subject in a scene;
Including
Defining the segments having the plurality of intervals includes defining a distance limit for each of the plurality of segments; and
47. A method according to any one of claims 34 to 46, wherein at least one distance limit of the plurality of spaced segments is different from a distance limit of at least one other spaced segment.   48. A method according to any one of claims 34 to 47, wherein forming an image having an intensity value distribution indicative of a distance to a subject includes normalizing the intensity value distribution.   49. The method of claim 48, wherein normalizing the intensity value distribution includes modifying the distribution to change the emissivity of the subject.   50. A method according to claim 48 or 49, wherein normalizing the intensity value distribution comprises modifying the distribution for different subject distances.   51. A method according to any one of claims 34 to 50, wherein detecting radiation comprises filtering radiant waves according to color and forming an image comprises forming a color image. A method for generating a mixed image including a target subject superimposed on a separately obtained background image,
Forming a first image showing a distance of a scene including a target subject to a subject in the scene according to any one of claims 34 to 51;
Determining the distance to the subject indicated by the first image;
Defining an area of distance including the distance to at least one subject;
Removing from the first image a subject outside the distance region to form a filtered image;
Superimposing the filtered image on the background image;
Including methods. A method for generating a blended image of a scene that includes a desired inset image inserted into a selected region in a blended image, comprising:
Specify an area in the above scene,
Generating a first image indicating a distance of the scene including the designated area to a subject in the scene according to any one of claims 34 to 51;
Determining the position of the boundary of the region in the first image and the distance to the region, as indicated by the first image;
Measuring the inset image to fit within the boundary of the region by the distance to the region;
Superimposing the measured inset image on the region in the first image to form a mixed image;
Including methods. Identifying a subject to be inserted into the first image, the distance to the target subject being shorter than the distance to the region,
Filtering the first image to form a filtered image of the target subject;
54. The method of claim 53, comprising superimposing a filtered image of the subject of interest on an inset image in the region.

Images