JP2013521710A - System and method for spatially adjusted scene lighting - Google Patents

Abstract

A scene lighting system is provided that creates spatially uniform or controlled brightness levels for machine vision applications. The system includes a camera, a plurality of light sources that selectively illuminate different regions within the camera's field of view, and a processing unit connected to the camera and the light sources. The focal region of the light source in the camera's field of view is sampled to determine the average region luminance and compared to the target luminance level. The processing unit controls the light source to increase or decrease the illumination level to bring it closer to the target luminance level in the field of view. This light source modulation can be repeated for successive video images until the target luminance level is reached. Once the goal is reached, for some applications, the iterative feedback control is fixed, but for other applications, the iterative process continues periodically or continuously to incorporate different scene or lighting state changes.
[Selection] Figure 1

Description

Description of Federally Assisted Research and Development The US government has a lump sum license for the present invention, with restrictions, contracts for the Ministry of Health and Social Welfare, the Public Health Service, the Center for Disease Control (CDC), and the Department of Defense (US Army) The request is granted by number W81XWH-05-C-0045, United States Department of Defense United States Congressional Research Unit proposal number W81XWH-06-2-0037, and United States Department of Transportation United States Congressional Research Unit approval decision number DTNH22-05-H-01424 As defined by the conditions of grant number 1 R43 CE 00151-01, the patent owner has the right to request that another person be licensed under reasonable conditions.

TECHNICAL FIELD The present invention relates to an apparatus and method for adjusting the illumination of an object, and in particular by adjusting a plurality of light sources for machine vision, for example, in a video camera image of an object such as a human eye or face. And an apparatus and method for producing spatially adjusted brightness, such as substantially uniform brightness or variable brightness.

  The invention particularly relates to illumination schemes for machine vision applications where the reflectivity of surfaces and objects varies within the field of view of the camera. Conventional lighting schemes often create shadows and / or flashes (eg, hot spots) that appear on a two-dimensional single plane in a scene that includes curved surfaces or objects. These problems are exacerbated when the illumination is limited to a limited number of light sources and / or when the light sources must be placed near the scene being illuminated.

  Applications that include machine vision are becoming increasingly commonplace. Some have arisen as a result of technological advances in the electronics and software development industries, reducing the cost of cameras and information processing units. A few examples within the scope of machine vision applications include: object recognition, distance measurement, food inspection, quality control in the assembly line, barcode reading, object counting, safety monitoring, biometrics. Sectors and industries that use machine vision include the military industry, medical care, security, semiconductor processing, manufacturing, robotics, and the toy industry.

  Almost all image processing techniques and algorithms are affected by improper illumination of areas in the image. If the illumination level is too low, as a result, the luminance change for identifying the boundary of the object is insufficient, or the reflectance changes locally. Reduced signal intensity or light intensity also causes dominance of detector noise in the image. A low signal-to-noise ratio generally makes the image grain rough and difficult to process.

  Conversely, if the illumination level is too high, the camera or detector pixels will be saturated. Also, fully saturated pixels do not provide information about changes in the brightness level of the image processing algorithm that distinguishes edges or boundaries. In some types of video cameras, saturated pixels bleed over, increasing the apparent brightness of nearby pixels.

  In most cases, information content is lost in image areas with too few or too many photons. The amount of image processing cannot recover lost information. In these cases, the illumination of the entire spatial area of the video image must be improved to produce a reliable machine vision application.

  For example, in order to accurately measure the object size, the edge of the object is detected so that the edge of the object is recognized as a region having a sharp inclination in color or brightness. When the field of view of the camera at the edge of an object is distorted by shadows, the reliability and accuracy of the edge detection algorithm deteriorates.

  Machine vision applications involving object recognition are particularly sensitive to lighting conditions. Dark corners, color changes due to lighting, brightness changes resulting from various angles of surface lighting, shadows, and hot spots can render unrecognizable objects due to lighting conditions.

  Controlled illumination of objects is difficult, especially when the light source is sealed and close to the illuminated object. This confinement may be due, for example, to the desire to reduce power consumption by miniaturizing the device and / or illuminating only the camera's field of view. Such a case is a case of illuminating an eye using glasses or an apparatus attached to a head-mounted device. Examples of these types of systems or devices are William C. et al. See US Pat. No. 7,515,054B2, which discloses a biosensor, communications, and controller application granted to Torch that is facilitated by monitoring eye movements.

  The present invention relates to an apparatus and method for controlling illumination of an object. In particular, the present invention provides a device that produces a substantially uniform brightness in a video camera image, eg, directed to a human eye or face, by adjusting spatially controlled brightness, eg, multiple light sources for machine vision. And a method.

  In view of the above background, the devices, systems and methods described herein provide improved illumination methods and systems for machine vision applications. This method typically comprises two or more sources of electromagnetic radiation that illuminate various areas of the scene at angles different from the viewing angle of the camera. Each of these illumination sources comprises one or more illumination devices, such as incandescent bulbs, arc lamps, or light emitting diodes (LEDs). The camera may be a component of a frame capturer and a processing system that generates feedback signals for individually controlling the illumination source based on the captured image. The target brightness level may be constant, and is a function of space (eg, grading the brightness across the scene) and / or time (eg, the scene is illuminated alternately by illumination sources at different wavelengths). Case). Controlled and / or uniform illumination allows machine vision applications, including position measurement and object identification, to be performed more easily, quickly and reliably.

  According to one embodiment, there is provided a system and method for generating camera images with spatially substantially uniform and / or controlled scene brightness, particularly when the reflectance in the scene varies spatially. ing.

  According to another embodiment, a system and method is provided for generating an image having a controlled or substantially uniform brightness from a curved and / or non-coplanar image plane of the camera.

  For example, the system and method use one or more illumination sources that are located towards the line-of-sight of the camera. Illumination from an acute angle can reveal fine structures such as surface cracks or indentations.

  According to yet another embodiment, a system and method are provided for reducing the effects of shadows caused by three-dimensional structures occurring on a surface. For example, the effect of this shadow can be reduced by using multiple electromagnetic radiation sources that illuminate the scene from a contrasting angle.

  According to yet another embodiment, systems and methods are provided that reduce or prevent the effects of so-called “glints” or bright spots resulting from a point source of illumination. The effect of flashing can be prevented by using a source at an angle away from the camera's viewpoint and directing light into the scene.

  According to another embodiment, a device configured to be worn on a person's head; a camera positioned on the device and positioned to view a first eye of the person wearing the device A system for monitoring the human eye comprising: a plurality of light sources on a device that selectively illuminates each focal region of the human face around the first eye and within the field of view of the camera; Yes. In an alternative embodiment, the camera and / or light source is mounted remotely from a person, for example, in a vehicle dashboard or other interior structure.

  The controller can be connected to a camera and a light source. The controller is configured to sample the luminance at each focal region of the light source using a camera and modulate the light source based on the sampled luminance to provide a desired luminance level within each focal region. For example, the controller is configured to sample luminance from a plurality of pixels of the camera corresponding to a first focal region in the field of view of the camera illuminated by the first light source, the controller being sampled. The combined luminance is determined to determine the average luminance provided by the first light source and to modulate the first light source to provide a desired luminance level within the first focal region. Similarly, the controller samples the luminance from the second or additional focal region illuminated by the second or additional light source and provides a desired luminance level within the corresponding focal region. It is configured to modulate the light source.

  In one embodiment, the controller is configured to perform at least one amplitude modulation of current and voltage to the light source to provide a desired brightness level within each focal region. Additionally or alternatively, the controller is configured to perform at least one pulse width modulation of current and voltage to the light source to provide a desired brightness level within each focal region.

  Optionally, a processing unit may be connected to the camera to receive the first eye image, for example to monitor and / or analyze the first eye image. The processing unit may comprise a controller or may be one or more other processors.

  According to yet another embodiment, a feedback control system for providing spatially controlled illumination is provided, the system comprising a camera for measuring scene brightness in two or more spatial regions of the field of view. A light source that selectively illuminates a corresponding area in the camera's field of view and an average brightness in each area of the camera's field of view, and the corresponding light source is modulated into one or more brightness levels into the field of view. And a processor that provides a desired brightness level.

  In one example, the system includes an electronic display, and the camera and / or light source is mounted relative to the display such that the camera and light source face the face of the person viewing the electronic display. In another example, the camera and / or light source is mounted on a vehicle dashboard or other structure such that the camera and / or light source is directed toward the face of the vehicle operator.

  In yet another example, a camera and / or light source is mounted on a structure near the assembly line or conveyor belt to obtain an image of an object facing along the assembly line or conveyor belt. In still another example, a camera and / or a light source are mounted on a structure near a plurality of storage areas so as to obtain an image of an object located in the storage area. For example, a light source can be mounted near an assembly line, conveyor belt, or storage area to selectively illuminate each focal region within the camera's field of view, for example, to facilitate object identification based on images acquired by the camera Can be.

  In any of these examples, the light source is operating substantially continuously or intermittently. For example, the light source is deactivated when the camera is not activated. For example, if a camera is used to sample the luminance in the area selectively illuminated by each light source to individually obtain an image of the field of view, the light source acquires an image in which the camera is activated and / or Alternatively, the light source is activated only while sampling the luminance. For example, when the luminance of the light source is controlled using pulse width modulation, the light source may be activated intermittently during this period.

  According to yet another embodiment, a method for controlling illumination of a first human eye is provided. The camera is positioned so that the person's first eye is within the field of view of the camera toward the person's face, and each focal region of the person's face that is within the field of view of the camera around the first eye A plurality of light sources that selectively illuminate at least a part of the human face. The brightness is sampled using a camera at each focal region of the light source and the light source is modulated based on at least partially sampled luminance to provide a desired luminance level within each focal region.

  According to yet another embodiment, a scene lighting system is provided that generates spatially uniform or controlled brightness levels for machine vision applications. For example, the target brightness level is substantially the same across the entire field of view of the camera, eg, a substantially uniform brightness occurs, or varies depending on the position function, for example, to create a controlled brightness level in the scene. In addition or alternatively, the target luminance level may change with time.

  In an exemplary embodiment, the system includes a camera, a plurality of light sources that selectively illuminate different regions within the camera's field of view, and a processing unit connected to the cameras and light sources. The focal region of the light source in the camera field of view is sampled and the average local luminance is measured and compared to the target luminance level. The processing unit controls the light source to raise or lower the illumination level and converge toward the target luminance level in the field of view. This modulation of the light source is repeated for each successful video image or for periodic images until the target luminance level is achieved. Once the target brightness level is achieved, iterative feedback control is fixed for some machine vision applications. For other applications, the iterative process continues periodically or continuously to configure different scene or lighting condition changes.

  According to yet another embodiment, a method is provided for controlling illumination of a scene, the method directing the camera towards the scene such that one or more objects in the scene are within the field of view of the camera. Illuminating the scene with a plurality of light sources that selectively illuminate each focal region within the field of view of the camera; sampling the luminance within each focal region of the light source using the camera; Modulating the light source based at least in part to provide a desired brightness level within each focal region.

  For example, the scene includes at least a portion of a person's face, and the one or more objects include at least one eye of the person. The camera and / or light source can be worn on a device on or off the person's head.

  In another example, the scene comprises an electronic display, and the camera and / or light source is positioned relative to the display such that the camera and light source face the face of the person viewing the electronic display. In this example, the light source can be modulated to provide a desired brightness level for each focal region of the human face within the camera's field of view.

  In yet another example, a camera and / or light source is mounted on the vehicle dashboard such that the camera is directed toward the operator's face of the vehicle. In this example, the scene includes at least a portion of the operator's face and the light source is directed toward the operator's face to selectively illuminate each focal region of the operator's face within the camera's field of view.

  In yet another example, a camera and / or light source is directed toward an assembly line or conveyor belt and acquires an image of an object that is directed along the assembly line or conveyor belt. In yet another example, a camera and / or light source is directed toward a plurality of storage areas, and an image of an object placed in the storage area is acquired. The light source is directed to the assembly line, conveyor belt, or storage area and selectively illuminates each focal area within the camera's field of view, for example, a substantially uniform brightness level within the field of view, or a brightness that varies within the field of view. Provides a level. For example, the light source is modulated to provide a higher brightness level for focal regions that are further from the camera than other focal regions.

  Other aspects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings.

Illustrative embodiments of the invention are shown in the drawings.
FIG. 1 is a perspective view showing spatially uniform illumination around an eye and a face area using three light emitting diodes by controlling a processing unit that acquires an image from a camera. FIG. 2 is a diagram illustrating an example of illumination patterns generated by three individual light sources and around the eyes and face area. FIG. 3 is an example of a camera image around the eye and face area, and three pixel clusters are sampled to measure the average illuminance in the three areas within the camera field of view. FIG. 4 is an example of controlled lighting of a grocery or warehouse shelf where a spatial gradient is required in the lighting pattern. The illumination gradually increases in brightness toward the right side of the image and is divided into four horizontal regions. FIG. 5 is an example of a time sequence of feedback and signals used to control illumination using amplitude modulation and pulse width modulation techniques. FIG. 6 is a flowchart illustrating an exemplary algorithm that can be used to control the illumination levels of multiple light sources to produce spatially uniform brightness. FIG. 7 is a perspective view showing still another embodiment of an apparatus for monitoring a person based on the movement of the person's eyes.

  Referring to FIG. 1, an exemplary embodiment of a system 10 is shown that provides feedback controlled illumination of a human eye and / or a facial region near the eye. In general, system 10 includes a plurality of light sources 12 (three light sources are shown), one or more cameras 14 (one camera is shown), light sources 12 and / or cameras. 14 includes a processing unit 16 connected to 14.

  The components of the system 10 include a single device or two or more individual devices. For example, as shown in FIG. 7, the camera 14 and the light source 12 are provided on a frame 18 or other device configured to be worn on a person's head. In the illustrated embodiment, the frame 18 has a pair of bridge pieces 18a, a rim 18b extending above or around each eye and defining an opening 18c, and / or a pair of glasses, for example glasses. An ear support 18d is provided. Optionally, the aperture 18c may or may not be provided with a desired lens, such as a prescription lens, a shaded lens, a polarizing lens, and / or a protective lens, but such a lens may be This operation is not necessary. Alternatively, the components of system 10 are disclosed in helmets, masks, goggles, pull-down masks, and US Pat. Nos. 6,163,281, 6,542,081, or 7,488,294. It may be provided in other devices configured to be worn on a human head, such as other devices (not shown).

  In a further alternative of the present application, as will be further described below, the components are provided in individual devices, such as stationery or moving structures, to monitor people, objects, and / or other scenes. It is supposed to be. For example, the camera 14 and / or the light source 12 may be provided away from a person and monitored. In one exemplary embodiment, the camera 14 and / or light source 12 may be in a dashboard, cockpit, or other interior area of the vehicle, in a vehicle driver, pilot, or other operator, passenger, or vehicle. Can be worn in the direction of other people. The position of the camera 14 and / or the light source 12 is substantially within the vehicle, for example, towards the face such that the camera 14 and / or the light source 12 is, for example, one eye or both eyes of an operator or other person. It may be fixed or adjustable. In another exemplary embodiment, camera 14 and / or light source 12 may be mounted on or adjacent to a computer display or other electronic device, for example, to monitor one or both eyes of a user of the electronic device. Thus, the user may be able to control or operate the electronic device based on at least part of the movement of one eye or both eyes.

  Alternatively, referring to FIGS. 1 and 7, processing unit 16 may also be performed by frame 18 or separately from frame 18 and / or frame 18 (and as described herein). It can also be performed remotely from other components of the system 10 if structures other than the frame 18 are provided. For example, as shown in FIG. 7, the processing unit 16 may be provided in a separate case from the frame 18 and may include one or more cables 17 extending from the frame 18 (one cable for clarity). (Only the cable is shown). Cable 17 may comprise a separate cable or light source 12, camera 14, and / or other components on frame 18, and a wire set connected to processing unit 16. Individual cables or wire sets are captured in cable 15 from each light source 12, camera 14, 15, etc., for example, within frame 18, eg, along rim 18b, for example, to capture the entire profile of frame 18. It may be embedded until it is as small as desired.

  Still referring to FIG. 1, a camera 14 is mounted and / or positioned on a frame 18 that is mounted with a frame 18 or otherwise monitored by the system 10 as shown. And has a field of view directed at the first eye 20 of the person. For example, as shown in FIG. 7, the camera 14 is offset from each opening 18c of the frame 18 and is described, for example, in pending patent application Ser. No. 12 / 551,547 filed Jan. 13, 2010. As is done, the camera 14 is placed away from the overall field of view of the person wearing the frame 18.

  In an exemplary embodiment, the camera 14 includes a detector that includes a CCD or CMOS or other active area including, for example, a rectangular or other pixel array, and the camera 14 captures an image and video representing this image. A signal is generated. The active area of the camera 14 has, for example, a square, a rectangle, a circle, an ellipse, and other desired shapes. The active area of the camera 14 is substantially flat or curved, for example in a curved surface facing the direction of the eye 22. Exemplary CMOS devices that can be used include Omnivision model number OV7740 or Mocron model number MT9V032. In addition, the camera 14 may include one or more filters, lenses, etc. (not shown), if desired, such as unwanted light intensity and / or focusing the image on the active area. For example, the wavelength can be filtered.

  Optionally, a second camera may be provided that has a field of view directed to the second eye (not shown) of the person being monitored by this system 10. For example, as shown in FIG. 7, a pair of cameras 14 are mounted on a frame 18, for example, on the lower region of the rim 18b under each eye, to minimize interference with human vision and Eyes can be monitored. In addition or alternatively, multiple cameras (or multiple cameras for each eye, not shown) for each eye of the person are provided to provide, for example, distant or overlapping fields of view You may make it do. Another option is to provide one or more cameras 15 on the frame 18 facing away from the person wearing the frame 18, as shown in broken lines in FIG. As disclosed in the described patent, an image around a person may be acquired.

  The light source 12 can be mounted at a plurality of locations on the frame 18, for example, around the opening 18 c near the camera 14. For example, as shown in the figure, three light sources 12a, 12b and 12c are provided. For example, the first and second light sources 12a and 12b are on the upper region of the rim 18b, and the third light source 12c is a rim. It may be provided on the lower region of 18b. Only two or more light sources (not shown) may be provided and controlled using the individually described systems and methods if necessary. As shown in FIG. 7, if the system 10 has a second camera 14, a second eye and / or face area (for a person wearing an additional light source set 12 on the frame 18 and wearing the frame 18 ( (Not shown) may be illuminated. Further, if the system 10 has multiple cameras facing a single eye (not shown), these cameras may share multiple light sources and, alternatively, multiple light source sets. May be provided to illuminate each field of view (not shown) of the camera.

  In an exemplary embodiment, each light source 12 emits light of a relatively narrow or wide wavelength, such as one or more wavelengths of infrared light, for example, about 640 to 700 nanometers, high band visible light such as white light, etc. You may provide the comprised light emitting diode. Optionally, the light source 12 may comprise a lens, filter, diffuser, or other structure (not shown) that facilitates, for example, illuminating each focal region of the human eye and / or face. good. The light sources 12 are spaced apart from one another in, for example, one or more arrays disposed around each opening 18c of the frame 18, for example, a substantially uniform or variable level of the human eye and / or face. A desired brightness level, such as the brightness of the human eye, may be provided to provide an image of a human eye and / or face captured using the camera 14.

  The processing unit 16 includes one or more controllers or processors that operate various components of the system 10, such as, for example, one or more hardware components and / or software modules. For example, the processing unit 16 comprises a separate or integrated controller (not shown) that controls the light source 12 and / or the camera 14 and receives and / or processes signals from the camera 14 and others. Optionally, one or more components of the processing unit 16 may be on the frame 18, eg, on the ear support 18 d or rim 18 b, similar to the embodiments described in the references described herein. It is carried on.

  The processing unit 16 may comprise a camera 14, 15, a filter for editing and / or processing image signals, and a memory for storing image signals from others. In addition, the processing unit 16 includes one or more power supplies (not shown) that operate, for example, components of the system 10. Optionally, frame 18 and / or processing unit 16 may comprise one or more transmitters and / or receivers (not shown) for transmitting data, receiving instructions, etc. good. Additionally or alternatively, system 10 may include components that are remote from frame 18 and / or processing unit 16, similar to the embodiments described in the references described herein. good. For example, the system 10 may include one or more receivers, processors, and / or, for example, in the same room, near the monitor station, or remote from the processing unit 16 and / or frame 18 at a more remote location. A display (not shown) may be provided.

  Referring to FIG. 1, for example, a system 10 for performing automatic pupil tracking, where spatially nearly uniform illumination is beneficial, shows an eye 20 of a person taking an image and a surrounding face region 22. Yes. The face area 22 including the eye 20 is visualized by the camera 14, where an image is sent to the processing unit 16. The pupil 26 is located at the center of the iris 27. Therefore, the iris 27 is located in the sclera 28 or the white eye region. During the normal viewing, both the voluntary muscle and the involuntary muscle are controlled, so that the position of the pupil 26 changes. While the pupil 26 is moving, the regions associated with the pupil 26, iris 27, and sclera 28 change within the camera's field of view. In part, the change in size and shape is due to the radius of curvature of the eye 20.

  When illuminating with one or more light sources, multiple factors may affect the light intensity detected from different regions within the camera's field of view. These factors are: a) the distance between the light source and the field of view, b) the intensity of each light source, c) the divergence angle of each light source, d) the reflectance of the illuminated surface (at the illumination wavelength), e) the curvature of that surface. And f) the efficiency of the camera when converting light into a signal, and the optical efficiency and spatial uniformity associated with this camera. In addition, a three-dimensional structure may create a shadow when illuminated from a single light source or a few light sources. In the case of the region 22 around the eye 20, shadows may be formed by structures including wrinkles, eyelashes, and skin wrinkles.

  As described further below, the brightness levels of various regions within the camera field of view, such as the camera 14 shown in FIG. 1, can be used in feedback mode to control the intensity of different illumination sources. Illuminating a scene from different angles using multiple lights can reduce the deleterious effects of shadows. Flash can be reduced or prevented by using multiple illumination sources and / or by placing the multiple illumination sources at strategic locations, eg, sufficiently away from the viewing angle of the camera. In the example shown in FIG. 1, infrared light emitting diodes (LEDs) are used as illumination sources. As shown in the figure, LED 12a selectively illuminates the upper left region of the field of view of camera 14, LED 12b selectively illuminates the upper right region of the field of view of camera 14, and LED 12c selectively illuminates the field of view of camera 14. Illuminate the lower area of

  FIG. 2 is a diagram showing each focal region illuminated by different light sources. In this example, three light sources illuminate the pupil 26 and the face region 22 around the eye 20. As shown in FIG. 2, it is understood that the light source generally does not create a sharp area. Rather, the dashed line represents a “focus area”, ie, an area where a particular light source has a selective or enhanced effect on illumination of an area relative to the surrounding area in the camera's field of view. Referring to FIG. 1, LED 12a selectively illuminates region 121 of FIG. 2, which is located in the upper left region of the camera's field of view. Similarly, LED 12b in FIG. 1 illuminates region 122 in FIG. 2 located in the upper right region of the camera's field of view. The LED 12c in FIG. 1 illuminates the region 123 in FIG. 2 located in the lower region of the camera's field of view. The size of the illuminated area or focal area depends on the divergence angle of the light source and / or the distance between the light source and the illumination surface. The approximate shape of the illuminated area depends on the light profile of the light source and the angle between the position of the light source and the vector perpendicular to the illumination surface. The average brightness of each focal region can be controlled by adjusting the intensity of the associated light source.

  FIG. 3 shows an example of a pixel cluster, which can be sampled to determine the average brightness in the region near the pupil 26. In this example, the brightness of the upper left region in the camera field of view is evaluated using the measured average intensity of the 8 × 8 element cluster of the pixel 101. An 8 × 8 element cluster of another pixel 102 is used to calculate the brightness of the upper right region in the camera's field of view. The 8 × 32 element cluster of the third pixel 103 is used to evaluate the brightness of the lower region in the camera field of view.

  However, it is obvious that the luminance can be evaluated from an arbitrary number of viewing areas. The luminance assessment can be determined from any number of pixels in a cluster of any size, space, shape within each focal region. For example, the same pixel can be sampled during each sampling within each focal region, and different pixels selected at random, for example within each focal region, can be sampled if desired. Luminance may be sampled using the actual video signal while the eye 20 is being monitored, or may be sampled outside the data stream used to monitor the eye 20. For example, periodic frames from a series of video signals may be sampled in addition to being recorded, processed, or monitored to predict and / or adjust brightness levels using the systems and methods described herein. Can do.

  For example, with further reference to FIGS. 1-3, processing unit 16 periodically samples predetermined pixels 101, 102, 103 (shown in FIG. 3) in focal regions 121, 122, 123 (shown in FIG. 2). Thus, the average luminance level in any one of the focal regions 121, 122, and 123 can be evaluated. The processing unit 16 modulates the light sources 12a, 12b, 12c (shown in FIG. 1) to provide a desired brightness level within each focal region 121, 122, 123. For example, it is preferable that the luminance is substantially uniform while the eye 14 is illuminated and imaged by the camera 14. The processing unit 16 samples the set of pixels 101, 102, 103 to determine the average brightness of each focal region 121, 122, 123 and increases or decreases the intensity of the light sources 12a, 12b, 12c, for example, the average brightness Can be maintained substantially uniformly and / or within a desired range.

  Alternatively, one or more cameras can be used to sample the luminance in the camera's field of view to provide multiple cameras to monitor a human eye (not shown). For example, in one embodiment, while using a first camera dedicated to luminance sampling, for example, as described above, an eye image is acquired using a second camera, for example, in this specification or book. Used for other purposes, such as monitoring a person's awareness, physical and / or mental state, controlling one or more devices, as described in the cited references in the specification be able to. In another embodiment, multiple cameras are used to sample the luminance in the focal region of each light source and average or compare the sampled luminance from multiple cameras to provide feedback control to the light source can do.

  In other applications, it is desirable to provide illumination for scenes that are spatially variable. FIG. 4 shows an example of controlling illumination in situations where a (non-uniform) spatial gradient of brightness 242 is desired. In this example, the camera 241 is used to image objects on a series of shelves or transport system 240 as commonly found in, for example, grocery stores, large wholesale stores, and manufacturing plants. The camera 241 and the light sources 231 to 234 can be individually mounted on, for example, one anchor remote from the shelf or the transport system 240 and can be substantially stationary or moved.

  In such a case, some objects and characters on the objects appear smaller than others in the image acquired using the camera 241. This is probably due to the fact that the size of the object has actually decreased, or because the object is far from the camera. Often, the performance of image processing algorithms is improved by increasing (without saturating) the brightness of objects that appear smaller in the camera's field of view. Spatially controlled illumination helps image processing algorithms compensate for spatial variations in the resolution of lenses and other optical components by increasing the signal-to-noise ratio. Most lenses, especially small lenses, have a lower spatial resolution and / or a lower light collection capability nearer the outer edge compared to the central area of the lens.

  In the example shown in FIG. 4, the visual field is divided into four vertical regions, so that the luminance has a horizontal gradient. The leftmost region 221 is illuminated by the LED 231 and requires the minimum brightness to perform reliable image processing. The second area 222 from the left is illuminated by the LED 232 and requires the next lowest brightness after the minimum brightness to perform reliable image processing. Region 223 is illuminated by LED 233 and requires greater brightness due to the presence of smaller objects. Finally, region 224 is illuminated by LED 234 and requires the greatest illumination since there are small objects. This object is farthest from the camera and / or has a low optical resolution near the edge of the image acquired using the camera 241.

  A processing unit (not shown) can be connected to the camera 241 to sample the luminance levels in the vertical regions 221 to 224 and / or the light sources 231 to 231 as described herein. 234 to modulate the intensity of the light sources 231 to 234 according to the sampled luminance level. For example, as described above, the processing unit can modulate the light sources 231 to 234 to increase the luminance level of each region 221 to 224 and easily monitor the image from the camera 241.

  FIG. 5 is an example of a time sequence of signals used to monitor and control illumination in various regions of the camera's field of view. The solid line 141 represents the measured average brightness of individual pixel clusters, and the dashed line 140 represents the target light intensity for the focal region of the light source. Dot 142 represents the time to collect a new camera image and sample the luminance. If the measured average brightness is below the target intensity, another scheme can be used to increase the intensity of the light source associated with the corresponding region. Conversely, if the measured average light intensity is above the target intensity, the same scheme can be used to reduce the brightness of the light source associated with the corresponding region.

  In the exemplary embodiment, solid line 143 shows a scheme for controlling the light intensity using the voltage or current amplitude dividing the light source. This is commonly referred to as “amplitude modulation”. In another embodiment, solid line 144 illustrates a scheme for controlling the light intensity by varying the duration or “dwell time” of the control voltage or current. This is commonly referred to as “pulse width modulation”. Optionally, both schemes can be used simultaneously.

  If desired, save energy by turning off the lights when not needed, such as when the camera is not converting the light into a usable signal, or when the entire device is not in use, and / or For example, the overall illumination intensity can be reduced for safety reasons. For example, the light sources 231-234 shown in FIG. 4 (or in other embodiments described herein) can be operated intermittently, for example, switched off when the camera 241 is not operating. . In one exemplary embodiment, the camera 241 is operated periodically to obtain an image of its field of view as well as sample brightness from, for example, acquired images. In this example, a camera 241 having a controlled light source brightness as described herein acquires an image using, for example, amplitude modulation and / or pulse width modulation while the camera is operating. The light sources 231 to 234 are switched on only during a certain period. In another exemplary embodiment, the camera 241 can be operated periodically to acquire an image of the field of view and sample the brightness separately. In this example, for example, by using amplitude modulation and / or pulse width modulation during the operation period, the light sources 231 to 234 are intermittently operated only during the period in which the camera 241 operates to acquire an image and / or samples the luminance. I am doing so.

  FIG. 6 is a flowchart of an exemplary algorithm that can be used to provide feedback control, eg, to produce controlled or uniform illumination, such as using the system 10 of FIGS. 1 and 7 and the system of FIG. The average brightness of each region (denoted “n”) is measured sequentially within the video image. For example, in step 310, new images can be collected from cameras illuminated with multiple light sources.

  In step 320, the average brightness of the first focal region corresponding to the first light source can be sampled by sampling a plurality of pixels from a new image in the first region. For example, the actual luminance level of a plurality of pixels in the first region can be acquired and averaged to determine the average luminance level of the first region. In step 330, this average luminance can be compared to the target intensity for the first region. If the average brightness is higher than the target intensity (branch 330a), in step 332, the output of the first light source is reduced. If the average brightness is lower than the target intensity (branch 330b), in step 334, the output of the first light source is increased. Thus, in step 336, the first light source is modulated to the desired intensity. In step 338, the number “n” is incremented and in step 340 it is confirmed that there is another focal region that has not yet been sampled. If “n” is less than or equal to the total number of focal regions and light sources, this process, ie, steps 320 through 336, is repeated for the second focal region and light sources, etc. If all focal regions have been sampled from a new image and a modulated light source, the procedure is repeated with the new image (starting again from step 310). Thus, with each sampling, the output to the entire array of light sources included in the system can be modulated to a desired level. This process is repeated periodically as new video images are collected, or every other image taken using, for example, a camera that provides sampled brightness, or every 10 images.

  The target luminance level may be the same for the entire region through the field of view of the camera, such as the system 10 shown in FIGS. 1 and 7, for example, and a scene having substantially uniform luminance may be created. Alternatively, the target brightness may also be selected to vary different spatial regions in the camera's field of view to partially compensate for the effects of reduced object size in the image, as in the system shown in FIG. Anyway. Optionally, increase the target brightness in one or more regions to create a spatially gradient pattern, for example to enhance analysis, for example, to crack detailed cracks on the surface or subtle color changes. Can manifest.

  Also, the target brightness level can be selected and changed as a function of time. When using a set of illumination sources that emit at different wavelengths, different target luminance levels will be required as each wavelength is selected to reveal different structures in the camera image.

  In some applications, brightness is dominated by electromagnetic sources that are part of the applied lighting system. In another application, light from a feedback-controlled lighting system can be superimposed on an ambient light source such as the sun or room light. In the latter case, when the ambient light level changes, it is necessary to convert it back to the desired luminance level.

  Another example of an application using dynamic lighting with feedback control includes a safety checkpoint that recognizes a vehicle or a person when approaching the camera's field of view. In this case, either a visible or infrared illumination source is dynamically modulated to prevent shadows and / or illuminate the object for uniform scene brightness while preventing hot spots in the image. .

  Another example of an application using dynamic lighting with feedback control is an assembly line or conveyor belt system such as those used to separate products or other foods such as apples and fish (not shown) It is a separation process within. As an object with various reflective properties, sizes, shapes, and / or orientations enters the camera's field of view, the illumination is dynamically adjusted to better measure dimensions and / or recognize. In this example, a camera (or multiple cameras as described herein) and multiple light sources are mounted on a stationary support near the assembly line or conveyor belt to produce the assembly line or conveyor belt. Directed to the object to be transported (not shown). Alternatively, a plurality of stationary cameras may be provided, for example, stationary to each camera, or mounted on an assembly line or conveyor belt so that the light source moves with objects on the assembly line or conveyor belt A plurality of light source sets may be provided.

  Another example of an application using dynamic lighting with feedback control includes an alignment system, such as used to accurately align a tire and a drive shaft. Another example of an application using dynamic lighting with feedback control is in the field of face recognition.

  The above disclosure regarding exemplary embodiments has been presented for purposes of illustration and description. It is not intended to exclude or limit the invention to the precise form disclosed. Many variations and modifications of the embodiments described herein will be apparent to those skilled in the art in light of the above disclosure.

  Further, in describing representative embodiments, the specification provides methods and / or processes as specific sequence steps. However, the scope of the method or process does not depend on the specific order of steps described herein, and the method or process is not limited to a specific sequence of steps. Those skilled in the art will appreciate that other sequence steps are possible. Accordingly, the specific order of the steps described herein is not a limitation on the scope of the claims.

  While the invention is susceptible to various modifications, it should be understood that the invention is not limited to the specific forms and methods disclosed herein, and that this modified form and specific examples are shown in the drawings. Have been described in detail. However, the invention is not limited to the specific forms or methods disclosed herein, but rather covers all modifications, equivalents, and variations that are within the scope of the claims.

Claims (70)

In a system that monitors the human eye:
A device configured to be worn on a human head;
A camera mounted on the device and arranged to view a first eye of a person wearing the device;
A plurality of light sources on the device that selectively illuminate each focal region of the person's face within the field of view of the camera around the first eye;
A controller connected to the camera and the light source, wherein the camera is used to sample the brightness of each focal region of the light source, and at least partially modulate the light source based on the sampled brightness to A controller configured to provide a desired brightness level within each focal region;
A system characterized by comprising.   The system of claim 1, wherein the controller is loaded into the device.   The system of claim 1, wherein the controller is located remotely from the device.   The system of claim 1, wherein the controller samples brightness from a plurality of pixels of the camera corresponding to a first focal region in the field of view of the camera that is illuminated with a first light source. And the controller combines the sampled luminances to determine an average luminance provided by the first light source, and the controller modulates the first light source to modulate the first focal region. Providing a desired brightness level within the system.   5. The system of claim 4, wherein the controller is configured to amplitude modulate at least one of a current and a voltage to the first light source to provide a desired brightness level. system.   5. The system of claim 4, wherein the controller is configured to pulse width modulate at least one of a current and a voltage to the first light source to provide a desired brightness level. System.   The system according to claim 1, wherein the controller is configured to activate the light source and acquire an image of the first eye only while the camera is operating. system.   The system of claim 1, wherein the desired brightness level of the light source is substantially uniform.   The system of claim 1, wherein the light source comprises an LED.   The system of claim 1, wherein the camera comprises one of a CCD and a CMOS detector.   The system according to claim 1, further comprising a processing unit connected to the camera, wherein the system receives a first eye image from the camera. In a feedback control system with spatially controlled lighting:
A camera for measuring scene brightness in two or more spatial regions of the field of view;
A light source that selectively illuminates a corresponding region in the field of view of the camera;
A processor that calculates an average luminance in each region within the field of view of the camera and modulates the corresponding light source to one or more target luminance levels to provide a desired luminance level within the field of view;
A system characterized by comprising.   13. The system according to claim 12, wherein the camera and the light source are loaded into a device configured to be worn on a person's head, and the camera has a first eye of a person wearing the device. Positioned on the device for viewing, the light source being around the first eye and selectively illuminating each focal region of the person's face within the field of view of the camera A system characterized by being arranged.   14. The system of claim 13, wherein the processor is located remotely from the device.   13. The system of claim 12, wherein the processor samples luminance from a plurality of pixels of the camera corresponding to a first focal region in the field of view of the camera that is illuminated with a first light source. And the processor combines the sampled luminances to determine an average luminance provided by the first light source, and the processor modulates the first light source to modulate the first focal region. Providing a desired brightness level within the system.   13. The system of claim 12, wherein the processor is configured to amplitude modulate at least one of a current and a voltage to the light source to provide a desired brightness level. .   13. The system of claim 12, wherein the processor is configured to pulse width modulate at least one of a current and a voltage to the light source to provide a desired brightness level. system.   13. The system of claim 12, wherein the processor is connected to the light source and configured to activate the light source and acquire the first eye image only while the camera is operating. System characterized by that.   19. The system of claim 18, wherein the processor is configured to pulse width modulate at least one of the current and voltage to the light source to provide a desired brightness level during the camera operation. System characterized by that.   13. The system of claim 12, wherein the processor is connected to the light source and does not activate the light source when the camera is not running.   13. The system according to claim 12, wherein the desired brightness level of the light source is substantially uniform.   The system of claim 12, wherein a desired brightness level of the light source is variable based at least in part on the position of each focal region within the field of view.   The system of claim 12, wherein the light source comprises an LED.   The system of claim 12, wherein the camera comprises one of a CCD and a CMOS detector.   13. The system of claim 12, further comprising an electronic display, wherein the camera and light source are mounted relative to the display such that the camera and light source face the face of the person viewing the electronic display. A system characterized by that.   26. The system of claim 25, wherein the camera and light source are mounted on the display.   13. The system according to claim 12, wherein the camera is mounted on a dashboard of the vehicle so that the camera faces the face of the operator of the vehicle.   28. The system of claim 27, wherein the light source is mounted on a vehicle such that the light source faces the operator's face to selectively select each focal region of the operator's face within the field of view of the camera. A system characterized by lighting.   13. The system of claim 12, wherein the camera is loaded on a structure near an assembly line or conveyor belt and acquires an image of an object in a direction along the assembly line or conveyor belt. System.   13. The system according to claim 12, wherein the camera is mounted on a plurality of structures in the vicinity of a storage area, and acquires an image of an object located in the storage area.   31. The system of claim 30, wherein the light source is loaded near the storage area and selectively illuminates each focal area of the storage area within the field of view of the camera.   32. The system of claim 31, wherein the processor is connected to the light source and modulates the light source based on an average brightness in each focal region to provide a substantially uniform brightness level in the field of view. Feature system.   32. The system of claim 31, wherein the processor is connected to the light source and modulates the light source based on an average brightness in each focal region to provide a variable brightness level in the field of view. System.   34. The system of claim 33, wherein the processor is connected to the light source to modulate the light source and provide higher brightness in a focal region that is further from the camera than other focal regions. . In a method for controlling illumination of a person's first eye:
Positioning the camera toward the person's face so that the person's first eye is within the field of view of the camera;
Illuminating at least a portion of the person's face with a plurality of light sources around the first eye and selectively illuminating each focal region of the person's face within the field of view of the camera;
Sampling the brightness of each focal region of the light source using the camera;
Modulating the light source based at least in part on the sampled luminance to provide a desired luminance level in each focal region;
A method characterized by comprising.   36. The method of claim 35, wherein the camera is loaded into a device configured to be worn on a person's head, the camera placing the device on the person's head to place the person's face. A method characterized by being positioned so as to face.   36. The method of claim 35, wherein the controller is connected to the camera to sample the brightness of each focal region, the controller is connected to the light source, and based at least in part on the sampled brightness. A method comprising modulating a light source.   38. The method of claim 37, wherein the camera is mounted on a device configured to be worn on a human head and the controller is located remotely from the device. 36. The method of claim 35, wherein the luminance sampling step:
Sampling luminance from a plurality of pixels of the camera corresponding to a first focal region in the field of view of the camera illuminated by a first light source;
Determining an average brightness provided by the first light source;
With
Modulating the light source modulates the first light source to provide a desired brightness level in the first focal region based at least in part on the determined average brightness; how to.   36. The method of claim 35, wherein the step of modulating the light source modulates an amplitude of at least one of a current and a voltage to the light source to provide a desired brightness level in each focal region. And how to.   36. The method of claim 35, wherein modulating the light source comprises modulating a pulse width of at least one of current and voltage to the light source to provide a desired brightness level in each focal region. Feature method.   36. The method of claim 35, wherein the light source is modulated to provide a substantially uniform brightness within the field of view of the camera.   36. The method of claim 35, further comprising monitoring the image of the first eye using the camera. 44. The method of claim 43, wherein the first eye image is monitored:
a) measuring the sleepiness level of the person,
b) at least partially controlling the electronic device;
c) providing biological self-control from said person;
Performing at least one of the following: In a method for controlling the lighting of a scene:
Directing the camera to the scene so that one or more objects of the scene are within the field of view of the camera;
Illuminating the scene with a plurality of light sources that selectively illuminate each focal region within the field of view of the camera;
Sampling the brightness of each focal region of the light source using the camera;
Modulating the light source based at least in part on the sampled luminance to provide a desired luminance level in each focal region;
A method characterized by comprising.   46. The method of claim 45, wherein the scene includes at least a portion of a person's face, and the one or more objects comprise at least one eye of the person.   46. The method of claim 45, wherein the camera is loaded into a device on the person's head.   46. The method of claim 45, wherein the plurality of light sources are loaded into the human head device.   46. The method of claim 45, wherein the camera is loaded at a location remote from the person's head.   46. The method of claim 45, wherein the plurality of light sources are loaded at a location remote from the person's head.   46. The method of claim 45, wherein the luminance in each focal region is sampled using a controller connected to the camera, and the controller is connected to the light source and is at least partially in the sampled luminance. Modulating the light source based on the method. 46. The method of claim 45, wherein the luminance sampling step:
Sampling the luminance from a plurality of pixels of the camera corresponding to a first focal region in the field of view of the camera illuminated by a first light source;
Determining an average brightness provided by the first light source from brightness sampled from the plurality of pixels;
With
Modulating the light source modulates the first light source to provide a desired brightness level in the first focal region based at least in part on the determined average brightness. Method. 53. The method of claim 52, further comprising the step of sampling the luminance:
Sampling the luminance from a plurality of pixels of the camera corresponding to a second focal region in the field of view of the camera illuminated by a second light source;
Determining an average brightness provided by the second light source;
With
The step of modulating the light source modulates the second light source to provide a desired brightness level in the second focal region based at least in part on the determined average brightness. Method.   46. The method of claim 45, wherein the step of modulating the light source alters an amplitude of at least one of a current and a voltage to the light source to provide a desired brightness level in each focal region. And how to.   46. The method of claim 45, wherein modulating the light source alters a pulse width of at least one of current and voltage to the light source to provide a desired brightness level in each focal region. Feature method.   46. The method of claim 45, wherein the light source is modulated to provide a substantially uniform brightness within the field of view of the camera.   46. The method of claim 45, wherein the light source is modulated to provide a predetermined brightness that varies in different portions within the field of view of the camera.   46. The method of claim 45, wherein the camera and light source are loaded with respect to the electronic display so that they face the face of a person looking at the electronic display, and the light source is modulated to be within the field of view of the camera. Providing a desired luminance level for each focal region of the person's face.   46. The method of claim 45, wherein the camera is mounted on a dashboard of the vehicle so as to face the operator of the vehicle.   60. The method of claim 59, wherein the light source is mounted on the vehicle to face the operator's face to selectively illuminate each focal region of the operator's face within the camera's field of view. Feature method.   46. The method of claim 45, wherein the camera is directed to an assembly line or conveyor belt and obtains an image of an object oriented along the assembly line or conveyor belt.   46. The method of claim 45, wherein the camera is oriented in a plurality of storage areas and acquires an image of an object located in the storage areas.   63. The method of claim 62, wherein the light source is directed toward the storage area and selectively illuminates each focal area of the storage area within the field of view of the camera.   64. The method of claim 63, wherein the light source is modulated to provide a substantially uniform brightness within the field of view.   64. The method of claim 63, wherein the light source is modulated to provide a variable brightness level within the field of view.   66. The method of claim 65, wherein the light source is modulated to provide a higher brightness level for a focal region that is further from the camera than other focal regions.   46. The method of claim 45, further comprising obtaining an image using the camera, wherein the brightness of each focal region of the light source is sampled from the image obtained using the camera.   68. The method of claim 67, wherein the light source is switched off when the camera is not acquiring an image.   46. The method of claim 45, further comprising the step of acquiring an image using the camera, wherein the brightness of each focal region of the light source is sampled separately from the image acquired using the camera. .   70. The method of claim 69, wherein the light source is active only while the camera is acquiring images and only when sampling the brightness of each focal region of the light source.

Images