JP4913356B2 - Visual information processing apparatus and visual information processing method thereof - Google Patents

Description

  The present invention relates to a visual information processing apparatus and a visual information processing method thereof, and more particularly to a visual information processing apparatus using a spatial light modulation element and a high-speed imaging element and a visual information processing method thereof.

  Today, imaging devices and cameras using imagers (solid-state imaging devices) such as CMOS image sensors are spreading their application fields in various forms regardless of consumer use or industrial use.

  In particular, in the field of industrial imaging devices, for example, imaging devices capable of simultaneously achieving ultra-high speed of 1000 frames per second and high resolution have been realized and used as sensors for various image analysis and control systems. Yes.

  On the other hand, recently, a device called a light spatial modulation element has been developed and put into practical use and is widely used in a projector apparatus or the like (for example, Patent Document 1). There are two types of spatial light modulators: a transmissive type such as a liquid crystal light valve, and a reflective type called DMD (Digital Micro-mirror Device).

  In a projector apparatus using a transmissive light spatial modulation element, an image signal is applied to a liquid crystal light valve, and light from a light source is transmitted through the liquid crystal light valve to project an image on a front screen or the like.

On the other hand, a projector apparatus using a reflective spatial modulation element projects an image on a front screen or the like after reflecting light from a light source to a DMD. The DMD has a large number of minute mirrors corresponding to the number of pixels. By changing the angle of the mirror for each pixel, the amount of light projected on the screen is turned on and off for each pixel, and a two-dimensional image is displayed. To form. Although the change in the mirror angle changes in a binary manner depending on the on / off signal, the continuous light intensity is expressed for the human eye by setting the on / off period by pulse width modulation. Can do. Further, by changing the duty ratio of the pulse width modulation for each pixel and each frame, a continuous moving image can be formed both spatially and temporally.
JP 2003-114479 A

  Conventionally, light spatial modulation elements such as liquid crystal light valves and DMDs are often used exclusively for projector apparatuses. With the development of high-brightness and high-speed system technology, it is not only used as a projector for presentations, but is also widely used as a projector for large-sized TVs or theater movies. Is used as an image forming element of a projector apparatus.

  The visual information processing apparatus according to the present invention does not use these light spatial modulation elements as conventional image forming elements.

  Image formation (imaging) itself is performed by an imager (solid-state imaging device) such as a CMOS image sensor. At the same time, spatial and temporal modulation is performed on the light from the object incident on the imager using a spatial light modulator such as a liquid crystal light valve or DMD.

  An object of the visual information processing apparatus according to the present invention is to realize new visual information processing that has not been obtained conventionally by combining a spatial light modulator and an imager.

  Specifically, an object of the present invention is to provide a visual information processing apparatus capable of realizing high resolution for a specific fixed or moving object by combining a spatial light modulator and an imager, and the visual information thereof It is to provide a processing method.

  Another object of the present invention is to provide a visual information processing apparatus capable of extracting a predetermined luminance variation for each pixel and generating a luminance variation spectrum image by combining an optical spatial modulation element and an imager, and the visual information thereof. It is to provide an information processing method.

  Furthermore, another object of the present invention is to provide visual information that can perform spatial filter processing such as edge enhancement on an entire image or a specific fixed or moving object by combining a spatial light modulator and an imager. It is to provide a processing device and a visual information processing method thereof.

  Another object of the present invention is to provide a visual information processing that can secure an appropriate brightness dynamic range even for an object having a very large difference in brightness by combining a spatial light modulator and an imager. An apparatus and a visual information processing method thereof are provided.

In the visual information processing device according to the present invention, all the pixel regions are grouped into a plurality, and only one pixel among the plurality of pixel signals in each group is sequentially switched on to reflect incident light from the imaging target or Light spatial modulation element that transmits and outputs, Imager that picks up incident light from the imaging object output from the light spatial modulation element with a resolution of the same number of pixels as the number of groups, and switching of the one pixel The pixel signals corresponding to the number of groups, which are sequentially input from each group of the spatial light modulators to the imager, are combined to increase the resolution of the imager to the same resolution as the spatial light modulators. And a resolution processing unit.

  According to the visual information processing apparatus and the visual information processing method thereof according to the present invention, new visual information processing that has not been obtained in the past can be realized by combining a spatial light modulator and an imager.

  An embodiment of a visual information processing apparatus and visual information processing method according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a diagram showing the relationship between an optical spatial modulation element and an imager that are commonly used in a visual information processing apparatus according to an embodiment of the present invention. In the embodiment of the present invention, the spatial light modulator 10 is used as a form for spatially modulating incident light from the imaging object 100 as shown in FIG. In this case, the spatial light modulator 10 can take two forms, a reflective type and a transmissive type. FIG. 1A shows a configuration using a reflective spatial light modulator 10a, for example, a DMD. FIG. 1B shows a form using a transmissive spatial light modulator 10b, for example, a transmissive liquid crystal light valve. In any form, incident light from the imaging object 100 is modulated (spatial modulation) for each pixel by the spatial light modulation elements 10a and 10b, and then output to the imager 20.

  In the following description, a transmissive spatial light modulator is used in the drawings, but this is for avoiding duplication of description, and does not exclude a reflective spatial light modulator.

(1) 1st Embodiment FIG. 2: is the figure which showed the structural example of the visual information processing apparatus 1 which concerns on 1st Embodiment.

  The visual information processing apparatus 1 according to the first embodiment is configured to perform processing for improving the resolution of the imager 20 by combining the imager 20 and the spatial light modulator 10.

  The visual information processing apparatus 1 includes a light spatial modulation element 10, an image feature extraction unit 2b, and a high resolution processing unit 30 (image data processing unit).

  The spatial light modulator 10 according to the present embodiment sequentially switches pixels that reflect or transmit incident light within a predetermined group based on a control signal from the high resolution processing unit 30 and outputs the pixels to the imager 20. .

  FIG. 3 is a diagram illustrating an operation concept of high-resolution processing by the visual information processing apparatus 1.

  In the visual information processing apparatus 1 according to the first embodiment, the resolution of the spatial light modulator 10 is configured to be higher than the resolution of the imager 20. In the example of FIG. 3, the spatial light modulator 10 has 16 × 16 (256) pixels, the imager 20 has 4 × 4 (16) pixels, and the spatial light modulator 10 has a resolution 16 times that of the imager 20. is doing.

  For example, when imaging an object in a dark place without a light source or the like, a more sensitive imager 20 is required. In general, increasing the pixel area is the most effective for improving the sensitivity of the imager 20, and many high-sensitivity imagers have a larger pixel area than a normal imager. For this reason, in general, a high-sensitivity imager is limited in the number of pixels in a predetermined area, and the resolution is often lower than that of a normal imager.

  In the visual information processing apparatus 1 according to the first embodiment, even such a high-sensitivity and low-resolution imager is realized by combining with the light spatial modulation element 10.

  The resolution of the imager 20 illustrated in FIG. 3 is 1/16 compared with the resolution of the spatial light modulator 10. That is, in this example, 16 pixels of the spatial light modulator 10 correspond to one pixel of the imager 20.

  First, the pixels of the spatial light modulation element 10 are grouped every 16 (4 × 4) elements. Then, only one pixel in the group is sequentially turned on. In the example of FIG. 10, the pixel is sequentially switched (scanned) and turned on from the upper left pixel in the group of (4 × 4) elements. For other groups, the pixels are turned on in the same switching sequence.

  With such a pixel switching method, the number of pixels corresponding to the number of groups is output from each group every time one pixel is switched. In the example of FIG. 3, one pixel from each of the 16 groups of the spatial light modulators 10 is turned on, and a total of 16 pixels are output to the imager 20. The imager 20 first generates one piece of low-resolution image data using these 16 pixels. In the spatial light modulator 10, the pixels in the group are sequentially switched and turned on, and the imager 20 sequentially generates low-resolution image data. When switching of all the pixels in the group is completed, the imager 20 generates 16 low-resolution images.

  Next, by combining these 16 pieces of low resolution image data, one piece of high resolution image data is generated. Since each of the 16 pieces of low-resolution image data and the pixel positions in the group have a one-to-one correspondence, it is possible to generate one piece of high-resolution image data by combining the 16 pieces of low-resolution image data. . In this example, the resolution can be increased to 16 times the resolution of the imager 20.

  In the example of FIG. 3, a mode is shown in which 16 pixels in the group are turned on one by one. On the other hand, for example, 4 (2 × 2) pixels of 16 pixels may be turned on. In this case, the imager 20 generates four pieces of low-resolution image data. By combining these four pieces, it is possible to generate image data with a resolution four times higher. Compared to the previous example, the resolution of the image data after synthesis is 1/4, but the number of image data to be synthesized is also 1/4, and the processing time is shortened.

  Thus, in the visual information processing apparatus 1 according to this embodiment, the resolution of the imager 20 can be improved, and the resolution to be improved can be easily changed as described above.

(2) Second Embodiment FIG. 4 is a diagram illustrating a configuration example of a visual information processing device 1a according to a second embodiment.

  The visual information processing device 1a according to the second embodiment combines the imager 20 and the spatial light modulation element 10 to extract a spectrum component obtained by extracting frequency components such as luminance fluctuations of the imaging target for each pixel and imaging the spectrum image. It is configured so that it can be generated.

  The visual information processing device 1a includes a light spatial modulation element 10, an imager 20, and a spectrum image generation unit 40 (image data processing unit).

  The spatial light modulator 10 according to this embodiment is turned on / off by a predetermined periodic signal for each pixel based on a control signal from the spectrum image generation unit 40, and reflects or transmits incident light from the imaging target 100. Output to the imager 20.

  FIG. 5 is a diagram showing an operation concept of spectrum image generation processing by the visual information processing apparatus 1a.

  In the second embodiment, the spatial light modulator 10 and the imager 20 are basically configured with the same number of pixels.

  The spatial light modulator 10 is divided into a plurality of groups having a plurality of pixels. In the example of FIG. 5, grouping is performed every 4 (2 × 2) pixels.

  An on / off control signal having a different frequency is applied to each of the four pixels in the group. For example, a control signal for switching on / off at frequencies 1 to 4 is applied to the pixels 1 to 4, respectively.

  When the luminance of the imaging object fluctuates at a predetermined frequency, if this fluctuation frequency matches the frequency of the on / off control signal applied to the spatial light modulator 10, the imager 20 holds the luminance of the corresponding pixel. Is done. On the other hand, when the frequency of the luminance fluctuation of the imaging object and the frequency of the on / off control signal applied to the spatial light modulator 10 do not match, the imager 20 suppresses the luminance of the corresponding pixel. That is, since the light spatial modulation element 10 functions as a correlator and the integrator is configured by the capacitance component included in the imager 20, it is possible to extract the fluctuation frequency of the imaging object for each pixel.

  For example, when a fluorescent lamp is lit with a commercial power supply of 50 Hz in a complicated background, in the visual information processing apparatus 1a according to the present embodiment, by matching one of the on / off control frequencies with 50 Hz, A spectrum image in which only the fluorescent lamp portion is emphasized can be generated.

  The on / off control frequency within each group can be arbitrarily set, and can be set as appropriate according to the physical condition of the imaging object and the frequency to be measured.

  For example, when detecting a crack or the like of a structure, if it is known in advance that the crack or the like fluctuates at a specific resonance frequency, the crack or the like can be obtained by matching one of the on / off control frequencies with the resonance frequency. It is possible to generate a spectrum image corresponding to the position. As a result, even a minute crack or the like that is difficult to visually recognize can be visualized by extracting the resonance frequency and imaging it.

  Conversely, even when the resonance frequency of the observation object is unknown, if the frequency of a specific part in the image is different from the frequency in the surrounding range, an image (extracted from the specific part) (Spectrum image) can also be generated.

  For example, a printed wiring board is vibrated at a known frequency, and a part having a frequency component different from the vibration frequency is extracted as an image, so that it is possible to detect a soldering failure portion of a mounted component as image information. .

(3) Third Embodiment FIG. 6 is a diagram illustrating a configuration example of a visual information processing device 1b according to a third embodiment. The visual information processing device 1b according to the third embodiment is configured such that spatial filtering processing such as edge enhancement can be realized by simple arithmetic processing by combining the imager 20 and the spatial light modulator 10. .

  The visual information processing apparatus 1b includes a light spatial modulation element 10, an imager 20, and a spatial filtering processing unit 50 (image data processing unit).

  The spatial light modulation element 10 according to the present embodiment is turned on / off for each pixel based on a control signal from the spatial filtering processing unit 50, and reflects or transmits incident light from the imaging object 100 and outputs the reflected light to the imager 20. To do.

  FIG. 7 is a diagram showing an operation concept of the spatial filtering process by the visual information processing apparatus 1b.

  In the third embodiment, the number of pixels of the spatial light modulator 10 is configured to be larger than the number of pixels of the imager 20. In the example of FIG. 7, the number of pixels of the spatial light modulator 10 is configured to be four times the number of pixels of the imager 20.

  The spatial light modulator 10 is divided into a plurality of groups having a plurality of pixels. In the example of FIG. 7, grouping is performed every 4 (2 × 2) pixels.

  For the four pixels in the group, an on / off control signal is applied to the corresponding pixel based on a predetermined spatial pattern. For example, as illustrated in FIG. 7, in the spatial pattern 1, all four pixels in the group are turned on. In the space pattern 2, two pixels in the right half are turned on and two pixels in the left half are turned off. On the other hand, in the space pattern 3, the two pixels in the left half are turned on and the two pixels in the right half are turned off.

  In the visual information processing device 1b according to the present embodiment, edges in the X direction and the Y direction are obtained only by subtraction from the image data of the imager 20 obtained when the light spatial modulation element 10 is set to the spatial patterns 1, 2, and 3. Spatial filtering processing for emphasis processing can be realized.

  Specifically, as shown in the equation (1) of FIG. 7, the image of the imager 20 obtained in the case of the space pattern 2, I (x, y, t + Δt), is obtained in the case of the space pattern 1. By subtracting 1/2 of I (x, y, t) of the image of the imager 20 to be generated, image data in which the edge in the X direction is emphasized can be generated. Strictly speaking, the image data obtained from the spatial pattern 1 and the image data obtained from the spatial pattern 2 are different in time, but by switching the pixels of the spatial light modulator 10 at high speed, the equation (1) in FIG. The arithmetic expression on the right side can be approximated on the left side.

  Similarly, as shown in the equation (2) of FIG. 7, the image of the imager 20 obtained at the time of the spatial pattern 3, I (x, y, t + 2Δt), is obtained at the time of the spatial pattern 1. By subtracting 1/2 of I (x, y, t) of the image of the imager 20, image data in which the edge in the Y direction is emphasized can be generated.

  According to the visual information processing device 1b according to the third embodiment, a plurality of spatial patterns are formed using the light spatial modulation element 10, and subtraction is performed on the image data of the imager 20 obtained according to each spatial pattern. The spatial filtering process can be performed by a simple operation such as the above.

  In addition, a spatial matching process or the like can be performed by generating an image that matches a wider area, such as a template image, as a spatial pattern by the light spatial modulation element 10. Also, not only an image prepared in advance but also a modulation pattern based on image information of the previous frame is prepared, so that processing such as block matching can be performed.

(4) Fourth Embodiment FIG. 8 is a diagram illustrating a configuration example of a visual information processing device 1c according to a fourth embodiment. The visual information processing apparatus 1c according to the fourth embodiment is configured to be able to control the exposure time of incident light from the reflecting object 100 for each pixel.

  The visual information processing apparatus 1c includes a light spatial modulation element 10, an imager 20, and a control unit 60 (image data processing unit).

  The spatial light modulator 10 controls the exposure time for each pixel based on a control signal from the controller 60 and outputs reflected light or transmitted light to the imager 20. In the case of a reflective spatial light modulator 10a, for example, DMD, the exposure time is controlled for each pixel by controlling the on-time, that is, the time when the reflective surface of the micromirror faces the imager 20 for each pixel. Can do. In the case of the transmissive spatial light modulator 10b, the exposure time can be equivalently controlled for each pixel by controlling the transmission amount for each pixel by controlling the voltage applied to the pixel.

  FIG. 9 is a diagram showing an operation concept of exposure time control by the visual information processing apparatus 1c. The imager 20 alone has a limit on the dynamic range with respect to brightness due to problems such as element accuracy, and it is generally difficult to capture an object with extremely high luminance and an object with very low luminance simultaneously with a clear image. . Although it is possible in principle to expand the dynamic range by providing a sensitivity control circuit for each pixel of the imager 20, the number of pixels and the pixel area are reduced due to restrictions in mounting the sensitivity control circuit, The performance of the original imager such as resolution and sensitivity is sacrificed.

  On the other hand, in the visual information processing apparatus 1c according to the present embodiment, the exposure time is controlled by the light spatial modulation element 10, and reflected light or transmitted light whose exposure time is controlled is output to the conventional imager 20. It is. Luminance information of the image data captured by the imager 20 is extracted for each pixel by the control unit 60 and fed back to the spatial light modulator 10. If the brightness of a specific area of the image data captured by the imager 20 is too high, the exposure time of the pixels in that area is decreased. Conversely, if the brightness of another specific area is too low, the image exposure in that area is exposed. Increase time. In this way, by controlling the luminance of the image observed by the imager 20 in units of pixels, a substantially wide dynamic range can be realized.

  In this embodiment, since the imager 20 itself can be used as it is, the resolution and sensitivity inherent in the imager 20 are not sacrificed.

  As a modified example of the visual information processing apparatus 1c according to the fourth embodiment, a visual information processing apparatus 1c having an automatic spatial sensitivity correction function can be provided.

  In general, the imager 20 does not necessarily have completely uniform sensitivity over the entire range of the pixel region, and there is a spatial difference in sensitivity. Similarly, color filter arrays such as RGB and CMY used for color imaging also have spatial sensitivity (transmittance) non-uniformity.

  In the visual information processing apparatus 1c according to the present embodiment, since the exposure time for each pixel can be controlled by the light spatial modulation element 10, the spatial nonuniformity of the imager 20 and the color filter array can be easily corrected. can do.

  Further, as another modified example of the visual information processing apparatus 1c according to the fourth embodiment, it is assumed that the luminance non-uniformity due to the deviation of the position of the illumination light that occurs in the case of indoor imaging or the like is corrected. You can also. When the luminance gradually decreases from a region close to the illumination light toward a region far from the illumination light, it is necessary to correct the luminance to be uniform especially when generating a binary image. In the visual information processing apparatus 1c according to the present embodiment, the luminance information of the imager 20 is monitored, and the exposure time of the light spatial modulation element 10 is controlled, so that an image having uniform luminance over the entire pixel regardless of the position of the illumination light. Data can be generated.

  According to the visual information processing apparatus and the visual information processing according to each of the above-described embodiments, a new visual information processing that has not been obtained in the past can be performed by combining an optical spatial modulation element and an imager (solid-state imaging element). Can be realized.

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined.

The figure which shows the relationship between the light spatial modulation element and imager in the 1st thru | or 4th embodiment of the visual information processing apparatus which concerns on this invention. The figure which shows the system configuration example of 1st Embodiment of the visual information processing apparatus which concerns on this invention. The figure explaining the operation | movement concept of the high resolution process in 1st Embodiment of the visual information processing apparatus which concerns on this invention. The figure which shows the system configuration example of 2nd Embodiment of the visual information processing apparatus which concerns on this invention. The figure explaining the operation | movement concept of the spectrum image generation process in 2nd Embodiment of the visual information processing apparatus which concerns on this invention. The figure which shows the system configuration example of 3rd Embodiment of the visual information processing apparatus which concerns on this invention. The figure explaining the operation | movement concept of the spatial filtering process in 3rd Embodiment of the visual information processing apparatus which concerns on this invention. The figure which shows the system configuration example of 4th Embodiment of the visual information processing apparatus which concerns on this invention. The figure explaining the operation | movement concept of exposure time control in 4th Embodiment of the visual information processing apparatus which concerns on this invention.

Explanation of symbols

1, 1a, 1b, 1c Visual information processing apparatus 10 Light spatial modulation element 10a Light spatial modulation element (reflection type)
10b Light spatial modulation element (transmission type)
20 Imager 30 High Resolution Processing Unit 40 Spectrum Image Generation Unit 50 Spatial Filtering Processing Unit 60 Control Unit

Claims (8)

An optical spatial modulation element in which all pixel regions are grouped into a plurality, and only one pixel among a plurality of pixel signals in each group is sequentially switched on to reflect or transmit incident light from an imaging object; ,
An imager for imaging incident light from the imaging object output from the spatial light modulator with a resolution of the same number of pixels as the number of groups;
The number of pixel signals corresponding to the number of groups, which are sequentially input from each group of the spatial light modulators to the imager every time the one pixel is switched, is combined, and the resolution of the imager is set to the spatial light modulator. A high-resolution processor that increases the same resolution,
A visual information processing apparatus comprising: The light spatial modulation element further turns on and off the plurality of pixel signals in each group at different frequencies to reflect or transmit incident light from the imaging object, and outputs the reflected light.
When the light spatial modulation element turns on and off a plurality of pixel signals in each group at different frequencies, the imager outputs incident light from the imaging object output from the light spatial modulation element, Capture image with the same resolution as that of the spatial light modulation element to generate image data,
A spectrum image generation unit that generates a spectrum image corresponding to the different frequency by frequency-resolving the luminance variation of the imaging object corresponding to the different frequency,
The visual information processing apparatus according to claim 1, further comprising: The light spatial modulation element further sequentially switches a plurality of pixel signals in each group based on a predetermined plurality of spatial patterns and reflects or transmits incident light from the imaging target,
When the light spatial modulation element sequentially switches and outputs a plurality of pixel signals in each group based on a predetermined plurality of spatial patterns, the imager outputs the imaging object output from the light spatial modulation element The incident light from is imaged with the same number of pixels as the number of groups to generate image data,
A spatial filter unit that calculates a plurality of signals sequentially output from an imager based on the plurality of spatial patterns, and performs spatial filtering using a plurality of pixels in the group as processing units;
The visual information processing apparatus according to claim 1, further comprising: The visual information processing apparatus according to claim 1, wherein the light spatial modulation element is a digital micromirror device (DMD) or a liquid crystal light valve. A spatial light modulation element in which all pixel areas are grouped into a plurality of groups, and sequentially switching only one pixel among a plurality of pixel signals in each group to turn on and reflect or transmit incident light from the imaging object. And steps to
Imaging incident light from the imaging object output from the spatial light modulator with an imager having a resolution equal to the number of pixels as the number of groups;
The number of pixel signals corresponding to the number of groups, which are sequentially input from each group of the spatial light modulators to the imager every time the one pixel is switched, is combined, and the resolution of the imager is set to the spatial light modulator. Step to the same resolution,
A visual information processing method characterized by comprising: The light spatial modulation element further turns on and off the plurality of pixel signals in each group at different frequencies to reflect or transmit incident light from the imaging object, and outputs the reflected light.
When the light spatial modulation element turns on and off a plurality of pixel signals in each group at different frequencies, the imager outputs incident light from the imaging object output from the light spatial modulation element, Imaging with an imager having the same resolution as that of the light spatial modulation element to generate image data,
The frequency variation of the luminance variation of the imaging object corresponding to the different frequency is generated, and a spectrum image corresponding to the different frequency is generated.
The visual information processing method according to claim 5, further comprising a step. The light spatial modulation element further sequentially switches a plurality of pixel signals in each group based on a predetermined plurality of spatial patterns and reflects or transmits incident light from the imaging target,
When the light spatial modulation element sequentially switches and outputs a plurality of pixel signals in each group based on a predetermined plurality of spatial patterns, the imager outputs the imaging object output from the light spatial modulation element The incident light from is captured by an imager having the same number of pixels as the number of groups to generate image data,
Calculating a plurality of signals sequentially output from the imager based on the plurality of spatial patterns, and performing spatial filtering using a plurality of pixels in the group as processing units;
The visual information processing method according to claim 5, further comprising a step. 8. The visual information processing method according to claim 5, wherein the spatial light modulator is a digital micromirror device (DMD) or a liquid crystal light valve.

Images